KR102246418B1 - Lane keeping method and apparatus thereof - Google Patents

Abstract

Disclosed are a lane keeping control method and apparatus therefor. The lane keeping control device includes: a camera that acquires a front image in real time; A prediction unit for predicting _{road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) by applying the front image to a pre-learned road curve coefficient regression model unit; And a lateral controller for deriving a control input of the vehicle by applying it to a behavior model that simulates the behavior of the vehicle based on the predicted road curve coefficients (c ₀ , c ₁ , c ₂ , c _{3 ).}

Description

Lane keeping control method and apparatus thereof TECHNICAL FIELD

The present invention relates to a lane keeping control method and apparatus using a forward image.

Recently, research on the advanced driver assistant system (ADAS) is being actively researched for driver convenience and active safety. In particular, the lane keeping system (LKS), which autonomously maintains the lane by controlling the vehicle's steering system, depends on the coefficients calculated and obtained from the vision sensor. Currently, LKS cannot provide reliable road coefficients in complex driving situations such as urban areas excluding highways, and artificial intelligence-based research is being conducted to solve this problem.

(01) Korean Registered Patent Publication No. 10-1848198 (2018.04.05.)

An object of the present invention is to provide a lane maintenance control method and apparatus capable of enabling lane maintenance control by providing a highly reliable road coefficient without a complex and classical image pre-processing process using neural network learning for a forward image.

In addition, the present invention is to provide a lane maintenance control method and apparatus capable of reliable lane maintenance control in a robust and complex situation through virtual road curve coefficient estimation even when lane maintenance is impossible based on an existing camera sensor.

In addition, the present invention is to provide a lane maintenance control method and apparatus for controlling lane maintenance by estimating a curve coefficient of a road model even on a road in which lane information does not exist.

According to an aspect of the present invention, an apparatus for controlling lane maintenance using an image is provided.

According to an embodiment of the present invention, a camera for acquiring a front image in real time; A prediction unit for predicting _{road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) by applying the front image to a pre-learned road curve coefficient regression model unit; And a lateral controller for deriving a control input of the vehicle by applying it to a behavior model that simulates the lateral behavior of the vehicle based on the predicted road curve coefficients (c ₀ , c ₁ , c ₂ , c _{3 ).} A maintenance control device may be provided.

The road curve coefficient regression model unit classifies and predicts the road curve coefficient according to an attribute, but _{a first group of the road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) is a coefficient related to a control error (c ₀ , c ₁ ), and the second group may be coefficients (c ₂ , c ₃ ) related to the shape of the road lane model.

The vehicle behavior model is a lateral model, and the vehicle motion state information includes a lateral position offset at a front distance, a lateral velocity offset at the vehicle center of gravity, a yaw angle offset at the vehicle center of gravity, and It may include the yaw angular velocity at the vehicle's center of gravity.

The control input may be a steering angle.

According to another embodiment of the present invention, a method for controlling lane maintenance using an image is provided.

According to an embodiment of the present invention, (a) obtaining a front image in real time; _{(b) predicting road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) by applying the front image to a previously learned road curve coefficient regression model unit; And (c) applying the predicted road curve coefficients (c ₀ , c ₁ , c ₂ , c ₃ ) to a behavior model that simulates the lateral behavior of the vehicle to derive a control input of the vehicle. A lane keeping control method may be provided.

In the step (b), the road curve coefficient is classified and predicted according to attributes using one front image, but the first group of _{the road curve coefficients (c 0} , c ₁ , c ₂ , c _{3) is controlled.} These are coefficients related to the error (c ₀ , c ₁ ), and the second group may be coefficients (c ₂ , c ₃ ) related to the shape of the road lane model.

By providing a lane maintenance control method and an apparatus therefor according to an embodiment of the present invention, it is possible to control lane maintenance by providing reliable road coefficients without a complex and classical image pre-processing process using neural network learning for a front image. There is an advantage to be able to do.

In addition, the present invention has the advantage of enabling reliable lane maintenance control in a robust and complex situation through virtual road curve coefficient estimation even when lane maintenance is impossible based on an existing camera sensor.

In addition, the present invention has the advantage of enabling lane maintenance control through estimation of a curve coefficient of a road model even on a road in which lane information does not exist.

1 is a block diagram schematically showing the configuration of a lane keeping control apparatus according to an embodiment of the present invention.
2 is a diagram schematically showing the configuration of a road curve coefficient regression model unit according to an embodiment of the present invention.
3 is a diagram illustrating an ROI according to an embodiment of the present invention.
4 is a diagram illustrating a transverse model according to an embodiment of the present invention.
5 is a flow chart showing a lane maintenance control method according to an embodiment of the present invention.
Figure 6 is a graph showing to compare the control results of the present invention and the prior art.

Singular expressions used in the present specification include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various elements or various steps described in the specification, and some of the elements or some steps It may not be included, or it should be interpreted that it may further include additional components or steps. In addition, terms such as "... unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically showing the configuration of a lane keeping control apparatus according to an embodiment of the present invention, and FIG. 2 is a diagram schematically showing the configuration of a road curve coefficient regression model unit according to an embodiment of the present invention 3 is a diagram illustrating an ROI according to an exemplary embodiment of the present invention, and FIG. 4 is a diagram illustrating a transverse model according to an exemplary embodiment of the present invention.

Referring to FIG. 1, a lane maintenance control apparatus 100 according to an embodiment of the present invention includes a camera 110, a prediction unit 120, a memory 130, a state estimator 140, and a lateral controller 150. Consists of including.

The camera 110 is mounted on a vehicle and is a means for photographing a front image of the vehicle.

The prediction unit 120 is a means for predicting _{the road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) by applying the forward image to the road curve coefficient model unit.

The road curve coefficient model unit is an artificial intelligence model such as a CNN, and a pre-learning process may be preceded by a training set.

2 illustrates the configuration of the road curve coefficient model unit. 2 is a description based on the assumption that the road curve coefficient model unit according to an embodiment of the present invention is a CNN model, and thus, it is shown that it is composed of an input layer, a plurality of convolutional layers, a plurality of FC layers, and an output layer. However, it is natural that the composition may be different when implemented with other artificial intelligence models.

The road curve coefficient regression model unit according to an embodiment of the present invention includes a first model unit and a second model unit, respectively. That is, the road curve coefficient regression model unit according to an embodiment of the present invention may predict the road curve coefficients classified according to each attribute through a plurality of model units (first and second model units), respectively.

For example, the road curve coefficient regression model unit according to an embodiment of the present invention _{classifies four road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) into two groups according to attributes (or classification criteria). can do. For example, the attribute (or classification criterion) may be a coefficient for a control error and a coefficient for a shape of a road lane model.

When learning and predicting four road curve coefficients through one model unit, there is a problem that the prediction is relatively inaccurate. Accordingly, according to an embodiment of the present invention, it is possible to improve prediction accuracy by classifying road curve coefficients according to attributes (or classification criteria) and then learning and predicting each classified group unit through a separate model unit.

The four road curve coefficients c ₀ , c ₁ , c ₂ , and c ₃ represent the transverse center distance error, direction angle error, curvature, and curvature change, respectively. That is, _{it can be seen that among c 0} , c ₁ , c ₂ , and c ₃ , c ₀ and c ₁ are coefficients related to the error of the lateral behavior control. In addition, _{it can be seen that c 2} and c ₃ are coefficients related to the shape of the road lane model.

Accordingly, the road curve regression model unit according to an embodiment of the present invention includes a first model unit and a second model unit, respectively, and any one of the first model unit and the second model unit includes coefficients related to the control error (c ₀ , c ₁ ) is learned and predicted, and the other one of the first model unit and the second model unit may learn and predict _{coefficients (c 2} , c _{3) related to the shape of the road lane model.}

Each of the first model unit and the second model unit may precede the training process using a training set. Here, the training set may be a front image and a road curve coefficient. In other words, the first model unit performs a learning process using a training set including the _{front image and road curve coefficients (c 0} , c _{1) related to the control error, and the second model unit} The learning process may be performed using a training set including shape-related road curve coefficients (c ₂ , c _{3 ).}

The error functions of the first model part and the second model part can be expressed as in Equation 1.

이며, j는 영상 인덱스를 나타내며,

는 전체 영상 개수를 나타내고, T는 전치 행렬(matrix transpose)을 나타낸다. here,

And j represents the image index,

Denotes the total number of images, and T denotes a matrix transpose.

In addition, each normalization function of the first model part and the second model part can be expressed as Equation 2 below.

이며,

는 전체 훈련 가능한 엔트리 개수를 나타내고, k는 훈련 가능한 엔트리 인덱스를 나타내며,

는 가중치를 나타내며,

는 정규화 파라미터를 나타낸다. here,

Is,

Denotes the total number of trainable entries, k denotes the trainable entry index,

Represents the weight,

Denotes the normalization parameter.

In addition, the optimization function for adaptive prediction of the first model unit and the second mobile unit may be expressed as Equation 3.

이며,

이고,

이며,

과

는 각각 모멘트를 나타내고,

과

는 각각 0 ~ 1 사이의 수를 나타내며, 위첨자 't'는 시간 t에서의 값을 나타내며,

는 초기 학습률(0.001)을 나타내며,

는 매우 작은 양의 정수를 나타낸다. here,

Is,

ego,

Is,

and

Each represents a moment,

and

Represents a number between 0 and 1, respectively, the superscript't' represents the value at time t,

Represents the initial learning rate (0.001),

Represents a very small positive integer.

That is, each model unit (ie, a first model unit and a second model unit) may include an input layer, a plurality of convolution layers, a plurality of FC layers, and an output layer, respectively. The first model unit receives the front image as an input of the input layer, and matches and sets the _{road curve coefficients (i.e., c 0} , c _{1) to be learned from the corresponding front image as the output layer.} A training process for optimizing a plurality of FC layers can be repeated.

The second model unit receives the front image as an input of the input layer, and matches the _{road curve coefficients (i.e., c 2} , c _{3) to be learned from the corresponding front image as the output layer.} It is possible to repeat the training process to optimize the FC layer of.

In addition, in an embodiment of the present invention, all of the front images are not used as they are, and an ROI may be set to use an input image as an ROI (see FIG. 3).

As described above, the predictor 120 may predict road curve coefficients by applying the front image to the road curve regression model unit on which the training has been completed.

The memory 130 includes various instructions (program codes) necessary to perform a method of controlling lane maintenance using a road curve coefficient predicted using a forward image according to an embodiment of the present invention, and derived from this process. It is a means for storing various data, etc.

The state estimator 140 is a means for estimating the motion state information of the vehicle based on the predicted road curve coefficients (c ₀ , c ₁ , c ₂ , c _{3 ).} That is, the state estimator 140 may estimate vehicle motion state information using data having different periods. The camera 110 and the sensors inside the vehicle have different update cycles. Accordingly, the state estimator 140 may estimate the motion state information of the vehicle using data of different periods as described above.

Although not shown in detail in FIG. 1, the state estimator 140 according to an embodiment of the present invention may acquire data from various sensors mounted in a vehicle or acquire data of each sensor from an ECU of a vehicle.

The transverse controller 150 is a means for deriving a control input (eg, steering angle) using the predicted road curve coefficient. That is, the lateral controller 150 may derive a lateral control input of the vehicle by using the state information and reference information estimated by the state estimator 140.

The clothoid road curve model can be expressed as Equation 1. That is, the road curve model can be expressed in the form of a third-order polynomial obtained after image processing for a lane marker. Assuming that the arc length is s, the clothoid curve model can be expressed as in Equation 4.

Here, c ₀ , c ₁ , c ₂ , and c ₃ denote a transverse center distance error, a direction angle error, a curvature, and a change in curvature, respectively.

In an embodiment of the present invention, the above-described road curve coefficients may be predicted by applying a forward image to the above-described road curve regression model unit instead of using the road curve model.

_{Using the road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) predicted by using a single front image, the state estimator 140 derives the lateral motion state information of the vehicle by applying it to the lateral model. By doing so, you can control lane keeping. Here, in deriving the lateral motion state information of the vehicle, a control input (eg, steering angle) input from the lateral controller 150 may be further used.

4 is a diagram illustrating a transverse model according to an embodiment of the present invention. In FIG. 4, L is a look-ahead, and represents a distance viewed from the center of gravity of the vehicle in the longitudinal direction. When the translational and rotational motions are summarized through Newton's second law, the state space expression of a dynamic model that simulates the transverse model in which L is considered can be expressed as Equation 2.

이며,

,

,

,

,

이다. here,

Is,

,

,

,

,

to be.

5 is a flow chart showing a lane maintenance control method according to an embodiment of the present invention.

In step 510, the lane keeping control device 100 receives a front image through the camera 110.

_{In step 515, the lane keeping control apparatus 100 predicts road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) by applying the front image to the previously learned road curve coefficient regression model unit. Since this is the same as already described above, a redundant description will be omitted.

In step 520, the lane keeping control device 100 applies the vehicle control input to a behavior model that simulates the lateral behavior of the vehicle based on the _{predicted road curve coefficients (c 0} , c ₁ , c ₂ , c _{3 ).} To derive. Since this has already been described above, a redundant description will be omitted.

6 is a graph comparing a control result according to an embodiment of the present invention and the related art. As shown in FIG. 6, it can be seen that the control result according to an embodiment of the present invention is accurately controlled more similar to that steered by a person.

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded in the computer-readable medium may be specially designed and configured for the present invention, or may be known to and usable by a person skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes magneto-optical media and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

So far, the present invention has been looked at around the embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: lane keeping control device
110: camera
120: prediction unit
130: memory
140: processor

Claims (7)

A camera that acquires a front image in real time;
_{In predicting the road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) by applying the front image to a pre-learned deep learning-based road curve coefficient regression model unit, control error of the road curve coefficient according to the attribute A prediction unit that classifies and predicts the coefficients (c ₀ , c ₁ ) related to the road lane model and the coefficients related to the shape of the road lane model (c ₂ , c _{3 ).} Classified and learned according to; And
Including a lateral controller for deriving a control input of the vehicle by applying it to a behavior model that simulates the behavior of the vehicle based on the predicted road curve coefficients (c ₀ , c ₁ , c ₂ , c _{3 ),}
The road curve coefficient regression model unit includes a first model unit and a second model unit,
_{The first model unit learns coefficients (c 0} , c ₁ ) related to the front image and control error as a training set, and the second model unit learns coefficients (c2, c3) related to the shape of the front image and the road lane model. ), the lane keeping control device, characterized in that the learning is performed using the training set.
delete The method of claim 1,
The behavior model that simulates the behavior of the vehicle is a model that simulates the lateral behavior, and the behavior model derives information on the lateral motion state of the vehicle using the predicted road curve coefficient. .
The method of claim 1,
The lane keeping control device, wherein the control input is a steering angle.
(a) obtaining a front image in real time;
_{(b) In predicting the road curve coefficients (c 0} , c ₁ , c ₂ , c ₃ ) by applying the front image to the previously learned deep learning-based road curve coefficient regression model unit, the road curve coefficient is applied to the property. Accordingly, classifying and predicting coefficients related to control errors (c ₀ , c ₁ ) and coefficients related to the shape of the road lane model (c ₂ , c ₃ )-The deep learning-based road curve coefficient regression model unit calculates the road curve coefficients. Classified and learned according to attributes; And
(c) applying to a behavior model that simulates the behavior of the vehicle based on the predicted road curve coefficients (c ₀ , c ₁ , c ₂ , c _{3) to derive a control input of the vehicle,}
Before step (a),
_{The first model unit of the deep learning-based road curve regression model unit is trained using coefficients (c 0} , c ₁ ) related to the front image and the control error as a training set, and coefficients related to the shape of the front image and the road lane model Using (c ₂ , c ₃ ) as a training set, the second model unit of the deep learning-based road curve regression model unit is trained.
delete A computer-readable recording medium product on which a program code for performing the method according to claim 5 is recorded.

Images