KR102656945B1 - Apparatus for sensor fusion of vehicle and method thereof - Google Patents

Abstract

The present invention relates to a sensor fusion device and method for a vehicle, which fuses camera object information and radar object information obtained respectively through a camera sensor and a radar sensor mounted on the vehicle through a predefined fusion algorithm to produce primary fusion object information. A first sensor fusion unit that generates a first sensor fusion unit, where the camera object information and radar object information are information of dynamic objects located around the vehicle, and the camera object information, primary fusion object information, and mounted on the vehicle. Perform preprocessing to synchronize the LiDAR object information acquired through the LiDAR sensor, correct the LiDAR object information by adding attribute information of the object to the synchronized LiDAR object information, and then create the corrected LiDAR object. A second sensor fusion unit that generates final fusion object information by updating the first fusion object information based on the information, wherein the lidar object information includes information on dynamic objects and static objects located around the vehicle, the second sensor It is characterized by including a fusion portion.

Description

Sensor fusion device and method for vehicle {APPARATUS FOR SENSOR FUSION OF VEHICLE AND METHOD THEREOF}

The present invention relates to a sensor fusion device and method for a vehicle, and more particularly, to a sensor fusion device and method for a vehicle that fuses object information obtained by a plurality of sensors mounted on the vehicle.

An autonomous vehicle refers to a vehicle that recognizes the surrounding environment through external information detection and processing functions when driving, determines its own driving path, and drives independently using its own power. Self-driving vehicles can drive to their destination on their own, preventing collisions with obstacles in the driving path and adjusting vehicle speed and driving direction according to the shape of the road, without the driver having to operate the steering wheel, accelerator pedal, or brakes. there is. For example, acceleration can be performed on a straight road, and deceleration can be performed on a curved road by changing the driving direction in response to the curvature of the road.

The positioning system applied to autonomous vehicles is based on road map information constructed using GPS (Global Positioning System) and various sensors (Radar, LiDAR, Camera, etc.), GPS location data acquired while driving, and The current location of the vehicle is determined through sensor data obtained through sensors. In order to ensure the stability of autonomous driving, it is important to accurately determine the current location of the vehicle, and for this purpose, DGPS (Differential GPS) and RTK-DGPS (Real Time Kinematic-DGPS), which can improve the location accuracy of GPS, are used. Sometimes it happens. In addition, map matching technology that compares sensor data with a pre-constructed road map is sometimes applied to correct errors in GPS location data that inevitably occur.

In addition, along with the current location of the vehicle, objects existing around the vehicle must be accurately detected through sensors mounted on the vehicle and used to control the vehicle's driving. As the sensing system applied to autonomous vehicles becomes more advanced, the sensor's Considering the current level of technology, which is increasing in number, an architecture that can detect surrounding objects more accurately by fusing surrounding object information acquired through each sensor is required.

The background technology of the present invention is disclosed in Republic of Korea Patent Publication No. 10-2017-0098071 (published on August 29, 2017).

The purpose of one aspect of the present invention is to develop a vehicle in which detection errors for objects can be minimized by fusing object information obtained through a plurality of object detection sensors mounted on a vehicle, especially camera sensors, radar sensors, and lidar sensors. To provide a sensor fusion device and method.

The sensor fusion device for a vehicle according to one aspect of the present invention fuses camera object information and radar object information obtained through a camera sensor and a radar sensor mounted on the vehicle, respectively, through a predefined fusion algorithm to produce primary fusion object information. A first sensor fusion unit that generates, wherein the camera object information and the radar object information are information of dynamic objects located around the vehicle, and the camera object information, the first fusion object information, And performing preprocessing to synchronize the LiDAR object information obtained through the LiDAR sensor mounted on the vehicle, and correcting the LiDAR object information by adding attribute information of the object to the synchronized LiDAR object information. Then, a second sensor fusion unit that updates the first fusion object information based on the corrected LiDAR object information to generate final fusion object information, wherein the LiDAR object information includes dynamic objects located around the vehicle and It is characterized in that it includes a second sensor fusion unit containing information about the static object.

In the present invention, the second sensor fusion unit performs coordinate synchronization by converting the coordinates of the camera object information and the primary fusion object information into the coordinate system of the LiDAR sensor, and the coordinate-synchronized camera object information and the primary fusion object information. The preprocessing is performed by performing time synchronization to match the times of the fusion object information and the lidar object information.

In the present invention, the second sensor fusion unit is characterized by adding attribute information according to the type of the object to the synchronized lidar object information using the camera object information.

In the present invention, the second sensor fusion unit updates the first fusion object information by adding static object information reflected in the corrected lidar object information to the first fusion object information to perform the final fusion. It is characterized by generating object information.

In the present invention, the second sensor fusion unit tracks the object for the final fusion object information using a predefined tracking algorithm and then outputs the location information of the object through post-processing.

In the present invention, the camera sensor includes a front camera sensor, the radar sensor includes a front radar and a corner radar, and the LIDAR sensor includes a front LIDAR sensor.

In the sensor fusion method for a vehicle according to an aspect of the present invention, the first sensor fusion unit fuses camera object information and radar object information respectively acquired through a camera sensor and a radar sensor mounted on the vehicle through a predefined fusion algorithm. Generating primary fusion object information, wherein the camera object information and the radar object information are information on dynamic objects located around the vehicle, wherein a second sensor fusion unit performs the camera object information and the primary fusion. A step of performing preprocessing to synchronize object information and LiDAR object information acquired through a LiDAR sensor mounted on the vehicle, wherein the LiDAR object information is information on dynamic objects and static objects located around the vehicle Comprising a step, wherein the second sensor fusion unit corrects the LiDAR object information by adding attribute information of the object to the synchronized LiDAR object information, and the second sensor fusion unit corrects the corrected LiDAR object information. Characterized by including the step of updating the first fusion object information based on lidar object information.

According to one aspect of the present invention, the present invention detects objects around the vehicle by first fusion of camera object information and radar object information, then correcting the first fusion result through lidar object information and performing tracking post-processing. Errors can be minimized.

1 is a block diagram illustrating a sensor fusion device for a vehicle according to an embodiment of the present invention.
Figure 2 is a block diagram showing the detailed configuration of a sensor fusion device for a vehicle according to an embodiment of the present invention.
Figure 3 is an example diagram showing the FOV (Field of View) of a front camera, front radar, corner radar, and front lidar applied to a sensor fusion device for a vehicle according to an embodiment of the present invention.
Figure 4 is a functional block diagram for explaining the operation of the preprocessing unit and the measurement value fusion unit of the second sensor fusion unit in the sensor fusion device for a vehicle according to an embodiment of the present invention.
Figure 5 is a flowchart illustrating a sensor fusion method for a vehicle according to an embodiment of the present invention.

Hereinafter, embodiments of a sensor fusion device and method for a vehicle according to the present invention will be described with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawing may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

FIG. 1 is a block diagram for explaining a sensor fusion device for a vehicle according to an embodiment of the present invention, and FIG. 2 is a block diagram showing the detailed configuration of a sensor fusion device for a vehicle according to an embodiment of the present invention. , FIG. 3 is an exemplary diagram showing the FOV (Field of View) of the front camera, front radar, corner radar, and front lidar applied to the sensor fusion device of a vehicle according to an embodiment of the present invention, and FIG. This is a functional block diagram for explaining the operation of the preprocessing unit and the measurement value fusion unit of the second sensor fusion unit in the sensor fusion device for a vehicle according to an embodiment of.

Referring to Figures 1 and 2, the sensor fusion device for a vehicle according to an embodiment of the present invention may include a first sensor fusion unit 100 and a second sensor fusion unit 200, and the second sensor fusion unit 200 The unit 200 may include a pre-processing unit 210, a measurement fusion unit 220, a tracking unit 230, and a post-processing unit 240.

The first sensor fusion unit 100 may fuse camera object information and radar object information obtained through a camera sensor and a radar sensor mounted on the vehicle, respectively, through a predefined fusion algorithm to generate primary fusion object information. . Here, the camera object information and radar object information refer to information on dynamic objects (e.g., other vehicles, bicycles, pedestrians, etc.) moving around the vehicle, such as the position, direction (heading angle), acceleration, and Information about speed, shape and size, type, and existence of a dynamic object may correspond to dynamic object information. Accordingly, the first sensor fusion unit 100 can generate primary fusion object information by fusing camera object information and radar object information through a predefined fusion algorithm, and generates predetermined information during the object fusion process according to the fusion algorithm. If necessary (e.g. vehicle attitude information, etc.), it can be received through C-CAN communication.

The second sensor fusion unit 200 performs preprocessing to synchronize camera object information, primary fusion object information, and LiDAR object information acquired through a LiDAR sensor mounted on the vehicle, and synchronized LiDAR object information. After correcting the LiDAR object information by adding attribute information of the corresponding object, the final fusion object information can be generated by updating the first fusion object information based on the corrected LiDAR object information. Here, the LIDAR object information may include information on dynamic objects and static objects (e.g., traffic lights, signposts, fixed obstacles, etc.) located around the vehicle.

In this embodiment, the camera sensor may include a front camera sensor, the radar sensor may include a front radar and a corner radar, and the LiDAR sensor may include a front LiDAR sensor. Figure 3 shows an example of the FOV (Field of View) of each sensor, however, the FOV of each sensor may be designed in various ranges based on the designer's intention and experimental results.

Hereinafter, the function of the second sensor fusion unit 200 will be described in detail through its sub-configurations. As shown in FIG. 2, the second sensor fusion unit 200 may include a pre-processing unit 210, a measurement fusion unit 220, a tracking unit 230, and a post-processing unit 240. FIG. 4 is a functional block diagram for explaining the operation of the pre-processing unit 210 and the measurement fusion unit 220. The following description of the operation of the pre-processing unit 210 and the measurement fusion unit 220 will refer to FIG. 4. do.

The pre-processing unit 210 may perform pre-processing to synchronize camera object information, primary fusion object information, and LiDAR object information acquired through a LiDAR sensor mounted on the vehicle. Here, synchronization of object information may mean coordinate synchronization and time synchronization.

Specifically, the preprocessor 210 can perform coordinate synchronization by converting the coordinates of the camera object information and the first fusion object information into the coordinate system of the LiDAR sensor. To this end, the preprocessor 210 includes camera object information and 1 A transformation matrix for changing the coordinates of car fusion object information into LiDAR coordinates may be predefined. Additionally, the preprocessor 210 may perform time synchronization to match the times of the coordinate-synchronized camera object information and primary fusion object information with the lidar object information. Furthermore, the preprocessor 210 may compensate for the position of the corresponding object by considering the movement of the host vehicle and the movement of the corresponding object reflected in each object information.

The Measuring Fusion Unit (200) may correct the LiDAR object information by adding attribute information of the object to the LiDAR object information for which coordinates and time synchronization have been achieved by the preprocessor (210). The state in which the LiDAR object information is corrected refers to the state in which the attribute information of the object is added after synchronization of the LiDAR object information acquired by the LiDAR sensor is achieved. At this time, the measurement fusion unit 220 can use the camera object information to add attribute information (Flag) according to the type of object (e.g., car, truck, bicycle, pedestrian, etc.) to the synchronized lidar object information. there is.

Additionally, the measurement fusion unit 220 may update the first fusion object information based on the corrected LIDAR object information to generate final fusion object information. Specifically, the measurement fusion unit 220 updates the primary fusion object information by adding static object information reflected in the corrected LiDAR object information to the primary fusion object information to generate final fusion object information. can do. Since the first fusion object information is fusion information of camera object information and radar object information, which are dynamic object information, by adding the static object information reflected in the lidar object information to the first fusion object information, the final fusion object information includes the surrounding areas of the vehicle. The dynamic object and static object information of is reflected, and the attribute information of the corresponding object for the final fused object information is also included.

The tracking unit 230 may perform tracking of the corresponding object for the final fused object information using a predefined tracking algorithm. As shown in FIG. 2, the tracking unit 230 can track the object through prediction, data association, and update methods according to a general target tracking algorithm. The primary fusion object information and lidar object information are information acquired through separate tracking processes, and the tracking error for the corresponding object can be reduced by performing additional tracking through the tracking unit 230.

Afterwards, the post-processing unit 240 can finally output the position information of the object being tracked through post-processing, and the direction (heading angle), acceleration, speed, and shape along with the position information of the object. Information on size and type can also be output.

Figure 5 is a flowchart for explaining a sensor fusion method for a vehicle according to an embodiment of the present invention.

When describing the sensor fusion method of a vehicle according to an embodiment of the present invention with reference to FIG. 5, first, the first sensor fusion unit 100 receives camera object information and camera object information respectively acquired through a camera sensor and a radar sensor mounted on the vehicle. Radar object information is fused through a predefined fusion algorithm to generate primary fusion object information (S100). Camera object information and radar object information refer to information on dynamic objects located around the vehicle.

Next, the second sensor fusion unit 200 performs preprocessing to synchronize the camera object information, the first fusion object information, and the LiDAR object information acquired through the LiDAR sensor mounted on the vehicle (S200). Lidar object information includes information on dynamic objects and static objects located around the vehicle.

In step S200, the second sensor fusion unit 200 performs coordinate synchronization by converting the coordinates of the camera object information and the first fusion object information into the coordinate system of the LiDAR sensor, and coordinate-synchronized camera object information and the first fusion object. Preprocessing is performed by performing time synchronization to match the time of the information and the lidar object information.

Next, the second sensor fusion unit 200 corrects the LiDAR object information by adding attribute information of the object to the LiDAR object information synchronized through step S200 (S300). In step S300, the second sensor fusion unit 200 uses camera object information to add attribute information according to the type of the object to the LiDAR object information synchronized through step S200.

Subsequently, the second sensor fusion unit 200 updates the first fusion object information based on the lidar object information corrected through steps S200 and S300 (S400). In step S400, the second sensor fusion unit 200 updates the primary fusion object information by adding static object information reflected in the corrected lidar object information to the primary fusion object information. Final fusion object information is generated through step S400.

Subsequently, the second sensor fusion unit 200 tracks the object for the final fusion object information using a predefined tracking algorithm in step S400, and then outputs the location information of the object through post-processing ( S500), along with the location information of the object, information on direction (heading angle), acceleration, speed, shape, size, and type can also be output.

In this way, in this embodiment, the detection error for objects around the vehicle can be minimized by first fusion of camera object information and radar object information, then correcting the first fusion result through lidar object information and performing tracking post-processing. there is.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will recognize that various modifications and other equivalent embodiments can be made therefrom. You will understand. Therefore, the true technical protection scope of the present invention should be determined by the scope of the patent claims below.

100: First sensor fusion unit
200: Second sensor fusion unit
210: preprocessing unit
220: Measurement fusion unit
230: Tracking unit
240: Post-processing unit

Claims (11)

A first sensor fusion unit that generates primary fusion object information by fusing the camera object information and radar object information as dynamic object information acquired through a camera sensor and a radar sensor respectively mounted on the vehicle through a predefined fusion algorithm; and
Perform preprocessing to synchronize the camera object information, the primary fusion object information, and the LiDAR object information acquired through the LiDAR sensor mounted on the vehicle, and use the camera object information to After correcting the LiDAR object information by adding attribute information according to the type of the object to the object information, the first fusion object information is updated based on the corrected LiDAR object information to generate final fusion object information. It includes a second sensor fusion unit;
The update is performed by adding static object information of the corrected lidar object information to the primary fusion object information,
The sensor fusion device for a vehicle, wherein the camera sensor includes a front camera sensor, the radar sensor includes a front radar and a corner radar, and the lidar sensor includes a front lidar sensor.
According to paragraph 1,
The second sensor fusion unit performs coordinate synchronization by converting the coordinates of the camera object information and the first fusion object information into the coordinate system of the LiDAR sensor, and the coordinate-synchronized camera object information and the first fusion object information A sensor fusion device for a vehicle, characterized in that the preprocessing is performed by performing time synchronization to match the time of the lidar object information.
delete delete According to paragraph 1,
The second sensor fusion unit tracks the object for the final fusion object information using a predefined tracking algorithm and then outputs the location information of the object through post-processing. Fusion device.
delete The first sensor fusion unit generates first fusion object information by fusing dynamic object information obtained through a camera sensor and a radar sensor mounted on the vehicle, respectively, and camera object information and radar object information through a predefined fusion algorithm. step;
A second sensor fusion unit performing preprocessing to synchronize the camera object information, the primary fusion object information, and the LiDAR object information obtained through a LiDAR sensor mounted on the vehicle;
The second sensor fusion unit corrects the LiDAR object information by adding attribute information according to the type of the object to the synchronized LiDAR object information using the camera object information; and
A step of the second sensor fusion unit updating the first fusion object information based on the corrected lidar object information to generate final fusion object information,
The update is performed by adding static object information of the corrected lidar object information to the primary fusion object information,
The sensor fusion method for a vehicle, wherein the camera sensor includes a front camera sensor, the radar sensor includes a front radar and a corner radar, and the lidar sensor includes a front lidar sensor.
In clause 7,
In the step of performing the preprocessing, the second sensor fusion unit,
Coordinate synchronization is performed by converting the coordinates of the camera object information and the first fusion object information into the coordinate system of the LiDAR sensor, and the time of the coordinate-synchronized camera object information and the first fusion object information and the LiDAR object information A sensor fusion method for a vehicle, characterized in that the preprocessing is performed by performing time synchronization to match.
delete delete In clause 7,
The second sensor fusion unit tracks the object for the final fusion object information using a predefined tracking algorithm, and then outputs location information of the object through post-processing. Characterized by a sensor fusion method for a vehicle.