KR102680011B1 - Apparatus and method for recognizing object - Google Patents

Abstract

An object recognition method according to the present invention includes the steps of acquiring location information of an own vehicle; activating detection logic to detect an object based on the location information of the own vehicle; determining a region of interest for detecting the object in an image captured by a camera; Detecting the object using a detector on the determined region of interest and recognizing the detected object by determining whether it is a true positive or a false positive through a pre-trained recognizer. Includes steps for generating results.

Description

Object recognition apparatus and method {APPARATUS AND METHOD FOR RECOGNIZING OBJECT}

The present invention relates to an object recognition device and method.

A sign detection system according to the prior art, as disclosed in Korean Patent Publication Nos. 10-2015-0085988, 10-2015-0082823, and 10-2015-0081480, receives images from a camera and uses a learned classifier ( It consists of logic that detects the candidate area of the sign (for detection) and then recognizes the detected candidate area using a learned classifier (for recognition).

In the case of this prior art, since trained detectors and recognizers are used, the quantity and quality of the database for training must be very good, and even if this is met, there is a problem in that non-detection and false detection occur very frequently.

Additionally, in the case of camera images, detectors trained so that the size of objects in the image vary depending on the distance between the camera and the object must be applied in various sizes, which causes the problem that the processing speed for object recognition becomes very slow.

In addition, since various detectors and recognizers are used to recognize dozens of types of signs, complexity greatly increases and processing speed inevitably decreases.

This problem also applies to the TLR (Traffic Light Recognition) system. Compared to signs with many features, traffic lights lack image features, so the difficulty in recognizing them is very high.

Meanwhile, Korean Patent Publication No. 10-1339255 discloses setting a lane detection region of interest (ROI) using a precision map. At this time, since lane detection does not use a trained detector, there is a limitation in the above patent that a precise map can only be used to set the area of interest.

In summary, Traffic Sign Recognition (TSR) and TLR according to the prior art have the following problems.

First, in the case of the prior art, the size of signs and traffic lights is small, so a high-pixel camera must be used. In this case, there is a problem that the detection time takes a long time because the image size is large.

In addition, since the size of the object in the image varies depending on the distance between the camera and the target object, detectors of various sizes must be used. For this, the processing time increases by one time each time the size of the detector is changed.

Additionally, since small objects must be detected, the detector must be tuned very precisely in advance, which increases the non-detection rate.

Additionally, in order to detect various target objects, the number of detectors and recognizers increases, which ultimately increases processing complexity and processing time.

Lastly, if the detector threshold is lowered to reduce the non-detection rate, numerous false detections will occur. In other words, it is a very difficult problem to accurately recognize only signs and traffic lights that do not have many image features in various background environments such as urban centers, and it is not easy to lower the non-detection rate no matter how perfectly the detector training is performed.

The basic and fundamental cause of the above-mentioned problem is as shown in FIG. 1.

Figure 1 is a diagram to explain the problems of the object recognition method according to the prior art.

As shown in Figure 1, the exact location of signs and traffic lights within the currently input camera image is unknown, so the entire image must be scanned (P1) through detectors of various sizes, and as a result, the processing time required for object recognition is reduced. In addition, when there is a diverse background environment (P2), false detections occur and the accuracy of object recognition decreases.

The embodiment of the present invention acquires the location information of the own vehicle based on a precision positioning technique, sets an area of interest from the image, detects and recognizes the object, and reduces the processing speed and non-detection and false detection problems, which are technical problems of TSR and TLR. Provides an object recognition device and method that can solve the problem.

However, the technical challenge that this embodiment aims to achieve is not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-described technical problem, a method for recognizing an object from an image captured by a camera of the own vehicle according to the first aspect of the present invention includes the steps of acquiring location information of the own vehicle; activating detection logic to detect an object based on the location information of the own vehicle; determining a region of interest for detecting the object in an image captured by a camera; Detecting the object using a detector on the determined region of interest and recognizing the detected object by determining whether it is a true positive or a false positive through a pre-trained recognizer. Includes steps for generating results.

In addition, a device for recognizing an object from an image captured by a camera of a vehicle according to the second aspect of the present invention includes a communication module that receives the image captured by the camera, a memory storing a program for recognizing the object, and the memory. Includes a processor that executes stored programs. At this time, as the processor executes the program, it activates detection logic for detecting an object based on the location information of the own vehicle, determines a region of interest for detecting the object in the captured image, and detects the object based on the detector. The object is detected, and a recognition result of the object is generated by determining whether the detected object is a true positive or a false positive through a pre-trained recognizer.

According to one of the means for solving the problem of the present invention described above, only some areas where signs or traffic lights are likely to exist are searched, and there is no need for redundant searches such as the image pyramid technique or the use of detectors of various sizes, which are required for object recognition. Processing time can be minimized.

In addition, since the detection technique only needs to be performed in a narrow area of interest where signs clearly exist, the non-detection rate and false detection rate can be minimized.

Additionally, since what types of signs exist are known in advance through precise maps, there is no need to use all kinds of detectors and recognizers.

Figure 1 is a diagram to explain the problems of the object recognition method according to the prior art.
Figure 2 is an exemplary diagram of a method for recognizing an object according to an embodiment of the present invention.
Figure 3 is a block diagram of an object recognition device according to an embodiment of the present invention.
Figure 4 is a diagram showing the location of the own vehicle and the location of the object within the precision map.
Figure 5 is an example of changing the area of interest by changing the positioning position.
Figure 6 is an example diagram showing the result of object recognition through an embodiment of the present invention.
Figure 7 is a flowchart of an object recognition method according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted.

Throughout the specification, when a part "includes" a certain element, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

The present invention relates to an object recognition device 100 and method.

According to an embodiment of the present invention, location information of the own vehicle is acquired based on a precision positioning technique, an area of interest is set from the image, and the object is detected and recognized, thereby solving the non-detection and false detection problems that are technical problems of TSR and TLR. This can solve the problem of reduced processing speed.

Figure 2 is an exemplary diagram of a method for recognizing an object according to an embodiment of the present invention.

In other words, as shown in FIG. 2, if the approximate area of interest (P3) is determined for the location of the traffic lights and signs in the input image and the approximate size of the sign in the image is known, the problems caused by the prior art can be solved. It becomes possible.

Hereinafter, with reference to FIGS. 3 to 6, an apparatus 100 for recognizing an object from an image captured by a camera 10 of a vehicle according to an embodiment of the present invention will be described.

Figure 3 is a block diagram of an object recognition device 100 according to an embodiment of the present invention.

The object recognition device 100 according to an embodiment of the present invention includes a communication module 110, a memory 120, and a processor 130.

The communication module 110 receives images captured by the camera 10. This camera 10 may be a front camera pre-installed in the vehicle. Alternatively, it may be a smartphone camera or a camera that captures separate images. In this case, the captured images may be transmitted to the vehicle's communication module through a wireless or wired network.

This communication module 110 may include both a wired communication module and a wireless communication module. The wired communication module can be implemented as a power line communication device, telephone line communication device, cable home (MoCA), Ethernet, IEEE1294, integrated wired home network, and RS-485 control device. Additionally, wireless communication modules can be implemented with WLAN (wireless LAN), Bluetooth, HDR WPAN, UWB, ZigBee, Impulse Radio, 60GHz WPAN, Binary-CDMA, wireless USB technology, and wireless HDMI technology.

Preferably, the communication module 110 may be implemented as CAN (Controller Area Network).

A program for recognizing objects is stored in the memory 120. Here, the object refers to a traffic light or sign, but is not necessarily limited thereto, and may be a variety of identifiable objects for obtaining traffic information.

At this time, the memory 120 is a general term for non-volatile storage devices and volatile storage devices that continue to retain stored information even when power is not supplied.

For example, memory 120 may include compact flash (CF) cards, secure digital (SD) cards, memory sticks, solid-state drives (SSD), and micro SD. This includes NAND flash memory such as cards, magnetic computer storage devices such as hard disk drives (HDD), and optical disc drives such as CD-ROM, DVD-ROM, etc. You can.

The processor 130 executes a program stored in the memory 120, and by executing the program, an object can be recognized from the captured image.

Specifically, the processor first obtains location information of the own vehicle. At this time, the processor 130 can perform precise positioning of the own vehicle using satellite navigation, dead reckoning based on vehicle behavior, and map matching techniques.

The processor 130 obtains first location information of the own vehicle based on satellite navigation. Satellite navigation is a positioning method based on GNSS (Global Navigation Satellite System) such as GPS, DGPS, and Network-RTK, and is a technique that allows absolute coordinates to be transmitted through satellites.

In the case of location navigation, the error is variable at 5 to 50 m, and there is a disadvantage that the acquired location information cannot be completely trusted in areas where multi-paths occur, such as urban areas.

In order to compensate for this shortcoming, the processor 130 acquires second location information by correcting the first location information based on dead reckoning based on vehicle behavior. Dead reckoning based on vehicle behavior utilizes vehicle sensors and an inertial measurement unit (IMU) that measures vehicle speed, steering angle, wheel odometer, yaw rate, acceleration, etc. to provide first position information measured by satellite navigation. This is a way to correct it.

Dead reckoning based on vehicle behavior is calculated by accumulating the movement of the own vehicle, and errors accumulate over time. The first location information can be corrected with an error of about 1 to 10 m, but error accumulation continues to occur.

To this end, the processor 130 may obtain third location information by correcting the second location information based on a map matching technique using a pre-stored precision map for autonomous driving and apply it as the final location information of the own vehicle.

The map matching technique determines the vehicle's location relatively based on the results of comparison between a precision map for autonomous driving and data acquired through one or more sensors installed on the vehicle (e.g., camera, stereo camera, AVM camera, lidar, etc.). This is a method of obtaining estimated third location information.

Here, a precision map refers to a map constructed with three-dimensional coordinates of lanes, road surface information, sign information, and traffic light information as similar to the real world as possible.

Through this process, the processor 130 can ultimately obtain location information at a level that enables autonomous driving in cm units.

After obtaining the location information of the own vehicle, the processor 130 activates detection logic to detect an object based on the location information of the own vehicle.

Figure 4 is a diagram showing the location of the own vehicle and the location of the object within the precision map.

Specifically, the processor compares the position of the own vehicle in the precision map (P5) with the 3D location information (P4) of the object built in the precision map, and as a result of the comparison, the object exists within the position of the own vehicle and a preset distance (P6). It determines whether or not it is present, and activates the detection logic if it is within a preset distance (P6).

When the detection logic is activated through this process, if there are no objects such as signs or traffic lights nearby, the detection logic is deactivated and does not operate, which can reduce the processing time for object recognition and reduce the phenomenon of false detection. Possible effects can also be expected.

Meanwhile, when the detection logic is activated, the processor 130 determines a region of interest for detecting an object in the image captured by the camera 10. At this time, the processor 130 time-synchronizes the captured image and the precision map based on the time stamp of the image. This is because the position of the own vehicle changes continuously, so images and precise maps obtained from the same location are needed.

Time synchronization moves or rotates the precision map based on the time stamp of the image, and the difference in vehicle position according to the time difference can be calculated through dead-reckoning trajectory information.

After performing time synchronization, the processor 130 maps the 3D location information of the object expressed in the vehicle-centered coordinate system in the precision map to the image coordinate system through Equation 1.

Here, (X, Y, Z) means three-dimensional coordinates in reality, and (u, v) means the projected coordinates of the pixel. And A refers to the camera matrix or unique parameters of the camera, (c _x , c _y ) refers to the principal point of the image _center , and ( _f It refers to distances (focal lengths).

Meanwhile, if the longitudinal and lateral positioning of the vehicle is accurate, the region of interest (ROI) in the image can be accurately determined. However, due to the nature of the map matching technique that utilizes lanes, etc., the accuracy in the longitudinal direction may be relatively lower than the accuracy of the lateral positioning.

Therefore, errors may be included in determining the region of interest due to longitudinal positioning errors.

To solve this problem, in one embodiment of the present invention, the processor 130 changes the area of interest by changing the longitudinal position of the vehicle, as shown in FIG. 5. And when an object is detected in the changed area of interest, the processor 130 may determine the area to be the area of interest.

Figure 5 is an example of changing the area of interest by changing the positioning position.

The change in the longitudinal position of the vehicle moves within a certain range from the currently estimated positioning position, moving from closer to farther away (③ → ②, ④ → ①, ⑤). At this time, if the longitudinal positioning is accurate, determination of the area of interest may be completed in one attempt.

If the processor 130 attempts to detect an object by moving in the longitudinal direction a certain number of times (for example, 5 times), but is not detected, the result is treated as a failure, and if the failure process accumulates more than a preset number of times, The processor 130 may determine that the location information of the own vehicle was obtained incorrectly.

This embodiment of the present invention can also be used to correct longitudinal positioning error of the own vehicle by referring to the determined area of interest.

Once the area of interest is determined, the processor 130 may detect an object within the determined area of interest based on a determined detector. At this time, the detector may be trained in advance according to the type of object registered in the precision map (for example, the type of sign or traffic light).

Accordingly, the processor 130 can select and use a detector corresponding to the type of object among the learned detectors. For example, if the type of object is a speed sign, the detector corresponding to the speed sign can be selected and used among speed signs, caution signs, and instruction signs.

Additionally, the processor 130 may calculate the size of the object included in the image by substituting the size of the object registered in the precision map into Equation 1. Accordingly, an embodiment of the present invention can detect an object using only a detector of a fixed size.

In this way, if an object is detected from the image, the processor 130 determines whether the detected object is a true positive or a false positive through a recognizer learned in advance, and results in recognition of the object. can be created.

Figure 6 is an example diagram showing the result of object recognition through an embodiment of the present invention.

Referring to FIG. 6, the area of interest (P6) corresponding to the traffic light and sign included in the image can be determined, and the traffic light and sign (P7) included within the area of interest (P6) can be accurately detected.

For reference, the components shown in FIG. 3 according to an embodiment of the present invention may be implemented in the form of software or hardware such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and perform certain roles. can do.

However, 'components' are not limited to software or hardware, and each component may be configured to reside on an addressable storage medium or may be configured to run on one or more processors.

Thus, as an example, a component may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, and sub-processes. Includes routines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

Components and the functionality provided within them may be combined into a smaller number of components or further separated into additional components.

Hereinafter, an object recognition method in the object recognition device 100 according to an embodiment of the present invention will be described with reference to FIG. 7.

Figure 7 is a flowchart of an object recognition method according to an embodiment of the present invention.

The object recognition method according to an embodiment of the present invention first acquires location information of the own vehicle based on satellite navigation, dead reckoning based on vehicle behavior, and map matching techniques (S110).

Next, detection logic for detecting an object is activated based on the location information of the own vehicle (S120). This detection logic compares the position of the own vehicle on the precision map with the 3D location information of the object constructed within the precision map, and can activate the detection logic if the object is within a preset distance.

Next, a region of interest for detecting an object in the image captured by the camera 10 is determined (S130). At this time, in determining the area of interest, the area of interest may be changed by changing the longitudinal position of the own vehicle, and if an object is detected in the changed area of interest, the area may be determined as the area of interest.

Additionally, if an object is not detected in the area of interest that has changed more than a preset number of times, it may be determined that the location information of the own vehicle has been obtained incorrectly.

Next, an object is detected using a detector on the determined area of interest (S140). The detector is learned in advance according to the type of object included in the precision map, and the detector corresponding to the type of object can be selected and used among the learned detectors.

Next, a pre-trained recognizer determines whether the detected object is a true positive or a false positive to generate a final recognition result (S150).

Meanwhile, in the above description, steps S110 to S150 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present invention. Additionally, some steps may be omitted or the order between steps may be changed as needed. In addition, even if other omitted content, the content already described in FIGS. 2 to 6 can also be applied to the object recognition method of FIG. 7.

According to one of the embodiments of the present invention, only some areas where signs or traffic lights are likely to exist are searched, and there is no need for redundant searches such as the image pyramid technique or the use of detectors of various sizes, which are required for object recognition. Processing time can be minimized.

In addition, since the detection technique only needs to be performed in a narrow area of interest where signs clearly exist, the non-detection rate and false detection rate can be minimized.

Additionally, since what types of signs exist are known in advance through precise maps, there is no need to use all kinds of detectors and recognizers.

Meanwhile, an embodiment of the present invention may also be implemented in the form of a computer program stored on a medium executed by a computer or a recording medium containing instructions executable by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery medium.

Although the methods and systems of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

10: Camera
100: object recognition device
110: Communication module
120: memory
130: processor

Claims (13)

In a method of recognizing an object from an image captured by a vehicle's camera,
Obtaining location information of the own vehicle;
activating detection logic to detect an object based on the location information of the own vehicle;
determining a region of interest for detecting the object in an image captured by a camera;
detecting the object using a detector on the determined area of interest; and
Generating a recognition result by determining whether the detected object is a true positive or a false positive through a pre-trained recognizer,
The step of acquiring the location information of the own vehicle is,
Obtaining first location information of the own vehicle based on satellite navigation;
Obtaining second location information by correcting the first location information based on dead reckoning based on vehicle behavior, and
Obtaining third location information corrected for the second location information based on a map matching technique using a pre-stored precision map for autonomous driving and applying it as the location information,
The step of activating the detection logic based on the location information of the own vehicle includes:
Comparing the position of the own vehicle in the precise map and 3D location information of an object constructed in the precise map;
Determining whether the object exists within a preset distance as a result of the comparison, and
When it is determined that the distance is within a preset distance, activating the detection logic;
The step of determining the area of interest is,
Time-synchronizing the captured image and the precise map based on a time stamp of the image;
Comprising the step of mapping 3D location information of objects in the time-synchronized precise map into an image coordinate system.
Object recognition method.
delete According to claim 1,
The vehicle behavior-based dead-reckoning method is an object recognition method that corrects the first location information using a vehicle sensor and IMU installed in the own vehicle.
According to claim 1,
The map matching technique is an object recognition method that obtains third location information in which the relative position of the own vehicle is estimated based on a comparison result between the precision map for autonomous driving and data acquired through one or more sensors installed in the own vehicle. .
delete delete According to claim 1,
The step of determining the region of interest is,
changing the area of interest by changing the longitudinal position of the own vehicle;
An object recognition method comprising determining the changed region of interest as the region of interest when the object is detected in the changed region of interest.
According to claim 7,
When the object is detected in the changed area of interest, the step of determining the area of interest is:
An object recognition method in which, when an object is not detected in an area of interest that has changed more than a preset number of times, it is determined that the location information of the own vehicle has been obtained incorrectly.
According to claim 1,
The detector is trained in advance according to the type of the object included in the precision map,
The step of detecting the object is,
An object recognition method that selects and uses a detector corresponding to the type of the object among the learned detectors.
According to claim 1,
An object recognition method further comprising calculating the size of the object included in the precision map.
In a device that recognizes an object from an image captured by a vehicle's camera,
A communication module that receives images captured by a camera,
Memory where a program for recognizing objects is stored and
Including a processor that executes the program stored in the memory,
As the processor executes the program, it activates detection logic for detecting an object based on the location information of the own vehicle, determines a region of interest for detecting the object in the captured image, and detects the object based on the detector. Detecting an object, determining whether the detected object is a true positive or a false positive through a pre-trained recognizer, and generating a recognition result of the object,
The processor acquires first location information of the own vehicle based on satellite navigation, acquires second location information corrected for the first location information based on dead reckoning based on vehicle behavior, and then generates a pre-stored precision map for autonomous driving. Obtain third location information that corrects the second location information based on a map matching technique using and apply it as the location information,
The processor determines whether the object exists within a preset distance as a result of comparing the position of the own vehicle in the precision map and the 3D position information of the object constructed in the precision map, and determines whether the object exists within a preset distance as a result of the determination. If present within the distance, activate the detection logic,
The processor time-synchronizes the captured image and the precision map based on the time stamp of the image, and maps the 3D location information of the object in the time-synchronized precision map into an image coordinate system.
Object recognition device.
delete delete