JP7687189B2 - Object detection device and object detection method - Google Patents

Description

The present invention relates to an object detection device and an object detection method, and in particular to an object detection device and an object detection method that can perform appropriate tracking processing on a detected object.

Devices that detect objects such as pedestrians from images captured by a camera mounted on a vehicle and display them with a detection frame to inform the driver of the presence of people are becoming widespread. In such devices, the position of people in the captured image changes due to the movement of pedestrians or vehicles, so a tracking process is performed to make it possible to display a detection frame that follows the movement.

Patent document 1 describes a technology for tracking objects around a vehicle.

JP 2007-279808 A

Several methods have been proposed for tracking processing. In general, tracking processing with high tracking performance imposes a relatively large processing load, such as calculation processing, while tracking processing with low tracking performance imposes a relatively small processing load. For this reason, applying tracking processing with high tracking performance to the tracking of all detected objects increases the processing load when there are a large number of detected objects, and it may not be possible to perform tracking processing for all objects or it may affect other processing. Furthermore, applying tracking processing with low tracking performance to the tracking of all detected objects may cause tracking to be interrupted, making it difficult to grasp the objects, and thus jeopardizing the safety of vehicle driving.

The present invention aims to provide an object detection device and an object detection method that can appropriately perform tracking processing of an object detected from a captured image.

The object detection device according to the present invention includes an object detection unit that acquires an image captured by an imaging unit that captures an image of the outside of the vehicle from the imaging unit and detects an object present in the captured image, a determination unit that determines a collision risk indicating the degree of risk of a collision between the vehicle and the object, and a tracking processing unit that selects a tracking method having a tracking performance based on the collision risk of the object and performs tracking processing of the object using the selected tracking method.

The object detection method according to the present invention is carried out by an object detection device, which performs the following steps: acquiring an image captured by an imaging unit that captures an image of the outside of the vehicle, and detecting an object present in the captured image; determining a collision risk indicating the degree of risk of a collision between the vehicle and the object; selecting a tracking method having a tracking performance based on the collision risk of the object, and performing a tracking process for the object using the selected tracking method.

According to the present invention, it is possible to appropriately perform tracking processing of an object detected from a captured image.

1 is a block diagram illustrating a configuration of an object detection system including an object detection device according to an embodiment. 3A and 3B are schematic diagrams illustrating display screens of a display unit according to the embodiment; 4 is a flowchart illustrating an operation of the object detection device according to the embodiment. 11 is a schematic diagram illustrating a display screen according to a comparative example of the embodiment; FIG.

[Embodiment]
<Configuration>
FIG. 1 is a block diagram illustrating a configuration of an object detection system including an object detection device according to an embodiment.

As shown in FIG. 1, the object detection system 10 according to the embodiment includes an object detection device 11, an imaging unit 12, and a display unit 13.

The object detection device 11 is a control device that controls the object detection system 10, and is, for example, a processing device constituted by a CPU (Central Processing Unit) and the like. The control device included in the object detection device 11 loads stored programs into an internal memory (not shown) and executes them. The object detection device 11 includes, as its configuration and functions realized by the programs, an object detection unit 111, a determination unit 112, a tracking processing unit 113, and a display control unit 114.

The imaging unit 12 is a camera provided in the vehicle and is disposed facing forward of the vehicle. The imaging unit 12 is a far-infrared camera or a visible light camera. Although not shown, the imaging unit 12 includes a lens, an imaging element, an A-D (Analog to Digital) conversion element, and the like. The imaging unit 12 outputs data constituting the captured image to the object detection unit 111 and the display control unit 114.

The display unit 13 is a display device, a so-called monitor, that displays various information under the control of the display control unit 114. The display unit 13 has a display panel, such as a liquid crystal panel or an organic EL (Organic Electro-Luminescence) panel. The display unit 13 is disposed in a position that is easily visible to the driver of the vehicle. The display unit 13 may be a display device dedicated to the object detection system 10, or may be a display device shared with other devices in the vehicle, such as a navigation device or an infotainment system.

The object detection unit 111 acquires an image captured by the imaging unit 12, which captures images of the outside of the vehicle, and detects objects present in the captured image. The objects detected by the object detection unit 111 include at least one of a person, a cyclist who is a person riding a bicycle, and an animal. The object detection unit 111 performs pattern matching on the captured image using an object recognition dictionary to perform object detection processing. The object recognition dictionary includes, for example, a person recognition dictionary for recognizing people, a cyclist recognition dictionary for recognizing cyclists, and an animal recognition dictionary for recognizing animals.

When the imaging unit 12 is a far-infrared camera, the object recognition dictionary referred to by the object detection unit 111 is a dictionary that has been machine-learned from far-infrared images, or so-called thermal images, of the object to be detected. For example, in the case of a person recognition dictionary, the dictionary is a dictionary that has been machine-learned from thermal images of people photographed from a vehicle.

The determination unit 112 determines the collision risk level, which indicates the degree of risk of a collision between the vehicle and the object, based on information about the respective movements of the vehicle and the object. The vehicle and the object move freely. For example, the vehicle and the object may move in different directions and at different speeds. The vehicle and the object may move in the same direction and at the same speed. The vehicle and the object may move in the same direction but at different speeds. The vehicle and the object may move in different directions but at the same speed. The collision risk level is also referred to as the possibility of collision, the possibility of contact, the impact of the possibility of collision, or the impact of the possibility of contact.

The information related to movement includes at least one of the following, or a combination of several: the distance from the vehicle to the object, the position of the object relative to the vehicle's direction of travel, the object's direction of travel, the object's speed of travel, the object's relative speed to the vehicle, the vehicle's direction of travel, the vehicle's speed of travel, and the predicted collision time until the vehicle collides with the object.

The determination unit 112 calculates the distance from the vehicle to the object based on the lowest position of the range indicating the detected object, i.e., the vertical positional relationship in the captured image of the ground contact position. The determination unit 112 acquires the vehicle's traveling direction and vehicle movement speed via a CAN (Controller Area Network) provided in the vehicle. The determination unit 112 may acquire the vehicle's traveling direction and vehicle movement speed based on positioning information received by a GNSS (Global Navigation Satellite System) receiving unit (not shown).

The determination unit 112 obtains the object's position relative to the vehicle's traveling direction, the object's traveling direction, the object's moving speed, and the object's relative speed to the vehicle from the transition of the object's position in the captured image in addition to the vehicle's traveling direction and vehicle moving speed.

The determination unit 112 determines (calculates) the collision risk level, which is expressed as a level indicating the possibility of a collision between the vehicle and the object, based on information on the movement of the vehicle and the object. The collision risk level determined by the determination unit 112 may be determined to be high if there is no actual possibility of a collision between the vehicle and the object, but if the vehicle is in a state where there is a high need for caution. Therefore, the collision risk level can also be replaced with the impact on the vehicle, the level of caution required, etc. A state where the impact on the vehicle is high is a state where the vehicle must slow down due to the positional relationship of the vehicle to the object, etc. A state where the level of caution required is high is a state where the driver of the vehicle needs to be careful of a person, such as when a detected person is not moving but is in a position where there is a possibility of crossing the road, etc.

The tracking processing unit 113 performs tracking processing of the detected object. The tracking processing unit 113 selects a tracking method having tracking performance based on the collision risk of the detected object, and performs tracking processing. That is, the tracking processing unit 113 selects a tracking method having tracking performance based on the collision risk calculated by the determination unit 112 from among a plurality of tracking methods, and performs tracking processing using the selected tracking method. As a result, when the collision risk with the object is high, a tracking method with high tracking performance is selected, and when the collision risk with the object is low, a tracking method with low tracking performance is selected. That is, an appropriate tracking method based on the collision risk is selected. The tracking processing unit 113 performs tracking processing of the object using the tracking method selected based on the collision risk with the object. The tracking method is sometimes called a tracking algorithm.

In detail, when the level indicating a high collision risk of the object is equal to or higher than a predetermined threshold, the tracking processing unit 113 selects a tracking method with high tracking performance and performs tracking processing of the object using the selected tracking method. When the level indicating a high collision risk of the object is less than a predetermined threshold, the tracking processing unit 113 selects a tracking method with low tracking performance and performs tracking processing of the object using the selected tracking method, or stops tracking processing. Tracking is sometimes referred to as chasing or tracking.

The tracking processing unit 113 will now be described in more detail. The tracking processing unit 113 applies either a tracking method with high tracking performance or a tracking method with low tracking performance to the object detected by the object detection unit 111, depending on the collision risk (impact) calculated by the determination unit 112. Note that the tracking method with high tracking performance imposes a relatively large processing load, while the tracking method with low tracking performance imposes a relatively small processing load.

When the edges of other objects present in the background of the object to be detected are not complex, or when the object to be detected is not overlapped by other objects, the tracking of the object to be detected is unlikely to be interrupted, so there is no problem even if the tracking processing is performed using a tracking method with low tracking performance.

On the other hand, when the tracking processing of an object by the tracking processing unit 113 is performed in a case where the edges of other objects present in the background of the object to be detected are complex, or when the object to be detected overlaps with other objects, the tracking of the object to be detected is likely to be interrupted by a tracking method with low tracking performance. Therefore, in order to perform uninterrupted tracking in such cases, it is preferable for the tracking processing unit 113 to apply a tracking method with high tracking performance. In particular, it is dangerous for tracking of an object with a high collision risk with the vehicle to be interrupted. For this reason, for an object with a high collision risk with the vehicle, a tracking process with high tracking performance is performed regardless of the state of the object's background, etc., to prevent interruption of tracking of the object. In addition, the tracking processing unit 113 may not perform tracking processing for an object with a low collision risk with the vehicle, instead of performing tracking processing using a tracking method with low tracking performance. For example, tracking processing may not be performed for an object located in a place different from the traveling direction of the vehicle or an object located in a distant place different from the traveling direction of the vehicle.

Next, examples of tracking methods selectively used by the tracking processing unit 113 will be described below.
<Example of a tracking method with high tracking performance>
Boosting method, MIL (Multiple Instance Learning) method, KCF (Kernelized Correlation Filter) method, TLD (Tracking Learning Detection) method, etc.
<Examples of tracking methods with poor tracking performance>
Kalman filter, optical flow, mean shift method, etc.

The processing by the tracking processing unit 113 is summarized below. When the risk of collision with an object is high, the tracking processing unit 113 performs tracking processing of the object using a tracking method with high tracking performance. When the risk of collision with an object is low, the tracking processing unit 113 performs tracking processing of the object using a tracking method with low tracking performance, or does not perform tracking processing.

Returning to FIG. 1, the display control unit 114 of the object detection device 11 causes the display unit 13 to display, in addition to the captured image captured by the imaging unit 12, a detection frame surrounding the object detected by the object detection unit 111 for which tracking processing is being performed. This makes it possible to inform the driver of the vehicle of the presence of the detected object.

When the level indicating a high collision risk of an object is equal to or higher than a predetermined threshold, the display control unit 114 may display a warning message on the display unit 13 to inform the driver of the danger. The warning message may be issued as sound.

As described above, according to the embodiment, it is possible to provide an object detection device and method capable of tracking an object using a tracking method that has tracking performance based on the collision risk of the object.

＜Effects＞
FIG. 2 is a schematic diagram illustrating a display screen of a display unit according to an embodiment. Time passes from the top to the bottom of FIG. 2. That is, this is an example showing the process in which the object detection system 10 provided in the vehicle detects a person and draws and displays a detection frame around the detected person. In FIG. 2, the outer frame displayed to surround the person is the detection frame that is displayed when the person is detected. The detection frame is drawn to include the detected person as a result of the person being detected. Therefore, when tracking of a person is interrupted, the drawing of the detection frame is also interrupted during the period when tracking of the person is interrupted.

The person shown in FIG. 2 is in the direction of travel of the vehicle, and therefore the determination unit 112 has determined that there is a high risk of collision with the person. Specifically, the risk of collision with the person is determined to be equal to or greater than a predetermined threshold based on the position and direction of movement of the person, in addition to the direction of travel and speed of the vehicle. Therefore, a tracking method with high tracking performance is applied to the person shown in FIG. 2. This makes it possible to track a person (object) even when the person is in a position with other vehicles in the background, that is, when the edge shape including the person and its surroundings is complex. In particular, even when the person overlaps with another vehicle at the bottom of FIG. 2, detection of the person and tracking of the person can be performed without interruption.

In addition, in the embodiment, the object detection device 11 does not always perform tracking processing with high tracking performance, so the processing load can be reduced. Furthermore, the object detection device 11 performs tracking processing with high tracking performance only for objects that have a high risk of collision with the vehicle, so it can properly warn the driver.

In addition, the object detection device 11 performs tracking processing with low tracking performance (slow processing speed) (or does not perform tracking processing) for objects that have a low risk of collision with the vehicle. This allows the object detection device 11 to reduce the processing load compared to when performing tracking processing with high tracking performance.

<Operation>
3 is a flowchart illustrating the operation of the object detection device according to the embodiment. The process shown in FIG. 3 is started when a condition that requires object detection using a far-infrared camera or a visible light camera is met, such as when the engine or power of a vehicle equipped with the object detection system 10 is turned on, or when the headlights of the vehicle are turned on.

As shown in FIG. 3, when processing starts, the object detection unit 111 starts acquiring an image captured by the imaging unit 12, which captures images of the outside of the vehicle, from the imaging unit 12 (step S101). Also, when processing starts, the object detection unit 111 starts detecting an object (person/cyclist/animal, etc.) from the image captured by the imaging unit 12 (input image) (step S101). The object detection unit 111 detects where an object is located in the captured image by using, for example, HOG (Histograms of Oriented Gradients) features or CNN (YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), etc.) as a method for detecting (detecting) an object.

The object detection unit 111 determines whether or not an object has been detected (step S102). If an object has not been detected in step S102 (step S102: No), the process proceeds to step S107. If an object has been detected in step S102 (step S102: Yes), the process proceeds to step S103.

If an object is detected in step S102, the determination unit 112 determines the collision risk level, which indicates the level of risk of a collision between the vehicle and the object (step S103). That is, the determination unit 112 determines the possibility that the detected object will come into contact with the vehicle, that is, the collision risk level.

In step S103, the determination unit 112 acquires information such as the vehicle speed and steering angle, and information from an acceleration sensor, etc., via the vehicle's CAN (Controller Area Network) in order to grasp the vehicle's behavior. The determination unit 112 also detects the distance to the detected object from the grounding position of the object or by using a distance sensor, etc. The determination unit 112 determines the movement direction and movement speed of the detected object from this time-series information (data). The determination unit 112 determines the impact of the possibility of a collision between the detected object and the vehicle (collision risk) based on one or a combination of the following conditions:

- The detected object is in front of the vehicle or is approaching the front.
The detected object is present in the direction of travel of the vehicle or is moving in the direction of travel.
- The movement speed of the detected object is above a certain value.
- The vehicle's movement speed is above a certain value.
The relative speed between the detected object and the vehicle is equal to or greater than a certain value.
The predicted time to collision between the detected object and the vehicle (time-to-collision) is below a certain value.
- The distance to the detection object is less than a certain value.

The determination unit 112 scores the above-mentioned conditions for each detected object, calculates the collision risk, and determines whether the level indicating a high collision risk is equal to or higher than a threshold. The determination unit 112 may also determine the level of collision risk from the position and movement direction of the detected object relative to the vehicle's traveling direction, without quantifying the collision risk.

After determining the collision risk with the detected object in step S103, the tracking processing unit 113 determines whether the determined collision risk is equal to or greater than a predetermined threshold (step S104). The determination in step S104 may be a determination of whether the collision risk is high.

If it is determined in step S104 that the collision risk is equal to or greater than the predetermined threshold (step S104: Yes), the tracking processing unit 113 selects a tracking method with high tracking performance that is equal to or greater than the predetermined tracking performance, and starts the object tracking process (step S105).

When the tracking processing unit 113 determines that the collision risk of the object is not equal to or greater than the predetermined threshold, i.e., is less than the predetermined threshold (step S104: No), it selects a tracking method with low tracking performance that is below the predetermined tracking performance and starts tracking processing of the object, or stops tracking processing (step S106).

The tracking process executed in step S105 or step S106 detects the position of the object to be tracked in the captured image in the next frame of the captured image or later, if an object is detected. The tracking process continues until the object to be tracked can no longer be detected.

After steps S105 and S106, the object detection device 11 determines whether or not to end the process (step S107). The process ends when a condition for ending object detection using the far-infrared camera or visible light camera is met, such as the engine or power of the vehicle equipped with the object detection system 10 being turned off, or the headlights of the vehicle being turned off.

<Features>
Here, the features of the embodiment will be described.
In an object detection system 10, an imaging unit 12 such as an on-board camera detects an object such as a person, a cyclist, or an animal, and issues a warning to a driver when it determines that there is a risk of collision.
The system determines the risk of collision based on the detected object's location and the vehicle's location/speed, etc.
For objects with a high collision risk, a highly accurate tracking process is performed.
For objects with a low collision risk, tracking processing with low tracking performance is performed, or no tracking processing is performed at all.

[Comparative Example]
FIG. 4 is a schematic diagram illustrating a display screen of a display unit according to a comparative example of the embodiment.
Time passes from the top to the bottom of Fig. 4. That is, this is an example showing the process in which the object detection system provided in the vehicle detects a person and draws and displays a detection frame around the detected person.
In FIG. 4, the outer frame displayed so as to surround a person is a detection frame that is displayed when that person is detected.

The object detection device in the comparative example does not have the function of selecting a tracking method having tracking performance based on the collision risk of the object and performing object tracking processing using the selected tracking method.

Therefore, as shown in FIG. 4, the object detection device according to the comparative example does not select a tracking method with high tracking performance even if the detected person is in the direction of travel of the vehicle, and does not use a tracking method with high tracking performance. As a result, in the object detection device according to the comparative example, for example, when a person is located in a position with other vehicles in the background, that is, when the edge shape including the person and its surroundings is complex, the tracking process is interrupted. In other words, this means that tracking of the person is being performed in the top row of FIG. 4, but tracking of the person is interrupted in the bottom row of FIG. 4.

In the comparative example shown in FIG. 4, even though the detected person is at high risk of collision or requires caution, there are periods when tracking is interrupted and the detection frame is not displayed. For example, in cases where it is difficult to find a person visually, such as at night, the interruption of tracking can lead to the driver of the vehicle being unable to grasp the person's condition. The present invention solves the problems that arise in situations such as the comparative example.

Some or all of the above processes may be executed by a computer program. The above-mentioned programs can be stored and provided to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be provided to the computer by various types of temporary computer readable media. Examples of temporary computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer readable medium can provide the program to the computer via a wired communication path such as an electrical wire or optical fiber, or via a wireless communication path.

The invention has been specifically described above based on the embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention.

REFERENCE SIGNS LIST 10 object detection system 11 object detection device 111 object detection section 112 determination section 113 tracking processing section 114 display control section 12 imaging section 13 display section

Claims (5)

an object detection unit that acquires an image captured by an imaging unit that captures an image of the outside of the vehicle and detects an object present in the captured image;
a determination unit for determining a collision risk level indicating a degree of risk of a collision between the vehicle and the object;
a tracking processing unit that selects a tracking method having a tracking performance based on the collision risk of the object, and performs tracking processing of the object using the selected tracking method;
An object detection device comprising: the determination unit determines the collision risk based on at least one of a distance from the vehicle to the object, a position of the object with respect to a traveling direction of the vehicle, a traveling direction of the object, a moving speed of the object, a relative speed of the object with respect to the vehicle, a traveling direction of the vehicle, a moving speed of the vehicle, and a predicted collision time until the vehicle collides with the object, or a combination of several of these.
The object detection device according to claim 1 . The tracking processing unit includes:
When the collision risk of the object is equal to or greater than a predetermined threshold, the tracking method having a predetermined tracking performance or higher is selected to perform the tracking process of the object;
When the collision risk of the object is less than the predetermined threshold, the tracking method having a tracking performance less than a predetermined value is selected to perform the tracking process of the object, or the tracking process is not performed.
The object detection device according to claim 1 . The tracking processing unit is
When the collision risk of the object is equal to or greater than a predetermined threshold, the tracking method having a large processing load is selected to perform the tracking process of the object;
When the collision risk of the object is less than the predetermined threshold, the tracking method having a small processing load is selected to perform the tracking process of the object, or the tracking process is not performed.
The object detection device according to claim 3 . acquiring an image captured by an imaging unit that captures an image of the outside of the vehicle, and detecting an object present in the captured image;
determining a collision risk indicating a degree of risk of a collision between the vehicle and the object;
selecting a tracking method having a tracking performance based on the collision risk of the object, and performing a tracking process of the object using the selected tracking method;
The object detection method is executed by the object detection device.

Images