WO2025013285A1 - Object detection device - Google Patents

Abstract

This object detection device for a vehicle comprises at least one processor (105a), wherein the processor: calculates the width and height of an object from detection results of a stereo camera (101) mounted on a vehicle (201); calculates an amount of change in each of the width and height of the object obtained from the stereo camera; determines a reference feature quantity to be used to calculate the relative distance between the vehicle and the object, from among the width and height of the object calculated from detection results of a monocular camera (102) mounted on the vehicle, on the basis of the amounts of change in the width and height of the object obtained from the stereo camera; calculates the reference feature quantity from detection results of the monocular camera when the object moves from the recognition range (202-S) of the stereo camera to the recognition range (203-M) of the monocular camera; and calculates the relative distance on the basis of the reference feature quantity and the feature quantity that is among the width and height of the object obtained from the stereo camera and that corresponds to the reference feature quantity.

Description

Object detection device

The present invention relates to an object detection device for vehicles.

There has been active development of driving support technologies that ensure safety by equipping automobiles with various sensors and monitoring the area around the vehicle. For example, there are lane departure warning systems that use cameras to monitor the vehicle ahead and recognize white lines to warn the driver not to deviate from the lane, and collision mitigation brakes that monitor obstacles ahead and automatically apply the brakes if there is a possibility of a collision.

These support technologies are expanding beyond just the front of the vehicle to monitor the rear, sides, etc. Accordingly, detection methods are no longer limited to cameras; radar, sonar, and other devices are appropriately combined to monitor objects that require the vehicle's attention. This is called FUSION technology, and it is important to combine sensors and optimize the recognition technology processing of each sensor. Furthermore, since each sensor has different monitoring directions, ranges, accuracy, and environmental conditions, it may become necessary to hand over the monitoring of an object (target) from one sensor to another when the target is approaching the vehicle.

For example, the technology described in Patent Document 1 monitors the rear of a vehicle using a stereo camera and a monocular camera. The width (or height) of a following vehicle in the recognition range of the stereo camera is determined and stored, and when the following vehicle moves from the recognition range of the stereo camera into the recognition range of the monocular camera, the distance to the following vehicle is determined based on the stored width (or height) and the width (or height) of the following vehicle determined from the recognition result of the monocular camera.

JP 2008-123462 A

However, when an object to be detected moves from the recognition range of a first sensor to the recognition range of a second sensor in such a way that the object's attitude relative to each sensor changes, the width and height of the object detected by each sensor (apparent dimensions as seen by each sensor) may be perceived as having changed significantly.

For example, the width of a motorcycle seen from the front is narrower than that of an automobile, but when the motorcycle makes a lane change or other movement that makes the side visible to each sensor, the apparent change in vehicle width at each sensor tends to be greater than that of an automobile. Under such circumstances, if the technology of Patent Document 1 is used to calculate distance based on the vehicle width from the recognition results of a monocular camera, the monocular camera may recognize the vehicle width as having changed significantly more than the vehicle width obtained from the recognition results of a stereo camera, and the distance to the object may be calculated as a value closer than the actual value. Conversely, if the monocular camera recognizes the vehicle width as having changed slightly, the distance to the object may be calculated as a value farther than the actual value. In other words, the distance calculation based on Patent Document 1 may cause differences in the distance to the actual object, and there is room for improvement. Note that this problem is due to the characteristics of a monocular camera, which is prone to distance errors due to dimensional changes in the image of the object, and can be pointed out not only in the case of changes in the width (vehicle width) of the object pointed out above, but also in the case of the height of the object detected by the sensor as having changed in appearance.

The object of the present invention is to provide an object detection device that can prevent the calculated distance between the vehicle and a detected object from deviating from the actual distance when distance calculation is handed over from the first sensor to the second sensor.

The present application includes multiple means for solving the above problem, and an example thereof is an object detection device for a vehicle having at least one processor, the processor calculates multiple feature quantities of an object from the detection result of a first sensor mounted on the vehicle, calculates changes in the multiple feature quantities obtained from the first sensor, determines a reference feature quantity to be used in calculating the relative distance between the vehicle and the object from the feature quantities of the object calculated from the detection result of a second sensor mounted on the vehicle based on the changes in the multiple feature quantities, calculates the reference feature quantity from the detection result of the second sensor when the object moves from the recognition range of the first sensor to the recognition range of the second sensor, and calculates the relative distance based on the reference feature quantity and the feature quantity corresponding to the reference feature quantity from the multiple feature quantities obtained from the first sensor.

The present invention makes it possible to prevent the calculated distance between the vehicle and a detected object from deviating from the actual distance when distance calculation is handed over from the first sensor to the second sensor.

1 is a configuration diagram including an object detection device according to a first embodiment of the present invention and its peripheral devices. FIG. 1 is a diagram showing an example of the locations where a stereo camera 101 and a monocular camera 102 for monitoring the rear are installed. FIG. 2 is a software configuration diagram of the ECU 105. FIG. 1 is an explanatory diagram of a two-dimensional bounding box. 1 shows a processing flow of rear monitoring executed by the ECU 105 (processor 105a) according to the first embodiment. 4A to 4C are diagrams illustrating the operation of rear monitoring according to the first embodiment; 13 shows a process flow of rear monitoring executed by the ECU 105 (processor 105a) according to the second embodiment. FIG. 13 is a diagram illustrating an example of a weighting map. FIG. 13 is a diagram illustrating an example of a weighting map. FIG. 13 is a diagram illustrating an example of a weighting map. FIG. 1 is an explanatory diagram of a three-dimensional bounding box. FIG. 11 is a diagram showing a processing flow of rear monitoring according to the second embodiment. FIG. 1 is a diagram showing an example of the locations where a stereo camera 101 and a monocular camera 102 for monitoring the forward direction are installed. FIG. 13 is a diagram illustrating the operation of forward monitoring according to the fourth embodiment.

Below, an embodiment of the present invention will be described with reference to the drawings. The subject of this embodiment is a detection device for detecting objects approaching a vehicle, such as an automobile or motorcycle, while the vehicle is traveling on a road. Note that, although multiple embodiments will be described below, parts common to each embodiment will be given the same reference numerals, and repeated explanations may be omitted. Note that, to make the explanation clearer, the drawings may be shown more diagrammatically than the actual aspect, but this is merely an example and does not limit the interpretation of the present invention.

First Embodiment
FIG. 1 is a block diagram including an object detection device according to a first embodiment of the present invention and its peripheral devices. The object detection device according to this embodiment includes a first sensor 101, a second sensor 102, a first processing circuit 103, a second processing circuit 104, and an electronic control device (hereinafter, sometimes referred to as ECU) 105. The ECU 105 can be connected to a vehicle control device (another ECU) 107 and an alarm device (monitor, speaker, etc.) 108 via a CAN interface (CAN IF) 106. The first sensor 101 is a sensor mounted on the host vehicle to monitor the rear of the host vehicle, and is any one of a stereo camera, a radar, a sonar, a LiDAR, and a monocular camera. In this embodiment, the first sensor 101 is a stereo camera. The camera image of the stereo camera 101 is image-processed by the first processing circuit 103 and input to the ECU 105.

The second sensor 102 is a sensor mounted on the vehicle to monitor the rear side of the vehicle, and is preferably a monocular camera. The camera image of the monocular camera 102 is processed by the second processing circuit 104 and input to the ECU 105.

The ECU 105 is equipped with a memory 105b which is a data storage device, and at least one processor 105a which executes various processes according to the programs stored in the memory 105b, and processes information on an object (e.g., another vehicle) approaching the vehicle using the processing results of the first processing circuit 103 and the second processing circuit 104 (in other words, the detection results of the first sensor 101 and the second sensor 102) within the ECU 105. This information includes relative distance data between the vehicle and the object. Then, information on an object approaching from the rear side of the vehicle is sent to the CAN bus via the CAN IF 106, and output to the vehicle control device 107 and the alarm device 108.

Here, a system including signal input units (not shown) that input signals from the stereo camera 101 and the monocular camera 102, the first processing circuit 103 and the second processing circuit 104, the ECU 105, and the CAN IF 106 is shown as an example, but these are not necessarily devices contained within a single unit. For example, the cameras 101 and 102 and the corresponding processing circuits 103 and 104 may each exist as a single unit, and be connected to some control device including the ECU 105 by a signal line. Also, the first processing circuit 103 and the second processing circuit 104 may be contained within the ECU 105, or the first processing circuit 103, the second processing circuit 104, and the ECU 105 may be contained within the vehicle control device 107.

Figure 2 shows an example of the installation locations of the stereo camera 101 and monocular camera 102 for rear monitoring. The stereo camera 101 can be installed, for example, on the rear bumper to monitor the rear of the vehicle 201. On the other hand, the monocular camera 102 can be installed, for example, on the door mirror to monitor the rear side of the vehicle. Area 202-S in the figure indicates the recognition area of the stereo camera 101, and area 203-M indicates the recognition area of the monocular camera 102. Here, the recognition area may be the camera viewing angle or angle of view, or it may indicate an area within the camera viewing angle that can be processed by the image processing unit. However, it is desirable to install the monocular camera 102 so that the optical axis is as far away from the vehicle 201 as possible so that the vehicle 201 is not reflected as much. Furthermore, the stereo camera 101 has a range that can be viewed in stereo, which is the shaded area 202-S where the fields of view of the left and right cameras overlap. The stereo camera 101 may be installed inside the vehicle 201. For example, the stereo camera 101 may be installed on the ceiling on the rear seat side so that it can monitor the rear of the vehicle through the rear window. This can prevent water droplets, mud, etc. from adhering to the stereo camera 101 in rainy weather.

FIG. 3 is a diagram showing the software configuration of the ECU 105 inside the ECU 105. The ECU 105 can function as a first feature amount calculation unit 301, a second feature amount calculation unit 302, a sensor recognition area movement determination unit 303, an object distance estimation unit 304, and a collision risk determination unit 305 by executing software stored in the memory 105b with the processor 105a.

The first feature amount calculation unit 301 (processor 105a) assigns a recognition ID to the following vehicle traveling behind the vehicle detected in the recognition area 202-S of the stereo camera 101, calculates multiple feature amounts (width and height) of the following vehicle from the detection result (camera image) of the stereo camera 101, and calculates the amount of change in the multiple feature amounts caused by the movement of the following vehicle. The first feature amount calculation unit 301 can also determine, from the feature amounts of the following vehicle calculated from the detection result of the monocular camera 102, a feature amount (hereinafter sometimes referred to as a "reference feature amount") to be used in calculating the relative distance between the vehicle and the following vehicle, based on the amount of change in the multiple feature amounts caused by the movement of the following vehicle. The reference feature amount is a feature amount obtained by the monocular camera 102 that is used in calculating the relative distance when the following vehicle is present in the recognition area 203-M of the monocular camera 102.

The feature amount of the following vehicle can be calculated, for example, using bounding box detection, which is known as a general technique for detecting objects. The first feature amount calculation unit 301 (processor 105a) uses deep learning or the like to assign a frame line (called a two-dimensional bounding box) 402, which is displayed as the smallest square (rectangle) circumscribing the object (following vehicle 401) in the camera images of the stereo camera 101 and the monocular camera 102, as shown in FIG. 4. The first feature amount calculation unit 301 (processor 105a) can then calculate the width and height (feature amount) of the following vehicle from the width and height of the two-dimensional bounding box. The first feature amount calculation unit 301 (processor 105a) can also calculate the amount of change in the width and height (feature amount) of the following vehicle by monitoring the change in the width and height of the two-dimensional bounding box caused by the movement of the following vehicle. The calculated feature amount and the data on the amount of change are used, for example, by the sensor recognition area movement determination unit 303 and the object distance estimation unit 304. Note that instead of the width and height of the bounding box, the width and height of the following vehicle may be calculated from, for example, the edges of the following vehicle in the camera image of the stereo camera 101.

The second feature amount calculation unit 302 (processor 105a) calculates multiple feature amounts (width and height) of the following vehicle traveling to the rear side of the host vehicle, detected in the recognition area 203-M of the monocular camera 102, from the detection result (camera image) of the monocular camera 102. The feature amounts can be calculated in the same way as the first feature amount calculation unit 301, and can be calculated, for example, based on a two-dimensional bounding box. The calculated feature amounts include reference feature amounts. The calculated feature amount data is used, for example, by the sensor recognition area movement determination unit 303 and the object distance estimation unit 304.

The sensor recognition area movement determination unit 303 (processor 105a) determines whether the same following vehicle has moved from the recognition area 202-S of the stereo camera 101 to the recognition area 203-M of the monocular camera 102, and if it determines that the same following vehicle has moved, it assigns the following vehicle in the recognition area 203-M the same recognition ID as when it was in the recognition area 202-S (inherits the recognition ID).

When the sensor recognition area movement determination unit 303 determines that the following vehicle has moved from the stereo camera recognition area 202-S to the monocular camera 102 recognition area 203-M, the object distance estimation unit 304 (processor 105a) calculates the relative distance between the following vehicle and the vehicle based on a reference feature value used to calculate the relative distance between the following vehicle and the vehicle from among the feature values of the following vehicle obtained from the monocular camera 102, and a feature value corresponding to the reference feature value from among the multiple feature values of the following vehicle obtained from the stereo camera 101.

The collision risk determination unit 305 (processor 105a) determines whether there is a possibility of a collision between the host vehicle and the following vehicle from the relative distance between the host vehicle and the following vehicle calculated by the object distance estimation unit 304, and transmits permission or denial of collision avoidance control by the vehicle control device 107 based on the determination result via the CAN IF 106. If the collision risk determination unit 305 determines that there is a possibility of a collision based on the determination result, it may transmit a message to that effect to the alarm device 108.

Figure 5 shows the flow of processing for rear monitoring executed by the ECU 105 (processor 105a) according to the first embodiment. First, when a vehicle following the host vehicle is detected by the stereo camera 101, the processor 105a assigns a recognition ID to the following vehicle and calculates the feature values (width and height) of the following vehicle based on the camera image of the stereo camera 101 (S501). The processor 105a then calculates the amount of change in the feature values (width and height) that occurs due to the movement of the following vehicle, such as a lane change, based on the camera image of the stereo camera 101 (S502).

Next, based on the amount of change in the width and height of the following vehicle, processor 105a determines a reference feature (i.e., one of the width and height of the following vehicle) to be used in the calculation of the relative distance from the width and height (feature) of the following vehicle obtained from monocular camera 102 in the subsequent processing (S503). In this embodiment, processor 105a selects as the reference feature the feature (width or height) with the least amount of change from the feature of the following vehicle obtained from stereo camera 101 until just before the following vehicle goes out of recognition area 202-S of stereo camera 101 (S503). Thereafter, the feature corresponding to the feature selected in S503 from the feature of the following vehicle obtained from monocular camera 102 is used in the calculation of the relative distance in S506. That is, for example, if it is determined in S503 that the amount of change in height is small among the features captured by the stereo camera 101, the height captured by the monocular camera 102 is used as a feature in the calculation of the relative distance in S506 using the image from the monocular camera 102.

Note that the method of selecting a reference feature (a feature used when calculating relative distance) from an image of the monocular camera 102 described here is just one example, and other selection methods may be used as long as the spirit of this embodiment can be reproduced. For example, if the amount of change in the width of the feature captured by the stereo camera 101 exceeds a predetermined threshold, height may be selected as the feature captured by the monocular camera 102 and used as the reference feature.

Then, the processor 105a judges whether the following vehicle has moved out of the recognition area 202-S of the stereo camera 101 and into the recognition area 203-M of the monocular camera 102 based on the recognition ID of the following vehicle (S504), and proceeds to S505 if it is judged that the following vehicle has moved into the recognition area 203-M of the monocular camera 102. For example, if it is judged that a following vehicle with a certain recognition ID that was detected in the recognition area 202-S of the stereo camera 101 has moved out of the recognition area 202-S, and a following vehicle is detected in the recognition area 203-M of the monocular camera 102 shortly thereafter, the same recognition ID is passed on to the following vehicle, and it is deemed that the following vehicle with that recognition ID has moved from the recognition area 202-S to the recognition area 203-M.

Next, after determining that the following vehicle has moved into the recognition area 203-M of the monocular camera 102, the processor 105a calculates the feature quantities (width and height) of the following vehicle based on the camera image of the monocular camera 102 (S505). Note that since the feature quantities used to calculate the relative distance have already been determined in S503, here it is also possible to calculate only the determined feature quantities from the camera image of the monocular camera 102.

Then, the processor 105a calculates the relative distance to the following vehicle (object) located to the rear side using the feature calculated in S505 that corresponds to the feature determined in S503 and one of the following formulas (1) and (2) (S506). Whether formula (1) or (2) is used to calculate the relative distance depends on the feature determined in S503. If the determined feature is "width", formula (1) is used, and if it is "height", formula (2) is used.

The following formula (1) is a formula for calculating the relative distance (first relative distance) dw when the feature determined in S503 is "width". In formula (1), dw is the relative distance (first relative distance) from the host vehicle to the following vehicle (object), f is the focal length of the monocular camera 102, W is the width of the following vehicle (object) calculated by the stereo camera 101, and △x is the width of the following vehicle (object) in the image of the monocular camera 102.

The following formula (2) is a formula for calculating the relative distance (second relative distance) dh when the feature determined in S503 is "height." In formula (2), dh is the relative distance (second relative distance) from the host vehicle to the following vehicle (object), f is the focal length of the monocular camera 102, H is the height of the following vehicle (object) calculated by the stereo camera 101, and △y is the height of the following vehicle (object) in the image of the monocular camera 102.

Then, the processor 105a determines whether or not there is a possibility that the vehicle on the rear side will collide with the vehicle on the rear side based on the calculation results of the relative distances dw, dh between the vehicle and the vehicle on the rear side (S507). If it is determined that there is a possibility of a collision, it makes a decision to permit control intervention to avoid the collision and transmits the decision (S508). If it is determined that there is no possibility of a collision, it makes a decision to prohibit control intervention to avoid the collision and transmits the decision (S509).

(Operation)
6 is a diagram illustrating the operation of rear monitoring according to this embodiment. Reference numerals 401-404 in the diagram indicate the positions of a vehicle (motorcycle) following the host vehicle 201, and the following vehicle moves from position 401 to position 404 after changing lanes.

The stereo camera 101 detects a following vehicle at position 401 behind the vehicle, and the processor 105a (first feature amount calculation unit 301) calculates the feature amounts (width and height) of the following vehicle and the amount of change therein from position 401 through position 402 until the following vehicle leaves the recognition area 202-S (S501, 502). At this time, the processor 105a (first feature amount calculation unit 301) may also calculate the relative speed and relative position between the vehicle 201 and the following vehicle. When the stereo camera 101 detects a following vehicle, a recognition ID is assigned to the following vehicle.

Next, the processor 105a (first feature calculation unit 301) determines the feature (width or height) that will be used as a reference for calculating the relative distance using the camera image of the monocular camera 102, depending on the change in the feature (width and height) of the following vehicle in the recognition area 202-S that includes the area from position 401 to position 402 (S503).

Next, assume that a following vehicle traveling at position 401 behind the vehicle 201 starts changing lanes and moves to position 403 to the rear side of the vehicle 201. At that time, the processor 105a (sensor recognition area movement determination unit 303) determines that the following vehicle has moved from recognition area 202-S to recognition area 203-M by the stereo camera 101 and the monocular camera 102 (S504). Whether the following vehicle detected by the stereo camera 101 and the monocular camera 102 is the same or not can be determined, for example, if the relative position difference between the following vehicles detected simultaneously by both the stereo camera 101 and the monocular camera 102 is within a predetermined value, it can be determined that it is the same following vehicle. If it is the same following vehicle, the recognition ID is also passed on.

Next, when the following vehicle at position 403 on the rear side of the vehicle 201 is detected by the monocular camera 102, the processor 105a (second feature amount calculation unit 302) calculates data related to the following vehicle. The data calculated at this time includes the feature amounts (width and height) of the following vehicle calculated from the camera image of the monocular camera 102 (S505).

The processor 105a (object distance estimation unit 304) calculates the relative distances dw, dh between the following vehicle at position 403 and the host vehicle 201 based on, for example, the characteristic amounts (width Δx and height Δy) of the following vehicle captured by the monocular camera 102 at position 403, the characteristic amounts (width W or height H) of the following vehicle captured by the stereo camera 101 selected in S503, and the above formula (1) or (2) (S506).

The presence or absence of a collision is determined from the calculation results of the relative distances dw, dh between the following vehicle 403 and the vehicle 201 (S507). The result of this determination is transmitted from the CAN IF 106 to the vehicle control device 107 and the alarm device 108 (S508, S509). Note that monitoring by the monocular camera 102 (calculation of the relative distance) may continue even if the following vehicle moves to position 404 and completes the lane change.

(effect)
(1) As described above, in this embodiment, the processor 105a calculates a plurality of feature amounts (width and height) of an object (following vehicle) from the detection result of the first sensor (stereo camera 101) mounted on the host vehicle 201, calculates the amount of change in each of the plurality of feature amounts (width and height) obtained from the first sensor (stereo camera 101), and calculates the relative distance between the host vehicle 201 and the object (following vehicle) from among the feature amounts of the object calculated from the detection result of the second sensor (monocular camera 102) mounted on the host vehicle 201. A reference feature used in calculating the distance is determined based on the amount of change in the multiple feature values. When the object (following vehicle) moves from recognition range 202-S of the first sensor (stereo camera 101) to recognition range 203-M of the second sensor (monocular camera 102), the reference feature value is calculated from the detection result of the second sensor (monocular camera 102), and the relative distance is calculated based on the reference feature value and one of the multiple feature values obtained from the first sensor (stereo camera 101) that corresponds to the reference feature value.

In this way, when the relative distance in the recognition range 203-M of the second sensor (monocular camera 102) is calculated based on the feature amount (reference feature amount) determined from the change amount of the feature amount of the object obtained from the first sensor (stereo camera 101) among the feature amounts of the object obtained from the second sensor (monocular camera 102) and the feature amount of the object obtained from the first sensor (stereo camera 101), the relative distance can be calculated based on the feature amount (reference feature amount) corresponding to the change amount of the feature amount obtained from the first sensor among the feature amounts obtained from the second sensor while using the accurate feature amount of the object obtained from the first sensor (stereo camera 101), so that the calculation accuracy of the relative distance by the second sensor can be improved. In other words, it is possible to suppress the deviation of the relative distance between the host vehicle and the following vehicle from the actual distance when the distance calculation is handed over from the first sensor (stereo camera 101) to the second sensor (monocular camera 102). In addition, since the accuracy of the relative distance calculation is improved, it is possible to reduce the malfunction or non-operation of the collision avoidance operation.

It is preferable that the feature of the object calculated and used in the calculation (1) above is related to the dimensions of the object.

(2) In the above (1), the first sensor is preferably a stereo camera, radar, sonar, LiDAR, or monocular camera, and the second sensor is preferably a monocular camera. Note that, instead of the stereo camera 101, any of the sensors radar, sonar, LiDAR, or monocular camera may be used, but when considering the balance between monetary cost and calculation accuracy, it is most preferable to use the stereo camera 101 (the same applies to each of the following embodiments).

(3) In the above (1), it is preferable that the multiple feature amounts are the width and height of the object, and the processor 105a determines the reference feature amount based on the amount of change in the width and height of the object obtained from the stereo camera 101 (first sensor).

(4) In (1) above, it is preferable that the multiple features are the width and height of the object, and the processor 105a determines the feature among the width W and height H of the object that has the least amount of change as the reference feature based on the amount of change in the width W and height H of the object obtained from the first sensor. Specifically, in this embodiment, the processor 105a calculates the width W and height H of the following vehicle (object) from the detection results of the stereo camera (first sensor) 101 mounted on the host vehicle 201, calculates the change in the width W and height H of the following vehicle obtained from the stereo camera 101, and when the following vehicle moves out of the recognition range 202-S of the stereo camera 101 and into the recognition range 203-M of the monocular camera (second sensor) 102 mounted on the host vehicle 201, selects the feature value of the width and height of the following vehicle (object) that has the least change based on the change in the width W and height H of the following vehicle (object) obtained from the stereo camera (first sensor) 101, and calculates the relative distances dw and dh based on the feature value (Δx or Δy) corresponding to the selected feature value out of the width Δx and height Δy of the following vehicle (object) obtained from the monocular camera (second sensor) 102.

When the object detection device is configured in this manner, when the object to be detected moves out of the recognition range 202-S of the stereo camera 101 and into the recognition range 203-M of the monocular camera 102, the relative distance is calculated based on features that change little even if the object moves due to a lane change, etc., so that the accuracy of the relative distance calculated by the monocular camera 102 can be prevented from decreasing.

In the above, the processor 105a "determines the feature with the least amount of change among the width and height of the following vehicle (object) obtained from the stereo camera (first sensor) 101 as the reference feature, and calculates the relative distances dw and dh based on the feature (Δx or Δy) corresponding to the reference feature among the width Δx and height Δy of the following vehicle (object) obtained from the monocular camera (second sensor) 102," but instead, the processor 105a may be configured to "determine the height Δy of the following vehicle (object) obtained from the monocular camera (second sensor) 102 as the reference feature when the amount of change in the width W of the following vehicle (object) obtained from the stereo camera (first sensor) 101 exceeds a predetermined threshold, and use this to calculate the relative distance dh."

When configured in this manner, the relative distance dh is calculated based on the "height," which changes less with object movement than the "width," so as described above, the accuracy of the relative distance calculated by the monocular camera 102 can be prevented from decreasing.

Second Embodiment
In this embodiment, an example will be described in which the relative distance d is calculated by weighting the relative distance calculated from equation (1) (first relative distance dw) and the relative distance calculated from equation (2) (second relative distance dh) according to the amount of change in the feature of the following vehicle calculated from the camera image of the stereo camera 101 (for example, the rate of change in the width of the following vehicle).

FIG. 7 shows the flow of the rear monitoring process executed by the ECU 105 (processor 105a) according to the second embodiment.

First, steps S501 and S502 at the beginning of the flow are the same as those in the first embodiment shown in Figure 5.

Next, the processor 105a calculates a weight from the amount of change calculated in S502 and a weighting map. FIG. 8 is an example of a weighting map. When using this weighting map, in S502, the rate of change of the width of the following vehicle as seen by the stereo camera 101 is calculated as the "amount of change". Various methods for calculating the rate of change can be used, for example, a method that calculates the ratio of the width of the following vehicle just before it leaves the recognition area 202-S to the width when the following vehicle enters the recognition area 202-S. The rate of change calculated in S502 is converted into a weight wf (0<wf≦1) of the second relative distance dh by the map in FIG. 8. In FIG. 8, the weighting is performed such that the weight wf of the second relative distance dh increases as the rate of change (amount of change) of the width of the following vehicle obtained from the stereo camera 101 increases.

Note that this weighting map based on the rate of change of the characteristic amount (width or height) is only one example, and any map can be set. In other words, other weighting may be added as long as the purpose of this embodiment can be reproduced. For example, in addition to or instead of the weighting in FIG. 8, the weighting may be different depending on the type of the following vehicle (object), as in the weighting diagram based on the type of following vehicle shown in FIG. 9. As shown in FIG. 9, for example, a motorcycle has a larger ratio of length to width than a passenger car, so it is preferable to make the weight wf of the second relative distance dh larger than that of a passenger car. Furthermore, the weighting may be changed depending on the relative distance between the host vehicle and the following vehicle (object), as in the weighting diagram based on the relative distance between the host vehicle and the following vehicle (object) shown in FIG. 10. For example, in the example of FIG. 10, the weighting may be set lighter because the rate of change of the characteristic amount (width and height) is more difficult to capture due to the influence of noise, etc., as the relative distance between the host vehicle and the following vehicle (object) increases.

S504 and 505 are the same as in the first embodiment.

In S506-W, the processor 105a first calculates the first relative distance dw and the second relative distance dh from the feature amount (width W and height H) obtained by the stereo camera 101 in S501, the feature amount (width Δx and height Δy) obtained by the monocular camera 102 in S505, and equations (1) and (2) (in the first embodiment, one of the two relative distances dw, dh was calculated, but in this embodiment, two relative distances dw, dh are calculated). Next, the processor 105a calculates the relative distance d between the host vehicle and the following vehicle based on the calculated first relative distance dw and second relative distance dh, the weight wf calculated in S503-W, and the following equation (3).

The following formula (3) is a formula for calculating the relative distance d from the weight wf and the first and second relative distances dw and dh. In formula (3), wf represents the weight of the second relative distance dh, dw represents the first relative distance calculated from formula (1) above, and dh represents the second relative distance calculated from formula (2) above. In other words, in this embodiment, the relative distance d is calculated as the average (weighted average) of the first and second relative distances, taking into account the weight wf according to the amount of change in the width of the following vehicle obtained from the stereo camera 101.

The remaining steps S507, 508, and 509 are the same as in the first embodiment.

In the above explanation, weighting was performed according to the amount of change (rate of change) in the width of the following vehicle obtained from the stereo camera 101, but weighting may also be performed according to the amount of change (rate of change) in the height of the following vehicle obtained from the stereo camera 101. In this case, it is preferable to use a weighting map that converts the amount of change (rate of change) in the height of the following vehicle into a weight hf of the first relative distance dw (proportion hf of distance measurement based on width). It is also preferable to use the following formula (4) instead of the above formula (3).

In this embodiment configured as described above, the processor 105a calculates a first relative distance dw between the host vehicle and the following vehicle (object) based on the width of the following vehicle (object) obtained from the stereo camera 101 and the width of the following vehicle (object) obtained from the monocular camera 102 (second sensor), calculates a second relative distance dh between the host vehicle and the following vehicle (object) based on the height of the following vehicle (object) obtained from the stereo camera 101 and the height of the following vehicle (object) obtained from the monocular camera 102 (second sensor), and calculates the relative distance d by weighting the first relative distance dw and the second relative distance dh according to the amount of change in the width (or height) of the following vehicle (object) obtained from the stereo camera 101 (first sensor) and summing the two.

In this way, by calculating the relative distance as a weighted average of the first relative distance dw and the second relative distance dh, the relative distance is calculated taking into account the amount of change in both the width and height of the following vehicle obtained from the stereo camera 101 (first sensor), thereby improving the accuracy of the relative distance.

In particular, in the above embodiment, the relative distance d is calculated so that the weight of the height of the following vehicle relative to its width increases as the rate of change of the width of the following vehicle increases. As a result, when the rate of change of the width of the following vehicle increases, the relative distance is calculated with an emphasis on the height, which has a relatively small rate of change, improving the accuracy of the calculation of the relative distance.

Third Embodiment
In the above embodiment, the use of a two-dimensional bounding box for object detection has been mentioned, but in this embodiment, a case where a three-dimensional bounding box (3D bounding box) is used will be described. In this embodiment, a following vehicle is detected using a 3D bounding box, and when the detection confidence level is equal to or lower than a predetermined value and the confidence level of the object detection is low, the flow of the first embodiment or the second embodiment (FIGS. 5 and 7) is executed. The detection confidence level is an index value indicating how accurate the features (width, height, length, type) of the detected object are, and here, the larger the value, the higher the confidence level. For example, if the features (width, height, length, type) of the object are continuously captured during object detection, the detection confidence level can be said to be high.

First, 3D bounding box detection will be explained here. 3D bounding box detection is object detection that uses AI (artificial intelligence) learning of objects in the camera images of the stereo camera 101 and monocular camera 102 by the first processing circuit 103 and the second processing circuit 104 to extract the features of the object, capture it three-dimensionally, and display a three-dimensional frame (3D bounding box) that surrounds the object in a cube, as shown in FIG. 11. With this 3D bounding box detection, the first feature amount calculation unit 301 and the second feature amount calculation unit 302 can obtain the feature amounts (width, height, length, type) of the object, detection confidence information, and the distance from the vehicle to the object from the detected cube frame, and assign a recognition ID.

FIG. 12 shows the processing flow for rear monitoring according to this embodiment.

First, when the following vehicle is detected by the stereo camera 101 using a 3D bounding box, the processor 105a calculates the features of the following vehicle (width, height, length, type), the distance to the following vehicle, and the detection confidence level (S551).

The processor 105a then compares the detection confidence level obtained in S551 with a predetermined value (S552), and if the detection confidence level is equal to or lower than the predetermined value, executes the processing from S501 onward in FIG. 5 or FIG. 7.

In other words, in this embodiment, the processor 105a assigns a 3D bounding box to the following vehicle (object) on the image obtained by the stereo camera 101, and if the detection confidence of the 3D bounding box is equal to or lower than a predetermined value, it performs the processing required to calculate the relative distance from S501 onward in FIG. 5 or FIG. 7. In other words, according to this embodiment, the flow in FIG. 5 or FIG. 7 is executed only when the confidence of the 3D bounding box detection is low, and the present invention can also be applied to systems that use 3D bounding boxes for object detection.

Fourth Embodiment
Next, an embodiment in which both the stereo camera 101 and the monocular camera 102 are installed in front of the host vehicle, and a vehicle ahead of the host vehicle is detected and the distance is estimated will be described.

FIG. 13 shows an example of the installation locations of the stereo camera 101 and monocular camera 102 for forward monitoring. The stereo camera 101 is installed at the front of the vehicle interior to monitor the area ahead of the vehicle 201. Meanwhile, the monocular camera 102 is installed at a position where it can monitor the area ahead of the vehicle 201 and monitor the blind spot of the stereo camera 101 located in front of the vehicle 201. The recognition areas obtained thereby are, for example, 204-S and 205-M, respectively. Here, the recognition area may be the camera viewing angle or field of view, or it may be an area within the camera viewing angle that can be processed by the image processing unit. With the stereo camera 101, the range that can be viewed in stereo is the area 204-S where the fields of view of the left and right cameras overlap. For example, the range at a predetermined distance from the front of the vehicle 201 is the blind spot of the stereo camera 101.

The hardware configuration of the object detection device is the same as that shown in Figure 1.

The software configuration of the ECU 105 is also the same as that shown in FIG. 3. However, the sensor recognition area movement determination unit 303 of this embodiment determines whether a vehicle ahead of the vehicle has entered the blind spot of the recognition area 204-S of the stereo camera 101 and moved to the recognition area 205-M of only the monocular camera 102, and if it determines that the same vehicle has moved, it assigns the following vehicle in the recognition area 205-M the same recognition ID as when it was in the recognition area 204-S (inherits the recognition ID).

The processing flow for forward monitoring executed by ECU 105 (processor 105a) is the same as that shown in Figure 5, although the following vehicle must be interpreted as the vehicle ahead.

FIG. 14 shows an explanatory diagram of the forward monitoring operation according to this embodiment. Reference numerals 601-604 in the figure indicate the positions of a vehicle (motorcycle) ahead of the host vehicle 201, and the vehicle ahead moves from position 601 to position 604 after changing course.

The stereo camera 101 detects a vehicle ahead of the host vehicle at position 601, and the processor 105a (first feature amount calculation unit 301) calculates the feature amounts (width and height) of the vehicle ahead and the amount of change therein from position 601 through position 603 until the vehicle ahead enters the blind spot of the recognition area 204-S (S501, 502). At this time, the processor 105a (first feature amount calculation unit 301) may also calculate the relative speed and relative position between the host vehicle 201 and the vehicle ahead. When the stereo camera 101 detects a vehicle ahead, a recognition ID is assigned to the vehicle ahead.

Next, the processor 105a (first feature calculation unit 301) determines the feature (width or height) that will be used as a reference for calculating the relative distance using the camera image of the monocular camera 102, depending on the change in the feature (width and height) of the vehicle ahead in the recognition area 204-S that includes the positions 601 to 603 (S503).

Next, assume that a vehicle ahead that was traveling at position 601 ahead of the vehicle 201 starts to change course and moves to position 604 ahead of the vehicle 201. At that time, the processor 105a (sensor recognition area movement determination unit 303) determines that the vehicle ahead has moved into the area where the blind spot area of recognition area 204-S and recognition area 205-M overlap in the stereo camera 101 and monocular camera 102 (S504). Whether the vehicle ahead detected by the stereo camera 101 and the monocular camera 102 are the same or not can be determined, for example, if the relative position difference between the vehicle ahead detected simultaneously by both the stereo camera 101 and the monocular camera 102 is within a predetermined value, it can be determined that the vehicle ahead is the same. If the vehicle ahead is the same, the recognition ID is also passed on.

Next, when a vehicle ahead at position 604 ahead of the vehicle 201 is detected by the monocular camera 102, the processor 105a (second feature amount calculation unit 302) calculates data related to the vehicle ahead. The data calculated at this time includes the feature amounts (width and height) of the vehicle ahead calculated from the camera image of the monocular camera 102 (S505).

The processor 105a (object distance estimation unit 304) calculates the relative distances dw, dh between the vehicle ahead at position 604 and the host vehicle 201 based on, for example, the features of the vehicle ahead captured by the monocular camera 102 at position 604 (width Δx and height Δy), the features of the vehicle ahead captured by the stereo camera 101 selected in S503 (width W or height H), and the above formula (1) or (2) (S506).

The presence or absence of a collision is determined based on the calculation results of the relative distances dw, dh between the vehicle ahead 604 and the vehicle itself 201 (S507). The result of this determination is transmitted from the CAN IF 106 to the vehicle control device 107 and the alarm device 108 (S508, S509).

As described above, even in the case of forward monitoring using the stereo camera 101 and the monocular camera 102 as in this embodiment, when an object to be detected enters the blind spot of the recognition range 204-S of the stereo camera 101 and moves into the recognition range 205-M of only the monocular camera 102, the relative distance is calculated based on features that change little even if the object moves due to a lane change, etc., so that the accuracy of the relative distance calculated by the monocular camera 102 can be prevented from decreasing. In other words, when the distance calculation is handed over from the stereo camera (first sensor) 101 to the monocular camera (second sensor) 102, the relative distances dw and dh between the host vehicle and the following vehicle can be prevented from deviating from the actual distance. Furthermore, since the accuracy of the relative distance calculation is improved, the malfunction or non-operation of the collision avoidance operation can be reduced.

It goes without saying that weighting as in the second embodiment or a 3D bounding box as in the third embodiment may also be used for forward monitoring.

The present invention is not limited to the above-described embodiments, and includes various modifications that do not deviate from the gist of the invention. For example, the present invention is not limited to those that include all of the configurations described in the above-described embodiments, and also includes those in which some of the configurations are omitted. Also, it is possible to add or replace some of the configurations of one embodiment with the configurations of another embodiment.

Furthermore, the configurations of the ECU 105 and the functions and execution processes of the configurations may be realized in part or in whole by hardware (for example, by designing the logic that executes each function in an integrated circuit). The configurations of the ECU 105 may be realized as programs (software) that are read and executed by an arithmetic processing device (for example, a CPU) to realize the functions of the ECU 105 configuration. Information related to the programs may be stored, for example, in semiconductor memory (flash memory, SSD, etc.), magnetic storage devices (hard disk drives, etc.), and recording media (magnetic disks, optical disks, etc.).

In addition, in the above explanation of each embodiment, the control lines and information lines are those that are considered necessary for the explanation of the embodiment, but they do not necessarily show all the control lines and information lines related to the product. In reality, it can be considered that almost all components are interconnected.

101...Stereo camera (first sensor), 102...Monocular camera (second sensor), 105...ECU, 105a...Processor, 201...Own vehicle, 202-S...Stereo camera recognition area (recognition range), 203-M...Monocular camera recognition area (recognition range), 204-S...Stereo camera recognition area (recognition range), 205-M...Monocular camera recognition area (recognition range), 401-404...Position of rear vehicle, 601-604...Position of front vehicle

Claims (12)

An object detection device for a vehicle comprising at least one processor,
The processor,
Calculating a plurality of feature amounts of an object from a detection result of a first sensor mounted on the vehicle;
calculating changes in the plurality of feature quantities obtained from the first sensor,
determining a reference feature quantity to be used in calculating a relative distance between the vehicle and the object, among the feature quantities of the object calculated from the detection result of a second sensor mounted on the vehicle, based on an amount of change in the plurality of feature quantities;
an object detection device comprising: when the object moves from the recognition range of the first sensor to the recognition range of the second sensor, the object detection device calculates the reference feature from the detection result of the second sensor; and calculates the relative distance based on the reference feature and a feature corresponding to the reference feature among the plurality of feature values obtained from the first sensor. 2. The object detection device according to claim 1,
The first sensor is any one of a stereo camera, a radar, a sonar, a LiDAR, and a monocular camera;
The object detection device, wherein the second sensor is a monocular camera. 2. The object detection device according to claim 1,
the plurality of feature amounts are a width and a height of the object,
The object detection device, wherein the processor determines the reference feature amount based on an amount of change in width and height of the object obtained from the first sensor. 2. The object detection device according to claim 1,
the plurality of feature amounts are a width and a height of the object,
The object detection device, wherein the processor determines, as the reference feature, a feature that has a smaller amount of change among the amount of change in width and height of the object obtained from the first sensor. 2. The object detection device according to claim 1,
the plurality of feature amounts are a width and a height of the object,
The object detection device, wherein the processor determines a height of the object as the reference feature when a change in width of the object obtained from the first sensor exceeds a predetermined threshold. 2. The object detection device according to claim 1,
the plurality of feature amounts are a width and a height of the object,
the reference feature amounts are a width and a height of the object,
The processor,
calculating a first relative distance between the vehicle and the object based on a width of the object obtained from the first sensor and a width of the object obtained from the second sensor;
calculating a second relative distance between the vehicle and the object based on the height of the object obtained from the first sensor and the height of the object obtained from the second sensor;
an object detection device comprising: a first sensor for detecting an object from the object; a second sensor for detecting an object from the object; a first sensor for detecting a width of the object; a height of the object; 7. The object detection device according to claim 6,
an object detection device, characterized in that a weight of the second relative distance is set to be larger as a change in width of the object obtained from the first sensor increases. 7. The object detection device according to claim 6,
The object detection device according to claim 1, wherein weighting based on the amount of change in width and height of the object obtained from the first sensor varies depending on the type of the vehicle. 7. The object detection device according to claim 6,
an object detection device, characterized in that a weight based on an amount of change in width and height of the object obtained from the first sensor varies depending on the magnitude of the first relative distance and the second relative distance. 2. The object detection device according to claim 1,
the first sensor is a stereo camera that monitors a rear area of the vehicle;
The object detection device according to claim 1, wherein the second sensor is a monocular camera that monitors a rear side of the vehicle. 2. The object detection device according to claim 1,
the first sensor is a stereo camera that monitors a front of the vehicle;
the second sensor is a monocular camera that monitors a front of the vehicle and is installed at a position where it can monitor a blind spot of the stereo camera located in front of the vehicle. 2. The object detection device according to claim 1,
the first sensor is a stereo camera,
The processor assigns a three-dimensional bounding box to the object in the image obtained by the stereo camera, and, if a detection confidence of the three-dimensional bounding box is equal to or lower than a predetermined value, executes a calculation necessary to calculate the relative distance.

Images