JP7135579B2 - Object detection device - Google Patents

Description

The present invention relates to an object detection device that is mounted on a vehicle and configured to detect objects existing around the vehicle.

As a device of this type, for example, one described in Patent Document 1 is known. The device described in Patent Literature 1 detects the three-dimensional position of an object such as an obstacle in the surroundings by using an image captured by a camera and the behavior of the own vehicle. Specifically, this device first acquires an image captured by the camera at the current time and a captured image captured immediately before this captured image. Next, this device extracts a plurality of feature points from each of the two captured images, and associates each feature point between the plurality of captured images. After that, this device estimates the three-dimensional position of the feature point in the real space based on the amount of movement of the feature point between the captured images and the amount of movement of the moving object.

JP 2014-142241 A

For example, there may be a situation in which the direction of relative movement between the vehicle and the object substantially coincides with the imaging direction of the camera, and the object is positioned in front of the camera. In such a situation, it is difficult to obtain sufficient parallax necessary for estimating the three-dimensional position. In particular, when the object is an elongated pole-shaped obstacle extending in the vertical direction, such a problem occurs remarkably. In this case, this is because the change in the state of the object in the captured image is small before and after the relative movement between the object and the camera as the vehicle travels.

The present invention has been made in view of the circumstances exemplified above. That is, for example, the present invention provides a configuration capable of better detecting objects existing around the own vehicle.

An object detection device (20) according to claim 1 is configured to detect an object (B) existing around the own vehicle (10) by being mounted on the own vehicle (10).
This object detection device
an image information acquisition unit (271) for acquiring image information corresponding to a photographed image of the surroundings of the own vehicle;
a feature point extraction unit (273) for extracting feature points (FP) in the captured image based on the image information acquired by the image information acquisition unit;
a three-dimensional information acquisition unit (274) that acquires the three-dimensional position of the feature point based on the plurality of captured images from different viewpoints;
a point group extraction unit (275) for extracting a point group (G) consisting of a plurality of the feature points arranged under a predetermined condition from the result of extraction of the feature points by the feature point extraction unit;
a ranging information acquisition unit (276) that acquires ranging information corresponding to the distance to the ranging point (DP) measured by the ranging sensor (22);
Corresponding to the first point group (G1) and the second point group (G2), which are the two point groups extracted by the point cloud extraction unit, and the ranging information acquired by the ranging information acquisition unit an obstacle detection unit ( 277) and
with
In the obstacle detection unit, the first point group and the second point group are configured by a plurality of the feature points for which the three-dimensional position was not acquired, and the distance measurement information acquisition unit acquires and when the range-finding point corresponding to the range-finding information and the first point group and the second point group are in the predetermined positional relationship in the captured image, the first point group and the second point detecting the area between the group as the obstacle;
The predetermined positional relationship is such that a reference feature point (FP0), which is the feature point from which the three-dimensional position is acquired, exists between the first point group and the second point group, and the reference feature point A difference between a reference distance, which is the distance from the host vehicle, and a ranging distance, which is the distance of the ranging point from the host vehicle, is equal to or less than a predetermined value .
The object detection device (20) according to claim 2 is configured to detect an object (B) existing around the own vehicle (10) by being mounted on the own vehicle (10).
This object detection device
an image information acquisition unit (271) for acquiring image information corresponding to a photographed image of the surroundings of the own vehicle;
a feature point extraction unit (273) for extracting feature points (FP) in the captured image based on the image information acquired by the image information acquisition unit;
a point group extraction unit (275) for extracting a point group (G) consisting of a plurality of the feature points arranged under a predetermined condition from the result of extraction of the feature points by the feature point extraction unit;
a ranging information acquisition unit (276) that acquires ranging information corresponding to the distance to the ranging point (DP) measured by the ranging sensor (22);
Corresponding to the first point group (G1) and the second point group (G2), which are the two point groups extracted by the point cloud extraction unit, and the ranging information acquired by the ranging information acquisition unit an obstacle detection unit ( 277) and
with
The predetermined positional relationship includes that the distances of the first point group and the second point group from the range-finding point in the captured image are equal to or less than a predetermined distance.
The object detection device (20) according to claim 3 is configured to detect an object (B) existing around the own vehicle by being mounted on the own vehicle (10).
This object detection device
an image information acquisition unit (271) for acquiring image information corresponding to a photographed image of the surroundings of the own vehicle;
a feature point extraction unit (273) for extracting feature points (FP) in the captured image based on the image information acquired by the image information acquisition unit;
a point group extraction unit (275) for extracting a point group (G) consisting of a plurality of the feature points arranged under a predetermined condition from the result of extraction of the feature points by the feature point extraction unit;
a ranging information acquisition unit (276) that acquires ranging information corresponding to the distance to the ranging point (DP) measured by the ranging sensor (22);
Corresponding to the first point group (G1) and the second point group (G2), which are the two point groups extracted by the point cloud extraction unit, and the ranging information acquired by the ranging information acquisition unit an obstacle detection unit ( 277) and
with
The predetermined positional relationship is
The distance of the first point group or the second point group from the range-finding point in the captured image is a predetermined distance or less, and
An interval in the photographed image between the first point group and the second point group is equal to or less than a predetermined interval.

In the above configuration, the image information acquisition unit acquires the image information corresponding to the captured image of the surroundings of the own vehicle. The feature point extraction unit extracts the feature points in the captured image based on the image information acquired by the image information acquisition unit. As is well known, the three-dimensional positions of the feature points can be obtained by parallax calculation based on the plurality of captured images from different viewpoints.

However, as described above, a situation may occur in which it is difficult to obtain sufficient parallax necessary for obtaining or estimating the three-dimensional position of the feature point. In such a situation, parallax calculation becomes difficult, and thus the three-dimensional position of the feature point becomes difficult to obtain. Then, it becomes difficult to detect the obstacles around the host vehicle that correspond to the feature points.

In this regard, for example, when the distance measurement information is obtained in substantially the same direction as the plurality of feature points for which the three-dimensional positions were not obtained due to insufficient parallax, the distance measurement information There is a high possibility that the object exists at the position corresponding to . In this case, the distance measurement information corresponding to the distance to the object is acquired by the distance measurement information acquisition unit, and the plurality of features are obtained in the vicinity of the distance measurement point corresponding to the acquired distance measurement information. The points are arranged in a predetermined manner.

Therefore, in the above configuration, the point group extraction unit extracts the point group composed of the plurality of feature points arranged under the predetermined condition from the extraction result of the feature points by the feature point extraction unit. The ranging information acquisition unit acquires the ranging information corresponding to the distance to the ranging point measured by the ranging sensor. The obstacle detection unit includes the first point group and the second point group, which are the two point groups extracted by the point group extraction unit, and the distance measurement acquired by the distance measurement information acquisition unit. A region between the first point group and the second point group is detected as the obstacle when the range-finding points corresponding to the information are in the predetermined positional relationship in the captured image.

In the above configuration, the first point group and the second point group arranged under the predetermined condition are extracted, and the first point group, the second point group, and the range-finding point are located in the captured image. , the area between the first point group and the second point group can be detected as the obstacle. Therefore, according to the above configuration, it is possible to more satisfactorily detect the object existing around the own vehicle.

In addition, in each column of the application documents, each element may be given a reference sign with parentheses. In this case, the reference numerals simply indicate an example of the correspondence relationship between the same element and the specific configuration described in the embodiment described later. Therefore, the present invention is not limited in any way by the description of the reference numerals.

BRIEF DESCRIPTION OF THE DRAWINGS It is a top view which shows schematic structure of the vehicle which mounts the object detection apparatus which concerns on embodiment. 2 is a block diagram showing a schematic functional configuration of the object detection device shown in FIG. 1; FIG. 3 is a conceptual diagram showing an outline of the operation of the object detection device shown in FIG. 2; FIG. 3 is a flowchart showing a first operation example of the object detection device shown in FIG. 2; 3 is a flowchart showing a first operation example of the object detection device shown in FIG. 2; 3 is a flowchart showing a second operation example of the object detection device shown in FIG. 2;

(embodiment)
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described based on the drawings. It should be noted that if various modifications applicable to one embodiment are inserted in the middle of a series of explanations related to the embodiment, there is a risk that the understanding of the embodiment will be hindered. Therefore, the modified examples will be collectively described after the series of descriptions regarding the embodiment, not in the middle of the description.

(Overall configuration of vehicle)
Referring to FIG. 1, a vehicle 10 is a so-called four-wheel vehicle and includes a substantially rectangular vehicle body 11 in a plan view. Hereinafter, an imaginary straight line passing through the center of the vehicle 10 in the vehicle width direction and parallel to the vehicle full length direction of the vehicle 10 will be referred to as a vehicle center axis X. As shown in FIG. In FIG. 1, the vehicle width direction is the horizontal direction in the drawing. The vehicle length direction is a direction orthogonal to the vehicle width direction and orthogonal to the vehicle height direction. The vehicle height direction is a direction that defines the vehicle height of the vehicle 10, and is parallel to the direction of gravity when the vehicle 10 is placed on a horizontal plane. Furthermore, an arbitrary direction orthogonal to the vehicle height direction in which the vehicle 10 can move by running is sometimes referred to as the “translational direction” of the vehicle 10 .

For convenience of explanation, "front", "rear", "left", and "right" of the vehicle 10 are defined as indicated by arrows in FIG. That is, the vehicle length direction is synonymous with the front-rear direction. Further, the vehicle width direction is synonymous with the left-right direction. Note that the vehicle height direction may not be parallel to the gravity action direction depending on the mounting conditions or running conditions of the vehicle 10 . However, in many cases, the vehicle height direction is along the direction of gravity action, so the "translational direction" orthogonal to the vehicle height direction is "horizontal direction", "in-plane direction", "approach direction", and " It may also be referred to as "heading" or "tracking".

A front bumper 13 is attached to a front portion 12 that is a front end portion of the vehicle body 11 . A rear bumper 15 is attached to a rear surface portion 14 that is a rear end portion of the vehicle body 11 . A door panel 17 is attached to a side portion 16 of the vehicle body 11 . In the specific example shown in FIG. 1, a total of four door panels 17 are provided, two on each side. A door mirror 18 is attached to each of the pair of left and right door panels 17 on the front side.

An object detection device 20 is mounted on the vehicle 10 . The object detection device 20 is mounted on the vehicle 10 so as to detect an object B existing outside and around the vehicle 10 . Hereinafter, the vehicle 10 equipped with the object detection device 20 may be abbreviated as "own vehicle". That is, the object detection device 20 is configured to detect the object B existing in the translational direction of the own vehicle.

Specifically, the object detection device 20 includes an imaging unit 21, a sonar sensor 22, a radar sensor 23, a vehicle speed sensor 24, a shift position sensor 25, a steering angle sensor 26, an object detection ECU 27, and a display unit 28. and an audio output unit 29 . ECU is an abbreviation for Electronic Control Unit. The details of each part constituting the object detection device 20 will be described below. For the sake of simplification of illustration, FIG.

The imaging unit 21 is provided to capture an image of the surroundings of the own vehicle and generate and acquire image information corresponding to the captured image. In this embodiment, the imaging unit 21 is a digital camera device, and includes an image sensor such as a CCD. CCD is an abbreviation for Charge Coupled Device.

In this embodiment, the vehicle 10 is equipped with a plurality of imaging units 21, that is, a front camera CF, a rear camera CB, a left camera CL, and a right camera CR. When not specifying any of the front camera CF, the rear camera CB, the left camera CL, and the right camera CR, hereinafter, the singular expression “imaging unit 21” or “plural imaging units 21” is sometimes used.

The front camera CF is provided to acquire image information corresponding to an image in front of the vehicle. Specifically, in the present embodiment, the front camera CF is attached to a rearview mirror (not shown) arranged in the vehicle interior of the vehicle 10 .

The rear camera CB is attached to the rear surface portion 14 of the vehicle body 11 so as to obtain image information corresponding to an image behind the own vehicle. The left camera CL is attached to the left side door mirror 18 so as to acquire image information corresponding to the left side image of the vehicle. The right camera CR is attached to the right side door mirror 18 so as to obtain image information corresponding to the right image of the vehicle.

Each of the imaging units 21 is electrically connected to the object detection ECU 27 . That is, each of the plurality of imaging units 21 acquires image information under the control of the object detection ECU 27 and outputs the acquired image information so that the object detection ECU 27 can receive the image information.

The sonar sensor 22 is a ranging sensor that measures the distance to a point on the object B, and is attached to the vehicle body 11 . That is, the sonar sensor 22 is provided so as to output a signal corresponding to the distance from the object B by receiving the received wave including the reflected wave from the object B among the search waves transmitted toward the outside of the own vehicle. It is Specifically, in the present embodiment, the sonar sensor 22 is a so-called ultrasonic sensor that can transmit search waves, which are ultrasonic waves, toward the outside of the vehicle and can receive received waves including ultrasonic waves. is configured to

The object detection device 20 has at least one sonar sensor 22 . Specifically, in this embodiment, a plurality of sonar sensors 22 are provided. The plurality of sonar sensors 22 are arranged so as to be shifted from the vehicle center axis X to one side in the vehicle width direction. Moreover, at least some of the plurality of sonar sensors 22 are provided so as to emit survey waves along the direction intersecting with the vehicle center axis X. As shown in FIG.

Specifically, the front bumper 13 is equipped with a first front sonar SF1, a second front sonar SF2, a third front sonar SF3, and a fourth front sonar SF4 as sonar sensors 22. FIG. Similarly, the rear bumper 15 is equipped with a first rear sonar SR1, a second rear sonar SR2, a third rear sonar SR3, and a fourth rear sonar SR4 as sonar sensors 22. As shown in FIG. A first side sonar SS1, a second side sonar SS2, a third side sonar SS3, and a fourth side sonar SS4 are mounted as sonar sensors 22 on the side portion 16 of the vehicle body 11. As shown in FIG.

1st front sonar SF1, 2nd front sonar SF2, 3rd front sonar SF3, 4th front sonar SF4, 1st rear sonar SR1, 2nd rear sonar SR2, 3rd rear sonar SR3, 4th rear sonar SR4, 1st side sonar SS1, Hereinafter, when not specifying any of the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4, the singular expression "sonar sensor 22" or "plurality of sonar sensors 22” is sometimes used.

One sonar sensor 22 is called a "first sonar sensor" and another sonar sensor 22 is called a "second sonar sensor", and "direct waves" and "indirect waves" are defined as follows. A received wave that is received by the first sonar sensor and is caused by a reflected wave from the object B of the search wave emitted from the first sonar sensor is referred to as a "direct wave". On the other hand, a received wave that is received by the first sonar sensor and that is caused by a wave reflected by the object B in the search wave emitted from the second sonar sensor is referred to as an "indirect wave."

The first front sonar SF1 is provided at the left end of the front surface of the front bumper 13 so as to emit a survey wave to the left front of the vehicle. The second front sonar SF2 is provided at the right end of the front side surface of the front bumper 13 so as to transmit a survey wave to the right front of the vehicle. The first front sonar SF1 and the second front sonar SF2 are arranged symmetrically with the vehicle center axis X interposed therebetween.

The third front sonar SF3 and the fourth front sonar SF4 are arranged in the width direction of the vehicle at positions near the center of the front surface of the front bumper 13 . The third front sonar SF3 is arranged between the first front sonar SF1 and the vehicle center axis X in the vehicle width direction so as to emit survey waves substantially forward of the vehicle. The fourth front sonar SF4 is arranged between the second front sonar SF2 and the vehicle center axis X in the vehicle width direction so as to emit survey waves substantially forward of the vehicle. The third front sonar SF3 and the fourth front sonar SF4 are arranged symmetrically with the vehicle center axis X interposed therebetween.

As described above, the first front sonar SF1 and the third front sonar SF3 mounted on the left side of the vehicle body 11 are arranged at different positions in plan view. Also, the first front sonar SF1 and the third front sonar SF3, which are adjacent to each other in the vehicle width direction, are in a positional relationship such that the reflected wave of the survey wave emitted by one of them by the object B can be received by the other as a received wave. is provided.

That is, the first front sonar SF1 is arranged so as to be able to receive both the direct wave corresponding to the search wave emitted by itself and the indirect wave corresponding to the search wave emitted by the third front sonar SF3. Similarly, the third front sonar SF3 is arranged so as to be able to receive both the direct wave corresponding to the search wave emitted by itself and the indirect wave corresponding to the search wave emitted by the first front sonar SF1.

Similarly, the third front sonar SF3 and the fourth front sonar SF4 mounted near the center of the vehicle body 11 in the vehicle width direction are arranged at different positions in plan view. Further, the third front sonar SF3 and the fourth front sonar SF4, which are adjacent to each other in the vehicle width direction, are in a positional relationship such that the reflected wave of the survey wave transmitted by one of them by the object B can be received by the other as a received wave. is provided.

Similarly, the second front sonar SF2 and the fourth front sonar SF4 mounted on the right side of the vehicle body 11 are arranged at positions different from each other in plan view. Further, the second front sonar SF2 and the fourth front sonar SF4, which are adjacent to each other in the vehicle width direction, are in a positional relationship such that the reflected wave of the survey wave transmitted by one of them by the object B can be received by the other as a received wave. is provided.

The first rear sonar SR1 is provided at the left end of the rear surface of the rear bumper 15 so as to transmit a survey wave to the left rear of the vehicle. The second rear sonar SR2 is provided at the right end portion of the rear surface of the rear bumper 15 so as to transmit a survey wave to the right rear of the vehicle. The first rear sonar SR1 and the second rear sonar SR2 are arranged symmetrically with the vehicle center axis X interposed therebetween.

The third rear sonar SR3 and the fourth rear sonar SR4 are arranged in the vehicle width direction at positions near the center of the rear surface of the rear bumper 15 . The third rear sonar SR3 is arranged between the first rear sonar SR1 and the vehicle center axis X in the vehicle width direction so as to transmit a survey wave substantially behind the own vehicle. The fourth rear sonar SR4 is arranged between the second rear sonar SR2 and the vehicle center axis X in the vehicle width direction so as to emit survey waves substantially behind the vehicle. The third rear sonar SR3 and the fourth rear sonar SR4 are arranged symmetrically with the vehicle center axis X interposed therebetween.

As described above, the first rear sonar SR1 and the third rear sonar SR3 mounted on the left side of the vehicle body 11 are arranged at different positions in plan view. Further, the first rear sonar SR1 and the third rear sonar SR3, which are adjacent to each other in the vehicle width direction, are provided in a positional relationship such that the reflected waves of the survey waves emitted by one of them and reflected by the object B can be received as received waves by the other. ing.

That is, the first rear sonar SR1 is arranged so as to be able to receive both the direct wave corresponding to the search wave emitted by itself and the indirect wave corresponding to the search wave emitted by the third rear sonar SR3. Similarly, the third rear sonar SR3 is arranged so as to be able to receive both the direct wave corresponding to the search wave emitted by itself and the indirect wave corresponding to the search wave emitted by the first rear sonar SR1.

Similarly, the third rear sonar SR3 and the fourth rear sonar SR4, which are mounted near the center of the vehicle body 11 in the vehicle width direction, are arranged at different positions in plan view. Further, the third rear sonar SR3 and the fourth rear sonar SR4, which are adjacent to each other in the vehicle width direction, are provided in a positional relationship such that the reflected waves of the survey waves emitted by one of them and reflected by the object B can be received by the other as received waves. ing.

Similarly, the second rear sonar SR2 and the fourth rear sonar SR4 mounted on the right side of the vehicle body 11 are arranged at different positions in plan view. Further, the second rear sonar SR2 and the fourth rear sonar SR4, which are adjacent to each other in the vehicle width direction, are provided in a positional relationship such that the reflected waves of the survey waves emitted by one of them and reflected by the object B can be received as received waves by the other. ing.

The first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 are designed to transmit survey waves to the side of the vehicle from the side of the vehicle, which is the outer surface of the side portion 16. is provided. The first side sonar SS1, the second side sonar SS2, the third side sonar SS3, and the fourth side sonar SS4 are each provided so as to receive only direct waves.

The first side sonar SS1 is arranged between the left side door mirror 18 and the first front sonar SF1 in the front-rear direction so as to emit a search wave to the left of the vehicle. The second side sonar SS2 is arranged between the right side door mirror 18 and the second front sonar SF2 in the front-rear direction so as to transmit a search wave to the right of the vehicle. The first side sonar SS1 and the second side sonar SS2 are provided symmetrically with the vehicle center axis X interposed therebetween.

The third side sonar SS3 is arranged between the left rear door panel 17 and the first rear sonar SR1 in the front-rear direction so as to emit a search wave to the left of the vehicle. The fourth side sonar SS4 is arranged between the right rear door panel 17 and the second rear sonar SR2 in the front-rear direction so as to transmit a search wave to the right of the vehicle. The third side sonar SS3 and the fourth side sonar SS4 are provided symmetrically with the vehicle center axis X interposed therebetween.

Each of the multiple sonar sensors 22 is electrically connected to the object detection ECU 27 . That is, each of the plurality of sonar sensors 22 transmits a survey wave under the control of the object detection ECU 27, generates a signal corresponding to the reception result of the received wave, and outputs the signal so that it can be received by the object detection ECU 27. ing. Information included in the signal corresponding to the reception result of the received wave is hereinafter referred to as "ranging information". The ranging information includes information related to the reception intensity of received waves and distance information. “Distance information” is information related to the distance between each of the plurality of sonar sensors 22 and the object B. FIG. Specifically, for example, the distance information includes information related to the time difference between the transmission of the search wave and the reception of the received wave.

The radar sensor 23 is a laser radar sensor or millimeter wave radar sensor that transmits and receives radar waves, and is attached to the front portion 12 of the vehicle body 11 . The radar sensor 23 is configured to generate a signal corresponding to the position and relative velocity of the reflection point and output the signal so that it can be received by the object detection ECU 27 . A “reflection point” is a point on the surface of the object B that is presumed to have reflected the radar wave. "Relative velocity" is the relative velocity of the reflection point, that is, the object B that reflected the radar wave, with respect to the own vehicle.

Vehicle speed sensor 24 , shift position sensor 25 and steering angle sensor 26 are electrically connected to object detection ECU 27 . The vehicle speed sensor 24 is provided so as to generate a signal corresponding to the running speed of the own vehicle and output the signal so as to be receivable by the object detection ECU 27 . The running speed of the own vehicle is hereinafter simply referred to as "vehicle speed". The shift position sensor 25 is provided so as to generate a signal corresponding to the shift position of the host vehicle and output the signal so as to be receivable by the object detection ECU 27 . The steering angle sensor 26 is provided so as to generate a signal corresponding to the steering angle of the host vehicle and output the signal so that it can be received by the object detection ECU 27 .

The object detection ECU 27 is arranged inside the vehicle body 11 . The object detection ECU 27 is a so-called in-vehicle microcomputer, and includes a CPU, ROM, RAM, non-volatile RAM, etc. (not shown). Non-volatile RAM is, for example, flash ROM or the like. The CPU, ROM, RAM and nonvolatile RAM of the object detection ECU 27 are hereinafter simply referred to as "CPU", "ROM", "RAM" and "nonvolatile RAM".

The object detection ECU 27 is configured so that various control operations can be realized by the CPU reading and executing a program from the ROM or the nonvolatile RAM. This program includes those corresponding to the routines described below. Also, the RAM and the non-volatile RAM are configured to be able to temporarily store processing data when the CPU executes the program. Furthermore, the ROM and/or the non-volatile RAM pre-store various data used when executing the program. Various data include, for example, initial values, lookup tables, maps, and the like.

The object detection ECU 27 executes an object detection operation based on signals and information received from each of the plurality of sonar sensors 22, each of the plurality of imaging units 21, the vehicle speed sensor 24, the shift position sensor 25, the steering angle sensor 26, and the like. is configured to In addition, the object detection ECU 27 controls the operations of the display unit 28 and the audio output unit 29 to perform notification operations associated with the object detection state.

The display unit 28 and the audio output unit 29 are arranged in the vehicle interior of the vehicle 10 . Also, the display unit 28 and the audio output unit 29 are electrically connected to the object detection ECU 27 . The display unit 28 is configured to perform a notification operation associated with the object detection state by displaying a display screen or an indicator. The audio output unit 29 is configured to output a notification operation accompanying the object detection state by audio output.

(Functional configuration of object detection ECU)
Referring to FIG. 2, the object detection ECU 27 includes an image information acquisition unit 271, a movement information acquisition unit 272, a feature point extraction unit 273, and a three-dimensional coordinate information acquisition unit as functional components realized on a microcomputer. It has a section 274 , a point group extraction section 275 , a distance measurement information acquisition section 276 and an obstacle detection section 277 .

Details of the functional configuration of the object detection ECU 27 in this embodiment will be described below with reference to the block diagram of FIG. 2 and the operation schematic diagram of FIG. FIG. 3 shows how an image captured by the rear camera CB, including the pole-shaped object B existing behind the vehicle 1, is processed. In FIG. 3, VA is the angle of view of the captured image. The angle of view VA is the maximum range of the captured image corresponding to the field of view of the imaging unit 21 . Further, HL is a horizontal line, that is, the horizon line, DL1 is a first division line that is a solid road division line, and DL2 is a second division line that is a broken road division line. BD is a distant building.

The image information acquisition unit 271 is provided so as to acquire image information corresponding to a photographed image of the surroundings of the own vehicle. Specifically, the image information acquisition section 271 receives image information generated by the imaging section 21 from the imaging section 21 and stores the received image information in time series in the non-volatile RAM.

The movement information acquisition unit 272 is provided so as to acquire the movement direction and movement amount of the own vehicle during the object detection operation. Specifically, the movement information acquisition unit 272 calculates the movement direction and movement amount of the own vehicle based on the outputs of the vehicle speed sensor 24, the shift position sensor 25, and the steering angle sensor 26, and displays the calculation results in time series. is stored in the non-volatile RAM.

The feature point extraction section 273 is provided to extract feature points FP in the captured image based on the image information acquired by the image information acquisition section 271 . The feature point FP is a point that characterizes the shape of the object B in the captured image. In other words, the feature point FP is a characteristic point, ie, a pixel within the angle of view VA. Specifically, for example, the feature point FP is a pixel with a large change in luminance between adjacent pixels.

Note that the feature point FP and its extraction method are well known at the time of filing the application. Specifically, a well-known method (for example, Sobel filter, Laplacian filter, Canny method, etc.) can be used as a method of detecting feature points FP. Therefore, in this specification, the detailed description of the method of extracting the feature points FP by the feature point extraction unit 273 will be omitted. Note that "extraction" of the feature points FP can also be expressed as "detection".

The three-dimensional coordinate information acquisition section 274 is provided so as to acquire the three-dimensional position of the feature point FP based on a plurality of captured images from different viewpoints. In this embodiment, one rear camera CB, one front camera CF, one left camera CL, and one right camera CR are provided. Therefore, in the present embodiment, the three-dimensional coordinate information acquisition unit 274 combines the image information acquired and stored by the image information acquisition unit 271 with the movement direction and movement amount acquired and stored by the movement information acquisition unit 272. Based on this, the three-dimensional coordinate information of the feature point FP on the object B is obtained.

That is, the three-dimensional coordinate information acquisition unit 274 is configured to recognize the object B existing around the own vehicle using a moving stereo or monocular moving stereo technique. Specifically, the three-dimensional coordinate information acquisition unit 274 determines the direction of the imaging unit 21 based on the history of image information in one imaging unit 21 directed in a specific direction and the amount of movement of the vehicle. It recognizes an object B existing in the direction of Motion stereo is also referred to as SFM. SFM is an abbreviation for Structure from Motion.

Moving stereo or SFM has already become a well-known technology at the time of the filing of this application. See, for example, US Pat. Nos. 7,433,494, 8,027,514, etc. As necessary, these descriptions are incorporated by reference as appropriate to the extent permitted by domestic laws and regulations, as long as they are not technically inconsistent. Therefore, the details of the moving stereo are omitted in this specification.

The point group extraction unit 275 is provided so as to extract a point group G composed of a plurality of feature points FP arranged under a predetermined condition from the extraction result of the feature points FP by the feature point extraction unit 273 . In this embodiment, the "predetermined condition" includes that the plurality of feature points FP are arranged in a straight line. Also, the "predetermined condition" includes that the plurality of feature points FP are arranged so as to intersect the horizontal line HL. Moreover, in this embodiment, the point group extraction unit 275 extracts the point group G in a predetermined search range RS. The search range RS is a predetermined range in the captured image with reference to the ranging point DP corresponding to the ranging information acquired by the ranging information acquiring section 276 . In this embodiment, the shape of the search range RS is set to be rectangular. The width of the search range RS is set to a predetermined width in consideration of the ranging error of the ranging point DP by triangulation. The height of the search range RS is set to a predetermined height such that the search range RS intersects the horizontal line HL in consideration of the computational load.

The distance measurement information acquisition unit 276 is provided to acquire distance measurement information corresponding to the distance to the distance measurement point DP measured by the sonar sensor 22 . That is, the ranging information acquisition unit 276 acquires two-dimensional position information corresponding to the relative position of the object B with respect to the own vehicle in the translational direction of the own vehicle, in other words, acquires ranging information corresponding to the ranging point DP. , the acquired results are stored in a non-volatile RAM in chronological order. Specifically, the ranging information acquisition unit 276 detects the reflected wave of the probe wave, which is an ultrasonic wave emitted toward the outside of the vehicle, by the object B using the sonar sensor 22 mounted on the vehicle body 11 of the vehicle. It is configured to acquire ranging information based on the received result.

In this embodiment, the ranging information acquisition unit 276 is provided so as to acquire and store ranging information based on the result of the sonar sensor 22 receiving the reflected wave of the search wave from the object B. FIG. Specifically, the distance measurement information acquisition unit 276 acquires the distance measurement information corresponding to the distance measurement point DP and stores it in time series in the non-volatile RAM. The “distance measurement point DP” is a point on the surface of the object B that is estimated to have reflected the search wave transmitted from the sonar sensor 22 , and corresponds to the “reflection point” on the radar sensor 23 . Incidentally, the method of obtaining the distance measuring point DP is well known at the time of filing of the present application. Therefore, the detailed description of such a technique is omitted in this specification.

The obstacle detection unit 277 is provided to detect an object B around the vehicle as an obstacle. Specifically, the obstacle detection unit 277 is configured to detect the object B recognized by the three-dimensional coordinate information acquisition unit 274 as an obstacle. That is, the obstacle detection unit 277 detects an obstacle by moving stereo based on the feature points FP with sufficient parallax necessary for obtaining or estimating the three-dimensional positions of the feature points FP. It's like

Further, the obstacle detection unit 277 utilizes a plurality of feature points FP whose three-dimensional positions have not been acquired by the three-dimensional coordinate information acquisition unit 274 due to insufficient parallax to detect obstacles. is configured as Specifically, the obstacle detection unit 277 detects the first point group G1, the second point group G2, and the distance measurement point DP corresponding to the distance measurement information acquired by the distance measurement information acquisition unit 276 in the vicinity thereof. However, when there is a predetermined positional relationship in the captured image, the area between the first point group G1 and the second point group G2 is detected as an obstacle. A first point group G<b>1 and a second point group G<b>2 are two point groups G extracted by the point group extraction unit 275 . In this embodiment, the "predetermined positional relationship" includes that the distance of the first point group G1 and the second point group G2 from the distance measuring point DP in the captured image is equal to or less than a predetermined distance.

(action/effect)
An overview of the operation of the object detection device 20, that is, the object detection ECU 27 of the present embodiment will be described below, together with the effects achieved by the configuration of the present embodiment, with reference to FIGS. 1 to 3. FIG.

Each of the plurality of imaging units 21, that is, the front camera CF, the rear camera CB, the left camera CL, and the right camera CR captures an image of the surroundings of the own vehicle, and generates and acquires image information corresponding to the captured image. do. Further, each of the plurality of imaging units 21 outputs the acquired image information so that it can be received by the object detection ECU 27 .

Each of the plurality of sonar sensors 22 measures the distance to a point on an object B existing around the vehicle by receiving received waves including reflected waves of the search waves transmitted toward the outside of the vehicle. do. Further, each of the plurality of sonar sensors 22 outputs the acquired distance measurement information so that it can be received by the object detection ECU 27 .

The radar sensor 23 generates a signal corresponding to the position and relative speed of the reflection point on the object B, and outputs the signal so that it can be received by the object detection ECU 27 . The vehicle speed sensor 24 generates a signal corresponding to the vehicle speed and outputs the signal so that it can be received by the object detection ECU 27 . The shift position sensor 25 generates a signal corresponding to the shift position of the host vehicle and outputs the signal so that the object detection ECU 27 can receive the signal. The steering angle sensor 26 generates a signal corresponding to the steering angle of the host vehicle and outputs the signal so that it can be received by the object detection ECU 27 .

The object detection ECU 27 receives image information from each of the plurality of imaging units 21 . The object detection ECU 27 also receives ranging information from each of the plurality of sonar sensors 22 . The object detection ECU 27 also receives output signals from the radar sensor 23 , vehicle speed sensor 24 , shift position sensor 25 and steering angle sensor 26 . The object detection ECU 27 executes an object detection operation based on signals and information received from each of the plurality of sonar sensors 22, each of the plurality of imaging units 21, the vehicle speed sensor 24, the shift position sensor 25, the steering angle sensor 26, and the like. do.

Specifically, the image information acquisition unit 271 acquires image information corresponding to the captured images around the host vehicle from each of the plurality of imaging units 21 . In addition, the image information acquisition unit 271 stores the acquired image information in the nonvolatile RAM in chronological order. Movement information acquisition section 272 calculates the movement direction and movement amount of the host vehicle based on the outputs of vehicle speed sensor 24 , shift position sensor 25 , and steering angle sensor 26 . In addition, the movement information acquisition unit 272 stores the calculation results in time series in the nonvolatile RAM. The feature point extraction unit 273 extracts feature points FP in the captured image based on the image information acquired by the image information acquisition unit 271 .

Sufficient parallax may be obtained to enable good parallax calculation at the feature points FP extracted from each of a plurality of captured images from different viewpoints. In this case, the three-dimensional coordinate information acquisition unit 274 acquires the three-dimensional position of the feature point FP by parallax calculation based on a plurality of captured images from different viewpoints. The obstacle detection section 277 detects obstacles based on the three-dimensional position acquired by the three-dimensional coordinate information acquisition section 274 . That is, the obstacle detection unit 277 detects an obstacle by moving stereo based on the feature points FP with sufficient parallax.

However, as described above, a situation may occur in which it is difficult to obtain sufficient parallax necessary for obtaining or estimating the three-dimensional position of the feature point FP. In such a situation, it becomes difficult to obtain the three-dimensional position of the feature point FP because the parallax calculation becomes difficult. Then, it becomes difficult to detect obstacles around the own vehicle corresponding to such feature points FP. In particular, such problems are likely to occur in situations where the following (1) and (2) are established. (1) The direction of relative movement between the subject vehicle and the object B should substantially match the direction in which the image capturing unit 21 captures the image of the object B. FIG. (2) The object B is positioned in front of the imaging unit 21 . As a typical example, FIGS. 1 and 3 show a situation in which a pole-shaped object B exists directly behind the vehicle that is reversing, and the object B is positioned in front of the rear camera CB.

In this respect, for example, there may be a case where distance measurement information is acquired in substantially the same azimuth as a plurality of feature points FP whose three-dimensional positions have not been acquired due to insufficient parallax. Specifically, the ranging information corresponding to the distance to the object B is acquired by the ranging information acquisition unit 276, and a plurality of feature points are obtained in the vicinity of the ranging point DP corresponding to the acquired ranging information. FPs are arranged in a predetermined manner. In this case, there is a high possibility that the object B exists at the position of the ranging point DP corresponding to the ranging information.

Therefore, in the present embodiment, the point group extraction unit 275 extracts a point group G composed of a plurality of feature points FP arranged under a predetermined condition from the extraction result of the feature points FP by the feature point extraction unit 273 . Specifically, the point group extracting unit 275 extracts, as a point group G, feature points FP that intersect the horizontal line HL from among the plurality of linearly arranged feature points FP.

In this embodiment, as conditions for extracting the point group G by the point group extraction unit 275, the plurality of feature points FP are arranged in a straight line, and the plurality of feature points FP are arranged so that they intersect the horizontal line HL. It is mentioned that they are arranged. This takes into account the external shape of the pole-like object B, such as the typical example described above.

That is, for example, as shown in FIG. 3, the plurality of feature points FP corresponding to the first marking line DL1 are arranged in a straight line. However, they do not intersect the horizontal line HL and are arranged only in the area below the horizontal line HL. The same applies to the plurality of feature points FP corresponding to the second lane marking DL2. A plurality of feature points FP corresponding to the distant building BD are also arranged in a straight line, but do not intersect the horizontal line HL, and are arranged only in the area above the horizontal line HL. The same applies to a plurality of feature points FP corresponding to ceiling beams or pipes.

On the other hand, a plurality of feature points FP corresponding to the left edge in the photographed image of the pole-shaped object B located in front of the rear camera CB are arranged in a straight line. Furthermore, these plurality of feature points FP are arranged in the vertical direction while crossing the horizontal line HL. That is, these plurality of feature points FP are arranged on the first edge straight line L1 intersecting the horizontal line HL so as to straddle the horizontal line HL. Therefore, the point group extracting unit 275 extracts the first point group G1 from these plurality of feature points FP.

Similarly, a plurality of feature points FP corresponding to the right edge of the captured image of the pole-shaped object B located in front of the rear camera CB are arranged in a straight line. Furthermore, these plurality of feature points FP are arranged in the vertical direction while crossing the horizontal line HL. That is, these plurality of feature points FP are arranged on the second edge straight line L2 intersecting the horizontal line HL so as to straddle the horizontal line HL. Therefore, the point group extracting section 275 extracts the second point group G2 from these plurality of feature points FP.

The ranging information acquisition unit 276 acquires ranging information corresponding to the distance to the ranging point DP measured by the sonar sensor 22 from the sonar sensor 22 . In addition, the ranging information acquisition unit 276 stores the acquired ranging information in the non-volatile RAM in chronological order. The point group extraction unit 275 extracts the point group G within a search range RS, which is a predetermined range in the captured image with reference to the distance measuring point DP. By limiting the search range RS to a part of the field angle VA instead of the entire field angle VA, the calculation load is reduced and the search accuracy is improved.

The obstacle detection unit 277 extracts the first point group G1 and the second point group G2, which are the two point groups G extracted by the point group extraction unit 275, and the distance measurement information acquired by the distance measurement information acquisition unit 276. is in a predetermined positional relationship in the captured image, the area between the first point group G1 and the second point group G2 is detected as an object B, that is, an obstacle.

As described above, the first point group G1 and the second point group G2 arranged under predetermined conditions are extracted, and the first point group G1 and the second point group G2 and the range-finding point DP are predetermined in the captured image. There may be a positional relationship. In this case, according to the configuration of this embodiment, the area between the first point group G1 and the second point group G2 can be detected as an obstacle. Therefore, according to such a configuration, it is possible to more satisfactorily detect the object B as an obstacle existing around the own vehicle.

In addition, in the configuration of this embodiment, the obstacle detection unit 277 detects an obstacle by a moving stereo technique, which is normal logic, based on the feature points FP whose three-dimensional positions are obtained by parallax calculation. . On the other hand, the obstacle detection unit 277 detects obstacles using additional logic based on the feature points FP for which the three-dimensional positions have not been obtained. Such additional logic uses the extraction results of the feature points FP and the distance measurement information, so it can also be called sensor fusion or simply fusion.

That is, when the following two conditions are satisfied, the obstacle detection unit 277 uses additional logic to detect the area between the first point group G1 and the second point group G2 as an obstacle. (Condition 1) The first point group G1 and the second point group G2 are composed of a plurality of feature points FP whose three-dimensional positions have not been obtained. (Condition 2) The distance measurement point DP corresponding to the distance measurement information acquired by the distance measurement information acquisition unit 276 and the first point group G1 and the second point group G2 have a predetermined positional relationship in the captured image. . According to such a configuration, the application target of the additional logic is limited, so the occurrence of erroneous detection of obstacles can be suppressed satisfactorily.

(Operation example)
A specific operation example corresponding to the above outline of operation according to the configuration of the present embodiment will be described below with reference to the flowchart shown in FIG. In the drawings, "step" is simply abbreviated as "S". The CPU of the object detection ECU 27 repeatedly activates the routine shown in FIG. 4 at predetermined time intervals while a predetermined activation condition is satisfied.

When this routine is activated, first, in step 401, the CPU acquires image information. The image information acquired here is, for example, image information acquired by the rear camera CB while the host vehicle is moving backward. That is, when the host vehicle is moving backward, the CPU acquires image information from the rear camera CB.

Next, at step 402, the CPU extracts feature points FP based on the acquired image information. Subsequently, at step 403, the CPU acquires the movement direction and movement amount of the host vehicle. Subsequently, at step 404, the CPU acquires or calculates the three-dimensional position of the feature point FP by parallax calculation.

Among the large number of feature points FP extracted in step 402, as described above, there are those whose three-dimensional positions are obtained by parallax calculation and those whose three-dimensional positions are not obtained. Therefore, following step 404, the CPU once executes steps 405 and 406, and then terminates this routine. At step 405, the CPU determines whether or not there is an object B around the host vehicle by a monocular moving stereo method, which is normal logic. Further, at step 406, the CPU determines whether or not there is an object B around the host vehicle by sensor fusion, which is additional logic.

The processing of step 405 is normal processing in monocular moving stereo. Such processing is already well known at the time of filing of this application. Therefore, in this specification, the detailed description of the processing contents of step 405 is omitted.

FIG. 5 shows details of the processing contents of step 406 in FIG. Such processing contents are shown as steps 501 to 510 in FIG. These will be described in order below.

First, at step 501, the CPU extracts residual feature points. The remaining feature points are feature points FP extracted in step 402 that are not used to determine whether or not there is an object B around the host vehicle. It should be noted that, at the time when the process of step 501 is executed for the first time, the remaining feature points are all the feature points FP extracted in step 402 excluding the first used feature points. The first used feature point is associated with the object B around the host vehicle in step 405 by the monocular moving stereo technique. At the time when the process of step 501 is executed for the first time, the remaining feature points include feature points FP whose three-dimensional positions have not been obtained by parallax calculation. Also includes those that were not associated with the object B in .

Next, at step 502, the CPU acquires distance measurement information. That is, the CPU acquires one distance measuring point DP. Subsequently, at step 503, the CPU sets a search range RS around the acquired distance measuring point DP. Also, at step 504, the CPU extracts the point group G within the search range RS.

At step 505, the CPU determines whether the parallax calculation is established for the point group G extracted at step 504 or not. If the parallax calculation is established (that is, step 505 =YES), the CPU advances the process to step 506 . At step 506, the CPU determines an object B, ie, an obstacle, for the point group G extracted this time, and excludes it from the remaining feature points by the monocular moving stereo method. After that, the CPU advances the process to step 507 .

At step 507, the CPU determines whether or not there are remaining feature points. If there are no remaining feature points (that is, step 507=NO), the CPU terminates the processing of step 406 in FIG. On the other hand, if there are remaining feature points (that is, step 507 =YES), the CPU returns the process to step 501 . In this case, in subsequent step 502, a range-finding point DP different from that previously obtained is obtained.

If the parallax calculation does not hold for the point group G extracted in step 504 (that is, step 505 =NO), the CPU advances the process to step 508 . At step 508, the CPU determines whether or not the positional relationship condition is satisfied. The positional relationship condition is that the positional relationship between the first point group G1 and the second point group G2 extracted this time and the range-finding point DP is in a predetermined positional relationship.

If the positional relationship condition is satisfied (that is, step 508 =YES), the CPU advances the process to step 509 . At step 509, the CPU determines whether the point group G extracted this time, ie, the first point group G1 and the second point group G2, is an object B, ie, an obstacle. Specifically, the area between the first point group G1 and the second point group G2 is determined as the object B, that is, the obstacle. Also, the CPU excludes the point group G extracted this time from the remaining feature points. After that, the CPU advances the process to step 507 . The contents of processing in step 507 are the same as above.

If the positional relationship condition does not hold (that is, step 508 =NO), the CPU advances the process to step 510 . At step 510, the CPU determines that the point group G extracted this time does not correspond to the object B, that is, the obstacle, and excludes it from the remaining feature points. After that, the CPU advances the process to step 507 . The contents of processing in step 507 are the same as above.

(Modification)
The present invention is not limited to the above embodiments. Therefore, the above embodiment can be modified as appropriate. A representative modified example will be described below. In the following description of the modified example, differences from the above embodiment will be mainly described. Moreover, in the above-described embodiment and modifications, the same or equivalent portions are denoted by the same reference numerals. Therefore, in the description of the modification below, the description in the above embodiment can be used as appropriate for components having the same reference numerals as those in the above embodiment, unless there is a technical contradiction or special additional description.

The present invention is not limited to the specific device configurations shown in the above embodiments. That is, for example, the vehicle 10 equipped with the object detection device 20 is not limited to a four-wheeled vehicle. Specifically, the vehicle 10 may be a three-wheeled vehicle, or a six-wheeled or eight-wheeled vehicle such as a freight truck.

The type of vehicle 10 may be an automobile equipped only with an internal combustion engine, an electric vehicle or a fuel cell vehicle without an internal combustion engine, or a so-called hybrid vehicle. The shape and structure of the vehicle body 11 are also not limited to a box shape, that is, a substantially rectangular shape in plan view. The number of door panels 17 is also not particularly limited.

There is also no particular limitation on the application target of the object detection device 20 . That is, for example, the object detection device 20 can be applied to parking assistance. Alternatively, the object detection device 20 can be suitably applied to semi-automatic driving or automatic driving corresponding to levels 2 to 5 in the definition of automatic driving.

The image sensor that constitutes the imaging unit 21 is not limited to a CCD sensor. That is, for example, CMOS sensors can be used instead of CCD sensors. CMOS is an abbreviation for Complementary MOS.

The arrangement and number of imaging units 21 are not limited to the above example. That is, for example, the front camera CF can be arranged outside the vehicle compartment. Specifically, for example, the front camera CF can be attached to the front portion 12 of the vehicle body 11 . The number of front cameras CF may be one or two. That is, the object detection device 20 may have a compound-eye stereo camera configuration. For example, the left camera CL and the right camera CR may be arranged at positions different from the door mirrors 18 . Alternatively, the left camera CL and right camera CR may be omitted.

The arrangement and number of sonar sensors 22 are not limited to the above specific examples. That is, for example, referring to FIG. 1, when the third front sonar SF3 is arranged at the center position in the vehicle width direction, the fourth front sonar SF4 is omitted. Similarly, when the third rear sonar SR3 is arranged at the central position in the vehicle width direction, the fourth rear sonar SR4 is omitted. The third side sonar SS3 and fourth side sonar SS4 may be omitted.

The object detection ECU 27 constitutes a main part of the object detection device 20 in the above embodiment. Therefore, the imaging unit 21, the sonar sensor 22, the radar sensor 23, the vehicle speed sensor 24, the shift position sensor 25, the steering angle sensor 26, the display unit 28, and the audio output unit 29 are not components of the object detection device 20, but rather It may be understood to be an ancillary element of sensing device 20 . Alternatively, the imaging unit 21 and the sonar sensor 22 may be grasped as components of the object detection device 20 as components of the image information acquisition unit 271 and the ranging information acquisition unit 276, respectively.

In the above-described embodiment, the object detection ECU 27 is configured such that the CPU reads a program from the ROM or the like and starts it. However, the invention is not limited to such a configuration. That is, for example, the object detection ECU 27 may be a digital circuit, such as an ASIC such as a gate array, capable of performing the above operations. ASIC is an abbreviation for APPLICATION SPECIFIC INTEGRATED CIRCUIT.

The present invention is not limited to the specific functional configurations and operation examples shown in the above embodiments. That is, for example, the movement information acquisition unit 272 may use the output of an acceleration sensor (not shown) to acquire the amount of movement of the host vehicle.

In the above specific example, the operation of detecting the object B using the image captured by the rear camera CB has been described. However, the invention is not limited to such aspects. That is, for example, the present invention can be suitably applied to the operation of detecting an object B using an image captured by the front camera CF. Similarly, the present invention can also be suitably applied to the operation of detecting an object B using images captured by the left camera CL and the right camera CR.

The feature point FP may be composed of a single pixel, or may be composed of an aggregate of a plurality of pixels. Saturation information may be used in place of or along with luminance information when detecting feature points FP. However, when image information is acquired through optical filters of RGB colors, saturation information can be synonymous with luminance information in image information of RGB colors.

The processing content of the three-dimensional coordinate information acquisition unit 274 is not limited to monocular movement stereo. Specifically, for example, compound eye stereo processing or integrated processing of SFM and compound eye stereo may be used. Compound-eye stereo processing, or integration processing of SFM and compound-eye stereo, is already publicly known or well-known at the time of filing of this application.

The height of the search range RS in extracting the point group G may be the same as the height of the angle of view VA. That is, the search range RS may limit the extraction range of the point group G in the width direction of the angle of view VA. Also, the shape of the search range RS is not limited to a rectangular shape. That is, for example, the search range RS may be an ellipse centered on the distance measuring point DP or a circle with a predetermined radius.

The ranging information acquisition unit 276 may acquire ranging information based on the output of the radar sensor 23 instead of or together with the output of the sonar sensor 22 . That is, ultrasonic waves or electromagnetic waves can be used as probe waves. A reflection point acquired by the radar sensor 23 may be used as the ranging point DP. In this case, the relative position information regarding the reflection point can be used as a substitute for the ranging point DP acquired by the sonar sensor 22 . Alternatively, the relative position information about the reflection points can be used as a correction factor for the ranging point DP acquired by the sonar sensor 22 .

In the above-described embodiment, the predetermined positional relationship, that is, the positional relationship condition, is that the distance of the first point group G1 and the second point group G2 from the distance measuring point DP in the captured image is a predetermined distance or less. However, the invention is not limited to such aspects.

Specifically, for example, the predetermined positional relationship may include an AND condition of condition MA1 and condition MB1 below. (Condition MA1) The distance of the first point group G1 or the second point group G2 from the distance measuring point DP in the captured image is equal to or less than a predetermined distance. (Condition MB1) The interval in the captured image between the first point group G1 and the second point group G2 is equal to or less than a predetermined interval.

For example, as shown in FIG. 3, a feature point FP whose three-dimensional position can be acquired may occur at the base of an elongated pole-shaped object B that extends in the vertical direction. Such a feature point FP is called a reference feature point FP0. The base of the object B is also called a position where the object B intersects with the road surface or the ground. Note that the reference feature point FP0 can be generated not only at the base of the object B, but also in correspondence with stains or patterns on the surface of the object B. FIG.

As described above, not only are the first point group G1 and the second point group G2 and the distance measuring point DP in the vicinity thereof in a predetermined positional relationship in the photographed image, but also the direction substantially the same as that of the distance measuring point DP. When the reference feature point FP0 is detected in , the existence probability of the object B, that is, the obstacle increases. Therefore, the predetermined positional relationship may further include a condition regarding the reference feature point FP0.

Specifically, the predetermined positional relationship may include an AND condition of condition MA2 and condition MB2 below. (Condition MA2) The reference feature point FP0, which is the feature point FP whose three-dimensional position is acquired, exists between the first point group G1 and the second point group G2. (Condition MB2) The distance difference ΔD is equal to or less than a predetermined value ΔD0. The distance difference ΔD is the difference between the reference distance, which is the distance of the reference feature point FP0 from the own vehicle, and the distance measurement distance, which is the distance of the distance measurement point DP from the own vehicle.

That is, in this modified example, the positional relationship condition includes the second positional relationship condition in addition to the first positional relationship condition. The first positional relationship condition is the positional relationship condition in the above embodiment. The second positional relationship condition is an AND condition of condition MA2 and condition MB2, and is a condition regarding reference feature point FP0.

FIG. 6 shows a part of the flow chart of FIG. 5 modified in accordance with such a modification. Note that the processing of steps 501 to 506 in FIG. 5 is the same in this modified example. That is, the processing of this modification corresponding to FIG. 6 also includes the processing of steps 501 to 506 in FIG. The processing of this modified example differs from the processing of the above-described embodiment only in the contents after step 508 . Therefore, in FIG. 6, the processing of steps 501 to 506 in FIG. 5 is omitted.

At step 508, the CPU determines whether or not the first positional relationship condition is satisfied. The first positional relationship condition is that the positional relationship between the first point group G1 and the second point group G2 extracted this time and the distance measuring point DP is in a predetermined positional relationship. That is, the processing contents of step 508 in this modified example are the same as those in the above-described embodiment.

If the first positional relationship condition does not hold (that is, step 508 =NO), the CPU advances the process to step 510 . At step 510, the CPU determines that the point group G extracted this time does not correspond to the object B, that is, the obstacle, and excludes it from the remaining feature points. After that, the CPU advances the process to step 507 . The processing contents in step 507 are the same as those in the above embodiment.

If the first positional relationship condition is satisfied (that is, step 508 =YES), the CPU advances the process to step 511 . At step 511, the CPU determines whether or not the reference feature point FP0 exists between the first point group G1 and the second point group G2. If the reference feature point FP0 does not exist between the first point group G1 and the second point group G2 (that is, step 511=NO), the CPU advances the process to step 510.

If the reference feature point FP0 exists between the first point group G1 and the second point group G2 (that is, step 511=YES), the CPU advances the process to step 512. At step 512, the CPU determines whether or not the distance difference ΔD is equal to or less than a predetermined value ΔD0. That is, step 511 and step 512 determine whether the second positional relationship condition is met.

If the distance difference ΔD is less than or equal to the predetermined value ΔD0 (that is, step 512=YES), the CPU advances the process to step 509. At step 509, the CPU determines whether the point group G extracted this time, ie, the first point group G1 and the second point group G2, is an object B, ie, an obstacle. After that, the CPU advances the process to step 507 . On the other hand, if the distance difference ΔD exceeds the predetermined value ΔD0 (that is, step 512=NO), the CPU advances the process to step 510.

The expression "acquire" and similar expressions such as "estimation", "detection", "detection", and "calculation" can be appropriately replaced within a technically consistent range. Both "detection" and "extraction" can be interchanged as long as they are not technically inconsistent. The inequality sign in each determination process may or may not have an equal sign. That is, for example, "less than a predetermined value" and "not more than a predetermined value" can be interchanged within a technically consistent range.

Needless to say, the elements constituting the above-described embodiments are not necessarily essential, unless explicitly stated as essential or clearly considered essential in principle. In addition, when numerical values such as the number, numerical value, amount, range, etc. of a constituent element are mentioned, unless it is explicitly stated that it is particularly essential or when it is clearly limited to a specific number in principle, The present invention is not limited to the number of . Similarly, when the shape, direction, positional relationship, etc. of a component, etc. are mentioned, unless it is explicitly stated that it is particularly essential, or when it is limited to a specific shape, direction, positional relationship, etc. in principle , the shape, direction, positional relationship, etc., of which the present invention is not limited.

Modifications are also not limited to the above examples. Also, multiple variants can be combined with each other. Furthermore, all or part of the above embodiments and all or part of the modifications may be combined with each other.

10 vehicle 20 object detection device 271 image information acquisition unit 272 movement information acquisition unit 273 feature point extraction unit 274 three-dimensional coordinate information acquisition unit 275 point group extraction unit 276 ranging information acquisition unit 277 obstacle detection unit B object

Claims (9)

An object detection device (20) configured to detect an object (B) existing around the own vehicle (10) by being mounted on the own vehicle (10),
an image information acquisition unit (271) for acquiring image information corresponding to a photographed image of the surroundings of the own vehicle;
a feature point extraction unit (273) for extracting feature points (FP) in the captured image based on the image information acquired by the image information acquisition unit;
a three-dimensional information acquisition unit (274) that acquires the three-dimensional position of the feature point based on the plurality of captured images from different viewpoints;
a point group extraction unit (275) for extracting a point group (G) consisting of a plurality of the feature points arranged under a predetermined condition from the result of extraction of the feature points by the feature point extraction unit;
a ranging information acquisition unit (276) that acquires ranging information corresponding to the distance to the ranging point (DP) measured by the ranging sensor (22);
Corresponding to the first point group (G1) and the second point group (G2), which are the two point groups extracted by the point cloud extraction unit, and the ranging information acquired by the ranging information acquisition unit an obstacle detection unit ( 277) and
with
In the obstacle detection unit, the first point group and the second point group are configured by a plurality of the feature points for which the three-dimensional position was not acquired, and the distance measurement information acquisition unit acquires and when the range-finding point corresponding to the range-finding information and the first point group and the second point group are in the predetermined positional relationship in the captured image, the first point group and the second point detecting the area between the group as the obstacle;
The predetermined positional relationship is such that a reference feature point (FP0), which is the feature point from which the three-dimensional position is acquired, exists between the first point group and the second point group, and the reference feature point The difference between the reference distance, which is the distance from the own vehicle, and the ranging distance, which is the distance of the ranging point from the own vehicle, is less than or equal to a predetermined value,
Object detection device. An object detection device (20) configured to detect an object (B) existing around the own vehicle (10) by being mounted on the own vehicle (10),
an image information acquisition unit (271) for acquiring image information corresponding to a photographed image of the surroundings of the own vehicle;
a feature point extraction unit (273) for extracting feature points (FP) in the captured image based on the image information acquired by the image information acquisition unit;
a point group extraction unit (275) for extracting a point group (G) consisting of a plurality of the feature points arranged under a predetermined condition from the result of extraction of the feature points by the feature point extraction unit;
a ranging information acquisition unit (276) that acquires ranging information corresponding to the distance to the ranging point (DP) measured by the ranging sensor (22);
Corresponding to the first point group (G1) and the second point group (G2), which are the two point groups extracted by the point cloud extraction unit, and the ranging information acquired by the ranging information acquisition unit an obstacle detection unit ( 277) and
with
The predetermined positional relationship includes that the distance of the first point group and the second point group from the range-finding point in the captured image is equal to or less than a predetermined distance ,
Object detection device. An object detection device (20) configured to detect an object (B) existing around the own vehicle (10) by being mounted on the own vehicle (10),
an image information acquisition unit (271) for acquiring image information corresponding to a photographed image of the surroundings of the own vehicle;
a feature point extraction unit (273) for extracting feature points (FP) in the captured image based on the image information acquired by the image information acquisition unit;
a point group extraction unit (275) for extracting a point group (G) consisting of a plurality of the feature points arranged under a predetermined condition from the result of extraction of the feature points by the feature point extraction unit;
a ranging information acquisition unit (276) that acquires ranging information corresponding to the distance to the ranging point (DP) measured by the ranging sensor (22);
Corresponding to the first point group (G1) and the second point group (G2), which are the two point groups extracted by the point cloud extraction unit, and the ranging information acquired by the ranging information acquisition unit an obstacle detection unit ( 277) and
with
The predetermined positional relationship is
The distance of the first point group or the second point group from the range-finding point in the captured image is a predetermined distance or less, and
Including that the interval in the captured image between the first point group and the second point group is equal to or less than a predetermined interval ,
Object detection device. further comprising a three-dimensional information acquisition unit (274) that acquires the three-dimensional position of the feature point based on the plurality of captured images from different viewpoints,
In the obstacle detection unit, the first point group and the second point group are configured by a plurality of the feature points for which the three-dimensional position was not acquired, and the distance measurement information acquisition unit acquires and when the range-finding point corresponding to the range-finding information and the first point group and the second point group are in the predetermined positional relationship in the captured image, the first point group and the second point detecting the area between the group as the obstacle;
The object detection device according to claim 2 or 3 . The predetermined positional relationship is such that a reference feature point (FP0), which is the feature point from which the three-dimensional position is acquired, exists between the first point group and the second point group, and the reference feature point The difference between the reference distance, which is the distance from the own vehicle, and the ranging distance, which is the distance of the ranging point from the own vehicle, is less than or equal to a predetermined value,
The object detection device according to claim 4 . The point group extraction unit extracts the plurality of characteristic points arranged in a straight line as the point group,
The object detection device according to any one of claims 1 to 5 . The point cloud extraction unit extracts a plurality of the feature points arranged to intersect a horizontal line (HL) as the point cloud,
The object detection device according to any one of claims 1 to 6 . The point cloud extraction unit extracts the point cloud within a predetermined range in the captured image with reference to the ranging point corresponding to the ranging information acquired by the ranging information acquisition unit. ,
The object detection device according to any one of claims 1 to 7 . The ranging information acquisition unit transmits a reflected wave from the object of the search wave, which is an ultrasonic wave transmitted toward the outside of the vehicle, to the ranging sensor mounted on the vehicle body (11) of the vehicle. obtaining the ranging information based on the results received by
The object detection device according to any one of claims 1 to 8.

Images