JP2017207874A - Image processing apparatus, imaging apparatus, moving body device control system, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the accuracy in detecting a road surface.SOLUTION: An image processing device comprises: a creation part that creates, from a distance image having a distance value according to a road surface in a plurality of photographed images photographed respectively by a plurality of imaging parts and the distance to an obstacle blocking the road surface, vertical direction distribution data indicating a distribution of the frequency of the distance value with respect to the vertical direction of the distance image; an estimation part that estimates the height of the road surface according to the distance on the basis of the vertical direction distribution data; and a modification part that modifies to lower the height of the road surface estimated by the estimation part in the distance of the obstacle.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing device, an imaging device, a mobile device control system, an image processing method, and a program.

  Conventionally, in the safety of automobiles, body structures of automobiles have been developed from the viewpoint of how to protect pedestrians and protect passengers when they collide with pedestrians and automobiles. However, in recent years, with the development of information processing technology and image processing technology, technology for detecting people, cars, etc. at high speed has been developed. Automobiles that apply these technologies to automatically apply a brake before a collision to prevent the collision are already on the market.

  In order to automatically apply a brake, it is necessary to measure the distance to an object such as a person or another vehicle. For this reason, measurement using an image of a stereo camera has been put into practical use.

  In the measurement using the image of the stereo camera, a technique for recognizing an object such as a person or another vehicle by performing the following image processing is known (for example, see Patent Document 1).

  First, from a plurality of images captured by a stereo camera, a V-Disparity image having the vertical coordinate of the image as one axis, the parallax of the image as the other axis, and the frequency value of the parallax as a pixel value Is generated. Next, a road surface is detected from the generated V-Disparity image. After removing noise using the detected road surface, a U-Disparity image having the horizontal coordinate of the image as the vertical axis, the parallax of the image as the horizontal axis, and the parallax frequency as a pixel value is generated. Based on the generated U-Disparity image, an object such as a person or other vehicle is recognized.

  However, the conventional technology is affected by the parallax of obstacles such as side walls and guard rails when there are side walls or guard rails of highways in front of the stereo camera due to, for example, a curved road surface. There is a problem that the road surface cannot be measured accurately.

  Therefore, an object is to provide a technique for improving the accuracy of detecting a road surface.

  In the image processing apparatus, a distance value with respect to a vertical direction of the distance image is determined from a distance image having a distance value corresponding to a road surface in a plurality of captured images captured by a plurality of imaging units and a distance of an obstacle that blocks the road surface. A generation unit that generates vertical distribution data indicating a frequency distribution, an estimation unit that estimates the height of the road surface according to the distance based on the vertical distribution data, and the estimation unit A correction unit that corrects the height of the road surface at the obstacle distance to a low level.

  According to the disclosed technology, it is possible to improve the accuracy of detecting a road surface.

It is a figure which shows the structure of the vehicle equipment control system as a moving body apparatus control system which concerns on embodiment of this invention. It is a schematic diagram which shows the structure of an imaging unit and an image analysis unit. It is a functional block diagram of a mobile device control system. It is a figure for demonstrating the parallax image data and the V map produced | generated from the parallax image data. It is a figure which shows the image example of the picked-up image as a reference | standard image imaged with one imaging part, and the V map corresponding to the picked-up image. It is a figure which shows the image example of the picked-up image as a reference | standard image imaged with one imaging part, and the V map of the predetermined area | region corresponding to the picked-up image. It is a flowchart which shows an example of a road surface estimation process. It is a figure explaining the example in case the road surface which has a side wall is curving. It is a flowchart which shows an example of the termination | terminus segment correction process which concerns on 1st Embodiment. It is a figure explaining the example of the process which corrects the road surface of the segment far from a termination | terminus segment. It is a flowchart which shows an example of the termination | terminus segment correction process which concerns on 2nd Embodiment. It is a figure explaining the process which searches the termination | terminus segment of 2nd Embodiment. It is a flowchart which shows an example of the termination | terminus segment correction process which concerns on 3rd Embodiment. It is a figure explaining the correspondence of a picked-up image and a V map. It is a figure explaining the corresponding | compatible relationship between the picked-up image of obstructions, such as a side wall, and a V map. It is a flowchart which shows an example of the road surface estimation process which concerns on 4th Embodiment. It is a figure explaining an example of the sample point extraction process which concerns on 4th Embodiment. It is a figure explaining the termination | terminus segment correction process in an uphill. It is a flowchart which shows an example of the termination | terminus segment correction process which concerns on 5th Embodiment.

  Hereinafter, a mobile device control system having an image processing apparatus according to an embodiment will be described.

[First Embodiment]
<Configuration of in-vehicle device control system>
FIG. 1 is a diagram showing a configuration of an in-vehicle device control system as a mobile device control system according to an embodiment of the present invention.

  The in-vehicle device control system 1 is mounted on a host vehicle 100 such as an automobile as a moving body, and includes an imaging unit 500, an image analysis unit 600, a display monitor 103, and a vehicle travel control unit 104. Then, the relative height information (relative relative) of the road surface (moving surface) in front of the host vehicle is obtained from the captured image data of the front region (imaging region) in the traveling direction of the host vehicle in which the front of the moving body is captured by the imaging unit 500. (Information indicating an inclination state) is detected, the three-dimensional shape of the traveling road surface in front of the host vehicle is detected from the detection result, and the mobile object and various in-vehicle devices are controlled using the detection result. The control of the moving body includes, for example, warning notification, control of the handle of the own vehicle 100 (own moving body), or braking of the own vehicle 100 (own moving body).

  The imaging unit 500 is installed near a room mirror (not shown) of the windshield 105 of the host vehicle 100, for example. Various data such as captured image data obtained by imaging by the imaging unit 500 is input to an image analysis unit 600 as image processing means.

  The image analysis unit 600 analyzes the data transmitted from the imaging unit 500, and is on the traveling road surface in front of the own vehicle with respect to the road surface portion where the own vehicle 100 is traveling (the road surface portion located directly below the own vehicle). The relative height (position information) at each point is detected, and the three-dimensional shape of the traveling road surface in front of the host vehicle is grasped. Also, recognition objects such as other vehicles in front of the host vehicle, pedestrians, and various obstacles are recognized.

  The analysis result of the image analysis unit 600 is sent to the display monitor 103 and the vehicle travel control unit 104. The display monitor 103 displays captured image data and analysis results obtained by the imaging unit 500. For example, the vehicle travel control unit 104 notifies the driver of the host vehicle 100 of a warning based on the recognition result of other objects in front of the host vehicle such as other vehicles, pedestrians, and various obstacles. Driving control such as controlling the steering wheel and brakes of the host vehicle.

<Configuration of Imaging Unit 500 and Image Analysis Unit 600>
FIG. 2 is a diagram illustrating the configuration of the imaging unit 500 and the image analysis unit 600.

  The imaging unit 500 is configured with a stereo camera including two imaging units 510a and 510b as imaging means, and the two imaging units 510a and 510b are the same. The imaging units 510a and 510b respectively include imaging lenses 511a and 511b, sensor substrates 514a and 514b including two-dimensionally arranged image sensors 514a and 513b, and analog output from the sensor substrates 514a and 514b. It comprises signal processing units 515a and 515b that generate and output captured image data obtained by converting an electrical signal (an electrical signal corresponding to the amount of light received by each light receiving element on the image sensors 513a and 513b) into a digital electrical signal. ing. Luminance image data and parallax image data are output from the imaging unit 500.

  In addition, the imaging unit 500 includes a processing hardware unit 510 including an FPGA (Field-Programmable Gate Array) or the like. In order to obtain a parallax image from the luminance image data output from each of the imaging units 510a and 510b, the processing hardware unit 510 obtains the parallax value of the corresponding image portion between the captured images captured by the imaging units 510a and 510b. A parallax calculation unit 511 is provided as parallax image information generation means for calculation.

  The parallax value here refers to a comparison image for an image portion on the reference image corresponding to the same point in the imaging region, with one of the captured images captured by the imaging units 510a and 510b as a reference image and the other as a comparison image. The positional deviation amount of the upper image part is calculated as the parallax value of the image part. By using the principle of triangulation, the distance to the same point in the imaging area corresponding to the image portion can be calculated from the parallax value.

  The image analysis unit 600 includes an image processing board and the like, and includes a storage unit 601 configured by a RAM, a ROM, or the like that stores luminance image data and parallax image data output from the imaging unit 500, and recognition processing of an identification target. A CPU (Central Processing Unit) 602 that executes a computer program for performing parallax calculation control, a data I / F (interface) 603, and a serial I / F 604 are provided.

  The FPGA that configures the processing hardware unit 510 performs parallax images by performing processing that requires real-time processing on image data, for example, gamma correction, distortion correction (parallelization of left and right captured images), and parallax calculation by block matching. Are generated and written to the RAM of the image analysis unit 600. The CPU 602 of the image analysis unit 600 is responsible for the control of the image sensor controller of each of the imaging units 510a and 510b and the overall control of the image processing board, as well as detection processing of the three-dimensional shape of the road surface, guardrails and other various objects (identification targets). A program for executing a detection process of the object) is loaded from the ROM, and various processes are executed by inputting the luminance image data and the parallax image data stored in the RAM, and the processing result is displayed as the data I / F 603 or the serial I / F. Output from F604 to the outside. In executing such processing, the vehicle I / F 603 is used to input vehicle operation information such as the vehicle speed, acceleration (mainly acceleration generated in the longitudinal direction of the vehicle), steering angle, yaw rate, etc. It can also be used as a processing parameter. Data output to the outside is used as input data for controlling various devices of the host vehicle 100 (brake control, vehicle speed control, warning control, etc.).

  The imaging unit 500 and the image analysis unit 600 may be configured as the imaging device 2 that is an integrated device.

  FIG. 3 is a functional block diagram of the in-vehicle device control system 1 realized by the processing hardware unit 510, the image analysis unit 600, and the vehicle travel control unit 104 in FIG. Note that the functional unit realized by the image analysis unit 600 is realized by processing that causes the CPU 602 of the image analysis unit 600 to execute one or more programs installed in the image analysis unit 600.

  Hereinafter, processing in the present embodiment will be described.

<Parallax image generation processing>
The parallax image generation unit 11 performs parallax image generation processing for generating parallax image data (parallax image information). In addition, the parallax image generation part 11 is comprised by the parallax calculating part 511 (FIG. 2), for example.

  In the parallax image generation processing, first, the luminance image data of one imaging unit 510a of the two imaging units 510a and 510b is set as reference image data, and the luminance image data of the other imaging unit 510b is set as comparison image data, and these are used. The parallax between them is calculated to generate and output parallax image data. The parallax image data indicates a parallax image in which pixel values corresponding to the parallax value d calculated for each image portion on the reference image data are represented as pixel values of the respective image portions.

  Specifically, the parallax image generation unit 11 defines a block composed of a plurality of pixels (for example, 16 pixels × 1 pixel) centered on one target pixel for a certain row of reference image data. On the other hand, in the same row of the comparison image data, a block having the same size as the block of the defined reference image data is shifted by one pixel in the horizontal line direction (x direction) to show the feature of the pixel value of the block defined in the reference image data. Correlation values indicating the correlation between the feature amount and the feature amount indicating the feature of the pixel value of each block in the comparison image data are calculated. Then, based on the calculated correlation value, a matching process is performed for selecting a block of comparison image data that is most correlated with the block of reference image data among the blocks in the comparison image data. Thereafter, a positional deviation amount between the target pixel of the block of the reference image data and the corresponding pixel of the block of the comparison image data selected by the matching process is calculated as the parallax value d. The parallax image data can be obtained by performing such processing for calculating the parallax value d for the entire area of the reference image data or a specific area.

  As the feature amount of the block used for the matching process, for example, the value (luminance value) of each pixel in the block can be used, and as the correlation value, for example, the value (luminance) of each pixel in the block of the reference image data Value) and the sum of absolute values of the differences between the values (luminance values) of the pixels in the block of comparison image data corresponding to these pixels, respectively. In this case, it can be said that the block having the smallest sum is most correlated.

  When the matching processing in the parallax image generation unit 11 is realized by hardware processing, for example, SSD (Sum of Squared Difference), ZSSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), ZSAD ( Methods such as Zero-mean Sum of Absolute Difference and NCC (Normalized Cross Correlation) can be used. In the matching process, only a parallax value in units of pixels can be calculated. Therefore, when a sub-pixel level parallax value less than one pixel is required, an estimated value needs to be used. As the estimation method, for example, an equiangular straight line method, a quadratic curve method, or the like can be used.

<V map generation processing>
The V map generation unit 12 performs V map generation processing for generating a V map (V-Disparity Map, an example of “vertical direction distribution data”) based on the parallax pixel data. Each piece of parallax pixel data included in the parallax image data is indicated by a set (x, y, d) of an x-direction position, a y-direction position, and a parallax value d. This is converted into three-dimensional coordinate information (d, y, f) in which d is set on the X-axis, y is set on the Y-axis, and frequency f is set on the Z-axis, or this three-dimensional coordinate information (d, y, f) 3D coordinate information (d, y, f) limited to information exceeding a predetermined frequency threshold is generated as parallax histogram information. The parallax histogram information of this embodiment is composed of three-dimensional coordinate information (d, y, f), and this three-dimensional histogram information distributed in an XY two-dimensional coordinate system is called a V map.

  More specifically, the V map generation unit 12 calculates a parallax value frequency distribution for each row region obtained by dividing a parallax image into a plurality of vertical directions. Information indicating the parallax value frequency distribution is parallax histogram information.

  FIG. 4 is a diagram for explaining parallax image data and a V map generated from the parallax image data. 4A is a diagram illustrating an example of the parallax value distribution of the parallax image, and FIG. 5B is a diagram illustrating a V map illustrating the parallax value frequency distribution for each row of the parallax image of FIG. 4A.

  When parallax image data having a parallax value distribution as shown in FIG. 4A is input, the V map generation unit 12 calculates a parallax value frequency distribution that is a distribution of the number of pieces of data of each parallax value for each row, and This is output as parallax histogram information. The two-dimensional orthogonal coordinate system in which the parallax value frequency distribution information of each row obtained in this way is obtained by taking the y-direction position on the parallax image (the vertical position of the captured image) on the Y-axis and the parallax value d on the X-axis. By expressing the above, a V map as shown in FIG. 4B can be obtained. This V map can also be expressed as an image in which pixels having pixel values corresponding to the frequency f are distributed on the two-dimensional orthogonal coordinate system.

  FIG. 5 is a diagram illustrating an example of a captured image as a reference image captured by one imaging unit and a V map corresponding to the captured image. Here, FIG. 5A is a captured image, and FIG. 5B is a V map. That is, the V map shown in FIG. 5B is generated from the captured image as shown in FIG. 5A. In the V map, no parallax is detected in the area below the road surface, and therefore the parallax is not counted in the area A indicated by the oblique lines.

  In the image example shown in FIG. 5A, a road surface 401 on which the host vehicle is traveling, a preceding vehicle 402 that exists in front of the host vehicle, and a utility pole 403 that exists outside the road are shown. The V map shown in FIG. 5B includes a road surface 501, a preceding vehicle 502, and a utility pole 503 corresponding to the image example.

  This image example is a virtual road surface obtained by extending the front road surface of the host vehicle to a relatively flat road surface, that is, the front road surface of the host vehicle is parallel to the road surface portion immediately below the host vehicle. This corresponds to the case where the reference road surface (virtual reference moving surface) is matched. In this case, in the lower part of the V map corresponding to the lower part of the image, the high-frequency points are distributed in a substantially straight line having an inclination such that the parallax value d decreases toward the upper part of the image. Pixels showing such a distribution are pixels that are present at almost the same distance in each row on the parallax image, have the highest occupation ratio, and project an identification object whose distance continuously increases toward the top of the image. It can be said that.

  Since the imaging unit 510a captures an area in front of the host vehicle, as shown in FIG. 5A, the parallax value d of the road surface decreases as the image moves upward. Also, in the same row (horizontal line), the pixels that project the road surface have substantially the same parallax value d. Accordingly, the high-frequency points distributed in a substantially straight line on the V map correspond to the characteristics of the pixels displaying the road surface (moving surface). Therefore, it is possible to estimate that the pixels of the points distributed on or near the approximate straight line obtained by linearly approximating high-frequency points on the V map are the pixels displaying the road surface with high accuracy. Further, the distance to the road surface portion projected on each pixel can be obtained with high accuracy from the parallax value d of the corresponding point on the approximate straight line.

  FIG. 6 is a diagram illustrating an example of a captured image as a reference image captured by one imaging unit, and a V map of a predetermined region corresponding to the captured image.

  The V map generation unit 12 may generate a V map using all pixels in the parallax image, or a predetermined area (for example, FIG. 6A is an example of a captured image that is a basis of the parallax image) (for example, The V map may be generated using only pixels in a region where the road surface can be seen. For example, as the road surface becomes farther away, the road surface becomes narrower toward the vanishing point. Therefore, as shown in FIG. 6A, an area corresponding to the width of the road surface may be set. Thereby, it is possible to prevent noise caused by an object (for example, the utility pole 403) located in an area other than the area where the road surface can be captured from being mixed into the V map.

<Road estimation>
The road surface estimation unit 13 estimates (detects) the road surface based on the parallax image generated by the parallax image generation unit 11. Hereinafter, the process of the road surface estimation part 13 is demonstrated. FIG. 7 is a flowchart illustrating an example of a road surface estimation process.

  The road surface estimation unit 13 divides the V map into a plurality of segments according to the value of the parallax d that is the horizontal axis of the V map (step S11). By dividing and processing into a plurality of segments, it is possible to capture a road surface having a complicated shape such as a hill shape that rises and falls halfway from a flat road.

  Subsequently, the road surface estimation unit 13 extracts sample points from the pixels of the V map (step S12). For example, the road surface estimation unit 13 may extract one or more sample points for each coordinate position of each parallax value d. Alternatively, the road surface estimation unit 13 may extract the most frequent point as the sample point at the coordinate position of each parallax value d. Alternatively, the road surface estimation unit 13 has a frequency at a plurality of d coordinates including each parallax value d (for example, the coordinate position of each parallax value d and at least one of the left and right coordinate positions of each parallax value d). The most frequent point with the largest number may be extracted as a sample point.

  Subsequently, the road surface estimation unit 13 excludes inappropriate points from the extracted sample points (step S13). The road surface estimation unit 13 calculates an approximate straight line corresponding to each extracted sample point using, for example, the least square method. Then, sample points whose Euclidean distance is more than a predetermined threshold are removed from the calculated approximate straight line.

  Subsequently, the road surface estimation unit 13 detects the shape (position, height) of the road surface based on the sample points not excluded in Step S13 (Step S14). The road surface estimation unit 13 calculates an approximate straight line from the sample points of each segment by, for example, the least square method, and detects (estimates) the calculated approximate straight line as a road surface. Alternatively, the road surface shape detection unit 133 may detect (estimate) the curve shape calculated from the sample points of each segment using polynomial approximation or the like as the road surface.

  Subsequently, the road surface estimation unit 13 supplements the road surface when the detected (estimated) road surface is inappropriate (step S15). For example, when an inappropriate road surface that cannot be captured by a stereo camera is estimated due to noise, the road surface estimation unit 13 is based on the default road surface or road surface data estimated in a previous frame. To supplement (interpolate) inappropriate road surfaces.

  Subsequently, the road surface estimation unit 13 corrects (smoothing processing) each road surface estimated in each segment so that the respective road surfaces are continuous (step S16). The road surface estimation unit 13, for example, in each road surface estimated in two adjacent segments, the inclination of each road surface so that the end point (end point) of one road surface and the start point (end point) of the other road surface coincide with each other. And change the intercept.

<End segment correction>
The terminal segment correction unit 14 corrects the height of the road surface at the obstacle distance estimated by the road surface estimation unit 13 to be low.

  In the first embodiment, the terminal segment modification unit 14 divides the V map into a plurality of segments according to the parallax value, and a pixel (hereinafter referred to as a parallax frequency value) included in each divided segment is equal to or greater than a predetermined value. The “parallax point” is an example of “distance point”).

  Then, the end segment correction unit 14 determines the segment (second segment) that is the farthest from the host vehicle among the segments (end candidates) whose number of parallax points is equal to or greater than a predetermined threshold.

  Then, if there is a first segment that is farther away from the host vehicle than the second segment, the terminal segment correction unit 14 corrects the height of the road surface in the second segment.

  Next, with reference to FIGS. 8 and 9, a detailed example of the processing of the terminal segment correction unit 14 will be described.

  FIG. 8 is a diagram illustrating an example in the case where a road surface having a side wall is curved. FIG. 8A is a diagram illustrating an image example of a captured image as a reference image captured by one imaging unit when a road surface having a side wall is curved. FIG. 8B is a diagram illustrating an example of a V map of a predetermined area corresponding to the captured image of FIG.

  As shown in FIG. 8A, when there is an obstacle such as a side wall in the distance from the host vehicle and the distance cannot be seen, as shown in FIG. 8B, the distant from the segment where the parallax of the obstacle exists. In the segment, there is no parallax distribution (pixel).

  In this case, in the segment where the parallax of the obstacle exists, the road surface estimation unit 13 may erroneously detect the height of the road surface as 521 in FIG. 8B due to the parallax of the obstacle.

  Further, in a segment farther than the segment where the parallax of the obstacle exists, the road surface estimation unit 13 detects the road surface as indicated by 522 in FIG. 8B according to an inappropriate pixel due to noise or the like, for example. There is a case.

  Therefore, the terminal segment correction unit 14 performs the following processing. FIG. 9 is a flowchart illustrating an example of a terminal segment correction process according to the first embodiment.

  The terminal segment correction unit 14 divides the V map into a plurality of segments (step S101). Note that the size, shape, and number of each segment to be divided may be any.

  Subsequently, the terminal segment correction unit 14 counts the parallax points having a value equal to or greater than a predetermined value (for example, 1) in each divided segment (step S102). The predetermined value may be 2 or more in order to remove noise, or may be an average value of the parallax frequency in each segment.

  Subsequently, the end segment correcting unit 14 sets a segment having a count number equal to or greater than a predetermined threshold as segment A (end candidate) and a segment having a count number less than the predetermined threshold as segment B (non-end candidate) (step S103). ). Thus, in the example of FIG. 8B, the first segment to the fourth segment that are relatively close to the host vehicle are segment A, and the fifth to seventh segments that are relatively far from the host vehicle are segment. B is determined.

  Subsequently, the end segment correction unit 14 evaluates in order from the segment far from the host vehicle, and determines the segment A when the segment B is changed to the segment A as the end segment (step S104). By searching in order from the segment farthest from the host vehicle, the end segment is correctly detected even if the segment closer to the host vehicle than the end segment is determined to be segment B by noise, for example. it can.

  Subsequently, the terminal segment correction unit 14 corrects the road surface so that the road surface of the terminal segment becomes low (step S105). The terminal segment modification unit 14 sets the lower road surface among the road surface estimated by the road surface estimation unit 13 for the terminal segment and the road surface estimated by a method different from the road surface estimation unit 13 as the road surface of the terminal segment. Here, the road surface present at a low height may be, for example, a road surface with a low y coordinate at the end point (left end) of the end segment, or a road surface with a low y coordinate at the middle point.

  For example, the terminal segment correction unit 14 may use the road surface calculated as described below as a road surface estimated by a method different from that of the road surface estimation unit 13.

・ Road surface estimated from the previous segment of the end segment extended to the end segment ・ Road surface estimated from previous captured image (frame) for the end segment (historic road surface)
If the segment becomes discontinuous as a result of correcting the end segment, the above-described smoothing process may be performed so that each segment is continuous.

  Thereby, the road surface in the terminal segment is corrected as indicated by 523 in FIG.

  Subsequently, the terminal segment correction unit 14 corrects the road surface of the segment that is farther from the host vehicle than the terminal segment (step S106). The end segment correction unit 14 performs, for example, the following processing on the road surface of a segment that is farther from the host vehicle than the end segment. FIG. 10 is a diagram illustrating an example of processing for correcting the road surface of a segment farther from the end segment.

  As shown in FIG. 10A, the road surface is cut off at the end segment (the road surface of the segment farther than the end segment is discarded). Thereby, it is possible to prevent the U map used for the subsequent clustering from using the parallax of the segment that is farther from the host vehicle than the terminal segment.

  As shown in FIG. 10B, the road surface of the corrected end segment is extended in a segment that is farther from the host vehicle than the end segment.

  ・ As shown in FIG. 10 (C), the road surface estimated in the previous frame for the segment farther from the host vehicle than the end segment is used as the base point of the end point (left end) of the corrected end segment road surface. Are connected in a continuous manner.

<Road height table calculation processing>
The road surface height table calculation unit 15 calculates the road surface height (relative height with respect to the road surface portion directly below the host vehicle) based on the road surface in each segment corrected by the smoothing processing unit 135 to form a table. A road surface height table calculation process is performed.

  The road surface height table calculation unit 15 calculates the distance from the road surface information in each segment to each road surface portion displayed in each row area (each position in the vertical direction of the image) on the captured image. In addition, each surface portion in the traveling direction of the own plane of the virtual plane that is extended forward in the traveling direction of the host vehicle so that the road surface portion located directly below the host vehicle is parallel to the surface is displayed in which row area in the captured image. The virtual plane (reference road surface) is represented by a straight line (reference straight line) on the V map. The road surface height table calculation unit 15 can obtain the height of each road surface portion in front of the host vehicle by comparing the road surface in each segment with a reference straight line. In a simple manner, the height of the road surface portion existing in front of the host vehicle can be calculated from the Y-axis position on the road surface in each segment by a distance obtained from the corresponding parallax value. The road surface height table calculation unit 15 tabulates the height of each road surface portion obtained from the approximate straight line for the necessary parallax range.

  Note that the height from the road surface of the object projected on the captured image portion corresponding to the point where the Y-axis position is y ′ at a certain parallax value d is y0 on the road surface at the parallax value d. Then, it can be calculated from (y′−y0). In general, the height H from the road surface of the object corresponding to the coordinates (d, y ′) on the V map can be calculated from the following equation. However, in the following equation, “z” is a distance (z = BF / (d−offset)) calculated from the parallax value d, and “f” is the focal length of the camera (y′−y0). The value converted to the same unit as the unit. Here, “BF” is a value obtained by multiplying the base line length of the stereo camera and the focal length, and “offset” is a parallax value when an object at infinity is photographed.

H = z × (y′−y0) / f
<Clustering, rejection, tracking>
The clustering unit 16 sets a set (x, y, d) of the x-direction position, the y-direction position, and the parallax value d in each piece of parallax pixel data included in the parallax image data, x on the X axis, d, Z on the Y axis. Frequency is set on the axis, and XY two-dimensional histogram information (frequency U map) is created.

  Based on the height of each road surface portion tabulated by the road surface height table calculation unit 15, the clustering unit 16 is configured to generate a parallax image whose height H from the road surface is within a predetermined height range (for example, 20 cm to 3 m). A frequency U map is created only for the point (x, y, d). In this case, an object existing within the predetermined height range from the road surface can be appropriately extracted.

  In the frequency U map, the clustering unit 16 detects a region where the frequency is higher than a predetermined value and where the parallax is dense as an object region, and determines from the coordinates on the parallax image and the actual size (size) of the object. Individual information such as the predicted object type (person or pedestrian) is given.

  The rejection unit 17 rejects information on an object that is not a recognition target, based on the parallax image, the frequency U map, and the individual information of the object.

  The tracking unit 18 determines whether or not the detected object is a tracking target when the detected object appears continuously in a plurality of parallax image frames.

<Driving support control>
Based on the detection result of the object by the clustering unit 16, the control unit 19 performs driving support control such as, for example, notifying a driver of the host vehicle 100 or controlling a handle or a brake of the host vehicle. Do.

[Second Embodiment]
In the first embodiment, the example in which the end segment is detected based on the number of parallax points in each segment has been described. In the second embodiment, an example in which the end segment correction unit 14 detects the end segment based on the number of parallax points for each parallax d of each segment will be described. Note that the second embodiment is the same as the first embodiment except for a part thereof, and thus description thereof will be omitted as appropriate.

  Below, the difference with 1st Embodiment is demonstrated about the process of the vehicle equipment control system 1 which concerns on 2nd Embodiment.

  As in the first embodiment, the terminal segment modification unit 14 according to the second embodiment first divides the V map into a plurality of segments according to the disparity values, and the disparity frequency included in each divided segment. Pixels (parallax points) whose value is greater than or equal to a predetermined value are counted.

  The end segment correction unit 14 then selects the segment (distance candidate) having the longest distance from the host vehicle among the segments (termination candidates) in which the number of parallax values d is equal to or greater than a predetermined threshold. 2nd segment) is determined.

  Then, as in the first embodiment, if there is a first segment that is farther from the vehicle than the second segment, the terminal segment correction unit 14 reduces the height of the road surface in the second segment. Correct it.

  FIG. 11 is a flowchart illustrating an example of a terminal segment correction process according to the second embodiment.

  The terminal segment correction unit 14 divides the V map into a plurality of segments (step S201).

  Subsequently, the terminal segment correction unit 14 counts d coordinates having a predetermined number (for example, 1) of pixels (parallax points) having a value equal to or greater than a predetermined value (for example, 1) for each of the divided d coordinates of each segment. (Step S202). The predetermined value may be 2 or more in order to remove noise, or may be an average value of the parallax frequency in each segment.

  Further, the predetermined number may be set in consideration of road surface dispersion, or some predetermined value may be set in advance. For example, when it is known that the normal height when dispersed in the vertical direction (y direction) of the parallax of the road surface is 20 cm, a value obtained by converting the actual distance 20 cm into the number of pixels may be set as the predetermined value. Thereby, noise can be removed.

  Subsequently, the terminal segment correction unit 14 sets a segment having a d-coordinate count number equal to or greater than a predetermined threshold as segment A, and a segment having a d-coordinate count number less than the predetermined threshold as segment B (step S203).

  Subsequently, the terminal segment correction unit 14 evaluates in order from the segment far from the host vehicle, and determines the segment A when the segment B is changed to the segment A as the terminal segment (step S204).

  Subsequently, the terminal segment correction unit 14 corrects the road surface so that the road surface of the terminal segment becomes low (step S205).

  Subsequently, the terminal segment correction unit 14 corrects the road surface of the segment that is farther from the host vehicle than the terminal segment (step S206).

  Note that the processing of step S201 and step S204 to step S206 is the same as the processing of step S101 and step S104 to step S106 of FIG. 9 according to the first embodiment.

  Next, with reference to FIG. 12, the process of step S202 and step S203 will be described. FIG. 12 is a diagram illustrating processing for searching for a termination segment according to the second embodiment.

  When the V map in a segment has a shape as shown in FIG. 12A, the number of pixels having a value greater than or equal to a predetermined value is counted in the vertical direction for each d coordinate as shown in FIG. 12B. At that time, d coordinates whose number is less than the predetermined number are determined as noise, and the number of d coordinates not determined as noise is counted. Then, a segment in which the number of d coordinates that have not been determined as noise is equal to or greater than a predetermined threshold is determined as segment A (terminal candidate).

  In the example of FIG. 12B, an example is shown in which the number of parallax points at each d coordinate is converted into a histogram for determination, but it is not always necessary to form a histogram. For example, the following processing may be performed.

  The end segment correction unit 14 sequentially determines from the d coordinate of the end point (left end) of the target segment, and increments the d coordinate counter when the count number of each d coordinate (bin) exceeds a predetermined number 524. Then, the next d coordinate determination is performed. Then, when the d-coordinate counter becomes equal to or greater than a predetermined threshold, the target segment is set as segment A (end candidate).

  According to the second embodiment, it is possible to capture a parallax point group of obstacles such as side walls that appear together in the vertical direction (y direction) of the V map. Therefore, the accuracy of searching for the end segment is improved.

[Third Embodiment]
In the third embodiment, an example will be described in which a predetermined area in a luminance image (reference image data) corresponding to each segment of the V map is image-recognized and the end segment is determined based on the recognition result.

  In the first actual form and the second actual form, since the end segment is determined based on the parallax point on the V map, for example, when the parallax point cannot be detected because there is no unevenness or white line on the road surface, There is a possibility of misjudging it.

  According to the third embodiment, it is possible to correctly determine whether a segment is a terminal segment even in the case of a segment where a parallax point cannot be detected.

  Note that the third embodiment is the same as the first embodiment except for a part thereof, and thus description thereof will be omitted as appropriate. Below, the difference with 1st Embodiment is demonstrated about the detail of the process of the vehicle equipment control system 1 which concerns on 3rd Embodiment.

  With reference to FIGS. 13 to 15, the end segment correction processing by the end segment correction unit 14 of the third embodiment will be described.

  FIG. 13 is a flowchart illustrating an example of a terminal segment correction process according to the third embodiment.

  First, the end segment correction unit 14 sets each area in the captured image (step S301). FIG. 14 is a diagram for explaining the correspondence between a captured image and a V map. FIG. 14 shows an example in which a region where a straight road surface can be seen is divided into three regions 591, 592, and 593.

  Subsequently, the end segment correction unit 14 recognizes each region of the captured image (step S302).

  Note that any method may be used as the image recognition method in step S302. For example, a template for an object such as a vehicle, a person, a road surface, or a raindrop may be prepared in advance, and the object may be recognized by template matching.

  Also, for example, “Csurka, G., Bray, C., Dance, C. and Fan, L,“ Visual categorization with bags of keypoints, ”Proc. ECCV Workshop on Statistical Learning in Computer Vision, pp.1-22, The Bag-of-Visual-Words method used for recognition using vectors called visual words that quantize local feature quantities such as SIFT (Scale-Invariant Feature Transform) as described in 2004. May be used. Alternatively, as described in “Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.,“ Imagenet classification with deep convolutional neural networks, ”In Advances in neural information processing systems, pp. 1097-1105, 2012. A method using Deep Learning in which a neural network is multilayered may be used.

  When recognizing an image, a predetermined area of the reference image may be divided into smaller small areas, and the recognition process may be executed sequentially while performing raster scanning. In this case, since the area including noise can be removed rather than subjecting the entire segment to recognition processing, recognition accuracy can be improved.

  Subsequently, the end segment correcting unit 14 determines the end segment based on the result of the image recognition in step S302 (step S303).

  The parallax point of each segment of the V map is due to an object appearing in an area corresponding to each segment in the luminance image. Therefore, if it is determined by image recognition whether or not an obstacle such as a side wall is reflected in each area in the luminance image, it can be determined whether each segment of the V map corresponding to each area is a terminal segment.

  For example, when an obstacle such as a side wall is reflected to the upper end of the highest position area among the areas in the luminance image, the end segment correcting unit 14 determines that the segment including the lower end of the obstacle is the end segment. May be determined.

  FIG. 15 is a diagram for explaining a correspondence relationship between a captured image of an obstacle such as a side wall and a V map.

  For example, when an obstacle is shown (exists) in a part of the region 592 of the captured image and the entire region 593, the parallax extends in the vertical direction in the segment 592a on the V map. A disparity point does not appear in the segment 593a that is distributed and farther away. In this case, it is determined that the segment 592a is a terminal segment.

  According to the third embodiment, the accuracy of searching for a terminal segment is improved even in bad weather, for example, in a situation where noise is generated in which parallax is dispersed due to rain reflection.

  Note that the above-described image recognition may be performed using a parallax image instead of a captured image (luminance image). Alternatively, both a captured image and a parallax image may be used. In this case, for example, the recognition result may be weighted to recognize the obstacle, or it may be determined that the obstacle exists when the average value of the recognition result scores is larger than a predetermined value.

  Further, the end segment may be determined based on a value obtained by weighted addition of the end segment determination result of the third embodiment and the end segment determination result of the first embodiment and the second embodiment.

[Fourth Embodiment]
In the fourth embodiment, when estimating the road surface of each segment, based on the data of the end segment determined in the previously captured image (past frame), the termination determined in the captured image captured this time An example of correcting the road surface of the segment will be described.

  Note that the fourth embodiment is the same as the first embodiment except for a part thereof, and thus description thereof will be omitted as appropriate. Below, the difference with 1st Embodiment is demonstrated about the detail of the process of the vehicle equipment control system 1 which concerns on 4th Embodiment.

  FIG. 16 is a flowchart illustrating an example of a road surface estimation process according to the fourth embodiment.

  The road surface estimation unit 13 divides the V map into a plurality of segments according to the value of the parallax d that is the horizontal axis of the V map (step S21).

  Subsequently, the road surface estimation unit 13 sets the segment with the shortest distance from the host vehicle as the processing target segment in the V map (step S22).

  Subsequently, the road surface estimation unit 13 extracts sample points from the pixels included in the processing target segment (step S23).

  FIG. 17 is a diagram for explaining an example of sample point extraction processing according to the fourth embodiment. For example, in the second segment, the road surface estimation unit 13 determines the end point position (the road surface 561 of the previous segment is the second segment) of the previous segment (the segment adjacent to the second segment that has already detected the road surface). The reference point 562 is a position in contact with the segment. Then, the road surface estimation unit 13 sets a predetermined range corresponding to the reference point 562 as the sample point search region 563.

  For example, as shown in FIG. 17A, the road surface estimation unit 13 searches between two straight lines 564 and 565 starting from a reference point and extending between two straight lines 564 and 565 at a predetermined angle to the adjacent segment. The region 563 may be used. In this case, the road surface estimation unit 13 may determine the two straight lines 564 and 565 corresponding to the angle at which the road surface can tilt. Or the road surface estimation part 13 is good also considering the search area | region 563 as a rectangular area | region which has the height according to the y coordinate of the reference point 562 like FIG.17 (B), for example.

  Subsequently, the road surface estimation unit 13 excludes inappropriate points from the extracted sample points (step S24).

  Subsequently, the road surface estimation unit 13 detects the shape (position, height) of the road surface in the processing target segment based on the sample points not excluded in step S24 (step S25).

  Subsequently, when the detected (estimated) road surface is inappropriate, the road surface estimation unit 13 supplements the road surface in the segment to be processed (step S26).

  Subsequently, the end segment correction unit 14 determines whether or not the segment to be processed is the end segment (step S27). In step S27, for example, the terminal segment correction unit 14 acquires the vehicle speed of the host vehicle 100 using the data I / F 603, calculates the movement amount of the obstacle according to the vehicle speed, and calculates the calculated movement amount of the obstacle. Depending on, the end segment in the current frame may be determined.

  When the segment to be processed is the end segment (YES in step S27), the end segment correcting unit 14 corrects the road surface of the end segment based on the history road surface of the end segment (step S28), Proceed to processing. In step S28, the end segment correction unit 14 may use the end segment road surface calculated in the previous frame as the end segment road surface calculated in the current frame. Alternatively, the end segment correction unit 14 acquires the vehicle speed of the host vehicle 100 using, for example, the data I / F 603, calculates the amount of movement of the road surface according to the vehicle speed, and ends according to the calculated amount of movement of the road surface. The road surface of the segment may be corrected.

  Note that the terminal segment correction unit 14 corrects the road surface of the terminal segment by the processing described in the above first to third embodiments and corrects it when the history road surface is not stored due to startup or the like. The completed road surface may be stored as a history road surface.

  When the segment to be processed is not the end segment (NO in step S27), the road surface estimation unit 13 determines whether or not there is a next segment that is farther from the vehicle than the segment to be processed in the V map. (Step S29).

  If there is a next segment (YES in step S29), the next segment is selected as a processing target (step S30), and the process proceeds to step S23.

  When there is no next segment (NO in step S29), the road surface estimation unit 13 corrects (smoothing processing) each road surface estimated in each segment so that each road surface is continuous (step S31).

  In step S23, the road surface estimation unit 13 determines a sample point search region in the segment to be processed in order from the segment closer to the host vehicle, with reference to the road surface of the previous segment, as shown in FIG. Shall. In this way, when the road surface estimated in the previous segment affects the road surface estimated in the next segment, the road surface estimation process and the terminal segment correction process are executed sequentially, and the next segment is processed. The road surface of the previous segment must be corrected.

  According to the fourth embodiment, since the road surface estimation process and the terminal segment correction process are sequentially performed on each segment, for example, the sample point search region in the segment to be processed is set based on the road surface of the previous segment. Can be determined.

[Fifth Embodiment]
In the fifth embodiment, an example will be described in which when an uphill is erroneously determined to be an obstacle such as a side wall, the terminal segment correction process is not performed.

  Note that the fifth embodiment is the same as the first embodiment except for a part thereof, and thus description thereof will be omitted as appropriate. Below, the difference with 1st Embodiment is demonstrated about the detail of the process of the vehicle equipment control system 1 which concerns on 5th Embodiment.

  In the first embodiment and the second embodiment, when the front side of the host vehicle is an uphill, the road surface height estimated in a certain segment on the V map penetrates upward, and the disparity point in a farther segment There may be no.

  FIG. 18 is a diagram for explaining an end segment correction process on an uphill. As shown in FIG. 18 (A), in an uphill scene in which the road surface is divided into the first to seventh segments in order of increasing distance from the host vehicle on the V map, and the road surface height increases in the sixth segment. Suppose there is. In this case, since there are few parallax points in the seventh segment, in the processing described in the first embodiment and the second embodiment, the sixth segment is determined to be the terminal segment, and the road surface 541 in the terminal segment has a height. There is a problem that the road surface 542 is corrected.

  Therefore, the terminal segment correction unit 14 according to the fifth embodiment performs the following processing.

  FIG. 19 is a flowchart illustrating an example of a terminal segment correction process according to the fifth embodiment.

  The terminal segment correction unit 14 divides the V map into a plurality of segments (step S501).

  The end segment correction unit 14 determines whether or not there is a segment that is an uphill in each divided segment (step S502).

  As shown in FIG. 18B, the terminal segment correction unit 14 determines that this segment is an uphill segment when a predetermined number (eg, 1) or more of parallax points are included in a predetermined range 551 of a certain segment. May be. The predetermined range 551 may be, for example, a range having a position higher than the reference with the end (left end) of the road surface of the previous segment as a reference.

  If there is an uphill segment (YES in step S502), the process ends.

  When there is no segment that is uphill (NO in step S502), the terminal segment correction unit 14 searches for a terminal segment (step S503). Note that the processing for searching for the end segment may be the same as that in the first embodiment or the second embodiment described above, for example.

  Subsequently, the terminal segment correction unit 14 corrects the road surface so that the road surface of the terminal segment becomes low (step S504).

  Subsequently, the end segment correction unit 14 corrects the road surface of the segment that is farther from the host vehicle than the end segment (step S505).

  According to the fifth embodiment, when the road surface is an uphill, control is performed so that the terminal segment correction process is not performed, so that the road surface can be estimated more appropriately.

<Summary>
According to each embodiment described above, among the segments in the V map, a segment in front of a segment whose value corresponding to the number of parallax points is less than a predetermined threshold is determined as the end segment, and the road surface height in the end segment is determined. Modify the height to be lower. Thereby, even when the parallax of an obstacle such as a side wall or a guard rail is detected, it is possible to improve the accuracy of detecting the road surface.

  Since the distance value (distance value) and the parallax value can be handled equivalently, in the present embodiment, the parallax image is used as an example of the distance image, but the present invention is not limited to this. For example, a distance image may be generated by integrating distance information generated using a detection device such as a millimeter wave radar or a laser radar with a parallax image generated using a stereo camera. Further, the detection accuracy may be further enhanced by using a stereo camera and a detection device such as a millimeter wave radar or a laser radar in combination with the detection result of the object by the stereo camera described above.

  The system configuration in the above-described embodiment is an example, and it goes without saying that there are various system configuration examples depending on applications and purposes. Moreover, it is also possible to combine a part or all of each embodiment mentioned above.

  For example, in the road surface estimation processing, processing such as removal point removal, road surface supplementation, smoothing processing, and the like are not essential, and therefore these processing may not be performed.

  The functional units of the processing hardware unit 510, the image analysis unit 600, and the vehicle travel control unit 104 may be realized by hardware, or the CPU executes a program stored in a storage device. It is good also as a structure implement | achieved by. The program may be recorded and distributed on a computer-readable recording medium by a file in an installable or executable format. Examples of the recording medium include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, and the like. Further, a recording medium such as a CD-ROM in which each program is stored, and the HD 504 in which these programs are stored can be provided as a program product (Japan or overseas).

1 In-vehicle device control system (an example of “device control system”)
100 own vehicle 101 imaging unit 103 display monitor 106 vehicle travel control unit (an example of “control unit”)
11 Parallax image generation unit (an example of a “distance image generation unit”)
12 V map generator (an example of “generator”)
13 Road surface estimation part (an example of "estimation part")
14 End segment correction part (an example of "correction part")
15 Road surface height table calculation unit 16 Clustering unit (an example of “object detection unit”)
17 Rejecting unit 18 Tracking unit 19 Control unit 2 Imaging device 510a, 510b Imaging unit 510 Processing hardware unit 600 Image analysis unit (an example of “image processing device”)

Japanese Patent Laid-Open No. 2015-075800

Claims (13)

A frequency distribution of distance values with respect to a vertical direction of the distance image is shown from a distance image having a distance value according to a distance of a road surface and an obstacle that blocks the road surface in a plurality of captured images respectively captured by a plurality of imaging units. A generator for generating vertical distribution data;
An estimation unit that estimates the height of the road surface according to the distance based on the vertical distribution data;
A correction unit that corrects the height of the road surface at the distance of the obstacle estimated by the estimation unit to be low;
An image processing apparatus comprising: The correction unit divides the frequency distribution of distance values in the vertical direction of the distance image into a plurality of segments corresponding to the distance values, and the value corresponding to the divided distribution is less than a predetermined threshold value. Correcting the height of the road surface in the second segment having the distance value smaller than that of the first segment to be lower than the height estimated by the estimation unit;
The image processing apparatus according to claim 1. In the second segment, the number of distance values in the vertical direction in which the frequency value of each distance value included in each segment is greater than or equal to a predetermined value is equal to or greater than a predetermined threshold value. Among the segments that are greater than or equal to the threshold value, the segment that has the largest distance value included in the segment,
The image processing apparatus according to claim 2. The second segment includes the distance included in the segment among the segments in which the number of distance points in the vertical direction in which the frequency value of the distance value included in each segment is equal to or greater than a predetermined value is equal to or greater than a predetermined threshold. The segment with the largest value,
The image processing apparatus according to claim 2. The correction unit recognizes the obstacle in the captured image, and corrects the height of the road surface at the distance of the obstacle estimated by the estimation unit to be low.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The correction unit corrects the height of the road surface at the distance of the obstacle based on the height of the road surface estimated by the estimation unit according to the plurality of previously captured images.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The correction unit determines whether or not the road surface is an uphill, and if it is an uphill, does not correct the height of the road surface at the distance of the obstacle estimated by the estimation unit,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The correction unit corrects the height of the road surface at the obstacle distance estimated by the estimation unit to a road surface height at a distance smaller than the distance of the obstacle estimated by the estimation unit. To
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The correction unit discards the road height data at a distance greater than the distance of the obstacle,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A plurality of imaging units;
A distance image generating unit that generates a distance image having a distance value according to a parallax of an obstacle that blocks the road surface and the road surface in the plurality of captured images from a plurality of captured images respectively captured by the plurality of imaging units;
A generating unit that generates vertical distribution data indicating a frequency distribution of distance values with respect to a vertical direction of the distance image;
An estimation unit that estimates the height of the road surface according to the distance value based on the vertical direction distribution data;
A correction unit for correcting the height of the road surface at the obstacle distance value estimated by the estimation unit to be low;
An imaging apparatus comprising: A plurality of imaging units mounted on a moving body and imaging the front of the moving body;
A distance having a distance value corresponding to a parallax of a moving surface, an obstacle that blocks the moving surface, and an object on the moving surface in the plurality of captured images from a plurality of captured images respectively captured by the plurality of imaging units. A distance image generation unit for generating an image;
A generating unit that generates vertical distribution data indicating a frequency distribution of distance values with respect to a vertical direction of the distance image;
An estimation unit that estimates the height of the moving surface according to the distance value based on the vertical direction distribution data;
A correction unit that corrects the height of the moving surface at the obstacle distance value estimated by the estimation unit to be low;
An object detection unit for detecting the object based on the moving surface corrected by the correction unit and the distance image;
A control unit that controls the moving body based on the data of the object detected by the object detection unit;
A mobile device control system comprising: Computer
A frequency distribution of distance values with respect to a vertical direction of the distance image is shown from a distance image having a distance value according to a distance of a road surface and an obstacle that blocks the road surface in a plurality of captured images respectively captured by a plurality of imaging units. Generating vertical distribution data;
Estimating the height of the road surface according to the distance based on the vertical distribution data;
Correcting the height of the road surface at the estimated distance of the obstacle to be low;
An image processing method for executing. On the computer,
A frequency distribution of distance values with respect to a vertical direction of the distance image is shown from a distance image having a distance value according to a distance of a road surface and an obstacle that blocks the road surface in a plurality of captured images respectively captured by a plurality of imaging units. Generating vertical distribution data;
Estimating the height of the road surface according to the distance based on the vertical distribution data;
Correcting the height of the road surface at the estimated distance of the obstacle to be low;
A program that executes

Images