JP6763198B2 - Image processing equipment, imaging equipment, mobile device control systems, image processing methods, and programs - Google Patents

Description

The present invention relates to an image processing device, an imaging device, a mobile device control system, an image processing method, and a program.

In terms of automobile safety, the body structure of automobiles has been developed from the viewpoint of how to protect pedestrians and occupants when they collide with pedestrians or automobiles. However, in recent years, with the development of information processing technology and image processing technology, technology for detecting people, automobiles, etc. at high speed has been developed. Automobiles that apply these technologies to automatically brake before a collision and prevent a collision have already been released.

In order to automatically apply the brakes, it is necessary to measure the distance to an object such as a person or another vehicle, and for that purpose, measurement using an image of a stereo camera has been put into practical use.

In the measurement using the image of this stereo camera, a technique for recognizing an object such as a person or another vehicle by performing the following image processing is known (see, for example, Patent Document 1).

First, from a plurality of images captured by a stereo camera, a V-Disparity image in which the coordinates in the vertical direction of the image are one axis, the disparity of the image is the other axis, and the value of the frequency of parallax is a pixel value. To generate. Next, the road surface is detected from the generated V-Disparity image. After removing noise using the detected road surface, a U-Disparity image is generated in which the horizontal coordinates of the image are the vertical axis, the parallax of the image is the horizontal axis, and the parallax frequency is the pixel value. Then, based on the generated U-Disparity image, an object such as a person or another vehicle is recognized.

However, the prior art has a problem that when detecting a road surface, it may be determined that there is an uphill in front of the own vehicle when an object is placed on the road surface.

Therefore, it is an object of the present invention to provide a technique for improving the accuracy of detecting the road surface.

In the image processing apparatus, the frequency of the distance value with respect to the vertical direction from the distance image having the distance value corresponding to the distance between the road surface and the object on the road surface in the plurality of captured images taken by the plurality of imaging units. The height of the first point at which the frequency of the distance value is equal to or higher than a predetermined threshold based on the vertical distribution data and the generation unit that generates the vertical distribution data showing the distribution of the above, and the height of the first point. When the difference from the height of the second point having a small distance value, the difference from the distance value of the first point is within a predetermined value, and the frequency of the distance value is equal to or greater than a predetermined threshold value is greater than or equal to the predetermined value. It is provided with an estimation unit that estimates the road surface based on the points other than the first point when it is determined to be present.

According to the disclosed technology, it is possible to improve the accuracy of detecting the road surface.

It is a schematic diagram which shows the structure of the in-vehicle device control system which concerns on embodiment. It is a schematic diagram which shows the structure of the image pickup unit and the image analysis unit. It is a functional block diagram of a mobile device control system. It is a figure for demonstrating the parallax image data, and the V map generated from the parallax image data. It is a figure which shows the image example of the photographed image as a reference image imaged by one image pickup part, and the V map corresponding to the photographed image. It is a figure which shows the image example of the photographed image as a reference image imaged by one image pickup unit, and the V map of a predetermined area corresponding to the photographed image. It is a functional block diagram which shows an example of the road surface estimation part 13. It is a flowchart which shows an example of the sample point removal process. It is a figure explaining an example of the sample point removal processing. It is a figure explaining another example of the sample point removal process. It is a figure which shows an example of the flowchart of the sampling point extraction process which concerns on 2nd Embodiment. It is a figure explaining the process of superimposing the point in the present V map, and the point in the previous V map. It is a figure which shows an example of the flowchart of the road surface shape detection processing which concerns on 3rd Embodiment. It is a figure explaining the process of determining whether or not there is a discrepancy point by an object on a road surface in a V map.

Hereinafter, a mobile device control system having an image processing device according to the embodiment will be described.

[First Embodiment]
<Configuration of in-vehicle device control system>
FIG. 1 is a diagram showing a configuration of an in-vehicle device control system as a mobile device control system according to an embodiment of the present invention.

The in-vehicle device control system 1 is mounted on the own vehicle 100 such as a moving vehicle, and includes an image pickup unit 500, an image analysis unit 600, a display monitor 103, and a vehicle travel control unit 104. Then, the relative height information (relative) of the road surface (moving surface) in front of the own vehicle is obtained from the captured image data of the front region (imaging region) in the traveling direction of the own vehicle in which the front of the moving body is imaged by the imaging unit 500. Information indicating the inclination status) is detected, the three-dimensional shape of the traveling road surface in front of the own vehicle is detected from the detection result, and the moving body and various in-vehicle devices are controlled by using the detection result. The control of the moving body includes, for example, notification of a warning, control of the handle of the own vehicle 100 (self-moving body), or braking of the own vehicle 100 (self-moving body).

The image pickup unit 500 is installed near, for example, a room mirror (not shown) of the windshield 105 of the own vehicle 100. Various data such as captured image data obtained by imaging the imaging unit 500 are input to the image analysis unit 600 as an image processing means.

The image analysis unit 600 analyzes the data transmitted from the imaging unit 500, and on the traveling road surface in front of the own vehicle with respect to the road surface portion (the road surface portion located directly below the own vehicle) on which the own vehicle 100 is traveling. The relative height (position information) at each point is detected, and the three-dimensional shape of the traveling road surface in front of the own vehicle is grasped. It also recognizes objects to be recognized such as other vehicles, pedestrians, and various obstacles in front of the own vehicle.

The analysis result of the image analysis unit 600 is sent to the display monitor 103 and the vehicle travel control unit 104. The display monitor 103 displays the captured image data and the analysis result obtained by the imaging unit 500. The vehicle travel control unit 104 notifies, for example, a warning to the driver of the own vehicle 100 based on the recognition result of the recognition target object such as another vehicle, a pedestrian, and various obstacles in front of the own vehicle by the image analysis unit 600. It also performs driving support control such as controlling the steering wheel and brakes of the own vehicle.

<Structure of image pickup unit 500 and image analysis unit 600>
FIG. 2 is a diagram showing the configurations of the image pickup unit 500 and the image analysis unit 600.

The image pickup unit 500 is composed of a stereo camera including two image pickup units 510a and 510b as image pickup means, and the two image pickup units 510a and 510b are the same. Each of the imaging units 510a and 510b is an analog output from the sensor substrates 514a and 514b including the imaging lenses 511a and 511b and the image sensors 513a and 513b in which the light receiving elements are arranged in two dimensions, and the sensor substrates 514a and 514b, respectively. It is composed of signal processing units 515a and 515b that generate and output captured image data obtained by converting an electric signal (an electric signal corresponding to the amount of light received by each light receiving element on the image sensors 513a and 513b) into a digital electric signal. ing. Luminance image data and parallax image data are output from the imaging unit 500.

Further, the imaging unit 500 includes a processing hardware unit 510 including an FPGA (Field-Programmable Gate Array) or the like. In order to obtain a parallax image from the brightness image data output from the imaging units 510a and 510b, the processing hardware unit 510 obtains the parallax value of the corresponding image portion between the captured images captured by the imaging units 510a and 510b, respectively. A parallax calculation unit 511 is provided as a parallax image information generation means for calculation.

The parallax value referred to here is a comparison image with respect to an image portion on the reference image corresponding to the same point in the imaging region, with one of the captured images captured by the imaging units 510a and 510b as a reference image and the other as a comparison image. The amount of misalignment of the upper image portion is calculated as a parallax value of the image portion. By using the principle of triangulation, it is possible to calculate the distance from this parallax value to the same point in the imaging region corresponding to the image portion.

The image analysis unit 600 includes a storage means 601 composed of an image processing board or the like and composed of a RAM, a ROM, or the like for storing brightness image data and parallax image data output from the image pickup unit 500, and recognition processing of an identification target. It includes a CPU (Central Processing Unit) 602 that executes a computer program for performing disparity calculation control and the like, a data I / F (interface) 603, and a serial I / F 604.

The FPGA that constitutes the processing hardware unit 510 performs processing that requires real-time performance for image data, such as gamma correction, distortion correction (parallelization of left and right captured images), and parallax calculation by block matching, to perform a parallax image. Information is generated and written to the RAM of the image analysis unit 600. The CPU 602 of the image analysis unit 600 is responsible for controlling the image sensor controllers of the imaging units 510a and 510b and overall control of the image processing substrate, as well as detecting the three-dimensional shape of the road surface, guard rails, and various other objects (identification targets). A program that executes detection processing of the object) is loaded from the ROM, various processes are executed by inputting the brightness image data and the parallax image data stored in the RAM, and the processing results are used as data I / F603 or serial I /. Output from F604 to the outside. When executing such processing, the data I / F603 is used to input vehicle operation information such as vehicle speed, acceleration (mainly acceleration generated in the front-rear direction of the vehicle), steering angle, yaw rate, etc. of the own vehicle 100, and various types are input. It can also be used as a processing parameter. The data output to the outside is used as input data for controlling various devices of the own vehicle 100 (brake control, vehicle speed control, warning control, etc.).

The image pickup unit 500 and the image analysis unit 600 may be configured as an image pickup device 2 which is an integrated device.

FIG. 3 is a functional block diagram of the in-vehicle device control system 1 realized by the processing hardware unit 510, the image analysis unit 600, and the vehicle travel control unit 104 in FIG. The functional unit realized by the image analysis unit 600 is realized by a process of executing one or more programs installed in the image analysis unit 600 by the CPU 602 of the image analysis unit 600.

Hereinafter, the processing in this embodiment will be described.

<Parallax image generation processing>
The parallax image generation unit 11 performs a parallax image generation process for generating parallax image data (parallax image information). The parallax image generation unit 11 is composed of, for example, a parallax calculation unit 511 (FIG. 2).

In the parallax image generation process, first, the brightness image data of one of the two imaging units 510a and 510b is used as the reference image data, and the brightness image data of the other imaging unit 510b is used as the comparison image data. The difference between the two is calculated using the difference, and the difference image data is generated and output. This parallax image data shows a parallax image in which the pixel value corresponding to the parallax value d calculated for each image portion on the reference image data is represented as the pixel value of each image portion.

Specifically, the parallax image generation unit 11 defines a block composed of a plurality of pixels (for example, 16 pixels × 1 pixel) centered on one pixel of interest for a row of reference image data. On the other hand, in the same row of the comparative image data, a block having the same size as the defined reference image data block is shifted one pixel at a time in the horizontal line direction (x direction) to show the characteristics of the pixel value of the block defined in the reference image data. Correlation values indicating the correlation between the feature amount and the feature amount indicating the characteristics of the pixel value of each block in the comparative image data are calculated. Then, based on the calculated correlation value, a matching process is performed to select the block of the comparison image data that has the most correlation with the block of the reference image data among the blocks of the comparison image data. After that, the amount of positional deviation between the pixel of interest of the block of reference image data and the corresponding pixel of the block of comparative image data selected by the matching process is calculated as the parallax value d. Parallax image data can be obtained by performing the process of calculating the parallax value d for the entire area of the reference image data or a specific area.

As the feature amount of the block used for the matching process, for example, the value (brightness value) of each pixel in the block can be used, and as the correlation value, for example, the value (brightness value) of each pixel in the block of the reference image data. The sum of the absolute values of the differences between the value) and the value (brightness value) of each pixel in the block of comparative image data corresponding to each of these pixels can be used. In this case, it can be said that the block with the smallest sum is the most correlated.

When the matching process in the parallax image generation unit 11 is realized by hardware processing, for example, SSD (Sum of Squared Difference), ZSD (Zero-mean Sum of Squared Difference), SAD (Sum of Absolute Difference), ZSAD ( Methods such as Zero-mean Sum of Absolute Difference) and NCC (Normalized cross correlation) can be used. Since only the parallax value for each pixel can be calculated in the matching process, it is necessary to use the estimated value when the parallax value at the sub-pixel level of less than one pixel is required. As the estimation method, for example, an isometric straight line method, a quadratic curve method, or the like can be used.

<V map generation process>
The V-map generation unit 12 executes a V-map generation process for generating a V-map (V-Disparity Map, an example of "vertical distribution data") based on the parallax pixel data. Each parallax pixel data included in the parallax image data is indicated by a set (x, y, d) of the x-direction position, the y-direction position, and the parallax value d. This is converted into three-dimensional coordinate information (d, y, f) in which d is set on the X-axis, y is set on the Y-axis, and frequency f is set on the Z-axis, or the three-dimensional coordinate information (d, y, f). Three-dimensional coordinate information (d, y, f) limited to information exceeding a predetermined frequency threshold value is generated as parallax histogram information. The disparity histogram information of the present embodiment consists of three-dimensional coordinate information (d, y, f), and the distribution of the three-dimensional histogram information in the two-dimensional coordinate system of XY is called a V-map.

Specifically, the V-map generation unit 12 calculates the parallax value frequency distribution for each row region obtained by dividing the parallax image into a plurality of parts in the vertical direction. The information indicating the parallax value frequency distribution is the parallax histogram information.

FIG. 4 is a diagram for explaining the parallax image data and the V map generated from the parallax image data. Here, FIG. 4A is a diagram showing an example of the parallax value distribution of the parallax image, and FIG. 5B is a diagram showing a V map showing the parallax value frequency distribution for each row of the parallax image of FIG. 4A.

When the parallax image data having the parallax value distribution as shown in FIG. 4A is input, the V map generation unit 12 calculates the parallax value frequency distribution which is the distribution of the number of data of each parallax value for each row. This is output as parallax histogram information. The information of the parallax value frequency distribution of each row obtained in this way is a two-dimensional Cartesian coordinate system in which the y-direction position (vertical position of the captured image) on the parallax image is taken on the Y-axis and the parallax value d is taken on the X-axis. By representing above, a V-map as shown in FIG. 4B can be obtained. This V-map can also be expressed as an image in which pixels having pixel values corresponding to the frequency f are distributed on the two-dimensional Cartesian coordinate system.

FIG. 5 is a diagram showing an image example of a captured image as a reference image captured by one imaging unit and a V map corresponding to the captured image. Here, FIG. 5A is a photographed image, and FIG. 5B is a V map. That is, the V map shown in FIG. 5B is generated from the captured image as shown in FIG. 5A. In the V map, parallax is not detected in the region below the road surface, so that the parallax is not counted in the shaded area A.

In the image example shown in FIG. 5A, the road surface 401 on which the own vehicle is traveling, the preceding vehicle 402 existing in front of the own vehicle, and the utility pole 403 existing outside the road are projected. Further, in the V map shown in FIG. 5B, there are a road surface 501, a preceding vehicle 502, and a utility pole 503 corresponding to an image example.

This image example is a virtual image obtained by extending the road surface in front of the own vehicle to a relatively flat road surface, that is, the surface in which the front road surface of the own vehicle is parallel to the road surface portion directly below the own vehicle to the front of the own vehicle. This is the case when it matches the reference road surface (virtual reference moving surface). In this case, in the lower part of the V map corresponding to the lower part of the image, the high frequency points are distributed in a substantially straight line with an inclination so that the parallax value d becomes smaller toward the upper part of the image. Pixels showing such a distribution are pixels that are present at approximately the same distance in each row on the parallax image, have the highest occupancy rate, and project an identification object whose distance is continuously increased toward the upper part of the image. It can be said that.

Since the image pickup unit 510a captures the area in front of the own vehicle, the parallax value d of the road surface becomes smaller toward the upper side of the image as shown in FIG. 5A. Further, in the same line (horizontal line), the pixels projecting the road surface have substantially the same parallax value d. Therefore, the high-frequency points distributed in a substantially linear shape on the V-map correspond to the characteristics of the pixels that project the road surface (moving surface). Therefore, it can be estimated that the pixels of the points distributed on or near the approximate straight line obtained by linearly approximating the high frequency points on the V map are the pixels projecting the road surface with high accuracy. Further, the distance to the road surface portion projected on each pixel can be obtained with high accuracy from the parallax value d of the corresponding point on the approximate straight line.

FIG. 6 is a diagram showing an image example of a captured image as a reference image captured by one imaging unit and a V-map of a predetermined region corresponding to the captured image.

The V-map generation unit 12 may generate a V-map using all the pixels in the parallax image, or may generate a V-map in a predetermined region (for example, FIG. 6A is an example of a captured image on which the parallax image is based). The V-map may be generated using only the pixels in the area where the road surface can be captured). For example, since the road surface becomes narrower toward the vanishing point as the distance increases, a region corresponding to the width of the road surface may be set as shown in FIG. 6A. This makes it possible to prevent noise from an object (for example, a utility pole 403) located in a region other than the region where the road surface can be captured from being mixed in the V map.

<Road surface estimation>
The road surface estimation unit 13 estimates (detects) the road surface based on the V map generated by the V map generation unit 12.

An example of the functional configuration of the road surface estimation unit 13 will be described with reference to FIG. 7. FIG. 7 is a functional block diagram showing an example of the road surface estimation unit 13.

The road surface estimation unit 13 includes a sample point extraction unit 131, a sample point removal unit 132, a road surface shape detection unit 133, a road surface supplement unit 134, and a smoothing processing unit 135.

Hereinafter, the processing of each functional unit of the road surface estimation unit 13 will be described.

《Sample point extraction process》
The sample point extraction unit 131 extracts sample points used for estimating the road surface from the V map generated by the V map generation unit 12.

In the following, an example will be described in which the V map is divided into a plurality of segments according to the parallax value (distance value from the own vehicle), and the sample point extraction process and the road surface shape detection process described later are performed. Sample point extraction processing, road surface shape detection processing, and the like may be performed without dividing the V-map.

The sample point extraction unit 131 divides the V map into a plurality of segments according to the parallax value on the horizontal axis of the V map.

Then, the sample point extraction unit 131 extracts a plurality of sample points in each segment. The sample point extraction unit 131 may, for example, extract one or more sample points for each coordinate position of each parallax value d. Alternatively, at the coordinate position of each parallax value d, the mode with the highest frequency may be extracted as a sample point. Alternatively, the most frequent mode point in a plurality of d coordinates including each parallax value d (for example, the coordinate position of each parallax value d and at least one coordinate position of at least one of the left and right of each parallax value d). May be extracted as a sample point. Alternatively, even if the pixel is not a pixel (parallax point) whose frequency value is equal to or higher than a predetermined value, for example, if points having a high parallax frequency are concentrated in a predetermined range around the pixel, it is accidental. Since there is a possibility that the parallax point is missing, the pixel may be used as a sample point.

《Sample point removal process》
The sample point removing unit 132 removes sample points based on the parallax of an object other than the road surface among the plurality of sample points extracted by the sample point extraction unit 131.

FIG. 8 is a flowchart showing an example of the sample point removal process. The sample point removing unit 132 performs the following processing for each sample point in each segment.

The sample point removing unit 132 is larger than the d-coordinate value of the sample point to be processed (close to the own vehicle) and close to the sample point to be processed (the difference between the d-coordinate values is predetermined). Extract sample points (within the value) (step S101).

Subsequently, the sample point removing unit 132 determines whether or not the difference between the height of the sample point to be processed (value of the y coordinate) and the height of the adjacent sample point is equal to or greater than a predetermined value (). Step S102).

When the height difference is not equal to or greater than a predetermined value (NO in step S102), the sample point removing unit 132 ends the process.

When the difference in height is equal to or greater than a predetermined value (YES in step S102), the sample point removing unit 132 has a value d-coordinated with respect to the sample point to be processed and the sample point to be processed in the segment to be processed. Excludes sample points with a small value (step S103), and ends the process. As a result, noise due to parallax of the object is removed, and the road surface can be estimated accurately.

FIG. 9 is a diagram illustrating an example of the sample point removal process. In FIG. 9, it is assumed that the white circle is the parallax point and the black circle is the sample point. In FIG. 9, in the segment to be processed, there is a difference 703 between the height of the sample point 701 to be processed and the height of the sample point 702 that is closer to the vehicle than the sample point 701 and is closer to the sample point 701. It is assumed that the value is equal to or higher than a predetermined value. In this case, the sample point 701 and the sample point 704 that is farther from the own vehicle than the sample point 701 in the segment to be processed are excluded.

Here, the predetermined value regarding the height difference 703 may be, for example, a value obtained by multiplying the segment length (the difference between the value of the d-coordinate at the left end of the segment and the value of the d-coordinate at the right end) by a predetermined ratio. .. Further, these threshold values may be set individually for each segment, or the same value may be set for all the segments.

When the d coordinate of the sample point to be processed is the right end of the segment, the height difference 703 from the sample point corresponding to the end point of the segment to the right, which is close to the own vehicle, may be calculated.

Note that this sampling point removal process may be performed only on a predetermined segment. For example, when focusing only on the pulling up of a distant road surface (uphill), it may be carried out only for the segment in which the distance from the own vehicle in the V map is equal to or more than a predetermined value.

In the above, the removal process was performed on each sample point, but the same process may be performed on each discriminant point in each segment. In this case, after performing the removal process, the sample point extraction unit 131 searches for the sample point with the noise removed.

The sample point removing unit 132 may remove points that are not suitable for linear approximation from each sample point. For example, an approximate straight line of each sample point may be calculated, and sample points whose Euclidean distance is separated by a predetermined threshold value or more from the calculated approximate straight line may be removed.

<Modification example>
In the process of step S103, instead of excluding all the sample points whose d-coordinate value is smaller than the sample points of the process target in the segment to be processed, the following process may be performed.

FIG. 10 is a diagram illustrating another example of the sampling point removal process. Instead of step S103, the sample point removing unit 132 calculates the height difference from the adjacent sample points for each sample point whose d coordinate value is smaller than that of the sample points to be processed in the segment to be processed. Then, only the sample points whose calculated difference is equal to or larger than the predetermined value are removed. As shown in FIG. 10, if the height difference 706 between each sample point 705 whose d-coordinate value is smaller than the sample point 701 to be processed and the adjacent sample point 702 is not equal to or more than a predetermined value, the sample point 705 is set. Not removed. As a result, when a certain sample point is accidentally generated as noise, the sample point on the road surface farther from the own vehicle than the sample point can be prevented from being removed.

As described above, according to the sample point removal process of the first embodiment, when the difference in y-coordinates between adjacent sample points is equal to or greater than a predetermined value, the sample points included in the predetermined range are removed as noise. As a result, when the parallax due to the object is selected as the sample point, the problem of removing the sample point and estimating the height of the road surface to be high can be avoided.

<< Road surface shape detection processing >>
The road surface shape detection unit 133 extracts the shape of the road surface from each segment of the V map generated by the V map generation unit 12 based on the sample points extracted by the sample point extraction unit 131 and not removed by the sample point removal unit 132. (Position, height) is detected.

The road surface shape detection unit 133 calculates an approximate straight line from the sample points of each segment by, for example, the least squares method, and detects (estimates) the calculated approximate straight line as the road surface.

《Road surface supplement processing》
The road surface supplementing unit 134 determines whether or not the road surface detected (estimated) by the road surface shape detecting unit 133 is inappropriate, and if it is determined to be inappropriate, the road surface supplementing unit 134 supplements the road surface.

The road surface supplementing unit 134 determines whether or not an inappropriate road surface that cannot be photographed by a stereo camera is estimated due to noise. Then, when the road surface supplementing unit 134 determines that an inappropriate road surface has been estimated, it supplements (interpolates) the inappropriate road surface based on the data of the default road surface or the road surface estimated in the previous (past) frame. To do.

When the road surface supplementing unit 134 detects data on a steeply rising road surface where, for example, a part of the slope is equal to or higher than a predetermined value in the V map, the slope of the road surface increases as the distance from the own vehicle increases. Judge that the road surface is a steep downhill. Then, the road surface supplementing unit 134 removes the data of the road surface and supplements with the data of the default road surface or the like instead.

《Smoothing process》
The smoothing processing unit 135 modifies each road surface estimated in each segment so that the road surfaces are continuous. The smoothing processing unit 135 determines the inclination and intercept of each road surface so that the end point (end point) of one road surface and the start point (end point) of the other road surface coincide with each of the estimated road surfaces in the two adjacent segments. To change.

<Road height table calculation process>
The road surface height table calculation unit 15 calculates the road surface height (the height relative to the road surface portion directly below the own vehicle) based on the road surface in each segment corrected by the smoothing processing unit 135 and creates a table. Performs road surface height table calculation processing.

The road surface height table calculation unit 15 calculates the distance from the road surface information in each segment to each road surface portion projected in each row region (each position in the vertical direction of the image) on the captured image. In addition, each surface portion in the own vehicle traveling direction of the virtual plane extending forward in the own vehicle traveling direction so as to be parallel to the surface of the road surface portion located directly under the own vehicle is projected on each row area in the captured image. It is decided in advance, and this virtual plane (reference road surface) is represented by a straight line (reference straight line) on the V map. The road surface height table calculation unit 15 can obtain the height of each road surface portion in front of the own vehicle by comparing the road surface in each segment with the reference straight line. Simply, the height of the road surface portion existing in front of the own vehicle can be calculated from the Y-axis position on the road surface in each segment by the distance obtained from the corresponding parallax value. The road surface height table calculation unit 15 tabulates the height of each road surface portion obtained from the approximate straight line for the required parallax range.

The height of the object projected on the captured image portion corresponding to the point where the Y-axis position is y'at a certain parallax value d from the road surface is such that the Y-axis position on the road surface at the parallax value d is y0. Then, it can be calculated from (y'−y0). In general, the height H from the road surface of an object corresponding to the coordinates (d, y') on the V map can be calculated by the following formula. However, in the following equation, "z" is the distance (z = BF / (d-offset)) calculated from the parallax value d, and "f" is the focal length of the camera (y'-y0). It is a value converted to the same unit as the unit. Here, "BF" is a value obtained by multiplying the baseline length of the stereo camera by the focal length, and "offset" is a parallax value when an object at infinity is photographed.

H = z × (y'−y0) / f
<Clustering, rejection, tracking>
The clustering unit 16 sets the pair (x, y, d) of the x-direction position, the y-direction position, and the differential value d in each of the differential pixel data included in the differential image data on the X-axis and d, Z on the Y-axis. Frequency is set on the axis, and XY two-dimensional histogram information (frequency U map) is created.

The clustering unit 16 is a parallax image in which the height H from the road surface is in a predetermined height range (for example, 20 cm to 3 m) based on the height of each road surface portion tabulated by the road surface height table calculation unit 15. Create a frequency U map only for points (x, y, d). In this case, an object existing in the predetermined height range can be appropriately extracted from the road surface.

In the frequency U map, the clustering unit 16 detects a region where the frequency is higher than a predetermined value and the parallax is dense as the region of the object, and from the coordinates on the parallax image and the actual size (size) of the object. Add individual information such as the predicted object type (people and pedestrians).

The rejection unit 17 rejects the information of the object that is not the recognition target based on the parallax image, the frequency U map, and the individual information of the object.

The tracking unit 18 determines whether or not the detected object is a tracking target when it appears consecutively in frames of a plurality of parallax images.

<Running support control>
Based on the detection result of the object by the clustering unit 16, the control unit 19 performs driving support control such as notifying the driver of the own vehicle 100 of a warning and controlling the steering wheel and brake of the own vehicle. Do.

[Second Embodiment]
In the first embodiment, an example of estimating the road surface based on the frame photographed this time has been described. In the second embodiment, an example of estimating the road surface by using a previously photographed frame will be described. According to the second embodiment, the road surface using the overlapping sample points of the sample point group in the V map based on the frame photographed this time and the sample point group in the V map based on the previously photographed frame. To estimate. As a result, noise that appears accidentally can be removed, so that the accuracy of road surface estimation is improved.

Since the second embodiment is the same as the first embodiment except for a part, the description thereof will be omitted as appropriate.

Hereinafter, the difference between the processing of the in-vehicle device control system 1 according to the second embodiment and the first embodiment will be described. In the second embodiment, for example, the configuration may not have the sample point removing unit 132.

《Sample point extraction process》
The sample point extraction unit 131 according to the second embodiment extracts sample points used for estimating the road surface from the V map generated by the V map generation unit 12.

The sample point extraction process by the sample point extraction unit 131 of the second embodiment will be described with reference to FIGS. 11 and 12. FIG. 11 is a diagram showing an example of a flowchart of the sampling point extraction process according to the second embodiment.

The sample point extraction unit 131 extracts sample points from the V map generated based on the frame taken this time (step S201). This process may be the same as the sampling point extraction process of the first embodiment.

Subsequently, the sample point extraction unit 131 extracts the sample points on the V map extracted based on the frame shot this time and the sample points on one or more V maps extracted based on the frame shot previously. Superimpose (step S202).

FIG. 12 is a diagram illustrating a process of superimposing points in the current V-map and points in the previous V-map.

FIG. 12A is a diagram showing an example of a sample point or a disparity point in the current V map. FIG. 12B is a diagram showing an example of a sample point or a difference point in the previous V map. FIG. 12C is a diagram showing an example of superposed sample points or disparity points.

As shown in FIG. 12, the sample points on the V map extracted based on the frame shot this time and the sample points on one or more V maps extracted based on the frame shot previously are on the V map. It may be superimposed on the same coordinates (d, y). Alternatively, for example, the vehicle speed (speed) of the own vehicle 100 is acquired from CAN (Controller Area Network) or GPS (Global Positioning System) using data I / F603, and the movement of an object or the road surface is captured according to the acquired vehicle speed. May be superimposed. For example, the amount of movement of an object or a road surface may be calculated from the vehicle speed and superimposed on the coordinates (d, y) corresponding to the calculated amount of movement.

Subsequently, the sample point extraction unit 131 extracts sample points from the superimposed sample point group (step S203).

For example, the sample point extraction unit 131 may extract all of the superimposed sample point groups as sample points. As a result, sample points on the road surface are extracted from the previous frame, and in this frame, when the parallax of the road surface is missing, the possibility of extracting the parallax of the object as the sample point is reduced. Can be done.

Alternatively, the sample point extraction unit 131 is a sample point on the V map extracted based on the frame taken this time, and is a sample point on one or more V maps extracted based on the frame taken before. Coordinates that are may be extracted as sample points. As a result, only the sample points detected at the same coordinates for a plurality of frames are extracted, so that the sample points that accidentally appear in the current frame can be removed as noise. In this case, coordinates that appear as sample points more than a predetermined number of times may be extracted as sample points for a predetermined number of frames.

Note that the sample point extraction unit 131 may superimpose the discriminant points instead of superimposing the sample points. In this case, the sample point extraction unit 131 extracts the discriminant points from the superposed discriminant point group in the same manner as the above-described processing.

The appearance of parallax may vary due to the influence of the external environment such as sunshine conditions and the faint white lines that mark the road surface. In this case, if the parallax from the road surface required for road surface estimation is missing and only the parallax of the object is detected, the parallax of the object is detected as a sample point, so that the height of the road surface is highly erroneously estimated.

According to the second embodiment, the accuracy of the road surface estimation is improved by extracting the sample points based on the sample points or the discriminant points of the current frame and the previous frame.
[Third Embodiment]
In the first embodiment, an example of removing a sample point by an object on a road surface in a V map has been described. In the third embodiment, when there is a discrepancy point due to an object on the road surface in the V map, an example in which the height of the road surface is high and a constraint is provided so as not to be misestimated will be described.

Since the third embodiment is the same as the first embodiment or the second embodiment except for a part, the description thereof will be omitted as appropriate.

Hereinafter, the difference between the processing of the in-vehicle device control system 1 according to the third embodiment and the first embodiment or the second embodiment will be described. In the third embodiment, for example, the configuration may not have the sample point removing unit 132.

<< Road surface shape detection processing >>
The road surface shape detection unit 133 according to the third embodiment is restricted so that the height of the road surface is not erroneously estimated when there is a discrepancy point due to an object on the road surface in the V map generated by the V map generation unit 12. Is provided.

The road surface shape detection process by the road surface shape detection unit 133 of the third embodiment will be described with reference to FIGS. 13 and 14. FIG. 13 is a diagram showing an example of a flowchart of the road surface shape detection process according to the third embodiment.

The road surface shape detection unit 133 determines whether or not there is a discrepancy point due to an object on the road surface in the segment to be processed (step S301).

FIG. 14 is a diagram illustrating a process of determining whether or not there is a discrepancy point due to an object on the road surface in the V map. FIG. 14A is a diagram showing an example of a V map when there is a discrepancy point due to an object on the road surface. FIG. 14B is a diagram showing an example of a V map when there is no discrepancy point due to an object on the road surface.

When an object is present on the road surface, as shown in FIG. 14A, the number of disparity points included in the lower left region 651a in the segment to be processed is relatively large.

When there is no object on the road surface, for example, when the road surface is uphill, the number of discrimination points included in the lower left region 651b in the segment to be processed is reduced as shown in FIG. 14 (B). This is because no object is reflected farther than the road surface at each y coordinate in the parallax image.

Therefore, in step S301, the road surface shape detecting unit 133 may determine that there is a discriminant point due to an object on the road surface when the number of discriminant points included in the predetermined region of the segment to be processed is a predetermined number or more. ..

Alternatively, in step S301, the road surface shape detecting unit 133 determines that there is a discriminant point due to an object on the road surface when the shape of the region including the predetermined discriminant point group in the segment to be processed is, for example, a rectangle. You may.

When there is no discrepancy point due to an object on the road surface (NO in step S301), the road surface shape detection unit 133 is set to a sample point extracted by the sample point extraction unit 131 or a sample point not removed by the sample point removal unit 132. Based on this, the shape (position, height) of the road surface in the segment to be processed is detected (step S302), and the processing is terminated.

When there is a discrepancy point due to an object on the road surface (YES in step S301), the road surface shape detection unit 133 detects the shape of the road surface in the segment to be processed according to the constraint that the height of the road surface is high and is not erroneously estimated. (Step S303), and the process ends.

In step S303, when the road surface shape detection unit 133 has a discrepancy point due to an object on the road surface in the segment to be processed, for example, the following processing is performed.
-In the segment to be processed, a plurality of road surfaces are estimated according to a plurality of sample point search methods, and the road surface existing at a lower position among the estimated plurality of road surfaces is adopted.
-Calculate the road surface that extends the road surface estimated in the adjacent segment that is closer to the own vehicle than the segment to be processed to the end (left end) of the segment to be processed. Then, of the calculated road surface and the road surface estimated in the segment to be processed, the road surface existing at a lower position is adopted.
-The default road surface or the historical road surface in the segment to be processed is set as the road surface of the segment to be processed without performing the road surface estimation based on the sample points in the segment to be processed.

<Summary>
According to each of the above-described embodiments, the road surface is detected from the parallax distribution on the V map so as not to be affected by the parallax of the object on the road surface. As a result, it is possible to improve the accuracy of detecting the road surface.

Since the distance value (distance value) and the parallax value can be treated equivalently, the parallax image is described as an example of the distance image in the present embodiment, but the present invention is not limited to this. For example, a distance image may be generated by integrating the distance information generated by using a detection device such as a millimeter wave radar or a laser radar with the parallax image generated by using a stereo camera. Further, the stereo camera and a detection device such as a millimeter wave radar or a laser radar may be used in combination and combined with the detection result of the object by the stereo camera described above to further improve the detection accuracy.

The system configuration in the above-described embodiment is an example, and it goes without saying that there are various system configuration examples depending on the application and purpose. It is also possible to combine some or all of the above-described embodiments.

For example, since the processing of the road surface supplementing unit 134, the smoothing processing unit 135, and the like is not essential, the configuration may not have these functional units.

Further, each functional unit of the processing hardware unit 510, the image analysis unit 600, and the vehicle travel control unit 104 may be configured to be realized by hardware, or the CPU executes a program stored in the storage device. It may be a configuration realized by. The program may be recorded and distributed on computer-readable recording media in an installable or executable format file. Further, examples of the recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc and the like. In addition, a recording medium such as a CD-ROM in which each program is stored, and an HD504 in which these programs are stored can be provided as a program product in Japan or abroad.

1 In-vehicle device control system (an example of "device control system")
100 Own vehicle 101 Imaging unit 103 Display monitor 106 Vehicle travel control unit (an example of "control unit")
11 Parallax image generator (an example of "distance image generator")
12 V-map generator (an example of "generator")
13 Road surface estimation section (an example of "estimation section")
131 Sample point extraction unit 132 Sample point removal unit 133 Road surface shape detection unit 134 Road surface supplement unit 135 Smoothing processing unit 15 Road surface height table calculation unit 16 Clustering unit (an example of "object detection unit")
17 Rejection unit 18 Tracking unit 19 Control unit 2 Image pickup device 510a, 510b Image pickup device 510 Processing hardware unit 600 Image analysis unit (an example of "image processing device")

Japanese Unexamined Patent Publication No. 2015-075800

Claims (8)

Vertical showing the distribution of the frequency of the distance values with respect to the vertical direction of the distance images from the distance images having the distance values corresponding to the distances of the road surface and the objects on the road surface in the plurality of captured images taken by the plurality of imaging units. A generator that generates directional distribution data and
Based on the vertical distribution data, the distance value of the first point having the height of the first point where the frequency of the distance value is equal to or higher than a predetermined threshold value and the distance value smaller than the first point, and the distance value of the first point. When it is determined that the difference from the height of the second point where the difference from the distance value is within the predetermined value and the frequency of the distance value is equal to or greater than the predetermined threshold value is equal to or greater than the predetermined value, the first point is excluded. An estimation unit that estimates the road surface based on points ,
An image processing device characterized by comprising. The estimation unit divides the distribution of the frequency of the distance value in the vertical direction of the distance image into a plurality of segments according to the distance value, and determines the height of the first point and the height of the second point. Based on the points excluding the first point and the point having a distance value smaller than the first point in the segment including the first point when it is determined that the difference between the two is equal to or more than a predetermined value. , Estimate the road surface,
The image processing apparatus according to claim 1 . The estimation unit applies the frequency distribution of the distance values based on the plurality of captured images captured this time to the frequency distribution of the distance values based on the plurality of captured images previously captured at the same coordinate position or image. The road surface is estimated based on the vertical distribution data superimposed on the coordinate positions according to the moving speed of the processing device.
The image processing apparatus according to claim 1 or 2 . When the estimation unit detects the distribution of the frequency of distance values by an object on the road surface based on the vertical distribution data, the road surface existing at a lower position among a plurality of road surfaces estimated by a plurality of methods. Is estimated as the road surface,
The image processing apparatus according to any one of claims 1 to 3 , wherein the image processing apparatus is characterized. With multiple imaging units
A distance image generation unit that generates a distance image having a distance value corresponding to the parallax of the road surface and the object on the road surface in the plurality of captured images from the plurality of captured images captured by the plurality of imaging units.
A generator that generates vertical distribution data showing the distribution of the frequency of distance values with respect to the vertical direction of the distance image, and a generator.
Based on the vertical distribution data, the distance value of the first point having the height of the first point where the frequency of the distance value is equal to or higher than a predetermined threshold value and the distance value smaller than the first point, and the distance value of the first point. When it is determined that the difference from the height of the second point where the difference from the distance value is within the predetermined value and the frequency of the distance value is equal to or greater than the predetermined threshold value is equal to or greater than the predetermined value, the first point is excluded. An estimation unit that estimates the road surface based on points ,
An imaging device comprising. A plurality of imaging units mounted on the moving body to image the front of the moving body,
A distance image generation unit that generates a distance image having a distance value corresponding to the parallax of the moving surface and the object on the moving surface in the plurality of captured images from the plurality of captured images captured by the plurality of imaging units. ,
A generator that generates vertical distribution data showing the distribution of the frequency of distance values with respect to the vertical direction of the distance image, and a generator.
Based on the vertical distribution data, the distance value of the first point having the height of the first point where the frequency of the distance value is equal to or higher than a predetermined threshold value and the distance value smaller than the first point, and the distance value of the first point. When it is determined that the difference from the height of the second point where the difference from the distance value is within the predetermined value and the frequency of the distance value is equal to or greater than the predetermined threshold value is equal to or greater than the predetermined value, the first point is excluded. An estimation unit that estimates the moving surface based on points ,
An object detection unit that detects an object in the plurality of captured images based on the moving surface estimated by the estimation unit and the distance image.
A control unit that controls the moving body based on the data of the object detected by the object detection unit, and
Mobile device control system equipped with. The computer
From a distance image having a distance value corresponding to the distance of the road surface in a plurality of captured images taken by a plurality of imaging units, vertical distribution data showing the frequency distribution of the distance value with respect to the vertical direction of the distance image is generated. Steps and
Based on the vertical distribution data, the distance value of the first point having the height of the first point where the frequency of the distance value is equal to or higher than a predetermined threshold value and the distance value smaller than the first point, and the distance value of the first point. When it is determined that the difference from the height of the second point where the difference from the distance value is within the predetermined value and the frequency of the distance value is equal to or greater than the predetermined threshold value is equal to or greater than the predetermined value, the first point is excluded. The step of estimating the road surface based on the points and
Image processing method to execute. On the computer
From a distance image having a distance value corresponding to the distance of the road surface in a plurality of captured images taken by a plurality of imaging units, vertical distribution data showing the frequency distribution of the distance value with respect to the vertical direction of the distance image is generated. Steps and
Based on the vertical distribution data, the distance value of the first point having the height of the first point where the frequency of the distance value is equal to or higher than a predetermined threshold value and the distance value smaller than the first point, and the distance value of the first point. When it is determined that the difference from the height of the second point where the difference from the distance value is within the predetermined value and the frequency of the distance value is equal to or greater than the predetermined threshold value is equal to or greater than the predetermined value, the first point is excluded. The step of estimating the road surface based on the points and
A program that executes.

Images