JP5299296B2 - Vehicle periphery image display device and vehicle periphery image display method - Google Patents

Description

  The present invention relates to a vehicle periphery image display device and a vehicle periphery image display method for converting an image from an imaging device that captures the periphery of a vehicle into a bird's-eye view image and displaying the image to a vehicle occupant.

  2. Description of the Related Art Conventionally, there has been proposed a device that displays a bird's-eye image obtained by combining a plurality of video images of a camera mounted on a vehicle or changes the viewpoint, and then displays the image on a vehicle occupant.

  For example, in the apparatus described in Patent Document 1, coordinate conversion is performed in order to make a vehicle peripheral image from each camera that captures the front, rear, and left and right sides of the vehicle an overhead image (bird's-eye image). Then, the partial images obtained by the coordinate conversion are combined to generate an overhead image in which the periphery of the host vehicle is viewed from above, and is output to the display.

  In the apparatus described in Patent Document 2, spatial data is created by mapping an input image captured by one or more cameras to a predetermined spatial model in a three-dimensional space, and the spatial data is referred to Then, an image viewed from an arbitrary virtual viewpoint in the three-dimensional space is created and displayed.

JP 2009-101984 Japanese Patent No. 3286306

  However, the apparatus of Patent Document 1 merely converts the vehicle periphery image from each camera into a bird's-eye view image viewed from a virtual viewpoint above the vehicle using the ground as a reference plane. For this reason, when a three-dimensional object exists in the vehicle peripheral image photographed by the camera, the three-dimensional object is distorted (extended) in the converted bird's-eye view image. For this reason, an unnatural and difficult-to-understand image is displayed for the vehicle occupant.

  Also, in the case of the device of Patent Document 2, since only the input image from the camera is mapped to a predetermined space model (a situation or a plane), the same problem (three-dimensional object) is found in the device of Patent Document 2. Distortion).

  The present invention has been made in view of such a point, and even if a three-dimensional structure (three-dimensional object) is present in a photographed image, the vehicle occupant can easily understand it intuitively. An object of the present invention is to provide a vehicle periphery image display device and a display method capable of displaying a bird's-eye view image.

In order to achieve the above object, a vehicle periphery image display device according to claim 1 is provided.
A display device provided in the passenger compartment of the vehicle;
A photographing means mounted on the vehicle for photographing an image around the vehicle;
Image generating means for generating a presentation image for presentation to the driver of the vehicle based on the image photographed by the photographing means;
Display control means for displaying a presentation image generated by the image generation means on a display device,
Furthermore, using an image photographed by the photographing means, the image has a three-dimensional structure extraction means for specifying a region in which a three-dimensional structure extending in the height direction from the ground is projected,
The image generation means
A first image for performing a first image conversion for converting an image photographed by the photographing means into a bird's-eye view image using the ground as a reference plane, excluding the region of the three-dimensional structure specified by the three-dimensional structure extracting means. Conversion means;
This is different from the first image conversion by the first image conversion means, and the distortion of the three-dimensional structure is reduced as compared with the case where the three-dimensional structure is converted into a bird's-eye view image by the first image conversion. A second image conversion unit that converts the image of the three-dimensional structure by applying the second image conversion; and a presentation image obtained by integrating the images converted by the first image conversion unit and the second image conversion unit It is intended to generate a,
In the second image conversion performed by the second image conversion unit, the height of each part of the three-dimensional structure from the ground is calculated using the image captured by the imaging unit, and the height corresponding to the calculated height is calculated. It is converted into a bird's-eye view image as a reference plane .

  As described above, the vehicle periphery image display apparatus according to the first aspect includes the three-dimensional structure extraction means for specifying the region where the three-dimensional structure is displayed in the photographed image. The first image conversion unit of the image generation unit converts the captured image into a bird's-eye image with the ground as a reference plane, excluding the region of the three-dimensional structure, while the second image conversion unit The three-dimensional structure is subjected to image conversion by the second image conversion in which distortion of the three-dimensional structure is reduced as compared with the first image conversion. Then, the images converted by the first and second image conversion means are integrated to generate a presentation image.

  Therefore, even if a three-dimensional structure (three-dimensional object) is present in the photographed vehicle periphery image, a bird's-eye view image with reduced distortion of the three-dimensional structure can be displayed as compared with the conventional case. For this reason, the vehicle occupant can quickly and easily grasp the situation around the vehicle from the displayed image.

  Note that, for example, a single camera is used as a photographing means to perform stereo measurement on two images taken at different timings, or a plurality of cameras are used to take a stereo image on a plurality of images taken simultaneously. By performing processing or using a distance image sensor of the TOF (time of flight) method, the distance to the object to be imaged can be obtained. If the distance to the object to be imaged is obtained, the installation parameters of the image taking means (height from the ground, angle of depression, etc.) are known, and the height from the ground of the object to be imaged can be obtained from the distance. At this time, the imaging target having a height higher than the ground can be regarded as a three-dimensional structure, and thus, the three-dimensional structure can be extracted from the captured image.

In the second image conversion performed by the second image conversion unit, the height of each part of the three-dimensional structure from the ground is calculated using the image captured by the imaging unit, and the height corresponding to the calculated height is calculated. Ru der converts the bird's-eye image as a reference plane is.

When an image including a three-dimensional structure is converted into a bird's-eye view image with the ground as a reference plane, the image is obtained by projecting the three-dimensional structure onto the ground, and the three-dimensional structure is distorted. On the other hand, as described in claim 1 , the height of each part of the three-dimensional structure is calculated and converted into a bird's-eye view image having a height corresponding to the height as a reference plane. The distortion of the three-dimensional structure can be reduced as compared with the bird's-eye view image in which the three-dimensional structure is projected on the ground.

According to a second aspect of the present invention, when the second image conversion means converts the three-dimensional structure into a bird's-eye view image, the second image conversion means preferably performs image conversion in the order from a place where the height from the ground is low to a place where the height is high. The image (original image) which image | photographed the three-dimensional structure by carrying out the bird's-eye-view conversion of each part of the three-dimensional structure by using the height of each part of the three-dimensional structure as a reference plane as described in Claim 1 May be mapped to the same pixel of the bird's-eye view image. In such a case, the pixel conversion at a higher height is overwritten in the bird's-eye view image by performing the image conversion at a higher height later. Therefore, it is possible to appropriately convert the three-dimensional structure into a bird's-eye view image.

According to a third aspect of the present invention, the second image conversion means divides the three-dimensional structure into a plurality of small areas, and adds a restriction to the height difference between the small areas adjacent in the height direction. You may make it calculate the height from the ground of each place of a structure. When the distance to each part of the three-dimensional structure is simply detected and the height of the three-dimensional structure from the ground is calculated based on the distance, the height calculated by the distance detection error or the like May contain errors. In such a case, the shape of the three-dimensional structure in the bird's-eye view image becomes jagged, and the shape cannot be correctly converted into a bird's-eye view image. Therefore, as described above, if the height from the ground of each part of the three-dimensional structure is calculated while limiting the difference in height between adjacent small regions, an error in calculating the height Can be reduced.

According to a fourth aspect of the present invention, when the second image conversion means converts the three-dimensional structure into a bird's-eye image, the bird's-eye image viewed from a direction inclined by a predetermined angle from a direction perpendicular to the ground, As described above, image conversion of a three-dimensional structure may be performed. This is because in the case of a three-dimensional structure, it is easier to recognize the three-dimensional structure if it is converted to a bird's-eye image viewed from an oblique direction, rather than being converted to a bird's-eye image viewed from directly above.

According to the fifth aspect of the present invention, before the three-dimensional structure extraction unit extracts the three-dimensional structure, the image generation unit converts the entire image into a bird's-eye view image using the ground as a reference plane by the first image conversion unit. The bird's-eye view image may be converted into a presentation image.

  Thereby, when the three-dimensional structure is not included in the photographed image, or is included, but the three-dimensional structure has not yet been extracted, a bird's-eye image similar to the conventional one is displayed, When the three-dimensional structure is extracted, a bird's-eye view image with reduced distortion is displayed for the three-dimensional structure. Therefore, it is possible to always display an image showing the situation around the vehicle.

According to a sixth aspect of the present invention, there is provided a mapping table that indicates to which pixel of the presented image each pixel of the image photographed by the photographing means is mapped at predetermined height ranges from a height corresponding to the ground. Preferably, the storage means is stored, and the first image conversion means and the second image conversion means perform the first and second image conversions with reference to the mapping table. This makes it possible to convert the presentation image (bird's-eye view image) at high speed.

As described in claim 7 , in the image photographed by the photographing means, since it is located on the back side of the three-dimensional structure, the complementing means for complementing the image with respect to the defective region where the image information is missing in the presented image You may have. By converting the captured image into a bird's-eye view image using the ground as a reference plane, excluding the region of the three-dimensional structure, a portion hidden behind the three-dimensional structure in the captured image is converted into a bird's-eye view image. It may be a display area. In such a case, the image information in this display area is lost. For this reason, for example, when a defect area where image information is missing is filled with the same pixel value as the surrounding area, or when a lane line extends in both directions across the defect area, the lane line is drawn in the defect area. Alternatively, the image of the missing area may be complemented.

The vehicle periphery image display method according to claims 8 to 14 describes a processing method executed in the vehicle periphery image display device according to claims 1 to 7 , and the operation and effects thereof are as follows. Since it is the same as that of Claims 1-7, description is abbreviate | omitted.

It is a block diagram which shows the structure of the vehicle periphery image display apparatus by 1st Embodiment of this invention. It is a figure for demonstrating the principle which detects the distance from a camera to a three-dimensional structure. It is a figure for demonstrating the method of calculating the height from the ground of a three-dimensional structure. It is a flowchart which shows the process for the periphery image display in the vehicle periphery image display apparatus of FIG. It is a figure which shows an example of the image image | photographed with the camera mounted in the vehicle. It is a figure which shows the image which converted the whole image of FIG. 5 into the bird's-eye view image which uses the ground as a reference plane. 6 is a diagram illustrating an image obtained by converting the three-dimensional structure in FIG. 5 into a bird's-eye image using height information and converting a region other than the three-dimensional structure into a bird's-eye image with the ground as a reference plane. It is a figure which shows the image which complemented the defect | deletion area | region behind the three-dimensional structure in FIG.

(First embodiment)
First, a first embodiment of the present invention will be described in detail based on the drawings. FIG. 1 is a configuration diagram showing a configuration of a vehicle peripheral image display device according to a first embodiment of the present invention. In the present embodiment, an example will be described in which a CCD camera as a photographing unit is mounted on the rear part of the vehicle and photographs the state behind the vehicle.

  As shown in FIG. 1, the vehicle periphery image display device 100 includes two CCD cameras 10 a and 10 b, an image processing ECU 20, and a display device 30.

  The two CCD cameras 10a and 10b are arranged at the rear part of the vehicle, specifically, at the rear door or trunk of the vehicle, with the optical axis pointing downward from the direction parallel to the ground. Installed to be included. In particular, in the present embodiment, the two CCD cameras 10a and 10b are installed at the rear of the vehicle so as to have the same height and the same depression angle from the road surface in a state of being separated by a predetermined distance. Accordingly, the photographing range behind the vehicle by the two CCD cameras 10a and 10b is shifted by the distance between the cameras in the vehicle width direction, and most of the photographing ranges overlap each other.

  The image processing ECU 20 uses the images output from the CCD cameras 10a and 10b, and in one of the images (for example, an image taken by the CCD camera 10a), a three-dimensional structure (three-dimensional object) extending in the height direction from the ground. ) Has a three-dimensional structure estimation unit 21 that identifies an area where the image is displayed.

  A method of estimating and specifying the three-dimensional structure in the image taken by the CCD camera 10a in the three-dimensional structure estimation unit 21 will be described with reference to FIGS.

  In the present embodiment, first, the distance to each part of the three-dimensional structure is measured based on the principle of triangulation from the parallax generated when the same three-dimensional structure is photographed by the two CCD cameras 10a and 10b (stereo). Image processing). Specifically, as shown in FIG. 2, if the parallax d, which is the difference in position of the three-dimensional structure in the images taken by the two CCD cameras 10a and 10b, can be obtained, the three-dimensional structure The distance Z to can be obtained by the following formula 1.

(Equation 1)
Z = f · B / d
In Equation 1, B is the inter-camera distance, and f is the focal length of the CCD cameras 10a and 10b.

  The parallax d can be calculated by sequentially searching which pixel corresponds to which pixel between images taken by the two CCD cameras 10a and 10b. For example, in the captured image (first image) of the CCD camera 10a which is a reference when calculating the parallax d (distance Z), a small area including the pixels for which the parallax d (distance Z) is calculated and the surrounding pixels Set. Then, a small area similar to this small area is set as a captured image (second image) of the CCD camera 10b, and the position of the most similar small area is determined by pattern matching. If the position of the corresponding pixel can be obtained between the first image and the second image by such processing, the parallax d can be calculated for the pixel.

  Then, by executing such processing for all the pixels of the first image serving as a reference, the distance to the shooting target is obtained for all the pixels for which the corresponding pixels can be searched for in the second image in the first image. Can do. In the present embodiment, the two CCD cameras 10a and 10b are mounted on the vehicle at the same height and the same depression angle. Therefore, in the first image and the second image, the corresponding pixels are not shifted in the vertical direction, but are present in positions shifted only in the horizontal direction. For this reason, the search for the corresponding pixel position can be performed relatively easily.

  The example in which the distance to the object to be photographed is obtained using the two CCD cameras 10a and 10b has been described above, but three or more CCD cameras may be used. On the other hand, even when only one CCD camera is provided, it is possible to perform stereo measurement by using a so-called structure from motion (SFM) technique and obtain the distance to the object to be photographed.

  In this SFM technique, feature points such as corner points and edges that are easy to correspond are extracted from two or more images taken at different timings by one CCD camera and having different shooting positions. When the extracted feature points are associated with each other in two or more images, based on the positional relationship, a basic matrix indicating the relative positional relationship (rotation and translational position) between the CCD cameras at each photographing position or 3 Find the double tensor matrix. And based on the relative positional relationship shown by those matrices, the distance to each imaging object is calculated from the principle of triangulation (stereo measurement).

  Further, by using a TOF (time of flight) type distance image sensor, the distance of each pixel to the photographing target may be directly obtained. Note that the TOF type distance image sensor measures the distance to the reflective object for each pixel by, for example, modulating and projecting infrared light and measuring the phase difference with the reflected light from the reflective object for each pixel. It is possible.

  When the distance is determined for each pixel of the image captured by the CCD camera 10a, the three-dimensional structure estimation unit 21 determines the height from the ground of the target object captured by each pixel based on the determined distance. Is calculated.

Since the installation height and the depression angle of the CCD camera 10a from the ground are determined in advance, the distance from the CCD camera 10a to the ground on which each pixel of the CCD camera 10a is projected takes into consideration the angle of view of the CCD camera 10a and the like. Can be calculated in advance. The three-dimensional structure estimation unit 21 stores in advance a distance Z ₀ to the ground of each pixel.

Here, as shown in FIG. 3, it is assumed that a pixel of the CCD camera 10a projects the tip of a three-dimensional structure. The distance to the tip of the three-dimensional structure is assumed to be a distance Z based on the distance obtained for each pixel of the image taken by the CCD camera 10a. In this case, the ratio between the distance Z and the distance Z ₀ is the height obtained by subtracting the height h from the height H from the ground of the CCD camera 10a to the tip of the three-dimensional structure, high from the ground of the CCD camera 10a It becomes the same as the ratio with the height H. For this reason, the height h to the tip of the three-dimensional structure on which a certain pixel of the CCD camera 10a is projected can be calculated by the following formula 2.

(Equation 2)
h = H × (1−Z / Z ₀ )
Using the distance obtained for each pixel of the image photographed by the CCD camera 10a, the height h from the ground of the photographing object projected by each pixel is calculated by performing the calculation of Equation 2. Can do. If the calculated height h of the imaging target is higher than a threshold for determining whether or not the height of the imaging target is higher than the height of the ground, the imaging target is not the ground but a three-dimensional structure extending in the height direction from the ground. Can be considered part of That is, it is possible to determine whether the shooting target is the ground or the three-dimensional structure based on the calculated height h of the shooting target, and thereby the three-dimensional structure is displayed in the shot image. Can be identified.

  In the three-dimensional structure estimation unit 21, the height calculated for each pixel is given to the image generation unit 22 of the image processing ECU 20. The image generation unit 22 refers to the mapping table 23 based on the height calculated for each pixel given from the three-dimensional structure estimation unit 21, and converts the image captured by the CCD camera 10a into a bird's-eye view image. The presentation image to be presented to the vehicle occupant is generated. Below, the conversion process into the bird's-eye image in the image generation part 22 is demonstrated in detail.

  First, the mapping table 23 will be described. The mapping table 23 shows the relationship between the pixel positions of the image captured by the CCD camera 10a and the bird's-eye view image. That is, the mapping table 23 indicates a positional relationship in which the pixel value of each pixel of the captured image is mapped to which pixel position of the bird's-eye image in order to convert the captured image into a bird's-eye image. With respect to the bird's-eye view image, the relationship between the pixel positions of the captured image and the bird's-eye view image is uniquely determined by determining in advance from which viewpoint the bird's-eye view image is to be used. By storing the relationship between the pixel positions in the mapping table 23, conversion from the captured image to the bird's-eye view image can be performed at high speed.

In particular, in the present embodiment, the mapping table 23 has a relationship between pixel positions of a captured image and a bird's-eye image at each height from a height corresponding to the ground to a predetermined height range (for example, 5 to 10 cm). Is remembered. For convenience of explanation, except for the height corresponding to the ground, the height at which the positional relationship between the pixel positions is stored is referred to as a height threshold, and in order from a position one step higher than the height of the ground to the highest position, It will be expressed as h ₁ , h ₂ ,..., h _n .

When the image generation unit 22 obtains the height h from the ground of the object to be imaged on which each pixel is projected from the three-dimensional structure estimation unit 21, the positional relationship between the pixel positions is stored as the height h. Each of the height thresholds h ₁ , h ₂ ,... H _n corresponding to each height is sequentially compared. In this case, the height h is lower than the height threshold h _1, the pixels associated with that height, it can be considered that reflects the ground. On the other hand, the height h is equal to or height threshold h ₁ above, is then compared to the height threshold h _2. Thereafter, in the same manner, the comparison process is repeated to specify a height threshold value that exceeds the height h. At this time, if the height h is equal to or higher than the height threshold h ₂ , but is less than the height threshold h ₃ , the height h is used for conversion of the pixel position of the pixel having the height information of the height h. to a is given, it is determined that the relationship between pixel positions of the captured image and the bird's-eye image corresponding to the height threshold h _2. Such a process has a height threshold h ₁ or more of the height, it is possible to specify the pixel area that reflects the three-dimensional structure.

Then, the image generation unit 22 first determines the height of the ground in the mapping table 23 with respect to a pixel that is assumed that the height h is less than the height threshold value h ₁ and the ground is projected instead of the three-dimensional structure. With reference to the relationship between the pixel positions of the corresponding captured image and bird's-eye view image, the pixel value is mapped on the image plane for generating the bird's-eye view image. The relationship between the pixel positions of the captured image and the bird's-eye image corresponding to the height of the ground is determined so that the conversion to the bird's-eye image is performed using the ground as a reference plane. Therefore, the captured image can be converted into a bird's-eye view image with the ground as a reference plane, except for the pixel region in which the three-dimensional structure is projected.

Next, for a pixel having height information not less than the height threshold h ₁ and less than the height threshold h ₂ , refer to the relationship between the pixel positions of the captured image and the bird's-eye image corresponding to the height threshold h ₁ in the mapping table 23. Then, the pixel value is mapped on the image plane for generating the bird's-eye view image. Such processing is repeated thereafter from the height threshold h ₂ to a height threshold h _n. Thereby, it is possible to perform image conversion from a captured image to a bird's-eye image in consideration of height information, and to generate a bird's-eye image as a presentation image.

  The display device 30 is installed on, for example, an instrument panel in a vehicle interior of the vehicle, and displays a bird's-eye view image generated by the image generation unit 22. In the present embodiment, in the bird's-eye view image displayed on the display device 30, since the distortion of the three-dimensional structure is reduced as compared with the conventional case, the vehicle occupant can calculate from the image displayed on the display device 30. The situation around the vehicle (around the rear) can be grasped quickly and easily.

  Next, the display process of the surrounding image in the vehicle surrounding image display apparatus 100 of this embodiment is demonstrated using the flowchart of FIG. Note that the process shown in the flowchart of FIG. 4 is periodically and repeatedly executed in the vehicle periphery image display device 100.

  First, in step S110, images simultaneously captured by the CCD cameras 10a and 10b are input. FIG. 5 shows an example of an image taken by the CCD camera 10a. In the image shown in FIG. 5, a triangular pyramid-shaped three-dimensional structure exists in a part of the parking area behind the vehicle.

  Next, the process proceeds to step S120, and a three-dimensional structure extraction process is executed. Specifically, as described above, the parallax d of each pixel is obtained between the images captured by the CCD cameras 10a and 10b, and the distance Z to the imaging target of each pixel in the captured image of the CCD camera 10a is obtained. calculate. Furthermore, using the calculated distance Z, the height h of each pixel from the ground to be imaged is calculated.

  In the subsequent step S130, a pixel on the bird's-eye image generation screen is displayed so that a subject to be photographed having a height less than the height threshold h1, that is, a pixel displaying the ground is converted into a bird's-eye image with the ground as a reference plane. Map values. In step S140, pixel values are mapped on the bird's-eye image generation screen for conversion into a bird's-eye image in order from the lower part to the higher part with respect to the pixels displaying the three-dimensional structure. In subsequent step S150, the generated bird's-eye view image is displayed on display device 30.

  As described above, in the present embodiment, when performing image conversion from a captured image to a bird's-eye view image, for each pixel of the captured image, the height h from the ground of the imaging target captured by each pixel is used. Bird's eye view conversion is performed with a height corresponding to the height as a reference plane. For example, as shown in FIG. 5, conversion to a bird's-eye view image is performed in addition to pixel positions ((X1, Y1) and (X2, Y2)) in the captured image, and height information (H (X1, X1, Y2)) at each pixel position. Y1) and H (X2, Y2)).

  For this reason, as compared with the conventional case, the bird's-eye view image is uniformly compared with the case of performing the conversion to the bird's-eye view image using the ground as a reference plane regardless of whether or not the photographed image includes a three-dimensional structure. The distortion of the three-dimensional structure can be reduced. For example, FIG. 6 shows an image obtained by converting the entire photographed image shown in FIG. 5 into a bird's-eye view image with the ground as a reference plane, and FIG. 7 shows the photographed image of FIG. The image which converted into the bird's-eye view image in consideration of the height information of each pixel is shown. As is apparent from the images of FIGS. 6 and 7, the vehicle periphery image display device 100 according to the present embodiment displays an image in which the distortion of the three-dimensional structure is reduced as compared with the conventional bird's-eye view image. Can do. In FIG. 7, the pixel positions (X′1, Y′1) and (X′2, Y′2) are the pixel positions (X1, Y1), ( It shows the pixel position of the bird's-eye view image corresponding to (X2, Y2).

  In the present embodiment, when the image generation unit 22 converts a captured image including a three-dimensional structure into a bird's-eye image, first, the corresponding position in the bird's-eye image with respect to pixels having height information corresponding to the ground. The pixel value is mapped to. In addition, with regard to the pixels that project the three-dimensional structure, the pixel values are mapped to the corresponding positions in the bird's-eye view image in the order of the pixels that project the high part from the pixels that project the low part from the ground. doing.

  At this time, in the present embodiment, the pixel values are mapped on the image plane for generating the bird's-eye image with reference to the relationship between the pixel positions of the captured image and the bird's-eye image corresponding to each height threshold. There is a possibility that a plurality of pixels, particularly a three-dimensional structure in the captured image, are mapped to the same pixel in the bird's-eye view image. However, in the present embodiment, the mapping of the pixel values of the pixels that project a higher part is performed later so that even if the pixel values of a plurality of pixels are mapped to the same position, The pixel value of the pixel showing the high part is overwritten on the bird's eye image generation screen. Therefore, it is possible to appropriately convert the three-dimensional structure into a bird's-eye view image.

  As shown in FIG. 7, when the image conversion of the three-dimensional structure into the bird's-eye view image is performed in consideration of the height information, the image is displayed on the display device 30 because it is located on the back side of the three-dimensional structure. In the presented image, a defective area in which image information is lost occurs. This missing area may be displayed as it is, but the missing area may be supplemented so that the vehicle occupant can more easily see the image.

  When complementing the defect area, for example, if the defect area is filled with the same pixel value as the surrounding area, or if the lane markings extend in both directions with the defect area in between, the lane markings in the defect area Or draw. FIG. 8 shows an image in which the missing area in FIG. 7 is complemented.

  The preferred embodiment of the present invention has been described above, but the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  For example, in the above-described embodiment, when the three-dimensional structure estimation unit 21 calculates the height h from the ground of the photographing target on which each pixel of the photographed image is projected, first, the three-dimensional structure is projected. The pixel area is extracted for each three-dimensional structure. Then, the three-dimensional structure grasped from the pixel region is divided into a plurality of small regions (for example, Delaunay division), and the difference in height between the small regions adjacent to each other in the height direction is limited. You may make it calculate the height from the ground of each place. Simply detect the distance to each part of the 3D structure from the parallax between the two captured images, and calculate the height of the 3D structure from the ground based on the distance. There may be an error in the calculated height due to an error or the like. In such a case, the shape of the three-dimensional structure in the bird's-eye view image becomes jagged, and the shape cannot be correctly converted into a bird's-eye view image.

  On the other hand, as described above, if the height from the ground of each part of the three-dimensional structure is calculated while limiting the difference in height between adjacent small regions, the height is calculated. Can be reduced.

  Further, in the above-described embodiment, when the three-dimensional structure is converted into a bird's-eye view image, the pixels in the mapping table so as to be a bird's-eye view image viewed from a direction inclined by a predetermined angle from a direction perpendicular to the ground. A positional relationship may be determined. This is because in the case of a three-dimensional structure, it is easier to recognize the three-dimensional structure if it is converted to a bird's-eye image viewed from an oblique direction, rather than being converted to a bird's-eye image viewed from directly above.

  Further, for example, when the vehicle periphery image display device 100 includes one camera and obtains height information of a three-dimensional structure using the SFM technique, several times are required before obtaining the height information. Need an image of the frame. In such a case, that is, before the three-dimensional structure is extracted, the entire image captured by the camera may be converted into a bird's-eye view image with the ground as a reference plane, and the bird's-eye view image may be used as a presentation image.

  As a result, when the three-dimensional structure is not included in the photographed image, or when the three-dimensional structure is not extracted yet, the same bird's-eye image is displayed. When the three-dimensional structure is extracted, a bird's-eye view image with reduced distortion is displayed for the three-dimensional structure. Therefore, it is possible to always display an image showing the situation around the vehicle.

  Furthermore, in the above-described embodiment, the distortion of the three-dimensional structure is reduced by performing bird's-eye view image conversion considering height information for the three-dimensional structure. However, regarding the three-dimensional structure, the bird's-eye view image as the presentation image may be generated by performing only the scale conversion without performing the conversion to the bird's-eye view image.

  Since the CCD camera 10a captures the three-dimensional structure so as to look down at a predetermined depression angle, even if the three-dimensional structure is displayed as it is without performing bird's-eye view conversion, there is almost no sense of incongruity for the vehicle occupant. Conversely, since the distortion of the three-dimensional structure associated with the conversion to the bird's-eye view image does not occur, a display that is easy for the vehicle occupant to understand can be performed.

  The scale conversion of the three-dimensional structure and the generation of the presentation image are performed as follows. First, in the captured image, a region of the three-dimensional structure is specified, and a tangential portion where the three-dimensional structure is in contact with the ground is specified. Next, with respect to the pixels projecting the ground, excluding the pixel region projecting the three-dimensional structure, conversion to a bird's-eye view image using the ground as a reference plane is performed. In the bird's-eye view image, a tangent portion with the ground three-dimensional structure is specified. Then, the three-dimensional structure is scaled so that the tangent portion of the captured image has the same length as the tangent portion of the bird's-eye image, and the three-dimensional structure is displayed on the bird's-eye image so that the tangent portions coincide with each other. The presentation image is generated by writing the pixel value of the existing pixel.

10a, 10b CCD camera 20 Image processing ECU
30 Display device
100 Vehicle periphery image display device

Claims (14)

A display device provided in the passenger compartment of the vehicle;
A photographing means mounted on the vehicle for photographing an image around the vehicle;
Image generating means for generating a presentation image for presentation to the driver of the vehicle based on the image photographed by the photographing means;
A vehicle surrounding image display device comprising: display control means for displaying the presentation image generated by the image generation means on the display device;
Using an image photographed by the photographing means, and having a three-dimensional structure extracting means for specifying an area in which the three-dimensional structure extending in the height direction from the ground is projected in the image;
The image generating means includes
Except for the region of the three-dimensional structure specified by the three-dimensional structure extraction means, a first image conversion is performed for converting the image photographed by the photographing means into a bird's-eye image with the ground as a reference plane. First image conversion means;
This is different from the first image conversion by the first image conversion means, and is more distorted than the case where the three-dimensional structure is converted into a bird's-eye image by the first image conversion. A second image conversion unit that converts the three-dimensional structure into an image by applying a second image conversion to reduce the image, and converted by the first image conversion unit and the second image conversion unit The presenting image is generated by integrating images ,
The second image conversion performed by the second image conversion means calculates the height from the ground of each place of the three-dimensional structure using the image photographed by the photographing means, and corresponds to the calculated height. A vehicle peripheral image display device that converts a bird's-eye view image having a height as a reference plane . The said 2nd image conversion means performs image conversion in order of the location where the height from the ground is low, when converting the said three-dimensional structure into a bird's-eye view image . Vehicle periphery image display device. The second image conversion means divides the three-dimensional structure into a plurality of small areas, and limits the difference in height between the small areas adjacent in the height direction, while restricting the ground in various places of the three-dimensional structure. The vehicle surrounding image display device according to claim 1 , wherein a height from the vehicle is calculated . The second image conversion means converts the three-dimensional structure into a bird's-eye image so that the bird's-eye image is viewed from a direction inclined by a predetermined angle from a direction perpendicular to the ground. The vehicle periphery image display device according to any one of claims 1 to 3 , wherein image conversion of a structure is performed . The image generation means converts the entire image into a bird's-eye image with the ground as a reference plane by the first image conversion means before the three-dimensional structure is extracted by the three-dimensional structure extraction means. the vehicle periphery image display device according to any one of claims 1 to 4, characterized in that said presentation image. Storage means storing a mapping table indicating which pixel of the presented image each pixel of the image photographed by the photographing means is mapped every predetermined height range from the height corresponding to the ground;
Said first image converting means and said second image transformation means, by referring to the mapping table, according to any one of claims 1 to 5, characterized in that performing the first and second image conversion Vehicle periphery image display device. Since the image photographed by the photographing means is located on the back side of the three-dimensional structure, it has a complementing means for complementing the image with respect to a defect area where image information is missing in the presented image. vehicle periphery image display device according to any one of claims 1 to 6. An image acquisition step for acquiring an image from a photographing means mounted on the vehicle and capturing an image around the vehicle;
An image generation step for generating a presentation image for presentation to the driver of the vehicle based on the image acquired by the image acquisition step;
A display step for displaying the presentation image generated by the image generation step to a vehicle occupant, and a vehicle periphery image display method comprising:
Using the image acquired in the image acquisition step, and in the image, a three-dimensional structure extraction step for specifying a region in which a three-dimensional structure extending in the height direction from the ground is projected,
The image generation step includes
First image conversion for converting the image acquired by the image acquisition step into a bird's-eye image using the ground as a reference plane, excluding the region of the three-dimensional structure specified in the three-dimensional structure extraction step. Performing a first image conversion step;
The first image conversion in the first image conversion step is different from the first image conversion, and the distortion of the three-dimensional structure is higher than that in the case where the three-dimensional structure is converted into a bird's-eye image in the first image conversion. Applying a second image conversion to reduce the image, and converting the three-dimensional structure into an image,
Said first image transformation step, by integrating the converted image by the second image conversion step generates the presentation image,
The second image conversion uses the image acquired in the image acquisition step to calculate the height of each part of the three-dimensional structure from the ground, and uses the height corresponding to the calculated height as a reference plane. The vehicle periphery image display method characterized by converting into a bird's-eye view image . The said 2nd image conversion step WHEREIN: When converting the said three-dimensional structure into a bird's-eye view image, image conversion is performed in order of the location where a height from a ground is low to a high location. Vehicle periphery image display method. In the second image conversion step, the three-dimensional structure is divided into a plurality of small areas, and a difference in height between small areas adjacent to each other in the height direction is limited, and the ground at various places of the three-dimensional structure is added. The vehicle periphery image display method according to claim 8, wherein a height from the vehicle is calculated . In the second image conversion step, when the three-dimensional structure is converted into a bird's-eye view image, the three-dimensional structure is formed so that a bird's-eye view image viewed from a direction inclined by a predetermined angle from a direction perpendicular to the ground is obtained. 11. The vehicle periphery image display method according to claim 8 , wherein image conversion of a structure is performed . In the image generation step, before the three-dimensional structure is extracted by the three-dimensional structure extraction step, the entire image is converted into a bird's-eye image using the ground as a reference plane by first image conversion. The vehicle surrounding image display method according to any one of claims 8 to 11 , wherein the presentation image is used as the presentation image . Storage means storing a mapping table indicating which pixels of the image to be mapped to each pixel of the image acquired by the image acquisition step at predetermined height ranges from a height corresponding to the ground ,
The said 1st image conversion means and the said 2nd image conversion means perform 1st and 2nd image conversion with reference to the said mapping table, The one of Claim 8 thru | or 12 characterized by the above-mentioned. Vehicle periphery image display method. In the image acquired by the image acquisition step, a position on the back side of the three-dimensional structure
As a result, the image is complemented for the missing area where the image information is missing in the presented image.
The vehicle surrounding image display method according to claim 8, further comprising a complementing step .

Images