JP2012147149A - Image generating apparatus - Google Patents

Abstract

An overhead view image capable of recognizing a three-dimensional object existing on a road surface with a good sense of distance is generated from a photographed image of an in-vehicle camera.
A peripheral overhead image generation unit that generates a peripheral overhead image from a photographed image using first projective transformation, and a stereoscopic object that exists in the field of view of an in-vehicle camera and includes the position of the stereoscopic object. A three-dimensional object detection unit that outputs object information, a three-dimensional object extraction unit that extracts a three-dimensional object image, which is an image region in which a three-dimensional object is captured, from a captured image based on the three-dimensional object information; A three-dimensional object overhead image generation unit that generates a three-dimensional object overhead image from a three-dimensional object image using the second projective transformation that reduces the image, and an image composition unit that synthesizes the three-dimensional object overhead image with the peripheral overhead image based on the three-dimensional object information An image generation apparatus provided.
[Selection] Figure 1

Description

  The present invention relates to an image generation apparatus that generates, as a display image, a bird's-eye view image from an upper virtual viewpoint above a camera viewpoint by projective transformation of a captured image acquired by an in-vehicle camera that captures a peripheral region of the vehicle.

  In a conventional overhead image generation device, a captured image acquired by an in-vehicle camera is projected onto a projection plane parallel to the road surface, that is, an overhead image from directly above is generated by projective transformation with the virtual viewpoint positioned vertically upward. Is done. Therefore, the bird's-eye view image is displayed on the monitor, thereby assisting the driver to grasp the road surface condition around the vehicle. However, the overhead image obtained through such projective transformation has a problem that the image is more distorted in a distant region than in the vicinity region of the in-vehicle camera, making it difficult for the driver to grasp the sense of distance.

  In order to solve such a problem, for example, in the image processing apparatus described in Patent Document 1, an acquisition unit that acquires a captured image (camera image) captured by an in-vehicle camera, and the captured image is projected onto a plane. A first image conversion unit that converts the image into a planar projection image; and a second image conversion unit that projects the camera image onto a curved surface and converts the image into a curved projection image (for example, a cylindrical projection image). The image conversion unit converts a first image region having a predetermined width in the captured image into the planar projection image, and the second image conversion unit is outside the first image region in the width direction in the captured image. Is converted into the curved projection image. Therefore, by giving an image converted by this apparatus to the display unit, an image having both a planar projection image close to an image that the driver can actually see visually and a curved projection image with a wide field of view is displayed. For this reason, a driver's visibility fall is suppressed. However, the device according to Patent Document 1 is not limited to a solid object (not just a person or a car, but a construction work) that is arranged on the road surface, even though the display of road markings and shoulders drawn on the road surface is easy to see. Road obstacles such as triangular cones for use) become distorted shapes extending in the shooting direction.

  In addition, in the image generation device described in Patent Document 2, a near-view screen set on the road surface as a virtual projection plane, and the near-view screen connected to the side far from the real camera and have a predetermined upward tilt angle. The virtual stereoscopic projection plane having the far-view screen set in the above is set, and the virtual camera is set at a position higher than the real camera that acquires the camera video data from the image projected on the real imaging plane. Then, coordinate conversion is performed between each pixel position of the single camera video data acquired by the real camera and each pixel position on the virtual imaging plane of the virtual camera via the virtual stereoscopic projection plane, and the virtual position is converted according to the coordinate conversion. Each pixel of the camera-captured image is moved onto the virtual imaging plane of the camera, and an image projected on the virtual imaging plane when the virtual stereoscopic projection plane is viewed from the virtual camera is displayed on the monitor. As a result, the monitor image is a composite image of a close-up bird's-eye view image in which a sense of distance is easily recognized and a perspective image of a distant view that is easy to recognize a sense of distance, in which a distant object looks small and a close object looks large. . However, even in this apparatus, when a captured image of a three-dimensional object arranged on the road surface is converted by a virtual projection plane that is horizontal to the road surface, the shape becomes long and the visibility of the three-dimensional object is inevitably lowered. .

  Furthermore, in the vehicle surrounding monitoring system according to Patent Document 3, the overlapping area captured by the plurality of in-vehicle cameras is displayed while the overhead view display image created based on the original images captured by the plurality of in-vehicle cameras is displayed. When the obstacle detection device detects an obstacle present in the image, the image displayed on the display device is switched from the overhead view display image to the original images of a plurality of in-vehicle cameras that capture the overlapping area. . In this configuration, when an obstacle is detected, the type of the obstacle is identified intuitively and instantaneously through the original image, but it is not possible to satisfy the desire to see the obstacle well through the overhead image. .

JP 2008-181330 A (paragraph number [0010-0016], FIG. 5) JP 2008-83786 A (paragraph number [0007-0012], FIGS. 1 and 2) JP 2007-235529 A (paragraph number [0007-0008], FIG. 11)

  In view of the above circumstances, an object of the present invention is to generate a bird's-eye view image capable of recognizing a three-dimensional object existing on a road surface with a good sense of distance from a photographed image of an in-vehicle camera.

A feature of the image generation apparatus according to the present invention is that a bird's-eye view image generated by projective transformation of a captured image acquired by an in-vehicle camera that captures a peripheral area of a vehicle is projected from an upper virtual viewpoint as a display image. A peripheral overhead image generation unit that generates a peripheral overhead image from the captured image using projective transformation;
A three-dimensional object detection unit that detects a three-dimensional object existing in the field of view of the in-vehicle camera and outputs three-dimensional object information including the position of the three-dimensional object; and the three-dimensional object is extracted from the captured image based on the three-dimensional object information. A three-dimensional object overhead image is generated from the three-dimensional object image using a three-dimensional object extraction unit that extracts a three-dimensional object image that is a captured image region and a second projective transformation that reduces distortion of the three-dimensional object in the peripheral overhead image. A three-dimensional object overhead image generation unit, and an image composition unit that synthesizes the three-dimensional object overhead image with the surrounding overhead image based on the three-dimensional object information.

  According to this configuration, when a captured image of the in-vehicle camera is converted into a peripheral overhead view image from the upper virtual viewpoint using the first projective transformation, if the presence of a three-dimensional object is detected in the field of view, it is extracted from the captured image. A three-dimensional object overhead image is generated from the three-dimensional object image using a second projective transformation different from the first projective transformation, and the three-dimensional object overhead image is synthesized with a peripheral overhead image. The second projective transformation has parameters that reduce distortion of the three-dimensional object in the peripheral overhead image compared to the first projective transformation used for generating the peripheral overhead image. Therefore, the three-dimensional object in the synthesized peripheral bird's-eye view image has a natural shape that is easier to visually recognize, rather than having an elongated shape as in the conventional case. From this, the driver can recognize a three-dimensional object existing on the road surface with a good sense of distance from the surrounding bird's-eye view image.

  As is clear from the above description, the first projective transformation is a bird's-eye view image from the upper virtual viewpoint, that is, a projective transformation that projects a photographed image from above onto a projection plane parallel to the road surface. The projected image of the placed three-dimensional object has an extended distortion. For this reason, it is preferable that the second projective transformation uses a projection surface that is not parallel to the road surface and is inclined with respect to the road surface as much as possible. Therefore, in one preferred embodiment of the present invention, the second projective transformation employs a projective transformation in which the photographed image is projected from a photographing viewpoint onto a projection plane perpendicular to the road surface.

  In order to set projection transformation parameters that minimize the distortion of the three-dimensional object on the road surface included in the captured image in the overhead image, the shape and the position of the projection surface and the three-dimensional object are used for the projection transformation. Relationship is an important factor. Therefore, in a preferred embodiment of the present invention, based on the three-dimensional object information including the position and the shape of the three-dimensional object in the three-dimensional object in the shooting space, the shape of the projection surface or the three-dimensional object projected on the projection surface and The conversion parameter of the second projective transformation, which is the relative position or both, is set. The shape of the projection surface is preferably a curved surface or a bent surface as described in Patent Documents 1 and 2 described above. At that time, in particular, it is preferable that the front side of the boundary area between the three-dimensional object and the road surface is a projection plane part parallel to the road surface, and the rear side thereof is a vertical plane or an inclined plane close to the vertical plane. Thereby, the visual defect that the top of the three-dimensional object extends in the overhead image can be eliminated.

  Since the coordinates of the three-dimensional object are different between the peripheral overhead image generated using different projective transformations, that is, the three-dimensional object overhead image and the three-dimensional object overhead image, even if the three-dimensional object overhead image is synthesized with the peripheral overhead image, The three-dimensional object shown in the image remains in the surrounding overhead image after the image composition. In particular, when the surrounding bird's-eye view image combines a plurality of in-vehicle camera shot images with different shooting directions, the disagreement of the position of the three-dimensional object between the peripheral bird's-eye view image and the three-dimensional object bird's-eye view image becomes significant. For this reason, in a preferred embodiment of the present invention, a three-dimensional object suppression unit is provided that performs a suppression process so that the region of the three-dimensional object image in the peripheral overhead image is not conspicuous. As such suppression processing, blurring processing, desaturation processing, blend processing with surrounding pixel groups, and the like can be used.

  In order to make a three-dimensional object around the vehicle more clearly visible, it is effective to make the three-dimensional object bird's-eye image synthesized with the peripheral bird's-eye image more noticeable. For this reason, in a preferred embodiment of the present invention, color correction processing is performed on the three-dimensional object overhead image so that the three-dimensional object overhead image is highlighted. For example, increasing the color saturation of a three-dimensional object is a preferred embodiment.

  When the surrounding bird's-eye view image combines a plurality of in-vehicle camera captured images having different capturing directions, a three-dimensional object may be reflected in the two captured images. In such a case, since a three-dimensional object can be extracted from each captured image, it is preferable to select only one of them as a more appropriate three-dimensional object. For this purpose, in a preferred embodiment in which a plurality of the in-vehicle cameras are installed to capture the entire periphery of the vehicle, the solid object is included in the captured images of two or more cameras. Is configured such that a three-dimensional object image having a smaller three-dimensional object region is extracted. The reason is that it has been experimentally shown that the smaller the area of the three-dimensional object region, the smaller the distortion of the three-dimensional object in the overhead image, in particular, the smaller the inclination.

Schematic description of a basic image processing process used in the image generation apparatus according to the present invention to separately generate and synthesize a peripheral bird's-eye view and a three-dimensional object bird's-eye view with different projective transformations from an image captured by an in-vehicle camera. It is a schematic diagram to do. It is the schematic diagram which illustrated the process which synthesize | combines a three-dimensional object bird's-eye view image with a periphery bird's-eye view image, and produces | generates the final bird's-eye view image for a display. The present invention is a functional block diagram of a vehicle periphery monitoring system incorporating an image generating apparatus according to the present invention. It is a functional block diagram of the image processing module which comprises a vehicle periphery monitoring system. It is the schematic diagram which illustrated the process of producing | generating a three-dimensional object bird's-eye view image using the projective transformation selected from the various projective transformations. It is a flowchart which shows a bird's-eye view image display routine.

  First, a basic image processing process used in the image generation apparatus according to the present invention to separately generate and synthesize a peripheral bird's-eye view and a three-dimensional object bird's-eye view using different projective transformations from an image taken by an in-vehicle camera will be described. . FIG. 1 is a basic conceptual diagram schematically illustrating generation of an overhead image using two different projective transformations. Here, for the sake of simplicity, only a photographed image by a back camera as an in-vehicle camera is shown. The generation of the overhead image used is shown.

  In order to monitor and display a bird's-eye view image as a vehicle periphery monitoring screen, first, a captured image of a peripheral region in the traveling direction of the own vehicle is acquired by the in-vehicle camera (# 1). At the same time, when a three-dimensional object is detected around the vehicle and a three-dimensional object is detected within the field of view of the in-vehicle camera, three-dimensional object information describing data such as the position, orientation, and size of the three-dimensional object Is generated (# 101). From the captured image of the back camera, the first projective transformation with the projection plane parallel to the road surface, that is, the viewpoint transformation with the virtual viewpoint set directly above (# 2). Through this first projective transformation process, a peripheral overhead image that is an overhead image from directly above the captured image is obtained. If the three-dimensional object information is generated, the three-dimensional object should appear in the surrounding overhead image. Therefore, the three-dimensional object area is calculated using position information from the three-dimensional object information, edge detection processing, and the like, and the three-dimensional object area is suppressed so that the three-dimensional object area is not conspicuous from the periphery. A suppressed peripheral overhead image is obtained (# 3). In addition, in the surrounding bird's-eye view generated in this way, when a three-dimensional object is present in front of the camera, distortion occurs such that the vicinity of the top of the three-dimensional object extends. As this suppression processing, image processing such as blurring, blending with surrounding pixels (such as α blending), and desaturation is performed.

  If the three-dimensional object information is generated, the three-dimensional object region is extracted from the photographed image using position information from the three-dimensional object information, edge detection processing, or the like (# 102). The extracted three-dimensional object image is subjected to the second projective transformation using a projection surface having an inclination angle that is nearly perpendicular to the road surface at least on the far side, and a three-dimensional object overhead image is obtained (# 103). . In the example of FIG. 1, a curved surface curved upward in the vertical direction from the road surface is used. In the second projective transformation using such a projection surface, the vicinity of the top of the three-dimensional object is not a virtual viewpoint for viewing the road surface from above, but an overhead view using a virtual viewpoint for viewing the road surface from a substantially horizontal direction. Therefore, the inconvenience of extending the vicinity of the top of the three-dimensional object as generated by the first projective transformation described above is avoided.

  In step # 3, the three-dimensional object bird's-eye view image is synthesized so as to be superimposed on the position where the three-dimensional object exists on the three-dimensional object suppression peripheral bird's-eye view image (# 4). The synthesized overhead image is sent to the monitor as a monitor display overhead image.

  In the basic conceptual diagram of FIG. 1, the final bird's-eye view image for the purpose of monitor display was generated using only the photographed image from the back camera. There is a high demand for an all-around overhead image that can be grasped. FIG. 2 is a basic concept diagram showing a process of creating an all-around overhead view image from four surrounding captured images from the back camera 1a, the left and right side cameras 1b and 1c, and the front camera 1d.

  FIG. 2 illustrates a procedure for generating an all-around bird's-eye view image displayed on a monitor for parking support in reverse parallel parking. In this example, it is assumed that a triangular cone is reflected as a three-dimensional object to be noted in the captured image of the back camera 1a. The rear shot image by the back camera 1a is projectively converted as a rear region image of the all-around overhead image from directly above the vehicle. Similarly, a captured image by the front camera, a left captured image by the left side camera 1b, and a right captured image by the front camera 1d are respectively projectively transformed as a left region image, a right region image, and a front region image of the all-around overhead image. Here, the projective transformation is performed using the mapping table 63. Since each map data value is different, a suitable map is set. However, each of these maps is created so as to bring about a projective transformation having a plane parallel to the road surface as a projection surface, and this projective transformation corresponds to the first projective transformation in the present invention.

  Of the four in-vehicle cameras, the rear shot image from the back camera 1a includes a three-dimensional object (triangular cone). The presence of this three-dimensional object is detected by the three-dimensional object detection function installed in this vehicle. The three-dimensional object information is output. Therefore, based on the three-dimensional object information, a region where the three-dimensional object is copied is extracted from the rear shot image as a three-dimensional object image. Projective transformation with a curved or curved surface (including spherical and cylindrical surfaces) whose projection side is substantially parallel to the road surface on the camera side and substantially perpendicular (perpendicular) to the road surface is extracted. By being performed on the three-dimensional object image, a three-dimensional object overhead image is generated. This projective transformation corresponds to the second projective transformation in the present invention. Again, this projective transformation is performed using the mapping table 63. In this second projective transformation, a projective transformation that makes it easier to see the three-dimensional object can be obtained according to the position, posture, and size of the detected three-dimensional object. Therefore, a plurality of mapping tables are prepared and the second projective transformation is performed. Therefore, it is convenient to select an optimal projective transformation. At this time, not only selecting a projection plane having a different three-dimensional shape, but also appropriately selecting a positional relationship between the projection plane and the three-dimensional object, it is possible to obtain an overhead image that makes it easier to see the three-dimensional object.

  Further, in the image of the rear area of the all-around bird's-eye view image to be combined with the generated three-dimensional bird's-eye view image (rear side bird's-eye view image segment), the three-dimensional object region calculated based on the three-dimensional object information in advance is subjected to the suppression process. A three-dimensional bird's-eye view image is synthesized with the rear bird's-eye view image segment including the three-dimensional object region subjected to the suppression process, and a rear bird's-eye view image segment synthesized as an all-around bird's-eye view image is output. The left overhead image segment, the right overhead image segment, and the front overhead image segment, including the rear overhead image segment, are synthesized, and an all-around overhead image finally displayed on the monitor is generated. Of course, the synthesis of the three-dimensional bird's-eye view image may be performed on a predetermined region of the all-around bird's-eye view image that has been generated previously.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, as illustrated in FIG. 2, the entire periphery is obtained from captured images and three-dimensional object detection information from four in-vehicle cameras, a back camera 1a, a front camera 1d, a left side camera 1b, and a right side camera 1c. An image generation device for creating a bird's-eye view image is incorporated in a vehicle for the construction of a vehicle periphery monitoring system. In the following description, these in-vehicle cameras 1a, 1b, 1c, and 1d may be simply referred to as the camera 1 as appropriate. When the vehicle periphery monitoring is operated, an image captured by the camera 1 or an overhead image generated using the captured image is displayed on the monitor.

  The camera 1 uses an image sensor such as a charge coupled device (CCD) or a CMOS image sensor (CIS) to take a two-dimensional image of 15 to 30 frames per second in time series, and digitally converts the captured image in real time. It is a digital camera that outputs. The camera 1 includes a wide angle lens. In particular, in the present embodiment, a viewing angle of 140 to 190 ° is ensured in the horizontal direction and the optical axis is installed in a vehicle having a depression angle of about 30 degrees.

  An ECU 20 that forms the core of the vehicle periphery monitoring system is installed inside the vehicle. As shown in FIG. 3, the ECU 20 includes a sensor input interface 23, a communication interface 70, and the like that transmit the signal input from the vehicle state detection sensor group as it is or evaluate and transfer it to the inside of the ECU 20, and process input information. A microprocessor and DSP (digital signal processor).

  The vehicle state detection sensor group connected to the sensor input interface 23 detects a driving operation or a vehicle running state. Although not shown, the vehicle state detection sensor group includes a steering sensor that measures the steering operation direction (steering direction) and the operation amount (steering amount), a shift position sensor that determines the shift position of the shift lever, and an accelerator pedal. An accelerator sensor that measures the amount of operation, a brake sensor 27 that detects the amount of operation of the brake pedal, a distance sensor that detects the travel distance of the host vehicle, and the like are included.

  The communication interface 70 used as an input / output interface employs an in-vehicle LAN as a data transmission line, and includes control units such as a monitor 21, a touch panel 21T, a power steering unit PS, a transmission mechanism T, and a brake device BK. Connected to allow data transmission. In addition, a speaker 22 is also provided as an audio information output device.

  In addition, the ECU 20 is provided with various functional units constructed in the form of hardware and / or software. As the functional unit particularly related to the present invention, the three-dimensional object detection unit of the present invention is provided. Three-dimensional object detection module 30, image processing module 50, display control unit 71, and sound processing module 72. The monitor display image generated by the image processing module 50 is converted into a video signal by the display control unit 71 and sent to the monitor 21. The voice guide generated by the voice processing module 72, an emergency warning sound, and the like are played by the speaker 22.

The three-dimensional object detection module 30 includes an ultrasonic evaluation unit 31 that performs detection of three-dimensional objects by evaluating detection signals from a plurality of ultrasonic sensors 3, and an image that performs three-dimensional object detection using captured images from the in-vehicle camera 1. A recognition unit 32 is included. The ultrasonic sensors 3 are disposed at both ends and the center of the front, rear, left side, and right side of the vehicle, and objects (obstacles) existing in the vicinity of the vehicle are reflected through the reflected waves from them. Can be detected. By processing the return time and amplitude of the reflected wave in each ultrasonic sensor 3, not only the distance from the vehicle to the object and the size of the object can be estimated, but also the detection results of all the ultrasonic sensors 3 are processed over time. Thus, it is also possible to estimate the movement of the object and the outer shape in the lateral direction. The image recognition unit 32 implements an object recognition algorithm that is known per se, and detects a three-dimensional object around the vehicle from an input captured image, particularly a captured image that is continuous over time.
For detecting a three-dimensional object, any one of the ultrasonic evaluation unit 31 and the image recognition unit 32 may be used, but the image recognition unit 32 excellent in detecting the form of the three-dimensional object and the distance to the three-dimensional object That is, more accurate three-dimensional object detection can be performed by providing both of the ultrasonic evaluation units 31 excellent in calculating the position of the three-dimensional object and performing a cooperative operation. Thereby, the three-dimensional object detection module 30 can output three-dimensional object information describing the position, posture, size, color tone, and the like of the detected three-dimensional object. Therefore, any one or a combination of the ultrasonic evaluation unit 31 and the image recognition unit 32 functions as a three-dimensional object detection unit in the present invention, and further, another three-dimensional object using a laser radar. It also includes a detection device.

  FIG. 4 shows a functional block diagram of the image processing module 50 of the ECU 20. The image processing module 50 has a function of generating an image such as an overhead image converted by projective transformation from a captured image acquired by the camera 1 that captures the periphery of the vehicle.

  The image processing module 50 includes a captured image memory 51, a preprocessing unit 52, an image generation unit 60, a three-dimensional object information acquisition unit 53, an image composition unit 55, and a frame memory 56. The captured image acquired by the camera 1 is developed in the captured image memory 51, and the preprocessing unit 52 adjusts the luminance balance, color balance, and the like between the captured images individually acquired by the camera 1. The three-dimensional object information acquisition unit 53 receives the three-dimensional object information output from the three-dimensional object detection module 30, and reads the position, posture, size, color tone, and the like of the three-dimensional object described in the three-dimensional object information.

The image generation unit 60 includes a normal image generation unit 61, a projective transformation selection unit 62, a mapping table 63, a surrounding bird's-eye view image generation unit 64, a three-dimensional object bird's-eye view image generation unit 65, a three-dimensional object extraction unit 66, and a three-dimensional object suppression unit 67. It is out. The normal image generation unit 61 adjusts the captured image to an image quality suitable for monitor display as a vehicle peripheral image as it is. The vehicle peripheral image displayed on the monitor may be one selected by the driver from images captured by the back camera 1a, left and right side cameras 1b and 1c, and the front camera 1d, or may be a combination of a plurality of captured images. .

The surrounding bird's-eye view image generation unit 64 performs the first projective transformation using the photographed image developed in the photographed image memory 51 and generates a surrounding bird's-eye view image. As the first projective transformation, a virtual viewpoint located above the vehicle and a projective transformation with a plane that coincides with or parallel to the road surface are set as default projections. It can be changed according to
The three-dimensional object overhead image generation unit 65 does not use the photographed image developed in the photographed image memory 51 as it is, but uses the three-dimensional object image obtained by extracting the image area of the three-dimensional object shown in the photographed image. The second projective transformation is performed to generate a three-dimensional object overhead image. The three-dimensional object image is extracted from the captured image by the three-dimensional object extraction unit 66 based on the position data read from the three-dimensional object information by the three-dimensional object information acquisition unit 53. For this reason, the three-dimensional object extraction unit 66 converts the position data of the three-dimensional object into the position coordinates of the captured image system, calculates a rough area where the three-dimensional object is copied, and uses the edge detection filter in the calculated area. It has a function of detecting a boundary line of a three-dimensional object estimated by using the above. The three-dimensional object extraction unit 66 extracts a region surrounded by the boundary line, or a region widened by a predetermined length from the boundary line, as a three-dimensional object image, and provides the three-dimensional object overhead image generation unit 65 with the three-dimensional object image. Further, as the second projective transformation executed by the three-dimensional object overhead image generation unit 65, a plurality of types of projective transformations are prepared. The projective transformation selection performed by the three-dimensional object overhead image generation unit 65 is performed by the projective transformation selection unit 62 based on the relative position of the three-dimensional object with respect to the camera 1, its posture, its size, and the like.

  In this embodiment, the projective transformation in the peripheral bird's-eye view image generation unit 64 and the three-dimensional object bird's-eye view image generation unit 65 is performed by map transformation using a mapping table, and thus various projection transformations used here are used. A mapping table is stored in advance so as to be selectable. Such an assembly composed of a plurality of mapping tables that can be selected and the individual mapping table is referred to herein as a mapping table 63 and is also schematically shown in FIG. Each mapping table constituting the mapping table 63 (hereinafter abbreviated as a map) can be constructed in various forms. Here, the pixel data of the photographed image and the pixels of the projective transformation image (usually an overhead image) are used. It is constructed as a map in which the correspondence relationship with the data is described, and the destination pixel coordinates in the overhead image are described in each pixel of the captured image of one frame. Therefore, a different map is applied for each on-vehicle camera.

  The projective transformation selection unit 62 estimates the shape of the three-dimensional object after the projective transformation based on the data read from the three-dimensional object information, and harmonizes with the surrounding bird's-eye view image as much as possible. And projective transformation that reduces distortion such as stretching. Further, the three-dimensional object bird's-eye view image generation unit 65 can perform color correction processing such as saturation increase and contour enhancement processing as necessary so that the image region showing the three-dimensional object is highlighted.

FIG. 5 schematically shows a state in which an appropriate map for the three-dimensional object overhead image generation unit 65 is selected by the projective transformation selection unit 62 and a final overhead image is generated.
When the three-dimensional object information is output, on the other hand, a three-dimensional object image is extracted from the photographed image based on the three-dimensional object data (position, posture, shape, etc.) described in the three-dimensional object information. On the other hand, a rule selection process is performed using such three-dimensional object data as input parameters, and a map selection command for instructing an optimal type of projective transformation is determined. Based on this map selection command, the application map is set in the three-dimensional object overhead image generation unit. That is, based on the three-dimensional object data, a projection surface shape that defines projective transformation and a conversion parameter that represents the relative position between the projection plane and the three-dimensional object (more precisely, the three-dimensional object image as the projective transformation source image) are set. It is the composition which becomes.
Here, as the projective transformation for reducing the distortion of the three-dimensional object, as schematically shown in FIG. 5, a plurality of projective transformations having different projection surface shapes and different relative positions of the three-dimensional object are prepared in the form of a map. . Examples of the shape of the projection surface include a cylindrical curved surface, a curved surface formed of a plane bent halfway, and a spherical curved surface. Of course, a vertical plane perpendicular to the road surface and an inclined plane with respect to the road surface can also be used as the projection surface. Also, different projection transformations can be realized by arranging each projection plane at a different relative position to the three-dimensional object. As a particularly suitable arrangement, the three-dimensional object is projected in an inclined surface area or a curved surface area rising from the road surface of the projection surface, so that the entire three-dimensional object projection image is projected on the projection surface parallel to the road surface. The strain that is extended during the generation is suppressed. For this reason, an arrangement in which the projection position of the road surface side end portion of the three-dimensional object coincides with the region position where the projection surface rises from the road surface is convenient. In that case, the position of the road surface side end portion of the three-dimensional object can be detected by the edge detection processing, and therefore the three-dimensional object is used by using the projection surface arranged so that this position matches the rising position of the projection surface from the road surface. It is preferable to generate a bird's-eye view image.
As the types of prepared projective transformations increase, the selection rules become complicated. Therefore, in such a case, a neural network is constructed with the three-dimensional object data (position, orientation, shape, etc.) described in the three-dimensional object information as input parameters and a map selection command as an output parameter, and is incorporated into the projective transformation selection unit 62. Good. When a map for projective transformation suitable for a three-dimensional object image is selected, a three-dimensional object overhead image is generated from the three-dimensional object image using the map.

  The three-dimensional object suppression unit 67 has a blend function in order to perform a suppression process so as not to stand out with respect to the region of the three-dimensional object image in the peripheral overhead image generated by the peripheral overhead image generation unit 64. A typical process of this blend function is a process of replacing the pixel values of the region of the three-dimensional object image based on the pixel values of the peripheral region of the region of the three-dimensional object image, for example, the right region and the left region. If a plurality of cameras are used, a process using a road projection image by a camera on which the three-dimensional object is not reflected, for example, the pixel value of the region of the three-dimensional object image with the pixel value of the road projection image. The replacement process is also effective as a blend process. Of course, other processing may be employed. Further, instead of the blending function with surrounding pixels, a configuration using blurring processing or saturation reduction processing may be used.

  The image compositing unit 55 generates a three-dimensional object overhead image generated by the three-dimensional object bird's-eye image generation unit 65 in a region of the three-dimensional object of the peripheral bird's-eye view image subjected to the three-dimensional object suppression process by the three-dimensional object suppression unit 67 as necessary. Superimpose and combine images. The synthesized peripheral overhead view image is transferred to the frame memory 56 as a display overhead view image and displayed on the monitor 21 via the display control unit 71.

Next, the flow of the overhead image display by the vehicle periphery monitoring system incorporating the image generating apparatus configured as described above will be described with reference to the flowchart of FIG.
When the bird's-eye view image display routine for vehicle periphery monitoring starts, first, the display type of the bird's-eye view image set manually or by default according to the driver's request is read (# 01). Here, the display type of the bird's-eye view image defines a photographed image and a virtual viewpoint position used when generating the surrounding bird's-eye view image, a layout on the monitor screen of the generated display bird's-eye view image, and the like. A map for the first projective transformation used in the peripheral bird's-eye view image generation unit 64 is set for each captured image of the in-vehicle camera 1 to be used according to the display type of the read bird's-eye view image (# 02). A captured image of the in-vehicle camera 1 is acquired (# 03). An overhead image segment is generated from each captured image using each set map (# 04). In addition to combining the generated overhead image segments, a vehicle overhead image (which may be an illustration or a symbol) set in advance at the position of the host vehicle is arranged, and a surrounding overhead image is generated (# 05).

  It is checked whether or not the three-dimensional object information is output from the three-dimensional object detection module 30 (# 06). If the three-dimensional object information is not output (# 06 No branch), it is assumed that there is no noticeable three-dimensional object in any photographed image, and the surrounding overhead image generated in step # 05 is used as it is. An overhead image is generated (# 14). The generated display bird's-eye view image is displayed on the monitor 21 (# 15), and unless there is an instruction to end this bird's-eye view image display routine (# 16 No branch), the process returns to step # 01 again to repeat this routine.

  If the three-dimensional object information is output in the check of step # 06 (# 06 Yes branch), it is considered that the three-dimensional object to be noticed is reflected in the photographed image, and first, the three-dimensional object detected from the three-dimensional object information is detected. Data such as position, orientation, and size are read (# 07). Based on the read data regarding the three-dimensional object, the image area of the three-dimensional object in the captured image is calculated (# 08), and the three-dimensional object image is extracted and temporarily stored in the memory (# 09). Further, a three-dimensional object suppression process that eliminates or weakens the presence of the three-dimensional object is performed on the image area of the three-dimensional object in the peripheral bird's-eye view image using the pixel values of the peripheral area (# 10). ).

  In the surrounding bird's-eye view image, depending on the height of the three-dimensional object from the road surface and the distance from the camera 1, the upper end of the three-dimensional object extending upward is shown extended. In order to reduce the distortion of the three-dimensional object in the overhead view image from the upper virtual viewpoint, only the three-dimensional object image extracted from the photographed image is generated using another projective transformation, that is, the second projective transformation. To do. For this purpose, first, an optimal map for the second projective transformation is selected and set from various forms as shown in FIG. 5 based on the solid object information (# 11). Next, the three-dimensional object bird's-eye view image generation unit 65 performs projective transformation of the three-dimensional object image using the set map, and generates a three-dimensional object bird's-eye image whose distortion is reduced compared to the surrounding bird's-eye image (# 12). At that time, if necessary, color correction processing for increasing the saturation of the three-dimensional object region in the three-dimensional object bird's-eye view image or shape clarification processing for enhancing the contour may be performed. Further, the three-dimensional object bird's-eye view image is superimposed and synthesized on the image area of the three-dimensional object in the peripheral bird's-eye view image (# 13). The final peripheral bird's-eye view image generated in this way is generated as a display bird's-eye view image (# 14) and displayed on the monitor 21 (# 15).

[Another embodiment]

(1) In the above-described embodiment, an example in which one notable solid object is captured in one of the captured images acquired from the four cameras 1 has been described. There may be a case where a single three-dimensional object to be copied is shown in a plurality of captured images. Therefore, in an embodiment in which a plurality of in-vehicle cameras are installed to capture the entire periphery of the vehicle, when a three-dimensional object is included in the captured images of two or more cameras, the three-dimensional object having a larger area It is advantageous if a three-dimensional object image having a region is extracted for generating a three-dimensional object overhead image.
(2) Two different three-dimensional objects may be handled as three-dimensional objects to be noted. Further, in this case, a three-dimensional object overhead image may be generated using projective transformation that varies depending on the three-dimensional object, and may be combined with the peripheral overhead image.
(3) In the above-described embodiment, as the three-dimensional object detection method, one or a combination of three-dimensional object detection using ultrasonic waves and three-dimensional object detection by image recognition has been presented. It is also within the scope of the present invention to use an object detection method such as a laser radar method or an infrared method.

  The present invention can be used for all systems that monitor a vehicle periphery using an overhead image.

30: Three-dimensional object detection module 21: Monitor 50: Image processing module 53: Three-dimensional object information acquisition unit 54: Virtual three-dimensional object selection unit 55: Image composition unit 60: Image generation unit 61: Normal image generation unit 62: Projection conversion selection unit 63: Mapping table 64: Surrounding bird's-eye view image generation unit 65: Three-dimensional object overhead image generation unit 66: Three-dimensional object extraction unit 67: Three-dimensional object suppression unit 71: Display control unit

Claims (6)

An image generation device that outputs a bird's-eye view image generated by projective transformation of a captured image acquired by an in-vehicle camera that captures a peripheral area of a vehicle from an upper virtual viewpoint as a display image,
A peripheral overhead image generation unit that generates a peripheral overhead image from the captured image using the first projective transformation;
A three-dimensional object detection unit that detects a three-dimensional object existing in the field of view of the in-vehicle camera and outputs three-dimensional object information including the position of the three-dimensional object;
A three-dimensional object extraction unit that extracts a three-dimensional object image that is an image region in which the three-dimensional object is captured from the captured image based on the three-dimensional object information;
A three-dimensional object overhead image generation unit that generates a three-dimensional object overhead image from the three-dimensional object image using a second projective transformation that reduces distortion of the three-dimensional object in the peripheral overhead image;
An image compositing unit that synthesizes the three-dimensional object overhead image with the peripheral overhead image based on the three-dimensional object information;
An image generation apparatus comprising:   The image generation apparatus according to claim 1, wherein the second projective transformation is a projective transformation in which the photographed image is projected from a photographing viewpoint onto a projection plane perpendicular to a road surface.   The conversion parameter of the second projective transformation, which is the shape of the projection surface and / or the relative position between the projection surface and the projected three-dimensional object, or both, is set based on the three-dimensional object information. Image generation device.   4. The image generation device according to claim 1, further comprising a three-dimensional object suppression unit that performs a suppression process so that a region of the three-dimensional object image in the peripheral overhead image is not conspicuous.   The image generation apparatus according to claim 1, wherein a color correction process is performed on the three-dimensional object overhead image so that the three-dimensional object overhead image is highlighted.   A plurality of the in-vehicle cameras are installed for photographing the entire periphery of the vehicle, and when the three-dimensional object is included in the captured images of two or more cameras, the three-dimensional object region having a smaller area is provided. The image generation apparatus according to claim 1, wherein an object image is extracted.

Images