JP4907883B2 - Vehicle periphery image display device and vehicle periphery image display method - Google Patents

Description

  The present invention relates to a vehicle periphery image display device and a vehicle periphery image display method, and more particularly to a vehicle periphery image display device and a vehicle periphery image display method that provide an image that captures the periphery of the vehicle and enables a driver to drive safely. About.

  Conventionally, various technologies for realizing safer driving have been developed by imaging the periphery of a vehicle or the like with a monitoring camera provided around the vehicle or the like and providing the captured image data to the driver. It has become widespread.

  For example, a blind monitor that supports left and right confirmations at intersections and T-junctions with poor visibility displays images of a plurality of surveillance cameras provided near the front end of the vehicle.

  Further, many side blind monitors attached to side mirrors or the like monitor the front from the vicinity of the left front wheel, and are effective in passing through narrow roads, avoiding obstacles, or shifting in width.

  Furthermore, a back guide monitor installed in the vicinity of the rear vehicle number plate is widely used to support garage entry and parallel parking.

  Many of the surveillance cameras used for these blind monitors and the like have a relatively small angle of view and select a surveillance camera that is directed in a desired direction for monitoring by switching manually or automatically as appropriate.

  As a technique for enlarging the angle of view of the monitoring camera, for example, Patent Document 1 discloses a technique related to a monitoring apparatus using a fisheye lens. Patent Document 1 uses a fisheye lens for one of the rear monitoring cameras to obtain an image with a wide field of view, and at the same time, converts the fisheye image into a normal image and converts it into an image that does not feel uncomfortable for the driver. The technique which eliminates the blind spot behind a vehicle by superimposing on a mirror image is disclosed.

In Patent Document 2, a plurality of monitoring cameras for monitoring the periphery of the vehicle are provided around the vehicle, and an image obtained from the plurality of monitoring cameras is converted and combined into an image viewed from a virtual viewpoint above the vehicle, for example. Technology is disclosed.
JP 2003-196645 A Japanese Patent No. 3300334

  The above-described blind monitor, side blind monitor, or back guide monitor can be realized with a relatively simple configuration, and thus is a widely used technology today.

  However, monitoring images based on these techniques are premised on appropriate selection and switching according to driving conditions, and the space to be monitored is limited. For example, when a blind monitor is used to monitor the front left / right direction at an intersection or T-shaped road with poor visibility, it is not possible to monitor a motorcycle or a passerby approaching the side of the vehicle.

  In addition, when the side blind monitor is selected, it is impossible to monitor an obstacle behind the vehicle, a passerby approaching the vehicle rear, or the like.

  The back guide monitor is effective for driving support for garage parking and parallel parking, but because it monitors the space relatively close to the vehicle behind the vehicle, the approaching vehicle at a certain distance from the vehicle is not always sufficiently monitored. I can't do it.

  As described above, the blind monitor or the like supports the driver by providing the driver with an image of a specific monitoring range in a specific driving situation, and the monitoring direction and the monitoring range are limited. It is only providing partial images around the vehicle.

  Patent Document 1 discloses an embodiment in which the angle of view is enlarged using a fisheye lens in a rear monitoring camera, but basically a specific monitoring direction is monitored in the same manner as the blind monitor and the like.

  On the other hand, the vehicle periphery image display device disclosed in Patent Document 2 is provided with a plurality of monitoring cameras for monitoring the periphery of the vehicle, and images obtained from the plurality of monitoring cameras are, for example, virtual above the vehicle. The image is converted and combined into a bird's-eye view image viewed from the viewpoint, and a wide range of images around the vehicle is provided as a single image without selection or switching operation. For this reason, it is said that the convenience for the user is improved for the driver or the like.

  In the vehicle periphery image display device disclosed in Patent Document 2, it is estimated that a normal camera with a standard lens is used. For this reason, in order to obtain a composite image of all directions (360 degrees) around the vehicle, it is an embodiment in which eight surveillance camera images are combined.

  By using a wide-angle lens such as a fisheye lens for the surveillance camera, it is considered that even with fewer surveillance cameras, for example, about 4 surveillance cameras, it is possible to monitor all directions (360 degrees) around the vehicle. .

  By the way, in order to accurately convert an image of an actual surveillance camera, for example, a surveillance camera mounted on a rearview mirror, into an overhead image viewed from a virtual viewpoint above the vehicle, three-dimensional information of the object is necessary. .

  However, since the image information obtained by an actual surveillance camera is obtained as two-dimensional information obtained by projecting a three-dimensional object (three-dimensional information) around the vehicle in two dimensions, one-dimensional information is missing. In principle, it is possible to obtain three-dimensional information by imaging the same object using a plurality of surveillance cameras. However, if the object covers the entire circumference of the vehicle, Since the number increases and the processing becomes complicated, it is not a realistic solution.

  Therefore, when a two-dimensional image of a surveillance camera mounted on a rearview mirror or the like is converted into an overhead image viewed from a virtual viewpoint above the vehicle, predetermined assumptions are made with respect to the three-dimensional shape (space model) of the object. Is often done.

  Specifically, an object having a three-dimensional shape is treated as a projection image (two-dimensional shape) projected on a plane such as a road surface (see Patent Document 2).

  As a result, the driver who sees the bird's-eye view image may misidentify a three-dimensional object on the road as a pattern on the road surface. If the driver can observe the bird's-eye view image with a margin, it is possible to recognize that the object is a three-dimensional object based on the experience of the driver from the shape and color of the object image. Let's go. However, in scenes where the vehicle periphery image display device is effective, it is often impossible to expect the driver to observe the overhead image with a margin, and whether or not the object displayed in the overhead image is a three-dimensional object. There is a demand for a function that can instantly determine whether or not.

  Also, when there is a three-dimensional object in the common field of view of two cameras with different line-of-sight directions, the positions projected from the two cameras on the road surface are different, so the parts other than the part in contact with the road surface are separated. Will result in a composite image.

  For example, if one pedestrian enters the common field of view of two cameras, a matching image can be obtained on the pedestrian's foot on the road surface, but the pedestrian's torso and head Since this part is synthesized as a separated image, there is a risk of misunderstanding as if there are two pedestrians.

  The technique disclosed in Patent Document 2 cannot easily distinguish between a three-dimensional object and a planar object. In addition, the separation phenomenon of the three-dimensional object existing in the common visual field area of the two cameras cannot be prevented.

  Moreover, even if it is possible to provide a bird's-eye view image that reflects the entire periphery of the vehicle, it cannot be said that sufficient safety can be provided to the driver. There is a need for a vehicle periphery image display device that can automatically grasp a driving situation, display optimal driving assistance information according to the driving situation in addition to a bird's-eye view image, and perform driving assistance in an integrated manner.

  The present invention has been made in view of the above circumstances, and generates a bird's-eye view image that can identify and appropriately display a three-dimensional object around the vehicle, automatically grasps the driving situation, and responds to the driving situation. Another object of the present invention is to provide a vehicle periphery image display device and a vehicle periphery image display method capable of displaying the optimum driving support information in addition to the overhead image and performing integrated driving support.

In order to solve the above problems, a vehicle periphery image display device according to the present invention converts a plurality of images obtained from a plurality of fisheye cameras that capture the periphery of the vehicle into normal images. A normalization processing unit that generates a normalized image by performing a normalization process, an overhead image generation unit that generates an overhead view image that overlooks the periphery of the vehicle from a virtual viewpoint above the vehicle, and driving the vehicle. A support image generating unit that generates a driving support image for supporting, an image combining unit that combines the driving support image and the overhead image, a display unit that displays the driving support image together with the overhead image, and the vehicle A three-dimensional object detection unit that detects a position or direction of a surrounding three-dimensional object, and the support image generation unit images the three-dimensional object according to the detected position or direction of the three-dimensional object. Normalized image Selected as the driving support image, the image synthesizing unit, said as the obstacle included in the overhead image and said driving support image do not overlap with each other, depending on the position of the obstacle in the bird's-eye view image The display position of the driving support image combined with the overhead image is dynamically determined .

In addition, in order to solve the above-described problem, the vehicle periphery image display method according to the present invention provides a plurality of images obtained from a plurality of fisheye cameras that capture images of the periphery of the vehicle. A step of normalizing the image to generate a normalized image, a step of generating an overhead view image for overlooking the periphery of the vehicle from a virtual viewpoint above the vehicle from the normalized image, and for supporting driving of the vehicle Generating a driving assistance image; synthesizing the driving assistance image and the overhead image; displaying the driving assistance image together with the overhead image; and detecting a position or direction of a three-dimensional object around the vehicle And normalizing imaging the three-dimensional object according to the detected position or direction of the three-dimensional object in the step of generating the driving assistance image. Select an image as the driving support image, wherein in the step of combining, as the obstacle included in the overhead image and said driving support image do not overlap each other, the position of the obstacle in the bird's-eye view image Accordingly, the display position of the driving support image combined with the overhead image is dynamically determined .

  According to the vehicle periphery image display device and the vehicle periphery image display method according to the present invention, a bird's-eye view image capable of identifying and appropriately displaying a three-dimensional object around the vehicle is generated and the driving situation is automatically grasped. In addition, the optimum driving support information corresponding to the driving situation can be displayed in addition to the bird's-eye view image and integrated driving support can be performed.

  Embodiments of a vehicle periphery image display device and a vehicle periphery image display method according to the present invention will be described with reference to the accompanying drawings.

(1) Concept and Basic Configuration of Vehicle Peripheral Image Display Device and Vehicle Peripheral Image Display Method FIGS. 1A and 1B use the vehicle peripheral image display device and the vehicle peripheral image display method according to the embodiment of the present invention. 2 is a diagram showing an overall concept when generating and displaying a peripheral image of a vehicle 100. FIG.

  As shown in FIG. 1A, a plurality of monitoring cameras (fisheye cameras) 200 are attached to appropriate portions on the outer periphery of the vehicle 100. The number and mounting positions of the monitoring cameras 200 are not particularly limited. For example, the front monitoring camera 200a is mounted near the front number plate of the vehicle 100, and the rear monitoring camera 200b is mounted near the rear number plate. A left side monitoring camera 200c is attached to the left side mirror, and a right side monitoring camera 200d is attached to the right side mirror.

  Each monitoring camera 200a, 200b, 200c, and 200d (hereinafter simply referred to as the monitoring camera 200 when referring to these generic names) is a peripheral area of the vehicle 100, that is, as shown in FIG. An image of each of the front area Aoa, the rear area Aob, the left side area Aoc, and the right side area Aod is taken.

  Although the angle of view of the monitoring camera 200 is not particularly limited, it is preferable that the angle of view is wide because the entire collection of the vehicle 100 can be covered with a small number of vehicles. For example, the entire range (range of 360 degrees in the horizontal direction) around the vehicle 100 can be monitored by four monitoring cameras 200 using fisheye lenses having a horizontal angle of view of about 180 degrees or more.

  When the lens of the monitoring camera 200 has a super wide angle of view such as a fisheye lens, the captured image is an image with distortion. In the vehicle peripheral image display device and the vehicle peripheral image display method according to the embodiment of the present invention, The image is converted into an image without distortion (hereinafter referred to as a normalized image) by appropriate distortion correction means.

  The normalized images obtained by correcting the distortions for the captured images of the surveillance cameras 200a, 200b, 200c, and 200d are the corresponding regions, that is, the front region Aoa, the rear region Aob, the left side region Aoc, and the right side region Aod. Each area is an image viewed from the viewpoint where each monitoring camera 200 is mounted.

  Each normalized image is converted from a virtual viewpoint P provided above the vehicle 100 into an image in which the periphery of the vehicle is viewed from above. This conversion can be performed using an image conversion matrix having appropriate parameters. Furthermore, by combining the individual overhead images, an overhead image Pdd in which the entire periphery of the vehicle 100 is overhead viewed from the virtual viewpoint P is generated.

  The bird's-eye view image Pdd generated in this way is displayed on, for example, an image display unit (display unit) 140 provided in the front part of the driver's seat and provided to the driver. It is possible to instantly inform the driver of the situation of the passerby and the like, and driving safety can be improved.

  Further, in the vehicle periphery image display device and the vehicle periphery image display method according to the embodiment of the present invention, when a three-dimensional object such as a passerby exists in the overhead image Pdd, the three-dimensional object image is identified and the driving is performed. The safety of driving can be further improved by presenting it easily to the user.

  In addition, integrated driving support can be provided by monitoring driving conditions and combining and displaying driving support information and image information corresponding to the driving conditions on the overhead image Pdd.

  FIG. 2 is a diagram showing a basic configuration of the vehicle periphery image display device 1 according to the embodiment of the present invention.

  The vehicle peripheral image display device 1 corrects distortion of an image captured by a fisheye lens, an image buffer 2 that temporarily stores image data input from each of the monitoring cameras 200a, 200b, 200c, and 200d including the fisheye lens. The normalization processing unit 3 that outputs the normalized images (corresponding to the number of the monitoring cameras 200), and converts the individual normalized images into an image viewed from the virtual viewpoint P, and further synthesizes the entire periphery of the vehicle 100 An overhead view image generation unit 4 that generates an overhead view image Pdd that is seen from above, and an image composition unit 6 that synthesizes the overhead view image Pdd with other images and driving support images and outputs them to the image display unit 140 are provided.

  Further, the vehicle peripheral image display device 1 identifies whether or not a three-dimensional object exists in the generated overhead image Pdd. If a three-dimensional object exists, the image of the three-dimensional object is output to the image composition unit 6. The three-dimensional object identification unit 5 is provided.

  Furthermore, the vehicle periphery image display device 1 is based on information from the vehicle sensor 110 that monitors the vehicle speed, the steering angle, and the like, and information from the three-dimensional object detection unit 120 that detects obstacles around the vehicle 100. A region-of-interest estimation unit 7 is provided for estimating a direction of interest and a region of interest that would be of interest. In addition to generating an image of a region of interest (partial image) by extracting and enlarging the estimated image of the region of interest from the normalized image, a plurality of normalized images are input from the normalization processing unit 3 and the image composition unit 6 A support image generation unit 8 for outputting to

  In addition, the vehicle periphery image display device 1 includes a trajectory calculation unit 9 that calculates the trajectory of the vehicle 100 from the vehicle speed, the steering angle, and the like, for example, the predicted trajectories at the four corners of the front, rear, left, and right during low-speed driving. And a guide line generation unit 10 that generates a driving support image such as a guide line indicating a safety margin area around the vehicle 100 and outputs it to the image composition unit 6.

  The operation of the vehicle periphery image display device 1 configured as described above will be described with reference to the detailed configuration of each component.

(2) Detailed Configuration and Operation of Overhead Image Generation Unit and Solid Object Identification Unit FIG. 3 is a diagram illustrating a detailed configuration of the overhead view image generation unit 4.

  The overhead image generation unit 4 includes an overhead image conversion unit 40a, 40b, 40c, and 40d, a conversion parameter generation unit 41, and an overhead image integration unit 42.

  The overhead image generation unit 4 receives a normalized image. The normalized image is, for example, an image obtained by correcting images captured by four monitoring cameras 200 including fisheye lenses and mounted around the vehicle 100 using the normalization processing unit 3.

  Individual normalized images are converted from images viewed from the viewpoint of each monitoring camera 200 into individual overhead images viewed from the virtual viewpoint P in the overhead image conversion units 40a, 40b, 40c, and 40d. This conversion can be performed based on conversion parameters generated by parameters relating to each monitoring camera 200 and parameters relating to the virtual viewpoint P.

  The parameters related to each monitoring camera 200 are the position information and the line-of-sight direction of each monitoring camera 200, and the parameters related to the virtual viewpoint P are the position information of the virtual viewpoint P, the line-of-sight direction viewed from the virtual viewpoint, the field-of-view range, and the like. These conversion parameters are generated by the conversion parameter generation unit 41.

  Individual overhead images, for example, four overhead images based on the four monitoring cameras 200, are integrated into one overhead image Pdd in the overhead image integration unit 42. The integrated bird's-eye view image Pdd is an image in which the entire periphery of the vehicle 100 is seen from the virtual viewpoint P.

  The generated overhead image Pdd is output to the three-dimensional object identification unit 5 and the image composition unit 6.

  The three-dimensional object identification unit 5 determines whether or not a three-dimensional object is included in the overhead image Pdd. If a three-dimensional object is included, the three-dimensional object is extracted and output to the image composition unit 6. It is provided as a purpose.

  Objects to be imaged by the monitoring camera 200 are not limited to flat objects (including road patterns and the like) but include three-dimensional objects such as obstacles installed on the road and passers-by passing on the road.

  However, as shown in FIG. 4, for example, when the left side monitoring camera 200c captures a three-dimensional object Ob (for example, an infant traveling on the road), the image obtained from the left side monitoring camera 200c represents the three-dimensional object Ob. Obtained as a projected image projected on the road. That is, the three-dimensional information of the three-dimensional object Ob is lost, and is captured as a two-dimensional object existing on the road surface.

  For this reason, even if this projection image is converted into an overhead image Pdd viewed from the virtual viewpoint P, it is obtained as an overhead image Pod projected on the road surface as shown in FIG. Of course, it is also possible in principle to acquire three-dimensional information by imaging the same object using a plurality of surveillance cameras, but when the object extends all around the vehicle, Since the number of surveillance cameras increases and the processing becomes complicated, it is difficult to say that this is a realistic solution.

  Therefore, in the embodiment of the present invention, when a two-dimensional image obtained by the monitoring camera 200 is converted into a bird's-eye view image viewed from a virtual viewpoint above the vehicle, the three-dimensional shape (space model) of the target object is converted. Thus, it is assumed that the three-dimensional object has a two-dimensional shape projected on the road surface. For this reason, the image of the three-dimensional object obtained by the overhead image Pdd is expressed as a shape projected on the road surface.

  As a result, the driver who views the bird's-eye view image may misidentify the three-dimensional object Ob on the road as a pattern on the road surface. If the driver can observe the overhead image Pdd with a margin, it is possible to recognize that the object Ob is a three-dimensional object based on the experience of the driver from the shape and color of the target image. Will. However, in a scene where the vehicle periphery image display device 1 is effective, it is often impossible to expect the driver to observe the overhead image Ob with a margin. For this reason, it is set as the structure which provides separately the function which determines whether the target object projected on the bird's-eye view image Pdd in the solid-object identification part 5 is a solid object.

  FIG. 5 shows a detailed configuration of the three-dimensional object identification unit 5.

  The three-dimensional object identification unit 5 stores, for example, one frame of the overhead image Pdd input from the overhead image generation unit 4 and delays the overhead image Pdd delayed by one frame time by 1 frame time. A frame shift image generation unit 51 is provided that shifts the moving distance that the vehicle moves during the frame time.

  Further, the difference image processing unit 53 for obtaining a difference image Pdf between the bird's-eye view image Pdd (N) ′ of the previous frame shifted and the bird's-eye view image Pdd (N) of the new frame, and the bird's-eye view of the difference image and the new frame A correlation processing unit 54 that performs a correlation process with the image Pdd (N) and extracts a three-dimensional object image;

  In addition, a frame movement amount calculation unit 52 that calculates a movement amount Df per frame time from the vehicle speed input from the vehicle sensor 110 and a predetermined frame time is provided.

  The operation of the three-dimensional object identification unit 5 configured as described above will be described with reference to FIG.

  First, the overhead image Pdd (N-1) of the (N-1) th frame is input and stored in the frame memory unit 50. The frame memory unit 50 delays the overhead image Pdd (N-1) by one frame time. In the bird's-eye view image Pdd (N-1), for example, the image Im1 in which the three-dimensional object Ob is projected onto the road surface, the image Ims1 of the stop line drawn on the road surface, and “slow attention” also drawn on the road surface The image of the caution sign Ims2 is included.

  On the other hand, the frame movement amount calculation unit 52 calculates the movement amount Df of the vehicle 100 at the frame time Tf from the vehicle speed V input from the vehicle speed sensor 110 and the predetermined frame time Tf.

  Next, the frame shift image generation unit 51 performs a shift process uniformly on the bird's-eye view image Pdd (N−1) based on the movement amount Df to generate a shift bird's-eye view image Pdd (N) ′. The shifted overhead image Pdd (N) 'is for predicting the overhead image Pdd (N) of the (N) th frame.

  On the other hand, in the (N) th frame, a new overhead image Pdd (N) actually captured is input. The overhead image Pdd (N) includes a projection image Im2 of the three-dimensional object Ob captured from the position of the monitoring camera 200 in the (N) th frame, an image Ims1 of the stop line drawn on the road surface, and also on the road surface 2 includes an image Ims2 of a caution sign “Drawing attention” drawn in FIG.

  For the stop line image Ims1 and the caution sign image Ims2, since the position of the vehicle has moved by Df (N-1), the movement amount Df with respect to the overhead image Pdd (N-1) obtained in the frame (N-1). A similar image that is shifted only by the distance is obtained.

  On the other hand, the projection image Im2 of the three-dimensional object Ob is obtained at the (N-1) th frame even if the amount of movement Df is shifted because the line-of-sight direction of the monitoring camera 200 changes as the vehicle moves. It is different from the obtained overhead image Pdd (N-1). For example, although the image of the foot part approaching the road surface has substantially the same shape, the position of the image of the torso leg part and the head away from the road surface does not match even if the position is shifted by the movement amount Df.

  The three-dimensional object identification unit 5 identifies the three-dimensional object Ob using the phenomenon principle as follows.

  First, the difference image processing unit 53 calculates a difference between the actual overhead image Pdd (N) of the (N) th frame and the predicted overhead image Pdd (N) 'to generate a difference image Pdf. In the difference image Pdf, two projection images Im1 and Im2 having different line-of-sight directions with respect to the three-dimensional object Ob are obtained. However, since the foot image is the same on both sides, it is deleted by the difference process.

  In addition, it is good also as a form which added the noise removal process which removes the noise component which may arise in the case of a difference process after a difference process.

  Next, correlation processing is performed between the difference image Pdf and the actual overhead image Pdd (N) of the (N) th frame to generate a correlation image Pcr. Only the three-dimensional object Imsp included in the actual overhead image Pdd (N) in the (N) th frame is identified and extracted by the correlation process (from the (N-1) th frame image included in the difference image Pdf). The predicted three-dimensional object image Im1 is removed). Strictly speaking, a trunk leg portion and a head portion apart from the road surface are identified and extracted as a three-dimensional object.

  In this way, the three-dimensional object Imsp included in the actual overhead image Pdd (N) of the (N) th frame is identified and extracted, and is output to the image composition unit 6 as the correlation image Pcr.

  The image composition unit 6 performs appropriate processing such as coloring to facilitate visual recognition of the three-dimensional image Imsp, and then superimposes it on the actual overhead image Pdd (N) of the (N) frame. And output to the image display unit 140.

  According to the three-dimensional object identification process of the three-dimensional object identification unit 5 according to the present embodiment, identification of a three-dimensional object that is important for supporting safe driving is performed using difference information (difference image Pdf) between frames of the overhead image Pdd. Based on this, it becomes possible to identify and extract by a simple method. In addition, the image of the identified three-dimensional object Imsp is colored to display it easily and visually so that the driver's attention is given, and collisions with obstacles and passers-by three-dimensional objects around the vehicle, etc. Can be avoided in advance.

  By the way, even if the three-dimensional object is identified by the above-described three-dimensional object identification process, there are the following problems in the image display of the overlapping region of the monitoring camera 200 and its neighboring region.

  FIG. 7 explains the first problem and the solution.

  FIG. 7A shows a situation where there is a three-dimensional object Ob such as an obstacle or a passerby in the left rear area of the vehicle 100. The four corner areas around the vehicle 100 are areas in which the same area can be overlapped and monitored by both of the monitoring cameras 200a, 200b, 200c, and 200d. When the three-dimensional object Ob exists in this overlapping area, for example, the overlapping area Aobc on the left rear side of the vehicle 100, the monitoring camera 200b and the monitoring camera 200c have different line-of-sight directions with respect to the three-dimensional object Ob. And the shape will be different.

  For this reason, as shown in FIG. 7B, when the overhead image Pdd ′ is generated by simply superimposing and synthesizing the individual overhead images, the three-dimensional object Ob is displayed separately in the overhead image Pdd ′. Will be.

  In the image display as it is, there is a possibility that even if there is only one three-dimensional object Ob, the driver is misunderstood as if there are two three-dimensional objects.

  Therefore, in the vehicle peripheral image display device 1 according to the present embodiment, in the overlapping area of the monitoring camera 200, for example, the overlapping area Aobc, an image obtained from one of the monitoring cameras 200, for example, the area Aob monitored by the monitoring camera 200b. It is assumed that only the image is selected and the overhead image Pdd is synthesized. That is, the overhead image Pdd is obtained by using the image obtained from the monitoring camera 200b (image of the area Aob including the overlapping area Aobc) and the image of the non-overlapping area Aoc ′ obtained by removing the overlapping area Aobc from the image obtained from the monitoring camera 200c. Is to be synthesized. Hereinafter, this process is referred to as an overlapping area selection process.

  As a result, as shown in FIG. 7C, the overhead image Pdd displaying the overlapping area Aobc is generated based only on the image obtained from the monitoring camera 200b, and therefore the same three-dimensional object Ob is separated. It is not displayed and misidentification for the driver can be avoided.

  In the present embodiment, the overhead image generation unit 4 performs an overlapping area selection process.

  In addition, when the vehicle is traveling substantially straight, it is preferable to preferentially select images obtained from the monitoring cameras 200c and 200d on the left and right sides. As a result, even when the three-dimensional object on the side of the vehicle moves from the non-overlapping area to the overlapping area, it is possible to continue monitoring with the same monitoring camera 200, and avoid sudden image changes due to switching of the monitoring cameras. it can. For the same reason, when the vehicle moves in the left-right direction, such as when the vehicle turns significantly, it is preferable to preferentially select images obtained from the front and rear monitoring cameras 200a and 200b. In addition, the monitoring camera 200 to be preferentially selected may be appropriately changed and selected by the user by a command input from the instruction input unit 130.

  FIG. 8 explains the second problem relating to the overlapping region and its neighboring region and the solution thereof.

  FIG. 8A shows a situation where two three-dimensional objects Ob1 and Ob2 such as obstacles and passers-by exist in the left rear region of the vehicle 100. FIG. The three-dimensional object Ob1 is within the field of view of the monitoring camera 200b but is not visible from the monitoring camera 200c. In addition, the three-dimensional object Ob2 is in a position that is within the field of view of the monitoring camera 200c but cannot be seen from the monitoring camera 200b.

  In such a situation, when the overhead image Pdd ″ is simply generated based on the images from the monitoring cameras 200b and 200c, an image as shown in FIG. 8B is obtained.

  Next, when the overlapping area selection process is performed on the bird's-eye view image Pdd ″, the bird's-eye view image Pdd ′ shown in FIG. 8C is obtained. The solid object Ob2 is within the field of view of the monitoring camera 200b. Therefore, the overhead image Pdd ′ of the overlapping area Aobc generated from the image of the monitoring camera 200b does not include the projection image of the three-dimensional object Ob2 on the road surface.

  On the other hand, for the image of the monitoring camera 200c, only the image of the non-overlapping area Aoc 'is used to generate the overhead image Pdd.

  For this reason, as shown in FIG. 8C, when the three-dimensional object Ob2 exists in the non-overlapping area Aoc ′ adjacent to the overlapping area Aobc, a part of the three-dimensional object Ob2 near the road surface, for example, traffic Only the image Imc ′ of the person's foot is displayed in the overhead image Pdd ′.

  As a result, the driver overlooks a part of the image Imc ′ of the three-dimensional object Ob2, and misidentifies that there is only one three-dimensional object even though there are two three-dimensional objects such as obstacles and passers-by. The fear increases.

  Therefore, in order to solve this problem, the vehicle peripheral image display device 1 according to the present embodiment performs a three-dimensional object extension display process in addition to the overlapping area selection process.

  The three-dimensional object extension display processing will be described with reference to the example of FIG. If the image is extended, the image Imcsp of the three-dimensional object Ob2 is superimposed on the overlap area Aobc after the overlap area selection process.

  Whether or not the image Imc is an image of a three-dimensional object is determined by the three-dimensional object identification unit 5. In addition, it is determined whether or not the image Imc of the three-dimensional object Ob2 identified and extracted by the three-dimensional object identification unit 5 extends from the non-overlapping area Aoc ′ to the overlapping area Aobc (hereinafter, this determination is referred to as extension determination). ) Is determined by the overhead image generation unit 4 based on the position information of the identified three-dimensional object. This extension determination need not be performed on the entire region of each overhead image. Since the size of the three-dimensional object is usually limited to a certain range, it is sufficient to perform it in the overlapping region and its neighboring region. For example, the determination is made in the determination region Ad illustrated in FIG. Thereby, processing time can be shortened compared with the case where it determines by all the area | regions of each bird's-eye view image, for example, all the area | regions of Aoc.

  In the extension determination, when it is determined that the image Imc of the three-dimensional object Ob2 extends from the non-overlapping area Aoc ′ to the overlapping area Aobc, the determined extended three-dimensional object image Imcsp is superimposed on the overhead image Pdd ′. By combining, a final overhead image Pdd is generated.

(3) Detailed Configuration and Operation of Region of Interest Estimation Unit and Image Composition Unit FIG. 9 is a diagram illustrating specific input signals and output signals to the region of interest estimation unit 7 of the vehicle peripheral image display device 1.

  The region-of-interest estimation unit 7 includes information indicating the vehicle speed from the vehicle speed sensor 111 and the traveling direction of the vehicle from the gear sensor 112, that is, “forward” or “reverse” from the steering angle sensor 113. Is inputted with the steering angle of the driving wheel. In addition, the yaw rate (angular velocity around the central axis of the vehicle toward the ground) may be input from the yaw rate sensor.

  Also, the position information (direction, distance, etc.) of the obstacle detected by the three-dimensional object detection unit 120 included in the vehicle 100 is input to the region of interest estimation unit 7.

  In addition, command information is input from the instruction input unit 130 included in the vehicle 100 via appropriate input means such as a microphone, a keyboard, a switch, and a touch panel.

  The region-of-interest estimation unit 7 automatically determines a driving situation from various information input from the vehicle sensor 110, estimates a direction (interesting direction) in which the driver will be interested from this driving situation, and The direction information is output to the image composition unit 6. The direction of interest is, for example, forward, backward, left side, or right side.

  Further, the region of interest estimation unit 7 estimates the region of the obstacle (region of interest) that the driver will be interested in from the position information of the obstacle input from the three-dimensional object detection unit 120, and the region of interest information is obtained. The image is output to the image composition unit 6. The region-of-interest information is, for example, an azimuth range indicating the region, a distance range from the vehicle, or the like.

  FIG. 10 shows a detailed configuration of the image composition unit 6.

  The image composition unit 6 includes an image selection unit 60, a display image determination unit 61, a display position determination unit 62, and a composition unit 62.

  In the image selection unit 60, based on an instruction from the display image determination unit 61, three types of images input to the image composition unit 6, that is, from the overhead image Pdd, the normalized image Pnn, and the region-of-interest image Pii are selected. Single or multiple images are selected.

  In the case where there are a plurality of images selected by the image selection unit 60, the synthesis unit 62 arranges and synthesizes the images at the designated position based on the instruction from the display position determination unit 62. Further, if necessary, the image is further combined with a driving support image such as a guide line and an expected trajectory generated by the guide line generation unit 10 and then output to the image display unit 140.

  The display image determination unit 61 specifies an image to be displayed based on the direction-of-interest information and the region-of-interest information output from the region-of-interest estimation unit 7 and instructs the image selection unit 60 to select the selected image.

  In addition, when there are a plurality of selection images, the display position determination unit 62 determines the display positions of the plurality of selection images based on the direction of interest information and the region of interest information output from the region of interest estimation unit 7. Decide to be easily visible. In addition, when it is determined that the visibility is deteriorated with the size of the original image, the composing unit 63 is instructed to reduce and display a specific image so as to facilitate visual recognition.

  FIG. 11A illustrates a specific method of estimating the direction of interest and the region of interest in the region of interest estimation unit 7 in the form of a table.

  FIG. 11B illustrates a specific display image selection method and position determination method in the image composition unit 6 in the form of a table.

  The region-of-interest estimation unit 7 determines a driving situation from information such as the input vehicle speed, gear, and steering angle. For example, as illustrated in the first row of the table of FIG. 11A, when the driving situation is determined to be “low speed / forward”, there is a possibility that the vehicle is heading to an intersection or T-junction with poor visibility. high. Therefore, it is estimated that the direction of interest of the driver is “forward”.

  Further, when the vehicle is cued at “low speed / forward”, the driver's greatest interest is the normalized image Pnn (front) captured by the front monitoring camera 200a. The image Pdd is not considered particularly necessary. For this reason, as illustrated in the first row of the table of FIG. 11B, in selecting the display image, only the normalized image in the direction of interest (front) is selected, and the overhead image Pdd and the region-of-interest image are not displayed. It is supposed to be.

  12B and 13B, when the driving state is determined to be “low speed / forward”, the forward normalized image Pnn (front) is selected and displayed on the image display unit 140. It shows a state. The difference between FIG. 12 and FIG. 13 is that the orientation of the vehicle 100 in the overhead image Pdd is horizontal display in FIG. 12 and vertical display in FIG. This selection can be made by, for example, a manual instruction input from the instruction input unit 130. Note that since the overhead image Pdd is not selected in “low speed / forward”, FIGS. 12B and 13B are the same image.

  When the driving situation is determined to be “low speed / reverse” from information such as the input vehicle speed, gear, steering angle, etc., as illustrated in the second row of the table of FIG. The “interesting direction” is determined as “backward”. In this case, the image composition unit 6 selects two types of images, the overhead image Pdd and the normalized image Pnn (rear) behind.

  FIGS. 12C and 13C show that when the driving state is determined to be “low speed / reverse”, the overhead image Pdd and the rear normalized image Pnn (rear) are selected and the image display unit 140 shows a state displayed on the screen. After the normalized image Pnn (rear) is reduced to an appropriate size, the normalized image Pnn (rear) is arranged at a position where there is no hindrance to the backward display of the overhead image Pdd. Note that the display positional relationship between the overhead image Pdd and the normalized image Pnn (rear) behind is considered to be normally unchanged, and is thus “fixed”.

  If the driving situation is determined to be "low speed / left turn" or "low speed / right turn" from the input information such as vehicle speed, gear, turning angle, etc., the driver's "direction of interest" Or “right side”. In this case, the image composition unit 6 selects the overhead image Pdd and the normalized image Pnn (left side) or the normalized image Pnn (right side). The mutual positional relationship is “fixed” (the figure is omitted).

  When the three-dimensional object detection unit 120 detects an obstacle and the position information of the obstacle is input to the region-of-interest estimation unit 7, the region-of-interest estimation unit 7 estimates that the driving situation is “obstacle avoidance” and It is estimated that the existing region (direction, distance, etc.) is the “region of interest” for the driver.

  On the other hand, the support image generation unit 8 selects a normalized image Pnn that covers the region of interest from the four normalized images Pnn based on the estimated position information of the region of interest, and further selects the normalized image Pnn as the normalized image Pnn. Enlarging the region of interest included. An image in which the region of interest is enlarged is called a region of interest image (partial image) Pii.

  When it is estimated that the driving situation is “obstacle avoidance”, the image composition unit 6 selects the overhead image Pdd and the region-of-interest image Pii.

  12D and 13D, when the driving situation is determined to be “obstacle avoidance”, the bird's-eye view image Pdd and the region-of-interest image Pii are selected and displayed on the image display unit 140. It shows a state.

  Since it is unknown where the obstacle is, the obstacle in the overhead image Pdd and the region-of-interest image Pii do not overlap each other according to the positional relationship between the obstacle detected by the three-dimensional object detection unit 120 and the vehicle. The display position of the region-of-interest image Pii needs to be dynamically changed.

  FIG. 14 shows the details of the processing content at the time of “obstacle avoidance”.

  First, the three-dimensional object detection unit 120 detects an obstacle and outputs the position information of the obstacle to the region-of-interest estimation unit 7 (step ST1).

  Next, the region-of-interest estimation unit 7 estimates a region with an obstacle as the “region of interest” of the driver (step ST2). Further, the support image generation unit 8 selects the normalized image Pnn generated from the camera image covering the obstacle, and further expands the region of interest to generate the region of interest image Pii (step ST3).

  The image composition unit 6 determines the position of the region of interest image Pii so that the position of the obstacle in the overhead view image Pdd and the region of interest image Pii do not overlap. If the overlap cannot be avoided by simply arranging, any one of the images is appropriately reduced (step ST4).

  After determining the position and size of both images, the two images are superimposed and combined and output to the image display unit 140.

  FIG. 15 illustrates the positional relationship between the overhead image Pdd and the region-of-interest image Pii when the positions of obstacles such as passersby are different. 15A, 15B, and 15C, the passerby in the overhead image Pdd and the region-of-interest image Pii are displayed so as not to overlap each other.

  In this way, by dynamically changing the position in the region-of-interest image Pii in accordance with the position of the obstacle and displaying it easily for the driver, the position of the obstacle is always appropriately determined for the driver. It will be possible to let them know and contribute to improving driving safety.

  Note that the direction of interest and the range of the region of interest estimated by the region of interest estimation unit 7, the selection of the display image selected by the image composition unit 6, the display position of the image, and the like are input from the instruction input unit 130. It is configured so that it can be set and changed as appropriate by a command.

(4) Operation of Guide Line etc. Generation Unit and Trajectory Calculation Unit FIG. 16 is a diagram illustrating an example of a driving support image generated by the guide line etc. generation unit 10 and synthesized by the image synthesis unit 6.

  When parking in a narrow space such as a parking lot, a safety margin around the vehicle is important in order to avoid collision with other vehicles. In order to assist the driver in a driving situation such as parking, in the vehicle peripheral image display device 1 according to the present embodiment, a guide line indicating a safety margin area Asf1 around the vehicle 100 as shown in FIG. Lg1 is displayed together with the bird's-eye view image Pdd of the vehicle 100.

  Further, when parking, etc., it is necessary to park in consideration of the clearance when the driving door 150 is opened. In the vehicle periphery image display device 1 according to the present embodiment, as shown in FIG. 16B, a guide line Lg2 indicating the clearance area Asf2 when the driving door 150 is opened is displayed together with the overhead image Pdd of the vehicle 100. It is said.

  By displaying these guide lines Lg1 and Lg2 together with the overhead image Pdd, the driver can park at an appropriate and safe position.

  In addition to this, the vehicle periphery image display device 1 according to the present embodiment displays the predicted locus of the vehicle as a driving assistance image. When parking in a narrow space or parallel parking, it is convenient for the driver if, for example, predicted trajectories at the four corners of the vehicle are displayed together with the overhead image Pdd. By sequentially updating the expected trajectory and the like according to the state of the steering wheel (steering angle), it becomes possible to support the driver's appropriate steering operation.

  The trajectory calculation unit 9 calculates the expected trajectories at the four corners of the vehicle 100. Although the calculation method of the predicted trajectory at the four corners is not particularly limited, for example, it can be calculated by the following procedure.

If the position of the center of gravity of the vehicle is the coordinates (X, Y), the position of the center of gravity (X, Y) can be expressed by Equation 1.

  Equation 1 is applied to a two-wheel (for example, front wheel) drive model when there is no tire slip, such as during low-speed operation.

  FIG. 17 is a diagram illustrating each parameter of Equation 1. As shown in FIG. 17, the center of gravity position (X, Y) of the vehicle is represented by a fixed ground coordinate system. V represents the traveling speed (vehicle speed) of the vehicle 100 and is output from the vehicle speed sensor 111. β represents the front wheel steering angle (steering angle) and is output from the steering angle sensor 113. R represents the yaw rate (angular velocity around the Z axis) and is output from the yaw rate sensor 114. θ represents the difference between the reference axis (x axis) of the coordinate system fixed to the vehicle (vehicle fixed coordinate system) expressed by (x, y) and the reference axis (X axis) of the ground fixed coordinate system. Yes. The above-mentioned V, β, and r are all values at the start of the predicted trajectory calculation, and are treated as constant values in the following predictive calculation of the trajectory.

By integrating equation 1, the following equation 2 is obtained.

Here, (Xo, Yo) is a seating amount indicating the center of gravity position of the vehicle 100 at the start of prediction calculation, and (X (t), Y (t)) represents a predicted locus of the center of gravity position. . Further, θ (t) can be calculated by the following equation 3.

  Note that by making the traveling direction of the vehicle coincide with the X axis, θo = Xo = Yo = 0.

  Although the predicted position of the center of gravity can be calculated based on Equations 2 and 3, the following calculation is further required to obtain the predicted trajectories at the four corners of the vehicle.

As shown in FIG. 18, each vector indicating the positions of the four corners from the center of gravity position of the vehicle toward the left front end, right front end, left rear end, and right rear end is expressed by Equation 4 in terms of vehicle fixed coordinates. Indicate.

Next, conversion is performed from the vehicle fixed coordinate system to the ground fixed coordinate system using the following Expression 5.

According to Equation 5, for example, the predicted trajectory ^W P _fl at the left front end expressed in ground fixed coordinates is expressed by Equation 6 below.

^W P _□□ (t) can be calculated in the same manner for the other three end points, and the expected trajectories at the four corners of the vehicle 100 can be calculated.

  FIG. 19 exemplifies the expected trajectories of the right front end and the left rear end calculated based on the calculations of Expressions 2 to 6.

  The calculated predicted trajectories at the four corners are imaged in the guide line etc. generating unit 10, synthesized with the overhead image Pdd in the image synthesizing unit 6, and then output to the image display unit 140.

  The display of the guide lines Lg1 and Lg2 generated by the guide line etc. generating unit 10 and the predicted trajectory of each of the four corners of the vehicle 100 can be selected as appropriate according to a command input from the instruction input unit 130. It is said.

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined.

The figure which shows the operation | movement concept of the vehicle periphery image display apparatus and vehicle periphery image display method which concern on this invention. The figure which shows the basic structural example of embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure which shows the detailed structural example of the bird's-eye view image generation part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure explaining the display concept of the solid object in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure which shows the detailed structural example of the solid-object identification part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure explaining the operation principle of the three-dimensional object identification part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure explaining the overlapping area selection process of the solid-object identification part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure explaining the extension determination process of the solid-object identification part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure which shows the example of the input-output signal of the region-of-interest estimation part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure which shows the detailed structural example of the image synthetic | combination part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure explaining the processing content example of the region-of-interest estimation part and image synthetic | combination part in embodiment of the vehicle periphery image display apparatus which concerns on this invention with a table | surface form. The 1st figure which shows the composite image example of the image synthetic | combination part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The 2nd figure which shows the example of a synthesized image of the image synthetic | combination part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure explaining the example of the determination method of the display position of the image synthetic | combination part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The 3rd figure which shows the composite image example of the image synthetic | combination part in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure which shows the example of a display of the driving assistance image in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The 1st figure explaining the calculation method of the vehicle expected locus | trajectory in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The 2nd figure explaining the calculation method of the vehicle expected locus | trajectory in embodiment of the vehicle periphery image display apparatus which concerns on this invention. The figure which shows the example of a display of the vehicle estimated locus | trajectory in embodiment of the vehicle periphery image display apparatus which concerns on this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Vehicle periphery image display apparatus 2 Image buffer 3 Normalization process part 4 Overhead image generation part 5 Solid object identification part 6 Image composition part 7 Region of interest estimation part 8 Support image generation part 9 Trajectory calculation part 10 Guide line etc. generation part 100 Vehicle 110 vehicle sensor 120 three-dimensional object detection unit 130 instruction input unit 140 image display unit 200 surveillance camera (fisheye camera)

Claims (2)

A normalization processing unit that normalizes a plurality of images obtained from a plurality of fisheye cameras that capture the periphery of the vehicle into normal images and generates a normalized image;
From the normalized image, an overhead image generation unit that generates an overhead image that overlooks the periphery of the vehicle from a virtual viewpoint above the vehicle;
A support image generating unit that generates a driving support image for supporting driving of the vehicle;
An image synthesis unit that synthesizes the driving support image and the overhead image;
A display unit for displaying the driving support image together with the overhead image;
A three-dimensional object detection unit that detects the position or direction of the three-dimensional object around the vehicle;
With
The support image generation unit selects, as the driving support image, the normalized image capturing the solid object according to the detected position or direction of the solid object ,
The image combining unit is combined with the overhead image according to the position of the obstacle in the overhead image so that the obstacle and the driving support image included in the overhead image do not overlap each other. Dynamically determine the display position of the driving assistance image,
The vehicle periphery image display apparatus characterized by the above. Normalizing a plurality of images obtained from a plurality of fisheye cameras that capture the periphery of the vehicle into normal images, and generating normalized images;
Generating an overhead image from which the surroundings of the vehicle are viewed from a virtual viewpoint above the vehicle, from the normalized image;
Generating a driving support image for supporting driving of the vehicle;
Synthesizing the driving support image and the overhead image;
Displaying the driving assistance image together with the overhead view image;
Detecting a position or direction of a three-dimensional object around the vehicle;
With
In the step of generating the driving support image, the normalized image capturing the solid object is selected as the driving support image according to the detected position or direction of the solid object ,
In the synthesizing step, the obstacle included in the overhead image is synthesized with the overhead image according to the position of the obstacle in the overhead image so that the obstacle and the driving support image do not overlap each other. Dynamically determine the display position of the driving assistance image,
A vehicle periphery image display method characterized by the above.

Images