JP6724821B2 - Overhead video generation device, overhead video generation system, overhead video generation method and program - Google Patents

Description

The present invention relates to a bird's-eye view video generation device, a bird's-eye view video generation system, a bird's-eye view video generation method, and a program.

A technique is known in which a plurality of cameras installed around the vehicle captures an image of the vehicle periphery, and a plurality of captured images are subjected to viewpoint conversion processing to display a combined bird's-eye view image on a monitor (see, for example, Patent Document 1). ). In this technique, when a predetermined condition regarding the vehicle speed and the obstacle is satisfied, the peripheral image displayed on the display unit is switched to another peripheral image having a different display range.

JP2012-227699A

When the display range of the peripheral image displayed on the display unit is switched and enlarged, the obstacle is displayed relatively small. As the displayed obstacle becomes smaller, it may be difficult to correctly recognize the obstacle. Since the displayed obstacle becomes smaller, it may be difficult to correctly grasp the positional relationship between the obstacle and the vehicle. Thus, there is room for improvement in displaying the detected obstacle in a bird's-eye view image.

The present invention has been made in view of the above, and an object of the present invention is to display the detected positional relationship between an obstacle and a vehicle so that the positional relationship can be intuitively grasped.

In order to solve the above-mentioned problems and achieve the object, a bird's-eye view video generation device according to the present invention includes a video data acquisition unit that acquires peripheral video data of a periphery of a vehicle and a failure detected in the periphery of the vehicle. An obstacle information acquisition unit that acquires position information of an object, and the peripheral image acquired by the image data acquisition unit based on the position information of the obstacle acquired by the obstacle information acquisition unit, the vehicle and the obstacle. The bird's-eye view image generation unit and the bird's-eye view image generation unit generate a bird's-eye view image by performing viewpoint conversion processing so that a virtual viewpoint is located on an extension line from the vehicle to the obstacle so that the object can be viewed from above. And a display control unit for displaying the generated bird's-eye view image on the display unit.

A bird's-eye view image generation system according to the present invention includes at least one of the above-mentioned bird's-eye view image generation device, a plurality of cameras that photograph the periphery of the vehicle, the display unit, and a detection unit that detects obstacles and outputs obstacle information. Equipped with.

A bird's-eye view video generation method according to the present invention includes a video data acquisition step of acquiring peripheral video data of the surroundings of a vehicle, an obstacle information acquisition step of acquiring positional information of an obstacle detected in the vicinity of the vehicle, Based on the position information of the obstacle acquired in the obstacle information acquisition step, the peripheral image acquired in the image data acquisition step can be viewed from the vehicle to the obstacle so that the vehicle and the obstacle can be overlooked. Including an overhead-view video generation step of generating an overhead-view video subjected to viewpoint conversion processing so that a virtual viewpoint is located on an extension line of the above, and a display control step of displaying the overhead-view video generated in the overhead-view video generation step on a display unit. ..

A program according to the present invention includes a video data acquisition step of acquiring peripheral video data of a surrounding area of a vehicle, an obstacle information acquisition step of acquiring positional information of an obstacle detected around the vehicle, and the obstacle. Based on the position information of the obstacle acquired in the information acquisition step, the peripheral image acquired in the image data acquisition step is an extension line from the vehicle to the obstacle so that the vehicle and the obstacle can be overlooked. A bird's-eye view image generation step that generates a bird's-eye view image that has been subjected to viewpoint conversion processing so that a virtual viewpoint is located in Let the computer run as.

According to the present invention, the positional relationship between the detected obstacle and the vehicle can be displayed so that the user can intuitively understand the positional relationship.

FIG. 1 is a block diagram showing a configuration example of an overhead view video generation system according to the first embodiment. FIG. 2 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view image generation system according to the first embodiment. FIG. 3 is a diagram showing an example of an overhead view image generated by the overhead view image generation system according to the first embodiment. FIG. 4 is a schematic diagram illustrating the position of the virtual viewpoint in the bird's-eye view image generation system according to the first embodiment. FIG. 5 is a schematic diagram illustrating the position of the virtual viewpoint in the overhead view image generation system according to the first embodiment. FIG. 6 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the first embodiment. FIG. 7 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the first embodiment. FIG. 8 is a flowchart showing the flow of processing in the bird's eye view video generation system according to the first embodiment. FIG. 9 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view image generation system according to the second embodiment. FIG. 10 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view image generation system according to the third embodiment. FIG. 11 is a schematic diagram illustrating another example of the position of the virtual viewpoint in the overhead view video generation system according to the third embodiment. FIG. 12 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the third embodiment. FIG. 13 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the fourth embodiment.

Embodiments of an overhead view video generation device 40, an overhead view video generation system 1, an overhead view video generation method, and a program according to the present invention will be described in detail below with reference to the accompanying drawings. The present invention is not limited to the embodiments described below.

In the following description, the front-rear direction is a direction parallel to the traveling direction when the vehicle goes straight, and the side from the driver's seat to the windshield is the front side in the front-rear direction, and the side from the windshield to the driver's seat is the front-rear direction. The direction is “after”. The front-back direction is the X-axis direction. The left-right direction is a direction that is horizontally orthogonal to the front-rear direction. Towards the windshield, the left hand side is "left" and the right hand side is "right". The left-right direction is the Y-axis direction. The up-down direction is a direction orthogonal to the front-rear direction and the left-right direction. The vertical direction is the Z-axis direction. Therefore, the front-back direction, the left-right direction, and the vertical direction are three-dimensionally orthogonal. Front-rear, left-right, and top-bottom in the following description are front-rear, left-right, and top-bottom when the bird's-eye view image generation system 1 is mounted on a vehicle.

[First embodiment]
FIG. 1 is a block diagram showing a configuration example of an overhead view video generation system according to the first embodiment. The bird's-eye view image generation system 1 is mounted on a vehicle V. The bird's-eye view image generation system 1 may be a device that is portable and usable in the vehicle V, in addition to the device mounted on the vehicle V.

The overhead view video generation system 1 will be described with reference to FIG. The overhead view image generation system 1 includes a front camera 11, a rear camera 12, a left side camera 13, a right side camera 14, a sensor group (obstacle sensor) 20, a display panel (display unit) 30, and an overhead view image. And a generator 40.

The front camera 11, the rear camera 12, the left side camera 13, and the right side camera 14 will be described with reference to FIG. FIG. 2 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view image generation system according to the first embodiment. The front camera 11 is arranged in front of the vehicle V and takes an image of the periphery around the front of the vehicle V. The front camera 11 shoots a shooting range A1 of about 180°, for example. The front camera 11 outputs the captured video to the video data acquisition unit 42 of the overhead video generation device 40.

The rear camera 12 is arranged behind the vehicle V and photographs the periphery of the rear of the vehicle V as a center. The rear camera 12 shoots a shooting range A2 of about 180°, for example. The rear camera 12 outputs the captured video to the video data acquisition unit 42 of the overhead view video generation device 40.

The left side camera 13 is arranged on the left side of the vehicle V, and photographs the periphery of the left side of the vehicle V as a center. The left camera 13 shoots a shooting range A3 of about 180°, for example. The left camera 13 outputs the captured video to the video data acquisition unit 42 of the overhead view video generation device 40.

The right side camera 14 is arranged on the right side of the vehicle V and takes an image of the periphery around the right side of the vehicle V. The right camera 14 shoots a shooting range A4 (not shown) of about 180°, for example. The right camera 14 outputs the captured video to the video data acquisition unit 42 of the overhead view video generation device 40.

The front camera 11, the rear camera 12, the left-side camera 13, and the right-side camera 14 capture images in all directions of the vehicle V.

Returning to FIG. 1, the sensor group 20 is a detection unit that detects an obstacle Q around the vehicle V. The sensor group 20 can detect an obstacle Q in a range including the display range A of the overhead view image. In the present embodiment, the sensor group 20 includes a front sensor, a rear sensor, a left side sensor, and a right side sensor. Depending on the sensing method, the sensor group 20 is capable of sensing from a distance of several tens to several hundreds of meters, but when used for this purpose, it detects an obstacle Q at a distance of approximately 5 m from the vehicle V. For the sensor group 20, for example, various methods such as a combination of a plurality of methods of sensors such as an infrared sensor, an ultrasonic sensor, a millimeter wave sensor, and a sensor based on image recognition can be applied.

The front sensor is arranged in front of the vehicle V and detects an obstacle Q existing in a range centered on the front of the vehicle V. The front sensor detects an object having a height from the ground that may come into contact with the vehicle V when the vehicle V is moving forward. The front sensor detects an obstacle Q at a distance of about 5 m from the vehicle V, for example. The detection range of the front sensor overlaps the shooting range A1 of the front camera 11. The detection range of the front sensor may overlap a part of the detection range of the left side sensor and the right side sensor. The front sensor is composed of a combination of a plurality of sensors. As a result, the direction of the obstacle Q is subdivided and detected. The front sensor outputs the detected obstacle information of the obstacle Q to the obstacle information acquisition unit 43 of the overhead view image generation device 40.

The rear sensor is arranged behind the vehicle V and detects an obstacle Q existing in a range centered on the rear of the vehicle V. The rear sensor detects an object having a height from the ground that may come into contact with the vehicle V when the vehicle V is moving backward. The rear sensor detects an obstacle Q at a distance of, for example, about 5 m from the vehicle V. The detection range of the rear sensor overlaps the shooting range A2 of the rear camera 12. The detection range of the rear sensor may overlap a part of the detection range of the left side sensor and the right side sensor. The rear sensor is composed of a combination of a plurality of sensors. As a result, the direction of the obstacle Q is subdivided and detected. The rear sensor outputs the obstacle information of the detected obstacle Q to the obstacle information acquisition unit 43 of the overhead view image generation device 40.

The left side sensor is arranged on the left side of the vehicle V and detects an obstacle Q existing in a range centered on the left side of the vehicle V. The left side sensor detects an object having a height from the ground that may come into contact with the vehicle V when the vehicle V is moving forward or backward while steering. The left side sensor detects, for example, an obstacle Q at a distance of about 5 m from the vehicle V. The detection range of the left side sensor overlaps with the shooting range A3 of the left side camera 13. The detection range of the left side sensor may overlap a part of the detection range of the front sensor and the rear sensor. The left side sensor is composed of a combination of a plurality of sensors. As a result, the direction of the obstacle Q is subdivided and detected. The left side sensor outputs the obstacle information of the detected obstacle Q to the obstacle information acquisition unit 43 of the overhead view image generation device 40.

The right side sensor is arranged on the right side of the vehicle V and detects an obstacle Q existing in a range centered on the right side of the vehicle V. The right side sensor detects an object having a height from the ground that may come into contact with the vehicle V when the vehicle V is moving forward or backward while steering. The right side sensor detects an obstacle Q at a distance of about 5 m from the vehicle V, for example. The detection range of the right side sensor overlaps the shooting range A4 of the right side camera 14. The detection range of the right side sensor may overlap a part of the detection range of the front sensor and the rear sensor. The right side sensor is composed of a combination of a plurality of sensors. As a result, the direction of the obstacle Q is subdivided and detected. The right side sensor outputs the obstacle information of the detected obstacle Q to the obstacle information acquisition unit 43 of the overhead view image generation device 40.

The display panel 30 is a display including, for example, a liquid crystal display (LCD: Liquid Crystal Display) or an organic EL (Organic Electro-Luminescence) display. The display panel 30 displays the overhead view image 100 (see FIG. 3) based on the video signal output from the overhead view video generation device 40 of the overhead view video generation system 1. The display panel 30 may be dedicated to the bird's-eye view image generation system 1 or may be used together with another system including a navigation system, for example. The display panel 30 is arranged at a position where it can be easily viewed by the driver.

When the shape of the display panel 30 is a horizontally long rectangle, the display panel 30 may be divided into a plurality of display ranges. For example, the display panel 30 has a display range for displaying the bird's-eye view image 100 and a display range for displaying a navigation screen or an audio screen, which is arranged beside the display range of the bird's-eye view image 100. The display range in which the overhead view image 100 is displayed is a vertically long rectangular shape.

The overhead view video generation device 40 includes a control unit 41 and a storage unit 49.

The control unit 41 is, for example, an arithmetic processing unit including a CPU (Central Processing Unit) and the like. The control unit 41 loads the program stored in the storage unit 49 into the memory and executes the instructions included in the program. The control unit 41 includes a video data acquisition unit 42, an obstacle information acquisition unit 43, a vehicle information acquisition unit 44, an overhead view video generation unit 45, and a display control unit 48.

The video data acquisition unit 42 acquires peripheral video data of the surroundings of the vehicle V. More specifically, the video data acquisition unit 42 acquires peripheral video data output by the front camera 11, the rear camera 12, the left side camera 13, and the right side camera 14. The video data acquisition unit 42 outputs the acquired peripheral video data to the overhead view video generation unit 45.

The obstacle information acquisition unit 43 acquires the obstacle information of the obstacle Q detected around the vehicle V and specifies the position of the detected obstacle Q on the bird's-eye view image. More specifically, the obstacle information acquisition unit 43 acquires the obstacle information output by the sensor group 20. The obstacle information acquisition unit 43 subdivides and acquires the direction of the obstacle Q based on the mounting position of the sensor that outputs the obstacle information and the detection range of the obstacle by the sensor. The obstacle information acquisition unit 43 acquires obstacle information including the detected distance to the obstacle Q. The obstacle information acquisition unit 43 specifies the position of the obstacle Q on the overhead view image based on the acquired direction and distance of the obstacle Q.

The vehicle information acquisition unit 44 acquires vehicle information that is a trigger for displaying the bird's-eye view image 100, such as gear operation information of the vehicle V, from a CAN (Controller Area Network) or various sensors that sense the state of the vehicle V. .. In the present embodiment, the vehicle information includes information indicating the traveling direction of the vehicle V. The vehicle information acquisition unit 44 outputs the acquired vehicle information to the overhead view video generation unit 45.

The bird's-eye view video generation unit 45 generates the bird's-eye view video 100 with the virtual viewpoint P above the vehicle V by performing viewpoint conversion processing and combining the peripheral video data acquired by the video data acquisition unit 42.

The virtual viewpoint P will be described with reference to FIG. The virtual viewpoint P is located above the center of the vehicle V. The virtual viewpoint P is a viewpoint looking down from directly above the center of the vehicle V. The center of the vehicle V is the center of the vehicle V in the vehicle width direction and the center of the vehicle V in the front-rear direction. The position directly above the vehicle V is a position on the normal line of the reference plane of the vehicle V. The reference surface is a plane that is horizontal to the road surface when the vehicle V is located on a horizontal and flat road surface. The position of the virtual viewpoint P is (x, y, z).

The generated bird's-eye view image 100 will be described with reference to FIG. FIG. 3 is a diagram showing an example of an overhead view image generated by the overhead view image generation system according to the first embodiment. The bird's-eye view image 100 synthesizes and displays the peripheral image of the display range A as the virtual viewpoint P and the image obtained by the viewpoint conversion processing. The overhead view image 100 includes a front image 101, a rear image 102, a left side image 103, and a right side image 104. The display area of the front image 101 and the display area of the rear image 102 are the same size. The display area of the left side image 103 and the display area of the right side image 104 are the same size. At the center of the bird's-eye view image 100, a vehicle icon 200 indicating the vehicle V is displayed. The host vehicle icon 200 shows a form in which the vehicle V is viewed from directly above.

In FIG. 3, a combination boundary B1 between the front image 101 and the left side image 103, a combination boundary B2 between the front image 101 and the right side image 104, a combination boundary B3 between the rear image 102 and the left side image 103, and Although an oblique broken line showing the composite boundary B4 of the rear image 102 and the right side image 104 is shown for explanation, the broken line is not displayed in the overhead view image 100 actually displayed on the display panel 30. May not be displayed. The same applies to the other figures. In the following description, when there is no particular need to distinguish between the composite boundary B1, the composite boundary B2, the composite boundary B3, and the composite boundary B4, the composite boundary B will be described.

The bird's-eye view image generation unit 45, based on the position information of the obstacle Q acquired by the obstacle information acquisition unit 43, when there is an obstacle satisfying a predetermined condition, sets the surrounding image acquired by the image data acquisition unit 42 to the vehicle. The viewpoint conversion process is performed so that the virtual viewpoint PF is located apart from at least one of the vehicle width direction and the front-rear direction from the virtual viewpoint P located directly above the vehicle V so that the V and the obstacle Q can be overlooked. Generate a bird's-eye view image that is synthesized by going.

The predetermined condition is a condition for detecting the obstacle Q. In the present embodiment, the predetermined condition is that the obstacle Q is located in the traveling direction of the vehicle V. When the obstacle Q is located in the traveling direction of the vehicle V, it is determined that the predetermined condition is satisfied, and the obstacle Q is detected. For example, when the vehicle V is moving backward and the obstacle Q is located behind the front end of the vehicle V, it is determined that the predetermined condition is satisfied, and the obstacle Q is detected. More specifically, when the vehicle V is moving backward, the control unit 41 sets a predetermined condition when a sensor in the sensor group 20 that detects an obstacle Q behind the front end of the vehicle V detects the obstacle Q. It is determined that the condition is satisfied, and the obstacle Q is detected. In this embodiment, a case where the vehicle V is moving backward will be described.

The bird's-eye view video generation unit 45 includes a viewpoint conversion processing unit 451, a cutout processing unit 452, and a synthesis processing unit 453.

The viewpoint conversion processing unit 451 performs viewpoint conversion processing on the peripheral video data acquired by the video data acquisition unit 42 so as to look down the vehicle V from the upper virtual viewpoint P. More specifically, the viewpoint conversion processing unit 451 generates a viewpoint-converted image based on the peripheral image data captured by the front camera 11, the rear camera 12, the left side camera 13, and the right side camera 14. The viewpoint conversion processing method may be any known method and is not limited. The viewpoint conversion processing unit 451 outputs the peripheral video data subjected to the viewpoint conversion processing to the cutout processing unit 452.

Based on the position information of the obstacle Q acquired by the obstacle information acquisition unit 43, the viewpoint conversion processing unit 451 provides a bird's-eye view of the vehicle V and the obstacle Q when an obstacle satisfying a predetermined condition exists. From the virtual viewpoint P located directly above V, viewpoint conversion processing is performed so that the virtual viewpoint PF is located at a distance in at least either the vehicle width direction or the front-rear direction, and an image is generated. More specifically, the viewpoint conversion processing unit 451 determines the position of the virtual viewpoint so that the relative positional relationship between the vehicle V and the obstacle Q can be recognized when the obstacle Q located in the traveling direction of the vehicle V exists. An image that has undergone the viewpoint conversion processing is generated as a virtual viewpoint PF separated from the virtual viewpoint P. The viewpoint conversion processing unit 451 outputs the peripheral video data subjected to the viewpoint conversion processing to the cutout processing unit 452.

To be able to recognize the relative positional relationship between the vehicle V and the obstacle Q means that the vehicle V and the obstacle Q are represented without overlapping in the overhead view image. Furthermore, that the relative positional relationship between the vehicle V and the obstacle Q can be recognized means that when the vehicle V or the obstacle Q moves, the relative distance between the vehicle V and the obstacle Q changes. It means that the relative positional relationship between the vehicle V and the obstacle Q in the overhead view image changes.

The virtual viewpoint PF is preferably located at a position where the angle of view of the bird's-eye view image in the direction in which the obstacle is detected is approximately the same as the angle of view of the bird's-eye view image 100 of the virtual viewpoint P. The bird's-eye view image 100 of the virtual viewpoint P and the bird's-eye view image in which the virtual viewpoint is moved to the virtual viewpoint PA use the same image, but use an area on the appropriate virtual projection plane corresponding to the position of the virtual viewpoint as the bird's-eye view image. .. Therefore, when there is a large change in the angle of view of the bird's-eye view image in the direction in which the obstacle is detected, it is easy to grasp the position of the obstacle on the bird's-eye view image when the virtual viewpoint position is changed. Specifically, when the virtual viewpoint PA is behind the vehicle V, the rear image in the direction in which the virtual viewpoint PA is located has the same angle as the cut-out angle of view of the rear image 102 that is the virtual viewpoint P. The front image, which is in the opposite direction across the vehicle V, has a wider angle of view in front of the vehicle than the front image 101 which is the virtual viewpoint P. By doing so, it is possible to generate a bird's-eye view image that looks down from the virtual viewpoint PA.

The position of the virtual viewpoint PF separated from the virtual viewpoint P in the vehicle width direction or the front-rear direction is located on a curved surface centered on the vehicle V as shown in FIGS. 4 and 5. FIG. 4 is a schematic diagram illustrating the position of the virtual viewpoint in the bird's-eye view image generation system according to the first embodiment. FIG. 5 is a schematic diagram illustrating the position of the virtual viewpoint in the overhead view image generation system according to the first embodiment. Since the angle of view of the bird's-eye view image having the virtual viewpoint PF on such a curved surface does not change from the angle of view of the bird's-eye view image 100 having the virtual viewpoint P, the size of the obstacle Q displayed in the bird's-eye view image does not change. Therefore, even if the position of the virtual viewpoint is changed, a bird's-eye view image is generated that makes it easy to intuitively grasp the relative positional relationship between the obstacle Q and the vehicle V.

The virtual viewpoint PF is preferably located farther from the center of the vehicle V than the distance between the center of the vehicle V and the obstacle Q. Thereby, a bird's-eye view image in which the relative positional relationship between the obstacle Q and the vehicle V is easily recognized is generated.

In the present embodiment, as shown in FIGS. 6 and 7, when the obstacle Q located to the left rear of the vehicle V is detected, the viewpoint conversion processing unit 451 extends an extension line L61 from the vehicle V to the obstacle Q. An image subjected to the viewpoint conversion processing is generated as the virtual viewpoint PF located above. FIG. 6 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the first embodiment. FIG. 7 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the first embodiment. The position of the virtual viewpoint PF is represented by (xF, yF, zF).

The virtual viewpoint PF will be described. In the present embodiment, the virtual viewpoint PF is located on the left rear of the vehicle V. The virtual viewpoint PF is located to the left rear of the obstacle Q when viewed in the Z-axis direction. The virtual viewpoint PF is located to the left rear of the virtual viewpoint P when viewed in the Z-axis direction. The virtual viewpoint PF is farther from the center of the vehicle V than the distance between the center of the vehicle V and the obstacle Q. The virtual viewpoint PF is a viewpoint that looks down the vehicle V obliquely from the left rear.

The cutout processing unit 452 performs a cutout process of cutting out a predetermined range of video from the peripheral video data subjected to the viewpoint conversion processing. The cutout processing unit 452 cuts out the front cutout range from the peripheral image data from the front camera 11 that has undergone the viewpoint conversion processing. The cutout processing unit 452 cuts out the rear cutout range from the peripheral image data from the rear camera 12 that has undergone the viewpoint conversion processing. The cutout processing unit 452 cuts out the left-side cutout range from the peripheral image data from the left-sided camera 13 that has undergone the viewpoint conversion processing. The cutout processing unit 452 cuts out the right-side cutout range from the peripheral image data from the right-sided camera 14 that has been subjected to the viewpoint conversion processing. The cutout processing unit 452 outputs the video data of the video subjected to the cutout processing to the synthesis processing unit 453.

The front cutout range is a range in which the front image 101 is cut out from the peripheral image data that has undergone the viewpoint conversion process. The front cut-out range is a range from the front end of the vehicle V to the front. The rear cutout range is a range in which the rear image 102 is cut out from the peripheral image data that has undergone the viewpoint conversion process. The rear cutout range is a range behind the rear end of the vehicle V. The left side cutout range is a range in which the left side image 103 is cut out from the peripheral image data that has undergone the viewpoint conversion processing. The left side cut-out range is a range from the left side portion of the vehicle V to the left side. The right side cutout range is a range in which the right side image 104 is cut out from the peripheral image data that has undergone the viewpoint conversion processing. The right side cutout range is a range from the right side of the vehicle V to the right side.

The combining processing unit 453 combines the plurality of videos cut out by the cutting processing unit 452 to generate an overhead view video. The synthesis processing unit 453 outputs the generated bird's-eye view video to the display control unit 48.

When synthesizing the images on which the viewpoint conversion processing has been performed, the synthesizing processing unit 453 synthesizes the own vehicle icon imitating the form of the own vehicle viewed from the virtual viewpoint in the central portion of the synthesized bird's-eye view image. The own vehicle icon when the virtual viewpoint position is the virtual viewpoint P when looking down from directly above the vehicle V has a form as if the own vehicle was viewed from directly above like the own vehicle icon 200 shown in FIG. .. On the other hand, when the virtual viewpoint position is set to a position separated from the virtual viewpoint P, it is preferable that the host vehicle be viewed obliquely from the virtual viewpoint. In such a form, it is natural that the bird's-eye view image on the opposite side of the own vehicle from the virtual viewpoint is displayed such that a range near the own vehicle is shaded by the own vehicle icon. Even in such a case, it is preferable that the host vehicle icon has a display form in which a blind spot due to the host vehicle icon does not occur. For example, the own vehicle icon may be semi-transparent. For example, the own vehicle icon may have a frame shape indicating the outer shape.

Returning to FIG. 1, the display control unit 48 causes the display panel 30 to display the overhead view image.

The storage unit 49 stores data required for various processes in the bird's-eye view video generation device 40 and various process results. The storage unit 49 is a semiconductor memory device such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk.

Next, the flow of processing in the bird's-eye view video generation device 40 of the bird's-eye view video generation system 1 will be described with reference to FIG. FIG. 8 is a flowchart showing the flow of processing in the bird's eye view video generation system according to the first embodiment.

When the bird's-eye view video generation system 1 is activated, the control unit 41 causes the video data acquisition unit 42 to acquire peripheral video data. The control unit 41 causes the obstacle information acquisition unit 43 to acquire obstacle information.

The control unit 41 determines whether or not to start the overhead view image display (step S11). In the present embodiment, the control unit 41 determines whether or not to start the bird's-eye view image display based on the presence/absence of the backward trigger. The backward trigger means that the shift position is set to "reverse", for example. Alternatively, the backward trigger means that the traveling direction of the vehicle V is behind the vehicle V. When there is no backward trigger, the control unit 41 determines not to start the bird's eye view image display (No in step S11), and executes the process of step S11 again. When there is a backward trigger, the control unit 41 determines to start the bird's-eye view image display (Yes in step S11), and proceeds to step S12. The trigger for starting the overhead image display is not limited to the backward trigger, and any trigger such as user operation, obstacle detection result, and stop may be applied.

The control unit 41 determines whether or not the obstacle Q is detected (step S12). More specifically, the control unit 41 determines whether or not the obstacle information acquisition unit 43 has acquired the obstacle information of the obstacle Q satisfying a predetermined condition. When determining that the obstacle information of the obstacle Q satisfying the predetermined condition is acquired (Yes in step S12), the control unit 41 proceeds to step S13. When determining that the obstacle information of the obstacle Q satisfying the predetermined condition is not acquired (No in step S12), the control unit 41 proceeds to step S17.

The control unit 41 acquires the position information of the obstacle Q with respect to the vehicle V (step S13). More specifically, the control unit 41 causes the obstacle information acquisition unit 43 to acquire the position of the obstacle Q on the bird's-eye view image based on the acquired obstacle information. The control unit 41 proceeds to step S14.

The control unit 41 determines whether or not the obstacle Q is located within the display range A of the overhead view image (step S14). More specifically, the control unit 41 determines whether or not the obstacle Q is within the display range A of the overhead view image based on the position information of the obstacle Q acquired in step S13. When the control unit 41 determines that the obstacle Q is within the display range A of the overhead view image (Yes in step S14), the process proceeds to step S16. When the control unit 41 determines that the obstacle Q is not within the display range A of the overhead view image (No in step S14), the process proceeds to step S15.

The control unit 41 generates and displays the overhead view image 100 (step S15). More specifically, the control unit 41 generates a bird's-eye view image 100 by the bird's-eye view image generation unit 45 by performing viewpoint conversion processing so that the vehicle V is looked down from above with the virtual viewpoint P from the peripheral image data acquired by the image data acquisition unit 42. Then, it is displayed on the display panel 30. The control unit 41 proceeds to step S13.

The control unit 41 changes the virtual viewpoint position (step S16). More specifically, the control unit 41 causes the bird's-eye view image generation unit 45 to change the position of the virtual viewpoint P of the bird's-eye view image 100 to generate the bird's-eye view image as the virtual viewpoint PF, and display it on the display panel 30. The control unit 41 proceeds to step S18.

In the present embodiment, the control unit 41 causes the overhead image generation unit 45 to generate an image that has undergone the viewpoint conversion processing as the virtual viewpoint PF located on the extension line L61 from the vehicle V to the obstacle Q, and the display panel 30. To display.

The control unit 41 generates and displays the overhead view image 100 (step S17). The control unit 41 performs the same process as step S15.

The control unit 41 determines whether or not to end the overhead view image display (step S18). More specifically, the control unit 41 determines whether to end the bird's-eye view image display based on the presence/absence of the backward movement end trigger. The reverse end trigger means, for example, that the shift position has changed from "reverse" to another position. When there is the backward movement end trigger, the control unit 41 determines to end the bird's-eye view video display (Yes in step S18), and ends the process. If there is no backward end trigger, the control unit 41 determines not to end the overhead view image display (No in step S18), and returns to step S12 to continue the process.

In this way, the overhead view image generation system 1 changes the position of the virtual viewpoint to the virtual viewpoint PF located on the extension line L61 from the vehicle V to the obstacle Q when the obstacle Q is detected in the traveling direction. Generate a bird's eye view video.

As described above, in the present embodiment, when the obstacle Q is detected in the traveling direction, the overhead view video in which the position of the virtual viewpoint is changed to the virtual viewpoint PF is generated. Accordingly, in the present embodiment, when the obstacle Q is detected in the traveling direction, the position of the virtual viewpoint is changed to the position on the extension line L61 from the vehicle V to the obstacle Q, and the obstacle Q and the vehicle V are changed. It is possible to display a bird's-eye view image showing the relative positional relationship of the. According to the present embodiment, whether or not the obstacle Q contacts the vehicle V can be intuitively recognized by the overhead view image. As described above, the present embodiment can properly display the obstacle Q in the overhead view image. The present embodiment can improve the visibility of the obstacle Q in the overhead image. In the present embodiment, the obstacle Q around the vehicle can be displayed so that it can be appropriately confirmed.

In this embodiment, for example, it is possible to display a bird's-eye view image as a virtual viewpoint PF that guides the approaching vehicle V while standing behind the obstacle Q. As a result, it is possible to easily recognize whether or not the obstacle Q comes into contact with the vehicle V by the bird's-eye view image as compared with the bird's-eye view image 100 with the virtual viewpoint P located right above the vehicle V.

According to the present embodiment, the position where the angle of view of the generated bird's-eye view image is approximately the same as the angle of view of the bird's-eye view image 100 of the virtual viewpoint P is the virtual viewpoint PF. As a result, the size of the obstacle Q displayed in the bird's-eye view image as the virtual viewpoint PF is the same as the size of the obstacle Q displayed in the bird's-eye view image 100. According to this embodiment, even if the position of the virtual viewpoint is changed, it is possible to generate a bird's-eye view image in which the relative distance between the obstacle Q and the vehicle V can be easily recognized.

[Second embodiment]
The overhead view image generation system 1 according to the present embodiment will be described with reference to FIG. 9. FIG. 9 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view image generation system according to the second embodiment. The bird's-eye view video generation system 1 has the same basic configuration as the bird's-eye view video generation system 1 of the first embodiment. In the following description, the same components as those of the bird's-eye view video generation system 1 are designated by the same or corresponding symbols, and detailed description thereof is omitted. In the bird's-eye view video generation system 1 of this embodiment, the processing in the viewpoint conversion processing unit 451 of the bird's-eye view video generation unit 45 is different from that of the bird's-eye view video generation system 1 of the first embodiment.

When the position of the obstacle Q acquired by the obstacle information acquisition unit 43 is located in the traveling direction of the vehicle V, the viewpoint conversion processing unit 451 detects the obstacle Q on the extension line L62 from the vehicle V to the obstacle Q. An image subjected to the viewpoint conversion processing is generated as the virtual viewpoint PG located right above.

In this embodiment, when the obstacle Q located behind is detected, the viewpoint conversion processing unit 451 generates an image subjected to the viewpoint conversion processing as the virtual viewpoint PG positioned directly above the obstacle Q. The position of the virtual viewpoint PG is represented by (xG, yG, zG).

The virtual viewpoint PG will be described. In the present embodiment, the virtual viewpoint PG is located on the left rear of the vehicle V. The virtual viewpoint PG is located so as to overlap the obstacle Q when viewed in the Z-axis direction. The virtual viewpoint PG is located to the left rear of the virtual viewpoint P when viewed in the Z-axis direction. The virtual viewpoint PG is a viewpoint that looks down at the vehicle V obliquely from directly above the obstacle Q.

Next, the flow of processing in the bird's-eye view video generation device 40 of the bird's-eye view video generation system 1 will be described. The bird's-eye view video generation device 40 performs processing according to the flowchart shown in FIG. In this embodiment, the processing in step S16 is different from that in the first embodiment, and the processing in steps S11 to S15, step S17, and step S18 is the same as that in the first embodiment.

The control unit 41 changes the virtual viewpoint position (step S16). More specifically, the control unit 41 causes the bird's-eye view image generation unit 45 to generate the bird's-eye view image as the virtual viewpoint PG located on the extension line L62 from the vehicle V to the obstacle Q, and displays it on the display panel 30. The control unit 41 proceeds to step S18.

As described above, in the present embodiment, when the obstacle Q in the traveling direction is detected, the bird's-eye view image of the virtual viewpoint PG located right above the obstacle Q on the extension line L62 from the vehicle V to the obstacle Q. To generate. In the present embodiment, when an obstacle Q in the traveling direction is detected, the position of the virtual viewpoint is changed to a position directly above the obstacle Q so that the relative positional relationship between the obstacle Q and the vehicle V can be displayed more accurately. It is possible to display a bird's eye view video that was passed. According to the present embodiment, whether or not the obstacle Q contacts the vehicle V can be easily and accurately recognized from the overhead view image. As described above, the present embodiment can properly display the obstacle Q in the overhead view image. The present embodiment can improve the visibility of the obstacle Q in the overhead image. In the present embodiment, the obstacle Q around the vehicle can be displayed so that it can be appropriately confirmed.

In the present embodiment, for example, it is possible to display a bird's-eye view image as a virtual viewpoint PG that guides the approaching vehicle V while standing at the same position as the obstacle Q. This makes it possible to intuitively recognize whether or not the obstacle Q contacts the vehicle V by the overhead view image, compared to the overhead view image 100 with the virtual viewpoint P located directly above the vehicle V.

[Third embodiment]
The overhead view image generation system 1 according to the present embodiment will be described with reference to FIGS. 10 and 11. FIG. 10 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the third embodiment. FIG. 11 is a schematic diagram illustrating another example of the position of the virtual viewpoint in the overhead view video generation system according to the third embodiment. In the bird's-eye view video generation system 1 of this embodiment, the processing in the viewpoint conversion processing unit 451 of the bird's-eye view video generation unit 45 differs from that of the bird's-eye view video generation system 1 of the first embodiment. In this embodiment, an obstacle Q1 and an obstacle Q2 exist.

When there is an obstacle satisfying a predetermined condition, the viewpoint conversion processing unit 451 places the peripheral image acquired by the image data acquisition unit 42 at a position determined based on the extension line from the vehicle V to each of the plurality of obstacles. The viewpoint conversion processing is performed so that the virtual viewpoint is located. In the present embodiment, when there is an obstacle that satisfies a predetermined condition, the viewpoint conversion processing unit 451 displays the peripheral image between any two of the extension lines from the vehicle V to each of the plurality of obstacles Q. An image that has undergone the viewpoint conversion processing is generated as the virtual viewpoint PH located on the bisector of the corner.

The two extension lines may have the largest angle between the two extension lines, for example. Alternatively, the two extension lines may be, for example, extension lines to the two obstacles Q in the order of decreasing distance from the vehicle V.

In the present embodiment, as shown in FIG. 10, when a plurality of obstacles Q located on the left rear side of the vehicle V are detected, the viewpoint conversion processing unit 451 determines an extension line L63 from the vehicle V to the obstacle Q1. , An image that has undergone the viewpoint conversion processing is generated as the virtual viewpoint PH1 located on the bisector L65 at the corner between the extension line L64 extending from the vehicle V to the obstacle Q2. The position of the virtual viewpoint PH1 is represented by (xH1, yH1, zH1).

The virtual viewpoint PH1 will be described. In the present embodiment, the virtual viewpoint PH1 is located on the left rear of the vehicle V. The virtual viewpoint PH1 is located rearward of the obstacle Q1 and rearward left of the obstacle Q2 when viewed in the Z-axis direction. The virtual viewpoint PH1 is located to the left rear of the virtual viewpoint P when viewed in the Z-axis direction. The virtual viewpoint PH1 is farther from the center of the vehicle V than the distance between the center of the vehicle V and the obstacle Q1 and the distance between the center of the vehicle V and the obstacle Q2. The virtual viewpoint PH1 is a viewpoint that looks down the vehicle V diagonally from the left rear.

Alternatively, as shown in FIG. 11, when the viewpoint conversion processing unit 451 detects an obstacle Q1 located on the left rear side of the vehicle V and an obstacle Q2 located on the right side of the vehicle V, A bird's-eye view image is generated as a virtual viewpoint PH2 located on the bisector L68 at the corner between the extension line L66 extending from the center to the obstacle Q1 and the extension line L67 extending from the center of the vehicle V to the obstacle Q2. Good. The position of the virtual viewpoint PH2 is represented by (xH2, yH2, zH2).

The virtual viewpoint PH2 will be described. In the present embodiment, the virtual viewpoint PH2 is located on the right rear side of the vehicle V. The virtual viewpoint PH2 is located right behind the obstacle Q1 and behind the obstacle Q2 when viewed in the Z-axis direction. The virtual viewpoint PH2 is located to the right rear of the virtual viewpoint P when viewed in the Z-axis direction. The virtual viewpoint PH2 is farther from the center of the vehicle V than the distance between the center of the vehicle V and the obstacle Q1 and the distance between the center of the vehicle V and the obstacle Q2. The virtual viewpoint PH2 is a viewpoint that looks down at the vehicle V diagonally from the right rear.

Alternatively, as illustrated in FIG. 12, when the viewpoint conversion processing unit 451 detects an obstacle Q1 and an obstacle Q2 located in the left rear of the vehicle V and arranged in the vehicle width direction, the viewpoint conversion processing unit 451 causes an obstacle from the center of the vehicle V. The bird's-eye view image may be generated as the virtual viewpoint PH3 located on the bisector L71 of the angle between the extension line L69 extending to the object Q1 and the extension line L70 extending from the center of the vehicle V to the obstacle Q2. The position of the virtual viewpoint PH3 is represented by (xH3, yH3, zH3).

The virtual viewpoint PH3 will be described. In the present embodiment, the virtual viewpoint PH3 is located on the left rear of the vehicle V. The virtual viewpoint PH3 is located to the left rear of the obstacle Q1 and to the right rear of the obstacle Q2 when viewed in the Z-axis direction. The virtual viewpoint PH3 is located to the left rear of the virtual viewpoint P when viewed in the Z-axis direction. The virtual viewpoint PH3 is farther from the center of the vehicle V than the distance between the center of the vehicle V and the obstacle Q1 and the distance between the center of the vehicle V and the obstacle Q2. The virtual viewpoint PH3 is a viewpoint that looks down the vehicle V diagonally from the left rear.

Next, the flow of processing in the bird's-eye view video generation device 40 of the bird's-eye view video generation system 1 will be described. The bird's-eye view video generation device 40 performs processing according to the flowchart shown in FIG. In this embodiment, the processing in step S16 is different from that in the first embodiment, and the processing in steps S11 to S15, step S17, and step S18 is the same as that in the first embodiment.

The control unit 41 changes the virtual viewpoint position (step S16). More specifically, the control unit 41 is a bird's-eye view image generation unit 45, and is located on a bisector of a corner between any two of the extension lines from the vehicle V to each of the plurality of obstacles Q. An overhead image as the viewpoint PH is generated and displayed on the display panel 30. The control unit 41 proceeds to step S18.

As described above, in the present embodiment, when a plurality of obstacles Q in the traveling direction are detected, the corners between any two of the extension lines from the vehicle V to each of the plurality of obstacles Q are detected. An overhead image of the virtual viewpoint PH located on the bisector is generated. As a result, in the present embodiment, when a plurality of obstacles Q in the traveling direction are detected, the position of the virtual viewpoint is changed so that the plurality of obstacles Q do not overlap, and the plurality of obstacles Q and the vehicle V are not overlapped. A bird's-eye view image showing a relative positional relationship can be displayed. According to the present embodiment, it is possible to easily recognize whether or not a plurality of obstacles Q come into contact with the vehicle V from the overhead view image. As described above, the present embodiment can appropriately display the plurality of obstacles Q in the overhead view image. The present embodiment can improve the visibility of the plurality of obstacles Q in the overhead image. In the present embodiment, a plurality of obstacles Q around the vehicle can be displayed so that they can be appropriately confirmed.

In the present embodiment, for example, it is possible to display a bird's-eye view image with a virtual viewpoint PH that guides the approaching vehicle V while standing between a plurality of obstacles Q. As a result, it is possible to easily recognize whether or not a plurality of obstacles Q are in contact with the vehicle V by the bird's-eye view image, as compared with the bird's-eye view image 100 having the virtual viewpoint P located right above the vehicle V.

[Fourth Embodiment]
The overhead view image generation system 1 according to the present embodiment will be described with reference to FIG. FIG. 13 is a schematic diagram illustrating an example of the position of the virtual viewpoint in the overhead view video generation system according to the fourth embodiment. The bird's-eye view video generation system 1 of this embodiment is different from the bird's-eye view video generation system 1 of the first embodiment in the processing in the bird's-eye view video generation unit 45. In this embodiment, the obstacle Q1 is located on the left rear of the vehicle V, and the obstacle Q2 is located on the right rear of the vehicle V.

The bird's-eye view video generation unit 45 uses the vehicle V and at least some of the obstacles Q as the peripheral image acquired by the video data acquisition unit 42 based on the position information of the plurality of obstacles Q acquired by the obstacle information acquisition unit 43. In order to allow a bird's eye view, a video that has undergone the viewpoint conversion processing is generated as a virtual viewpoint PI1 different from the virtual viewpoint P. Further, the bird's-eye view image generation unit 45, based on the position information of the plurality of obstacles Q, allows the vehicle V and at least a part of the remaining obstacles Q to be looked down on the surrounding image, and is different from the virtual viewpoint PI1. As the PI2, an image subjected to the viewpoint conversion processing is generated. Then, the bird's-eye view video generation unit 45 synthesizes the two videos subjected to the viewpoint conversion processing to generate the bird's-eye view video.

The viewpoint conversion processing unit 451 extends the position of the virtual viewpoint from the vehicle V to the obstacle Q1 when the obstacle Q1 located on the left rear of the vehicle V and the obstacle Q2 located on the right rear of the vehicle V are detected. An image that has been changed to a position on the line L72 and subjected to the viewpoint conversion processing as the virtual viewpoint PI1, and a position of the virtual viewpoint is changed to a position on the extension line L73 from the vehicle V to the obstacle Q2 and the viewpoint as the virtual viewpoint PI2. The converted video is generated. The position of the virtual viewpoint PI1 is represented by (xI1, yI1, zI1). The position of the virtual viewpoint PI2 is represented by (xI2, yI2, zI2).

The virtual viewpoint PI1 will be described. In the present embodiment, the virtual viewpoint PI1 is located on the left rear of the vehicle V. The virtual viewpoint PI1 is located to the left rear of the obstacle Q1 when viewed in the Z-axis direction. The virtual viewpoint PI1 is located to the left rear of the virtual viewpoint P when viewed in the Z-axis direction. The virtual viewpoint PI1 is farther from the center of the vehicle V than the distance between the center of the vehicle V and the obstacle Q1. The virtual viewpoint PI1 is a viewpoint that looks down the vehicle V obliquely from the left rear.

The virtual viewpoint PI2 will be described. In the present embodiment, the virtual viewpoint PI2 is located on the right rear side of the vehicle V. The virtual viewpoint PI2 is located to the right rear of the obstacle Q2 when viewed in the Z-axis direction. The virtual viewpoint PI2 is located to the right rear of the virtual viewpoint P when viewed in the Z-axis direction. The virtual viewpoint PI2 is farther from the center of the vehicle V than the distance between the center of the vehicle V and the obstacle Q2. The virtual viewpoint PI2 is a viewpoint that looks down the vehicle V obliquely from the right rear.

The synthesis processing unit 453 synthesizes the video that has undergone the viewpoint conversion processing as the virtual viewpoint PI1 and the video that has undergone the viewpoint conversion processing as the virtual viewpoint PI2 to generate a bird's-eye view video. For example, the synthesis processing unit 453 synthesizes the viewpoint-converted video as the virtual viewpoint PI1 on the left side and the viewpoint-converted video as the virtual viewpoint PI2 on the right side to generate the overhead view video.

Next, the flow of processing in the bird's-eye view video generation device 40 of the bird's-eye view video generation system 1 will be described. The bird's-eye view video generation device 40 performs processing according to the flowchart shown in FIG. In this embodiment, the processing in step S16 is different from that in the first embodiment, and the processing in steps S11 to S15, step S17, and step S18 is the same as that in the first embodiment.

The control unit 41 changes the virtual viewpoint position (step S16). More specifically, the control unit 41 causes the bird's-eye view image generation unit 45 to display the peripheral image acquired by the image data acquisition unit 42 on the basis of the position information of the plurality of obstacles Q acquired by the obstacle information acquisition unit 43. An image that has undergone the viewpoint conversion processing is generated as a virtual viewpoint PI1 different from the virtual viewpoint P so that the V and at least the obstacle Q1 can be overlooked. The control unit 41 uses the bird's-eye view image generation unit 45 as a virtual viewpoint PI2 different from the virtual viewpoint PI1 so that the peripheral image can be viewed from the vehicle V and at least the obstacle Q2 based on the position information of the plurality of obstacles Q. Generates an image that has undergone viewpoint conversion processing. Then, the control unit 41 causes the overhead image generation unit 45 to combine the two images to generate an overhead image and display it on the display panel 30. The control unit 41 proceeds to step S18.

As described above, in the present embodiment, when the obstacle Q is detected on the left side and the right side in the traveling direction, the image in which the viewpoint conversion processing is performed with a plurality of virtual viewpoints is generated. Then, a plurality of images that have undergone the viewpoint conversion processing are combined to generate an overhead view image. In the present embodiment, when obstacles Q are detected on the left and right sides in the traveling direction, the images that have been subjected to the viewpoint conversion processing as a plurality of virtual viewpoints are combined to make a relative movement between the plurality of obstacles Q and the vehicle V. A bird's-eye view image showing various positional relationships can be displayed.

When obstacles Q are detected on the left and right sides in the traveling direction, if viewpoint conversion processing is performed from either the left or right virtual viewpoint to generate an overhead view image, the obstacle Q and the vehicle on the opposite side of the virtual viewpoint are detected. It may not be possible to represent the relative positional relationship of V.

On the other hand, according to the present embodiment, even when the obstacles Q are detected on the left and right sides in the traveling direction, for example, standing behind each of the obstacles Q to guide the approaching vehicle V. It is possible to display a bird's-eye view image that is a combination of the image of the virtual viewpoint PI1 and the image of the virtual viewpoint PI2. Whether or not the plurality of obstacles Q contact the vehicle V can be more appropriately recognized by the overhead view image. As described above, the present embodiment can appropriately display the plurality of obstacles Q in the overhead view image. The present embodiment can improve the visibility of the plurality of obstacles Q in the overhead image. In the present embodiment, the obstacle Q around the vehicle can be displayed so that it can be appropriately confirmed.

Now, the bird's-eye view image generation system 1 according to the present invention has been described so far, but it may be implemented in various different forms other than the above-described embodiment.

Each component of the illustrated bird's-eye view image generation system 1 is functionally conceptual, and does not necessarily have to be physically configured as illustrated. That is, the specific form of each device is not limited to the one shown in the figure, and all or part of the device may be functionally or physically dispersed or integrated in arbitrary units according to the processing load or usage condition of each device. May be.

The configuration of the bird's-eye view video generation system 1 is realized by, for example, a program loaded in a memory as software. In the above-described embodiment, the functional blocks have been described as those realized by cooperation of these hardware or software. That is, these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.

The components described above include those that can be easily conceived by those skilled in the art and those that are substantially the same. Furthermore, the configurations described above can be appropriately combined. In addition, various omissions, substitutions, or changes in the configuration are possible without departing from the scope of the present invention.

Although the predetermined condition is described as the obstacle Q being located in the traveling direction of the vehicle V, the predetermined condition is not limited to this. The predetermined condition may be, for example, that the vehicle V may be interfered with. The possibility of interfering with the vehicle V means, for example, being located in the traveling direction of the vehicle V and having a height above the ground that may contact the vehicle V.

The predetermined condition is that it is determined that the predetermined condition is satisfied when the obstacle Q is located behind the front end of the vehicle V when the vehicle V is moving backward, but the predetermined condition is not limited to this. For example, the predetermined condition is that when the vehicle V is moving backward, the obstacle Q is located behind the rear end of the vehicle V, or the obstacle Q is located behind the axle position of the drive wheels of the vehicle V. In this case, it may be determined that the predetermined condition is satisfied.

In step S11, the control unit 41 may determine whether or not to start the bird's-eye image display, for example, based on whether or not the operation to start the bird's-eye image display on the operation unit is detected.

The bird's-eye view image in which the position of the virtual viewpoint is changed may generate a bird's-eye view image in which the center of the own vehicle icon is shifted from the center of the bird's-eye view image so that the display area in the direction in which the obstacle Q is detected becomes large.

The position in the Z direction of the virtual viewpoint P may be, for example, the height of the line of sight of the driver sitting in the driver's seat, the height of the line of sight when the driver stands, the height of the roof of the vehicle V, or the like. ..

1 Overhead image generation system 11 Front camera 12 Rear camera 13 Left side camera 14 Right side camera 20 Sensor group (obstacle sensor)
30 Display panel (display section)
40 overhead view image generation device 41 control unit 42 image data acquisition unit 43 obstacle information acquisition unit 44 vehicle information acquisition unit 45 overhead view image generation unit 451 viewpoint conversion processing unit 452 cutout processing unit 453 synthesis processing unit 48 display control unit 49 storage unit P Virtual viewpoint Q Obstacle V Vehicle

Claims (5)

A video data acquisition unit that acquires peripheral video data of the surroundings of the vehicle,
An obstacle information acquisition unit that acquires position information of an obstacle detected in the vicinity of the vehicle,
Based on the position information of the obstacle acquired by the obstacle information acquisition unit, from the vehicle to the obstacle, the peripheral image acquired by the image data acquisition unit can be viewed from the vehicle and the obstacle. A bird's-eye view image generating unit that generates a bird's-eye view image by performing viewpoint conversion processing so that the virtual viewpoint is located at a position defined on the extension line of
A display control unit that causes the display unit to display the overhead image generated by the overhead image generation unit ;
Have a,
When there are a plurality of obstacles, the bird's-eye view image generation unit performs viewpoint conversion processing so that the virtual viewpoint is located at a position determined based on an extension line from the vehicle to each of the plurality of obstacles. Generate an overhead view video,
俯瞰映image generation apparatus. In the case where there are a plurality of obstacles, the bird's-eye view image generation unit positions the virtual viewpoint on the bisector of the corner between any two extended lines from the vehicle to each of the plurality of obstacles. To generate a bird's-eye view image that has undergone the viewpoint conversion process,
The bird's-eye view video generation device according to claim 1 . An overhead view image generation apparatus according to claim 1 or 2 ,
A bird's-eye view image generation system, comprising: a plurality of cameras that photograph the surroundings of the vehicle; the display unit; and a detection unit that detects an obstacle and outputs obstacle information. A video data acquisition step of acquiring peripheral video data of the surroundings of the vehicle,
An obstacle information acquisition step of acquiring position information of the obstacle detected in the vicinity of the vehicle,
Based on the position information of the obstacle acquired in the obstacle information acquisition step, the surrounding image acquired in the image data acquisition step can be viewed from the vehicle to the obstacle so that the vehicle and the obstacle can be overlooked. A bird's-eye view video generation step of generating a bird's-eye view video that has been subjected to viewpoint conversion processing so that the virtual viewpoint is located on the extension line of
A display control step of displaying the bird's-eye view image generated in the bird's-eye view image generating step on a display unit ;
Only including,
In the bird's-eye view image generation step, when there are a plurality of obstacles, the viewpoint conversion processing is performed so that the virtual viewpoint is located at a position determined based on an extension line from the vehicle to each of the plurality of obstacles. Generate an overhead view video,
俯瞰映image generating method overhead image generating device executes. A video data acquisition step of acquiring peripheral video data of the surroundings of the vehicle,
An obstacle information acquisition step of acquiring position information of the obstacle detected in the vicinity of the vehicle,
Based on the position information of the obstacle acquired in the obstacle information acquisition step, the surrounding image acquired in the image data acquisition step can be viewed from the vehicle to the obstacle so that the vehicle and the obstacle can be overlooked. A bird's-eye view video generation step of generating a bird's-eye view video that has been subjected to viewpoint conversion processing so that the virtual viewpoint is located on the extension line of
A display control step of displaying the bird's-eye view image generated in the bird's-eye view image generating step on a display unit ;
Including
In the bird's-eye view image generation step, when there are a plurality of obstacles, the viewpoint conversion processing is performed so that the virtual viewpoint is located at a position determined based on an extension line from the vehicle to each of the plurality of obstacles. Generate an overhead view video,
A program to be executed by a computer that operates as an overhead image generation device.

Images