JP4569285B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that processes an image captured by an in-vehicle camera and converts it into an image captured by a virtual camera.

  An apparatus that supports reverse parking by processing an image captured by an in-vehicle camera, converting the image to a virtual camera image, and superimposing icons indicating the vehicle position and the parking position (see, for example, Patent Document 1) is known. ).

Prior art documents related to the invention of this application include the following.
JP 2003-118522 A

  However, in the conventional apparatus, when the video of the real camera and the video of the virtual camera are suddenly switched, the object around the own vehicle on the image after switching, the object around the own vehicle on the original image, or the actual vehicle There is a problem that it is difficult for the driver to immediately recognize the correspondence relationship and the positional relationship with objects around the vehicle.

An actual imaging unit mounted on the host vehicle for capturing an image of the periphery of the host vehicle and outputting the actual video, and an image processing unit for converting the actual video by the actual imaging unit into a virtual video by a virtual imaging unit set at an arbitrary position and orientation And a display means for displaying a real video of the real imaging means or a virtual video of the image processing means, wherein the image processing means converts the display video of the display means to the first real video or the virtual video. on the way from the switching to the second real image or virtual image, by changing the optical properties of the virtual imaging lens of the taking lens and the virtual image pickup means of the actual imaging means, the position and capturing a first real image or virtual image An interpolated virtual image viewed from a position and direction between the direction and the position where the second real image or virtual image is captured and the direction is generated and displayed on the display means.

  According to the present invention, when switching an image obtained by imaging the surrounding environment of the host vehicle, the surrounding environment of the host vehicle can be immediately understood from the switched image.

<< First Embodiment of the Invention >>
FIG. 1 is a diagram showing a configuration of an embodiment. In this embodiment, four cameras 1 to 4 are mounted on the host vehicle, and real images around the host vehicle are captured. FIG. 2 shows vectors of the installation positions (viewpoints) and orientations of the four cameras 1 to 4, that is, the direction of the optical axis of the photographing lens (the line of sight). The front camera 1 is installed at the front end of the fender at the center of the left and right sides of the vehicle, and captures an image looking down on the road surface in front of the host vehicle as shown in FIG. The right-side camera 2 is installed on a fender on the right side of the vehicle, and captures an image looking down on the road surface on the right side of the host vehicle as shown in FIG. The left-side camera 3 is installed in a fender on the left side of the vehicle, and captures an image looking down on the road surface on the left side of the host vehicle as shown in FIG. The rear camera 4 is installed at the rear end of the roof or the rear end of the trunk lid at the center of the left and right sides of the vehicle, and captures an image looking down the road surface behind the host vehicle as shown in FIG. In this embodiment, a camera equipped with a wide-angle lens having a horizontal field angle of about 180 degrees will be described as an example.

  The image processing apparatus 5 includes a CPU, peripheral components such as a memory and an A / D converter, and performs processing such as distortion correction, enlargement, reduction, rotation, viewpoint conversion, and composition on the captured images by the cameras 1 to 4. Since these image processing methods are well known, description thereof will be omitted. For viewpoint conversion, conversion calculation may be performed in real time, but in this embodiment, a method of assigning color information of pixels of an input video to pixels of an output video by referring to a conversion table calculated in advance is adopted. To do. A conversion table is prepared in advance according to the type of viewpoint conversion and image composition. The display 6 displays the video generated by the image processing device 5.

  The image processing device 5 performs processing such as viewpoint conversion and composition on the actual video around the host vehicle by the cameras 1 to 4 mounted on the host vehicle, and converts it into a virtual video by a virtual camera set at an arbitrary position and orientation. . FIG. 7 shows vectors of the installation positions (viewpoints) and orientations of the virtual cameras VA, VB, VC, VD, VE, and VF of one embodiment, that is, the optical axis direction (sight line direction) of the photographing lens. The virtual camera VA is arranged so as to look down at the own vehicle 10 from above the own vehicle, and the virtual camera VB is arranged so as to see the own vehicle 10 in a substantially horizontal direction from the front of the own vehicle. In addition, the virtual camera VC is arranged so as to see the rear of the host vehicle from the left side of the vehicle fender, and the virtual camera VD is arranged so as to see the rear of the host vehicle from the right side of the vehicle fender. Furthermore, the virtual camera VE is disposed so as to look at the rear side of the host vehicle from the left and right center position on the vehicle fender, and the virtual camera VF is disposed so as to look down at the host vehicle 10 from directly above the center position of the left and right sides of the vehicle fender. The virtual camera VE is blocked by the host vehicle 10 and cannot capture an image behind the host vehicle, but captures an image when the host vehicle 10 is not present by synthesis.

  In the first embodiment, image processing is performed when switching from a bird's-eye view video of a virtual camera VA looking down at the host vehicle from above the host vehicle to a video of the virtual camera VB viewing the host vehicle in a substantially horizontal direction from the front of the host vehicle. Show. FIG. 8 shows a video example of the virtual camera VA, and FIG. 9 shows a video example of the virtual camera VB. These images are obtained by converting the viewpoints of the images captured by the four in-vehicle cameras 1 to 4 and then synthesizing them when entering the parking position drawn with a white line in a backward direction. Now, when the bird's-eye view video of the virtual camera VA shown in FIG. 8 is suddenly switched to the video of the virtual camera VB shown in FIG. 9, the object on the video shown in FIG. 9 and the object on the original video shown in FIG. The driver may be confused because the correspondence and position with the actual object cannot be immediately grasped. For example, although the other vehicle behind the host vehicle is not shown in the original image shown in FIG. 8, the other vehicle behind the host vehicle is shown in the image shown in FIG. 9 after switching.

  FIG. 10 is a top view and a side view showing the positional relationship between the host vehicle and the virtual camera, and FIG. 11 is a change in the position (viewpoint) and direction (optical axis direction of the photographing lens = line-of-sight direction) of the virtual camera when switching images. FIG. In this specification, the position of the virtual camera is represented by X, Y, and Z, the left and right direction of the host vehicle is the X axis, the left and right center is X = 0, the left direction is X> 0, and the right direction is X <0. To do. The front-rear direction of the host vehicle is the Y axis, the front wheel position is Y = 0, the rear of the host vehicle is Y> 0, and the front of the host vehicle is Y <0. Further, the height direction of the host vehicle is the Z axis, the ground contact point of the front wheels is Z = 0, the upper side is Z> 0, and the lower side is Z <0. In this specification, the direction of the virtual camera is represented by yaw, pitch, and roll. Here, yaw represents rotation around an axis parallel to the Z axis passing through the virtual camera, pitch represents rotation around an axis parallel to the X axis passing through the virtual camera, and roll is parallel to the Y axis passing through the virtual camera. Represents rotation around the axis. FIG. 11 shows the position (X, Y, Z) and direction (yaw, pitch, roll) of the virtual camera for each frame.

  In this embodiment, as shown in FIG. 11, the position of the virtual camera is VI1 to VI9 for each frame from the position of the virtual camera VA above the host vehicle to the position of the virtual camera VB in front of the host vehicle. Suppose that it has changed. As for the position (viewpoint) of the virtual camera, the distance Y to the host vehicle 10 gradually decreases and the height Z gradually decreases from the position of the VA in front of the host vehicle to the position of VB in front of the host vehicle. However, the position X in the left-right direction of the vehicle is constant. On the other hand, the direction of the virtual camera (the direction of the line of sight) gradually changes from the downward direction to the upward direction from the position of the VA in front of the host vehicle to the position of VB in front of the host vehicle, but the yaw and roll are constant. is there.

  Now, assuming that the pitch of the virtual camera VA is −35 degrees and the pitch of the virtual camera VB is −5 degrees, the pitch is upward by 3 degrees for each frame. The position of the virtual camera can be switched from VA to VB in 1/3 second. That is, virtual images of the virtual cameras VA, VI1, VI2,..., VI9, VB are displayed in order at 1/30 second intervals.

  FIG. 12 is a diagram illustrating a video switching display example according to the first embodiment. 12A shows a video example of the virtual camera VA similar to FIG. 8, and FIG. 12E shows a video example of the virtual camera VB similar to FIG. 12B, 12C, and 12D show examples of interpolation virtual images of the virtual cameras VI2, VI5, and VI8 that are interpolated between the virtual cameras VA and VB.

  As described above, when the video of the virtual camera VB suddenly switches from the video of the virtual camera VA, it takes time for the driver to understand the video of the virtual camera VB after switching, but the virtual camera VA By sequentially displaying the interpolated virtual videos of the virtual cameras VI1 to VI9 in an interpolated manner between the virtual video of the virtual camera VB and the virtual video of the virtual camera VB, approximately 1 from the virtual video of the virtual camera VA to the virtual video of the virtual camera VB is obtained. The video is gradually switched in 3 seconds, and the virtual video of the virtual camera VB after the switching can be immediately understood.

In the above-described embodiment, an example in which the position (X, Y, Z) and direction (yaw, pitch, roll) of the virtual camera are gradually switched in an interpolating manner has been described. Parameters such as the degree of distortion and the horizontal and vertical angles of view of the image may be changed. For example, the distortion characteristic of the lens is an approximate expression h = a * sin (b * θ) (1)
When a = b = 1 in the case of the virtual camera VA and a = b = 2 in the case of the virtual camera VB, different distortion characteristics are obtained by continuously changing these parameters a and b. The video can be connected without any sense of incongruity. In equation (1), θ is the incident angle, h is the distance from the center of the imaging surface of the light having the incident angle θ, and a and b are constants representing the distortion characteristics of the photographing lens.

<< Second Embodiment of the Invention >>
In the second embodiment, image processing when switching from the combined virtual video of the virtual cameras VC and VD shown in FIG. 13 to the video of the virtual camera VA shown in FIG. 8 will be described. In the synthesized virtual video shown in FIG. 13, the right half of the screen shows the virtual video of the virtual camera VC, and the left half of the screen shows the virtual video of the virtual camera VD.

  The configuration of the second embodiment is the same as the configuration shown in FIG. FIG. 14 is a top view and a side view showing the positional relationship between the host vehicle and the virtual camera, and FIG. 15 is a view showing changes in the position (viewpoint) and direction (line-of-sight direction) of the virtual camera when switching images. Note that the definitions of the position (X, Y, Z) and direction (yaw, pitch, roll) of the virtual camera are the same as the definitions shown in FIGS. In the second embodiment, the positions (X) of the virtual cameras VC and VD gradually change so that both cameras move toward the center of the left and right of the host vehicle, and at the same time, the directions of the virtual cameras VC and VD (the yaw) ) Gradually change and both cameras approach the direction of the back, and after integrating the virtual camera VE viewed from the left and right center position on the vehicle fender, the virtual camera position (Y, Z) The direction (pitch) gradually changes to reach the position of the virtual camera VA overlooking the host vehicle from above the host vehicle. For the position (X) and direction (yaw) of the virtual camera from the position of the virtual cameras VC and VD to the position of the virtual camera VE, the average value of the two virtual cameras VC and VD is used.

  Between the synthesized virtual video of the virtual cameras VC and VD and the virtual video of the virtual camera VE, the virtual videos of the virtual cameras (VC11, VD11) to (VC14, VD14) are displayed in an interpolated manner for each frame, and further the virtual camera Between the virtual video of VE and the virtual video of virtual camera VA, the virtual videos of virtual cameras VE1 to VE4 are displayed in an interpolated manner for each frame.

  FIG. 16 is a diagram illustrating a video switching display example according to the second embodiment. (a) shows an example of a combined virtual video of virtual cameras VC and VD similar to FIG. 13, and (c) shows an example of a virtual video of virtual camera VE. And (b) shows the example of a synthetic | combination virtual image | video of the virtual camera (VC11, VD11) of interpolation between virtual camera VC, VD and virtual camera VE. Further, (e) shows a virtual video example of the virtual camera VA similar to FIG. 8, and (d) shows a virtual video example of the virtual camera VE2 that is interpolated between the virtual camera VE and the virtual camera VA.

  As described above, when the virtual video of the virtual camera VA is suddenly switched from the combined virtual video of the virtual cameras VC and VD to the virtual video of the virtual camera VA, it takes time for the driver to understand the virtual video of the virtual camera VA after the switching. However, the virtual images of the virtual cameras (VC11, VD11) to (VC14, VD14), VE, VE11 to VE14 are interpolated between the synthesized virtual video of the virtual cameras VC and VD and the virtual video of the virtual camera VA. / Sequentially displaying every 30 seconds, the video gradually switches from frame to frame within about 1/3 second from the combined virtual video of the virtual cameras VC and VD to the virtual video of the virtual camera VA. The virtual video of the virtual camera VA can be immediately understood.

<< Third Embodiment of the Invention >>
In the second embodiment described above, the composite virtual video of the virtual cameras VC and VD is once switched to the virtual video of the virtual camera VE, and then further switched to the virtual video of the virtual camera VA. In the embodiment, image processing in the case of switching directly from the combined virtual video (see FIG. 13) of the virtual cameras VC and VD to the virtual video (see FIG. 8) of the virtual camera VA is shown. The configuration of the third embodiment is the same as the configuration shown in FIG.

  FIG. 17 is a top view and a side view showing the positional relationship between the host vehicle and the virtual camera, and FIG. 18 is a diagram showing changes in the position (viewpoint) and direction (line-of-sight direction) of the virtual camera when switching images. Note that the definitions of the position (X, Y, Z) and direction (yaw, pitch, roll) of the virtual camera are the same as the definitions shown in FIGS. In the third embodiment, the positions (X) of the virtual cameras VC and VD are gradually changed so that both cameras approach the center of the left and right of the host vehicle, and the directions (yaw) of the virtual cameras VC and VD. As the camera gradually changes, both cameras approach the back direction. Simultaneously with the change in position (X) and direction (yaw), the positions (Y, Z) of both virtual cameras VC, VD gradually change and approach the position (Y, Z) of virtual camera VA. At the same time, the direction (pitch) of both virtual cameras VC and VD gradually changes and approaches the direction (pitch) of the virtual camera VA. For the position (X) and direction (yaw) of the virtual camera from the position of the virtual cameras VC and VD to the position of the virtual camera VA, an average value of the two virtual cameras VC and VD is used.

  In the third embodiment, between the virtual images of the virtual cameras VC and VD and the virtual video of the virtual camera VA, the virtual images of the virtual cameras (VC11 and VD11) to (VC19 and VD19) are obtained for each frame. Is displayed in an interpolated manner. FIG. 19 is a diagram illustrating a video switching display example according to the third embodiment. FIG. 19A shows an example of a combined virtual video of virtual cameras VC and VD similar to FIG. 13, and FIG. 19E shows a virtual video example of a virtual camera VA similar to FIG. FIGS. 19B, 19C, and 19D show virtual cameras (VC12, VD12), (VC15, VD15), (VC18, interpolating) between the virtual cameras VC, VD and the virtual camera VA. An example of a synthesized virtual video of VD18) is shown.

  As described above, when the virtual video of the virtual camera VA is suddenly switched from the combined virtual video of the virtual cameras VC and VD to the virtual video of the virtual camera VA, it takes time for the driver to understand the virtual video of the virtual camera VA after the switching. However, the virtual images of the virtual cameras (VC11, VD11) to (VC19, VD19) are interpolated between the combined virtual images of the virtual cameras VC and VD and the virtual image of the virtual camera VA every 1/30 second frame. By sequentially displaying the images, the image is gradually switched for each frame in about 1/3 second from the synthesized virtual image of the virtual cameras VC and VD to the virtual image of the virtual camera VA, and the virtual camera VA after the switching is performed. Can immediately understand the virtual video.

<< Fourth Embodiment of the Invention >>
In the fourth embodiment, the virtual video of the virtual cameras VC and VD shown in FIG. 13 is once switched to the virtual video of the virtual camera VF and then switched to the virtual video of the virtual camera VA. The configuration of the fourth embodiment is the same as the configuration shown in FIG.

  FIG. 20 is a top view and a side view showing the positional relationship between the host vehicle and the virtual camera, and FIG. 21 shows changes in the position (viewpoint) and orientation (line-of-sight direction) of the virtual camera when switching images. Note that the definitions of the position (X, Y, Z) and direction (yaw, pitch, roll) of the virtual camera are the same as the definitions shown in FIGS. In the fourth embodiment, the positions (X) of the virtual cameras VC and VD are gradually changed so that both cameras are moved to the left and right center of the host vehicle, and the directions (yaw) of both cameras are gradually changed. Then approach the direction directly behind. Simultaneously with the change in position (X) and direction (yaw), the positions (Y, Z) of both virtual cameras VC, VD gradually change and approach the position (Y, Z) of virtual camera VF. At the same time, the direction (pitch) of both virtual cameras VC and VD gradually changes and approaches the direction (pitch) of the virtual camera VF. After the virtual cameras VC and VD are integrated with the virtual camera VF, the position (Y, Z) and direction (pitch) of the virtual camera approach the position (Y, Z) and direction (pitch) of the virtual camera VA.

For the position (X) and direction (yaw) of the virtual camera from the position of the virtual cameras VC and VD to the position of the virtual camera VF, the average value of the two virtual cameras VC and VD is used. Further, when the positions of the virtual cameras VC, VD, VA, VF are (X1, Y1), (X2, Y2), (X3, Y3), (X4, Y4), the X coordinate X4 of the virtual camera VF and Y coordinate Y4 is
X4 = (X1 + X2 + X3) / 3
Y4 = (Y1 + Y3) / 2 = (Y2 + Y3) / 2
And

  In the fourth embodiment, the synthesized virtual video of the virtual cameras (VC11, VD11) to (VC14, VD14) for each frame between the synthesized virtual video of the virtual cameras VC and VD and the virtual video of the virtual camera VF. Is displayed in an interpolated manner. FIG. 22 is a diagram illustrating a video switching display example according to the fourth embodiment. FIG. 22A shows an example of a synthesized virtual video of virtual cameras VC and VD similar to FIG. 13, and FIG. 22E shows an example of a virtual video of a virtual camera VA similar to FIG. FIG. 22B shows an example of a synthesized virtual video of an interpolated virtual camera (VC12, VD12) between the virtual cameras VC and VD and the virtual camera VF. FIGS. 22C and 22D are virtual images. An example of virtual images of virtual cameras VF1 and VF3 that are interpolated between the camera VF and the virtual camera VA is shown.

  As described above, when the virtual video of the virtual camera VA is suddenly switched from the combined virtual video of the virtual cameras VC and VD to the virtual video of the virtual camera VA, it takes time for the driver to understand the virtual video of the virtual camera VA after the switching. However, the virtual images of the virtual cameras (VC11, VD11) to (VC14, VD14), VF, VF1 to VF4 are interpolated between the synthesized video of the virtual cameras VC and VD and the virtual video of the virtual camera VA. By sequentially displaying every 30-second frame, the video is gradually switched in about 1/3 second from the synthesized virtual video of the virtual cameras VC and VD to the virtual video of the virtual camera VA, and the virtual camera VA after the switching Can immediately understand the virtual video.

<< Fifth Embodiment of the Invention >>
In the fifth embodiment, the virtual video of the virtual cameras VC and VD shown in FIG. 13 is first switched to the virtual video of the virtual camera VB, and then switched to the virtual video of the virtual camera VA. The configuration of the fifth embodiment is the same as the configuration shown in FIG.

  FIG. 23 is a top view and a side view showing the positional relationship between the host vehicle and the virtual camera, and FIG. 24 shows changes in the position (viewpoint) and orientation (line-of-sight direction) of the virtual camera when switching images. Note that the definitions of the position (X, Y, Z) and direction (yaw, pitch, roll) of the virtual camera are the same as the definitions shown in FIGS. In the fifth embodiment, the positions (X) of the virtual cameras VC and VD are gradually changed so that both cameras move toward the center of the left and right of the host vehicle, and the directions (yaw) of both cameras are gradually changed. Then approach the direction directly behind. Simultaneously with the change in the position (X) and the direction (yaw), the positions (Y) of both virtual cameras VC and VD gradually change and approach the position (Y) of the virtual camera VB. For the position (X) and direction (yaw) of the virtual camera from the position of the virtual cameras VC and VD to the position of the virtual camera VB, an average value of the two virtual cameras VC and VD is used. After the virtual cameras VC and VD are integrated into the virtual camera VB, the position (Y, Z) and direction (pitch) of the virtual camera approach the position (Y, Z) and direction (pitch) of the virtual camera VA.

  In the fifth embodiment, between the virtual images of the virtual cameras VC and VD and the virtual video of the virtual camera VB, the virtual images of the virtual cameras (VC11, VD11) to (VC14, VD14) for each frame. Is displayed in an interpolated manner. FIG. 25 is a diagram illustrating a video switching display example according to the fifth embodiment. FIG. 25A shows an example of a combined virtual video of virtual cameras VC and VD similar to FIG. 13, and FIG. 25E shows an example of a virtual video of a virtual camera VA similar to FIG. FIG. 25B shows an example of a synthesized virtual video of the virtual camera (VC12, VD12) that is interpolated between the virtual cameras VC, VD and the virtual camera VF, and FIGS. An example of virtual images of virtual cameras VB1 and VB3 that are interpolated between the camera VB and the virtual camera VA is shown.

  As described above, when the virtual video of the virtual camera VA is suddenly switched from the combined virtual video of the virtual cameras VC and VD to the virtual video of the virtual camera VA, it takes time for the driver to understand the virtual video of the virtual camera VA after the switching. However, the virtual images of the virtual cameras (VC11, VD11) to (VC14, VD14), VB, VB1 to VB4 are interpolated between the synthesized virtual video of the virtual cameras VC and VD and the virtual video of the virtual camera VA. / Sequentially displaying every 30-second frame, the video gradually switches in about 1/3 second from the synthesized virtual video of the virtual cameras VC and VD to the virtual video of the virtual camera VA, and the virtual camera after the switching The virtual image of VA can be immediately understood.

<< Sixth Embodiment of the Invention >>
An embodiment will be described in which the wire frame of the host vehicle is superimposed and displayed on an image around the host vehicle. FIGS. 26A to 26E show examples in which the wire frame of the host vehicle is superimposed on each video when switching from the virtual video of the virtual camera VA to the virtual video of the virtual camera VB. Thus, by superimposing and displaying the wire frame of the host vehicle on the virtual video around the host vehicle, the positional relationship with the parking frame on the video around the host vehicle and other vehicles can be easily grasped. It is possible to immediately understand the image around the vehicle displayed on the screen.

<< Seventh Embodiment of the Invention >>
An embodiment in which grid lines (lattices) are superimposed and displayed on an image around the host vehicle will be described. FIGS. 27A to 27E show examples in which grid lines are superimposed on each video when switching from the virtual video of the virtual camera VA to the virtual video of the virtual camera VB. This grid line converts the grid lines set on the ground (on the road map) at equal intervals to the grid lines viewed from the position (viewpoint) and direction (view direction) of the virtual camera, and converts them to the surroundings of the vehicle Superimposed on the video. Thereby, it is possible to sensuously grasp the parking frame on the image around the own vehicle and the distance from other vehicles, and it is possible to immediately understand the image around the own vehicle that is finally displayed.

<< Eighth Embodiment of the Invention >>
In the embodiment described above, when switching the video around the host vehicle, the interpolated video in the middle of switching is displayed to make it easier to understand the video around the host vehicle that is finally displayed. You may display the wire frame image of the own vehicle. For example, as shown in FIG. 28, when switching from the virtual image (a) of the virtual camera VA to the virtual image (e) of the VB of the virtual camera, (b) to (b) without displaying the interpolated virtual image during the switching. Instead, a wire frame image of the host vehicle is displayed as shown in d). In the wire frame image of the host vehicle, the wire frame of the host vehicle is converted into an overhead view as viewed from the position (viewpoint) and direction (view direction) of the virtual camera that captures the interpolated virtual image. As a result, the position and orientation of the virtual camera that captures the virtual video around the host vehicle, that is, the change in the viewpoint and line-of-sight direction of the host vehicle surrounding video can be easily grasped, and the host vehicle surrounding video that is finally displayed Can be understood immediately.

<< Ninth Embodiment of the Invention >>
Immediately after the driver activates the image processing apparatus, a composite virtual image of the virtual cameras VC and VD as shown in FIG. 13 is displayed. However, if the composite virtual video of the virtual cameras VC and VD is displayed immediately after the start-up, it is difficult for the driver to understand from which position (viewpoint) to which direction (line-of-sight direction) the composite virtual video is viewed. Therefore, as shown in FIG. 29 (a), first, an image of the virtual camera VF or VA, that is, an overhead view image looking down at the own vehicle from above the own vehicle is displayed, and then shown in (b) to (d). In this way, the position (viewpoint) and direction (line-of-sight direction) of the virtual camera are gradually changed to finally move to the positions of the virtual cameras VC and VD, and finally the virtual cameras VC and VD are shown as shown in (e). Displays the combined virtual video.

  In the embodiment described above, an example of switching from a virtual video of a virtual camera to a virtual video of another virtual camera has been described. However, when switching from a real video of a real camera to a virtual video of a virtual camera, or conversely, The present invention can also be applied when switching from a virtual image of a camera to a real image of a real camera.

  As described above, according to the embodiment, the real cameras 1 to 4 that are mounted on the own vehicle and that capture the surroundings of the own vehicle and output the real images, and the real images by the real cameras 1 to 4 can be set to arbitrary positions. An image processing device 5 for converting into virtual images by virtual cameras VA, VB, VC, VD, VE, and VF set in a direction, and a display for displaying real images of real cameras 1 to 4 or virtual images of the image processing device 5 6 and the image processing device 5 captures the first real video or virtual video while switching the display video on the display 6 from the first real video or virtual video to the second real video or virtual video. An interpolated virtual image viewed from a position and orientation intermediate between the position and orientation obtained and the position and orientation at which the second real image or virtual image is captured is generated and displayed on the display 6. As a result, when switching the image that captures the surrounding environment of the host vehicle, the object around the host vehicle on the switched image, the object around the host vehicle on the original image, or the object around the actual host vehicle Can be immediately recognized by the driver, and the surrounding environment of the host vehicle can be immediately understood from the image after switching.

  Further, according to the embodiment, the first real video or the virtual video is captured while the display video on the display device 6 is switched from the first real video or the virtual video to the second real video or the virtual video. A plurality of interpolated virtual images viewed from a plurality of intermediate positions and orientations from the position and orientation to the position and orientation from which the second real image or virtual image is imaged are generated, and these are sequentially switched to the display 6 Since it is displayed, it is possible to grasp the state of switching of the video, and to immediately understand the surrounding environment of the host vehicle from the video after switching.

  Further, according to the embodiment, the real cameras 1 to 4 are switched by the image processing device 5 while the display video of the display device 6 is switched from the first real video or virtual video to the second real video or virtual video. In addition, since the optical characteristics of the photographing lenses of the virtual cameras VA, VB, VC, VD, VE, and VF are continuously changed, it becomes easy to see the virtual video and the switching of the video can be easily understood.

  According to an embodiment, the image processing device 5 generates a real video or a virtual video by combining videos of the plurality of real cameras 1 to 4 or the virtual cameras VA, VB, VC, VD, VE, and VF, Since the information is displayed on the display device 6, the environment around the host vehicle can be efficiently grasped from the image of one display device 6.

  According to the embodiment, the virtual image of the single virtual camera A set by the image processing device 5 from the position above the own vehicle to the direction of the own vehicle from the composite images of the plurality of real cameras or virtual cameras. In the middle of switching, the interpolation virtual image of the single virtual camera B set in the direction of the host vehicle is generated from the position closer to the host vehicle than the virtual camera A and is displayed on the display unit 6. It is possible to reduce the ratio of the synthesized virtual video from among a plurality of interpolated virtual videos, making it easy to see the interpolated virtual video and making it easy to understand the video switching.

  According to one embodiment, when switching from a composite video of a plurality of real cameras or virtual cameras to a video of a single real camera or virtual camera, the position and orientation of the single real camera or virtual camera are combined. Since the average value of the positions and orientations of a plurality of real cameras or virtual cameras that capture images is used, it is easy to understand the switching of images, and the images after switching can be understood immediately.

  According to one embodiment, when the virtual image of the virtual camera C set in the direction of the host vehicle is displayed at a position close to the host vehicle by the image processing device 5, first from the host vehicle rather than the virtual camera C. A virtual image of the virtual camera D set in the direction of the host vehicle is displayed at a far position, and then an interpolated virtual image of the virtual camera E set in an intermediate position and direction between the virtual camera D and the virtual camera C is displayed. After that, since the virtual video of the virtual camera C is displayed, even when the virtual video of viewing the host vehicle from a position close to the host vehicle is displayed immediately after starting the image processing apparatus, the host vehicle in the virtual video is displayed. You can immediately understand the surrounding environment.

  According to one embodiment, the image processing device 5 superimposes a schematic diagram (wire frame) of the host vehicle viewed from the position and orientation of a virtual camera for capturing an interpolated virtual image on the interpolated virtual image and displays it. 6, it is easy to understand the switching between the vehicle position and the video on the interpolation virtual video, and the video after the switching can be understood immediately.

  According to an embodiment, the image processing device 5 converts grid lines (lattice lines) set at equal intervals on the road surface into grid lines viewed from the position and orientation of a virtual camera for capturing an interpolated virtual image. Since it is superimposed on the interpolated virtual image and displayed on the display device 6, it is possible to sensuously grasp the parking frame on the image around the host vehicle and the distance to other vehicles, and the final display is performed. You can immediately understand the video around your vehicle.

  According to an embodiment, the image processing device 5 displays a schematic diagram of the host vehicle viewed from the position and orientation of a virtual camera for capturing an interpolated virtual image on the display 6 instead of the interpolated virtual image. Therefore, it is easy to understand the switching between the vehicle position and the video on the interpolated virtual video, and the video after the switching can be understood immediately.

It is a figure which shows the structure of one embodiment. It is a figure which shows the installation position of a vehicle-mounted camera. It is a figure which shows the image | video imaged with the vehicle front camera. It is a figure which shows the image | video imaged with the vehicle right side camera. It is a figure which shows the image | video imaged with the vehicle left side camera. It is a figure which shows the image | video imaged with the vehicle rear camera. It is a figure which shows the position and direction of a virtual camera. It is a figure which shows the example of an image | video of virtual camera VA. It is a figure which shows the example of an image | video of virtual camera VB. It is a figure which shows the positional relationship of the own vehicle and virtual camera of 1st Embodiment. It is a figure which shows the change of the position and orientation of a virtual camera at the time of the video switch of 1st Embodiment. It is a figure which shows the example of a video switching display of 1st Embodiment. It is a figure which shows the synthetic | combination image | video of virtual camera VC and VD. It is a figure which shows the positional relationship of the own vehicle and virtual camera of 2nd Embodiment. It is a figure which shows the change of the position and orientation of a virtual camera at the time of the video switch of 2nd Embodiment. It is a figure which shows the example of a video switching display of 2nd Embodiment. It is a figure which shows the positional relationship of the own vehicle and virtual camera of 3rd Embodiment. It is a figure which shows the change of the position and orientation of a virtual camera at the time of image switching of 3rd Embodiment. It is a figure which shows the example of a video switching display of 3rd Embodiment. It is a figure which shows the positional relationship of the own vehicle and virtual camera of 4th Embodiment. It is a figure which shows the change of the position and orientation of a virtual camera at the time of image switching of 4th Embodiment. It is a figure which shows the example of a video switching display of 4th Embodiment. It is a figure which shows the positional relationship of the own vehicle and virtual camera of 5th Embodiment. It is a figure which shows the change of the position and orientation of a virtual camera at the time of image switching of 5th Embodiment. It is a figure which shows the example of a video switching display of 5th Embodiment. It is a figure which shows the example of a video switching display of 6th Embodiment. It is a figure which shows the example of a video switching display of 7th Embodiment. It is a figure which shows the example of a video switching display of 8th Embodiment. It is a figure which shows the example of a video switching display of 9th Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Front camera 2 Right side camera 3 Left side camera 4 Rear camera 5 Image processing apparatus 6 Display

Claims (9)

An actual imaging means mounted on the host vehicle for imaging the periphery of the host vehicle and outputting an actual video;
Image processing means for converting a real video by the real imaging means into a virtual video by a virtual imaging means set at an arbitrary position and orientation;
An image processing apparatus comprising display means for displaying a real video of the real imaging means or a virtual video of the image processing means,
Wherein the image processing means, in the course of switching the display image of the display means from a first real image or virtual image to a second real image or a virtual image, the virtual imaging of the imaging lens and the virtual image pickup means of the actual imaging means By changing the optical characteristics of the lens, the lens is viewed from an intermediate position and orientation between the position and orientation in which the first real image or virtual image is imaged and the position and orientation in which the second real image or virtual image is imaged. An interpolated virtual image is generated and displayed on the display means. The image processing apparatus according to claim 1.
The image processing means captures the first real video or virtual video while switching the display video of the display means from the first real video or virtual video to the second real video or virtual video. And a plurality of interpolated virtual images viewed from a plurality of intermediate positions and orientations from the direction to the position and orientation from which the second real image or virtual image is imaged, and sequentially switching them to the display means An image processing apparatus characterized by displaying. The image processing apparatus according to claim 1 or 2,
The image processing device is characterized in that a plurality of images of the real imaging unit or the virtual imaging unit are combined to generate a real video or a virtual video and displayed on the display unit. The image processing apparatus according to claim 3.
The image processing means switches from a composite image of the plurality of real imaging means or the virtual imaging means to a single virtual image of the virtual imaging means A set in the direction of the host vehicle from a position above the host vehicle. In the middle, a single interpolation virtual image of the virtual imaging means B set in the direction of the host vehicle is generated from a position closer to the host vehicle than the virtual imaging means A, and is displayed on the display means. An image processing apparatus. The image processing apparatus according to claim 3 or 4,
In the case of switching from a composite video of the plurality of real imaging means or the virtual imaging means to a single video of the real imaging means or the virtual imaging means, the position of the single real imaging means or the virtual imaging means And an orientation as an average value of the positions and orientations of the plurality of the real imaging means or the virtual imaging means for imaging the composite video. In the image processing device according to any one of claims 1 to 5,
When displaying the virtual image of the virtual imaging means C set in the direction of the own vehicle at a position close to the own vehicle, the image processing means first detects the virtual image at a position farther from the own vehicle than the virtual imaging means C. The virtual image of the virtual imaging means D set in the direction of the vehicle is displayed, and then the interpolation virtual of the virtual imaging means E set in the intermediate position and orientation between the virtual imaging means D and the virtual imaging means C An image processing apparatus that displays a virtual image of the virtual imaging means C after displaying the image. In the image processing device according to any one of claims 1 to 6,
The image processing means superimposes a schematic view of the host vehicle viewed from the position and orientation of the virtual imaging means for capturing the interpolated virtual image on the interpolated virtual image and displays the superimposed image on the display means. An image processing apparatus. In the image processing device according to any one of claims 1 to 6,
The image processing means converts grid lines (lattice lines) set at equal intervals on the road surface into grid lines viewed from the position and orientation of the virtual imaging means for imaging the interpolation virtual image, and the interpolation virtual An image processing apparatus, wherein the image processing apparatus displays the image on the display unit in a superimposed manner. In the image processing device according to any one of claims 1 to 6,
The image processing means displays a schematic diagram of the host vehicle viewed from the position and orientation of the virtual imaging means for capturing the interpolated virtual video on the display means instead of the interpolated virtual video. Image processing device.

Images