JP6392693B2 - Vehicle periphery monitoring device, vehicle periphery monitoring method, and program - Google Patents

Description

  The present invention relates to a technique for monitoring the periphery of a vehicle with a camera mounted on the vehicle.

  Conventionally, a system (Patent Document 1) that presents an image captured by a camera instead of a side mirror to a driver, or a system that is equipped with wide-angle cameras on the front, rear, left and right sides of a vehicle, synthesizes these images, and presents them to the driver (Patent Document 2). There is also a system that combines an image captured by a side camera and an image captured by a rear camera and presents the situation behind the vehicle to the driver (Patent Document 3).

JP-A-7-223487 JP 2007-183877 A JP 2013-141120 A

  Patent Document 1 describes an invention in which an image captured by an imaging unit is displayed and a region where a moving body is present is highlighted. Although this document discloses that a plurality of television cameras are provided, images taken by the television cameras are displayed on the left and right front pillars and the display of the instrument panel, and the images are not synthesized.

  Patent Document 2 describes an invention in which video images captured by a plurality of in-vehicle cameras are connected to a video image obtained by looking down from a virtual viewpoint above the vehicle. In the invention described in this document, since the positional relationship between the in-vehicle camera and the ground (the angle and the distance of the ground with respect to the in-vehicle camera) is known, it is possible to connect images taken by the in-vehicle cameras. Sometimes, when a vehicle is present in an image taken from behind, the positional relationship between the camera and the vehicle is unknown, so it cannot be connected cleanly.

  The invention described in Document 3 is an invention in which an image photographed by the left and right side cameras and an image photographed by the rear camera are synthesized. In the present invention, the image of the own lane farther than a certain distance (reference line) and the left and right lanes (total 3 lanes) are cut out from the rear camera image and synthesized, so Vehicles existing on the lane (a total of three lanes) can be displayed reliably. However, in the invention described in this document 3, when the vehicle approaches the reference line (for example, 15 m), the vehicle straddles the boundary between the rear image and the side mirror image, so the image is discontinuous. It becomes difficult to see.

  In view of the above background, an object of the present invention is to provide a system capable of presenting a composite image with little discomfort to a driver in a system that combines and displays rear images taken by a plurality of cameras.

  A vehicle periphery monitoring device of the present invention is provided on a side portion of a vehicle, and a side camera that captures the rear, a rear camera that is disposed on the rear of the vehicle and captures the rear at a wide angle, and a side camera that is captured by the side camera. A distance measuring unit that measures a distance to a rear vehicle based on an image and a rear camera image captured by the rear camera, and information on the distance to the rear vehicle, the side camera image and the rear camera image A viewpoint converting unit that performs viewpoint conversion so that the viewpoints match, an image combining unit that combines the side camera image and the rear camera image whose viewpoints match by the viewpoint conversion, and a display unit that displays the combined image With.

  As described above, since the viewpoint conversion is performed using the information on the distance to the rear vehicle so that the viewpoints of the side camera image and the rear camera image coincide with each other, the image of the rear vehicle can be synthesized without a sense of incongruity.

  In the vehicle periphery monitoring device of the present invention, the viewpoint conversion unit converts the rear camera image into an image viewed from the viewpoint of the side camera, and the image composition unit includes the side where the rear vehicle is hidden by a vehicle body. The rear camera image that has undergone the viewpoint conversion may be combined with a region of the camera image, and the display unit may display an image of the vehicle body that is reflected in the side camera image superimposed on the combined image.

  With this configuration, when the rear vehicle is hidden by the body of the host vehicle in the side camera image, it is possible to generate an image of the rear vehicle as seen from the side mirror by performing image composition. Further, when displaying the image of the rear vehicle, the driver can grasp the positional relationship between the host vehicle and the rear vehicle by displaying the vehicle body of the host vehicle in a superimposed manner. In addition, as a method of superimposing and displaying the rear vehicle and the vehicle body of the host vehicle, for example, the host vehicle may be displayed in a semi-transparent manner so that the rear vehicle can be seen through the host vehicle.

  In the vehicle periphery monitoring device of the present invention, the viewpoint conversion unit converts the rear camera image and the side camera image into an image viewed from a viewpoint at a predetermined position on the central axis of the vehicle along the traveling direction of the vehicle. Also good.

  With this configuration, a panoramic image can be generated from the rear camera image and the side camera image.

  In the vehicle periphery monitoring device of the present invention, the distance measuring unit obtains distances to the front and side surfaces of the rear vehicle, and the viewpoint conversion unit is a plane in which the front and side surfaces of the rear vehicle are perpendicular to the road surface. You may perform viewpoint conversion.

  Thus, by treating the front and side surfaces of the rear vehicle as different planes and performing viewpoint conversion using the distances to the respective planes, accurate viewpoint conversion can be performed.

  A vehicle periphery monitoring device of the present invention includes a vehicle shape storage unit that stores external shape information of a vehicle, and stores the external shape information of a vehicle corresponding to a rear vehicle reflected in the rear camera image or the side camera image. The vehicle shape specifying unit searched from the unit and the viewpoint conversion unit may perform viewpoint conversion using external shape information of the vehicle.

  With this configuration, more accurate viewpoint conversion can be performed based on the external shape of the vehicle. In addition, the external shape information of all vehicle types may be stored in the vehicle shape storage unit, but information on the external shape of a typical vehicle may be stored.

  In the vehicle periphery monitoring device of the present invention, the distance measuring unit obtains distances to a plurality of locations on the vehicle body of the rear vehicle (for the part where distances are not obtained at this time, interpolation is performed using the obtained points, etc.) The viewpoint conversion unit may perform the viewpoint conversion so that viewpoints of the side camera image and the rear camera image coincide with each other using information on distances to the plurality of locations.

  With this configuration, it is possible to improve the accuracy of viewpoint conversion by using distances to a plurality of locations on the vehicle body. As the number of points for which the distance is obtained is increased, the external shape of the vehicle can be accurately grasped, and the accuracy of viewpoint conversion can be improved.

  In the vehicle periphery monitoring device according to the present invention, the distance measuring unit includes a distortion correction unit that cuts out a part of the rear camera image to correct distortion, and the rear from either the side camera image or the rear camera image. A feature point search unit that detects a feature point of the vehicle and searches for a point corresponding to the feature point on the other epipolar line of the side camera image or the rear camera image; and a feature point in the side camera image; And a stereo distance measuring unit that obtains a distance to the rear vehicle based on a positional relationship with a corresponding feature point in the rear camera image corresponding thereto.

  With this configuration, it is possible to perform distance measurement by stereo viewing in an area where the fields of view overlap, making use of images captured by side cameras and rear cameras originally mounted for different purposes.

  The vehicle periphery monitoring method of the present invention is a method of displaying an image behind the vehicle, wherein the vehicle periphery monitoring device is provided on a side portion of the vehicle and acquires a side camera image from a side camera that captures the rear. The vehicle periphery monitoring device is provided at the rear of the vehicle, and acquires a rear camera image from a rear camera that captures the rear at a wide angle; and the vehicle periphery monitoring device captures a side camera image captured by the side camera; A step of measuring a distance to a rear vehicle based on a rear camera image photographed by the rear camera, and the vehicle periphery monitoring device using the information on the distance to the rear vehicle, the side camera image and the rear Performing viewpoint conversion so that the viewpoints of the camera images match; and Comprising a step of combining the camera image and the rear camera image, the vehicle periphery monitoring device, and displaying the synthesized image.

  The program of the present invention is a program for displaying an image of the rear of the vehicle, the computer being provided on the side of the vehicle, obtaining a side camera image from a side camera that captures the rear, and the rear of the vehicle A rear camera image obtained from a rear camera that captures the rear at a wide angle, and a distance to the rear vehicle based on the side camera image captured by the side camera and the rear camera image captured by the rear camera. Using the information on the distance to the vehicle behind the vehicle, performing a viewpoint conversion such that the viewpoints of the side camera image and the rear camera image match, and the viewpoints matched by the viewpoint conversion A step of combining the side camera image and the rear camera image and a step of displaying the combined image are performed. Make.

  According to the present invention, since the viewpoint conversion is performed so that the viewpoints of the side camera image and the rear camera image coincide with each other using the information on the distance to the rear vehicle, the image of the rear vehicle can be synthesized without a sense of incongruity. Has an effect.

It is a figure which shows the structure of the vehicle periphery monitoring apparatus of 1st Embodiment. It is a figure which shows the attachment position of a side camera and a rear camera. It is a figure which shows the plane assumed in the case of viewpoint conversion. It is a figure which shows the method of a synthesis | combination of a side camera image and a rear camera image. It is a figure which shows the example of the image which superimposed the image of the vehicle body of the own vehicle on the synthesized image. It is a figure which shows the hardware constitutions of a vehicle periphery monitoring apparatus. It is a flowchart which shows operation | movement of the vehicle periphery monitoring apparatus of 1st Embodiment. It is a figure explaining the background of 2nd Embodiment. It is a figure which shows the structure of the vehicle periphery monitoring apparatus of 2nd Embodiment. It is a figure which shows the example of the data memorize | stored in vehicle shape DB. It is a figure for demonstrating viewpoint conversion in the vehicle periphery monitoring apparatus of 3rd Embodiment. It is a flowchart which shows operation | movement of the vehicle periphery monitoring apparatus of 3rd Embodiment. It is a figure which shows the structure of a ranging part. (A) is a figure showing an example of a rear camera image. (B) is a figure which shows the image which cut out the left half of the rear camera image, and carried out distortion correction. (C) is a figure which shows the image which carried out projective transformation so that a back vehicle may become front direction. It is a figure which shows the example which the back vehicle approached back 5m. (A) is a figure which shows the example of a side camera image, (b) is a figure which shows the rear camera image which performed distortion correction. It is a figure which shows operation | movement of a ranging part. It is a figure which shows the other example of a ranging part. It is a figure which shows operation | movement of the other example of a ranging part.

Hereinafter, a vehicle periphery monitoring device according to an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating a configuration of a vehicle periphery monitoring device 1 according to the first embodiment. The vehicle periphery monitoring device 1 includes a side camera 10 and a rear camera 11 attached to a vehicle, an image photographed by the side camera 10 (referred to as a “side camera image”), and an image photographed by the rear camera 11 (referred to as “ The distance measuring unit 12 for obtaining the distance to the rear vehicle, the viewpoint converting unit 13 for performing image conversion so as to match the viewpoints of the side camera image and the rear camera image, and the viewpoint converted. The image combining unit 14 combines the side camera image and the rear camera image, and the monitor 15 displays the combined image.

  FIG. 2 is a diagram illustrating attachment positions of the side camera 10 and the rear camera 11 to the vehicle. A side camera 10 is attached to a position on the side of the vehicle where the side mirror is located. The side camera 10 is attached obliquely rearward and photographs the rear from the side of the vehicle. By displaying an image photographed by the side camera 10 on the monitor 15 in the vehicle, it is possible to show the driver the scenery reflected in the side mirror. The angle of view of the side camera 10 is generally 90 degrees or less. The rear camera 11 is attached to the rear part of the vehicle. The angle of view of the rear camera 11 is generally wider than the side camera and about 180 degrees at the maximum. Note that the installation positions and postures of the side camera 10 and the rear camera 11 are calibrated, and the relative positions and postures of the rear camera 10 and the side camera 11 are known.

  Returning to FIG. 1, the configuration of the vehicle periphery monitoring device 1 will be described. The distance measurement unit 12 performs stereo distance measurement on the distance from the nearest rear vehicle to, for example, the license plate, using the side camera image and the rear camera image. If the side of the vehicle is visible, the distance to the side can be measured. The configuration of the distance measuring unit 12 will be described in detail later.

  The viewpoint conversion unit 13 converts the side camera image and the rear camera image so that the viewpoints are at the same position. In the present embodiment, the rear camera image is converted into an image viewed from the viewpoint of the side camera. When the rear vehicle is on the right side of the host vehicle, the rear camera image is converted into an image from the viewpoint E1 of the right side camera. When the rear vehicle is on the left side of the host vehicle, the rear camera image is The image is converted into an image from the viewpoint E1 of the side camera.

  When the viewpoint conversion unit 13 performs the viewpoint conversion of the rear camera image, the front and side surfaces of the rear vehicle are planes that stand vertically from the road surface at the position of the distance to the front and side surfaces obtained by the distance measurement unit 12. Assuming that the viewpoint is changed. Here, it is assumed that the rear vehicle and the host vehicle are facing the same direction. That is, the front face of the host vehicle and the front face of the rear vehicle are parallel (the same applies to the side faces). FIG. 3 is a diagram illustrating a plane assumed in the viewpoint conversion. Assume that the front surface of the rear vehicle is plane A and the side surface is plane B. If the distance to one point (for example, the license plate) on the vehicle and the distance to the side surface are known, the plane A and the plane B can be assumed. If only the distance to the front is known, an appropriate vehicle width (for example, 1.8 m) is assumed and the position of the side plane is determined.

  The viewpoint conversion unit 13 converts the rear camera image into an image viewed from the viewpoint E1 of the side camera 10. The viewpoint conversion unit 13 includes a positional relationship (distance and angle) between the side camera 10 and the plane A, a positional relationship (distance and angle) with the plane B, a positional relationship (distance and angle) between the rear camera 11 and the plane A, and a plane. By performing viewpoint conversion using information on the positional relationship (distance and angle) with B, the degree of coincidence between the rear vehicle in the rear camera image after conversion and the rear vehicle in the side camera image is made as high as possible.

  The image synthesis unit 14 synthesizes the side camera image and the rear camera image converted into the viewpoint E1 of the side camera 10. FIG. 4 is a diagram for explaining how to synthesize a side camera image and a rear camera image. For range A, a side camera image is used. The range B is a range that is outside the range of the field of view of the side camera image or is not properly reflected behind the host vehicle even if it is within the range of the field of view. For the range B, the image composition unit 14 synthesizes the rear camera image that has undergone viewpoint conversion. Since the viewpoint conversion of the rear camera image is performed based on the distance to the front and side surfaces of the rear vehicle, the degree of coincidence between the rear vehicle in the side camera image and the rear vehicle in the rear camera image is high, A composite image in which the rear vehicle and the rear vehicle in the rear camera image are clearly connected is obtained.

  The image composition unit 14 superimposes the image of the vehicle body of the host vehicle reflected in the side camera image on the composite image obtained by combining the side camera image and the rear camera image. FIG. 5 is a diagram illustrating an example of an image in which an image of the vehicle body of the host vehicle is superimposed on the composite image. The car body of the host vehicle may be reflected in the conventional side mirror. By displaying the car body of the host vehicle in the same way, the driver can feel the distance to the rear vehicle in the same way as the conventional side mirror. Can be grabbed. Furthermore, since it is possible to see even a portion hidden behind the host vehicle, more information can be obtained from the side camera image than a conventional side mirror.

  FIG. 6 is a diagram illustrating a hardware configuration of the vehicle periphery monitoring device 1. The vehicle periphery monitoring device 1 includes a CPU 20, a RAM 21, a ROM 22, a monitor 15, a hard disk 24, a side camera 10, and a rear camera 11. The ROM 22 stores a program 23 for vehicle periphery monitoring. When the CPU 20 reads and executes the program 23, the vehicle periphery monitoring device 1 described above can be realized. Such a program 23 is also included in the scope of the present invention.

  FIG. 7 is a flowchart showing the operation of the vehicle periphery monitoring device 1 according to the first embodiment. First, the vehicle periphery monitoring device 1 captures the rear camera image by photographing the rear of the host vehicle with the rear camera 11 while capturing the rear camera image by photographing the rear of the host vehicle with the side camera 10 ( S10). Subsequently, the vehicle periphery monitoring device 1 measures the distance to the rear vehicle shown in the side camera image and the rear camera image (S11). A method for measuring the distance to the vehicle behind will be described later.

  Next, the vehicle periphery monitoring apparatus 1 converts the rear camera image into an image viewed from the viewpoint E1 of the side camera 10 (S12). Here, the viewpoint E1 of the left side camera 10 is converted to the viewpoint E1 of the side camera 10 on which the rear vehicle is greatly reflected. Subsequently, the vehicle periphery monitoring device 1 synthesizes the side camera image and the rear camera image subjected to viewpoint conversion (S13). At this time, the vehicle periphery monitoring device 1 combines the rear camera image with a portion that cannot be covered by the field of view of the side camera image.

  Next, the vehicle periphery monitoring apparatus 1 superimposes an image of the vehicle body of the host vehicle that is reflected in the side camera image on the synthesized image (S14). For example, the vehicle body of the host vehicle is made translucent and superimposed so that the scenery behind the vehicle including the vehicle behind can be seen. And the vehicle periphery monitoring apparatus 1 displays the superimposed image on the monitor 15 (S15).

  The configuration and operation of the vehicle periphery monitoring device 1 according to the first embodiment have been described above. The vehicle periphery monitoring device 1 according to the first embodiment uses the information on the distance to the rear vehicle to convert the rear camera image into an image viewed from the viewpoint E1 of the side camera 10, so that the rear shown in the side camera image The degree of coincidence between the image of the vehicle and the image of the rear vehicle in the rear camera image after conversion is increased, and when the two images are combined, the image of the rear vehicle can be combined without a sense of incongruity.

  Further, the vehicle periphery monitoring device 1 according to the first embodiment assumes that the front surface of the rear vehicle is the plane A and the side surface is the plane B, and information on the positional relationship (distance and angle) with the plane A and the plane B. By using it, it is possible to synthesize both the front and side surfaces of the rear vehicle without a sense of incongruity with a simple configuration.

(Second Embodiment)
Next, the vehicle periphery monitoring apparatus 2 of the 2nd Embodiment of this invention is demonstrated. The basic configuration of the vehicle periphery monitoring device 2 of the second embodiment is the same as that of the first embodiment, but in the vehicle periphery monitoring device 1 of the first embodiment, the front and side surfaces of the rear vehicle. The vehicle periphery monitoring device 2 according to the second embodiment is different in that viewpoint conversion is performed using information on the external shape of the rear vehicle.

  FIG. 8 is a diagram for explaining the background of the second embodiment. The front and side surfaces of the vehicle are not actually flat surfaces. For example, as shown in FIG. 8, the distance L5 from the rear camera 11 to the windshield of the rear vehicle and the distance L6 to the bumper are different. Of course, the same can be said for the side camera 10. Assuming that the front and side surfaces of the rear vehicle are completely flat, there is a possibility that the viewpoint conversion may be slightly affected. In the second embodiment, the external shape of the vehicle is grasped for each type of vehicle, and the external shape is determined. The viewpoint is converted using distance information obtained from the above information.

  FIG. 9 is a diagram illustrating a configuration of the vehicle periphery monitoring device 2 according to the second embodiment. The vehicle periphery monitoring device 2 includes a vehicle shape database (hereinafter referred to as “vehicle shape DB”) 16 in addition to the configuration of the vehicle periphery monitoring device 1 of the first embodiment. Moreover, the viewpoint conversion unit 13 includes a vehicle shape specifying unit 17 that specifies the vehicle shape of the rear vehicle.

  FIG. 10 is a diagram illustrating an example of data stored in the vehicle shape DB 16. The vehicle shape DB 16 stores vehicle type and external shape information. The vehicle shape specifying unit 17 of the viewpoint conversion unit 13 specifies the vehicle shape by searching the vehicle shape DB 16 for the vehicle type most similar to the image of the rear vehicle. In addition, the vehicle shape specific | specification part 17 may receive the vehicle type information from a back vehicle by vehicle-to-vehicle communication, and may specify the vehicle shape corresponding to the said vehicle type.

  The viewpoint conversion unit 13 obtains the distance from the side camera 10 and the rear camera 11 to each position of the rear vehicle (for example, a bumper, a hood, a windshield, etc.) based on the external shape information of the rear vehicle. Viewpoint conversion is performed based on the distance information. The process of combining the side camera image and the rear camera image after the viewpoint conversion and superimposing the image of the vehicle body of the host vehicle on the combined image is the same as in the first embodiment.

  Since the vehicle periphery monitoring device 2 according to the second embodiment performs image processing for converting the viewpoint using the external shape information of the rear vehicle, the difference in distance caused by the external shape of the vehicle body of the rear vehicle is also taken into consideration. Thus, when the degree of coincidence between the image of the rear vehicle reflected in the side camera image and the image of the rear vehicle reflected in the rear camera image is further increased, and the side camera image and the rear camera image are combined, an image of the rear vehicle that does not feel uncomfortable is obtained. Can be generated.

(Third embodiment)
Next, a vehicle periphery monitoring device according to a third embodiment of the present invention will be described. The basic configuration of the vehicle periphery monitoring device according to the third embodiment is the same as that of the vehicle periphery monitoring device 1 according to the first embodiment (see FIG. 1), but the vehicle according to the third embodiment. The periphery monitoring device is different in that the left and right side camera images and the rear camera image are combined and the rear panoramic image is combined.

  FIG. 11 is a diagram for explaining viewpoint conversion in the vehicle periphery monitoring device according to the third embodiment. As shown in FIG. 11, in the third embodiment, the left and right side camera images and the rear camera image are converted into images with a predetermined point E3 on the central axis 18 in the traveling direction of the vehicle as a viewpoint. At this time, the viewpoint conversion in consideration of the distance to the nearest rear vehicle is the same as in the first embodiment.

  FIG. 12 is a flowchart illustrating the operation of the vehicle periphery monitoring device according to the third embodiment. The vehicle periphery monitoring apparatus according to the third embodiment first captures the rear of the host vehicle by capturing the rear camera 11 with the side camera 10 and acquires the rear of the host vehicle with the rear camera 11. A camera image is acquired (S20). Subsequently, the vehicle periphery monitoring device measures the distance to the rear vehicle shown in the side camera image and the rear camera image (S21). A method for measuring the distance to the vehicle behind will be described later.

  Next, the vehicle periphery monitoring device converts the rear camera image and the left and right side camera images into an image viewed from the viewpoint of the predetermined point E3 (S22). Subsequently, the vehicle periphery monitoring device synthesizes the left and right side camera images and the rear camera image that have undergone viewpoint conversion, and generates a panoramic image behind the vehicle (S23). Here, as a method for synthesizing a panoramic image, for example, the method of Reference 3 can be considered for a case where the vehicle is not reflected or a range where the vehicle is not reflected (the rear camera image is used for the three lanes far from the reference line). Use). Regarding the range in which the vehicle is shown, a method of giving priority to the rear camera image and interpolating a portion not shown by the rear camera from the side camera image can be considered. Then, the vehicle periphery monitoring device displays the superimposed image on the monitor 15 (S24).

  The configuration and operation of the vehicle periphery monitoring device according to the third embodiment have been described above. The vehicle periphery monitoring apparatus according to the third embodiment can make the driver easily grasp the state of the rear by synthesizing the left and right side camera images and the rear camera image to generate and display a panoramic image. . In addition, since the information on the distance to the rear vehicle is used when the images are combined, a panoramic image including the rear vehicle image without any sense of incongruity can be provided.

[Rangefinder]
FIG. 13 is a diagram illustrating a configuration of the distance measuring unit 12. The distance measuring unit 12 includes a distortion correction unit 31 that corrects distortion of the rear camera image acquired by the rear camera 11, and a projection conversion unit 33 that performs projective conversion of the rear camera image so that the rear vehicle reflected in the image faces front. And have. The distortion correction unit 31 and the projective conversion unit 33 perform preprocessing for appropriately performing distance measurement using a wide-angle rear camera image.

  FIG. 14A to FIG. 14C are diagrams for explaining the correction of the rear camera image by the distortion correction unit 31 and the projective conversion unit 33. FIG. 14A is a diagram illustrating an example of the rear vehicle reflected in the rear camera image. As described above, the rear camera 11 captures images at a wide angle of about 180 degrees, so that a large distortion occurs at the peripheral edge of the image.

  The distortion correction unit 31 corrects distortion by cutting out half of the rear camera image, as shown by being surrounded by a thick line in FIG. FIG. 14B is a diagram illustrating a rear camera image subjected to distortion correction. The image in FIG. 14A is an image with an angle of view of about 180 degrees, but the image in FIG. 14B after distortion correction corresponds to an image with an angle of view of 40 degrees. Note that the right side of the rear camera image is cut out by the distortion correction unit 31 when the distance measurement is performed in cooperation with the right side camera 10, and the left side camera is cut out. When the distance measurement is performed in conjunction with 10, the left half of the rear camera image is cut out.

  The distance measuring unit 12 includes a distance estimating unit 32 that calculates an approximate distance (approximate distance) to the rear vehicle. The distance estimation part 32 calculates | requires the approximate distance to a back vehicle according to a well-known technique. For example, from the side camera image, the lateral edge existing in the lane is used, or the front of the vehicle is detected using a discriminator that has learned the image of the front of the vehicle in advance, and the lower end of the vehicle is used. The lower end of the rear vehicle is detected by the method. And the ranging part 12 calculates | requires the rough distance to a back vehicle based on the position of the detected lower end. Since the installation position and orientation of the side camera 10 are calibrated in advance and the road surface position can be calculated, the distance from the position of the lower end in the side camera image to the side camera 10 can be calculated.

  When the vehicle behind you is driving in the next lane or when you are overtaking, such as when you are not behind the vehicle, the vehicle behind you appears in the image captured by the rear camera image. As it gets closer to the vehicle, it appears in a direction rotated from the front, and the ratio of the side of the vehicle being visible increases. For example, in the image shown in FIG. 14B, the rear vehicle is shown rotating in a direction in which the left side can be seen. In this case, since it becomes difficult to appropriately match the feature points appearing in the image, the projective conversion unit 33 performs projective conversion so that the rear vehicle faces front.

  The projection conversion unit 33 cuts out an image of the front surface of the vehicle including feature points such as an emblem and a license plate from the rear camera image, as shown by surrounding the thick line in FIG. 14B. Here, it is assumed that the front surface of the vehicle is a plane. FIG. 15 is a diagram illustrating an example in which the rear vehicle approaches the rear 5 m. In FIG. 15, a straight line A is a line indicating a plane including the front surface of the rear vehicle. In the image photographed by the rear camera 11, the plane is reflected at an angle α from the front. The projective conversion unit 33 performs projective conversion on the clipped image so that the rear vehicle becomes an image facing the front, that is, the plane including the front surface of the rear vehicle is perpendicular to the shooting direction of the rear camera 11. In the example of FIG. 15, it is rotated by an angle α and converted into a plane as shown by a straight line B. When the distance measuring unit 12 estimates the distance to the rear vehicle, if the lower end of the rear vehicle or the front area of the vehicle is found, it may be used. Thereby, you may perform a corresponding point search after that using the wide area | region of the vehicle front.

  The projection conversion unit 33 uses data of an approximate distance to the rear vehicle in order to obtain the amount by which the front surface of the rear vehicle reflected in the rear camera image is tilted from the front. As described above, when the rear vehicle is not directly behind the host vehicle, the rear vehicle is traveling on the adjacent lane or trying to pass the host vehicle. It is about several meters away in the direction perpendicular to the axis. Therefore, if the approximate distance to the rear vehicle is known, the approximate direction of the rear vehicle can be known, and the degree of rotation of the rear vehicle can be grasped.

  The distance measuring unit 12 stereo-measures the distance to the rear vehicle based on the feature point searching unit 34 that searches for the corresponding feature point included in the side camera image and the rear camera image, and based on the position of the searched feature point. And a stereo distance measuring unit 35.

  The feature point search unit 34 detects a feature point of the rear vehicle from the side camera image. As the feature point, for example, an emblem, a license plate, a lamp, or the like can be used. In the case of nighttime, the lamp is effective. Subsequently, the feature point search unit 34 searches for feature points corresponding to the detected feature points from the rear camera image. At this time, the feature point search unit 34 obtains an epipolar line that is a straight line in which the detected feature point may exist in the rear camera image, and searches for the corresponding feature point on the epipolar line. Further, the feature point search unit 34 receives data of the approximate distance from the distance estimation unit 32 to the vehicle behind, obtains in which range the feature point exists based on the approximate distance, and searches on the epipolar line in the range. . Thereby, the process of searching for the corresponding feature point can be remarkably reduced.

  A method of searching for a corresponding feature point by the feature point search unit 34 is performed by comparing an image of a predetermined unit region including the feature point in the side camera image with an image of the predetermined unit region of the rear camera image. When the image similarity is higher than a predetermined threshold, it is determined that the image is a corresponding feature point (including a unit region). At this time, it is preferable that the unit area of the rear camera image is larger than the unit area of the side camera image, and the image of the unit area cut out from the rear camera image is reduced to the same size as the side camera image before comparison. . This is because the rear camera installed behind the side camera is closer to the rear vehicle, and the rear camera image shows the rear vehicle larger than the side camera image. At this time, the size of each unit area can be determined based on the approximate distance to the rear vehicle.

  In the description here, the procedure for detecting the feature point of the vehicle from the side camera image and searching for the feature point corresponding to the detected feature point from the rear camera image is described as an example. The feature point of the vehicle may be detected from the rear camera image, and the feature point corresponding to the detected feature point may be searched from the side camera image.

  The stereo distance measuring unit 35 measures the distance to the rear vehicle in stereo based on the positional relationship between the corresponding feature points. FIG. 16A is a diagram illustrating an example of a side camera image, and FIG. 16B is a diagram illustrating a rear camera image corrected by the distortion correction unit 31 to an image corresponding to an angle of view of 40 degrees. As shown in FIGS. 16A and 16B, an emblem was detected as a feature point from the side camera image and the rear camera image. As described above, the relative relationship between the installation positions and orientations of the side camera 10 and the rear camera 11 is known. The stereo distance measuring unit 35 calculates the distance to the rear vehicle based on the principle of triangulation based on the position of the emblem in the image and the mounting positions and mounting directions of the side camera 10 and the rear camera 11.

  FIG. 17 is a flowchart showing the operation of the distance measuring unit 12. First, the distance measuring unit 12 captures the vehicle periphery with the side camera 10 and the rear camera 11, and inputs the captured side camera image and rear camera image to the distance measuring unit 12 (S30). This step is the same as the step (S10, FIG. 7) of acquiring the side camera image and the rear camera image in the vehicle periphery monitoring apparatus 1 described above.

  When the distance measuring unit 12 receives the input of the rear camera image, the distortion correcting unit 31 cuts out half of the rear camera image and corrects the distortion of the cut out image (S31). As a result, the rear camera image is substantially corrected to an image having an angle of view of 40 degrees. Subsequently, the distance measuring unit 12 calculates an approximate distance to the rear vehicle shown in the side camera image (S32).

  Next, the projective transformation unit 33 calculates how much the rear vehicle is reflected from the front direction using the data of the approximate distance obtained in step S12, and uses the obtained amount of inclination to calculate whether the rear vehicle is The rear camera image is projectively transformed so as to face the front (S33).

  Subsequently, the feature point search unit 34 searches for a feature point of the rear vehicle from the side camera image (S34), and searches for a feature point corresponding to the searched feature point from the rear camera image (S35). When the corresponding feature point is searched, the distance to the rear vehicle is calculated based on the positional relationship between the feature points shown in the side camera image and the rear camera image (S36). The distance measuring unit 12 determines whether or not the rear vehicle is closer than a predetermined threshold based on the obtained distance data, and issues a warning to the driver when the rear vehicle is closer than the predetermined threshold. You may perform the process of.

  The distance measuring unit 12 appropriately searches for a corresponding feature point with the image captured by the side camera 10 by correcting the distortion of the image captured by the wide-angle rear camera 11 by the distortion correcting unit 31. be able to. In addition, since the distance measuring unit 12 performs the projective transformation so that the rear vehicle reflected in the rear camera image faces the front, it is possible to improve the accuracy of the feature point search. Thereby, the distance to a back vehicle can be stereo-measured using the existing hardware called the side camera 10 and the rear camera 11. FIG.

[Another example of distance measuring unit]
FIG. 18 is a diagram illustrating another example of the distance measuring unit 12. Although the basic configuration of the distance measuring unit 12 is the same as that described above, in this example, the distance measuring unit 12 includes a parallelization correcting unit 36 instead of the distance estimating unit 32 and the projective conversion unit 33. ing.

  The parallelization correction unit 36 is configured so that the side camera image and the rear camera image are parallel to each other so that the epipolar lines of the side camera image and the rear camera image are parallel based on the position where the side camera 10 and the rear camera 11 are attached and the shooting direction. It has a function to correct. In this way, by obtaining an image in which epipolar lines are parallel by the collimation correction unit 36, when searching for a feature point corresponding to a feature point detected in one image, the other image is set in the horizontal direction. Since the search may be performed, the feature point search processing by the feature point search unit 34 can be reduced.

  FIG. 19 is a flowchart showing the operation of another example of the distance measuring unit 12. First, the distance measuring unit 12 captures the vehicle periphery with the side camera 10 and the rear camera 11, and inputs the captured side camera image and rear camera image to the distance measuring unit 12 (S40). This step is the same as the step (S10, FIG. 7) of acquiring the side camera image and the rear camera image in the vehicle periphery monitoring apparatus 1 described above.

  When the distance measuring unit 12 receives the input of the rear camera image, the distortion correcting unit 31 cuts out half of the rear camera image and corrects the distortion of the cut out image (S41). As a result, the rear camera image is substantially corrected to an image having an angle of view of 40 degrees.

  Next, the parallelization correction unit 36 performs a process of converting the image so that the epipolar lines of the side camera image and the rear camera image whose distortion is corrected are parallel to each other (S42).

  Subsequently, the feature point search unit 34 searches for a feature point of the rear vehicle from the side camera image (S43), and searches for a feature point corresponding to the searched feature point from the rear camera image (S44). When the corresponding feature point is searched, the distance to the rear vehicle is calculated from the positional relationship between the feature points shown in the side camera image and the rear camera image (S45).

  The configuration of the distance measuring unit 12 has been described above. In the above description, the example in which the rear camera image is projectively transformed has been described. However, the side camera image may be subjected to projective transformation in the same manner as the rear camera image.

  Further, the projective conversion unit 33 may determine whether or not to perform the projective conversion based on the amount of inclination from the front of the rear vehicle. That is, when the amount of inclination is smaller than a predetermined threshold, projective transformation may not be performed. Note that the projective transformation unit 33 is not an essential component and may be omitted depending on circumstances.

  In this example, the example in which the size of the unit area serving as a comparison unit is adjusted when the feature point search unit 34 searches for the corresponding feature point in the side camera image 10 and the rear camera image 11 has been described. May be performed simultaneously with the projective transformation. Further, when block matching is performed by the feature point search unit 34, the similarity may be calculated while changing the scale including the scale parameter, and the result having the highest similarity may be adopted.

  As mentioned above, although the structure of the vehicle periphery monitoring apparatus which concerns on embodiment of this invention was demonstrated, this invention is not limited to above-described embodiment. In the above-described embodiment, the viewpoint conversion unit has been described by taking an example of performing line-of-sight conversion based on the external shape information of the rear vehicle. However, the viewpoint conversion unit obtains distances to a plurality of locations on the vehicle body of the rear vehicle, and The viewpoint conversion may be performed using the information on the distance to the location so that the viewpoints of the side camera image and the rear camera image coincide. Thus, even if the external shape information of the vehicle is not included, the rear vehicle image and the rear camera image reflected in the side camera image are also taken into account in consideration of the difference in distance caused by the difference in the external shape of the rear vehicle body. It is possible to further increase the degree of coincidence of the image of the rear vehicle reflected on the vehicle.

  The present invention is useful as an apparatus for monitoring the periphery of a vehicle based on an image captured by a camera.

DESCRIPTION OF SYMBOLS 1, 2 Vehicle periphery monitoring apparatus 10 Side camera 11 Rear camera 12 Distance measuring part 15 Monitor 16 Vehicle shape database 17 Vehicle shape specific | specification part 18 Center axis 20 of CPU CPU
21 RAM
22 ROM
23 Program 24 Hard disk 31 Distortion correction unit 32 Distance estimation unit 33 Projection conversion unit 34 Feature point search unit 35 Stereo distance measurement unit 36 Parallelization correction unit

Claims (9)

A side camera that is provided on the side of the vehicle and that captures the rear;
A rear camera which is provided at the rear of the vehicle and shoots the rear at a wide angle;
A distance measuring unit that measures a distance and an angle to the front surface of the rear vehicle based on a side camera image captured by the side camera and a rear camera image captured by the rear camera;
As a plane perpendicular to the front and side of the road surface of the rear vehicle, based on the distance and angle to the front, as the viewpoint of the side camera image and the rear camera images match, the side camera image and / Or a viewpoint conversion unit for converting the rear camera image ,
An image synthesis unit that synthesizes the side camera image and the rear camera image whose viewpoints are matched by the image conversion;
A display for displaying the synthesized image;
A vehicle periphery monitoring device. A side camera that is provided on the side of the vehicle and that captures the rear;
  A rear camera which is provided at the rear of the vehicle and shoots the rear at a wide angle;
  A vehicle shape storage unit storing external shape information of the vehicle;
  A distance measuring unit that measures a distance and an angle to a rear vehicle based on a side camera image captured by the side camera and a rear camera image captured by the rear camera;
  A vehicle shape identifying unit for retrieving external shape information of a vehicle corresponding to a rear vehicle reflected in the rear camera image or the side camera image from the vehicle shape storage unit;
  Using the information on the distance and angle to each position of the rear vehicle obtained from the external shape information of the vehicle, the side camera image and / or the side camera image and / or the rear camera image so that the viewpoints of the side camera image and the rear camera image coincide with each other. A viewpoint conversion unit for converting the rear camera image;
  An image synthesis unit that synthesizes the side camera image and the rear camera image whose viewpoints are matched by the image conversion;
  A display for displaying the synthesized image;
  A vehicle periphery monitoring device. The viewpoint conversion unit converts the rear camera image into an image viewed from the viewpoint of the side camera,
The image synthesizing unit synthesizes the rear camera image that has been subjected to the image conversion into an area of the side camera image in which the rear vehicle is hidden by a vehicle body, and is further reflected in the side camera image in the synthesized image. The vehicle periphery monitoring device according to claim 1, wherein an image of the vehicle body is superimposed.   The vehicle periphery according to claim 1 or 2, wherein the viewpoint conversion unit converts the rear camera image and the side camera image into an image viewed from a viewpoint at a predetermined position on a central axis of the vehicle along a traveling direction of the vehicle. Monitoring device. The distance measuring unit is
A distortion correction unit that cuts out a part of the rear camera image to correct distortion;
A feature point of the rear vehicle is detected from either the side camera image or the rear camera image, and a point corresponding to the feature point is searched from the other epipolar line of the side camera image or the rear camera image. A feature point search unit;
A stereo distance measuring unit for obtaining a distance and an angle to the rear vehicle based on a positional relationship between the feature points in the side camera image and the corresponding feature points in the rear camera image,
A vehicle periphery monitoring device according to any one of claims 1 to 4. A vehicle periphery monitoring method for displaying an image behind the vehicle,
A vehicle periphery monitoring device is provided on a side of the vehicle, and acquires a side camera image from a side camera that captures the rear; and
The vehicle periphery monitoring device is provided at the rear of the vehicle, and acquires a rear camera image from a rear camera that captures the rear at a wide angle;
The vehicle periphery monitoring device measuring a distance and an angle to the front surface of the rear vehicle based on a side camera image captured by the side camera and a rear camera image captured by the rear camera;
The vehicle periphery monitoring device assumes that the front and side surfaces of the rear vehicle are planes perpendicular to the road surface, and the viewpoints of the side camera image and the rear camera image match based on the distance and angle to the front surface. And converting the side camera image and / or the rear camera image ,
The vehicle periphery monitoring device combining the side camera image and the rear camera image whose viewpoints coincide with each other by the image conversion;
The vehicle periphery monitoring device displaying a synthesized image;
A vehicle periphery monitoring method comprising: A vehicle periphery monitoring method for displaying an image behind the vehicle,
  A vehicle periphery monitoring device is provided on a side of the vehicle, and acquires a side camera image from a side camera that captures the rear; and
  The vehicle periphery monitoring device is provided at the rear of the vehicle, and acquires a rear camera image from a rear camera that captures the rear at a wide angle;
  The vehicle periphery monitoring device measuring a distance and an angle to a rear vehicle based on a side camera image captured by the side camera and a rear camera image captured by the rear camera;
  The vehicle periphery monitoring device retrieves external shape information of a vehicle corresponding to a rear vehicle reflected in the rear camera image or the side camera image from a vehicle shape storage unit that stores the external shape information of the vehicle;
  The vehicle periphery monitoring device uses the distance and angle information to each position of the rear vehicle obtained from the external shape information of the vehicle so that the viewpoints of the side camera image and the rear camera image coincide with each other. Converting the side camera image and / or the rear camera image;
  The vehicle periphery monitoring device combining the side camera image and the rear camera image whose viewpoints coincide with each other by the image conversion;
  The vehicle periphery monitoring device displaying a synthesized image;
  A vehicle periphery monitoring method comprising: A program for displaying an image of the rear of the vehicle,
A step of obtaining a side camera image from a side camera that is provided on a side of the vehicle and shoots the rear;
Obtaining a rear camera image from a rear camera provided at the rear of the vehicle and photographing the rear at a wide angle;
Measuring the distance and angle to the front of the rear vehicle based on the side camera image captured by the side camera and the rear camera image captured by the rear camera;
As a plane perpendicular to the front and side of the road surface of the rear vehicle, based on the distance and angle to the front, as the viewpoint of the side camera image and the rear camera images match, the side camera image and And / or converting the rear camera image .
Combining the side camera image and the rear camera image whose viewpoints coincide with each other by the image conversion;
Displaying the synthesized image; and
A program that executes A program for displaying an image of the rear of the vehicle,
  A step of obtaining a side camera image from a side camera that is provided on a side of the vehicle and shoots the rear;
  Obtaining a rear camera image from a rear camera provided at the rear of the vehicle and photographing the rear at a wide angle;
  Measuring the distance and angle to the rear vehicle based on the side camera image captured by the side camera and the rear camera image captured by the rear camera;
  Searching external shape information of a vehicle corresponding to a rear vehicle reflected in the rear camera image or the side camera image from a vehicle shape storage unit storing external shape information of the vehicle;
  Using the information on the distance and angle to each position of the rear vehicle obtained from the external shape information of the vehicle, the side camera image and / or the side camera image and / or the rear camera image so that the viewpoints of the side camera image and the rear camera image coincide with each other. Converting the rear camera image into an image;
  Combining the side camera image and the rear camera image whose viewpoints coincide with each other by the image conversion;
  Displaying the synthesized image; and
  A program that executes