JP4682664B2 - Perimeter monitoring system - Google Patents

Description

  The present invention relates to a periphery monitoring system that displays a captured image of a periphery of a vehicle captured by a camera installed in the vehicle on a display device in a vehicle interior.

  In recent years, with the widespread use of navigation systems, the number of vehicles equipped with a monitor system that displays the situation around the vehicle on a display device in the passenger compartment is increasing. Examples of the situation around the vehicle include the position of other vehicles, the situation of obstacles, and road markings such as a center line and a stop line. When the vehicle is to be turned back, the surroundings are displayed on the display device such as a back monitor for monitoring the rear, a bumper under the front bumper, and a blind spot at the corner. Such a monitor system is intended to reduce the burden on the driver when driving the vehicle. In view of the large amount of information that the driver needs to acquire during driving, the monitor system is excellent in visibility and looks comfortable. Then, it is desirable to display an image from which the driver can acquire an image from which necessary information can be acquired. Of course, it is desirable that the field of view of the image is wide. In general, the human static field of view is about 200 degrees with one eye gazing at one point. Of these, it is said that it can be confirmed up to about 70 degrees including colors such as red, blue and yellow. Further, the moving object visual field when the object is viewed while moving is narrowed according to the moving object speed, and the vehicle speed is about 40 km / h, which is about 100 degrees, which is half of the static object visual field. To compensate for this, the driver usually tries to keep a wide viewing angle without gazing at a single point. However, the monitoring system as described above favors such human physiological limits and efforts. It is desired that it be stored.

  However, in general, the viewing angle of an image taken by a camera or the like is as narrow as about 50 to 65 degrees, and a sufficient viewing angle cannot be obtained simply by installing the camera in a vehicle. Various methods have been proposed to compensate for this. For example, as shown in FIGS. 15 to 16, there is a method in which a place that can be a blind spot from the driver's seat is photographed with priority, and this is displayed in different display frames of the display device.

  Patent Document 1 discloses one or several cameras, means for converting an image input from the camera into other coordinates by perspective transformation, and this converted image in relation to an image of the own vehicle. There has been proposed one having a means for synthesizing a single image and a display (display device) for displaying this image to an occupant (driver). The perspective transformation implemented in this document is the transformation from the screen image of the camera into road surface (plane) coordinates with the vehicle center as the origin, the left and right sides with respect to the vehicle traveling direction as the X axis, and the vehicle traveling direction as the Y axis. To do. Then, the synthesized image shows the own vehicle shown in the illustration, the other vehicle, the road sign, the object, etc., which are perspective-transformed, shown in plane coordinates, and this is displayed on the display device.

Japanese Patent Laid-Open No. 3-99952 (Page 2, FIGS. 1-3)

  Perspective transformation is also called viewpoint transformation, but taking the technique of Patent Document 1 as an example, screen coordinates viewed from a direction parallel to the XY plane composed of the X axis and the Y axis are expressed with respect to the XY plane. This is converted into planar coordinates viewed from the orthogonal direction. Therefore, an object standing perpendicular to the XY plane, such as a pole or a wall, is clearly visible in the screen coordinates, but in the plane coordinates, it is aggregated into points, lines, etc. It will not be possible or displayed as an image that makes the driver feel very uncomfortable. Even if it is not as extreme as this example, if an image taken from a certain viewpoint in a three-dimensional space is converted into a flat image, a solid object that is not on the conversion target plane will not be converted correctly, and the driver will feel uncomfortable. It becomes an image.

  In addition, a method of focusing on a place where a blind spot can be formed from the driver's seat as shown in FIGS. 15 to 16 without converting to plane coordinates, and displaying this in a separate display frame of the display device is employed. In this case, since the image is from a normal viewpoint, there is little discomfort with respect to each image. However, when divided and displayed as shown in FIG. 16, the driver cannot fully grasp the environment unless he / she is aware of the position of the correlation between the captured images.

  In order to solve this problem, the present applicants, in Japanese Patent Application No. 2004-281910, synthesize a captured image captured by a plurality of cameras installed in a vehicle and display it as a single image on a display device. In addition, a technique has been proposed in which a driver can grasp the situation around the vehicle at a glance in a wide range on a display device by correcting a horizontal or vertical shift that occurs between the boundaries of adjacent images.

  The above-described periphery monitoring system proposed by the present applicants will be specifically described with reference to an example in which the vehicle 100 exits from the alley as shown in FIG. At the intersection shown in FIG. 13, there is a street tree 112 on the right side of the stop line 111 drawn on the alley. There is a building 113 across the main street of the alley. There is a pedestrian crossing 114 on the left side of the stop line 111, and a person 115 crosses the pedestrian crossing 114. There is an installation 116 at the center of the intersection. When the driver stops the vehicle 100 at the position of the stop line 111 and confirms left and right, the driver's seat is provided at a position behind the front bumper, and thus it is difficult to confirm the left and right direction of the main street. In the vehicle 100 that is not equipped with the periphery monitoring system, the driver alerts vehicles, bicycles, motorcycles, and the like traveling on the main street while moving the vehicle 100 little by little from the alley. After moving forward to the visible position, check the left and right and merge as you can. At the intersection shown in FIG. 13, the roadside tree 112 blocks the right field of view of the driver in the drawing, so that the driver increases the amount of forward movement of the vehicle 100 in order to confirm the left-right direction.

  In this regard, in the case of the vehicle 100 equipped with the periphery monitoring system proposed by the present applicants, two cameras installed so that the viewing angles partially overlap the front bumper of the vehicle 100, And the situation in the right direction is captured, and the composite image obtained by combining the two captured images after adjusting the deviation between the horizontal direction and the vertical direction is displayed on the display device, so that the vehicle 100 is not moved forward. However, it is possible to grasp the situation in the left-right direction as a single image. At this time, the image of the display device is displayed without shifting the composite boundary position of the composite image in which the installation object 116 or the building 113 shown in FIG. 13 is displayed, and it is difficult for the driver to feel uncomfortable.

  At this time, if the deformation of the captured image caused by the distortion of the lens or the imaging device is combined without correction, the situation shown in FIG. 13 is captured and displayed on the display device as shown in FIG. The person 115 and the road width reflected at the edge of the image are displayed extremely small. In this case, it is difficult to distinguish a vehicle, a bicycle, a motorcycle, or the like running along the main street toward the host vehicle.

  On the other hand, for example, as shown in the projection view of FIG. 18, an image captured with the optical axis L of the camera directed to the left by θ5 degrees with respect to the target surface is directed frontward with the direction of the target surface facing right by θ5 If the viewpoint is converted into an image such as that shown in FIG. 19, the side surface of the imaging object that was shown at the end of the captured image can be enlarged and displayed, as shown in FIG. When this viewpoint conversion is adopted in the periphery monitoring system proposed by the present applicants, when the situation shown in FIG. 13 is captured by two cameras and a composite image is displayed on the display device, the display device is shown in FIG. The person 115 and the main street reflected on the edge of the screen can be enlarged and displayed.

  However, in actuality, the image of the person 115 and the actual image data reflected at the edge of the captured image is diffused and becomes an abnormally large image, and it is difficult to understand what is reflected because the image is blurred. The object is displayed too large, and the driver feels uncomfortable. For example, as shown in FIG. 21, the captured image P1 of the camera in which the optical axis L is moved from the front of the vehicle 100 to the left by θ5 degrees is two-dimensional, so that it faces the front of the vehicle 100. In the image P2 obtained by converting the viewpoint, the object shown at the end of the captured image is not treated as a three-dimensional object on the image, and the camera actually captures the captured image as if it were written on the same plane as the object in front. This is considered to be an image as if it was taken close to the object reflected at the edge. In the peripheral monitoring system, as described above, it is important to display a left-right image without a sense of incongruity when checking automobiles, bicycles, motorcycles, etc. that run from the left-right direction with respect to the vehicle 100. It is not preferable to reduce the resolution by enlarging the image too much.

  Therefore, the present invention has been made to solve the above-described problems, and an object of the present invention is to provide a periphery monitoring system that can display a situation around a vehicle on a display unit without a sense of incongruity.

The periphery monitoring system according to the present invention has the following configuration.
(1) In a periphery monitoring system that captures an image of a situation around a vehicle by an image capturing unit installed in the vehicle, performs image processing on the captured image and displays the image on a display unit, the captured image is divided into a plurality of regions, And a viewpoint conversion unit that performs viewpoint conversion by continuously moving the virtual viewpoint direction of the imaging unit in a direction different from the optical axis of the imaging unit.

(2) In the invention described in (1), the camera includes a storage unit that stores a correspondence relationship between the captured image and the virtual viewpoint direction, and the viewpoint conversion unit stores the captured image stored in the storage unit. And viewpoint conversion based on the correspondence between the virtual viewpoint direction and the virtual viewpoint direction.
(3) In the invention described in (1) or (2), the viewpoint conversion unit continuously moves the virtual viewpoint direction from one side of the captured image to the other side.
(4) In the invention described in (1) or (2), the viewpoint conversion unit changes the virtual viewpoint direction of the captured image based on a viewpoint conversion map.

(5) In the invention according to any one of (1) to (4), when the imaging unit includes a plurality and there are overlapping ranges where the captured images overlap, a composite boundary position is determined within the overlapping range. And combining means for combining the captured images.

  The periphery monitoring system of the present invention having the above configuration divides a captured image captured by the imaging unit into a plurality of regions, and continuously sets the virtual viewpoint direction of the imaging unit in a plurality of different directions from the optical axis of the imaging unit for each region. The viewpoint of the imaging means is displayed on the display means as if the driver saw the situation of the surroundings of the vehicle facing each direction in the display means. The image data of the subject in the place away from the camera will not be displayed small on the display means due to distortion of the lens or imaging device, or will not be diffused and blurred during viewpoint conversion, or will not be enlarged unnecessarily The situation around the vehicle can be displayed on the display means without a sense of incongruity.

Further, the periphery monitoring system of the present invention stores a correspondence relationship between the captured image and the virtual viewpoint direction in the storage unit, and determines the virtual viewpoint direction of the image data reflected in the divided area between the captured image and the virtual viewpoint direction. Since the movement is performed in accordance with the correspondence relationship, the processing time for viewpoint conversion can be shortened.
Further, the periphery monitoring system of the present invention continuously moves the virtual viewpoint direction from one side of the captured image to the other side, so that the driver shifts his / her line of sight from one side to the other. Since it is displayed on the display means as an image as if it has been seen, the situation around the vehicle can be displayed as a single image on the display means without any discomfort.
In addition, since the periphery monitoring system of the present invention changes the virtual viewpoint direction based on the viewpoint conversion map, the viewpoint can be converted with a small number of calculations.

  Further, in the periphery monitoring system according to the present invention, when there are a plurality of imaging units and the overlapped images are overlapped, the combined boundary position is determined within the overlapped range, and the captured images are combined. Can be displayed on the display means without interruption.

Next, an embodiment of the periphery monitoring system according to the present invention will be described with reference to the drawings. FIG. 1 is a schematic configuration diagram of the periphery monitoring system 1.
The perimeter monitoring system 1 includes a vehicle 2, a first camera (imaging means) 3, a second camera (imaging means) 4, a display (display means) 5, a perimeter monitoring control device (perimeter monitoring ECU) 6, and the like. .

  The periphery monitoring system 1 acquires an image in front of the vehicle using the first camera 3 and the second camera 4, and combines the captured images captured by the first and second cameras 3 and 4 as one image to display 5. To display. The viewing angle of the composite image is desirably about 120 to 200 degrees, which is about the human still body viewing angle. In the present embodiment, the first camera 3 and the second camera 4 capable of wide-angle shooting with a viewing angle of about 120 degrees are placed on the number plate mounted in front of the vehicle so that the viewing angles overlap each other as shown in FIG. It is placed near the center for wide-angle shooting. If the first and second cameras 3 and 4 are mounted horizontally, about half of the photographed image is obtained by photographing the sky, so that it is slightly downward from the horizontal, for example, about 15 to 30 degrees. It is arranged with a depression angle.

The display 5 is provided on the center console or panel surface in the room of the vehicle 2 and displays, for example, a composite image obtained by combining the captured images of the first and second cameras 3 and 4. The display 5 may be shared with the navigation system.
The peripheral monitoring ECU 6 performs viewpoint conversion on the captured images of the first and second cameras 3 and 4 and combines them, and displays the combined image on the display 5.

FIG. 2 is a block diagram of the periphery monitoring ECU 6.
The peripheral monitoring ECU 6 includes a video switching processing unit 11, a capture processing unit 12, a first capture buffer 13, a second capture buffer 14, a coordinate conversion processing unit (viewpoint conversion unit) 15, and a display coordinate conversion table storage unit (storage unit) 16. A display buffer 17 and display processing means 18.

  The peripheral monitoring ECU 6 switches the images captured by the first and second cameras 3 and 4 alternately at predetermined intervals by the video switching unit 11 and inputs the images to the capture processing unit 12. The capture processing unit 12 stores an image captured by the first camera 3 in the first capture buffer 13, while storing an image captured by the second camera 4 in the second capture buffer 14. That is, the images captured by the first and second cameras 3 and 4 are stored separately so that they can be distinguished and recognized. The coordinate conversion processing means 15 reads the image data from the first capture buffer 13 and the second capture buffer 14, performs viewpoint conversion according to the information of the coordinate conversion table stored in the coordinate conversion table storage means 16, and after the viewpoint conversion Are combined to create a composite image. The created composite image is input to the display buffer 17. The display processing means 18 inputs the composite image from the display buffer 17 and outputs it to the display 5, and displays the situation around the vehicle 2 on the display 5 as one image.

The coordinate conversion process that characterizes the present invention will be specifically described. The coordinate conversion process shown in FIG. 2 is performed in the same procedure for the captured images of the first and second cameras 3 and 4. Therefore, here, the coordinate conversion of the captured image of the second camera 4 will be mainly described, and the description of the coordinate conversion of the captured image of the first camera 3 will be omitted. FIG. 3 is an image diagram of the coordinate conversion processing shown in FIG.
The second capture buffer 14 has m pixels arranged in the vertical direction and n pixels arranged in the horizontal direction. The second capture buffer 14 includes a coordinate system having the second camera position at the center of the vehicle as the origin, the left and right sides with respect to the vehicle traveling direction as the X axis, and the vehicle traveling direction as the Y axis. That is, the second camera 4 is provided with a coordinate system such that the X value increases from the vehicle center side toward the right side and the Y value increases downward (road surface). The captured image input to the second capture buffer 14 is divided into a plurality of image data groups according to coordinate values.

  The coordinate conversion table is stored in the coordinate conversion table storage unit 16. The coordinate conversion table has the same width W and height H as the display buffer 17 and defines the correspondence between the pixels of the captured image and the pixels of the display image. When the coordinate conversion processing unit 15 initializes the value of the composite image to zero, the coordinate conversion processing unit 15 performs coordinate conversion processing on all the coordinates on the coordinate conversion table. As an example, (x2, y2) on the coordinate conversion table will be described. Information about the second capture buffer X = 100, Y = 80 is acquired from the coordinate conversion table (x2, y2). Based on this information, X = 100, Y = 80 pixels in the second capture buffer are acquired, and are copied and held in the display buffer (x2, y2). At the time of the coordinate conversion, the coordinate conversion processing unit 15 performs the divided region of the captured image based on the correspondence relationship between the captured image stored in the coordinate conversion table storage unit 16 and the virtual viewpoint direction of the second camera 4. In addition, the viewpoint conversion is performed by continuously moving the virtual viewpoint direction of the second camera 4 in a direction different from the optical axis L2 of the second camera 4. Here, the “virtual viewpoint direction” refers to a direction in which the actual viewpoint direction of the camera is virtually moved with respect to the optical axis of the camera.

FIG. 4 is a graph showing a viewpoint conversion map that defines the correspondence between the captured image and the virtual viewpoint direction of the second camera 4 in the coordinate conversion process. FIG. 5 is an example of the virtual viewpoint direction when creating the coordinate conversion table.
As shown in the viewpoint conversion map of FIG. 4, the correspondence between the captured image and the virtual viewpoint direction of the second camera 4 is mapped and stored in the coordinate conversion table storage unit 16. In the viewpoint conversion map, the virtual viewpoint direction is set on the vertical axis, and the pixels of the second buffer 14 having the left end of the captured image as the origin are set on the horizontal axis. For example, as shown in FIG. 9, out of the viewing angle R2 of the second camera 4, the viewing angle from the combined boundary position P to the camera optical axis L2 of the second camera 4 is θ5, and the imaging range from the combined boundary position P is Assuming that the viewing angle up to the right end n is α, as shown in FIG. 4, the viewpoint conversion map is a virtual viewpoint from the left end of the image on the vehicle center side to the composite boundary position P for the captured image of the second camera 4. Without changing the direction, the virtual viewpoint direction is continuously moved for each region divided by the X coordinate (or Y coordinate or XY coordinate) for the image data in the range from the composite boundary position P to the right end n of the imaging range. Go. The correspondence relationship between the captured image and the virtual viewpoint direction of the second camera 4 is defined by the straight line G1, the curves G2, and G3 on the graph, and is virtually determined from the camera optical axis L for each region obtained by dividing the captured image based on the XY coordinates. It defines the angle to move the viewpoint direction. The straight line G1 and the curves G2 and G3 are appropriately selected and used by the coordinate conversion processing unit 15 depending on the video portion that is important on the captured image.

  For example, the straight line G1 corresponds to the virtual viewpoint direction of the second camera 4 so that the virtual viewpoint direction of the captured image is continuously moved from one side of the captured image to the other with a constant angular change rate. It defines the relationship. This correspondence is suitable when importance is attached to the entire captured image of the second camera 4. When the straight line G1 is applied to the captured image, the result is as shown in FIG. 5, and the image data group included in the captured image of the second camera 4 converts the virtual viewpoint direction from the left end of the captured image to the composite boundary position P. Instead, every time the X coordinate value increases by 1 from the composite boundary position P toward the right edge of the image, the virtual viewpoint direction is moved by 0.5 degrees.

  Also, for example, the curves G2 and G3 shown in FIG. 4 indicate the angles when moving the virtual viewpoint direction of the captured image from one side to the other for each region divided by the X coordinate (or Y coordinate or XY coordinate). The correspondence relationship between the captured image and the virtual viewpoint direction of the second camera 4 is defined so as to change the rate of change. That is, the curve G2 has a small angle change rate in the virtual viewpoint direction in the vicinity of the combined boundary position P, and a large angle change rate in the virtual viewpoint direction in the vicinity of the right end n of the imaging range of the right camera 4. The correspondence relationship with the virtual viewpoint direction of the second camera 4 is defined. This correspondence is suitable when the portion near the composite boundary position P in the captured image of the second camera 4 is more important than the vicinity of the right end n of the imaging range. On the other hand, the curve G3 is a captured image so that the angle change rate in the virtual viewpoint direction is large near the composite boundary position P and the angle change rate in the virtual viewpoint direction is small near the right end n of the imaging range of the second camera 4. And the virtual viewpoint direction of the second camera 4 are defined. This correspondence is suitable when the right end portion of the captured image of the second camera 4 is more important than the vicinity of the composite boundary position P.

  In this way, if the correspondence relationship between the captured image and the virtual viewpoint direction of the second camera 4 is mapped on the assumption of an important point in the captured image, a plurality of coordinate conversion tables are stored in advance. The storage capacity can be reduced. The correspondence between the captured image and the virtual viewpoint direction of the second camera 4 is selected and set when the periphery monitoring system 1 is activated. The correspondence relationship between the captured image and the virtual viewpoint direction of the second camera 4 is, for example, a portion that is important for the driver to monitor the periphery of the vehicle 2 using an operation unit such as a touch panel of the display 5. The mode may be arbitrarily selected by switching the designated mode. In addition, for example, the steering angle of the direction indicator or the steering wheel, the braking force, and the like are detected, and the correspondence between the captured image and the virtual viewpoint direction of the second camera 4 is automatically determined based on the traveling state of the vehicle 2. You may make it select. Furthermore, when a portion to be emphasized is determined in advance, only one type of correspondence between the captured image and the virtual viewpoint direction of the second camera 4 may be stored to reduce the storage capacity.

  By the way, the virtual viewpoint direction of the captured image is set to change stepwise or continuously from θ5-α to θ5 degrees. θ5 is an angle formed by the camera optical axis L with respect to the vehicle traveling direction, and the virtual viewpoint direction of the horizontal coordinate P serving as a boundary portion for combining the left and right camera images is set to θ5. Thereby, the virtual viewpoint direction of the synthetic | combination boundary position P becomes the advancing direction of a vehicle, and a natural image | video can be obtained. Further, α corresponds to an angle at which the driver swings his / her head when checking the left and right safety. In the present embodiment, as an example, θ5 is set to 45 degrees and α is set to 60 degrees.

  Next, the operation of the periphery monitoring system 1 will be specifically described. The processing of the captured images of the first and second cameras 3 and 4 is the same. Here, considering that the road in Japan is right-hand traffic, the processing of the captured image of the second camera 4 that captures the situation in the right direction of the vehicle will be mainly described, and the processing of the captured image of the first camera 3 will be described. Will be omitted as appropriate.

FIG. 6 is an explanatory diagram showing the imaging range of the first camera 3 and the second camera 4.
The intersection shown in FIG. 6 is a T-junction where the alley 22 is connected to the main street 21. The wall 23 is constructed along the main street 21. Sidewalks 24 and 25 are provided on both sides of the alley 22, and a utility pole 26 stands on the sidewalk 24. In the vehicle 2 that is going out to the main street 21 from the alley 22, the first and second cameras 3 and 4 are installed with the optical axes L 1 and L 2 directed in different directions, and the first camera 3 moves in the left direction of the vehicle. The second camera 4 captures an image in the right direction of the vehicle.

FIG. 7 is a diagram illustrating a captured image of the first camera 3 illustrated in FIG. 6, and FIG. 8 is a diagram illustrating a captured image of the second camera 4 illustrated in FIG. 6.
As shown in FIG. 7, the captured image of the first camera 3 shows the main street 21 and the substantially left half of the wall 23, and the sidewalk 24 and the utility pole 26 on the left side of the vehicle 2. On the other hand, as shown in FIG. 8, the captured image of the second camera 4 shows the main street 21 and the substantially right half of the wall 23 and the sidewalk 25 on the right side of the alley 22. In the captured image of the second camera 4, the main street 21 becomes narrower as it goes to the right, and the road width at the right end of the image is w1. As the road width decreases, the sidewalk 25 is projected to the right by θ2 degrees. In this state, the main street 21 is displayed small at the right end of the image, and it is difficult to distinguish a vehicle, a bicycle, a motorcycle, or the like traveling toward the host vehicle 2 from the right direction.

FIG. 9 is an image diagram of viewpoint conversion.
Therefore, the virtual camera performs image processing assuming that the viewpoint of the second camera 4 is moved to the right by a predetermined angle from the front surface A (0) of the vehicle 2 and the right front of the vehicle 2 is imaged.

FIG. 10 is an image diagram when the viewpoints of the first camera 3 and the second camera 4 are changed so as to face only the front surface (A (0)) of the vehicle 2.
When the viewpoint is changed so that the viewpoint of the second camera 4 is directed only to the front A (0) of the vehicle 2, the captured image shown in FIG. 8 is displayed according to the same principle as that described in the column of the problem to be solved by the invention. The road width w1 of the main street 21 shown at the right end is enlarged and displayed to the road width w2 as shown in FIG. At the same time, the sidewalk 25 is enlarged and displayed as it goes to the right end of the image. Therefore, the boundary between the main street 21 and the sidewalk 25 is inclined to the right by θ3 degrees. θ3 degrees has a larger angle than θ2 degrees. Therefore, the image shown in FIG. 10 is an enlarged view of the main street 21 and the sidewalk 25 shown at the right end of the image shown in FIG. 8 from the viewpoint where the driver is facing the front A (0) of the vehicle 2. It's not too real.

  That is, the driver generally checks whether a vehicle, a bicycle, a motorcycle, or the like is running from the left-right direction of the vehicle 2 toward the own vehicle 2 with the face facing the left-right direction of the main street 21 at this time. The reduction ratio at which the road width of the main street 21 is reduced in the right direction due to the perspective difference is more in the case where the driver's face is directed to the right than when the driver's face is directed toward the front A (0). Should be smaller. Therefore, the angle θ3 at which the boundary between the main street 21 and the sidewalk 25 is inclined to the right should be smaller than the angle θ2 shown in FIG. Nevertheless, in the image shown in FIG. 10, the angle θ3 at which the boundary between the main street 21 and the sidewalk 25 tilts upward is larger than the angle θ2 shown in FIG. 8, and the driver actually confirms the left-right direction. There is a sense of incongruity between the visible situation and the image displayed on the display 5. In addition, the image shown in FIG. 10 is too blurry because the main street 21 and the sidewalk 25 on the right end are displayed too large, and from this point, it is difficult to confirm vehicles, bicycles, motorcycles, etc. that are approaching the vehicle 2 from the right direction. . In view of the fact that the surrounding monitoring system 1 assists the driver in checking left and right, it is desirable to display an image close to the situation that can be seen when the driver checks the left and right directions on the display 5.

  Therefore, as shown in FIG. 9, for example, image data included in the captured image of the second camera 4 is captured according to the correspondence relationship between the captured image defined by the straight line G <b> 1 illustrated in FIG. 4 and the virtual viewpoint direction of the second camera 4. When the virtual viewpoint direction is moved for each area divided by the XY coordinates of the image, the direction of the image data is as if the driver changed the face direction from the front A (0) to the right and looked at the subject. Can be changed continuously. Therefore, the image displayed at the right end of the display 5 is an image as if the driver viewed the face facing right.

FIG. 11 is a diagram showing images of the first camera 3 and the second camera 4 subjected to viewpoint conversion.
As a result of the viewpoint conversion, the captured image of the second camera 4 is displayed with the road width w3 of the main street 21 shown at the right end larger than the road width w1 of FIG. 8 and smaller than the road width w2 shown in FIG. Then, the boundary between the main street 21 and the sidewalk 25 is displayed to the upper right by θ4 degrees. The angle θ4 at which the boundary between the main street 21 and the sidewalk 25 is tilted upward is smaller than the angle θ2 shown in FIG. 8, and the road width w3 is not simply an enlargement of the road width w1 shown in FIG. It can be seen that it was enlarged.

  Note that the first camera 3 is also subjected to the same processing for images captured at the same timing as the second camera 4. Therefore, the left end of the main camera 21 is also displayed larger than that in FIG. The utility pole 26 shown in FIG. 11 is inclined obliquely in the lower left direction in the figure as compared with the utility pole 26 shown in FIGS. 8 and 10, and the captured image of the second camera 3 is leftward from the front A (0) by the driver. It can be seen that the viewpoint has been changed as if it was viewed with the face facing up, and the left end portion was enlarged.

  As shown in FIG. 6, the images of the first and second cameras 3 and 4 subjected to the image processing in this way have the combined boundary position P within the overlapping range S1 of the imaging ranges of the first and second cameras 3 and 4. It is determined. This is because the images of the first and second cameras 3 and 4 are connected and displayed without generating a blind spot. In the present embodiment, the front surface A (0) of the vehicle 2 is determined as the composite boundary position P. The virtual viewpoint direction of the composite boundary position P is matched with the vehicle traveling direction, and the captured images of the first and second cameras 3 and 4 are used in the left and right uniform areas of the display image, and the captured image and the virtual viewpoint of the second camera 4 are used. This is because image processing of the first and second cameras 3 and 4 is performed by sharing a viewpoint conversion map that defines the correspondence with the direction. Thus, if the viewpoint conversion map is shared, there is an advantage that the storage capacity can be reduced.

FIG. 12 is a diagram illustrating a combined image obtained by combining the images of the first camera and the second camera.
When the images of the first and second cameras 3 and 4 that have been subjected to image processing are combined at the combining boundary position P, one image in the front left and right direction of the vehicle 2 is displayed. Since this display image is close to the situation seen when the driver checks the left and right directions with the line of sight, the driver can check the situation in the left and right directions of the main street 21 without feeling uncomfortable. Further, although the main street 21 at the right end portion of the display image is displayed larger than the case shown in FIG. 8, the image data is not diffused and blurred or is displayed unnecessarily large unlike the case shown in FIG. 10. . Therefore, even if it is difficult for the driver to see the right and left direction directly when exiting from the alley 22 to the main street 21, if the driver sees the display content on the display 5, the vehicle, bicycle, motorcycle, etc. It is easy to recognize whether or not the vehicle travels toward the host vehicle 2.

  Therefore, according to the periphery monitoring system 1 of the present embodiment, the captured image captured by the first and second cameras 3 and 4 is divided into a plurality of regions, and the virtual images of the first and second cameras 3 and 4 are divided for each region. By performing the viewpoint conversion by continuously moving the viewpoint direction in different directions, the front and left and right directions of the vehicle 2 are displayed on the display 5 as if the driver viewed the front and right and left directions of the vehicle 2 facing each other. (See FIGS. 9, 11, and 12) For example, image data of a subject that is captured at a location away from the viewpoints of the first and second cameras 3 and 4 is displayed on the display 5 due to distortion of a lens or an imaging device. It is possible to display the front left and right directions of the vehicle 2 on the display 5 without a sense of incongruity without being displayed in a small size, or being diffused and blurred at the time of viewpoint conversion. That is, when the surroundings monitoring system 1 is used to monitor the situation around the vehicle 2 shown in FIG. 13, as shown in FIG. 14, the person 115 crossing the pedestrian crossing 114 reflected in the end portion of the display image, It is possible to appropriately display the roadside tree 112 on one screen in the size, shape, and direction as if the driver looked to the right from the front. In particular, since the roadside tree 112 does not hide the right direction of the main street as shown in FIGS. 17 and 20, the driver confirms the situation of the right direction of the main street in a wide range and approaches the host vehicle 100. The presence of vehicles, bicycles, motorcycles, etc. can be grasped without a sense of incongruity.

In addition, the periphery monitoring system 1 of the present embodiment stores in advance the correspondence relationship between the captured image shown in FIG. 4 and the virtual viewpoint direction of the second camera 4 in the coordinate conversion table storage unit 16 and divides it by XY coordinates. In order to move the virtual viewpoint direction of the image data included in the imaged area according to the correspondence between the captured image and the virtual viewpoint direction of the second camera 4 (see FIGS. 9 and 11), the viewpoint conversion processing time is shortened. can do.
In addition, the periphery monitoring system 1 according to the present embodiment, for example, continuously moves the virtual viewpoint direction of the image data of the second camera 4 from the combined boundary position P with the captured image captured by the first camera 3 toward the right end n. By moving the vehicle (refer to the straight lines G1, G2, and G3 in FIG. 4), the driver continuously shifts his / her line of sight from the front A (0) of the vehicle 2 to the right in the front right direction of the vehicle 2. Since it is displayed on the display 5 as an image as if it was seen (see FIGS. 9 and 11), the situation in the front left and right directions of the vehicle 2 can be displayed as a single image on the display 5 without a sense of incongruity.
Moreover, since the periphery monitoring system 1 of this embodiment changes a virtual viewpoint direction based on a viewpoint conversion map (refer FIG. 4), a viewpoint conversion can be carried out with few calculation frequency.

  Moreover, since the periphery monitoring system 1 of this embodiment determines the synthetic | combination boundary position P within the overlapping range S1 of the picked-up image imaged with the 1st camera 3 and the 2nd camera 4, and synthesize | combines a picked-up image, the vehicle front condition Can be displayed on the display 5 without a break (see FIG. 12).

  Although the embodiments of the present invention have been described, the present invention is not limited to the above-described embodiments, and various applications are possible.

(1) For example, in the above-described embodiment, the periphery monitoring system 1 installs the first and second cameras 3 and 4 in the front bumper of the vehicle 2, but one or more cameras may be installed. . That is, for example, only one wide-angle camera is installed on the front bumper, a third camera that projects the lower side of the bumper in addition to the first and second cameras 3 and 4 is installed on the front bumper, and 1 is installed on the rear bumper. Even when one or a plurality of cameras are installed or one or a plurality of cameras are installed on one side of the vehicle 2, by moving the virtual viewpoint direction of the captured image continuously, the front, rear, and side of the vehicle 2 This situation can be displayed on the display 5 without a sense of incongruity. Then, the captured images of the cameras provided on the front, rear, left and right of the vehicle 2 are moved in the virtual viewpoint direction to create a composite image and displayed on the display 5 as a single image. Can be confirmed in a wide range, and convenience in parallel parking or parallel parking is improved.

(2) For example, in the above-described embodiment, the captured images of the first and second cameras 3 and 4 are used except for the overlapping range S1, but the display processing unit 18 includes the composite image held in the display buffer 17. The image data of a necessary part may be cut out and output to the display 5. In this case, for example, when the driver moves an area desired to be viewed with an operation means such as a touch panel of the display 5, image data corresponding to the area moved by the display processing means 18 is appropriately input from the display buffer 17 and displayed. By outputting to 5, it is possible to display the situation around the vehicle 2 on the display 5 without a sense of incongruity according to the driver's request.

It is a schematic block diagram of the periphery monitoring system in one embodiment of this invention. It is a block diagram of periphery monitoring ECU shown in FIG. It is a conceptual diagram of the coordinate transformation process shown in FIG. It is a graph which shows the viewpoint conversion map which prescribes | regulates the correspondence of a captured image and the virtual viewpoint direction of the 2nd camera 4 in a coordinate conversion process. It is an example of the virtual viewpoint direction at the time of creating a coordinate conversion table. It is explanatory drawing which shows the imaging range of a 1st camera and a 2nd camera. It is a figure which shows the captured image of the 1st camera shown in FIG. It is a figure which shows the picked-up image of the 2nd camera shown in FIG. It is an image figure of viewpoint conversion. It is an image figure when a viewpoint is changed so that the viewpoints of the first camera and the second camera are directed only to the front of the vehicle. It is a figure which shows the image of the 1st camera and 2nd camera which carried out viewpoint conversion. It is a figure which shows the synthesized image which synthesize | combined the image of a 1st camera and a 2nd camera. It is explanatory drawing which shows the condition around a vehicle. It is a figure which shows an example of the captured image display form which imaged and displayed the condition shown in FIG. It is a figure which shows an example of the imaging | photography range of the conventional periphery monitoring system. It is a figure which shows an example of the conventional captured image display form. It is a figure which shows an example of the captured image display form which images the situation shown in FIG. 13 and displays it on a display apparatus. It is explanatory drawing which shows an example of a projection figure. It is an image figure of viewpoint conversion proposed by Japanese Patent Application No. 2001-529474. It is a figure which shows an example of a captured image display form. It is an image diagram when the viewpoint is changed so that the viewpoints of the first camera and the second camera are directed only to the front of the vehicle.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Perimeter monitoring system 2 Vehicle 3 1st camera (imaging means)
4 Second camera (imaging means)
5 Display (display means)
15 Coordinate conversion processing means (viewpoint conversion means)
16 Coordinate conversion table storage means (storage means)

Claims (7)

In a periphery monitoring system that captures an image of a situation around a vehicle by an image capturing unit installed in the vehicle, displays the captured image on a display unit after image processing,
The driver divides the captured image into a plurality of regions and performs viewpoint conversion by continuously moving the virtual viewpoint direction of the imaging unit in a direction different from the optical axis of the imaging unit for each region , thereby allowing the driver to perform the viewpoint conversion. to have a viewpoint conversion unit seen directly opposite in each direction around the situation of,
Having an area composition means for creating a composite image by arranging the areas subjected to viewpoint conversion in the horizontal direction;
The viewpoint conversion means continuously moves the virtual viewpoint direction from one of the captured images to the other with a changing rate of angle change;
Having a viewpoint conversion map that defines the correspondence between the captured image and the virtual viewpoint direction of the imaging means;
In the viewpoint conversion map, the virtual viewpoint direction is set on the vertical axis, and the processed image data pixels are set on the horizontal axis.
The changing angle change rate is defined in the viewpoint conversion map;
Perimeter monitoring system characterized by In the periphery monitoring system according to claim 1,
One of the captured images is a traveling direction of the vehicle . In surroundings monitoring system according to 請 Motomeko 2,
The other of the captured images corresponds to an angle at which the driver swings his / her head when checking the left and right safety with respect to the traveling direction of the vehicle . In surroundings monitoring system according to any one of claims 1 to 3,
The changing angle change rate is such that the angle change rate in the virtual viewpoint direction is small near one side of the captured image, and the angle change rate in the virtual viewpoint direction is large near the other side of the captured image. Or the angle change rate in the virtual viewpoint direction is large near one side of the captured image, and the angle change rate in the virtual viewpoint direction is small near the other side of the captured image. Perimeter monitoring system characterized by that. In the periphery monitoring system according to any one of claims 1 to 4 ,
Storage means for storing a correspondence relationship between the captured image and the virtual viewpoint direction;
The periphery monitoring system, wherein the viewpoint conversion unit performs viewpoint conversion based on a correspondence relationship between the captured image stored in the storage unit and the virtual viewpoint direction. In the periphery monitoring system according to any one of claims 1 to 5 ,
The viewpoint conversion means includes
Environment monitoring system characterized by varying the virtual viewpoint direction based the captured image to the viewpoint conversion map. In the periphery monitoring system according to any one of claims 1 to 6 ,
A peripheral monitoring system, comprising: a plurality of imaging means, and a synthesis means for determining a synthesis boundary position within the overlapping range and synthesizing the captured images when the imaging images have overlapping ranges.

Images