JP2010250640A - Image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device capable of improving visibility of a three-dimensional object. <P>SOLUTION: When cameras C_3 and C_4 out of cameras C_1 to C_4 are noticed, the cameras C_3 and C_4 capture a road surface from diagonally upward so as to partially have a common visual field VW_34. A CPU loads field images P_3 and P_4 output from the cameras C_3 and C_4 respectively, converts the loaded field images P_3 and P_4 into bird's eye view images BEV_3 and BEV_4 respectively, and combines the converted bird's eye view images BEV_3 and BEV_4 together while referring to a border line allocated to the common visual field VW_34. When a three-dimensional object is present in the common visual field VW_34, the CPU detects a size difference between three-dimensional images respectively appearing in the respective bird's eye view images BEV_3 and BEV_4 and adjusts the setting of the border line by referring to the detected difference. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and in particular, an image processing apparatus that converts a scene image output from each of a plurality of cameras that capture a reference plane from an oblique direction so as to partially have a common visual field into a bird's-eye view image. About.

  An example of this type of device is disclosed in Patent Document 1. According to this background art, the four cameras are mounted on the vehicle so that the fields of view of the two adjacent cameras partially overlap. The surroundings of the vehicle are captured by such a camera, and the object scene image output from each camera is converted into a bird's-eye view image.

  A boundary line is assigned to the partial image corresponding to the overlapping visual field in the converted bird's-eye view image. The four bird's-eye images respectively corresponding to the four cameras are combined with each other through the trimming process referring to such a boundary line. However, when an obstacle, that is, a three-dimensional object is detected from the overlapping visual field, the position of the boundary line is changed so as to avoid the three-dimensional object image. Thereby, the quality of the three-dimensional object image is maintained.

JP 2007-104373 A

  However, since the bird's-eye view image is created based on the road surface, the reproducibility of the three-dimensional object in the bird's-eye view image varies due to the difference in height between the camera and the three-dimensional object, the distance from the camera to the three-dimensional object, etc. To do. In the background art, the position of the boundary line is not controlled in consideration of such characteristics, and the visibility of the three-dimensional object is limited.

  Therefore, a main object of the present invention is to provide an image processing apparatus capable of improving the visibility of a three-dimensional object.

  The image processing apparatus according to the present invention (10: reference numerals corresponding to the embodiments; the same applies hereinafter) is respectively output from a plurality of cameras (C_3, C_4) that capture the reference plane obliquely from above so as to partially have a common visual field. Capture means (S1) for capturing a plurality of object scene images, and conversion means for converting the plurality of object scene images captured by the capture means into a plurality of bird's-eye images representing a state where the reference plane is captured from directly above ( S3), a combining means (S7) for combining a plurality of bird's-eye images converted by the converting means with reference to the weight assigned to the common visual field, and a plurality of images converted by the converting means when a three-dimensional object exists in the common visual field. First detection means (S35) for detecting a difference in size between a plurality of three-dimensional object images respectively appearing in the bird's-eye view image, and a first adjustment for adjusting the weight with reference to the difference detected by the first detection means Provide means (S41, S43, S61) Yeah.

  Preferably, the first adjustment unit adjusts the weight so that a larger three-dimensional object image is reproduced.

  Preferably, second detection means (S45) for detecting the movement amount of the three-dimensional object in relation to the detection processing of the first detection means, and second adjustment for adjusting the weight with reference to the movement amount detected by the second detection means. The adjusting means (S47 to S55, S61) and control means (S37, S39) for starting the second adjusting means instead of the first adjusting means when the difference detected by the first detecting means falls below the reference are further provided. .

  More preferably, the second adjusting means adjusts the weight so that the three-dimensional object image of the camera existing in the moving direction of the three-dimensional object is reproduced.

  Preferably, the weight includes a boundary line defined in the common visual field as a parameter, and the synthesis unit deletes the partial image outside the boundary line from each of the plurality of bird's-eye images (S73), and the deletion unit A combination unit (S79) for combining the plurality of partial bird's-eye images remaining after the deletion process.

  Preferably, the plurality of cameras are provided on the moving body, and further include display means (16) for displaying the synthesized bird's-eye view image created by the synthesizing means toward the operator of the moving body.

  According to the present invention, the weight referred to when combining a plurality of bird's-eye images is adjusted by paying attention to the difference between the cameras in the size of the three-dimensional object image. Thereby, the visibility of a three-dimensional object can be improved.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the basic composition of this invention. It is a block diagram which shows the structure of one Example of this invention. It is an illustration figure which shows the visual field captured by the some camera attached to the vehicle. (A) is an illustrative view showing an example of a bird's eye view image based on the output of the previous camera, (B) is an illustrative view showing an example of a bird's eye view image based on the output of the right camera, and (C) is an output of the left camera. It is an illustration figure which shows an example of the bird's-eye view image based on (D), and (D) is an illustration figure which shows an example of the bird's-eye view image based on the output of a back camera. It is an illustration figure which shows a part of creation operation of an all-around bird's-eye view image. It is an illustration figure which shows an example of the created all-around bird's-eye view image. It is an illustration figure which shows an example of the driving assistance image displayed by a display apparatus. It is an illustration figure which shows the angle of the camera attached to the vehicle. It is an illustration figure which shows the relationship between a camera coordinate system, the coordinate system of an imaging surface, and a world coordinate system. It is an illustration figure which shows an example of the obstruction which exists in a vehicle and its periphery. FIG. 10 is an illustrative view showing another portion of the operation of creating the driving assistance image. It is an illustration figure which shows an example of the operation | movement which reproduces an obstacle image. It is an illustration figure which shows another example of the operation | movement which reproduces an obstacle image. It is an illustration figure which shows another example of the operation | movement which reproduces an obstacle image. It is a flowchart which shows a part of operation | movement of CPU applied to the FIG. 2 Example. It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2; It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2;

Embodiments of the present invention will be described below with reference to the drawings.
[Basic configuration]

  Referring to FIG. 1, the image processing apparatus of the present invention is basically configured as follows. The capturing means 2 captures a plurality of object scene images respectively output from a plurality of cameras 1, 1,... Capturing the reference plane obliquely from above so as to partially have a common visual field. The converting unit 3 converts the plurality of scene images captured by the capturing unit 2 into a plurality of bird's-eye images representing a state where the reference plane is captured from directly above. The synthesizing unit 4 synthesizes the plurality of bird's-eye images converted by the converting unit 3 with reference to the weight assigned to the common visual field. The first detection unit 5 detects a difference in size between the plurality of three-dimensional object images respectively appearing in the plurality of bird's-eye images converted by the conversion unit 3 when a three-dimensional object exists in the common visual field. The first adjusting unit 6 adjusts the weight with reference to the difference detected by the first detecting unit.

The size of the three-dimensional object image that appears in the bird's-eye view image differs depending on the height of the three-dimensional object from the reference plane and the height difference between the camera 1 and the three-dimensional object. The weight referred to when combining a plurality of bird's-eye images is adjusted by paying attention to the difference between the cameras 1 in the size of such a three-dimensional object image. Thereby, the visibility of a three-dimensional object can be improved.
[Example]

  The steering assistance device 10 of this embodiment shown in FIG. 2 includes four cameras C_1 to C_4. The cameras C_1 to C_4 output the scene images P_1 to P_4 every 1/30 seconds in synchronization with the common timing signal. The output scene images P_1 to P_4 are given to the image processing circuit 12.

  Referring to FIG. 3, camera C_1 is installed at the center of the front portion of vehicle 100 with the optical axis of camera C_1 extending obliquely downward in front of vehicle 100. The camera C_2 is installed on the upper right side of the vehicle 100 in a posture in which the optical axis of the camera C_2 extends obliquely downward to the right of the vehicle 100. The camera C_3 is installed at the center of the rear part of the vehicle 100 in a posture in which the optical axis of the camera C_3 extends obliquely downward rearward of the vehicle 100. The camera C_4 is installed on the upper left side of the vehicle 100 such that the optical axis of the camera C_4 extends obliquely downward to the left of the vehicle 100. The object scene around the vehicle 100 is captured from such a direction that intersects the road surface in an oblique direction by the cameras C_1 to C_4.

  Camera C_1 has a field of view VW_1 that captures the front of the vehicle 100, camera C_2 has a field of view VW_2 that captures the right direction of the vehicle 100, camera C_3 has a field of view VW_3 that captures the rear of the vehicle 100, and camera C_4 It has a visual field VW_4 that captures the left direction of the vehicle 100. Also, the visual fields VW_1 and VW_2 have a common visual field VW_12, the visual fields VW_2 and VW_3 have a common visual field VW_23, the visual fields VW_3 and VW_4 have a common visual field VW_34, and the visual fields VW_4 and VW_1 have a common visual field VW_41.

  Returning to FIG. 2, the CPU 12p provided in the image processing circuit 12 generates the bird's-eye view image BEV_1 shown in FIG. 4A based on the object scene image P_1 output from the camera C_1, and is output from the camera C_2. The bird's-eye view image BEV_2 shown in FIG. 4B is generated based on the object scene image P_2. The CPU 12p also generates the bird's-eye view image BEV_3 shown in FIG. 4C based on the object scene image P_3 output from the camera C_3, and based on the object scene image P_4 output from the camera C_4, FIG. The bird's-eye view image BEV_4 shown in FIG.

  The bird's-eye view image BEV_1 corresponds to an image captured by the virtual camera looking down the visual field VW_1 in the vertical direction, and the bird's-eye view image BEV_2 corresponds to an image captured by the virtual camera looking down the visual field VW_2 in the vertical direction. The bird's-eye view image BEV_3 corresponds to an image captured by a virtual camera looking down the visual field VW_3 in the vertical direction, and the bird's-eye view image BEV_4 corresponds to an image captured by a virtual camera looking down the visual field VW_4 in the vertical direction.

  According to FIGS. 4A to 4D, the bird's-eye image BEV_1 has a bird's-eye coordinate system X1 and Y1, the bird's-eye image BEV_2 has a bird's-eye coordinate system X2 and Y2, and the bird's-eye image BEV_3 has a bird's-eye coordinate system. The bird's-eye view image BEV_4 has a bird's-eye coordinate system X4 / Y4. Such bird's-eye images BEV_1 to BEV_4 are held in the work area W1 of the memory 12m.

  Subsequently, the CPU 12p deletes a part of the images outside the boundary line BL from each of the bird's-eye images BEV_1 to BEV_4, and rotates / removes a part of the bird's-eye images BEV_1 to BEV_4 remaining after the deletion (see FIG. 5). Combine with each other by moving process. As a result, the all-around bird's-eye view image shown in FIG. 6 is obtained in the work area W2 of the memory 12m.

  In FIG. 6, the overlapping area OL_12 indicated by diagonal lines corresponds to the common visual field VW_12, and the overlapping area OL_23 indicated by diagonal lines corresponds to the common visual field VW_23. Further, the overlapping area OL_34 indicated by hatching corresponds to the common visual field VW_34, and the overlapping area OL_41 indicated by hatching corresponds to the common visual field VW_41.

  The display device 16 installed in the driver's seat extracts a part of the image D1 in which the overlapping areas OL_12 to OL_41 are located at the four corners from the all-around bird's-eye view image on the work area W2, and the vehicle image D2 simulating the upper part of the vehicle 100 Is pasted in the center of the extracted image D1. As a result, the driving assistance image shown in FIG. 7 is displayed on the monitor screen.

  Note that a part of the overlapping area OL_12 that forms the image D1 is defined as “reproduction overlapping area OLD_12”, and a part of the overlapping area OL_23 that forms the image D1 is defined as “reproduction overlapping area OLD_23”. To do. Similarly, a part of the overlapping area OL_34 that forms the image D1 is defined as “reproduction overlapping area OLD_34”, and a part of the overlapping area OL_41 that forms the image D1 is defined as “reproduction overlapping area OLD_41”. To do.

  Next, how to create the bird's-eye view images BEV_1 to BEV_4 will be described. However, since the bird's-eye images BEV_1 to BEV_4 are all created in the same manner, the creation procedure of the bird's-eye image BEV3 will be described as a representative of the bird's-eye images BEV_1 to BEV_4.

  Referring to FIG. 8, camera C_3 is disposed rearward and obliquely downward at the rear of vehicle 100. If the depression angle of the camera C_3 is “θd”, the angle θ shown in FIG. 7 corresponds to “180 ° −θd”. The angle θ is defined in the range of 90 ° <θ <180 °.

  FIG. 9 shows the relationship between the camera coordinate system X · Y · Z, the coordinate system Xp · Yp of the imaging surface S of the camera C_3, and the world coordinate system Xw · Yw · Zw. The camera coordinate system X, Y, Z is a three-dimensional coordinate system with the X, Y, and Z axes as coordinate axes. The coordinate system Xp · Yp is a two-dimensional coordinate system having the Xp axis and the Yp axis as coordinate axes. The world coordinate system Xw · Yw · Zw is a three-dimensional coordinate system having the Xw axis, the Yw axis, and the Zw axis as coordinate axes.

  In the camera coordinate system X, Y, Z, the optical center of the camera C3 is defined as the origin O, the Z axis is defined in the optical axis direction, the X axis is defined in the direction perpendicular to the Z axis and parallel to the road surface, and A Y axis is defined in a direction orthogonal to the Z axis and the X axis. In the coordinate system Xp / Yp of the imaging surface S, with the center of the imaging surface S as the origin, the Xp axis is defined in the horizontal direction of the imaging surface S, and the Yp axis is defined in the vertical direction of the imaging surface S.

  In the world coordinate system Xw / Yw / Zw, the intersection of the vertical line passing through the origin O of the camera coordinate system XYZ and the road surface is defined as the origin Ow, and the Yw axis is defined in the direction perpendicular to the road surface. An Xw axis is defined in a direction parallel to the X axis of Z, and a Zw axis is defined in a direction orthogonal to the Xw axis and the Yw axis. The distance from the Xw axis to the X axis is “h”, and the obtuse angle formed by the Zw axis and the Z axis corresponds to the angle θ described above.

  When coordinates in the camera coordinate system X, Y, and Z are expressed as (x, y, z), “x”, “y”, and “z” are X-axis components in the camera coordinate system X, Y, and Z, respectively. A Y-axis component and a Z-axis component are shown. When the coordinates in the coordinate system Xp / Yp of the imaging surface S are expressed as (xp, yp), “xp” and “yp” respectively represent the Xp-axis component and the Yp-axis component in the coordinate system Xp / Yp of the imaging surface S. Show. When coordinates in the world coordinate system Xw · Yw · Zw are expressed as (xw, yw, zw), “xw”, “yw”, and “zw” are Xw axis components in the world coordinate system Xw · Yw · Zw, The Yw axis component and the Zw axis component are shown.

A conversion formula between the coordinates (x, y, z) of the camera coordinate system X, Y, Z and the coordinates (xw, yw, zw) of the world coordinate system Xw, Yw, Zw is expressed by Formula 1.

Here, assuming that the focal length of the camera C_3 is “f”, the coordinates (xp, yp) of the coordinate system Xp / Yp of the imaging surface S and the coordinates (x, y, z) of the camera coordinate system X / Y / Z are The conversion formula between is expressed by Equation 2.

Also, Equation 3 is obtained based on Equation 1 and Equation 2. Equation 3 shows a conversion formula between the coordinates (xp, yp) of the coordinate system Xp / Yp of the imaging surface S and the coordinates (xw, zw) of the two-dimensional road surface coordinate system Xw / Zw.

  Also, a bird's-eye coordinate system X3 / Y3, which is a coordinate system of the bird's-eye image BEV_3 shown in FIG. 4C, is defined. The bird's-eye coordinate system X3 / Y3 is a two-dimensional coordinate system having the X3 axis and the Y3 axis as coordinate axes. When the coordinates in the bird's-eye view coordinate system X3 / Y3 are expressed as (x3, y3), the position of each pixel forming the bird's-eye view image BEV_3 is represented by the coordinates (x3, y3). “X3” and “y3” respectively indicate an X3 axis component and a Y3 axis component in the bird's eye view coordinate system X3 · Y3.

The projection from the two-dimensional coordinate system Xw / Zw representing the road surface to the bird's eye coordinate system X3 / Y3 corresponds to a so-called parallel projection. If the height of the virtual camera, that is, the virtual viewpoint is “H”, the conversion formula between the coordinates (xw, zw) of the two-dimensional coordinate system Xw · Zw and the coordinates (x3, y3) of the bird's-eye coordinate system X3 · Y3 is , Represented by Equation (4). The height H of the virtual camera is determined in advance.

Further, based on Equation 4, Equation 5 is obtained, Equation 5 is obtained based on Equation 5 and Equation 3, and Equation 7 is obtained based on Equation 6. Equation 7 corresponds to a conversion formula for converting the coordinates (xp, yp) of the coordinate system Xp / Yp of the imaging surface S into the coordinates (x3, y3) of the bird's eye coordinate system X3 / Y3.

  The coordinates (xp, yp) of the coordinate system Xp / Yp of the imaging surface S represent the coordinates of the object scene image P_3 captured by the camera C_3. Accordingly, the object scene image P_3 from the camera C3 is converted into the bird's-eye view image BEV_3 by using Equation 7. Actually, the object scene image P_3 is first subjected to image processing such as lens distortion correction, and then converted into a bird's-eye view image BEV_3 by Equation 7.

  Referring to FIG. 10 and FIG. 11, when obstacles (three-dimensional objects) 201 and 202 exist in the common left view VW_34 of the vehicle 100, the obstacles 201 and 202 captured by the camera C_3 are obstruction images. Appear in the bird's-eye view image BEV_3 as OBJ_3_1 and OBJ_3_2. Similarly, the obstacles 201 and 202 captured by the camera C_4 appear in the bird's-eye view image BEV_4 as the obstacle images OBJ_4_1 and OBJ_4_2.

  The postures of the obstacle images OBJ_3_1 and OBJ_4_1 are different from each other due to the difference between the viewpoint of the camera C_3 and the viewpoint of the camera C_4. Similarly, the postures of the obstacle images OBJ_3_2 and OBJ_4_2 are also different from each other due to the difference between the viewpoint of the camera C_3 and the viewpoint of the camera C_4.

  That is, the obstacle image OBJ_3_1 is reproduced so as to fall down along a straight line connecting the camera C_3 and the bottom of the obstacle 201, and the obstacle image OBJ_4_1 falls down along a straight line connecting the camera C_4 and the bottom of the obstacle 201. Is reproduced. Similarly, the obstacle image OBJ_3_2 is reproduced so as to fall down along a straight line connecting the camera C_3 and the bottom of the obstacle 202, and the obstacle image OBJ_4_2 falls down along a straight line connecting the camera C_4 and the bottom of the obstacle 202. Is reproduced as follows.

  Since the bird's-eye images BEV_1 to BEV_4 are created based on the road surface, the size of the obstacle images OBJ_3_1 and OBJ_4_1 is different from the height of the camera C_3 and the height of the camera C_4, or from the camera C_3 to the obstacle 201. And the distance from the camera C_4 to the obstacle 201 are different from each other. Similarly, the sizes of the obstacle images OBJ_3_2 and OBJ_4_2 also differ between the height of the camera C_3 and the height of the camera C_4, the difference between the distance from the camera C_3 to the obstacle 202 and the distance from the camera C_4 to the obstacle 202, and the like. Are different from each other.

  When reproducing such obstacles 201 and 202 on the all-around bird's-eye view image, the CPU 12p executes the following processing.

  First, it is determined for each of the common visual fields VW_12 to VW_41 whether any obstacle exists. According to FIG. 10, no obstacle exists in the common visual fields VW_12, VW_23 and VW_41. Therefore, the boundary line BL is set in the initial state in each of the common visual fields VW_12, VW_23, and VW_41. That is, the boundary line BL is set so as to connect the diagonals of the reproduction overlap areas OLD_12, OLD_23, and OLD_41 (see FIG. 11).

  On the other hand, obstacles 201 and 202 exist in the common visual field VW_34, and the obstacle images OBJ_3_1 and OBJ_3_2 are reproduced as the bird's-eye image BEV_3, while the obstacle images OBJ_4_1 and OBJ_4_2 are reproduced as the bird's-eye image BEV_4. For this reason, the boundary line BL is corrected in the following manner.

  First, the size difference between the obstacle images OBJ — 3 — 1 and OBJ — 4 — 1 is calculated as “ΔSZ — 1”, and the size difference between the obstacle images OBJ — 3 — 2 and OBJ — 4 — 2 is calculated as “ΔSZ — 2”. The size difference ΔSZ_1 is obtained by subtracting the size of the obstacle image OBJ_4_1 from the size of the obstacle image OBJ_3_1. Similarly, the size difference ΔSZ_2 is obtained by subtracting the size of the obstacle image OBJ_4_2 from the size of the obstacle image OBJ_3_2.

  If the size difference ΔSZ_1 exceeds the reference value REF, the obstacle image OBJ_3_1 is set as the selected image OBJ_1_SEL. On the other hand, when the size difference ΔSZ_1 falls below the reference value “−REF”, the obstacle image OBJ — 4_1 is set as the selected image OBJ — 1_SEL. That is, if the size difference between the obstacle image OBJ — 3_1 and the obstacle image OBJ — 4_1 is sufficiently large, the larger obstacle image is selected.

  If the size difference ΔSZ_1 is within the range of the reference value “−REF” to the reference value REF, that is, if the size difference between the obstacle image OBJ — 3_1 and the obstacle image OBJ — 4_1 is small, the obstacle is based on the obstacle images OBJ — 3_1 and OBJ — 4_1. A motion vector of the object 201 is detected.

  If the amount of the detected motion vector exceeds the threshold value THmv and the direction of the detected motion vector is the camera C_3 side, the obstacle image OBJ_3_1 is set as the selected image OBJ_1_SEL. If the amount of the detected motion vector exceeds the threshold value THmv and the direction of the motion vector is the camera C_4 side, the obstacle image OBJ_4_1 is set as the selected image OBJ_1_SEL. That is, if the movement of the obstacle 201 is large, an obstacle image corresponding to the camera in the direction in which the obstacle 201 moves is selected.

  If the detected motion vector amount is equal to or smaller than the threshold value THmv, the previous setting of the selected image OBJ_K_SEL is maintained.

  Such processing is also executed for the obstacle 202, whereby one of the obstacle images OBJ_3_2 and OBJ_4_2 is set as the selected image OBJ_2_SEL. The boundary line BL is set to the common visual field VW_34 so that the selection images OBJ_1_SEL to OBJ_2_SEL set in this way are reproduced.

  Therefore, as shown in the upper part of FIG. 12, when both of the obstacles 201 and 202 move toward the camera C_4, the size of the obstacle image OBJ_3_1 is sufficiently larger than the size of the obstacle image OBJ_4_1. If the size of the object image OBJ — 3_2 is substantially the same as the size of the obstacle image OBJ — 4_2, the boundary line BL is set as shown in the lower part of FIG. As a result, the obstacle images OBJ_3_1 and OBJ_4_2 are reproduced on the display device 12.

  Also, as shown in the upper part of FIG. 13, when the obstacles 201 and 202 are close to each other and the obstacle 201 moves toward the camera C_3, while the obstacle 202 moves toward the camera C_4, the obstacle If the size of the image OBJ_3_1 is substantially the same as the size of the obstacle image OBJ_4_1 and the size of the obstacle image OBJ_3_2 is substantially the same as the size of the obstacle image OBJ_4_2, the boundary line BL is set as shown in the lower part of FIG. . As a result, the obstacle images OBJ_3_1 and OBJ_4_2 are reproduced on the display device 12.

  Furthermore, as shown in the upper part of FIG. 14, when the obstacles 201 and 202 are close to each other and both the obstacles 201 and 202 move toward the camera C_3, the size of the obstacle image OBJ_3_1 is the obstacle image. If the size of the obstacle image OBJ_3_2 is substantially the same as the size of the obstacle image OBJ_4_2, the boundary line BL is set as shown in the lower part of FIG. As a result, the obstacle images OBJ — 3_1 and OBJ — 3_2 are reproduced on the display device 12.

  Specifically, the CPU 12p executes processing according to the flowcharts shown in FIGS. The control program corresponding to these flowcharts is stored in the flash memory 14 (see FIG. 1).

  In step S1 shown in FIG. 15, the scene images P_1 to P_4 are captured from the cameras C_1 to C_4. In step S3, bird's-eye view images BEV_1 to BEV_4 are created based on the captured object scene images P_1 to P_4. The created bird's-eye view images BEV_1 to BEV_4 are secured in the work area W1. In step S5, an obstacle detection process is executed. In step S7, an all-around bird's-eye view image is created based on the bird's-eye view images BEV_1 to BEV_4 created in step S3. The created all-around bird's-eye view image is secured in the work area W2. On the monitor screen of the display device 16, a driving support image based on the all-around bird's-eye view image secured in the work area W2 is displayed. When the process of step S7 is completed, the process returns to step S1.

The obstacle detection process shown in step S5 follows a subroutine shown in FIGS. First, in step S11, variables M and N are set to “1” and “2”, respectively. Here, the variable M is a variable that is updated in the order of “1” → “2” → “3” → “4”, and the variable N is “2” → “3” → “4” → “1”. It is a variable that is updated in order.

  In step S13, a difference image in the common visual field VW_MN is created based on the bird's-eye view image BEV_M based on the output of the camera C_M and the bird's-eye view image BEV_N based on the output of the camera C_N. In step S15, it is determined based on the difference image created in step S13 whether an obstacle exists in the common visual field VW_MN.

  If no obstacle is detected from the common visual field VW_MN, a boundary line is initialized in step S17. On the other hand, if an obstacle is detected from the common visual field VW_MN, boundary correction processing is executed in step S19. When the process of step S17 or S19 is completed, it is determined in step S21 whether or not the variable M indicates “4”. If the variable M is less than “4”, the variables M and N are updated in step S23, and then the process returns to step S13. On the other hand, if the variable M is “4”, the process returns to the upper layer routine.

  The boundary line correction process in step S19 shown in FIG. 16 is executed according to a subroutine shown in FIGS.

  First, in step S31, the number of obstacles existing in the common visual field VW_MN is specified as “Kmax”. In step S33, the variable K is set to “1”, and in step S35, the size difference between the obstacle images OBJ_M_K and OBJ_N_K is calculated as “ΔSZ_K”. The size difference ΔSZ_K is obtained by subtracting the size of the obstacle image OBJ_N_K from the size of the obstacle image OBJ_M_K.

  In step S37, it is determined whether or not the size difference ΔSZ_K exceeds the reference value REF. In step S39, it is determined whether or not the size difference ΔSZ_K is less than the reference value “−REF”. If “YES” in the step S37, the process proceeds to a step S41 to set the obstacle image OBJ_M_K as the selected image OBJ_K_SEL. If “YES” in the step S39, the process proceeds to a step S43 to set the obstacle image OBJ_N_K as the selected image OBJ_K_SEL.

  If NO in both steps S37 and S39, the process proceeds to step S45, and the motion vector of the obstacle 20K is detected based on the obstacle images OBJ_M_K and OBJ_N_K. In step S47, it is determined whether or not the amount of the detected motion vector exceeds a threshold value THmv. In step S49, it is determined whether or not the direction of the detected motion vector is on the camera C_M side.

  If NO in step S47, the selected image OBJ_K_SEL is set to the same setting as the previous time. If both step S47 and S51 are YES, it will progress to step S53 and will set the obstruction image OBJ_M_K as selection image OBJ_K_SEL. If “YES” in the step S47, if “NO” in the step S51, the process proceeds to a step S55 to set the obstacle image OBJ_N_K as the selected image OBJ_K_SEL.

  When the setting of the selected image OBJ_K_SEL is completed, the variable K is incremented in step S57, and it is determined in step S59 whether or not the incremented variable K exceeds the variable Kmax. If NO in step S59, the process returns to step S35, and if YES in step S59, the process proceeds to step S61. In step S61, the boundary line BL is set to the common visual field VW_MN so that the selected images OBJ_1_SEL to OBJ_Kmax_SEL are reproduced. When the setting of the boundary line BL is completed, the process returns to the upper hierarchy routine.

  The all-around bird's-eye view creation process in step S7 shown in FIG. 15 follows a subroutine shown in FIG. First, in step S71, the variable M is set to “1”. As described above, the variable M is a variable that is updated in the order of “1” → “2” → “3” → “4”. In step S73, an image outside the boundary line is deleted from the bird's-eye view image BEV_M, and in step S75, it is determined whether or not the variable M has reached “4”. If the variable M is less than “4”, the variable M is updated in step S77, and the process returns to step S73. On the other hand, if the variable M indicates “4”, the process proceeds to step S79, and part of the bird's-eye view images BEV_1 to BEV_4 remaining after the deletion process in step S73 is coupled to each other by coordinate transformation. When the joining process is completed, the routine returns to the upper layer routine.

  As can be seen from the above description, when attention is paid to the cameras C_3 and C_4 among the cameras C_1 to C_4, the cameras C_3 and C_4 grasp the road surface obliquely from above so as to partially have the common visual field VW_34. The CPU 12p captures the scene images P_3 and P_4 output from the cameras C_3 and C_4, respectively (S1), and converts the captured scene images P_3 and P_4 into the bird's-eye images BEV_3 and BEV_4, respectively (S3). Then, the converted bird's-eye images BEV_3 and BEV_4 are combined with reference to the boundary line BL (= weight) assigned to the common visual field VW_34 (S7).

  When a three-dimensional object exists in the common visual field VW_34, the CPU 12p detects a difference in size between the three-dimensional object images that appear in each of the bird's-eye images BEV_3 and BEV_4 (S35), and refers to the detected difference as a boundary. The setting of the line BL is adjusted (S41, S43, S61). The boundary line BL is set so that a larger three-dimensional object image is reproduced.

  The size of the three-dimensional object image appearing in each of the bird's-eye images BEV_3 and BEV_4 varies depending on the height of the three-dimensional object from the road surface and the height difference between each of the cameras C_3 and C_4 and the three-dimensional object. The boundary line BL that is referred to when the bird's-eye view images BEV_3 and BEV_4 are combined is adjusted by paying attention to the difference between the cameras C_3 and C_4 having such a three-dimensional object image size. Thereby, the visibility of a three-dimensional object can be improved.

  In this embodiment, when combining the bird's-eye view images BEV_1 to BEV_4, a part of the image outside the boundary line BL is deleted (see FIG. 5). However, two partial images representing the common visual field may be synthesized by weighted addition, and the weighting amount referred to at the time of weighted addition may be adjusted based on the difference in the size of the three-dimensional object image.

  In this embodiment, an obstacle is detected from the common visual field VW_MN based on the difference image created for the common visual field VW_MN (see steps S13 to S15 in FIG. 17). However, when detecting an obstacle, a stereo vision method or an optical flow method may be used, and an ultrasonic sensor, a millimeter wave sensor, or a microwave sensor may be used.

  The following are notes on the above-described embodiment. This annotation can be arbitrarily combined with the above-described embodiment as long as there is no contradiction.

  The coordinate transformation for generating a bird's eye view image from a captured image as described in the embodiment is generally called a perspective projection transformation. Instead of using the perspective projection conversion, a bird's eye view image may be generated from the captured image by a known plane projective conversion. When using planar projective transformation, a homography matrix (coordinate transformation matrix) for converting the coordinate value of each pixel on the captured image into the coordinate value of each pixel on the bird's eye view image is obtained in advance at the stage of camera calibration processing. . A method for obtaining a homography matrix is known. And when performing image conversion, what is necessary is just to convert a picked-up image into a bird's-eye view image based on a homography matrix. In any case, the captured image is converted into the bird's-eye view image by projecting the captured image onto the bird's-eye view image.

DESCRIPTION OF SYMBOLS 10 ... Steering assistance apparatus C1-C4 ... Camera 12 ... Image processing circuit 12p ... CPU
12m ... Memory 14 ... Flash memory 16 ... Display device 100 ... Vehicle 200 ... Obstacle

Claims (6)

Capture means for capturing a plurality of object scene images respectively output from a plurality of cameras capturing the reference plane obliquely from above so as to partially have a common visual field,
Conversion means for respectively converting a plurality of object scene images captured by the capturing means into a plurality of bird's-eye images representing a state in which the reference plane is captured from directly above;
Combining means for combining a plurality of bird's-eye images converted by the converting means with reference to weights assigned to the common visual field;
First detection means for detecting a difference in size between a plurality of three-dimensional object images respectively appearing in a plurality of bird's-eye images converted by the conversion means when a three-dimensional object exists in the common visual field; and the first detection An image processing apparatus comprising first adjusting means for adjusting the weight with reference to a difference detected by the means.   The image processing apparatus according to claim 1, wherein the first adjustment unit adjusts the weight so that a larger three-dimensional object image is reproduced. Second detection means for detecting the amount of movement of the three-dimensional object in relation to the detection processing of the first detection means;
Second adjustment means for adjusting the weight with reference to the amount of motion detected by the second detection means; and when the difference detected by the first detection means falls below a reference, the first adjustment means instead of the first adjustment means The image processing apparatus according to claim 1, further comprising a control unit that activates the second adjustment unit.   The image processing apparatus according to claim 3, wherein the second adjustment unit adjusts the weight so that a three-dimensional object image of a camera existing in a moving direction of the three-dimensional object is reproduced. The weight includes a boundary defined in the common field of view as a parameter,
The synthesizing unit is configured to delete a partial image outside the boundary line from each of the plurality of bird's-eye images, and to combine the plurality of partial bird's-eye images remaining after the deletion process of the deleting unit. The image processing apparatus according to claim 1, further comprising means. The plurality of cameras are provided on a moving body,
The image processing apparatus according to claim 1, further comprising display means for displaying a synthesized bird's-eye image created by the synthesizing means toward a driver of the moving body.

Images