JP2021025868A - Stereo camera - Google Patents

Abstract

To provide a stereo camera capable of measuring a distance precisely while including a wide angle in which parallax displacement becomes prominent because of windows of a vehicle.SOLUTION: In a stereo camera that is composed by including a left camera and a right camera and photographs the outside of a vehicle via a window formed from a light transmissive member, light axes of the left camera and the right camera are tilted mutually such that the maximum value of the amount of offset before and after transmitting through the window of an optical path in which an image is formed in image pickup devices of the left camera and the right camera through the window from the same light-point decreases as compared with a case where mutual light axes of the left camera and the right camera are parallel.SELECTED DRAWING: Figure 3

Description

The present invention relates to a stereo camera.

Since the stereo camera can acquire the distance information to the object as well as the visual information by the image, various objects around the car (obstacles such as people and cars, white lines (lanes, etc.), road surface, signals, signs, other vehicles (Tail lamps, headlights, etc.) can be grasped in detail. The accuracy required for various object recognition and distance measurement by such a stereo camera is increasing due to the improvement of social recognition of automatic control of vehicles.

One of the factors that reduce the distance measurement accuracy of stereo cameras is the windshield of automobiles. The windshield is formed in a curved shape, and the surface of the windshield has fine irregularities. If the refraction of light on the windshield becomes large due to this, it affects the distance measurement by the stereo camera.

Various techniques related to aiming (calibration) for correcting distance measurement errors have been proposed for stereo cameras mounted on vehicles. For example, there is known an example in which calibration parameters are obtained from the shooting results with and without glass (Patent Document 1). There is also known an example in which a laser range finder is used to calculate a chart installation error to obtain calibration parameters (Patent Document 2).

JP-A-2015-169583 Japanese Unexamined Patent Publication No. 2016-006406

However, the techniques of Patent Documents 1 and 2 are premised on the correction of the ranging error in the standard angle of view (for example, about 45 ° in the horizontal angle of view), and the correction of the ranging error in the wide angle range exceeding the horizontal angle of view of 80 °. No consideration is given. The wider the angle of view, the greater the parallax shift due to the refraction of the optical path when passing through the windshield. If the parallax shift becomes extremely large, it cannot be corrected by software as it is.

An object of the present invention is to provide a stereo camera capable of accurately measuring a distance including a wide-angle range in which parallax deviation becomes remarkable due to a vehicle window.

In order to achieve the above object, the present invention includes a left camera and a right camera, and has the same light spot in a stereo camera that captures the outside world of a vehicle through a window formed of a light transmitting member. The maximum value of the offset amount before and after passing through the window of the optical path that passes through the window and forms an image on the image pickup elements of the left camera and the right camera is such that the optical axes of the left camera and the right camera are parallel to each other. The optical axes of the left camera and the right camera are tilted from each other so as to decrease as compared with a certain case.

According to the present invention, it is possible to measure the distance with high accuracy including a wide-angle range in which the parallax deviation becomes remarkable due to the window of the vehicle.

A block diagram showing a schematic configuration of a stereo camera according to an embodiment of the present invention. A block diagram showing processing during manufacturing, installation, and use by a stereo camera according to an embodiment of the present invention. Schematic diagram imagining the process of optical axis adjustment and optical information calculation in one embodiment of the present invention. Schematic diagram explaining the principle of parallax shift due to windows A graph showing the relationship between the position of the light spot within the horizontal angle of view and the parallax shift when the left and right optical axes are parallel. A graph showing the relationship (before correction) between the position of the light spot in the horizontal angle of view and the parallax deviation of the stereo camera according to the embodiment of the present invention. Flow chart showing details of aiming process (Fig. 2) Schematic diagram showing an example of an aiming chart used in the aiming process A graph showing the relationship between the position of the light spot within the horizontal angle of view of the stereo camera and the parallax deviation after correction by the correction parameters calculated in the correction difference calculation process (Fig. 7). Flow chart showing details of processing by the stereo camera during driving (Fig. 2)

Embodiments of the present invention will be described below with reference to the drawings.

-Stereo camera-
FIG. 1 is a block diagram showing a schematic configuration of a stereo camera 100 according to an embodiment of the present invention. The stereo camera 100 shown in the figure is installed in the driver's cab of the vehicle, photographs the outside world of the vehicle through a window W (FIG. 4) formed of a light transmitting member (glass in the present embodiment), and outside the vehicle. Recognize an object and measure the distance to that object. The window W is typically the windshield of the driver's cab of the vehicle, and is often slightly curved in a convex shape from the inside of the vehicle to the outside of the vehicle. When the term "vehicle" is simply described in the present embodiment, it means a vehicle equipped with the stereo camera 100. Typical examples of objects outside the vehicle recognized by the stereo camera 100 include obstacles such as people and vehicles, white lines (lanes and the like), road surfaces, traffic lights, signs, tail lamps and headlights of other vehicles. The stereo camera 100 may also have a function of giving a warning to the driver driving the vehicle and, in some cases, controlling the brake and steering of the own vehicle based on the recognition result of an object outside the vehicle. The stereo camera 100 of the present embodiment is provided with an aiming function for calculating correction parameters for ranging errors at a vehicle factory or the like.

The stereo camera 100 includes a left camera 11, a right camera 12, and a computer unit 20. In addition to the left camera 11 and the right camera 12, the stereo camera 100 may be provided with yet another camera (not shown). The computer unit 20 includes at least one storage device 21 and at least one CPU 22.

The storage device 21 stores, for example, a captured image and a parallax value obtained by the CPU 22, an optical axis angle of the left camera 11 and the right camera 12, and each program and data required for control. In addition, known position information with respect to the stereo camera 100 for the specified chart for aiming (for example, FIG. 8) installed in the positional relationship set with respect to the left camera 11 and the right camera 12, and the light of the left camera 11 and the right camera 12. The axis angle is also stored in the storage device 21. As the storage device 21, a semiconductor memory such as a ROM or RAM or a magnetic storage device such as a hard disk can be used. A removable storage medium such as a memory card can also be used as the storage device 21. Further, the storage device 21 may be composed of a single storage device or a plurality of storage devices.

The CPU 22 is a circuit that loads a program stored in the storage device 21 and executes various arithmetic processes. The difference between the measured value and the theoretical value of the parallax value obtained by photographing the chart (for example, FIG. 8) by the CPU 22 is the correction parameter of the parallax deviation remaining after setting the angle of the optical axis of the left camera 11 and the right camera 12. Is calculated as (aiming is performed). The theoretical parallax value is a parallax value that should be obtained if the chart is photographed without the window W, and is determined by the CPU 22 from the known installation information of the chart based on the optical axis angles of the left camera 11 and the right camera 12. Calculated by optical path simulation. Through this aiming, the CPU 22 corrects the parallax value actually measured from the captured images of the left camera 11 and the right camera 12 by the correction parameter while the vehicle is running. As a result, the parallax shift due to the presence of the vehicle window between the subject and the left camera 11 and the right camera 12 is corrected, and the true distance to the object can be calculated accurately.

The CPU 22 includes processing units such as an image input interface 23, an image processing circuit 24, an arithmetic processing circuit 25, a CAN interface 26, and a control processing circuit 27. The CPU 22 may be single or plural. That is, the image input interface 23, the image processing circuit 24, the arithmetic processing circuit 25, the CAN interface 26, and the control processing circuit 27 may each be composed of separate CPUs 22, but a plurality of them are composed of the same CPU 22. It may be done. The image input interface 23 is connected to the left camera 11 and the right camera 12, and the CAN interface 26 is connected to the CAN (Controller Area Network) 30. The image input interface 23, the image processing circuit 24, the arithmetic processing circuit 25, the storage device 21, the CAN interface 26, and the control processing circuit 27 are connected to each other via an internal bus 28.

The image input interface 23 controls the left camera 11 and the right camera 12 according to a program stored in the storage device 21, and sequentially captures data of images captured by the left camera 11 and the right camera 12, respectively. The captured image data captured by the image input interface 23 is input to the image processing circuit 24 and the arithmetic processing circuit 25 via the internal bus 28, and is stored in the storage device 21 as needed.

The image processing circuit 24 compares the left and right captured images taken by the left camera 11 and the right camera 12 at the same time, and the left and right captured images are caused by the camera (for example, due to the tilt of the image pickup element and various aberrations of the lens). Correct deviations and perform noise interpolation. The left and right captured images are an image captured by the left camera 11 and an image captured by the right camera 12. The left and right captured images corrected by the image processing circuit 24 are stored in the storage device 21. Further, the parts corresponding to each other in the left and right captured images are determined, the parallax value (the amount of displacement of the positions on the left and right captured images) is calculated for each part, and the parallax image including the parallax value is generated and stored in the storage device 21. Remember in.

The arithmetic processing circuit 25 recognizes various objects around the vehicle based on the captured image and the parallax value stored in the storage device 21. As described above, the various objects are, for example, obstacles such as people and vehicles, white lines (lanes and the like), road surfaces, traffic lights, signs, tail lamps and headlights of other vehicles. The recognition result and the intermediate calculation result of these objects are stored in the storage device 21. The arithmetic processing circuit 25 also calculates command signals (for example, control values for buzzers, brakes, steering, etc.) to the vehicle according to recognition results of various objects based on captured images.

The CAN interface 26 outputs the output information (command signal, object recognition result, etc.) of the stereo camera 100 to the vehicle control system (not shown) via the CAN 30 which is an external in-vehicle network.

In order to prevent abnormal operation, the control processing circuit 27 monitors whether each circuit composed of the CPU 22 has caused abnormal operation, whether an error has occurred during data transfer, and the like.

-Full processing-
It is a block diagram which shows the process at the time of manufacturing, installation, and use by the stereo camera 100 which concerns on this Embodiment of FIG. As shown in the figure, the processing executed by the stereo camera 100 includes processing executed at the time of manufacturing the stereo camera 100, processing executed at the time of installation on the vehicle (aiming), execution at the time of running after delivery, and the like. The processing to be done is included.

The manufacturing process of the stereo camera 100 of the present embodiment includes the steps of optical axis adjustment S11 and optical information calculation S12 as characteristic steps. The purpose is to structurally suppress the parallax deviation (parallax error, FIG. 5) caused by being photographed through the window of the vehicle by the optical axis adjustment S11 (FIG. 6). This manufacturing process is carried out, for example, in a manufacturing factory of the stereo camera 100. Details of the processing at the time of manufacturing will be described later with reference to FIGS. 3 to 6.

The manufactured stereo camera 100 is installed in a specified place and in a specified posture in the driver's cab of the vehicle. At the time of this installation, aiming of the stereo camera 100 is carried out. After structurally suppressing the parallax deviation caused by shooting through the window of the vehicle with the optical axis adjustment S11 at the time of manufacturing, the correction parameter for correcting the parallax deviation that cannot be suppressed by the optical axis adjustment S11 is calculated. That is the purpose of the aiming process at the time of installation. The aiming step at the time of installation includes each step of imaging S21, image correction S22, and correction difference calculation S25. Each step at the time of this installation will be described later with reference to FIGS. 7-9.

After that, when the vehicle equipped with the stereo camera 100 is traveling, the parallax value actually measured from the images taken by the left camera 11 and the right camera 12 is corrected by the correction parameter to calculate the information of the outside world of the vehicle. The traveling process includes the steps of imaging S31, image processing S32, S33, and parallax calculation S34. The process during running will be described later with reference to FIG.

-Manufacturing (optical axis adjustment)-
FIG. 3 is a schematic view imagining the steps of the optical axis adjustment S11 and the optical information calculation S12 in the present embodiment. In the process of the optical axis adjustment S11 shown in FIG. 2, the left camera 11 is arranged so that the optical axes of the stereo cameras 100 are not parallel to each other when the stereo camera 100 is assembled (in the present embodiment, the optical axes are slightly opened in the horizontal direction toward infinity). And the right camera 12 is assembled. The installation angles (optical axis angles) of the left camera 11 and the right camera 12 at that time are preset by an optical path simulation (described later) based on the vehicle information I1. One of the features is that the angles of the optical axes of the left camera 11 and the right camera 12 are defined by the optical path simulation based on the vehicle information I1. In the step of the optical axis adjustment S11, the stereo camera 100 is assembled so that the left camera 11 and the right camera 12 have the optical axis angles obtained by the simulation.

The vehicle information I1 includes, for example, the curvature of the window W (FIG. 4), the radius of curvature of the window W, the thickness of the window W, the refractive index of the window W, and the design positional relationship between the stereo camera 100 installed in the vehicle and the window W ( Information such as distance and angle), which is stored in the storage device 21. The curvature and radius of curvature of the window W are, for example, the curvature and radius of curvature of the line of intersection between the plane including the optical axes of the left camera 11 and the right camera 12 and the window W. When used for photographing the front of a vehicle, for example, the stereo camera 100 is installed so as to face the front with a predetermined distance from the window W when viewed in a horizontal cross section. For example, the stereo camera 100 is installed at a predetermined position in a posture in which the bisectors Ax'of the corners of the left and right optical axes Ax face the straight direction of the vehicle. The left and right optical axes are the optical axis of the left camera 11 and the optical axis of the right camera 12.

In FIG. 3, the general optical axes of the left and right cameras (optical axes parallel to the left and right cameras) are represented by dashed arrows. As shown in the figure, in the present embodiment, the optical axes Ax (solid line) of the left camera 11 and the right camera 12 are inclined to the left and right toward infinity (upward in the figure). doing. The optical axis angles of the left camera 11 and the right camera 12 are set in particular according to the shape of the window W, and although exaggerated in FIG. 3, the angles formed by the left and right optical axes Ax are larger than 0 degrees and less than 1 degree ( For example, about a few minutes). The optical axis Ax passes through the center of the lenses 13 of the left and right cameras, and is orthogonal to the light receiving surface of the image sensor (for example, CMOS) 14 at the center. The angles of view of the left camera 11 and the right camera 12 of the stereo camera 100 are ultra-wide angles, and the horizontal angle of view of the stereo view by the images taken by the left camera 11 and the right camera 12 (the overlapping area of the images taken by the left and right cameras). The horizontal angle of view) is 80 degrees or more (150 degrees or more in this embodiment).

In the process of optical information calculation S12 shown in FIG. 2, the optical characteristic information I2 and the optical axis information I3 of the left camera 11 and the right camera 12 after the optical axis adjustment are specified by calculation. The calculation of the optical characteristic information I2 and the optical axis information I3 is an internal process of the stereo camera 100 and is executed by the CPU 22 (image processing circuit 24). Specifically, for example, a geometric subject is photographed by the left camera 11 and the right camera 12, and image processing is executed by, for example, the image processing circuit 24, whereby the optical characteristic information I2 and the optical axis information I3 are obtained from the captured image. Is calculated. The optical characteristic information I2 includes correction parameters used for correcting various aberrations such as distortion (for example, barrel distortion) of the lenses 13 of the left camera 11 and the right camera 12. Further, the optical axis information I3 includes the angle information of the actual optical axes of the left camera 11 and the right camera 12 after the optical axis adjustment.

-Optical path simulation-
FIG. 4 is a schematic diagram illustrating the principle of parallax shift due to the window. Here, for the sake of simplicity, a pinhole model will be used for explanation. First, consider a light ray that comes out of the light point P0, passes through the optical path L0, and enters the window W. It is assumed that the window W has a curvature that is convex from the driver's cab toward the outside world in a plan view. The optical path L0 is refracted at the refraction points P1 and P2 due to the difference between the material of the window W and the refractive index of air. As a result, the light emitted from the light spot P0 passes through the optical path L1 offset with respect to the optical path L0, passes through the pinhole P3 of the left camera 11 and the right camera 12, and is on the light receiving surface of the image sensor S0 of the left and right cameras. An image is formed at the image pickup point P4. Since the light beam passing through the optical path L0 is offset when passing through the window W and is incident on the imaging points P4 on the left and right light receiving surfaces through the optical path L1, it is based on the parallax X1 + X2 on the left and right captured images. The source of the incident light appears to be at the optical point P0'. That is, in reality, an object existing at the light point P0 at a distance y0 from the image sensor S0 appears to be at the apparent light point P0'by taking a picture through the window W, and the distance to the light point P0 is as it is. It is erroneously recognized as y0'.

Therefore, in the present embodiment, the positional deviation of the imaging point P4 with respect to the ideal imaging point P4'(parallax deviation ΔX1, ΔX2) is the curvature and refractive index of the window W, and the positional relationship between the window W and the stereo camera 100. Simulation calculation is performed based on the vehicle information I1 such as. The image pickup point P4'is the coordinates on the light receiving surface of the left and right image pickup elements S0 on which light emitted from the light point P0 and traveling in the optical path L0 is incident unless refracted by the window W. In the simulation, the calculation of the parallax deviation ΔX1 + ΔX2 is executed while changing the optical axis angles of the left camera 11 and the right camera 12 for each light point P0 in the stereo angle of view, and the optimum optical axis with the smallest maximum value of the parallax deviation ΔX1 + ΔX2 is executed. Identify the angle. By simulating the optical path in this way, the optimum optical axis angle is specified, and the left camera 11 and the right camera 12 are assembled in the above optical axis adjustment S11 so as to have the specified optical axis angle.

By inclining the optical axes of the left camera 11 and the right camera 12 with each other based on this optical path simulation, in the present embodiment, the image sensor 14 of the left camera 11 and the right camera 12 is transmitted through the window W from the same light point. The amount of offset before and after passing through the window W of the optical path to be imaged is suppressed. The maximum value of the offset amount (parallax deviation ΔX1, ΔX2) of the optical path of the light incident on the image sensor 14 is reduced as compared with the case where the left camera 11 and the right camera 12 are parallel to each other.

FIG. 5 is a graph showing the relationship between the position of the light spot in the horizontal angle of view and the parallax deviation when the left and right optical axes are parallel, and FIG. 6 is the light spot in the horizontal angle of view of the stereo camera of the present embodiment. It is a graph which showed the relationship (before correction) of a position and a parallax shift. In the stereo camera of the present embodiment, as described above, the left and right optical axes are tilted in the direction of opening toward infinity based on the simulation. The graphs shown in FIGS. 5 and 6 are raw data and show the relationship between the angle of view position and the parallax deviation (error) in a state where the parallax deviation is not corrected by image processing. Consider a case where the angle of view is widened (for example, the horizontal angle of view is about 150 degrees). In this case, when the left and right optical axes are parallel, the parallax deviation is small within the angle of view of 80 degrees, that is, within the range of 40 degrees (-40 degrees to 40 degrees) to the left and right from the front of the stereo camera. However, the parallax shift increases sharply from around 80 degrees, and reaches several tens of pixels at the end of the angle of view (the region near 150 degrees). If the parallax shift increases to this extent, it cannot be corrected by image processing.

On the other hand, in the present embodiment, by setting the optical axis angles of the left camera 11 and the right camera 12 to the angles obtained by simulation, the maximum value of the parallax deviation due to the offset of the optical path before and after passing through the window W is left and right. It decreases as shown in FIG. 6 as compared with the case where the optical axes are parallel. In the simulation results in the figure, it is estimated that parallax shift occurs in the negative direction from 40 degrees to the vicinity of the angle of view edge, but the maximum value (absolute value) of parallax shift is suppressed to less than 10 pixels (a few pixels). Be done.

-Aiming-
FIG. 7 is a flowchart showing the details of the aiming process shown in FIG. 2, and FIG. 8 is a schematic diagram showing an example of an aiming chart used in the aiming process.

In the aiming process, the stereo camera 100 is attached to the vehicle as designed, a chart (FIG. 8) is installed at a predetermined position (here, in front of the vehicle) so as to have a preset positional relationship with respect to the vehicle, and the stereo camera 100 is installed. The parameter for correcting the parallax deviation is estimated. In this case, for example, the storage device 21 stores known position information (installation information I5) about the specified chart for aiming installed in the positional relationship set with respect to the left camera 11 and the right camera 12. Then, the CPU 22 executes a correction for removing the influence of the window W based on the optical axis information I3 and the installation information I5 for the captured images of the charts captured by the left camera 11 and the right camera 12. In this case, as a correction parameter for correcting the parallax deviation remaining after the optical axis adjustment S11, it was calculated from the parallax value calculated from the captured image before the correction (before removing the influence of the window W) and the captured image after the correction. The difference in parallax values can be calculated by the CPU 22. Specifically, in the aiming step of this example, in addition to the basic steps of imaging S21 (FIG. 2), image correction S22, parallax calculation S23, chart analysis S24 and correction difference calculation S25, monocular image processing for improving accuracy is performed. The steps of S26 and chart installation parameter estimation S27 are included.

-Imaging S21
When starting the aiming process, first, the operation unit (not shown) is operated to switch the operation mode of the stereo camera 100 to the aiming mode. As a result, first, in the step of imaging S21 (FIG. 2), the captured image (raw data) of the chart (FIG. 8) taken by the left camera 11 and the right camera 12 is acquired by the CPU 22 (for example, the image input interface 23). The chart includes a portion in which a plurality of vertical bars are arranged at equal intervals in the left-right direction, a black frame portion, and a random pattern portion displayed inside the black frame. The vertical bar may be replaced with another pattern, but it is necessary that the same pattern is arranged at equal intervals in the left-right direction.

-Image correction S22
The captured image input in the step of the imaging S21 has distortion (for example, barrel distortion) due to the lens 13. In the subsequent step of image correction S22, the CPU 22 (for example, the image processing circuit 24) corrects the distortion of the left and right captured images based on the optical characteristic information I2 obtained in the step of optical information calculation S12 (FIG. 2) at the time of manufacturing. To do.

-Parallax calculation S23
In the subsequent step of parallax calculation S23, the CPU 22 (for example, the image processing circuit 24) calculates the parallax value for the entire stereo angle of view from the left and right captured images corrected for distortion.

-Chart analysis S24
In the step of chart analysis S24, the CPU 22 (for example, the image processing circuit 24) calculates the parallax deviation (parallax residual) that still remains after the optical axis adjustment S11 at the time of manufacture. Here, the CPU 22 first calculates (actually measures) the parallax value A for a random pattern in a vertical bar or a square frame arranged horizontally in the upper region of the chart. Since the positional relationship between the stereo camera 100 and the chart (installation information I5) is known, the CPU 22 next, based on the installation information I5 and the optical axis information I3 read from the storage device 21, when there is no refraction by the window W. The measured parallax value B (ideal value) is calculated. The parallax value B is the vertical bar at the imaging point P4'described in FIG. 4 based on the known position information (installation information I5) of the subject (vertical bar or random pattern) and the optical axis information I3 calculated at the time of manufacture. It is obtained by estimating for each (or random pattern).

-Correction difference calculation S25
Then, the CPU 22 (for example, the image processing circuit 24) calculates the difference between the parallax value A (measured value) and the parallax value B (ideal value) in the step of the correction difference calculation S25. The difference between the parallax values A and B becomes a correction parameter for the parallax deviation (residual of the parallax deviation) that remains even after the optical axis adjustment S11 at the time of manufacturing is performed. The magnitude of the parallax deviation (residual) due to the window W changes depending on the horizontal position of the angle of view. Therefore, the CPU 22 summarizes the parallax deviation correction parameters as a function or table for the horizontal position within the angle of view, and stores this as the optical axis characteristic I4 in the storage device 21. When calculating the correction parameter for the horizontal position, the parallax of the vertical bar in the upper region of the chart may be used as the basis. When calculating the correction parameter in consideration of the vertical position, the parallax of the random pattern in the square frame in the center of the chart may be used as the basis.

-Monocular image processing S26, chart installation parameter estimation S27
As described above, since the chart installation information I5 and the optical axis information I3 are used for the chart analysis S24 and the correction difference calculation S25, the chart installation accuracy may affect the calculation accuracy of the correction parameters. When it is necessary to consider the chart installation error (error of the positional relationship between the chart and the stereo camera 100), in the present embodiment, the parameters related to the chart installation information I5 are estimated in the steps of monocular image processing S26 and chart installation parameter estimation S27. To do. As a result, the installation information I5 can be corrected and the accuracy of the correction parameter for the parallax deviation can be improved.

The method for estimating monocular image processing and chart installation parameters is described in detail in International Publication No. WO2018 / 0425954, and is described in the same document in monocular image processing S26 and chart installation parameter estimation S27. The technology can be applied. Roughly speaking, the interval (number of pixels) of vertical lines is examined on the captured image of the chart, and the statistical value of the difference between the position of the chart and the interval on the design image of the vertical bar obtained from the size of the image sensor 14 (for example, average). Value) is calculated as the amount of deviation of the installation position of the chart. By correcting the installation information I5 using the deviation amount of the installation position of the chart as a correction parameter, the optical axis characteristic I4 that suppresses the influence of the installation error of the chart can be obtained in the aiming process. However, the steps of the monocular image processing S26 and the chart installation parameter estimation S27 can be omitted if the chart installation accuracy does not matter.

FIG. 9 is a graph showing the relationship between the position of the light spot in the horizontal angle of view and the parallax deviation after correction by the correction parameter calculated in the step of correction difference calculation S25. By suppressing the parallax deviation caused by hardware with the optical axis adjustment S11, as shown in the figure, the parallax deviation is softly suppressed to almost 0 for the entire angle of view using the correction parameters calculated in the aiming process. be able to. If an attempt is made to correct by image processing while the parallax deviation due to the hardware is large without going through the optical axis adjustment S11, the required image cache capacity increases more than necessary, which is not preferable.

-Running-
FIG. 10 is a flowchart showing details of processing (FIG. 2) by the stereo camera during traveling. While the vehicle is running, the CPU 22 corrects the parallax values actually measured from the images taken by the left camera 11 and the right camera 12 by the correction parameters obtained in the aiming step. Specifically, the CPU 22 first captures the outside world with the left camera 11 and the right camera 12 (imaging S31 in FIG. 2), and corrects the distortion of the left and right captured images using the optical characteristic information I2 (image correction S32). , The image is further corrected by using the optical axis characteristic I3 obtained by aiming (image correction S33). In the image correction S33, the distance from the actual image point (image point P4 in FIG. 4) to the actual light point (light point P0 in FIG. 4) is calculated in consideration of the refraction of the optical path by the window W. The left and right captured images are corrected. That is, the distortion of the image due to the window W that could not be completely removed by the optical axis adjustment S11 is removed by the image correction S33. Then, the CPU 22 calculates the parallax value from the left and right captured images after the image correction S33 (parallax calculation S34), detects the object within the angle of view, and calculates the distance to the object (object detection / distance measurement S35). The result (object information I6) is stored in the storage device 21 and output to the CAN 30.

-Effect-
(1) In the stereo camera 100 that captures the outside world through the window W, the windows W are made non-parallel by making the left and right optical axes non-parallel according to the simulation based on the vehicle information I1 such as the curvature of the window W in the optical axis adjustment S11. It is possible to suppress the parallax shift due to the refraction of the optical path when passing through. As a result, although it depends on the curvature and thickness of the window W, the parallax deviation due to the influence of the window W can be suppressed to a maximum of, for example, several pixels even in a wide angle range of the angle of view. In this way, even if the angle is wide, the parallax shift can be suppressed at the low data level, so even if the parallax shift cannot be completely removed by adjusting the optical axis, the residual is small, so the image processing makes it softer with a smaller cache capacity. It can be easily corrected. As described above, according to the stereo camera 100 of the present embodiment, it is possible to accurately measure the distance including the wide-angle range in which the parallax deviation becomes remarkable due to the window W of the vehicle.

(2) After the optical axis adjustment S11, the optical axis angles of the left camera 11 and the right camera 12 can be calculated by the CPU 22 from the captured images of the left camera 11 and the right camera 12 and stored in the storage device 21. The left and right optical axis angles are design values obtained by simulation, but in reality, errors may occur with respect to the design values when manufacturing or assembling the camera unit. Therefore, the actual optical axis angle after the optical axis adjustment S11 is calculated by the internal processing of the stereo camera 100 based on the actual captured image and stored as the optical axis information I3, so that the actual optical axis angle is used at the time of aiming. It is possible to calculate the parallax correction parameters with high accuracy.

(3) By tilting the optical axes of the left camera 11 and the right camera 12 so as to be separated from each other toward infinity, the parallax deviation that increases in the wide-angle range due to the refraction of the optical path when passing through the window W is rational. Can be suppressed to.

(4) The window W has a curvature, and is often formed in a gently convex shape in a plan view from the inside of the vehicle to the outside of the vehicle, for example. The curvature of the window W may differ depending on the vehicle type, and if the curvature is different, the angle of the window W with respect to the stereo camera 100 also changes, so that the degree of deviation of the optical path (the degree of occurrence of parallax deviation) also differs. Therefore, in the present embodiment, by setting the optical axis angles of the left camera 11 and the right camera 12 according to the shape of the window W, the parallax deviation can be flexibly suppressed according to the shape of the window W.

(5) There is also a technique for expanding (widening) the field of view by opening the left and right optical axes and combining the stereo viewing area in the center of the field of view with the left and right monocular field of view, but the field of view is expanded including the monocular field of view. When opening the left and right optical axes aiming at, the opening of the optical axes becomes excessively large. In this case, focusing on the stereo visual region, the opening of the optical axis is excessive and the parallax shift is rather widened. In the first place, the monocular visual region becomes large, which is disadvantageous in covering the wide-angle region in the stereo visual region. On the other hand, in the present embodiment, the angle formed by the optical axes of the left camera 11 and the right camera 12 is larger than 0 degrees and less than 1 degree, for example, about several minutes. By slightly opening the optical axis within the range for suppressing parallax deviation in this way, it is possible to obtain a wide-angle stereo angle of view by making wide use of the effective fields of view of the left camera 11 and the right camera 12. It is advantageous in widening the horizontal angle of view of the stereo view by the images taken by the left camera 11 and the right camera 12 to 80 degrees or more, and even if the horizontal angle of view of the stereo view is 150 degrees or more, the data level is low. The parallax shift can be suppressed to the extent that it can be easily corrected by software. As shown in FIG. 6, the parallax deviation can be suppressed at a low data level in the entire range of the ultra-wide angle of view including the angle of view of about 80 degrees and the angle of view of 150 degrees.

(6) The optical axis angles of the left camera 11 and the right camera 12 are set in the optical axis adjustment S11 so that the maximum value of the parallax deviation due to the offset of the optical path before and after passing through the window W is less than 10 pixels. The residual of the parallax can be easily corrected in the subsequent aiming process.

(7) In the aiming step, the parallax correction parameter can be calculated by the internal processing of the computer unit 20. Specifically, the influence of the window W is removed by the correction based on the optical axis information I3 and the installation information I5 for the left and right captured images, and the parallax value (actual measurement value) based on the captured image before the correction and the captured image after the correction are performed. The parallax correction parameter is calculated by calculating the difference of the parallax value (ideal value) based on. In carrying out the aiming process, it is not necessary to remove the window W from the vehicle or separately prepare an instrument such as a laser range finder, and the labor required for aiming can be reduced and the time can be shortened.

(8) A correction parameter for correcting the influence of the window W is obtained by the optical axis adjustment S11 and the aiming process, and after delivery, the parallax value actually measured from the left and right captured images while the vehicle is running is corrected by this correction parameter. With a wide viewing angle, you can accurately grasp obstacles outside the vehicle.

11 ... left camera, 12 ... right camera, 14 ... image sensor, 21 ... storage device, 22 ... CPU, 100 ... stereo camera, Ax ... optical axis, I1 ... vehicle information (window shape), I3 ... optical axis information ( Optical axis angle), I4 ... Optical axis characteristics (correction parameter), I5 ... Installation information (known position information about the chart), L0, L1 ... Optical path, P0 ... Optical spot, S0 ... Image sensor, W ... Window, Δx1 , Δx2 ... Misalignment

Claims (10)

It is a stereo camera that includes a left camera and a right camera and captures the outside world of the vehicle through a window formed of a light-transmitting member.
The maximum value of the amount of offset before and after passing through the window of the optical path that passes through the window from the same light spot and forms an image on the image sensor of the left camera and the right camera is the mutual light of the left camera and the right camera. A stereo camera in which the optical axes of the left camera and the right camera are inclined with respect to each other so that the axes are reduced as compared with the case where the axes are parallel. In the stereo camera of claim 1,
A CPU that calculates the optical axis angle of the left camera and the right camera from the images taken by the left camera and the right camera.
A stereo camera including a storage device that stores the calculated optical axis angles of the left camera and the right camera. In the stereo camera of claim 1, the stereo camera in which the optical axes of the left camera and the right camera are inclined so as to be separated from each other toward infinity. The stereo camera according to claim 1, wherein the optical axis angles of the left camera and the right camera are set according to the shape of the window. The stereo camera according to claim 1, wherein the angle formed by the optical axes of the left camera and the right camera is greater than 0 degrees and less than 1 degree. The stereo camera according to claim 1, wherein the horizontal angle of view of the stereo view of the images taken by the left camera and the right camera is 80 degrees or more. The stereo camera according to claim 1, wherein the horizontal angle of view of the stereo view of the images taken by the left camera and the right camera is 150 degrees or more. In the stereo camera of claim 6, the optical axis angles of the left camera and the right camera are set so that the maximum value of the parallax deviation due to the offset of the optical path before and after passing through the window is less than 10 pixels. .. In the stereo camera of claim 1,
A storage device that stores known position information about a specified chart for aiming installed in a positional relationship set with respect to the left camera and the right camera.
With respect to the captured images of the chart taken by the left camera and the right camera, corrections are made to remove the influence of the window based on the optical axis angles of the left camera and the right camera and the known position information of the chart. A stereo camera including a CPU that calculates the difference between the parallax value calculated from the captured image before correction and the parallax value calculated from the captured image after correction as a correction parameter for the parallax deviation remaining after setting the angle of the optical axis. .. The stereo camera according to claim 9, wherein the CPU corrects the parallax value actually measured from the images taken by the left camera and the right camera by the correction parameter while the vehicle is running.

Images