JP7312594B2 - Calibration charts and calibration equipment - Google Patents

Description

The present invention relates to a chart and a calibration tool used for calibrating an imaging device.

A game is known in which a user's body or markers are photographed with a camera, and the area of the image is replaced with another image and displayed on a display (see Patent Document 1, for example). Technologies for acquiring the position and movement of a subject or a camera itself and recognizing the attributes of a subject by detecting and analyzing images in captured images are widely used not only in cameras installed in game devices and information terminals, but also in systems including security cameras, in-vehicle cameras, and cameras installed in robots.

In order to ensure processing accuracy in these systems, calibration processing is performed to obtain in advance information unique to the imaging device, such as internal parameters, distortion correction coefficients, and external parameters. The internal parameters define the relationship between the positional coordinates of pixels in the captured image and the positional coordinates in the camera coordinate system with the optical center as the origin and the length as the unit, and represent the lens characteristics determined by the focal length, the relative position of the origin, the shear coefficient, and the scale factor.

The distortion correction coefficient is a coefficient for correcting barrel distortion and distortion in the circumferential direction due to the lens. The extrinsic parameters define the relationship between the camera coordinate system and the world coordinate system, and are used for the purpose of aligning the tilts of the images captured by each camera, particularly by simultaneously calibrating a multi-view camera such as a stereo camera. As a calibration technique, Zhang's method is widely known, in which a planar chart of a checkered pattern is photographed from a plurality of viewpoints, and parameters are optimized so that the positions of the feature points on the photographed image and the positions of the feature points on the plane of the chart in the real space have a correct correspondence relationship (see Non-Patent Document 1).

EP-A-0999518

Zhengyou Zhang, "A Flexible New Technique for Camera Calibration", Microsoft Research Technical Report, MSR-TR-98-71, December 2, 1998.

With Zhang's method, it is necessary to set up imaging devices and charts in several positions and postures, and then take photographs repeatedly, which is a heavy workload. Although the burden of the photographing work can be reduced by providing certain constraints, it is conceivable that such constraints impair versatility. In addition, it is difficult to obtain the accuracy of Zhang's method with a simple calibration method.

SUMMARY OF THE INVENTION The present invention has been made in view of these problems, and an object thereof is to provide a technique capable of highly accurately calibrating an imaging device with a small number of man-hours.

In order to solve the above problems, one aspect of the present invention relates to a calibration chart. This calibration chart is a chart for calibrating an image pickup device, and is characterized in that a plurality of chart surfaces representing chart patterns on the surface and forming a predetermined angle are inclined by a predetermined angle around independent axes.

Another aspect of the invention relates to a calibration instrument. This calibration instrument is characterized by fixing the above-described calibration chart and a fixture for fixing an imaging device for capturing an image of the calibration chart.

Any combination of the above constituent elements, and conversion of expressions of the present invention between methods, devices, systems, etc. are also effective as aspects of the present invention.

According to the present invention, it is possible to calibrate an imaging device with high accuracy with a small number of man-hours.

It is a figure which shows the structure of the calibration system applicable to this Embodiment. FIG. 2 is a diagram for explaining in more detail a chart for calibration in FIG. 1; FIG. 3 is a diagram showing the configuration of functional blocks of the calibration device according to the present embodiment; FIG. 4 is a flow chart showing a processing procedure in which the calibration device according to the present embodiment performs calibration based on the photographed image of the chart. FIG. 10 is a diagram illustrating an example of an imaging environment of a chart in which one side of two charts having the same height are matched to form a predetermined angle, and an image of the chart; FIG. 10 is a diagram for explaining a chart in which the chart surface is inclined by a predetermined angle around an independent axis in the embodiment; 4A and 4B are diagrams exemplifying a photographing environment of a chart and a photographed image thereof according to the present embodiment; FIG. 8 is a trihedral view of the chart shown in FIG. 7; FIG.

This embodiment relates to a chart used for calibration of an imaging device. As for the method of calibration using this chart and the apparatus for performing calibration, those disclosed in International Publication No. 2018/235163 can be applied. An outline of this will be described below.

FIG. 1 shows the configuration of the calibration system disclosed in the above document and applicable to the present embodiment. This calibration system includes an imaging device 12 that performs calibration, a calibration chart 200, a jig 14 that fixes the imaging device 12 and chart 200 in a predetermined positional relationship, and a calibration device 10 that acquires camera parameters through calibration.

The imaging device 12 has a camera that captures an image of an object, and a mechanism that generates output data of a captured image by performing general processing such as demosaic processing on the output signal of the camera, and sends the output data to the calibration device 10 . The camera includes a visible light sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, which is used in general digital cameras and digital video cameras, or an infrared sensor. When the camera is an infrared sensor, an infrared irradiation mechanism (not shown) may be further provided.

The imaging device 12 may have only one camera, or may be a so-called stereo camera in which two cameras are arranged left and right at a known interval. Alternatively, it may have three or more cameras. The configuration of the camera provided in the imaging device 12 is determined according to the content of information processing performed using the imaging device 12 . For example, when acquiring the position of an object such as a user or a controller in a three-dimensional space to realize an electronic game, it is conceivable to realize the imaging device 12 with a stereo camera.

In the case of introducing a TOF (Time Of Flight) technique for determining the position of an object based on the time it takes to irradiate infrared rays and detect the reflected light, it is conceivable that the imaging device 12 is realized by an infrared irradiation mechanism and an infrared camera. With a similar configuration, it is conceivable to irradiate infrared rays in a random pattern and photograph the reflected light with a stereo camera to obtain the parallax using the infrared pattern as a feature point and specify the position in the three-dimensional space. In addition, the monocular camera/multi-lens camera and the visible light camera/infrared camera may be combined in any way depending on the purpose of use and the technology to be introduced.

The calibration device 10 acquires data of an image of the calibration chart 200 photographed by the imaging device 12, performs calibration calculations based on the data, and derives internal parameters and external parameters. These parameters are commonly known. Using these parameters, the relationship between the pixel m (u, v) in the captured image and the position M (X, Y, Z) in the world coordinate system is expressed as follows.

where s is the scale factor and A is the internal parameter. [R|t] is an extrinsic parameter consisting of a rotation matrix R and a translation vector t for transforming the world coordinate system into the camera coordinate system. The internal parameter A is expressed as follows.

Here, f is the focal length, k _u and k _v are horizontal and vertical scale factors of the image plane, and (u ₀ , v ₀ ) are the position coordinates of the optical axis on the image plane. Note that the skew distortion is assumed to be 0 in the above equation.

Furthermore, when barrel distortion and distortion in the circumferential direction due to the lens are taken into account, the point at position coordinates (x, y) centered on the optical axis in the photographed image is corrected to position coordinates (x', y'), for example, by the following approximation formula.
x′=(1+k ₁ r ² +k ₂ r ⁴ )x+2p ₁ xy+p ₂ (r ² +2x ² )
y'=(1+ _k1r2 + _k2r4 ) ^y + _{p1 (} ^r2 + ^2y2 )+ ^2p2xy
where r ² =x ² +y ²
is. Also, k1 and k2 are parameters due to barrel distortion, and p1 and p2 are parameters due to circumferential distortion, which are collectively referred to as distortion correction coefficients.

The calibration device 10 obtains intrinsic parameters, distortion correction coefficients, and extrinsic parameters so that the image of the feature point in the captured image appears at a position that reflects the original position in the three-dimensional space. Hereinafter, those parameters may be collectively referred to as "camera parameters". A basic algorithm for camera parameter derivation processing may be an existing technique, and in particular, Zhang's method described in Non-Patent Document 1 can be used. That is, the image of the feature point represented on the chart 200 is extracted from the photographed image, and its position coordinates are obtained. Then, the initial values of the internal parameters are obtained based on the position coordinates and the positions of the feature points in three dimensions, and the internal parameters and the external parameters are finally determined by a nonlinear optimization process called bundle adjustment.

On the other hand, in this technique, the calibration chart 200 is a three-dimensional structure having a plurality of planes that are not parallel to the imaging surface. In the illustrated example, the chart 200 is in a state in which two boards having the same area are aligned with one side of each other, and the board is erected so as to form an angle θ (0<θ<180°) with the relevant side as an axis. The jig 14 fixes both so that the optical axis of the imaging device 12 intersects the axis of the chart 200 and is positioned and oriented at a predetermined distance from the chart 200 .

Of the chart 200, each plane on the imaging device 12 side represents a chart pattern including a checker pattern in which black and white are reversed between adjacent rectangles among the rectangles partitioned in a grid pattern, and a marker for identifying the plane. In this case, the feature points are the vertices of each rectangle. By arranging the feature points with variations in the depth direction in this way, it is possible to obtain information similar to that obtained by photographing a single plane chart from different viewpoints in a single photographing operation. Note that the pattern represented in the chart 200 is not limited to the checkered pattern, and it is sufficient if the feature points are distributed in a shape and arrangement that can be easily detected by an existing search method. For example, by adopting an orthogonal feature point array such as a circle grid in which black circles are arranged vertically and horizontally, feature point detection and attribute acquisition can be facilitated.

FIG. 2 is a diagram for explaining the calibration chart 200 in FIG. 1 in more detail. (a) shows chart patterns of checkers and markers represented on respective planes that constitute the chart 200. A chart pattern 212a is shown on the left side of the imaging device 12, and a chart pattern 212b is shown on the right side of the imaging device 12. FIG. (b) shows the state of the chart 200 viewed from the imaging device 12 when the two planes face the imaging device 12 at an angle θ as shown in the bird's-eye view on the right side of the figure.

In this technique, when the plane of the chart is tilted with respect to the imaging plane, the chart pattern appears to be a general checkered pattern. In the case of other chart patterns as well, the standard chart pattern determined in consideration of search processing and the like is made to appear in the photographed image with the plane of the chart 200 tilted. For this reason, as shown in (a), the chart patterns 212a and 212b shown on each plane are obtained by adding a reverse perspective to the original checker pattern based on the inclination.

Qualitatively, the scale of the pattern on the surface of the chart 200 is increased as it is positioned farther from the imaging surface. By doing so, an image in which figures of the same size are evenly arranged in the vertical and horizontal directions is captured even though they are at different distances in the real space. As a result, it is possible to obtain information necessary for calibration in one shot, and to avoid complication of feature point detection processing.

FIG. 3 shows the configuration of functional blocks of the calibration device 10. As shown in FIG. The calibration device 10 includes an image acquisition unit 34 that acquires captured image data, a feature point information acquisition unit 36 that associates the position coordinates of feature points in the captured image with the position coordinates of corresponding feature points in the 3D model, a projective transformation parameter storage unit 40 that stores transformation parameters used therein, a feature point information storage unit 42 that stores correspondence information between feature points, a calibration unit 38 that performs calibration using the correspondence information, and a camera parameter storage unit 44 that stores camera parameters obtained as a result of calibration. Prepare.

The image acquisition unit 34 acquires the data of the captured image of the chart 200 from the imaging device 12 . When the imaging device 12 is composed of a multi-lens camera, all the data of the images captured by each camera are acquired. The projective transformation parameter storage unit 40 stores, for each plane constituting the chart, transformation parameters for transforming indices given to all feature points in the chart pattern before perspective into position coordinates of feature points in the chart pattern on the chart 200 .

The feature point information acquisition unit 36 detects feature points and marker images from the captured image, and associates the position coordinates of the detected feature point images with the position coordinates of the feature points in the 3D model for each plane identified by the markers. At this time, the index is specified for each feature point, and the position coordinates of the 3D model are specified from the index using the transformation parameters stored in the projective transformation parameter storage unit 40 . As a result, the position coordinates in the photographed image and the position coordinates of the 3D model are associated with each feature point.

The acquired correspondence information is stored in the feature point information storage unit 42 . The calibration unit 38 uses the feature point correspondence information as input data to identify camera parameters by an existing algorithm. The specified parameters are stored in the camera parameter storage unit 44 . Alternatively, the parameters may be transmitted to the imaging device 12 . By holding camera parameters in the imaging device 12 , appropriate correction can be performed inside the imaging device 12 . Alternatively, an image captured by the imaging device 12 may be used to transmit to an information processing device that performs information processing such as a game. As a result, analysis can be performed after performing appropriate corrections such as aligning epipolar lines with the captured stereo images.

FIG. 4 is a flow chart showing a processing procedure in which the calibration device 10 performs calibration based on the photographed image of the chart. First, the image acquiring unit 34 acquires the data of the captured image from the imaging device 12 (S20). When the imaging device 12 is a multi-lens camera such as a stereo camera, all data of images shot by each camera are acquired. Alternatively, data of images shot multiple times under the same shooting conditions may be obtained. At this time, the image acquiring unit 34 may perform noise reduction processing by adding each frame image. For example, images for 16 frames are added for each corresponding pixel to obtain an average value, and an image having the average value as a pixel value is generated.

Next, the feature point information acquisition unit 36 extracts the image of feature points from the captured image (S22). Various methods have been put into practical use for detecting the vertices of the squares of the checker pattern as feature points, and any of them may be employed here. For example, according to the method using the Harris feature amount, it is possible to extract the feature value of a predetermined value or more as the vertex (corner) of the grid (C. Harris and M. Stephens, "A combined corner and edge detector", Proceedings of the 4th Alvey Vision Conference, 1988, pp. 147-151).

However, since the position coordinates obtained by this method may not be accurate enough, it is desirable to obtain the position coordinates (u, v) in the captured image with sub-pixel accuracy using two edge luminance gradients. For this processing, the cv::FindCornerSubPix() function of OpenCV, which is a typical open source image processing software library, can be used. Furthermore, the feature point information acquisition unit 36 obtains the index of each feature point based on the positional relationship with the image of the marker, and converts it into position coordinates on the chart 200 using the transformation parameters stored in the projective transformation parameter storage unit 30 . As a result, feature point information is generated by associating the position coordinates of the feature points on the captured image with the position coordinates of the 3D model of the chart 200 (S24).

At this time, the planes forming the chart 200 are identified by the shapes of the markers, and the transformation parameters associated with the respective planes are read out from the projective transformation parameter storage unit 30 and used. In the case of a multi-lens camera, the processing of S24 is repeated for images captured by all cameras (N of S26). When feature point information can be generated for images captured by all cameras (Y of S26), the calibration unit 38 performs calibration using the feature point information and acquires camera parameters (S28). The existing technology described in Non-Patent Document 1 or the like can be used for the processing of S28.

According to the technique described above, chart surfaces having a plurality of angles with respect to the imaging surface can be photographed at once, and feature points represented in the chart pattern and feature points appearing in the photographed image can be easily associated with each other. Therefore, it is possible to reduce the number of man-hours while maintaining the accuracy of calibration. On the other hand, due to the convenience of combining surfaces with different angles, it is conceivable that the image of the characteristic points will not appear uniformly in the image obtained by photographing them.

FIG. 5 exemplifies the photographing environment of the chart in which one side of the two charts having the same height is matched to form a predetermined angle as shown in FIG. 2, and the photographed image thereof. As shown in (a), the imaging device 12 is installed in front of the pattern of the chart 200 to take an image. At this time, for example, the illumination 204 is installed behind the imaging device 12 and projected onto the chart pattern so that the image can be captured clearly. However, as the distance between the surfaces of the chart 200 increases toward the inner side where the two planes of the chart 200 are joined, and the distance from the illumination 204 increases, the light is less likely to hit the surface.

As a result, the pattern appears dark like the area 206 in the photographed image of (b), and there is a possibility that the feature point extraction accuracy will be lower than in other areas. If the light of the illumination 204 is increased so as to obtain sufficient brightness in the region 206, it is conceivable that the feature point extraction accuracy will not be uniform due to surface highlights and the like. Therefore, in the present embodiment, a chart is formed which is inclined by a predetermined angle θ of 0°<θ<180° around an independent axis for each chart surface, in addition to the open angle around the common axis forming the sides of the chart surface.

FIG. 6 is a diagram for explaining a chart formed by inclining the chart surface at a predetermined angle around an independent axis. Similar to the chart 200 shown in FIG. 5, the upper chart 200 shows a chart of two planes forming a predetermined angle θ ₁ around the axis in the direction of the side to be joined, which is the Y-axis in the vertical direction in the figure. In this embodiment, the chart 210 is formed by tilting the chart plane by a predetermined angle θ ₂ about the axis perpendicular to the Y axis, which is the horizontal X-axis on one chart plane in the figure. This tilt corresponds to rotation in the pitch direction as viewed from the chart plane. Alternatively, the other chart plane may be tilted by a predetermined angle in the opposite direction about the horizontal X'-axis on the other chart plane.

FIG. 7 exemplifies the photographing environment of the chart of the present embodiment and the photographed image thereof. As shown in (a), the chart 214 has the shape of the original chart 200, i.e., two chart planes having an angle of θ ₁ about the vertical axis corresponding to the side to be joined, each inclined by ±θ ₂ /2 about the horizontal axis of half height. For example, θ ₂ /2=30°. Imaging device 12 is preferably placed so that its optical axis passes through the junction of the two surfaces. However, the position of the imaging device 12 is not limited as long as the chart patterns on the two sides are within the field of view.

For this chart 214, lights 216b and 216c are arranged behind each chart surface in addition to the light 216a on the imaging device 12 side. Since the two chart faces are tilted on an axis other than their common axis, there is an incident path between them such that light shining from behind one face reaches the front face of the other face. Therefore, by installing the lights 216a and 216b respectively behind the two surfaces, it is possible to project light from a near position on either surface. As a result, it is possible to brighten the back side of the chart 200 shown in FIG.

The positions of the lights 216a, 216b, 216c are optimized so that the entire chart pattern is illuminated uniformly. For example, the lights 216b, 216c behind each chart face are differentiated in the height direction so that sufficient incident light can be obtained from between the faces. It also optimizes the light intensity to avoid surface highlights. After obtaining the optimum positional relationship between the lights 216a, 216b, 216c and the chart 214, they are fixed by the jig 14 as shown in FIG. Also, the imaging device 12 can be fixed in the same manner as the jig 14 . (b) shows an image captured in this way. Compared with the photographed image of FIG. 5B, the pattern image is obtained with more uniform brightness in the entire area of the chart surface.

Note that the pattern represented in the chart 214 is preferably a predetermined pattern that has undergone conversion according to changes in the distance from the imaging plane to the plane so that the predetermined pattern can be obtained in the captured image. For example, as described above, a perspective corresponding to the orientation of each surface with respect to the imaging surface is applied so that the patterns are arranged in the vertical direction and the horizontal direction on the photographed image in a substantially uniform size. As a result, extraction of feature points and association with the 3D model of the chart pattern can be performed with high accuracy. The pattern represented in chart 214 also includes a different marker for each face to identify each face and to identify the index of the feature points, as described above.

By setting the tilt angle of each chart surface to a predetermined angle and fixing it together with the imaging device using a jig, the orientation of each surface with respect to the imaging surface is known, and the method of giving a perspective to the chart pattern can be uniquely determined. In this way, the structure including the chart itself and the fixture of the imaging device determines the conversion parameters necessary for calibration, so it can be realized as a calibration tool in which the shape of the entire structure itself is fixed. The calibration fixture may include illumination for optimized position and intensity. A calibration method using the chart 214 and a calibration device for realizing it may be the same as those described above.

FIG. 8 shows a trihedral view of the chart 214 shown in FIG. According to plan view 220a, one side of chart 210 faces upward and the chart pattern is visible. A light 216b is placed behind the other chart face where the back face is visible. This can illuminate an upward chart pattern. In addition, the imaging device 12 and the illumination 216a are arranged so as to face these charts 214 directly. According to the front view 220b, chart patterns are visible on both sides of the chart 214. FIG. By photographing with the imaging device 12 from this direction, an image as shown in FIG. 7B can be photographed.

According to the right side view 220c, the chart surface located in the back faces the right, so the chart pattern can be seen, and the chart surface located in the front faces the left, so the back surface can be seen. In addition, since each plane has an angle around the horizontal axis of 1/2 height, each plane intersects. A light 216c is placed behind the front chart surface where the back surface is visible. This can illuminate a rightward chart pattern.

According to the present embodiment described above, a chart composed of a plurality of planes forming a predetermined angle is used in calibration of the imaging device. As a result, information equivalent to that obtained by photographing one plane from different viewpoints can be obtained by photographing once. Here, the surfaces forming the chart are tilted at a predetermined angle of 0°<θ<180° around an independent axis in addition to opening around a common axis formed by the sides of each surface. As a result, the incident paths of light can be formed in a plurality of directions, so that the chart pattern can be illuminated uniformly, and a uniform image can be obtained in the photographed image. As a result, intrinsic parameters and extrinsic parameters can be derived accurately and efficiently based on the information of all regions of the captured image.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the above-described embodiments are examples, and that various modifications can be made to combinations of each component and each treatment process, and such modifications are also within the scope of the present invention.

For example, in the present embodiment, a chart having two planes is illustrated, but the number of planes of the chart is not limited. In the case of three or more surfaces, the incident path of light is formed between the surfaces by rotating adjacent surfaces not only around a common axis but also independently rotating around an axis or rotating only one of them. When the ambient light is sufficient, a clear image can be obtained in the entire area by forming the incident path without arranging the illumination as described above.

10 calibration device 12 imaging device 14 jig 34 image acquisition unit 36 feature point information acquisition unit 38 calibration unit 40 projective transformation parameter storage unit 42 feature point information storage unit 44 camera parameter storage unit 210 chart 214 chart 216a lighting.

Claims (8)

A chart for calibration of an imaging device,
A chart for calibration, characterized in that a plurality of chart surfaces representing a chart pattern on the surface and joined to each other at one side form a predetermined angle _θ1 about an axis in the direction of the side and are inclined at a predetermined angle θ2 about _an axis different from the axis. 2. The calibration chart according to claim 1, wherein one of the two chart surfaces is inclined around a horizontal axis on the chart surface. 2. A calibration chart according to claim 1, wherein said two chart surfaces are inclined in opposite directions about horizontal axes on each chart surface. 4. The calibration chart according to any one of claims 1 to 3, wherein the chart pattern is obtained by subjecting the predetermined pattern to conversion corresponding to a change in the distance from the imaging surface to the chart surface so that the predetermined pattern is obtained in the captured image. 5. The calibration chart according to any one of claims 1 to 4, wherein the chart pattern includes an image of a different marker for each chart surface. a calibration chart according to any one of claims 1 to 5;
a fixture for fixing an imaging device that captures the calibration chart;
A calibration instrument characterized by fixing the 7. The instrument for calibration according to claim 6, wherein illumination is fixed before and after said chart for calibration, respectively. 8. A calibration instrument according to claim 6 or 7, wherein illumination is fixed behind each of said plurality of chart surfaces.

Images