CN117830435A - Multi-camera system calibration method based on 3D reconstruction - Google Patents

Abstract

The invention provides a multi-camera system calibration method based on three-dimensional reconstruction, which comprises the following implementation steps: constructing a calibration scene of the multi-camera system to be calibrated; acquiring an image pair of a scene image; carrying out three-dimensional reconstruction on the calibration object; performing scale recovery on the three-dimensional sparse point cloud of the calibration object; constructing a descriptor standard library; acquiring an external reference calibration chart and an internal reference calibration chart of the multi-camera system; obtaining calibration results of internal parameters and external parameters of the multi-camera system; according to the method, after the three-dimensional reconstruction is carried out on the calibration object through the images of all scene images, the three-dimensional sparse point cloud with the real scale of the calibration object is obtained, so that the characteristic points in the images and the three-dimensional points of the calibration object can be accurately matched no matter the camera to be calibrated shoots the calibration object from any angle, the calibration accuracy is effectively improved, and meanwhile, the external parameters and the internal parameters of the camera are calibrated through the descriptor standard library, and the calibration efficiency is effectively improved.

Description

Multi-camera system calibration method based on 3D reconstruction

Technical Field

The present invention belongs to the field of computer technology and relates to a multi-camera system calibration method, and specifically to a multi-camera system calibration method based on three-dimensional reconstruction.

Background technique

A multi-camera system refers to a system consisting of multiple cameras that work together to capture images of the same scene from different perspectives or positions. The parameters of a multi-camera system include the intrinsic and extrinsic parameters of the multi-camera. The intrinsic parameters are the parameters that describe the imaging characteristics of the camera itself, including the focal length and the coordinates of the principal point. The extrinsic parameters are the parameters that describe the position and orientation of the camera coordinate system relative to a reference coordinate system such as a calibration object or a world coordinate system.

Accurately calibrating the intrinsic and extrinsic parameters of a multi-camera system is the basis for achieving multi-camera collaborative work. The technical ideas for multi-camera system calibration usually involve four key steps: selecting calibration objects, capturing calibration images, detecting feature points, and estimating camera parameters. In a space-constrained environment, each camera can clearly capture the image of the calibration object, and can match the feature points detected in the image with the actual points of the calibration object in three-dimensional space, which is the difficulty of multi-camera system calibration.

Reprojection error is a key indicator to measure the accuracy of multi-camera calibration. It is defined as the difference between the projected points and the actually observed image points after the three-dimensional points are reprojected back to the image plane using the calibration results including intrinsic and extrinsic parameters during the camera calibration process. It is generally believed in the industry that if the reprojection error is within 0.1 pixels, the calibration result is accurate.

The patent application with the application publication number CN116704045A and the name “Multi-camera system calibration method for monitoring starry sky background simulation system” discloses a multi-camera system calibration method, which determines the three-dimensional representation of the mark points on the calibration plate in the world coordinate system; sets each three-dimensional turntable to the initial zero position; uses the calibration plate to calibrate the high-precision cameras in each group of camera devices, and determines the intrinsic and extrinsic parameters of each high-precision camera; sets the mark points on the rotatable axis frame of the three-dimensional turntable in each group of camera devices; selects two groups of camera devices, and calibrates the three-dimensional turntables of the remaining groups of camera devices; selects another two groups of camera devices, and calibrates the three-dimensional turntables of the first two groups of camera devices; selects a reference coordinate system, determines the position and posture relationship of the camera coordinate systems corresponding to the remaining groups of camera devices in the reference coordinate system, and obtains the extrinsic parameters of the multi-camera system. The invention can quickly and accurately calibrate the multi-camera and turntable used for monitoring the starry sky system. However, the invention requires precise coordination between multiple cameras and a three-dimensional turntable in the process of obtaining the calibration results of the internal and external parameters, resulting in low calibration accuracy and efficiency.

Summary of the invention

The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a multi-camera system calibration method based on three-dimensional reconstruction to solve the technical problems of low calibration accuracy and efficiency in the prior art.

To achieve the above object, the technical solution adopted by the present invention includes the following steps:

(1) Construct a calibration scenario for the multi-camera system to be calibrated:

Construct a calibration scene for a multi-camera system consisting of a calibration object in the shape of a convex polyhedron, a chessboard placed side by side with it, and S cameras to be calibrated, S ≥ 2;

(2) Obtaining image pairs of scene images:

The calibration scene is photographed from N angles by a camera with an internal reference C, and a total of N scene images are obtained, including L complete chessboard images and L' incomplete chessboard images. Feature detection is performed on each scene image to obtain a feature point set of N scene images. The Z feature points and the corresponding Z descriptors contained in the n-th scene image, and the mapping function g(q _nz )=d _nz between the z-th feature point q _nz and the descriptor d _nz , and after matching q _nz with each feature point of the other N-1 scene images, the n-th scene image and the scene images whose matching feature points are greater than β form an image pair, and multiple groups of image pairs corresponding to the n-th scene image are obtained, where N=L+L', N≥20, L≥2;

(3) Perform three-dimensional reconstruction of the calibration object:

The calibration object is reconstructed in three dimensions through the image pairs corresponding to N scene images to obtain the 3D sparse point cloud X, X and feature point set composed of I 3D points of the calibration object. The corresponding relationship F and the camera pose composed of the rotation matrix set r = {r _l |1≤l≤L} and the translation vector set t = {t _l |1≤l≤L} of the calibration object coordinate system relative to the camera coordinate system when shooting L scenes that can fully display the chessboard, where r _l and t _l represent the rotation matrix and translation vector of the calibration object coordinate system relative to the camera coordinate system when shooting the l-th scene image that can fully display the chessboard, respectively, and the i-th 3D point in X is _Xi ;

(4) Scale restoration of the 3D sparse point cloud of the calibration object:

The scale of the 3D sparse point cloud X of the calibration object is restored through the camera's intrinsic parameter C and L scene images that fully reflect the chessboard, and the 3D sparse point cloud X' of the calibration object with the real scale is obtained;

(5) Build a descriptor standard library:

Through X and feature point set The corresponding relationship F, in the feature point set/> Search for a feature point corresponding to each 3D point _Xi , and through the mapping function g( _qnz )= _dnz between _qnz and _dnz , combine the descriptors of I feature points into a descriptor standard library J;

(6) Obtain the external parameter calibration map and the internal parameter calibration map of the multi-camera system:

The S cameras to be calibrated in the multi-camera system shoot the calibration object within the public field of view to obtain S external parameter calibration images B, and shoot the calibration object moved to the field of view of each camera to obtain S internal parameter calibration images E;

(7) Obtain the calibration results of the internal and external parameters of the multi-camera system:

Through the three-dimensional sparse point cloud X' of the calibration object with real scale, the descriptor standard library J and S intrinsic parameter calibration images E, the focal length _ax and _ay along the x-axis and y-axis of the image coordinate system, and the principal point coordinates _x0 and _y0 of each camera are calibrated to obtain the intrinsic parameter matrix _KS of the multi-camera system. And through _KS , X', J and S extrinsic parameter calibration images B, the rotation matrix _RS and translation vector _tS of the calibration object coordinate system relative to the camera coordinate system of each camera are calibrated to obtain the extrinsic parameters of the multi-camera system.

Compared with the prior art, the present invention has the following advantages:

1. The present invention shoots the calibration scene from different angles with a camera determined by the intrinsic parameters, and after three-dimensionally reconstructing the calibration object through image pairs corresponding to all scene images, a three-dimensional sparse point cloud of the calibration object with a real scale is obtained, ensuring that no matter from which angle the camera to be calibrated shoots the calibration object, the feature points in the image and the three-dimensional points of the calibration object can be accurately matched. Finally, the calibration of the intrinsic parameters and external parameters of multiple cameras is realized, avoiding the defect of the prior art that precise coordination between multiple cameras and a three-dimensional turntable is required, and effectively improving the accuracy of calibration.

2. The present invention obtains a descriptor standard library of descriptors including multiple feature points through the feature points corresponding to each three-dimensional point in the feature point set, and the mapping function between each feature point and the corresponding descriptor, and calibrates the external parameters and internal parameters of the camera through it. It can quickly match the feature points in the internal and external parameter calibration images and the three-dimensional points of the calibration object, effectively improving the calibration efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of an implementation of the present invention.

Detailed ways

The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Referring to Figure 1, the present invention comprises the following steps:

Step 1) Construct the calibration scene of the multi-camera system to be calibrated:

Construct a calibration scene for a multi-camera system consisting of a calibration object in the shape of a convex polyhedron, a chessboard placed side by side with it, and S cameras to be calibrated, S ≥ 2;

Since a convex polyhedron has multiple faces in space, cameras at different angles and positions can capture some features of the calibration object. In this embodiment, S=4, a sphere with a diameter of 1 meter is used as the calibration object. Its surface is rich in feature points. A chessboard is placed side by side with it so that feature points on the sphere and the chessboard can be captured simultaneously when shooting the scene. The chessboard is designed with a side length of 30mm for each square. This known size is crucial for restoring the scale of the 3D sparse point cloud in subsequent work.

Step 2) Obtain image pairs of scene images:

The calibration scene is photographed from N angles by a camera with an internal reference C, and a total of N scene images are obtained, including L complete chessboard images and L' incomplete chessboard images. Feature detection is performed on each scene image to obtain a feature point set of N scene images. The Z feature points and the corresponding Z descriptors contained in the n-th scene image, and the mapping function g(q _nz )=d _nz between the z-th feature point q _nz and the descriptor d _nz , and after matching q _nz with each feature point of the other N-1 scene images, the n-th scene image and the scene images whose matching feature points are greater than β form an image pair, and multiple groups of image pairs corresponding to the n-th scene image are obtained, where N=L+L', N≥20, L≥2;

In this embodiment, N = 30, L = 5, β = 100, The camera focal lengths along the x-axis and y-axis of the image coordinate system are 1373.77 and 1373.5 pixels, respectively, and the principal point coordinates are 968.45 and 542.036 pixels, respectively.

When the calibration scene is photographed from N angles by a camera with an intrinsic reference C, the overlap between images taken at adjacent angles is greater than 60%, and the resolution of the camera is 1920*1080 pixels.

Step 3) Perform three-dimensional reconstruction of the calibration object:

The SfM algorithm is used to reconstruct the calibration object in three dimensions through the image pairs corresponding to N scene images, and the 3D sparse point cloud X, X and the feature point set composed of I 3D points of the calibration object are obtained. The corresponding relationship F, F expression is:

(X _i ,q _nz )∈F

in, Represents a set of feature points of a three-dimensional sparse point cloud X and N scene images/> The Cartesian product of corresponds to the set of all possible ordered pairs of the three-dimensional point _Xi and the feature point _qnz in the image; ( _Xi , _qnz )∈F indicates that in the SfM algorithm, the three-dimensional point _Xi has a corresponding two-dimensional feature point _qnk in the nth image;

In the process of SfM algorithm, the camera pose consisting of the rotation matrix set r = {r _l |1≤l≤L} and the translation vector set t = {t _l |1≤l≤L} of the calibration object coordinate system relative to the camera coordinate system is obtained when shooting L images of the scene that can fully display the checkerboard. Among them, r _l and t _l respectively represent the rotation matrix and translation vector of the calibration object coordinate system relative to the camera coordinate system when shooting the scene image that can fully display the checkerboard. These camera poses prepare for the scale recovery of the three-dimensional sparse point cloud of the calibration object in the subsequent steps. Among them, r _l is a 3x3 matrix composed of three orthogonal unit column vectors, each column vector represents the direction of an axis in the camera coordinate system in the calibration object coordinate system; t _l is a 3x1 vector representing the position of the origin of the camera coordinate system relative to the origin of the calibration object coordinate system. In this implementation, I = 47402.

Step 4) Scale the 3D sparse point cloud of the calibration object:

The scale of the 3D sparse point cloud X of the calibration object is restored through the camera's intrinsic parameter C and L scene images that fully reflect the chessboard, and the 3D sparse point cloud X' of the calibration object with the real scale is obtained:

X' _i = scX _i

t _l,l+1 =t _l+1 -r _l,l+1 t _l

r _l,l+1 = r _l+1 r _l ^T

Where sc represents the scale factor, ‖·‖ ₂ represents the binary norm operation, r _l and t _l represent the rotation matrix and translation vector of the calibration object coordinate system relative to the camera coordinate system when the l-th image of the scene fully reflects the checkerboard, r _l,l+1 and t _l,l+1 represent the rotation matrix and translation vector of the camera coordinate system when the l-th image is taken relative to the l+1-th image of the scene fully reflects the checkerboard, t′ _l,l+1 represents the true scale of t _l,l+1 ; t′ _l,l+1 represents the true scale of t _l,l+1 ;

The steps for obtaining the real scale t′ l _,l+1 of t _l,l+1 are as follows: extract the corner points of the chessboard in each of L scene images containing a complete chessboard, and use the EPnP algorithm to estimate the camera's pose through the camera's internal parameter C, and then use the Gauss-Newton optimization method to optimize the pose, and obtain the rotation matrix and translation vector with real scale of the calibration object coordinate system to the camera coordinate system when each calibration scene is photographed, and finally calculate the translation vector t′ _l,l+1 with real scale of the camera coordinate system when the lth shooting is relative to the l+1th shooting of the scene image that fully reflects the chessboard through the rotation matrix and translation vector. In this implementation, sc=0.53754128.

Step 5) Build the descriptor standard library:

Through X and feature point set The corresponding relationship F, in the feature point set/> A feature point corresponding to each 3D point _Xi is searched in, and the descriptors of the I feature points are combined into a descriptor standard library J through the mapping function g( _qnz )= _dnz between _qnz and _dnz . The specific steps are as follows:

In the feature point set Search for the feature point {q _nz |(X _i ,q _nz )∈F} corresponding to each 3D point _Xi , and randomly select a feature point in {q _nz |(X _i ,q _nz )∈F}; for example, select the feature point q _nz , obtain the descriptor d _nk corresponding to the feature point through the mapping function g(q _nz )＝d _nz , and add the selected descriptor d _nk to the set J, that is, J＝J∪{d _nk };

Step 6) Obtain the external parameter calibration map and the internal parameter calibration map of the multi-camera system:

The mobile calibration object is located within the common field of view of S cameras, and each camera of the multi-camera system is used to simultaneously shoot the calibration object within the common field of view to obtain S external parameter calibration images B, and the mobile calibration object is sequentially placed within the field of view of each camera, so that the calibration object covers the field of view of each camera as much as possible, and the calibration object is shot to obtain S internal parameter calibration images E. In this embodiment, S=4.

Step 7) Obtain the calibration results of the multi-camera system internal and external parameters:

Through the calibration object with real-scale three-dimensional sparse point cloud X', descriptor standard library J and S internal parameter calibration images E, the focal length _ax and _ay of each camera along the x-axis and y-axis of the image coordinate system, and the principal point coordinates _x0 and _y0 are calibrated to obtain the intrinsic parameter matrix _Ks of the multi-camera system. And through _Ks , X', J and S external parameter calibration images B, the rotation matrix _RS and translation vector _tS of the calibration object coordinate system relative to the camera coordinate system of each camera are calibrated to obtain the external parameters of the multi-camera system. The specific steps are as follows:

(7a) The SIFT algorithm is used to extract features from each image of the intrinsic calibration image E, and H feature points and their corresponding H descriptors of the s-th image are obtained. The h-th descriptor v _sh is matched with the descriptor standard library J by the ANN algorithm. Then, the three-dimensional point X' _i corresponding to the closest descriptor in the matching result is matched with the h-th feature point u _sh to form a matching point pair, and W matching point pairs of the s-th image are obtained, where W ≤ H.

(7b) The camera projection matrix P _s of the multi-camera system is calculated through W matching point pairs, and the matrix _MS composed of the first three columns of P _s is subjected to QR decomposition. The upper triangular matrix K _s obtained by the decomposition is the intrinsic parameter matrix of the multi-camera system;

(7c) The SIFT algorithm is used to extract features from each image of the external calibration image B to obtain the feature points of each image and its corresponding descriptor. The ANN algorithm is used to match each descriptor with the descriptor standard library J. Then, the three-dimensional point _X'i corresponding to the closest descriptor in the matching result is matched with the feature point to form a matching point pair. The EPnP algorithm is used to estimate the pose of the multi-camera system through the internal parameter matrix _Ks and the matching point pair. The Gauss-Newton method is used to optimize the pose of the multi-camera system to obtain the rotation matrix _RS and translation vector _tS of the calibration object coordinate system relative to the camera coordinate system of each camera in the multi-camera system.

The experimental data in Table 1 is the reprojection error for verifying the accuracy of the internal and external parameter calibration results of this embodiment. It can be seen from Table 1 that the minimum and maximum values of the reprojection errors of the calibration results of the four cameras in this embodiment are 0.0237078 and 0.0541603, respectively, which are significantly lower than the reprojection error of 0.1 pixel believed by the industry, indicating that the calibration results of this embodiment are accurate.

Table 1.

Camera No. Reprojection error (in pixels) Camera 1 0.050013 Camera 2 0.0237078 Camera 3 0.0541603 Camera 4 0.0480274

Claims (9)

1. A multi-camera system calibration method based on three-dimensional reconstruction, characterized in that it includes the following steps: (1) Construct a calibration scenario for the multi-camera system to be calibrated: Construct a calibration scene for a multi-camera system consisting of a calibration object in the shape of a convex polyhedron, a chessboard placed side by side with it, and S cameras to be calibrated, S ≥ 2; (2) Obtaining image pairs of scene images: The calibration scene is photographed from N angles by a camera with an internal reference C, and a total of N scene images are obtained, including L complete chessboard images and L' incomplete chessboard images. Feature detection is performed on each scene image to obtain a feature point set of N scene images. The Z feature points and the corresponding Z descriptors contained in the n-th scene image, and the mapping function g(q _nz )=d _nz between the z-th feature point q _nz and the descriptor d _nz , and after matching q _nz with each feature point of the other N-1 scene images, the n-th scene image and the scene images whose matching feature points are greater than β form an image pair, and multiple groups of image pairs corresponding to the n-th scene image are obtained, where N=L+L', N≥20, L≥2; (3) Perform three-dimensional reconstruction of the calibration object: The calibration object is reconstructed in three dimensions through the image pairs corresponding to N scene images to obtain the 3D sparse point cloud X, X and feature point set composed of I 3D points of the calibration object. The corresponding relationship F and the camera pose composed of the rotation matrix set r = {r _l |1≤l≤L} and the translation vector set t = {t _l |1≤l≤L} of the calibration object coordinate system relative to the camera coordinate system when shooting L scenes that can fully display the chessboard, where r _l and t _l represent the rotation matrix and translation vector of the calibration object coordinate system relative to the camera coordinate system when shooting the l-th scene image that can fully display the chessboard, respectively, and the i-th 3D point in X is _Xi ; (4) Scale restoration of the 3D sparse point cloud of the calibration object: The scale of the 3D sparse point cloud X of the calibration object is restored through the camera's intrinsic parameter C and L scene images that fully reflect the chessboard, and the 3D sparse point cloud X' of the calibration object with the real scale is obtained; (5) Build a descriptor standard library: Through X and feature point set The corresponding relationship F, in the feature point set/> Search for a feature point corresponding to each 3D point _Xi , and through the mapping function g( _qnz )= _dnz between _qnz and _dnz , combine the descriptors of I feature points into a descriptor standard library J; (6) Obtain the external parameter calibration map and the internal parameter calibration map of the multi-camera system: The S cameras to be calibrated in the multi-camera system shoot the calibration object within the public field of view to obtain S external parameter calibration images B, and shoot the calibration object moved to the field of view of each camera to obtain S internal parameter calibration images E; (7) Obtain the calibration results of the internal and external parameters of the multi-camera system: Through the three-dimensional sparse point cloud X' of the calibration object with real scale, the descriptor standard library J and S intrinsic parameter calibration images E, the focal length _ax and _ay along the x-axis and y-axis of the image coordinate system, and the principal point coordinates _x0 and _y0 of each camera are calibrated to obtain the intrinsic parameter matrix _KS of the multi-camera system. And through _KS , X', J and S extrinsic parameter calibration images B, the rotation matrix _RS and translation vector _tS of the calibration object coordinate system relative to the camera coordinate system of each camera are calibrated to obtain the extrinsic parameters of the multi-camera system. 2. The method according to claim 1, characterized in that the camera intrinsic parameter C in step (2) is a matrix composed of the camera focal lengths _ax and _ay along the x-axis and y-axis of the image coordinate system expressed in pixels, and the principal point coordinates _x0 and _y0 , which can be expressed as follows: 3. The method according to claim 1 is characterized in that the feature detection of each scene image in step (2) adopts SIFT algorithm. 4. The method according to claim 1 is characterized in that the three-dimensional reconstruction of the calibration object described in step (3) adopts the SfM algorithm. 5. The method according to claim 1, characterized in that the X in step (3) is related to the feature point set The corresponding relationship F is expressed as: (X _i ,q _nz )∈F in, Represents a set of feature points of a three-dimensional sparse point cloud X and N scene images/> The Cartesian product of corresponds to the set of all possible ordered pairs of three-dimensional points _Xi and feature points _qnz in the image. 6. The method according to claim 1, characterized in that, when the lth shooting described in step (3) can completely display the checkerboard scene image, the rotation matrix r _l and translation vector t _l of the calibration object coordinate system relative to the camera coordinate system, wherein r _l is a 3x3 matrix composed of three orthogonal unit column vectors, each column vector represents the direction of an axis in the camera coordinate system in the calibration object coordinate system; t _l is a 3x1 vector representing the position of the origin of the camera coordinate system relative to the origin of the calibration object coordinate system. 7. The method according to claim 1, characterized in that the calibration object in step (4) has a three-dimensional sparse point cloud X' of real scale, and the calculation formula is: X' _i = scX _i t _l,l+1 =t _l+1 -r _l,l+1 t _l r _l,l+1 = r _l+1 r _l ^T Where sc represents the scale factor, ‖·‖ ₂ represents the binary norm operation, r _l and t _l represent the rotation matrix and translation vector of the calibration object coordinate system relative to the camera coordinate system when the l-th shot fully reflects the scene image of the checkerboard, r _l,l+1 and t _l,l+1 represent the rotation matrix and translation vector of the camera coordinate system when the l-th shot is relative to the l+1-th shot fully reflects the scene image of the checkerboard, and t′ _l,l+1 represents the actual scale of t _l,l+1 . 8. The method according to claim 1, characterized in that the step (5) in the feature point set Search for a feature point corresponding to each 3D point _Xi in the search method: In the feature point set Search for the feature point {q _nz |(X _i ,q _nz )∈F} corresponding to each 3D point _Xi , and randomly select a feature point in {q _nz |(X _i ,q _nz )∈F}. 9. The method according to claim 1, characterized in that the step (7) of calibrating the focal length _ax and _ay of each camera along the x-axis and y-axis of the image coordinate system and the principal point coordinates _x0 and _y0 is implemented by: (7a) The SIFT algorithm is used to extract features from each image in the intrinsic calibration image E, and H feature points and their corresponding H descriptors of the s-th image are obtained. The ANN algorithm is used to match the h-th descriptor v _sh with the descriptor standard library J. Then, the three-dimensional point X' _i corresponding to the closest descriptor in the matching result is matched with the h-th feature point u _sh to form a matching point pair, and W matching point pairs of the s-th image are obtained, where W ≤ H. (7b) The camera projection matrix _PS of the multi-camera system is calculated through W matching point pairs, and the matrix _MS composed of the first three columns of _PS is subjected to QR decomposition. The upper triangular matrix _Ks obtained by the decomposition is the intrinsic parameter matrix of the multi-camera system.