CN112541950A - Method and device for calibrating external parameter of depth camera - Google Patents

Abstract

The present application discloses a method for calibrating external parameters of a depth camera, which is characterized in that the method includes: acquiring depth image data, where the depth image data includes pixel coordinates and depth values; based on the acquired depth image data, the depth The pixel points in the image are converted into spatial three-dimensional points in the camera coordinate system; based on the three-dimensional points, the fitting plane of the three-dimensional points is obtained; according to the current pose relationship between the calibration plane parallel or overlapping with the fitting plane and the camera coordinate system , to obtain the parameters between the depth camera and the calibration plane; wherein, the calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located. In the calibration process of this application, there is no need to use any external tools, and only one image can be collected to give the calibration result, and the calibration can be carried out with the help of the natural horizontal and vertical ground and wall, which provides robustness for applications based on depth image data. Excellent and real-time application foundation.

Description

A method and device for calibrating external parameters of a depth camera

technical field

The present invention relates to the field of machine vision, in particular, to a method for calibrating external parameters of a depth camera.

Background technique

In the process of image measurement and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a space object and its corresponding point in the image, a geometric model of camera imaging must be established, and these geometric model parameters are camera parameters. Camera calibration is the process of calculating camera parameters through a certain method.

In the process of image pickup, the pose of the camera is not fixed, which makes the camera coordinate system not a stable coordinate system, and the origin of the camera coordinate system and the direction of each coordinate axis will change with the movement of the camera. , which requires the introduction of a stable and invariable coordinate system: the world coordinate system, which is an absolute coordinate system. The transformation from the camera coordinate system to the world coordinate system requires rotation transformation and translation transformation, and these rotation transformation and translation transformation become the external parameters of the camera.

A depth camera is a camera that can directly obtain depth image data. Any pixel data in the obtained depth image data is a distance value (depth value) that represents the distance value (depth value) of a spatial point based on the pixel point coordinates and the pixel point gray value. The data can be expressed as p(u, v, d), where u and v are the coordinates of the pixel point in the image coordinate system, and d is the depth value of the spatial three-dimensional point corresponding to the pixel point.

According to the composition principle of cameras, depth cameras include passive binocular cameras, active binocular cameras, time-of-flight (TOF) cameras, monocular structured light, etc. At present, the external parameter calibration of the depth camera usually calibrates the internal external parameters of the depth camera. For example, the depth camera composed of binocular cameras calibrates the relative rotation and translation between the two cameras. This calibration method is not suitable for Calibrate the external parameter relationship between the depth camera and the external coordinate system.

SUMMARY OF THE INVENTION

The present invention provides a method for calibrating external parameters of a depth camera, so as to calibrate the transformation relationship between the camera coordinate system and the calibration plane based on depth image data.

The present invention provides a method for calibrating external parameters of a depth camera, the method comprising:

Obtaining depth image data, the depth image data includes pixel coordinates and depth values;

Based on the acquired depth image data, transform the pixels in the depth image into three-dimensional spatial points in the camera coordinate system;

Based on the three-dimensional point, obtain the fitting plane of the three-dimensional point;

According to the current pose relationship between the calibration plane parallel or coincident with the fitting plane and the camera coordinate system, the parameters between the depth camera and the calibration plane are obtained;

Wherein, the calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located.

Preferably, the depth image data is at least one depth image, and the conversion of pixels in the depth image into three-dimensional spatial points in the camera coordinate system includes,

The three-dimensional point coordinates corresponding to the pixel points are obtained according to the mapping geometric model of the spatial three-dimensional point in the camera coordinate system and the two-dimensional image in the image coordinate system.

Preferably, obtaining the three-dimensional point coordinates corresponding to the pixel points according to the mapping geometric model of the spatial three-dimensional point under the camera coordinate system and the two-dimensional image under the image coordinate system, including:

For any pixel in the depth image,

Use the depth value of the pixel point as the z coordinate value of the three-dimensional point;

Obtain the first difference between the x-coordinate of the pixel point and the offset in the x-direction, multiply the ratio of the z-coordinate value to the focal length in the x-direction of the camera's internal parameters and the first difference, and the obtained result is used as the x-coordinate of the three-dimensional point value; the x-direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the x-direction;

Obtain the second difference between the y-coordinate of the pixel and the offset in the y-direction, multiply the ratio of the z-coordinate value to the y-direction focal length in the camera's internal parameters and the second difference, and use the result as the y-coordinate of the three-dimensional point value; the y-direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the y-direction;

Wherein, the offset of the origin of the camera coordinate system relative to the x direction in the image coordinate system, and the offset of the origin of the camera coordinate system relative to the y direction in the image coordinate system are obtained according to the camera internal parameters;

Convert all pixels in the depth image to 3D point coordinates to obtain a 3D point set.

Preferably, the method further includes, according to a screening strategy, screening the three-dimensional points in the three-dimensional point set to obtain a screened three-dimensional point set; wherein, the screening strategy includes any of the following conditions or any combination thereof:

(1) According to the orientation of the camera, remove 3D points in a certain range above the depth image;

(2) When there is an initial estimated value of the external parameter, the three-dimensional point in the camera coordinate system is transformed into the world coordinate system through the camera external parameter, and the three-dimensional point whose height in the height direction is greater than the preset height threshold is removed;

(3) For the depth camera of binocular stereo vision, remove the three-dimensional points whose depth value is greater than the preset depth threshold;

(4) For the TOF depth camera of the time of flight, according to the distance difference between the pixel point and its adjacent pixel points, remove the three-dimensional points whose distance difference is greater than the preset distance threshold.

Preferably, obtaining the fitting plane of the three-dimensional point based on the three-dimensional point includes,

According to the screened 3D point set, a random sampling consistent RANSAC algorithm is used to obtain the fitting plane equation of the 3D points, wherein the number of 3D points is greater than or equal to 3.

Preferably, the random sampling consistent RANSAC algorithm is used to obtain the fitting plane equation of the three-dimensional point, including,

Based on the filtered 3D point set, random selection is performed to obtain a current subset composed of the randomly selected 3D points, wherein the number of 3D points in the subset includes at least 3;

Based on the 3D points in the current subset, obtain a fitting plane estimate for the subset;

According to the obtained fitting plane estimation, obtain the degree of conformity of all 3D points in the screened 3D point set relative to the fitting plane estimation;

If the degree of conformity is not enough, returning to the step of randomly selecting based on the filtered three-dimensional point set;

If the degree of conformity is reached, use the estimated interior point of the fitting plane with the best degree of conformity to solve the fitting plane equation;

Wherein, the inliers include three-dimensional points whose distances from the three-dimensional points in the filtered three-dimensional point set to the fitting plane with the best degree of conformity are less than a preset distance threshold.

Preferably, obtaining the fitting plane estimation of the subset based on the three-dimensional points in the current subset, including,

If the number of 3D points in the current subset is equal to 3, substitute the coordinate values of the 3D points into the fitting plane equation, solve the unknowns in the fitting plane equation, and obtain the fitting plane estimation of the current subset;

If the number of 3D points in the current subset is greater than 3, the coordinate values of the 3D points are substituted into the fitting plane equation, and the unknowns in the fitting plane equation are solved by the least squares method to obtain the fitting plane estimate of the current subset.

Preferably, according to the obtained fitting plane estimation, the degree of conformity of all 3D points in the screened 3D point set relative to the fitting plane estimation is obtained, including:

Calculate the distance value estimated from each 3D point in the filtered 3D point set to the fitting plane,

Take the three-dimensional point whose calculated distance value is less than the set distance threshold as the interior point,

count the number of inliers,

Calculate the ratio of the number of inliers to the filtered 3D point cloud to obtain the inlier rate;

determining the degree of compliance according to the interior point rate;

The fitting plane equation is solved by using the estimated interior points of the fitting plane with the best degree of conformity, including, using the fitting plane estimate with the highest interior point rate as the best fitting plane estimate, and according to the best fitting plane estimate For the interior point, the unknowns in the fitted plane equation are re-solved by the method of least squares, and the fitted plane equation is obtained.

Preferably, the determining the degree of conformity according to the inlier rate includes,

The degree of conformity is determined according to the number of iterations, wherein the number of iterations satisfies:

where m is the number of 3D points in the subset, η is the set confidence level, ε is the interior point rate, is the proportion of interior points under the worst condition, or is set to the proportion under the worst condition in the initial state, and With the number of iterations, update to the current maximum inlier rate.

Preferably, the determining the degree of conformity according to the inlier rate includes determining the degree of conformity according to whether the probability that the v subsets are all inliers satisfies a set confidence level, wherein all the v subsets are inliers. The probability is:

where λ is the expectation of the number of times the subsets are all interior points in the current iteration.

Preferably, the parameters between the depth camera and the calibration plane are obtained according to the current pose relationship between the calibration plane that is parallel or coincident with the fitting plane and the camera coordinate system, including,

According to the fitted plane equation, the fitted plane normal vector is obtained,

According to the normal vector of the fitting plane, obtain the equation of the calibration plane that is parallel or coincident with the fitting plane;

Calculate the distance from the origin of the camera coordinate system to the calibration plane, and obtain the distance transformation of the camera coordinate system relative to the calibration plane;

Calculate the rotation of the camera coordinate system relative to the calibration plane, to the rotation transformation of the camera coordinate system relative to the calibration plane,

The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

Preferably, the depth image data includes image data of the bearing surface of the mobile robot body,

According to the fitted plane normal vector, the equation of the calibration plane associated with the fitted plane is obtained, including;

According to the normal vector of the fitting plane, the equation of the calibration plane parallel to the fitting plane and having a set distance is obtained; wherein, the calibration plane is the bearing ground of the mobile robot body in the world coordinate system;

The calculating the distance from the origin of the camera coordinate system to the calibration plane, and obtaining the distance transformation of the camera coordinate system relative to the calibration plane, including;

According to the equation parameters of the calibration plane, calculate the distance from the origin of the camera coordinate system to the calibration plane, and obtain the height transformation of the camera coordinate system relative to the calibration plane;

The calculation of the rotation of the camera coordinate system relative to the front view of the calibration plane, to the rotation transformation of the camera coordinate system relative to the calibration plane, includes,

According to the equation parameters of the calibration plane, the rotation of the y-axis of the camera coordinate system relative to the normal vector of the fitting plane is calculated, and the rotation transformation of the camera coordinate system relative to the calibration plane is obtained.

Preferably, the depth image data includes elevation image data perpendicular to the bearing surface of the mobile robot body,

According to the fitted plane normal vector, the equation of the calibration plane associated with the fitted plane is obtained, including;

According to the normal vector of the fitting plane, the equation of the calibration plane that is parallel to the fitting plane and has a set distance is obtained; wherein, the calibration plane is perpendicular to the bearing ground of the mobile robot body in the world coordinate system;

The calculating the distance from the origin of the camera coordinate system to the calibration plane, and obtaining the distance transformation of the camera coordinate system relative to the calibration plane, including;

According to the equation parameters of the calibration plane, calculate the distance from the origin of the camera coordinate system to the calibration plane, and obtain the distance transformation from the camera coordinate system to the calibration plane;

The calculation of the rotation of the camera coordinate system relative to the front view of the calibration plane, to the rotation transformation of the camera coordinate system relative to the calibration plane, includes,

According to the equation parameters of the calibration plane, the rotation of the z-axis of the camera coordinate system relative to the normal vector of the calibration plane is calculated, and the rotation transformation of the camera coordinate system relative to the calibration plane is obtained.

The present invention provides an electronic device for calibrating external parameters of a depth camera, the electronic device includes a memory and a processor, wherein,

The memory is used to store computer programs;

The processor is configured to execute the program stored in the memory to implement any of the above methods for calibrating external parameters of the depth camera.

The present invention also provides a computer-readable storage medium, where a computer program is stored in the storage medium, and the computer program is executed by a processor to perform any of the above-mentioned methods for calibrating external parameters of a depth camera.

The present application provides a method for calibrating external parameters of a depth camera. Based on the spatial three-dimensional points in the camera coordinate system corresponding to the depth image data, the fitting plane of the calibration plane is obtained, so as to obtain the fitting plane of the calibration plane according to the current relative relationship between the camera coordinate system and the fitting plane. Pose, obtain the distance transformation and rotation change between the camera coordinate system and the calibration plane, and obtain the external parameters of the camera relative to the calibration plane. In the calibration process of this application, there is no need to use any external tools in advance, and only one image can be collected to give the calibration result. In this way, when an image is collected in real time, the natural horizontal and vertical ground or wall can be used for real-time calibration, which is simple and easy. It has strong real-time performance and strong applicability, and provides an application foundation with good robustness and strong real-time performance for applications based on depth image data.

Description of drawings

FIG. 1 is a schematic diagram of the principle of depth camera extrinsic parameter calibration based on depth image data.

FIG. 2 is a schematic flowchart of an embodiment of the present application for calibrating external parameters of a depth camera.

FIG. 3 is a schematic diagram of calibration by fitting a plane.

FIG. 4 is a schematic diagram of performing calibration with a second fitting plane (wall surface) perpendicular to the fitting plane.

FIG. 5 is a schematic diagram of data association of a calibration method according to an embodiment of the present application.

FIG. 6 is a schematic diagram of a calibration device according to an embodiment of the present application.

Detailed ways

In order to make the objectives, technical means and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of the principle of depth camera extrinsic parameter calibration based on depth image data. The depth camera installed on the mobile robot body acquires depth image data. A representation of the depth image is shown in the figure. The depth image corresponding to the physical space ground in the figure (indicated by the thick solid line in the figure) is the dotted frame area in the image. Based on the depth image data of the space ground, a fitting plane Ax+By+Cz+D=0 is solved, and the fitted plane can be regarded as representing the plane where the space ground is located; since the world coordinate system is usually based on the ground , the plane where the xy axis is located is parallel to the ground plane, the z axis is perpendicular to the plane where the xy axis is located, and the translation trajectory (x, y coordinate changes) of the camera belongs to the same plane, so the rotation and height (distance) of the calibration camera relative to the ground are equivalent to In order to solve the rotation and height (distance) of the camera relative to the fitting plane, the rotation transformation and z-axis transformation of the camera relative to the world coordinate system are obtained, thereby obtaining the imaging model between the ground depth image and the physical space ground plane (horizontal plane). parameters; broadly speaking, the imaging model parameters between the depth image and any desired plane in physical space parallel or perpendicular to the bearing surface where the mobile robot body is located, that is, the imaging model parameters between the depth image and the calibration plane, which reflect An external parameter relationship between the depth camera and the external coordinate system can be regarded as an external parameter of the depth camera, which is referred to as the depth camera external parameter in this application.

Based on the above calibration principle, the depth camera external parameter calibration of the present application is based on the depth image collected by the depth camera, and is converted into a three-dimensional point in the camera coordinate system, and a fitting plane associated with the calibration plane is fitted by using the three-dimensional point. The extrinsic parameters of the depth camera are obtained by fitting the current pose relationship between the plane and the depth camera.

Referring to FIG. 2 , FIG. 2 is a schematic flowchart of an embodiment of the present application for calibrating external parameters of a depth camera.

Step 201: Acquire any depth image data output by the depth camera, and project the depth image into a three-dimensional point cloud through the camera's internal parameters, that is, project the depth image into a set of three-dimensional points.

In view of the calibration principle, a fitting plane associated with the calibration plane can be fitted by using the three-dimensional points of the depth image, and the external parameters of the depth camera can be obtained by fitting the current pose relationship between the plane and the depth camera. In this way, At least one depth image output by the depth camera can be used. For any pixel p(u, v, d) on the depth image, u and v are the coordinates of the pixel in the image coordinate system, and d is the gray value of the pixel point, which is also the depth value of the actual three-dimensional point:

According to the mapping geometric model between the three-dimensional coordinate point and the two-dimensional image plane (unit is pixel), such as the pinhole model, the pixel point in the depth image is converted into the three-dimensional point pc(x, y, z) in the camera coordinate system. Relationship:

Use the depth value of the pixel point as the z coordinate value of the three-dimensional point; obtain the first difference between the x coordinate of the pixel point and the offset in the x direction, where the offset in the x direction is the relative image of the origin of the camera coordinate system (camera optical center) For the offset of the coordinate system in the x-direction, multiply the ratio of the z-coordinate value to the focal length in the x-direction of the camera's internal reference by the first difference, and the obtained result is used as the x-coordinate value of the three-dimensional point;

Obtain the second difference between the y coordinate of the pixel and the offset in the y direction, where the offset in the y direction is the offset of the origin of the camera coordinate system relative to the image coordinate system in the y direction, and the z coordinate value and the y direction in the camera's internal parameters The ratio of the focal length is multiplied by the second difference, and the obtained result is taken as the y coordinate value of the three-dimensional point,

Wherein, the offset of the origin of the camera coordinate system relative to the x direction in the image coordinate system, and the offset of the origin of the camera coordinate system relative to the y direction in the image coordinate system are obtained according to the camera internal parameters;

Mathematically expressed as:

Among them, f _x and f _y are the internal parameters of the camera, respectively, and c _x and _cy are the offsets of the origin of the camera coordinate system in the camera coordinate system relative to the image coordinate system. These parameters correspond to the data in the camera internal parameter matrix K:

According to the above method, each pixel in the depth image can be transformed into a three-dimensional point in the camera coordinate system, and the obtained three-dimensional point set is also called a point cloud.

Step 202: Screen the three-dimensional point cloud, and remove the three-dimensional points with larger errors, so as to improve the accuracy and success rate of obtaining the fitting plane.

Taking the ground as an example, in this step, the three-dimensional point cloud calculated in step 201 is screened for the purpose of initially removing three-dimensional points that are obviously not on the ground and three-dimensional points with relatively large errors.

The point cloud screening strategy can add conditions and standards for point cloud screening according to different usage scenarios. Several conditions for point cloud screening are provided here. Each screening condition has its own application. In actual use, it can be one or any combination of the following screening conditions:

(1) Remove a certain range of 3D points above the depth image according to the camera orientation.

For example, when the camera is close to the head-up view, the 3D points in the upper half of the depth image will not appear on the ground, so 1/2 of the image can be used as the threshold to remove the 3D points converted from the pixels in the upper half of the depth image.

(2) When there is an initial estimated value of the external parameter, the three-dimensional point in the camera coordinate system can be transferred to the world coordinate system through the camera external parameter, and the points whose height in the height direction is greater than the preset height threshold can be removed.

(3) If the principle of the depth camera is to use binocular stereo vision, then due to the inherent characteristics of the binocular measurement principle, the depth measurement accuracy will decrease as the distance increases. That is to say, the depth accuracy of the three-dimensional points in the distance deteriorates, so a certain depth-direction distance threshold (depth threshold) can be set, and points exceeding the depth threshold can be removed.

(4) If the principle of the depth camera is to use the TOF method, then due to the characteristics of the TOF sensor itself, the camera is more susceptible to the influence of multi-path reflection, and there may be more flying spots in the depth image. The distance difference between pixel points, and remove the points whose distance difference is greater than the preset distance threshold.

Step 203, obtaining a fitting plane based on the screened 3D point cloud;

In this step, in one of the embodiments, a random sample consensus (RANSAC, Random Sample Consensus) algorithm is used to obtain the fitting plane. Specifically include:

Step 2031, based on the screened three-dimensional point cloud, perform random selection, and obtain a subset formed by randomly selected three-dimensional points, wherein the number of three-dimensional points is equal to the unknown number for determining the fitting plane, because the fitting plane equation can express It is: Ax+By+Cz+D=0, which can be transformed into ax+by+cz=1,

Therefore, it includes three unknowns a, b, c, so the number of three-dimensional points in the subset is at least three.

Step 2032, based on the three-dimensional points in the subset, generate an estimate of the fitting plane of the subset: use the three-dimensional points in the selected subset to substitute the plane equation, solve the unknowns in the fitting plane estimation equation, and obtain the equation of the fitting plane. estimate;

If the number m of 3D points in the selected subset is greater than the number of unknowns, the overdetermined equation system needs to be solved, and regression analysis can be used to solve the unknowns in the fitting plane estimation equation, for example, the linear least squares method is used, as follows:

Deform the fitting plane equation as: ax+by+cz=1, then there are:

非奇异性，则未知数a、b、c有唯一解，其中，x_m、y_m、z_m为三维点的坐标。When the matrix

Non-singularity, the unknowns a, b, and c have unique solutions, where x _m , y _m , and z _m are the coordinates of three-dimensional points.

Step 2033, based on the fitting plane estimation, obtain the degree of conformity of all the screened 3D points relative to the fitting plane estimation;

One of the embodiments is to calculate the interior point rate estimated by the fitted plane, and determine the degree of conformity according to the interior point rate, specifically:

Calculate the estimated distance from each point in the filtered 3D point cloud to the fitted plane,

Take the point where the calculated distance value is less than the set distance threshold as the interior point,

count the number of inliers,

Calculate the proportion of the number of interior points to the filtered 3D point cloud, and obtain the interior point rate.

Step 2034, determine whether the degree of conformity is satisfied, if so, execute step 2035, otherwise, return to step 2031, so as to randomly select a subset to perform fitting plane estimation, thereby performing an estimation-confirmation cycle;

In this step, in one of the embodiments, it is judged whether the inlier rate satisfies the preset condition,

In the second embodiment, under the condition of the confidence η, in the iterative loop process, at least one random selection is made, so that the selected m points are all interior points, which is beneficial to at least one time in the loop process. Get the best value for the fitted plane estimate. Therefore, the number of iterations i should satisfy the following conditions:

Among them, m is the size of the subset, that is, the number of three-dimensional points in the subset; the confidence level is generally set in the range of 0.95 to 0.99. ε is the interior point rate. In general, ε is usually unknown, so the proportion of interior points under the worst condition can be taken, or the proportion under the worst condition can be set in the initial state, and then with the number of iterations, Update to the current maximum inlier rate.

The third embodiment is to judge whether the probability that the subsets are all interior points meets the requirement of confidence. Specifically, the selected subset is regarded as two results of "all interior points" or "not all interior points". The binomial distribution of , and the probability of the former is p=1/ε ^m . For the case where p is small enough, it can be regarded as a Poisson distribution. Therefore, in the i cycles, the probability that there are v "subsets are all interior points" can be expressed as:

Among them, λ represents the expectation of the number of times that "the subsets are all interior points" in the i rounds.

For example, it is hoped that the probability that the selected subset "no one is all interior points" in this iterative loop is less than a certain confidence level, namely: p(0, λ)=e - ^λ <1-η, with confidence level Taking 95% as an example, λ is approximately equal to 3, which means that under 95% confidence, in i cycles, an average of 3 "good" subsets can be selected.

Step 2035, using the estimated interior point of the best fitting plane to solve the fitting plane equation;

Specifically, based on the estimated interior points of the fitting plane with the highest interior point rate, the fitting plane estimation is re-solved by the least squares method to obtain the final fitting plane equation.

Step 204 , based on the fitting plane parameters, calculate the distance from the origin of the camera coordinate system to the fitting plane and the rotation of the camera.

表示向量的叉乘，该拟合平面为世界坐标系下移动机器人本体的承载地面。相机坐标系原点到拟合平面的距离即为相机的高度变换，相机坐标系相对拟合平面正视的旋转对应为相机坐标系的Y轴相对拟合平面法向量的旋转；故，根据拟合平面方程Ax+By+Cz+D＝0，可得：Referring to Fig. 3, Fig. 3 is a schematic diagram of performing calibration by using the fitting plane as the calibration plane, the symbol in the figure

It represents the cross product of the vectors, and the fitting plane is the bearing ground of the mobile robot body in the world coordinate system. The distance from the origin of the camera coordinate system to the fitting plane is the height transformation of the camera, and the rotation of the camera coordinate system relative to the front view of the fitting plane corresponds to the rotation of the Y-axis of the camera coordinate system relative to the normal vector of the fitting plane; therefore, according to the fitting plane Equation Ax+By+Cz+D=0, we can get:

The normal vector n=(A, B, C) of the fitted plane (ground);

The distance from the camera coordinate system to the fitted plane

Camera coordinate system rotation transformation

Among them, the symbol × represents the cross product of vectors.

The above-mentioned embodiment can be extended to, according to the normal vector of the fitting plane, the equation of any calibration plane that is parallel to the fitting plane and has a set distance from the fitting plane can be obtained. The distance transformation and rotation transformation of the calibration plane are used to obtain the parameters between the depth camera and the calibration plane.

If the depth image is related to an elevation (eg, a wall) perpendicular to the bearing surface of the mobile robot body, for example, a depth image of a wall is obtained, the elevation can be used for calibration. Referring to FIG. 4, FIG. 4 is a schematic diagram of calibration with the fitting plane as the elevation. In the figure, the normal vector n is opposite to the front view direction of the camera coordinate system relative to the calibration plane, so it is -n. The distance from the origin of the camera coordinate system to the wall is the distance transformation of the camera, and the rotation of the camera coordinate system to the front view of the first calibration plane corresponds to the rotation of the z-axis of the camera coordinate system relative to the normal vector of the wall; therefore, according to the equation of the calibration plane A'x+B'y+C'z+D'=0, we can get:

Calibration plane normal vector -n = (A', B', C')

The distance from the camera coordinate system to the calibration plane

Camera coordinate system rotation transformation

Among them, the coincidence × represents the cross product of the vector.

The first calibration plane equation can be obtained through steps 201-203 based on the depth image.

The above embodiment can be extended to, according to the normal vector of the fitting plane, can obtain the equation of any calibration plane that is parallel to the fitting plane and has a set distance from the fitting plane. The distance transformation and rotation transformation of the plane, so as to obtain the parameters between the depth camera and the calibration plane.

The depth camera extrinsic parameter calibration method proposed in this application, according to different requirements, for example, when the obtained depth image includes the image of the mobile robot body bearing surface, the ground plane can be used for calibration, when the obtained depth image includes the mobile robot. When the body carries the image data of the vertical wall, the wall can be used for calibration without any external tools; there is no requirement for the principle of the depth camera itself, and it can be applied to the depth camera based on binocular, structured light or TOF principle External parameter calibration; the calibration method has simple operation, high calibration accuracy and strong robustness; only one depth image is needed for calibration, the calibration method has low computational complexity, and can be calibrated in real time; Converting the three-dimensional information in the camera coordinate system to the world coordinate system can easily perform operations such as ground removal and three-dimensional obstacle perception, which has strong application value.

Referring to FIG. 5 , FIG. 5 is a schematic diagram of data association in the calibration method according to an embodiment of the present application. Based on the depth image data, the pixels in the depth image are converted into spatial 3D points in the camera coordinate system; in order to improve the accuracy and success rate of obtaining the fitting plane, the spatial 3D points can be screened; when using the RANSAC algorithm , randomly select three-dimensional points as a subset, and obtain the fitting plane estimation of the subset based on the subset. Through the multiple randomly selected subsets of the algorithm, and the fitting plane estimation of each subset, according to the maximum interior point rate The best fitting plane estimation is selected according to the conditions; the fitting plane is recalculated based on the interior points of the best fitting plane; if the fitting plane is used as the calibration plane, the height of the camera coordinate system relative to the fitting plane is calculated according to the fitting plane transformation, and rotation transformation.

Referring to FIG. 6 , FIG. 6 is a schematic diagram of a calibration device according to an embodiment of the present application. The device includes,

a depth image acquisition module, which acquires depth image data, where the depth image data includes pixel coordinates and depth values;

The conversion module, based on the acquired depth image data, converts the pixel points in the depth image into spatial three-dimensional points in the camera coordinate system;

a fitting plane obtaining module, which obtains the fitting plane of the three-dimensional point based on the three-dimensional point;

The transformation calculation module obtains the parameters between the depth camera and the calibration plane according to the current pose relationship between the calibration plane that is parallel or coincident with the fitting plane and the camera coordinate system;

Wherein, the calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located.

The device further includes a screening module, which screens the three-dimensional points in the three-dimensional point set according to the screening strategy to obtain a screened three-dimensional point set.

The fitting plane acquisition module includes,

The random selection module performs random selection based on the screened set of three-dimensional points, and obtains a current subset formed by randomly selected three-dimensional points, wherein the number of three-dimensional points in the subset includes at least three;

The fitting plane estimation module obtains the fitting plane estimation of the subset based on the three-dimensional points in the current subset;

The degree of conformity obtaining module, according to the obtained fitting plane estimation, obtains the degree of conformity of all the three-dimensional points in the screened three-dimensional point set relative to the fitting plane estimate;

If the degree of conformity is not enough, return to the random selection module;

If the degree of conformity is reached, returning to the re-decomposition module;

Wherein, the inliers include three-dimensional points whose distances from the three-dimensional points in the filtered three-dimensional point set to the fitting plane with the best degree of conformity are less than a preset distance threshold.

The resolving module uses the estimated interior points of the fitted plane with the best fit to solve the fitted plane equation.

The fitting plane estimation module also includes, if the number of three-dimensional points in the current subset is equal to 3, substituting the three-dimensional point coordinate values into the fitting plane equation, solving the unknowns in the fitting plane equation, and obtaining the fitting plane of the current subset. estimate;

If the number of 3D points in the current subset is greater than 3, the coordinate values of the 3D points are substituted into the fitting plane equation, and the unknowns in the fitting plane equation are solved by the least squares method to obtain the fitting plane estimate of the current subset.

The degree of conformity obtaining module further includes calculating a distance value estimated from each three-dimensional point in the screened three-dimensional point set to the fitting plane,

Take the three-dimensional point whose calculated distance value is less than the set distance threshold as the interior point,

count the number of inliers,

Calculate the ratio of the number of inliers to the filtered 3D point cloud to obtain the inlier rate;

determining the degree of compliance according to the interior point rate;

or,

The degree of conformity is determined according to the number of iterations, wherein the number of iterations satisfies:

where m is the number of 3D points in the subset, η is the set confidence level, ε is the interior point rate, is the proportion of interior points under the worst condition, or is set to the proportion under the worst condition in the initial state, and With the number of iterations, it is updated to the current maximum inlier rate;

or,

The degree of conformity is determined according to whether the probability that the v subsets are all interior points satisfies the set confidence level, where the probability that the v subsets are all interior points is:

where λ is the expectation of the number of times the subsets are all interior points in the current iteration.

The resolving module further includes, according to the estimated interior point of the fitting plane with the highest interior point rate, by using the least squares method, resolving the unknowns in the fitting plane equation to obtain the fitting plane equation.

The transformation calculation module includes, according to the fitted plane equation, obtaining the fitted plane normal vector,

According to the normal vector of the fitting plane, obtain the equation of the calibration plane that is parallel or coincident with the fitting plane;

Calculate the distance from the origin of the camera coordinate system to the calibration plane, and obtain the distance transformation of the camera coordinate system relative to the calibration plane;

Calculate the rotation of the camera coordinate system relative to the calibration plane, to the rotation transformation of the camera coordinate system relative to the calibration plane,

The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

The depth image data includes image data of the bearing surface of the mobile robot body, and the transformation calculation module further includes:

According to the normal vector of the fitting plane, the equation of the calibration plane parallel to the fitting plane and having a set distance is obtained; wherein, the calibration plane is the bearing ground of the mobile robot body in the world coordinate system;

According to the equation parameters of the calibration plane, calculate the distance from the origin of the camera coordinate system to the fitting plane, and obtain the height transformation of the camera coordinate system relative to the calibration plane; calculate the rotation of the y-axis of the camera coordinate system relative to the normal vector of the fitting plane, and obtain the camera coordinate system The rotation transformation relative to the calibration plane,

The height transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

The depth image data includes elevation image data that is perpendicular to the bearing surface of the mobile robot body, and the transformation calculation module further includes, according to the normal vector of the fitting plane, obtaining a plane that is parallel to the fitting plane and has a set value. The equation of a fixed distance calibration plane; wherein, the calibration plane is perpendicular to the bearing ground of the mobile robot body in the world coordinate system;

According to the equation parameters of the calibration plane, calculate the distance from the origin of the camera coordinate system to the second fitting plane, and obtain the distance transformation of the camera coordinate system relative to the calibration plane; calculate the rotation of the z-axis of the camera coordinate system relative to the normal vector of the second fitting plane, Obtain the rotation transformation of the camera coordinate system relative to the calibration plane,

The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane.

The present application also provides an electronic device for calibrating external parameters of a depth camera, the electronic device includes a memory and a processor, wherein,

The memory is used to store computer programs;

The processor is configured to execute the program stored in the memory to realize the calibration step of any of the external parameters of the depth camera.

The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; may also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the following steps are implemented:

Obtaining depth image data, the depth image data includes pixel coordinates and depth values;

Based on the acquired depth image data, transform the pixels in the depth image into three-dimensional spatial points in the camera coordinate system;

Based on the three-dimensional points, obtaining a fitting plane associated with the calibration plane;

According to the current pose relationship between the fitting plane and the camera coordinate system, the parameters between the depth camera and the calibration plane are obtained;

Wherein, the calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located.

As for the apparatus/network side device/storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

It should be noted that the embodiments of the depth camera external parameter calibration method provided by the present application may not be limited to the above-mentioned embodiments. When the equation is overdetermined, a solution method other than the least squares method can also be used, for example, regression calculation and the like can be used. In addition, in order to reduce the computational complexity of the over-determined equation, it is also possible to improve the selection of reliable three-dimensional points, for example, supplemented by image feature points.

In this document, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such existence between these entities or operations. The actual relationship or sequence. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (15)

1. a calibration method of depth camera external parameter, is characterized in that, this method comprises, Obtaining depth image data, the depth image data includes pixel coordinates and depth values; Based on the acquired depth image data, transform the pixels in the depth image into three-dimensional spatial points in the camera coordinate system; Based on the three-dimensional point, obtain the fitting plane of the three-dimensional point; According to the current pose relationship between the calibration plane parallel or coincident with the fitting plane and the camera coordinate system, the parameters between the depth camera and the calibration plane are obtained; Wherein, the calibration plane includes any current plane that is parallel or perpendicular to the bearing surface of the mobile robot body where the depth camera is located. 2. The method according to claim 1, wherein the depth image data is at least one depth image, and the conversion of the pixel points in the depth image into the spatial three-dimensional points under the camera coordinate system comprises, The three-dimensional point coordinates corresponding to the pixel points are obtained according to the mapping geometric model of the spatial three-dimensional point in the camera coordinate system and the two-dimensional image in the image coordinate system. 3. The method according to claim 2, wherein, according to the spatial three-dimensional point under the camera coordinate system and the mapping geometric model of the two-dimensional image under the image coordinate system, the three-dimensional point coordinates corresponding to the pixel points are obtained, including , For any pixel in the depth image, Use the depth value of the pixel point as the z coordinate value of the three-dimensional point; Obtain the first difference between the x-coordinate of the pixel point and the offset in the x-direction, multiply the ratio of the z-coordinate value to the focal length in the x-direction of the camera's internal parameters and the first difference, and the obtained result is used as the x-coordinate of the three-dimensional point value; the x-direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the x-direction; Obtain the second difference between the y-coordinate of the pixel and the offset in the y-direction, multiply the ratio of the z-coordinate value to the y-direction focal length in the camera's internal parameters and the second difference, and use the result as the y-coordinate of the three-dimensional point value; the y-direction offset is the offset of the origin of the camera coordinate system relative to the image coordinate system in the y-direction; Wherein, the offset of the origin of the camera coordinate system relative to the x direction in the image coordinate system, and the offset of the origin of the camera coordinate system relative to the y direction in the image coordinate system are obtained according to the camera internal parameters; Convert all pixels in the depth image to 3D point coordinates to obtain a 3D point set. 4. The method of claim 3, further comprising, according to a screening strategy, screening the three-dimensional points in the three-dimensional point set to obtain a screened three-dimensional point set; wherein the screening strategy comprises: Any or any combination of the following conditions: (1) According to the orientation of the camera, remove 3D points in a certain range above the depth image; (2) When there is an initial estimated value of the external parameter, the three-dimensional point in the camera coordinate system is transformed into the world coordinate system through the camera external parameter, and the three-dimensional point whose height in the height direction is greater than the preset height threshold is removed; (3) For the depth camera of binocular stereo vision, remove the three-dimensional points whose depth value is greater than the preset depth threshold; (4) For the TOF depth camera of the time of flight, according to the distance difference between the pixel point and its adjacent pixel points, remove the three-dimensional points whose distance difference is greater than the preset distance threshold. 5. The method according to claim 4, wherein the obtaining the fitting plane of the three-dimensional point based on the three-dimensional point comprises: According to the screened 3D point set, a random sampling consistent RANSAC algorithm is used to obtain the fitting plane equation of the 3D points, wherein the number of 3D points is greater than or equal to 3. 6. The method according to claim 5, characterized in that, using a random sampling consistent RANSAC algorithm to obtain the fitting plane equation of the three-dimensional point, comprising: Based on the filtered 3D point set, random selection is performed to obtain a current subset composed of the randomly selected 3D points, wherein the number of 3D points in the subset includes at least 3; Based on the 3D points in the current subset, obtain a fitting plane estimate for the subset; According to the obtained fitting plane estimation, obtain the degree of conformity of all 3D points in the screened 3D point set relative to the fitting plane estimation; If the degree of conformity is not enough, returning to the step of randomly selecting based on the filtered three-dimensional point set; If the degree of conformity is reached, use the estimated interior point of the fitting plane with the best degree of conformity to solve the fitting plane equation; Wherein, the inliers include three-dimensional points whose distances from the three-dimensional points in the filtered three-dimensional point set to the fitting plane with the best degree of conformity are less than a preset distance threshold. 7. The method of claim 6, wherein the obtaining a fitting plane estimate of the subset based on the three-dimensional points in the current subset comprises: If the number of 3D points in the current subset is equal to 3, substitute the coordinate values of the 3D points into the fitting plane equation, solve the unknowns in the fitting plane equation, and obtain the fitting plane estimation of the current subset; If the number of 3D points in the current subset is greater than 3, the coordinate values of the 3D points are substituted into the fitting plane equation, and the unknowns in the fitting plane equation are solved by the least squares method to obtain the fitting plane estimate of the current subset. 8. The method according to claim 6, wherein, according to the obtained fitting plane estimation, obtaining the degree of conformity of all 3D points in the screened 3D point set relative to the fitting plane estimation, comprising: Calculate the distance value estimated from each 3D point in the filtered 3D point set to the fitting plane, Take the three-dimensional point whose calculated distance value is less than the set distance threshold as the interior point, count the number of inliers, Calculate the ratio of the number of inliers to the filtered 3D point cloud to obtain the inlier rate; determining the degree of compliance according to the inlier rate; The fitting plane equation is solved by using the estimated interior points of the fitting plane with the best degree of conformity, including, using the fitting plane estimate with the highest interior point rate as the best fitting plane estimate, and according to the best fitting plane estimate For the interior point, the unknowns in the fitted plane equation are re-solved by the method of least squares, and the fitted plane equation is obtained. 9. The method of claim 8, wherein the determining the degree of conformity according to the inlier rate comprises: The degree of conformity is determined according to the number of iterations, wherein the number of iterations satisfies:

where m is the number of 3D points in the subset, η is the set confidence level, ε is the interior point rate, is the proportion of interior points under the worst condition, or is set to the proportion under the worst condition in the initial state, and With the number of iterations, update to the current maximum inlier rate. 10 . The method according to claim 8 , wherein the determining the degree of conformity according to the inlier rate comprises: determining the conformity according to whether the probability that the v subsets are all inliers satisfies a set confidence level. 11 . degree, where the probability that the v subsets are all interior points is:

where λ is the expectation of the number of times the subsets are all interior points in the current iteration. 11. The method according to any one of claims 1 to 10, wherein the distance between the depth camera and the calibration plane is obtained according to the current pose relationship between the calibration plane that is parallel or coincident with the fitting plane and the camera coordinate system parameters, including, According to the fitted plane equation, the fitted plane normal vector is obtained, According to the normal vector of the fitting plane, obtain the equation of the calibration plane that is parallel or coincident with the fitting plane; Calculate the distance from the origin of the camera coordinate system to the calibration plane, and obtain the distance transformation of the camera coordinate system relative to the calibration plane; Calculate the rotation of the camera coordinate system relative to the calibration plane, to the rotation transformation of the camera coordinate system relative to the calibration plane, The distance transformation and the rotation transformation are used as parameters between the depth camera and the calibration plane. 12. The method according to claim 11, wherein the depth image data includes image data of the bearing surface of the mobile robot body, According to the fitted plane normal vector, the equation of the calibration plane associated with the fitted plane is obtained, including; According to the normal vector of the fitting plane, the equation of the calibration plane parallel to the fitting plane and having a set distance is obtained; wherein, the calibration plane is the bearing ground of the mobile robot body in the world coordinate system; The calculating the distance from the origin of the camera coordinate system to the calibration plane, and obtaining the distance transformation of the camera coordinate system relative to the calibration plane, including; According to the equation parameters of the calibration plane, calculate the distance from the origin of the camera coordinate system to the calibration plane, and obtain the height transformation of the camera coordinate system relative to the calibration plane; The calculation of the rotation of the camera coordinate system relative to the front view of the calibration plane, to the rotation transformation of the camera coordinate system relative to the calibration plane, includes, According to the equation parameters of the calibration plane, the rotation of the y-axis of the camera coordinate system relative to the normal vector of the fitting plane is calculated, and the rotation transformation of the camera coordinate system relative to the calibration plane is obtained. 13. The method according to any one of claims 11, wherein the depth image data includes elevation image data perpendicular to the bearing surface of the mobile robot body, According to the fitted plane normal vector, the equation of the calibration plane associated with the fitted plane is obtained, including; According to the normal vector of the fitting plane, the equation of the calibration plane that is parallel to the fitting plane and has a set distance is obtained; wherein, the calibration plane is perpendicular to the bearing ground of the mobile robot body in the world coordinate system; The calculating the distance from the origin of the camera coordinate system to the calibration plane, and obtaining the distance transformation of the camera coordinate system relative to the calibration plane, including; According to the equation parameters of the calibration plane, calculate the distance from the origin of the camera coordinate system to the calibration plane, and obtain the distance transformation from the camera coordinate system to the calibration plane; The calculation of the rotation of the camera coordinate system relative to the front view of the calibration plane, to the rotation transformation of the camera coordinate system relative to the calibration plane, includes, According to the equation parameters of the calibration plane, the rotation of the z-axis of the camera coordinate system relative to the normal vector of the calibration plane is calculated, and the rotation transformation of the camera coordinate system relative to the calibration plane is obtained. 14. An electronic device for calibrating external parameters of a depth camera, characterized in that the electronic device comprises a memory and a processor, wherein, The memory is used to store computer programs; The processor is configured to execute the program stored in the memory, and implement the method for calibrating the external parameters of the depth camera according to any one of claims 1-13. 15 . A computer-readable storage medium, wherein a computer program is stored in the storage medium, and the computer program is executed by a processor to perform the method for calibrating external parameters of a depth camera according to any one of claims 1-13 .

Images