CN116934873A - Camera external parameter calibration method, device, equipment and storage medium - Google Patents
Camera external parameter calibration method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116934873A CN116934873A CN202311021285.2A CN202311021285A CN116934873A CN 116934873 A CN116934873 A CN 116934873A CN 202311021285 A CN202311021285 A CN 202311021285A CN 116934873 A CN116934873 A CN 116934873A
- Authority
- CN
- China
- Prior art keywords
- camera
- calibration plate
- characteristic
- determining
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims description 49
- 239000011159 matrix material Substances 0.000 claims description 65
- 238000001514 detection method Methods 0.000 claims description 41
- 238000004422 calculation algorithm Methods 0.000 claims description 27
- 238000012545 processing Methods 0.000 claims description 12
- 238000013519 translation Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 description 10
- 238000005259 measurement Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 230000008719 thickening Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Pure & Applied Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The embodiment of the application provides a camera external parameter calibration method, device, equipment and storage medium. The method comprises the following steps: determining a first characteristic of a set calibration plate corresponding to the first sensor based on the point cloud data of the set calibration plate detected by the first sensor, wherein the set calibration plate is a two-dimensional code calibration plate, and the first characteristic comprises a first normal vector and a first center point; determining a second characteristic of the camera to be calibrated, which is obtained based on the set calibration plate, wherein the second characteristic comprises a second normal vector and a second center point; and obtaining an external parameter matrix corresponding to the camera to be calibrated based on the characteristic constraint equation, the first characteristic and the second characteristic. The embodiment of the application solves the problems of insufficient accuracy and reliability of the environment detection of the automatic driving vehicle and ensures the accuracy and reliability of the environment detection of the automatic driving vehicle.
Description
Technical Field
The embodiment of the application relates to the technical field of automatic driving, in particular to a camera external parameter calibration method, a device, equipment and a storage medium.
Background
With the continuous development of unmanned technologies, the running environment of vehicles is more open and complex, and continuous detection of the environment of the vehicles is required to ensure the running safety of the vehicles. In terms of environmental detection, it is difficult for single sensor data such as lidar or cameras to satisfy the demand. Therefore, in the related art, a multi-sensor data fusion algorithm is generally used to exert the advantages of each sensor, so as to cope with the sensing requirement of the complex scene. In the multi-sensor data fusion processing, it is necessary to ensure that the coordinate system corresponding to the data obtained by the camera and the other sensors is unified. Therefore, the external parameter matrix used for converting the data between different coordinate systems in the camera needs to be calibrated, namely external parameter calibration.
Because of the large size of the unmanned vehicle, the external parameter calibration method in the related art usually uses a checkerboard calibration plate for calibration. However, because the sparse distribution of the point cloud data obtained by the laser radar measurement is uneven, the condition that the points scanned to a certain edge of the checkerboard are few easily occurs, so that the fitting error of the calibration plate is large, the calibration precision is insufficient, and the problem of insufficient accuracy and reliability of the vehicle on environment detection is caused.
Disclosure of Invention
The embodiment of the application provides a camera external parameter calibration method, device, equipment and storage medium, which are used for solving the problems of insufficient accuracy and reliability of environment detection of an automatic driving vehicle.
In a first aspect, the present application provides a camera external parameter calibration method, where the camera external parameter calibration method includes:
determining a first characteristic of a set calibration plate corresponding to the first sensor based on the point cloud data of the set calibration plate detected by the first sensor, wherein the set calibration plate is a two-dimensional code calibration plate, and the first characteristic comprises a first normal vector and a first center point;
determining a second characteristic of the camera to be calibrated, which is obtained based on the set calibration plate, wherein the second characteristic comprises a second normal vector and a second center point;
and obtaining an external parameter matrix corresponding to the camera to be calibrated based on the characteristic constraint equation, the first characteristic and the second characteristic.
It can be seen that, through the point cloud data of the set calibration plate detected by the first sensor, the first characteristic of the set calibration plate corresponding to the first sensor is determined, the second characteristic of the camera to be calibrated obtained based on the set calibration plate is determined, and then the external parameter matrix corresponding to the camera to be calibrated is obtained based on the characteristic constraint equation, the first characteristic and the second characteristic. From this, can utilize the demarcation board characteristic that obtains with the different kind sensor of waiting to mark the camera, wait to mark the camera and carry out the exoparameter and mark, mark through using the dull and stereotyped as demarcation board that contains the two-dimensional code, need not carry out complicated processing such as thickening, fretwork to demarcation board, save the cost, simultaneously through the accurate detection two-dimensional code of camera, guarantee the precision of the second characteristic that the camera measurement obtained, and then guarantee the precision of exoparameter matrix to guarantee the accuracy and the reliability of autopilot vehicle environment detection.
Optionally, the first sensor is a laser sensor.
Therefore, by using the laser sensor as the first sensor matched with the camera to be calibrated, the laser sensor is convenient to detect by utilizing the characteristic of simple structure when the calibration plate is set to be a two-dimensional code calibration plate, the accuracy and the reliability of the first characteristic obtained by the first sensor are ensured, and the accuracy of the external parameter matrix obtained by calibration is further ensured.
Optionally, obtaining an external parameter matrix corresponding to the camera to be calibrated based on the feature constraint equation, the first feature and the second feature includes: acquiring a first characteristic and a second characteristic of a set number of groups; substituting the first features and the second features of the set number into a feature constraint equation, and determining an external parameter matrix by using a parameter matrix when the direction consistency, the direction matching degree and the distance matching degree of the first sensor and the camera to be calibrated are optimized.
Therefore, the direction consistency, the direction matching degree and the distance matching degree of the first sensor and the camera to be calibrated are used as indexes to be optimized, the obtained external parameter matrix is ensured to be optimal on the indexes, and the maximum unification of the camera and the first sensor coordinate system is further ensured, so that the accuracy and the reliability of the automatic driving vehicle on environment detection are ensured.
Optionally, the feature constraint equation comprises:
T(R,t)＝argmin T(R,t) (e d +e r +e t )；
wherein e d E is the index of direction consistency r E is the index of the matching degree of the direction t For the distance matching degree index, M is the set number, T (R, T) is the external reference matrix, R is the rotation matrix in the external reference matrix, T is the translation matrix in the external reference matrix, o l Is the coordinate of the first center point, n l Is a first normal vector o c Is the coordinates of the second center point, n c Is the second normal vector.
Therefore, as the three indexes of the direction consistency, the direction matching degree and the distance matching degree are all non-negative values, when the sum is minimum, namely when the three indexes of the first characteristic obtained by the first sensor and the second characteristic obtained by the camera to be calibrated are optimal, the fact that the external parameter matrix transfers the second characteristic to the same coordinate system of the first characteristic can be ensured, the maximum unification of the coordinate system is realized, and the accuracy and the reliability of the automatic driving vehicle on environment detection are ensured.
Optionally, determining the first characteristic of the set calibration plate corresponding to the first sensor based on the point cloud data of the set calibration plate detected by the first sensor includes: based on the point cloud data, determining a plane equation of a plane where the calibration plate is arranged and a first normal vector corresponding to the plane; determining edge points in the point cloud data based on the plane equation; a first center point in the calibration plate is determined based on the edge points.
Therefore, the first characteristic of the calibration plate can be conveniently determined through the point cloud data so as to be combined with the second characteristic, and the external parameter calibration of the camera to be calibrated is carried out. And the calculation amount is small, the calculation efficiency is high, and the structural accuracy is strong.
Optionally, determining edge points in the point cloud data based on the plane equation includes: projecting the point cloud data to a plane where a set calibration plate is positioned to obtain projected point cloud; and determining the same projection point which is different from two adjacent projection points in the projection point cloud as an edge point, and determining coordinates of the edge point.
Therefore, the relative positions of the projection points corresponding to the points originally located on the same plane are unchanged, and the relative positions of the projection points corresponding to the points other than the points on the same plane are changed, so that the edge points can be conveniently distinguished from the non-edge points, and the edge points can be rapidly identified, so that the first feature can be determined.
Optionally, setting the calibration plate as a circular calibration plate, wherein the circular calibration plate comprises a two-dimensional code; determining a first center point in the calibration plate based on the edge points, comprising: determining target points which have the same distance to all edge points and are positioned on a plane where the set calibration plate is positioned; the target point is determined as a first center point.
Therefore, the target point can be directly determined according to the circle centers of the circles formed by all the edge points of the set calibration plate conveniently by setting the set calibration plate to be circular, and then the first center point is determined, so that the calculation difficulty of the first center point is reduced, the calculation efficiency is improved, and the first characteristic is determined according to the calculation difficulty.
Optionally, determining the second feature of the camera to be calibrated based on the set calibration plate includes: determining a corresponding positioning algorithm according to the type of the set calibration plate; and determining a second characteristic based on the positioning algorithm and image data acquired by the camera to be calibrated.
Therefore, the second characteristic can be quickly obtained by utilizing the existing camera positioning algorithm by determining the corresponding camera positioning algorithm according to the type of the set calibration plate, the processing is convenient, the camera positioning algorithm is suitable for the type of the set calibration plate, and the accuracy of a camera measurement result is ensured.
In a second aspect, the present application provides a camera external parameter calibration device, including:
the first detection module is used for determining first characteristics of the set calibration plate corresponding to the first sensor based on the point cloud data of the set calibration plate detected by the first sensor;
the second detection module is used for determining a second characteristic of the camera to be calibrated, which is obtained based on the set calibration plate, wherein the second characteristic comprises a second normal vector and a second center point;
and the processing module is used for obtaining an external parameter matrix corresponding to the camera to be calibrated based on the characteristic constraint equation, the first characteristic and the second characteristic.
Optionally, the first detection module specifically includes that the first sensor is a laser sensor.
Optionally, the processing module is specifically configured to obtain a first feature and a second feature of the set number of groups; substituting the first features and the second features of the set number into a feature constraint equation, and determining an external parameter matrix by using a parameter matrix when the direction consistency, the direction matching degree and the distance matching degree of the first sensor and the camera to be calibrated are optimized.
Optionally, the processing module specifically includes a feature constraint equation, including:
T(R,t)＝argmin T(R,t) (e d +e r +e t )；
wherein e d E is the index of direction consistency r E is the index of the matching degree of the direction t For the distance matching degree index, M is the set number, T (R, T) is the external reference matrix, R is the rotation matrix in the external reference matrix, T is the translation matrix in the external reference matrix, o l Is the coordinate of the first center point, n l Is a first normal vector o c Is the coordinates of the second center point, n c Is the second normal vector.
Optionally, the first detection module is specifically configured to determine, based on the point cloud data, a plane equation for setting a plane in which the calibration plate is located and a first normal vector corresponding to the plane; determining edge points in the point cloud data based on the plane equation; a first center point in the calibration plate is determined based on the edge points.
Optionally, the first detection module is specifically configured to project the point cloud data onto a plane where the set calibration plate is located, so as to obtain a projected point cloud; and determining the same projection point which is different from two adjacent projection points in the projection point cloud as an edge point, and determining coordinates of the edge point.
Optionally, the first detection module is specifically configured to determine that distances from all edge points are the same and are located at a target point on a plane where the calibration plate is set if the calibration plate is set to be a circular calibration plate, and the circular calibration plate includes two-dimensional codes; the target point is determined as a first center point.
Optionally, the second detection module is specifically configured to determine a corresponding positioning algorithm according to the type of the set calibration plate; and determining a second characteristic based on the positioning algorithm and image data acquired by the camera to be calibrated.
In a third aspect, the present application also provides a control apparatus comprising:
at least one processor;
and a memory communicatively coupled to the at least one processor;
the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the control device executes the camera external parameter calibration method according to any embodiment of the first aspect of the present application.
In a fourth aspect, the present application also provides a computer readable storage medium, in which computer executable instructions are stored, which when executed by a processor are configured to implement a camera external parameter calibration method according to any one of the first aspect of the present application.
In a fifth aspect, the present application also provides a computer program product, which contains computer-executable instructions for implementing the camera exogenous calibration method according to any embodiment corresponding to the first aspect of the present application when the computer-executable instructions are executed by a processor.
Drawings
FIG. 1 is an application scene diagram of a camera extrinsic parameter calibration method according to an embodiment of the present application;
FIG. 2 is a flowchart of a camera extrinsic calibration method according to an embodiment of the present application;
FIG. 3a is a flowchart of a camera extrinsic calibration method according to yet another embodiment of the present application;
FIG. 3b is a flow chart of a method for determining edge points provided in the embodiment of FIG. 3 a;
FIG. 3c is a flow chart of a method of determining a first center point provided in the embodiment of FIG. 3 a;
FIG. 4 is a schematic diagram of a camera external parameter calibration device according to another embodiment of the present application;
fig. 5 is a schematic structural diagram of a control device according to another embodiment of the present application.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the application. Rather, they are merely examples of apparatus and methods consistent with aspects of embodiments of the application as detailed in the accompanying claims.
The following describes in detail the technical solutions of the embodiments of the present application and how the technical solutions of the embodiments of the present application solve the above technical problems with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
With the continuous development of unmanned technologies, the running environment of vehicles is more open and complex, and continuous detection of the environment of the vehicles is required to ensure the running safety of the vehicles. In terms of environmental detection, it is difficult for single sensor data such as lidar or cameras to satisfy the demand. Therefore, in the related art, a multi-sensor data fusion algorithm is generally used to exert the advantages of each sensor, so as to cope with the sensing requirement of the complex scene. In the multi-sensor data fusion processing, it is necessary to ensure that the coordinate system corresponding to the data obtained by the camera and the other sensors is unified. Therefore, the external parameter matrix used for converting the data between different coordinate systems in the camera needs to be calibrated, namely external parameter calibration.
The common external parameter calibration method comprises two calibration steps of calibration through a calibration room and calibration through a checkerboard calibration board, wherein the calibration steps are performed in the indoor calibration room, and the unmanned vehicle is large in size and large in requirement on a measurement range, so that the method is not suitable for calibration of the calibration room. Therefore, the external parameter calibration method in the related art usually uses a checkerboard calibration plate to calibrate, and uses scanning to the points of each side in the calibration plate, fitting each side and determining the intersection point of each side to finish the detection of the calibration plate, but because the sparse distribution of the point cloud data obtained by the laser radar measurement is uneven, the condition that the points scanned to a certain side of the checkerboard are few easily occurs, the error of the calibration plate when fitting the corresponding side and the intersection point is large, the calibration precision is insufficient, and the problem of insufficient accuracy and reliability of the vehicle on the environment detection is caused.
In order to solve the problems, the embodiment of the application provides a camera external parameter calibration method, which is characterized in that a two-dimensional code calibration plate is used for respectively detecting and positioning a first sensor and a camera to be calibrated, and then an external parameter matrix is determined based on the optimization of a detection result so as to complete external parameter calibration and improve the detection precision of the camera.
Fig. 1 is an application scene diagram of a camera external parameter calibration method provided by an embodiment of the present application. As shown in fig. 1, in the process of calibrating the external parameters of the camera, the first sensor 100 and the camera 110 to be calibrated are used to obtain the feature data of the calibration board 120, and the feature data are combined to obtain the external parameter matrix corresponding to the camera to be calibrated, so as to realize the external parameter calibration of the camera.
It should be noted that, in the scenario shown in fig. 1, the first sensor, the camera to be calibrated, and the calibration board are only illustrated by way of example, but the embodiment of the present application is not limited thereto, that is, the number of the first sensor, the camera to be calibrated, and the calibration board may be arbitrary.
The camera external parameter calibration method provided by the application is described in detail by a specific embodiment. It should be noted that the following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
Fig. 2 is a flowchart of a camera external parameter calibration method according to an embodiment of the application. As shown in fig. 2, the camera external parameter calibration method provided in this embodiment includes, but is not limited to, the following steps:
step 201, determining a first characteristic of the set calibration plate corresponding to the first sensor based on the point cloud data of the set calibration plate detected by the first sensor.
The first characteristic comprises a first normal vector and a first center point.
Specifically, the first sensor is a sensor which is matched with a camera to be calibrated and is used for detecting external environment data of the automatic driving vehicle. According to different requirements of practical measurement application, the first sensor can be different types of vehicle-mounted sensors, such as a common laser radar sensor, and also can be different types of sensors such as a prestarded structured light camera, a stereoscopic vision camera and the like. The vehicle-mounted sensor which can be used as the first sensor only needs to be capable of acquiring the point cloud data in the external environment is suitable for the scheme.
The first sensor can acquire point cloud data in an external environment, so that point cloud data in a preset calibration plate can be acquired.
The set calibration plate is a pre-prepared calibration plate containing patterns for positioning, and is different from the existing checkerboard calibration plate or hollowed calibration plate, the set calibration plate adopted in the scheme is a two-dimensional code calibration plate, and the patterns of two-dimensional codes in the specific two-dimensional code calibration plate are different according to the positioning algorithm used by a camera to be calibrated, such as an Apriltag code, an aro calibration plate, a calibration plate or any other existing two-dimensional code calibration plate. Through using the two-dimensional code calibration board but not the fretwork calibration board, utilize the two-dimensional code calibration board only to need the first sensor can distinguish the boundary (namely edge point) of two-dimensional code pattern place region and marginal region, and the point demand quantity of boundary only needs less quantity (if only needs three at least edge point), just can confirm corresponding first characteristic to when solving first sensor and being laser radar sensor, the scanning point cloud that probably appears distributes unevenly, leads to measuring first characteristic to have error problem.
The specific type of the first feature is a feature type which can be detected by the camera to be calibrated, so that an external parameter matrix required for converting the first feature into the second feature is determined by comparing the first feature with the second feature acquired by the camera to be calibrated.
The first feature is usually a first normal vector and a first center point, wherein the first normal vector is a normal vector of a plane where the set calibration plate is located, and the first center point is a center point of a two-dimensional code pattern in the set calibration plate.
Step S202, determining a second characteristic of the camera to be calibrated, which is obtained based on the set calibration plate.
Wherein the second feature comprises a second normal vector and a second center point.
Specifically, the camera to be calibrated can obtain the second characteristic in the set calibration plate based on the positioning algorithm through the shot image of the set calibration plate. According to different types of two-dimension codes in the set calibration plate, the specific positioning algorithm may have differences. The principle of the method is that the frame outline in the two-dimensional code is found out through identification processing of a shot image, then the coordinates (namely the second center point) of the center point of the two-dimensional code pattern are determined based on the coordinates of the frame outlines of all patterns in the two-dimensional code pattern in the shot image, and then the second normal vector of the set calibration plate is determined according to the deformation state of the quadrilateral outline in the two-dimensional code pattern and the center point of the two-dimensional code pattern.
The second normal vector is a normal vector of a plane where the set calibration plate is located, which is acquired by the camera to be calibrated, and the second center point is a center point of the two-dimensional code pattern in the set calibration plate.
And step 203, obtaining an external parameter matrix corresponding to the camera to be calibrated based on the characteristic constraint equation, the first characteristic and the second characteristic.
Specifically, the feature constraint equation is an equation for defining the first feature and the second feature relation from the viewpoint of setting the index. The set index may be a similarity of the first feature and the second feature in different dimensions, such as a direction consistency, a distance matching degree, and the like.
In the set index angle, the smaller the difference value obtained based on the comparison between the second characteristic and the first characteristic after the parameter matrix conversion, the closer the corresponding set index is to the ideal result.
To ensure that the coordinate system of the first sensor and the camera to be calibrated are unified, the set index in the characteristic constraint equation is required to be optimal, and the parameter matrix at the moment is the external parameter matrix. Under the condition that the first characteristic and the second characteristic are known, a characteristic constraint equation set is formed through a plurality of setting indexes, and the external parameter matrix can be obtained by combining the target with the optimal setting index, so that the external parameter calibration of the camera to be calibrated is completed.
According to the camera external parameter calibration method provided by the embodiment of the application, the first characteristic of the set calibration plate corresponding to the first sensor is determined based on the point cloud data of the set calibration plate detected by the first sensor, the second characteristic of the camera to be calibrated based on the set calibration plate is determined, and then the external parameter matrix corresponding to the camera to be calibrated is obtained based on the characteristic constraint equation, the first characteristic and the second characteristic. Therefore, the characteristic of the calibration plate obtained by the sensor of different types from the camera to be calibrated can be utilized, the camera to be calibrated is subjected to external parameter calibration, the flat plate containing the two-dimensional code is used as the calibration plate for calibration, the complex processing such as thickening and hollowed-out is not needed, the cost is saved, meanwhile, the two-dimensional code is accurately detected through the camera, the precision of the second characteristic obtained by the measurement of the camera is ensured, the precision of an external parameter matrix is further ensured, and the accuracy and the reliability of the environment detection of the automatic driving vehicle are further ensured
Fig. 3a is a flowchart of a camera external parameter calibration method according to another embodiment of the present application. As shown in fig. 3a, the camera external parameter calibration method includes:
step S301, determining a plane equation for setting a plane where the calibration plate is located and a first normal vector corresponding to the plane based on the point cloud data.
Wherein the first sensor is a laser sensor.
Specifically, since the laser sensor is the sensor most matched with the camera in the environment detection field of the automatic driving vehicle, the first sensor is used as the laser sensor in the present embodiment, but actually, the first sensor may be other sensors than the laser sensor, which is described with reference to the embodiment shown in fig. 2.
After the point cloud data are acquired, a plane equation of a plane where the calibration plate is set can be determined through an algorithm of the point cloud fitting plane. The algorithm of the point cloud fitting plane has a plurality of existing conventional algorithms, and can be arbitrarily selected by a person skilled in the art according to actual requirements, and is not repeated here.
The expression of the plane equation is typically: ax+by+cz+d=0, wherein (a, b, c, d) is a parameter of a plane equation, and (a, b, c) is a normal vector of the plane.
Therefore, after the plane equation of the set calibration plate is determined through the point cloud fitting plane algorithm, the corresponding normal vector, namely the first normal vector, can be conveniently determined.
Step S302, edge points in the point cloud data are determined based on a plane equation.
Specifically, the point cloud detected by the laser sensor is not only the point cloud on the set calibration plate, but also the point cloud outside the set calibration plate, and the point adjacent to the point cloud outside the set calibration plate, namely the edge point, is located at the edge of the set calibration plate, and the center point of the set calibration plate can be conveniently determined by determining the edge point.
Further, as shown in fig. 3b, a flowchart of a method for determining edge points includes the following specific steps:
and step 3021, projecting the point cloud data to a plane where the set calibration plate is located, so as to obtain a projected point cloud.
Specifically, all the point clouds obtained by monitoring the laser sensor are projected onto the plane where the set calibration plate is located, at the moment, the point originally located on the set calibration plate coincides with the projection point of the set calibration plate, and the point not located on the set calibration plate coincides with the projection point of the set calibration plate.
The calculation method of the projection point cloud can directly substitute the point cloud data into a plane equation of a set calibration plate, and then the point cloud corresponding to the projection point can be obtained.
In step S3022, the same projected point in the projected point cloud as the two adjacent projected points is determined as an edge point, and coordinates of the edge point are determined.
Specifically, because of the sparseness of the point cloud, the distance between the projection point of the point on the non-set calibration plate and the adjacent point on the set calibration plate is generally greater than the distance between the adjacent point on the set calibration plate (and because the object corresponding to the point on the non-set calibration plate is not generally an object parallel to the plane on which the set calibration plate is located, the distances between the points on the non-set calibration plate are also generally different from each other), and therefore, the judgment can be made according to the distance between the projection point cloud and the adjacent point.
For example, when there is a first projection point whose distance to two adjacent projection points (second projection point and third projection point) is the same (denoted as first distance), and a second projection point whose distance to a fourth projection point adjacent to the second projection point (denoted as second distance) is different from the first distance, it is possible to determine that the second projection point is an edge point, the first projection point and the third projection point are non-edge points on the set calibration plate, and the fourth projection point is a projection point of a point on the non-set calibration plate.
Step S303, determining and setting a first center point in the calibration plate based on the edge points.
Specifically, when the shape of the calibration plate is known to be set, the center point of the calibration plate, that is, the first center point, can be determined according to the edge points.
Further, as shown in fig. 3c, a flowchart of a method for determining a first center point includes the specific steps of:
step S3031, determining that the distances from all edge points are the same and are located at the target point of the plane where the set calibration plate is located.
The set calibration plate is a round calibration plate, and the round calibration plate comprises a two-dimensional code.
Specifically, when the calibration plate is set to be a round calibration plate, the coordinates of the point target point can be determined by combining the plane equation of the set calibration plate only by knowing the coordinates of the three edge points.
Step S3032, the target point is determined as the first center point.
Specifically, the target point is the first center point. The determination mode is not limited by the shape of the pattern in the set calibration plate, and can be conveniently and directly obtained by calculation by a laser sensor, so that the calculation difficulty is low, the calculation efficiency is high, and the reliability is high.
Step S304, determining a corresponding positioning algorithm according to the type of the set calibration plate.
Specifically, the two-dimensional code of the set calibration plate can be of various different types, in the prior art, aiming at the two-dimensional code calibration plate with various camera positioning detection, each two-dimensional code has a corresponding positioning algorithm, and a person skilled in the art can select the corresponding two-dimensional code calibration plate type and the corresponding positioning algorithm according to actual conditions and requirements, so that the method is not limited.
Step S305, determining a second feature based on the positioning algorithm and the image data acquired by the camera to be calibrated.
Specifically, the camera to be calibrated can determine the normal vector and the center point of the set calibration plate, namely the second normal vector and the second center point, through the collected image data based on the positioning algorithm corresponding to the two-dimensional code of the set calibration plate.
Step S306, the first characteristic and the second characteristic of the set group number are acquired.
Specifically, steps S301 to S305 are repeated until the first feature and the second feature of the set number of groups are acquired, so that the external parameter matrix is more accurately calculated by the plurality of groups of the first feature and the second feature. The more the number of the set groups, the higher the accuracy and reliability in calculating the extrinsic matrix.
Step S307, substituting the first features and the second features of the set number into a feature constraint equation, and determining an external parameter matrix by using a parameter matrix when the direction consistency, the direction matching degree and the distance matching degree of the first sensor and the camera to be calibrated are optimized.
Specifically, the first feature and the second feature are substituted into a feature constraint equation, and the feature constraint equation is optimized based on an objective function to calculate and obtain an extrinsic matrix.
In some embodiments, the feature constraint equation comprises:
T(R,t)＝argmin T(R,t) (e d +e r +e t )；
wherein e d E is the index of direction consistency r E is the index of the matching degree of the direction t For the distance matching degree index, M is the set number, T (R, T) is the external reference matrix, R is the rotation matrix in the external reference matrix, T is the translation matrix in the external reference matrix, o l Is the coordinate of the first center point, n l Is a first normal vector o c Is the coordinates of the second center point, n c Is the second normal vector.
T (R, T) is the objective function, argmin is the target function (e) d +e r +e t ) The objective function at the minimum takes the value. Substituting the first characteristic and the second characteristic of the set number into the above formula, and calculating to obtain corresponding R and R values, thereby obtaining the external parameter matrix.
According to the camera external parameter calibration method provided by the embodiment of the application, the plane equation of the plane where the calibration plate is located and the first normal vector corresponding to the plane are determined based on the point cloud data, then the edge point in the point cloud data is determined based on the plane equation, then the first center point in the calibration plate is determined based on the edge point, the corresponding positioning algorithm is determined according to the type of the calibration plate, so that the second characteristic is determined, and finally the obtained first characteristic and second characteristic of the set number are substituted into the characteristic constraint equation to determine the external parameter matrix. Therefore, the calibration plate can be set in a circular shape, the calculation data requirement during laser sensor detection is reduced, the detection precision is improved, the detection precision of a camera to be calibrated is guaranteed through a positioning algorithm corresponding to the two-dimensional code pattern, the accuracy of an external parameter matrix determined through a characteristic constraint equation is further guaranteed, and the accuracy and the reliability of automatic driving vehicle environment detection are guaranteed.
Fig. 4 is a schematic structural diagram of a camera external parameter calibration device according to an embodiment of the present application. As shown in fig. 4, the camera external parameter calibration apparatus 400 includes: a first detection module 410, a second detection module 420, and a processing module 430. Wherein:
a first detection module 410, configured to determine a first characteristic of a set calibration plate corresponding to a first sensor based on point cloud data of the set calibration plate detected by the first sensor;
a second detection module 420, configured to determine a second feature of the camera to be calibrated, where the second feature is obtained based on the set calibration board, and the second feature includes a second normal vector and a second center point;
and the processing module 430 is configured to obtain an extrinsic matrix corresponding to the camera to be calibrated based on the feature constraint equation, the first feature and the second feature.
Optionally, the first detection module 410 specifically includes that the first sensor is a laser sensor.
Optionally, the processing module 430 is specifically configured to obtain the first feature and the second feature of the set number of groups; substituting the first features and the second features of the set number into a feature constraint equation, and determining an external parameter matrix by using a parameter matrix when the direction consistency, the direction matching degree and the distance matching degree of the first sensor and the camera to be calibrated are optimized.
Optionally, the processing module 430 specifically includes a feature constraint equation, including:
T(R,t)＝argmin T(R,t) (e d +e r +e t )；
wherein e d E is the index of direction consistency r E is the index of the matching degree of the direction t For the distance matching degree index, M is the set number, T (R, T) is the external reference matrix, R is the rotation matrix in the external reference matrix, T is the translation matrix in the external reference matrix, o l Is the coordinate of the first center point, n l Is a first normal vector o c Is the coordinates of the second center point, n c Is the second normal vector.
Optionally, the first detection module 410 is specifically configured to determine, based on the point cloud data, a plane equation for setting a plane in which the calibration plate is located and a first normal vector corresponding to the plane; determining edge points in the point cloud data based on the plane equation; a first center point in the calibration plate is determined based on the edge points.
Optionally, the first detection module 410 is specifically configured to project the point cloud data onto a plane where the set calibration plate is located, so as to obtain a projected point cloud; and determining the same projection point which is different from two adjacent projection points in the projection point cloud as an edge point, and determining coordinates of the edge point.
Optionally, the first detection module 410 is specifically configured to, if the calibration plate is configured to be a circular calibration plate, wherein the circular calibration plate includes two-dimensional codes, determine that distances from all edge points are the same and are located at a target point on a plane where the calibration plate is configured; the target point is determined as a first center point.
Optionally, the second detection module 420 is specifically configured to determine a corresponding positioning algorithm according to the type of the set calibration plate; and determining a second characteristic based on the positioning algorithm and image data acquired by the camera to be calibrated.
In the embodiment, the camera external parameter calibration device solves the problems of insufficient accuracy and reliability of environment detection of an automatic driving vehicle in the prior art through the combination of the modules, and ensures the accuracy and reliability of the environment detection of the automatic driving vehicle.
Fig. 5 is a schematic structural diagram of a control device according to an embodiment of the present application, and as shown in fig. 5, the control device 500 includes: a memory 510 and a processor 520.
Wherein the memory 510 stores a computer program executable by the at least one processor 520. The computer program is executed by the at least one processor 520 to cause the control device to implement the camera exogenous calibration method provided in any of the embodiments above.
Wherein the memory 510 and the processor 520 may be connected by a bus 530.
The relevant descriptions and effects corresponding to the relevant description and effects corresponding to the method embodiments may be understood, and are not repeated herein.
An embodiment of the present application provides a computer-readable storage medium having stored thereon a computer program that is executed by a processor to implement the camera exogenous calibration method of any of the embodiments described above.
The computer readable storage medium may be, among other things, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
An embodiment of the application provides a computer program product comprising computer-executable instructions for implementing the camera exogenous calibration method of any of the embodiments as corresponds to fig. 2-3 a when executed by a processor.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof.
Claims (10)
1. The camera external parameter calibration method is characterized by comprising the following steps of:
determining a first characteristic of a set calibration plate corresponding to a first sensor based on point cloud data of the set calibration plate detected by the first sensor, wherein the set calibration plate is a two-dimensional code calibration plate, and the first characteristic comprises a first normal vector and a first center point;
determining a second characteristic of the camera to be calibrated, which is obtained based on the set calibration plate, wherein the second characteristic comprises a second normal vector and a second center point;
and obtaining an external parameter matrix corresponding to the camera to be calibrated based on the characteristic constraint equation, the first characteristic and the second characteristic.
2. The camera exogenous calibration method of claim 1, wherein the first sensor is a laser sensor.
3. The camera extrinsic parameter calibration method according to claim 1, wherein the obtaining an extrinsic parameter matrix corresponding to a camera to be calibrated based on the feature constraint equation, the first feature, and the second feature includes:
acquiring the first characteristic and the second characteristic of a set number of groups;
substituting the first features and the second features of the set number into the feature constraint equation, and determining the external parameter matrix by using the parameter matrix when the direction consistency, the direction matching degree and the distance matching degree of the first sensor and the camera to be calibrated are optimized.
4. A camera exogenous calibration method according to claim 3, wherein the characteristic constraint equation comprises:
T(R,t)＝argmin T(R,t) (e d +e r +e t )；
wherein e d E is the index of direction consistency r E is the index of the matching degree of the direction t For the distance matching degree index, M is the set number, T (R, T) is the external reference matrix, R is the rotation matrix in the external reference matrix, T is the translation matrix in the external reference matrix, o l N being the coordinates of the first centre point l For the first normal vector o c N being the coordinates of the second centre point c Is the second normal vector.
5. The camera exogenous calibration method according to any one of claims 1 to 4, wherein the determining the first characteristic of the set calibration plate corresponding to the first sensor based on the point cloud data of the set calibration plate detected by the first sensor includes:
based on the point cloud data, determining a plane equation of a plane where the set calibration plate is located and a first normal vector corresponding to the plane;
determining edge points in the point cloud data based on the plane equation;
and determining a first center point in the set calibration plate based on the edge points.
6. The camera exogenous calibration method according to claim 5, wherein the determining edge points in the point cloud data based on the plane equation includes:
projecting the point cloud data to a plane where the set calibration plate is positioned to obtain projected point cloud;
and determining the same projection point which is different from two adjacent projection points in the projection point cloud as an edge point, and determining the coordinates of the edge point.
7. The camera exogenous calibration method according to claim 5, wherein the set calibration plate is a circular calibration plate, and the circular calibration plate comprises a two-dimensional code;
the determining a first center point in the set calibration plate based on the edge point includes:
determining target points which have the same distance to all edge points and are positioned on the plane where the set calibration plate is positioned;
the target point is determined as the first center point.
8. The camera external parameter calibration method according to any one of claims 1 to 4, wherein the determining the second characteristic of the camera to be calibrated based on the set calibration plate includes:
determining a corresponding positioning algorithm according to the type of the set calibration plate;
and determining the second characteristic based on the positioning algorithm and the image data acquired by the camera to be calibrated.
9. A camera exogenous reference calibration device, comprising:
the first detection module is used for determining first characteristics of the set calibration plate corresponding to the first sensor based on the point cloud data of the set calibration plate detected by the first sensor;
the second detection module is used for determining a second characteristic of the camera to be calibrated, which is obtained based on the set calibration plate, wherein the second characteristic comprises a second normal vector and a second center point;
and the processing module is used for obtaining an external parameter matrix corresponding to the camera to be calibrated based on the characteristic constraint equation, the first characteristic and the second characteristic.
10. A control apparatus, characterized by comprising:
at least one processor;
and a memory communicatively coupled to the at least one processor;
wherein the memory stores instructions executable by the at least one processor to cause the control device to perform the camera exogenous calibration method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311021285.2A CN116934873A (en) | 2023-08-14 | 2023-08-14 | Camera external parameter calibration method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311021285.2A CN116934873A (en) | 2023-08-14 | 2023-08-14 | Camera external parameter calibration method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116934873A true CN116934873A (en) | 2023-10-24 |
Family
ID=88389757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311021285.2A Pending CN116934873A (en) | 2023-08-14 | 2023-08-14 | Camera external parameter calibration method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116934873A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118037857A (en) * | 2024-02-20 | 2024-05-14 | 北京集度科技有限公司 | Sensor external parameter calibration method and related device |
| CN118644560A (en) * | 2024-08-15 | 2024-09-13 | 杭州锐见智行科技有限公司 | Method, device, electronic device and storage medium for determining camera external parameters inside a vehicle |
-
2023
- 2023-08-14 CN CN202311021285.2A patent/CN116934873A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118037857A (en) * | 2024-02-20 | 2024-05-14 | 北京集度科技有限公司 | Sensor external parameter calibration method and related device |
| CN118644560A (en) * | 2024-08-15 | 2024-09-13 | 杭州锐见智行科技有限公司 | Method, device, electronic device and storage medium for determining camera external parameters inside a vehicle |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112907676B (en) | Calibration method, device and system of sensor, vehicle, equipment and storage medium | |
| CN116934873A (en) | Camera external parameter calibration method, device, equipment and storage medium | |
| CN109961468B (en) | Volume measurement method and device based on binocular vision and storage medium | |
| US20220276360A1 (en) | Calibration method and apparatus for sensor, and calibration system | |
| CN106548489B (en) | A kind of method for registering, the three-dimensional image acquisition apparatus of depth image and color image | |
| CN111383279B (en) | External parameter calibration method and device and electronic equipment | |
| WO2021098448A1 (en) | Sensor calibration method and device, storage medium, calibration system, and program product | |
| WO2010018009A1 (en) | 3d time-of-flight camera system and position/orientation calibration method therefor | |
| AU2018282302A1 (en) | Integrated sensor calibration in natural scenes | |
| CN108195305B (en) | Binocular detection system and depth detection method thereof | |
| CN111862208B (en) | Vehicle positioning method, device and server based on screen optical communication | |
| CN111105465B (en) | Camera device calibration method, device, system electronic equipment and storage medium | |
| US20220201164A1 (en) | Image registration apparatus, image generation system, image registration method, and image registration program product | |
| WO2021195886A1 (en) | Distance determination method, mobile platform, and computer-readable storage medium | |
| CN113496528B (en) | Method and device for calibrating position of visual detection target in fixed traffic roadside scene | |
| WO2022217988A1 (en) | Sensor configuration scheme determination method and apparatus, computer device, storage medium, and program | |
| CN116245937A (en) | Method and device for predicting stacking height of goods stack, equipment and storage medium | |
| CN111538008B (en) | Transformation matrix determining method, system and device | |
| CN116068507A (en) | Alignment verification in-vehicle sensors | |
| KR20230067242A (en) | Performance evaluation apparatus and method for autonomous emergency braking system | |
| CN109308714A (en) | Camera and laser radar information method for registering based on classification punishment | |
| CN111336938A (en) | Robot and object distance detection method and device thereof | |
| CN112750165B (en) | Parameter calibration method, intelligent driving method and device, equipment and storage medium thereof | |
| CN114596355B (en) | High-precision pose measurement method and system based on cooperative targets | |
| CN113256734B (en) | Calibration method, system and electronic equipment for vehicle perception sensor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |