KR102421067B1 - Simultaneous estimation apparatus and method for simultaneously estimating the camera pose and 3d coordinates of the target object - Google Patents

Abstract

A simultaneous estimation method for simultaneously estimating a camera posture and 3D coordinates of a target object, the simultaneous estimation method for estimating a camera posture and 3D coordinates of a target object simultaneously, (a) receiving coordinates of a known 3D reference point of a reference object step; (b) with respect to the known 3D reference point, setting a first projected 2D point projected on a first image acquired at a first time epoch through a camera, and by changing the attitude of the camera to the first time epoch setting a second projected 2D point projected on the second image when a second image is acquired at a second time epoch different from ; and (c) in response to the change in the posture of the camera, relation information between the projected 2D point and the 3D point on the image obtained based on the information set in step (b), and on the two images in which the change in the posture of the camera is considered. Using a cost function defined in consideration of information on the relative positional relationship with respect to the known 3D reference point of performing simultaneous estimation of estimated target 3D coordinates that are coordinates of an unknown 3D point of the target object in the second image, wherein the information about the relative positional relationship includes: the second projected 2D point and the second 1 may include relationship information between the projected 2D points.

Description

Simultaneous estimation apparatus and method for simultaneously estimating camera pose and 3D coordinates of a target

The present application relates to a simultaneous estimation apparatus and method for simultaneously estimating a camera posture and 3D coordinates of a target object.

Triangulation is needed to calculate the distance from the camera to the object. This is important in a variety of engineering applications such as surveying, metrology, astronomy, navigation, and target tracking. In particular, triangulation in the computer vision society has provided various useful methodologies such as SfM (Structure-from-Motion), stereo vision, SLAM (Simultaneous Localization And Mapping), and visual localization.

Triangulation assumes that 3D points in the real world coincide with corresponding image points observed by other camera images. The image point may be obtained by, for example, a feature detection and extraction algorithm known in the art. Such a 2D point (2D point) is generally referred to as a feature point, but may be simply referred to as a point instead of a feature point in the present application.

Solving triangulation problems in the absence of noise can be seen as simple. However, in practice this assumption is unrealistic because the 2D points measured in different images do not exactly match due to noise.

In the presence of noise, the triangulation problem estimates the best solution of the 3D point coordinates. The accuracy of coordinate estimation may decrease due to uncertainties in camera movement, calibration parameters, or pixel noise.

In most existing (conventional) methods, the pose (ie position and orientation) of the camera is assumed to be known (ie known in advance) prior to triangulation. The combination of previously reconstructed 3D reference points and their projected image points is called the 2D-3D correspondence, and can be used to estimate the camera pose. This method is widely known as a PnP (Perspective-n-Point) method.

Assuming that the exact coordinates of the 3D reference point are given, there are various versions of a method for accurate camera pose estimations. However, in reality, it can be said that it is difficult to consistently provide accurate coordinates of a 3D reference point. Therefore, it is required to develop a technique for estimating the correct camera posture even in a situation in which an error exists in the coordinates of the 3D reference point (that is, in a situation in which the coordinates of the 3D reference point are inaccurate).

The technology that is the background of the present application is disclosed in Korean Patent Publication No. 10-1645568.

The present application is to solve the problems of the prior art described above, and even in a situation in which an error (error, error) exists in the coordinates of the 3D reference point (that is, the situation in which the coordinates of the 3D reference point is inaccurate), the correct camera posture is estimated In addition, it is an object of the present invention to provide a simultaneous estimation apparatus and method for simultaneously estimating a camera posture and 3D coordinates of a target object, enabling simultaneous estimation of 3D coordinates of an unknown target object.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the simultaneous estimation method for simultaneously estimating the camera posture and the 3D coordinates of the target object according to an embodiment of the present application, (a) receiving the coordinates of the known 3D reference point of the reference object step; (b) with respect to the known 3D reference point, setting a first projected 2D point projected on a first image acquired at a first time epoch through a camera, and by changing the attitude of the camera to the first time epoch setting a second projected 2D point projected on the second image when a second image is acquired at a second time epoch different from ; and (c) in response to the change in the posture of the camera, relation information between the projected 2D point and the 3D point on the image obtained based on the information set in step (b), and on the two images in which the change in the posture of the camera is considered. Using a cost function defined in consideration of information on the relative positional relationship with respect to the known 3D reference point of performing simultaneous estimation of estimated target 3D coordinates that are coordinates of an unknown 3D point of the target object in the second image, wherein the information about the relative positional relationship includes: the second projected 2D point and the second 1 may include relationship information between the projected 2D points.

In addition, in the step (c), the cost function is an indirect measurement value in which a depth-related scale factor is taken into consideration, and a full state vector defined to include the estimated target camera posture-related state parameter and the estimated target 3D coordinate-related 3D coordinate estimation error. It may be defined as a total indirect measurement vector defined to be related to , and a relationship between an entire observation matrix indicating a connection relationship between the indirect measurement value and the total state vector.

Also, in step (c), the simultaneous estimation is performed through the minimization of the cost function, and the minimum error among the error values of the entire state vector that is a value of the cost function calculated by multiplying the entire indirect measurement vector and the entire observation matrix. Based on the overall state vector for calculating the value, the estimated camera posture and the estimated target 3D coordinates may be simultaneously estimated.

일 수 있다. 여기서,

는 추정오차를 갖는 추정된 전체 상태 벡터,

는 추정된 전체 관측 행렬,

는 추정된 전체 간접 측정 벡터이고,

의 노름(norm)이 수렴할때까지 전체 상태 벡터의 추정 값

는 업데이트될 수 있다.In addition, in step (c), the cost function is set to satisfy Equation 1 below, and [Equation 1] is

can be here,

is the estimated full state vector with estimation error,

is the estimated total observation matrix,

is the estimated total indirect measurement vector,

Estimate of the entire state vector until the norm of

can be updated.

In addition, the coordinates of the known 3D reference point in the step (a) may include the coordinates of the estimated 3D reference point in which an error in which an estimation error of the actual 3D point of the reference object exists.

In addition, in the step (c), the information about the relationship information and the location relationship may be information set based on Euclidean geometry.

Meanwhile, a simultaneous estimation apparatus for simultaneously estimating a camera posture and 3D coordinates of a target object according to an embodiment of the present application includes: an input unit for receiving coordinates of a known 3D reference point of a reference object; With respect to the known 3D reference point, set a first projected 2D point projected on a first image acquired at a first time epoch through a camera, and different from the first time epoch by changing the attitude of the camera a setting unit configured to set a second projected 2D point projected on the second image when a second image is obtained in a second time epoch; and relation information between a projected 2D point and a 3D point on an image obtained based on the information set in the setting unit in response to the change in the posture of the camera, and the known 3D reference point on the two images in which the change in the posture of the camera is considered Using a cost function defined in consideration of information on the relative positional relationship with and an estimator configured to simultaneously estimate the estimated target 3D coordinates, which are coordinates of the unknown 3D point of the target object, wherein the information on the relative positional relationship corresponds to the known 2D reference point projected on the second image. and relationship information between the second projected 2D point and the first projected 2D point projected on the first image.

In addition, the cost function includes a global state vector defined to include the estimated target camera posture-related state parameter and the estimated target 3D coordinate-related 3D coordinate estimation error, and a total state vector defined to relate a depth-related scale factor to the considered indirect measurement value. It may be defined as an indirect measurement vector and a relationship between an indirect measurement and an entire observation matrix indicating a connection relationship between the overall state vector.

In addition, the estimator performs the simultaneous estimation through the minimization of the cost function, and calculates a minimum error value among the error values of the entire state vector, which is a value of the cost function calculated by multiplying the entire indirect measurement vector and the entire observation matrix. Based on the overall state vector, the estimated camera posture and the estimated 3D coordinates may be simultaneously estimated.

일 수 있다. 여기서,

는 추정오차를 갖는 추정된 전체 상태 벡터,

는 추정된 전체 관측 행렬,

는 추정된 전체 간접 측정 벡터이고,

의 노름(norm)이 수렴할때까지 전체 상태 벡터의 추정 값

는 업데이트될 수 있다.In addition, the cost function is set to satisfy Equation 2 below, and [Equation 2] is

can be here,

is the estimated full state vector with estimation error,

is the estimated total observation matrix,

is the estimated total indirect measurement vector,

Estimate of the entire state vector until the norm of

can be updated.

Also, the known coordinates of the 3D reference point may include the coordinates of the estimated 3D reference point in which an error in which an estimation error of the actual 3D point of the reference object exists.

In addition, the relationship information and the information about the positional relationship may be information set based on Euclidean geometry.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, by providing a simultaneous estimation apparatus and method for estimating the camera posture and the 3D coordinates of the target object at the same time, there is an error (error, error) in the coordinates of the 3D reference point (that is, , it is possible to provide the effect of estimating the correct camera posture even when the coordinates of the 3D reference point are inaccurate) and simultaneously estimating the 3D coordinates of an unknown target object.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a diagram schematically showing a diagram of a conventional triangulation (top) method and a conventional PnP method (bottom).
2 is a diagram schematically illustrating a simultaneous estimation method by a simultaneous estimation apparatus for simultaneously estimating a camera posture and 3D coordinates of a target object according to an embodiment of the present application.
3 is a diagram illustrating a schematic configuration of a simultaneous estimation apparatus for simultaneously estimating a camera posture and 3D coordinates of a target object according to an embodiment of the present application.
4 is a diagram showing RMSEs of all experiments that increase the standard deviation of pixel noise as a result of an experiment of the present application.
5 is a view showing comparison results of RMSEs of each of the coordinates of a camera posture, a camera rotation, and an unknown 3D point as an experimental result of the present application.
6 is an experimental result of the present application, in constant Gaussian pixel noise and constant uncertainty of 3D reference coordinates, estimation of camera posture, camera rotation, and coordinates of unknown 3D points by different methods along different reference lines and depths It is a diagram comparing errors.
7 is a diagram depicting an experimental environment in which a matched 2D point is displayed on two consecutive images as an experimental result of the present application.
8 is a diagram illustrating a comparative example of RMSEs of a camera position, a camera rotation, and an unknown 3D point as an experimental result of the present application.
9 is a diagram schematically illustrating a configuration of a simultaneous estimation apparatus for simultaneously estimating a camera posture and 3D coordinates of a target object according to an exemplary embodiment of the present disclosure.
10 is a flowchart illustrating a simultaneous estimation room for simultaneously estimating a camera posture and 3D coordinates of a target object according to an exemplary embodiment of the present disclosure.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be “connected” to another part, it is not only “directly connected” but also “electrically connected” or “indirectly connected” with another element interposed therebetween. "Including cases where

Throughout this specification, when it is said that a member is positioned "on", "on", "on", "under", "under", or "under" another member, this means that a member is located on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

We utilize hybrid correspondences (i.e., 2D-2D correspondence and 2D-3D correspondence) to simultaneously record camera pose (i.e. position and orientation) and unknown (unknown, unknown) 3D coordinates of an object. We propose techniques that can be estimated. In this case, as a hybrid correspondence herein, coincident 2D-2D and 2D-3D points viewed at different time epochs (time base) with a single camera may be considered. In addition, an iterative least square method may be applied herein to minimize an image reprojection error. The least squares method may be otherwise referred to herein as the least squares method or the like.

1 is a diagram schematically showing a diagram of a conventional triangulation (top) method and a conventional PnP method (bottom). 2 is a diagram schematically illustrating a simultaneous estimation method by the simultaneous estimation apparatus 10 for simultaneously estimating a camera posture and 3D coordinates of a target object according to an embodiment of the present application. 3 is a diagram illustrating a schematic configuration of a simultaneous estimation apparatus 10 for simultaneously estimating a camera posture and 3D coordinates of a target object according to an embodiment of the present application.

1 and 2, the question mark '?' The indication may mean unknown parameters to be estimated (to be estimated, to be estimated, to be estimated).

That is, FIG. 2 shows an example of a geometrical model of the device 10 . Referring to FIG. 2 , points (blue points) corresponding to 2D-3D correspondence in the device 10 and points (yellow points) corresponding to 2D-2D correspondence in the device 10 are may mean a measurement value of (ie, a measurement value measured by the device). In addition, 3D coordinates (ie, P ₄ and P ₅ ) and R and t indicated in red in FIG. 2 mean an estimation target (estimation item, estimation parameter) estimated by the estimator 13 of the apparatus 10 . can do. In this case, the estimation target is marked with a question mark (?) to distinguish it from the measured value. Also, in Fig. 2, the uppercase letter P denotes the three-dimensional coordinates of the object (coordinates of the 3D point), and the lowercase letter p denotes the two-dimensional coordinates of the object projected onto the image (that is, the coordinates of the projected 2D point projected on the image). can represent

Hereinafter, the simultaneous estimation apparatus 10 for simultaneously estimating the camera posture and the 3D coordinates of the target object according to an embodiment of the present application will be referred to as the present apparatus 10 for convenience of description. In addition, the simultaneous estimation method of simultaneously estimating the camera posture and the 3D coordinates of the target object performed by the apparatus 10 will be referred to as a proposed method for convenience of description below. In addition, the content described with respect to the apparatus 10 herein may be equally applied to the description of the proposed method even if the content is omitted below, and vice versa.

1 to 3 , the device 10 can simultaneously estimate the camera posture (ie, the position and direction of the camera) and the 3D coordinates of the unknown points from the acquired image. It's about technology.

The proposed method by the apparatus 10 is a triangulation method as shown in the upper figure of FIG. 1 related to 2D-2D correspondence, and PnP (Perspective-n-Point) as shown in the lower figure of FIG. 1 related to 2D-3D correspondence. It can be said that all methods are closely related (fusion). This apparatus 10 may be referred to as a single estimator 10 based on hybrid correspondence (ie, 2D-2D correspondence and 2D-3D correspondence).

The proposed method has a distinct advantage of the multi-baseline technique, which can improve the depth accuracy proportional to the multi-baseline between cameras.

In addition, the proposed method, for example, [R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge university press, 2003.] by using only 2D matching points (that is, using only 2D-2D correspondence) with the basic matrix-based conventional methods such as It is possible to directly estimate a length scale parameter for .

The proposed method can be more robust against noise uncertainties of 3D reference points by using a mixture of 2D-2D correspondence and 2D-3D correspondence (ie, using hybrid correspondence). Specifically, compared with the proposed method, most conventional PnP methods rely only on 2D-3D correspondence for camera pose estimation. These conventional methods only consider the uncertainty of the 2D point, assuming that the 3D reference point coordinates are accurately given. Therefore, the proposed method is such that it is possible to obtain a more accurate camera posture even in a situation in which an error (error, error) exists in the coordinates of the 3D reference point (that is, a situation in which the coordinates of the 3D reference point are inaccurate). assumptions can be alleviated. That is, the proposed method can estimate a more accurate camera posture even when inaccurate 3D reference point coordinates are given.

In addition, the proposed method can be converted to a triangulation method or a PnP method (ie, a camera posture estimation method) according to the availability of measurements. If only 2D-2D correspondence is possible, the proposed method may be the same as the triangulation method. On the other hand, if only 2D-3D correspondence is possible, the proposed method may be the same as the PnP method.

Hereinafter, prior to performing a detailed description of the apparatus 10 , the technology underlying the apparatus 10 (triangulation technology and camera posture estimation technology using PnP) will be described in more detail.

A more detailed description of the triangulation technique follows.

노름을 사용한 문헌이 존재한다.The most extensive approach for two-view triangulation is the coordinates of a 3D point that minimizes the L ₂ norm of image reprojection errors. ) is to be found. This approach can generally be implemented by non-iterative polynomial methods. A maximum likelihood estimate (MLE) of unknown (unknown) coordinates can be obtained under the assumption that image points are perturbed by Gaussian pixel noise. Another two-view (two-view) method was studied by an iterative method, but based on the same assumptions. In addition, in the prior art, instead of minimizing the L ₂ norm of the image reprojection error, in order to minimize angular Eigenreprojection errors, L ₁ and

There is literature using gambling.

In addition to the polynomial methods mentioned above, there are also linear triangulation methods such as linear least-square (LLS) and linear-eigen (LE). However, these linear triangulation methods have no geometric meaning and do not correspond to a cost function that minimizes the L ₂ norm of the image reprojection error.

Therefore, in order to minimize the same errors of the cost function, a technique in which iterative linear methods are applied to LLS and LE has been previously proposed. These improved LLS and LE methods are called Iterative-LS method and Iterative-Eigen method, respectively. Both of these methods favor the simplicity of software implementation. However, these two methods sometimes fail to converge in unstable situations when some points are close to epipoles.

In addition, in order to improve the accuracy of estimating unknown 3D coordinates in the prior art, new triangulation approaches based on three or more views have been proposed. One of these prior documents introduced a polynomial method based on the minimization of the L ₂ norm of the image reprojection error, and in contrast to this, an iterative nonlinear least square method was proposed in any other prior document. .

However, all of the above-described conventional triangulation methods follow the assumption that the camera pose should be accurately known in advance, and detailed consideration of finding the unknown coordinates of the 3D point when the camera pose is incorrect is made. There is a problem that is not being lost.

Meanwhile, a more detailed description of the camera pose estimation technique is as follows.

In the early 1970s, the integration of a global positioning system (GPS) and an inertial navigation system (INS) was the main stream in vehicle ego-motion estimation. Recently, vision-based approaches have proven to be more accurate, cheaper and more versatile.

Traditional approaches involve using either a 2D-3D counterpart, known as structure-based pose estimation, or a 2D-2D counterpart, known as structure-less pose estimation. is focused on Each of these is one of the oldest and most important subjects in the fields of photogrammetry and computer vision. A widely used PnP method is based on 2D-3D correspondence. The P3P (Perspective-Three-Points) method corresponds to the minimal case of the PnP problem, and many variants of the basic P3P method have been studied since then.

There are other prior literatures using both 2D-3D and 2D-2D correspondence to solve the PnP problem. These hybrid methods make full use of all possible matches. The minimum case for various possible combinations will be described as follows. Illustratively, a joint formulation of posture estimation based on one 3D-3D correspondence and two 2D-3D correspondences has been proposed in the prior art. Here, compared with other non-hybrid PnP methods, one of three 2D-3D matches is replaced with one 3D-3D match. The superseded 3D-3D match contains 3D points whose coordinates need to be triangulated beforehand. However, this non-hybrid method has a problem in that it largely depends on the quality of triangulated points in a local frame.

In consideration of this, the proposed method or the apparatus 10 providing the proposed method combines the PnP method (ie, the camera posture estimation method) and the triangulation method, The coordinates of 3D points (unknown 3D points, unknown 3D points) and camera poses can be estimated simultaneously. To this end, the apparatus 10 may minimize the L ₂ norm of an image reprojection error with respect to a cost function. In this case, minimization may be implemented by a single estimator 10 (the present apparatus) based on an iterative least squares scheme. A standard perspective projection model may be used in the apparatus 10 , and two-view cases may be considered.

Hereinafter, coordinate frames and notations considered (applied) in the device 10 will be described.

Perspective projection (perspective projection) is expressed in Euclidean coordinates rather than homogeneous coordinates commonly used in computer vision society. This allows a meaningful definition of the distance between points, that is, the Euclidean distance. For this reason, in the present apparatus 10, the overall Euclidean coordinates can be used.

A description of several coordinate frames and mathematical notations considered in the present device 10 is as follows.

A V-frame represents a vision frame, also commonly referred to as an image plane (image plane). A P-frame represents a pixel frame representing visual measurements. C-frame represents a camera frame. N-frame (N-frame) represents a navigation frame (navigation frame).

The origin of the C-frame is at the optical center of the camera, and the XYZ axes may follow a forward-right-down convention. N-frames may be aligned in local north, east, and downward directions.

In the device 10, since the N-frame is set as a reference frame and the initial time can be arbitrarily allocated even if the C-frame is different with time, generality (generality) Without losing , it can be assumed that the initial position and orientation of the C-frame coincide with the initial position and orientation of the N-frame.

In describing the apparatus 10, small bold letters may be used to represent vectors. Ordinary letters can be used to represent coordinate elements. Superscripts and subscripts attached to small letters may represent reference frames and object identities, respectively.

는 N-프레임에 대한 카메라의 위치 벡터(position vector)를 나타낼 수 있다. 그리고, 벡터

의 좌표 원소(coordinate elements)는

로 표시될 수 있다.for example,

may represent a position vector of the camera with respect to the N-frame. and vector

The coordinate elements of

can be displayed as

는 이전 (k-1) 번째 시간 에포크(previous (k-1)-th time epoech)에서 현재 (k) 번째 시간 에포크(current (k)-th time epoech)까지의 카메라의 변환 벡터(translation vector)를 나타내며, 그 좌표 원소(coordinate elements)는 (k-1) 번째 시간 에포크에서 C-프레임에 대해 표현될 수 있다. 본원에서 시간 에포크(time epoech)는 시간 기점, 시점 등으로 달리 지칭될 수 있다.

is the translation vector of the camera from the previous ( k -1) -th time epoch to the current ( k ) -th time epoch , and the coordinate elements may be expressed for the C-frame in the ( k −1) th time epoch. A time epoch may be otherwise referred to herein as a time base, a point in time, or the like.

는 (k) 번째 시간 에포크에서 j 번째 포인트(j-th point, j 번째 지점)를 나타내고, 벡터

의 좌표 원소는

로 표시될 수 있다.

denotes the j-th point ( j - th point, j-th point) in the ( k )-th time epoch, the vector

The coordinate elements of

can be displayed as

는 C-프레임에서 N-프레임까지의 회전 행렬(rotation matrix)을 나타낸다.

는 (k-1) 번째 시간 에포크의 C-프레임에서 (k) 번째 시간 에포크의 C-프레임까지의 회전 행렬을 나타낸다.Bold capital letters can be used to indicate matrices.

denotes a rotation matrix from C-frame to N-frame.

denotes the rotation matrix from the C-frame of the ( k −1)-th time epoch to the C-frame of the ( k )-th time epoch.

Hereinafter, error models that can be considered (applied) in the device 10 will be described.

The relationships between estimates, estimation errors, measurements, measurements errors, and 3D points and their projected image points are set as shown in Equation 1 below. can be Here, the projected image point may be otherwise referred to as a projected 2D point.

[Equation 1]

와

는, 각각 실세계(real world)의 실제 3D 포인트(true 3D point)와 이미지에 실제 투영된 2D 포인트(true projected 2D point)를 나타낸다. 또한, 상부에 햇(upper hat)은 추정(estimate)을 나타내기 위해 사용되고, 상부에 틸데(upper tilde)는 측정(measurement)을 나타내기 위해 사용될 수 있다. 추정오차(estimation error)는 해당 진리값(truth value, 실제 값) 앞에

로 표시되고, 측정오차(measurement error)는

에 의해 표시될 수 있다.here,

Wow

represents a real 3D point in the real world and a true projected 2D point in the image, respectively. Also, an upper hat may be used to indicate an estimate, and an upper tilde may be used to indicate a measurement. The estimation error is preceded by the corresponding truth value (actual value).

is displayed, and the measurement error is

can be indicated by

, 실제 회전 행렬(true rotation matrix)인

과 C-프레임에서 N 프레임까지의 회전 행렬 오차(rotation matrix error)인

는 아래 식 2에 의해 관련이 있을 수 있다.The estimated rotation matrix is

, which is the true rotation matrix

and the rotation matrix error from C-frame to N frame,

can be related by Equation 2 below.

[Equation 2]

을 아래 식 3과 같이 나타낼 수 있다.Assuming that the rotation error is small,

can be expressed as Equation 3 below.

[Equation 3]

는 C-프레임과 N-프레임 사이의 작은 회전(small rotation)으로 정의되는 자세 오차 벡터(attitude error vector)를 나타낸다. 이는 각각 롤, 피치, 요(

)의 오차로 구성될 수 있다.

는 자세 오차 벡터인

에 의해 구성된 비대칭 행렬(skewsymmetric matrix)을 의미한다.here,

denotes an attitude error vector defined as a small rotation between the C-frame and the N-frame. These are roll, pitch, and yaw (

) can be composed of an error of

is the posture error vector

It means a skewsymmetric matrix constructed by .

Using Equations 2 and 3, a rotation matrix error can be derived (derived) as in Equation 4 below.

[Equation 4]

By a similar procedure, if Equation 4 is transposed, it can be expressed as Equation 5 below.

[Equation 5]

Hereinafter, the proposed method by the apparatus 10 will be described in more detail. In particular, the formulation of the proposed method for estimating the camera pose and the coordinates of the unknown 3D points simultaneously (simultaneously) will be described in more detail below.

In the proposed method, it may be required to know the coordinates of a single 3D reference point and its projected 2D point with feasible errors. In other words, the proposed method may know the coordinates of a single 3D reference point and its projected 2D point to be inaccurate with an error (provided that it is known to be inaccurate). can do). In addition, the proposed method may require that at least three new points be observed between two consecutive images (that is, subject to can). Here, the former condition may be related to 2D-3D correspondence, and the latter condition may be related to 2D-2D correspondence.

Under these conditions, the apparatus 10 may estimate unknown 3D coordinates of new points with an updated camera pose.

Hereinafter, error models developed by the apparatus 10 (or the proposed method of the apparatus) will be described. The error model developed by the proposed method can be explained in the following order by its geometric relations. That is, the error model developed by the proposed method is 2D-3D for unknown 3D points, 3D-3D between two views, and known 3D points ) can be described respectively in the order of the 2D-3D and full formulations. Each of these will be referred to as an error model related to the first to fourth parts in the following description.

First, with respect to the first part-related error model, a description of a 2D-3D error model for unknown 3D points is as follows.

는 아래 식 6과 같이 표현될 수 있다.Using a conventionally well-known pinhole camera model

can be expressed as Equation 6 below.

[Equation 6]

,

및

는 P-프레임(P-frame)의 실제 이미지 포인트(true image point)를 나타내며, 이는 일예로 종래에 기 공지된 코너 검출기(corner detector)와 같은 어떠한 특징의 검출(feature detection) 또는 추출 알고리즘(extraction algorithms)을 이용해 획득될 수 있다. here,

,

and

denotes a true image point of a P -frame, which is an example of any feature detection or extraction algorithm, such as a conventionally known corner detector. algorithms) can be obtained.

는 V-프레임(V-frame)의 실제 이미지 포인트(true image point)(즉, 실제 2D 포인트)를 나타낸다. P-프레임과 V-프레임의 각 축(axis)의 방향(direction)은 동일(identical)할 수 있다.

와

는 보정 툴(calibration tools)를 통해 획득할 수 있는 카메라의 초점 길이(focal length)와 주요 포인트(principal point)를 의미한다.

represents the true image point (ie, the real 2D point ) of the V -frame. A direction of each axis of the P- frame and the V -frame may be identical.

Wow

denotes a focal length and a principal point of a camera that can be obtained through calibration tools.

A linearized error model can be derived as shown in Equation 7 below by perturbing Equation 6 up to the first order.

[Equation 7]

는 3D 포인트(3D point)에서 차별화된(differentiated) 투시 투영(perspective projection)에 대한 야코비 행렬(Jacobian matrix)을 나타낸다.

는 깊이(depth)와 관련된 스케일 인수(scale factor)를 나타내고,

는

를 추출하기 위한 간접 측정(indirect measurement, 간접 측정치)를 나타낸다.here,

denotes a Jacobian matrix for a perspective projection differentiated from a 3D point.

represents a scale factor related to depth,

Is

Indicates an indirect measurement (indirect measurement) for extracting .

Equation 7 represents a cost function, and minimization may be made for the rear portion indicated by a wave in Equation 7.

, 3D 좌표 오차(3D coordinate error)

, 및 측정 노이즈(measurement noise, 측정치 노이즈)

는

와 같이 정의될 수 있다.Jacobian

, 3D coordinate error

, and measurement noise (measurement noise)

Is

can be defined as

는 평균(mean)

과 공분산 행렬(covariance matrix)

를 가진 가우스 분포(Gaussian distribution)를 나타낸다.here,

is the mean

and covariance matrix

represents a Gaussian distribution with .

The equation of Equation 8 below can minimize the image re-projection error even when multiplied by a scale factor. Therefore, the equation of Equation 8 below can handle the same estimation problem as Equation 7 above.

[Equation 8]

일 수 있다. 상기 식 8은 알려지지 않은 3D 좌표 오차(unknown 3D coordinate error)

의 오차 모델(error model)에 해당할 수 있다. 야코비 행렬

는 2D와 3D 포인트 사이의 관계(relationship)를 정의할 수 있다. here,

can be Equation 8 is an unknown 3D coordinate error

may correspond to an error model of Jacobian

may define a relationship between 2D and 3D points.

Based on Equation 8, in the proposed method (that is, the device 10) is the same 3D point (same 3D point) that can be seen (viewed) in two consecutive epochs as described below We can derive two new error models of

Second, with respect to the error model related to the second part, a description of the error model of 3D-3D between two views is as follows.

As an example, it is assumed that a single common 3D point captured by the same camera in two consecutive epochs is unknown ( That is, suppose that one common 3D point is unknown). This can be estimated in relation to the C -frame in the current epoch.

Hereinafter, for convenience of description, C -frames defined in two different time epochs may be denoted as C ( k −1 ) and C ( k ). As described above, the time epoch may mean a time base, a time point, or the like. That is, two consecutive epochs (or two different time epochs) may mean different consecutive time points (time points at different consecutive times).

Coordinates of the same 3D point viewed from two consecutive epochs can be expressed as Equation 9 below.

[Equation 9]

Based on Equation 9, which is expressed as a relation between true vectors, this equation can also be expressed as Equation 10 below which is expressed as a relation between estimated vectors.

[Equation 10]

는 추정된 회전 행렬(estimated rotation matrix)을 나타낸다.

에 포함된 회전 오차 행렬(rotation error matrix)

는 일예로 종래에 기 공지된 프사이 각도 접근법(psi-angle approach)에 기초하여 후술하는 식 25 내지 식 28에서와 같이 완전히 도출(fully derived)될 수 있다.At this time, in Equation 10,

denotes an estimated rotation matrix.

rotation error matrix contained in

As an example, based on a previously known psi-angle approach, can be fully derived as in Equations 25 to 28 to be described later.

에 포함된 추정 오차(estimation error)

는 아래 식 11과 같이 분해(decomposed)될 수 있다.Estimated coordinates are obtained by perturbing Equation 10 by first-order Taylor series expansion and using Equation 28 to be described later.

Estimation error included in

can be decomposed as in Equation 11 below.

[Equation 11]

는 두 개의 연속된 에포크(two consecutive epochs) 사이의 증분 자세 오차(incremental attitude error)를 나타낸다.here,

denotes the incremental attitude error between two consecutive epochs.

,

, 및

는 본 장치(10)에 의하여 아래 식 12와 같이 추정해야 하는 파라미터(parameters)(즉, 본 장치에 의해 추정되는 파라미터)를 의미할 수 있다.In Equation 11,

,

, and

may mean parameters to be estimated by the apparatus 10 as in Equation 12 below (ie, parameters estimated by the apparatus 10 ).

[Equation 12]

는 알려지지 않은 3D 포인트

, 변환 벡터(translation vector), 및 증분 자세 오차 벡터(incremental attitude error vector)를 포함하는 전체 상태 벡터(full state vector)를 나타낸다. here,

is an unknown 3D point

, a translation vector, and a full state vector including an incremental attitude error vector.

,

, 및

에 관한 간접 측정(indirect measurement, 간접 측정치)

는 아래 식 13을 통해 획득될 수 있다.Substituting Equation 11 into Equation 8, in the previous ( k -1)-th time epoch,

,

, and

indirect measurement of

can be obtained through Equation 13 below.

[Equation 13]

에 관한 간접 측정(indirect measurement, 간접 측정치)

는 아래 식 14와 같이 표현될 수 있다.Similarly, at the current ( k ) time epoch

indirect measurement of

can be expressed as Equation 14 below.

[Equation 14]

와 누적 잔차 벡터(stacked residual vector)

는 아래 식 15와 같이 구성될 수 있다.Combining Equations 13 and 14 above, a stacked observation matrix

and the stacked residual vector

can be configured as in Equation 15 below.

[Equation 15]

일 수 있으며,

일 수 있다. 즉,

은 (n x n) 제로 행렬을 나타내고,

는 알려지지 않은 3D 포인트의 수를 나타낸다.In Equation 15,

can be,

can be in other words,

denotes a (nxn) zero matrix,

represents the number of unknown 3D points.

From Equations 12 to 15, the subscript U is used to discriminate between the error model of known 3D points and the error model of unknown 3D points, It is a quote from the first letter of the word 'unknown'.

Third, with respect to the third part-related error model, a description of the error model of 2D-3D with respect to known 3D points is as follows. That is, hereinafter, an error model considering a relationship between a known 3D point and two consecutive camera frames will be described.

Assume that the 3D point of the C -frame is known in advance at the previous ( k −1) th time epoch. In this case, the same 3D point seen in the current ( k )-th time epoch can be estimated as in Equation 16 below using Equations 1 and 2 above.

[Equation 16]

에 포함된 추정 오차(estimation error)

는 식 16을 1차로(up to the first order) 섭동(perturbing)함으로써 아래 식 17과 같이 도출할 수 있다.

Estimation error included in

can be derived as Equation 17 below by perturbing Equation 16 up to the first order.

[Equation 17]

를 사전에(beforehand) 알고 있기 때문에

를 고려할 필요가 없다.Comparing this Equation 17 with the error model for the unknown 3D point of Equation 11 above, Equation 17 is

because we know beforehand

no need to consider

와

에 관한 간접 측정(indirect measurement, 간접 측정치)

는 아래 식 18을 통해 획득될 수 있다.Substituting Equation 17 into Equation 8, the state

Wow

indirect measurement of

can be obtained through Equation 18 below.

[Equation 18]

와 누적 잔차 벡터(stacked residual vector)

는 아래 식 19와 같이 구성될 수 있다.By Equation 18 above, a stacked observation matrix

and the stacked residual vector

can be configured as in Equation 19 below.

[Equation 19]

일 수 있다. 즉,

는 알려진 3D 포인트의 수를 나타낸다.In Equation 19,

can be in other words,

denotes the number of known 3D points.

In Equations 18 and 19, the subscript K may be cited from the first letter of the word 'known (known)'.

At this time, in contrast to the case in which the front element of the matrix (matrix) of H _U in Equation 15 is indicated by the J _M value, the front element in the matrix (matrix) of H _k in Equation 19 corresponding thereto is marked as 0, In the third part, since the error model for a known point is described, it is not necessary to know the corresponding value, so it may be indicated as 0.

In the above description, the first part can be said to describe a 2D-3D relationship in one time epoch among two time epochs (viewpoints). The second part is a relationship between 3D-3D (or between 2D-2D), and in particular, based on an unknown 3D point, describes the relative positional relationship of 3D points in which viewpoints that vary depending on viewpoints are considered (ie, unknown 3D points). It can be said that it describes the relationship between non-information and 2D-2D correspondence). The third part represents how the known 3D point is expressed based on the known 3D point, and what kind of relation does the point have with the information to be estimated by the device 10 (ie, the estimated target camera posture, the estimated target 3D coordinates) It can be said that this explains the existence of this. That is, the third part can be said to describe the relationship between the information (unknown information, unknown information) to be estimated by the apparatus 10 among the 2D-3D relationship and the known 3D point.

Fourth, with respect to the error model related to the fourth part, the description of the error model of the full formulation is as follows. That is, the description of the error models of the entire 2D-3D and 3D-3D is as follows. In other words, the entire formulation (the overall formulation of the proposed method) considered in the present device 10 will be described below.

, 전체 관측 행렬(full observation matrix)

및 전체 상태 벡터(full state vector)

는 아래 식 20과 같이 관련이 있을 수 있다.Using Equation 15 and Equation 19 above, a full indirect measurement vector

, the full observation matrix

and full state vector

can be related as in Equation 20 below.

[Equation 20]

Correspondingly, their estimates can be expressed as Equation 21 below in the C -frame. In Equation 20, Z may indicate how the measurement values are combined.

[Equation 21]

An iterative weighted least squares method, also called a Gauss-Newton method, may be applied based on Equation 22 below.

[Equation 22]

일 수 있다. 즉,

는 픽셀 노이즈를 나타낸다. 또한,

는 추정오차를 갖는 추정된 전체 상태 벡터,

는 추정된 전체 관측 행렬,

는 추정된 전체 간접 측정 벡터를 의미할 수 있다.here,

can be in other words,

represents pixel noise. In addition,

is the estimated full state vector with estimation error,

is the estimated total observation matrix,

may mean the estimated total indirect measurement vector.

는

의 노름(norm)이 수렴(converges)할 때까지 업데이트될 수 있다. 추정 변환 벡터(estimated translation vector)

와 추정 회전 행렬(estimated rotation matrix)

의 초기 값(initial values)은 각각 3x1 제로 벡터(zero vector, 영벡터)와 3x3 단위 행렬(identity matrix)로 설정될 수 있다. 초기 자세 오차(initial attitude error)

는 3x1 제로 벡터(zero vector, 영벡터)로 설정될 수 있다.In equation 22,

Is

may be updated until the norm of . estimated translation vector

and the estimated rotation matrix

Initial values of may be set to a 3x1 zero vector and a 3x3 identity matrix, respectively. initial attitude error

may be set as a 3x1 zero vector (zero vector).

Equation 22 above may mean a cost function defined (set, generated) by the device 10 . This cost function may be defined based on Equation 7 described above. The estimator 13 of the apparatus 10 may simultaneously estimate the camera posture and the 3D coordinates (3D point coordinates) of the unknown object by minimizing Equation 22.

The present device 10 may simultaneously estimate the camera posture and unknown 3D coordinates based on the error model developed by the proposed method described in the above-described first to fourth order.

Hereinafter, it will be described that the proposed method of the apparatus 10 can be converted to a triangulation method or a PnP method (ie, a camera posture estimation method) according to the availability of measurement. . That is, the apparatus 10 may be switched to a camera posture estimation mode, which is a triangulation mode or a PnP mode, depending on the availability of measurement.

First, the conversion to the triangulation method (mode) of the apparatus 10 will be described as follows.

If only 2D-2D correspondences are possible, the proposed method of the apparatus 10 may be converted into an Iterative Least Square-based triangulation algorithm.

는

에서 카메라 자세에 해당하는 야코비 행렬(Jacobian matrix)의 열(columns)을 제거함으로써 획득될 수 있다. 간접 측정 벡터(indirect measurement vector)

는 상기 식 15의

와 동일할 수 있다. For triangulation mode, you need to know the camera pose in advance. observation matrix

Is

It can be obtained by removing columns of the Jacobian matrix corresponding to the camera posture in . indirect measurement vector

is in the above formula 15

may be the same as

는 상기 식 21에 표시된 전체 상태(full states) 중에서 오직 알려지지 않은 3D 포인트(unknown 3D points)만 포함되도록 구성될 수 있다. 아래 식 23은 이 경우의 관측 행렬(observation matrix), 간접 측정 벡터(indirect measurement vector) 및 상태 벡터(state vector)를 요약한 것이라 할 수 있다.In this case, the state vector

may be configured to include only unknown 3D points among the full states shown in Equation 21 above. Equation 23 below can be said to be a summary of an observation matrix, an indirect measurement vector, and a state vector in this case.

[Equation 23]

일 수 있다. 즉,

은 알려지지 않은 3D 포인트의 수를 의미할 수 있다.here,

can be in other words,

may mean the number of unknown 3D points.

Based on Equation 23 above, the unknown 3D coordinates for the C -frame at the ( k )-th time epoch can be estimated by applying the least squares scheme.

Next, switching to the PnP method (ie, the camera posture estimation method and mode) of the present device 10 will be described as follows.

If only 2D-3D correspondences are possible, the proposed method of the apparatus 10 may be converted into a camera pose estimation algorithm like a conventional PnP algorithm. In this mode, the 3D point coordinates are already known, so there is no need to estimate them.

는

의 제로 행렬 부분(zero matrix parts)을 제거함으로써 획득될 수 있다. 간접 측정 벡터(indirect measurement vector)

는

와 동일하며, 상태 벡터(state vector)

는 상기 식 21에서 표시된 전체 상태(full states) 중에서 오직 카메라 자세 파라미터만 포함되도록 구성될 수 있다. 아래 식 24는 이 경우의 관측 행렬(observation matrix), 간접 측정 벡터(indirect measurement vector) 및 상태 벡터(state vector)를 요약한 것이라 할 수 있다.In this case, the observation matrix

Is

can be obtained by removing the zero matrix parts of indirect measurement vector

Is

Same as, state vector (state vector)

may be configured to include only the camera posture parameter among the full states indicated in Equation 21 above. Equation 24 below can be said to be a summary of an observation matrix, an indirect measurement vector, and a state vector in this case.

[Equation 24]

일 수 있다. 즉,

는 알려진 3D 포인트의 수를 의미할 수 있다.here,

can be in other words,

may mean the number of known 3D points.

The camera pose (ie, the camera pose and orientation) for the C -frame at the ( k −1) th time epoch can be estimated by applying the least squares scheme to Equation 24 above.

Meanwhile, in the above description, a description of the rotation error matrix derivable through Equations 25 to 28 is as follows. That is, the rotation error matrix between camera frames using navigation frames will be briefly described as follows.

According to Equation 2 described above, an estimate of the rotation matrix between camera frames in two successive epochs can be described as Equation 25 below.

[Equation 25]

는 회전 오차 행렬(rotation error matrix)을 나타낸다.here,

denotes a rotation error matrix.

라는 용어(term)가 추가되어 있음을 확인할 수 있다. 두 개의 다른 시간 에포크(two different time epochs)에서 카메라 프레임이 있기 때문에, C-프레임과 N-프레임 사이의 두 회전 행렬(two rotation matrices)도 고려되어야 한다. 상기 식 25를 마련하고 2차 항(second-order terms)을 무시하면, 상기 식 25는 아래 식 26과 같이 표현될 수 있다.Unlike Equation 2 above, Equation 25 has

It can be seen that the term has been added. Since there are camera frames at two different time epochs, the two rotation matrices between C -frames and N -frames must also be considered. By preparing Equation 25 and ignoring second-order terms, Equation 25 can be expressed as Equation 26 below.

[Equation 26]

It can be seen that there is no significant difference between the two N -frames, assuming that the distance between the camera positions in two consecutive epochs is much smaller than the radius of the earth. In this case, the following relationship can be maintained by Equation 27 below.

[Equation 27]

는 아래 식 28에 의해 획득될 수 있다.Then, substituting Equation 4 and Equation 5 into Equation 27,

can be obtained by Equation 28 below.

[Equation 28]

일 수 있다.here,

can be

Hereinafter, a block diagram of the apparatus 10 will be described with reference to FIG. 3 based on the above description.

2 and 3 , the apparatus 10 relates to a simultaneous estimation apparatus for simultaneously estimating a camera posture and 3D coordinates of a target object. Here, the target object may mean an object (object) having coordinates of an unknown 3D point. That is, the unknown 3D point corresponding to the 3D coordinate of the target object estimated by the apparatus 10 may mean, for example, 3D points indicated by P ₄ and P ₅ with respect to the capital letter P in FIG. 2 . In addition, the camera posture estimated by the apparatus 10 may refer to a rotation matrix denoted by R and a translation vector denoted by t in FIG. 2 .

The apparatus 10 may include an input unit 11 , a setting unit 12 , and an estimation unit 13 .

The input unit 11 may receive coordinates of a known 3D reference point (a reference 3D point) of the reference object. Here, the known 3D reference points may mean already known (known, known) 3D reference points (3D reference points). The known 3D reference point to which the input unit 11 is input may mean, for example, 3D points indicated by P ₁ , P ₂ , and P ₃ in relation to the capital letter P in FIG. 2 .

)가 고려(반영)되어 있는 오차가 존재하는 추정된 3D 기준점의 좌표(즉,

)를 포함할 수 있다. 즉, 입력받은 알려진 3D 기준점의 좌표는 오차가 있는 부정확한 3D 기준점의 좌표를 포함할 수 있다.The coordinates of the known 3D reference point input to the input unit 11 are the estimation error of the actual 3D point of the reference object (that is, in Equation 1 above).

) is the coordinates of the estimated 3D reference point where the error is considered (reflected) (i.e.,

) may be included. That is, the received coordinates of the known 3D reference point may include the coordinates of the inaccurate 3D reference point having an error.

The setting unit 12 is configured with a first projected 2D point (ie, on the first image) projected on the first image acquired at the first time epoch through the camera with respect to the known 3D reference point input from the input unit 11 . A first projected 2D point corresponding to a known 3D reference point projected on . In addition, when the second image is acquired at a second time epoch different from the first time epoch due to a change (updating) of the camera posture, the setting unit 12 is a second projected 2D point projected on the second image. (ie, a second projected 2D point corresponding to a known 3D reference point projected on the second image).

In the present specification, the setting unit 12 may be referred to as, for example, a calculation unit or the like. According to this, for example, when the second image is acquired at a second time epoch different from the first time epoch due to a change (updating) of the camera posture, the setting unit 12 (calculation unit) may set the second image projected on the second image. Two projected 2D points (ie, a second projected 2D point corresponding to a known 3D reference point projected on the second image) may be computed. That is, the setting unit 12 (the calculating unit) may calculate the location of the 2D point projected on the image corresponding to the known 3D reference point (the second projected 2D point) on the obtained second image. Hereinafter, even if omitted below, the description of the second image may be applied to the description of the first image in the same or similar manner.

Here, the first time epoch and the second time epoch may be two consecutive time epochs (ie, two consecutive time epochs). The first time epoch may mean, for example, the previous ( k −1)-th time epoch described above, and the second time epoch may mean the previous ( k )-th time epoch described above. As mentioned above, the temporal epoch may be referred to as a time base, a viewpoint (a point in a viewing direction), and the like.

Accordingly, the first image in the first time epoch means, for example, a C -frame defined in the ( k −1) th time epoch, which may be displayed as C ( k −1) described above. The second image in the second time epoch means, for example, a C -frame defined in the ( k )-th time epoch, which may be displayed as above-described C ( k ).

로 표시된 것을 의미할 수 있다. 제2 투영된 2D 포인트는 제2 이미지(이미지 평면, 평면 이미지) 상에 표시되는 2D 포인트를 의미하는 것으로서, 이는 일예로 도 2에 소문자 p와 관련하여

로 표시된 것을 의미할 수 있다.The first projected 2D point is not shown in FIG. 2 as an example and is omitted. However, the first projected 2D point refers to a 2D point displayed on the first image (image plane, flat image).

It may mean indicated by The second projected 2D point refers to a 2D point displayed on the second image (image plane, plane image), which is an example in relation to the lowercase letter p in FIG. 2 .

It may mean indicated by

The first projected 2D point and the second projected 2D point can be set by using a pinhole camera model as an example as described above, and in particular, detection of any feature, such as the known corner detector described above. (feature detection) or by obtaining using extraction algorithms.

When a second image is acquired at a second time epoch different from the first time epoch by changing (updating) the camera's posture, the estimator 13 responds to the change in the camera's posture. Two images (i.e., the first image and the second image) in which the relation information between the projected 2D point and the 3D point on the image obtained based on the information and the change in the camera's posture (i.e., R, t in Fig. 2) are considered Using a cost function (single cost function) defined in consideration of information about a relative positional relationship with respect to a known 3D reference point on Simultaneous estimation of the estimated target 3D coordinates (ie, the coordinates of the unknown 3D point of the unknown object) that are the coordinates of the unknown 3D point of the target object in the second image corresponding to may be performed.

Here, the relationship information (hereinafter simply referred to as relationship information) between the projected 2D point and the 3D point on the image obtained based on the information set in the setting unit 12 is the two images (that is, the first image and the second image). ) may refer to relationship information between a 2D point and a 3D point projected on any one image based on any one image. Such relationship information may be referred to as relationship information related to what is referred to as 2D-3D correspondence in the above description.

In this case, a known 3D point and an unknown 3D point may be considered as 3D points considered in the relationship information. The known 3D point may mean a 3D point of the reference object (ie, a known 3D reference point of the reference object), and the unknown 3D point may mean a 3D point of the target object (ie, an unknown 3D point of the target object).

Here, when an unknown 3D point is considered as the 3D point, the relationship information may refer to relationship information derived based on the description of the above-described first part-related error model. Accordingly, the relationship information may mean information including the above-described Equations 6 to 8, and consequently may mean information set (defined) as in Equation 8 above. For example, the relationship information may refer to relationship information between p ₄ and p ₅ for a lowercase letter p and P ₄ , P ₅ for an uppercase letter P in FIG. 2 . Such relationship information may mean relationship information (ie, relationship information about 2D-3D correspondence) about a 2D-3D error model with respect to unknown 3D points.

If a 3D point known as a 3D point is considered, the relationship information may mean relationship information derived based on the description of the third part-related error model. Accordingly, the relationship information may mean information including the above-described Equations 16 to 19, and consequently may mean information set (defined) as in Equation 19 above. Exemplarily, the relationship information may refer to relationship information between p ₁ , p ₂ , and p ₃ for a lowercase p in FIG. 2 and P ₁ , P ₂ , and P ₃ for a capital P in FIG. 2 . Such relationship information may refer to relationship information about a 2D-3D error model with respect to known 3D points (ie, relationship information about a 2D-3D correspondence).

In addition, the information on the relative positional relationship considered by the estimator 13 (hereinafter simply referred to as information on the positional relationship) includes the second projected 2D point corresponding to the known 2D reference point projected on the second image and the second projected 2D point. It may include relationship information between the first projected 2D points projected on one image.

In this case, the information on the positional relationship may be referred to as relationship information related to what is referred to as 2D-2D correspondence or 3D-3D in the above description.

The information on the positional relationship may refer to relationship information derived based on the description of the second part-related error model. Accordingly, the information on the positional relationship may mean information including Equations 9 to 15 described above, and consequently may mean information set (defined) as in Equation 15 above. Exemplarily, the information on the positional relationship may refer to relationship information between p ₄ and p ₅ with respect to the lowercase p that is respectively displayed on the two images in FIG. 2 .

The relationship information and the positional relationship information considered by the estimator 13 may be information (relationship information) set based on Euclidean geometry. In addition, all equations considered in the present device 10 may be set based on Euclidean geometry (ie, may be equations expressed based on Euclidean coordinates).

The estimator 13 uses a cost function defined in consideration of the relationship information and the information on the positional relationship to estimate the camera posture, which is the changed posture of the camera corresponding to the second time epoch, and the second time epoch. Estimation target 3D coordinates that are coordinates of an unknown 3D point of the target object in the second image may be simultaneously estimated. The estimator 13 may simultaneously estimate the estimated camera posture and the estimated 3D coordinates using a single cost function (ie, a single cost function).

The cost function may be set to satisfy Equation 22 above. The estimator 13 may pre-define a cost function, and for this purpose, the above-mentioned Equations 20 to 22 may be developed.

Referring to Equations 20 to 22, the cost function is an indirect measurement value ( This may be defined as a relation between the entire indirect measurement vector defined to be related to Z in Equation 8) and the entire observation matrix indicating the connection relationship between the indirect measurement value and the overall state vector.

, 및 회전 행렬(즉, 도 2에서 추정하고자 하는 값들 중 R로 표시된 값)과 관련된 증분 자세 오차인

가 포함될 수 있다. 또한, 추정 대상 3D 좌표 관련 3D 좌표 추정 오차는

로 표현되는 파라미터를 의미할 수 있다. At this time, the estimation target camera posture-related state parameter includes a transformation vector (ie, a value indicated by t among values to be estimated in FIG. 2 ).

, and the incremental posture error associated with the rotation matrix (that is, the value indicated by R among the values to be estimated in FIG. 2 ).

may be included. In addition, the 3D coordinate estimation error related to the estimated target 3D coordinate is

It may mean a parameter expressed as .

The estimator 13 may perform simultaneous estimation by minimizing the cost function expressed by Equation 22 above. In particular, the estimator 13 calculates the minimum error value among the error values of the entire state vector, which is a value of a cost function calculated by multiplying the entire indirect measurement vector and the entire observation matrix, based on the total state vector for calculating the estimated camera posture and 3D coordinates to be estimated can be simultaneously estimated.

That is, the estimator 13 can finally estimate (derive to be an optimal estimated value) the value of the estimated camera posture-related state parameter corresponding to the overall state vector for calculating the minimum error value as the estimated camera posture. have. In addition, the estimator 13 may finally estimate (derive to be an optimal estimated value) an estimation target 3D coordinate-related 3D coordinate estimation error value corresponding to the entire state vector for calculating the minimum error value as the estimation target 3D coordinate. can Since the description of the expression related to the cost function has been described in detail above, it will be omitted below.

The estimator 13 may estimate the position and direction of the camera as an estimation target camera posture through simultaneous estimation. In particular, the estimator 13 may estimate a rotation matrix denoted by R and a translation vector denoted by t in FIG. 2 as an estimation target camera posture. Also, the estimator 13 may estimate the coordinates of the 3D point of the unknown object as the estimation target 3D coordinates through simultaneous estimation.

As can be seen from the experimental results to be described later, through the simultaneous estimation of the estimated camera posture and the estimated target 3D coordinates, the present device 10 compares with the prior art in which only the camera posture is estimated or only 3D coordinates are estimated. Simultaneous estimation can be performed with higher accuracy. In addition, the apparatus 10 may derive the estimated value of the camera posture and the estimated coordinates of the unknown 3D point as more accurate values at a faster operation speed than those of the related art through simultaneous estimation.

Hereinafter, the performance evaluation result of the apparatus 10 (ie, the performance evaluation result of the proposed method) will be described. Hereinafter, an experiment for evaluating the performance of the apparatus 10 will be referred to as a present experiment.

In order to evaluate the performance of the proposed method, two types of experiments were performed by utilizing a synthetic dataset and a real dataset, respectively. This experiment is intended to quantitatively evaluate sensitivity to measurement noise, coordinate uncertainty, and dependence on baseline and depth. can

In this experiment, the proposed method was compared with the following approaches frequently used in recent SfM software. For camera pose estimation, MSAC-P3P, a Gaussian version of EPnP, and a robust version of DLS were compared. This method can be classified as a PnP algorithm. MSAC-P3P is a combination of MSAC and P3P for robust estimation of noise and outliers recently provided in the MATLAB Vision toolbox.

For triangulation, DLT was used to estimate unknown 3D coordinates after the camera pose was acquired by the aforementioned PnP algorithm. Throughout this experiment, it is assumed that camera calibration parameters are given in advance in this experiment. Hereinafter, the results of synthetic data evaluation will be described first, and then the results of real data evaluation will be described.

The description of the synthetic data evaluation results is as follows.

In the synthetic data experiment, a true camera trajectory was created for the first time. Then, at the initial time epoch, the 3D points for the C -frames are randomized from the volume [3 m, 7 m] × [-1 m, 1 m] × [-1 m, 1 m]. sampled randomly. The camera considered in this experiment has an image size of 1280 × 960 pixels, a focal length of 1013, and a virtual camera whose image center is a principal point. ) was assumed to be

2D-3D and 2D-2D correspondences were created by projecting all the points onto two camera image planes. For each PnP method, four 2D-3D correspondences were provided for camera pose estimation. However, DLT was given several tens of 2D-2D correspondences to estimate various unknown 3D coordinates.

In the proposed method, only one 2D-3D correspondence and the same amount of 2D-2D correspondence used in the DLT scheme can be given.

All plots (plots) presented in that part (i.e., descriptions of synthetic data evaluation results) were used to obtain meaningful accuracy statistics under different experiment conditions described below. , can be independently generated by running 100 trials with different random seed numbers.

The evaluation test results of sensitivity to measurement noise are as follows.

In this experiment, the accuracy of four methods, including the proposed method, was compared in the presence of measurement noise. Gaussian pixel noise is added to every projected 2D point. However, it was assumed that the coordinates of the 3D reference points were ideally accurate. After running each method, the accuracy of rotational angles, translation vectors and unknown 3D coordinates were compared with ground-truth.

4 is an experimental result of the present application (ie, an experimental result for evaluating the performance of the present device), showing RMSEs (Root Mean Square Errors) of all trials that increase the standard deviation of pixel noise. the drawing shown.

In particular, FIG. 4 is a diagram comparing camera position, camera rotation, and estimation errors of unknown 3D coordinates using different methods having different Gaussian pixel noise values. At this time, in FIG. 4 , the left figure shows the RMSE of the camera position, the middle figure shows the RMSE of camera rotation, and the right figure shows the RMSE of unknown 3D coordinates.

Referring to FIG. 4 , it can be confirmed that the proposed method is not resilient to pixel noise compared to the MSAC-P3P and DLS methods, but has better results than the EPNP method. However, it can be seen that the results are almost similar when the pixel noise is less than 0.5.

The evaluation experimental results of the dependence on the baseline and the depth are as follows.

)의 픽셀 노이즈 뿐만 아니라 실세계(real world)에서 3D 기준점 (

)의 좌표 불확실성(coordinate uncertainty)도 고려하였다. 따라서, 본 실험에서는 일예로 3D 기준점에 불확실성이 없는 경우와 3D 기준점에 불확실성이 있는 경우에 대한 두가지 유형의 실험이 수행되었다.In this experiment, the accuracy of the proposed method was compared with the existing methods (conventional methods) given different depths and baselines. In addition, in this experiment, a 2D point projected from the image plane (

) as well as the 3D reference point (

) was also considered. Therefore, in this experiment, as an example, two types of experiments were performed: a case in which there is no uncertainty in the 3D reference point and a case in which there is uncertainty in the 3D reference point.

와

)는 각각 두 방향(two directions)의 픽셀 노이즈와 세 방향(three directions)의 좌표 불확실성을 추가하는 데에 사용되었다. 시뮬레이션(Simulations)은 일예로 0.44m ~ 1.1m 범위의 4개의 다른 기준선과 3m ~7m 범위의 5개의 다른 깊이로 구성된 20개의 다른 조건(different conditions)에 대해 수행되었다.To determine individual dependences on different depths and baselines, constant standard deviations (

Wow

) were used to add pixel noise in two directions and coordinate uncertainty in three directions, respectively. Simulations were performed for 20 different conditions consisting of, for example, 4 different baselines ranging from 0.44 m to 1.1 m and 5 different depths ranging from 3 m to 7 m.

가 0.5 픽셀로 주어진 경우일 때의 RMSEs 비교 결과를 나타낸다.5 is an experimental result of the present application (ie, an experimental result for evaluating the performance of the present device), and is a view showing comparison results of RMSEs of camera posture, camera rotation, and coordinates of an unknown 3D point. In particular, in FIG. 5 only

It shows the result of comparison of RMSEs in the case where is given as 0.5 pixels.

In other words, Figure 5 shows the camera pose, camera rotation, and coordinates of unknown 3D points by different methods along different baselines and depths, at constant Gaussian pixel noise (eg 0.5 pixels). It is a diagram comparing estimation errors for In FIG. 5 , the upper three figures show a case where the depth is fixed at 7 m and the reference line is changed, and the lower three figures in FIG. 5 show a case where the base line is fixed at 1.1 m and the depth is changed.

That is, referring to FIG. 5 , in the top 3 plots of FIG. 5 (ie, in the top 3 graphs), the depth is fixed to 7 m, and the reference line may be gradually changed. In contrast, in the lower three configurations of FIG. 5 (ie, in the lower three graphs), the baseline is fixed at 1.1 m, and the depth can be varied.

As can be seen in all configurations (all graphs) of FIG. 5 , the performance of the EPNP is generally not good, but may be satisfactory when the depth is short.

It can be confirmed that the accuracy of the proposed method is better than that of DLS and MSAC-P3P. In particular, it can be seen that the results of DLS and the results of the proposed method are almost similar. This is because, for example, the DLS method is also based on the least squares approach.

On the other hand, it can be confirmed that the accuracy of MSAC-P3P, which is designed to be resistant to noise, provides unsatisfactory results compared to DLS and the proposed method. As can be seen in Fig. 5, if the pixel noise is not large, both the proposed method and DLS give more accurate estimates than MSAC-P3P when considering the long baseline and distant depth. You can confirm that you can provide it.

의 상수값(constant value)을 제외하고 도 5에서 사용한 것과 동일한 환경에서 얻어진 결과 그래프를 나타낸다.6 is a diagram for a known 3D reference point.

A graph of the results obtained in the same environment as used in FIG. 5 is shown except for the constant value of .

That is, FIG. 6 is an experimental result of the present application (ie, an experimental result for evaluating the performance of the apparatus), and shows constant Gaussian pixel noise (eg, 0.5 pixel) and uncertainty of constant 3D reference coordinates. (0.01 m), a diagram comparing the estimation errors for the coordinates of the camera pose, camera rotation, and unknown 3D point by different methods along different baselines and depths. In FIG. 6 , the upper three figures show a case where the depth is fixed at 7 m and the reference line is changed, and the lower three figures in FIG. 6 show a case where the base line is fixed at 1.1 m and the depth is changed.

가 0.01m 인 것을 제외하고, 도 6에서는 도 5의 생성시 고려된 조건과 동일한 조건(same conditions)으로, 카메라 자세, 카메라 회전 및 알려지지 않은 3D 포인트의 좌표 각각의 RMSEs가 비교될 수 있다.In other words,

RMSEs of each of the coordinates of the camera posture, camera rotation, and unknown 3D point may be compared in FIG. 6 under the same conditions as the conditions considered during generation of FIG.

Referring to FIG. 6 , it can be seen that the overall accuracy is decreased in all four methods when compared with FIG. 5 . Nevertheless, it can be confirmed that the proposed method shows better results than all other methods in estimating camera position, camera rotation and unknown 3D coordinates.

- However, when compared with the MSAC-P3P and DLS results of FIG. 5 , it can be confirmed that the accuracy of the MSAC-P3P is superior to the DLS of FIG. 4 . This result can be considered to be obtained because MSAC-P3P finds the best 3 2D-3D correspondence among the candidates. Therefore, it can be confirmed that the result of MSAC-P3P is more accurate than that of DLS.

As shown in FIGS. 5 and 6 , it can be confirmed that the RMSEs tend to decrease as the baseline increases or the depth decreases by all methods. In particular, since the proposed method inherits the properties of multi-baseline techniques such that the depth accuracy increases as the baseline length increases, it is distinctly robust when considering the uncertainty about the known 3D coordinates of the reference point. robustness.).

가 0.5 픽셀인 조건에 대하여 아래 [표 1]에 요약되어 있고,

가 0.5 픽셀이고

가 0.01m인 조건에 대하여 아래 표 2에 요약되어 있다.The overall result of the simulation is

It is summarized in [Table 1] below for the condition that is 0.5 pixel,

is 0.5 pixels

It is summarized in Table 2 below for the condition where is 0.01 m.

가 0.5 픽셀일 때, 상수 측정 노이즈에서 기준선 및 깊이에 따른 추정 오차를 비교한 시뮬레이션 결과를 나타낸다. 아래 표 2는

가 0.5 픽셀이고

가 0.01m일 때, 상수 측정 노이즈 및 3차원 기준 좌표 불확실성 하에서 다양한 기준선 및 깊이에 따른 추정 오차를 비교한 시뮬레이션 결과를 나타낸다.That is, Table 1 below is

When is 0.5 pixel, the simulation result comparing the estimation error according to the reference line and the depth in the constant measurement noise is shown. Table 2 below

is 0.5 pixels

When is 0.01 m, simulation results comparing estimation errors according to various baselines and depths under constant measurement noise and 3D reference coordinate uncertainty are shown.

In this case, in the table, MSAC-P3P and 2EPnP + GN may be abbreviated as MP3P and EPnP, respectively.

[Table 1]

[Table 2]

Hereinafter, the results of real data evaluation will be described.

In order to validate the performance of the proposed method in real world scenarios, a data set was collected using a stereo camera (eg, BumbleXB3) in this experiment. The sensor may output a rectified global shutter VGA (640×80) stereo image at 15 Hz having a baseline of 24 cm. This experiment was focused on evaluating the proposed method given unknown 3D points at different baselines and depths. In this experiment, camera calibration parameters were identified before the experiment by using the stereo camera API.

The data set may be composed of images, ground-truth values of camera poses, and target point coordinates. Images were captured by moving the camera at various baselines and depths. For evaluation, only left camera images were used as an example. Experiments were performed 16 times considering 4 different baselines and 4 depths.

In addition, a large chessboard having a width of 0.6 m and a height of 0.9 m was used as a target object. It may contain 36 small blocks measuring 0.1 m × 0.15 m. The 3D coordinates of the target points were calculated for ground-truth by stereo images with a baseline of 24 cm.

Since the accuracy of depth information is a dominant factor in coordinate estimation, in this experiment, depth was measured with a BOSCH GLM 50C, a precision range finder. The 3D coordinates of the reference point were calculated for the C -frame at the initial time epoch.

7 is an experimental result of the present application (ie, an experimental result for evaluating the performance of the present device), depicting an experimental environment in which matched 2D points are displayed in two consecutive images. it is one drawing

That is, FIG. 7 shows an example of an actual experimental environment. In this experiment, only the left image was used among the 3 cameras of BumblebeeXB3. A chessboard has a plane surface. 7A shows an example of a stereo camera and a chessboard provided at intervals of a constant depth. 7 (b) shows an illustration of the experiment, and a red arrow indicates that the camera and the object move in that direction. The object from the camera may be set at regular intervals of a constant depth. 7( c ) shows an example of Matched 2D points between two consecutive images.

Referring to FIG. 7 , a corner detector may be used to extract a 2D point from two consecutive images based on a sum of squared difference (SSD). In this experiment, known 3D reference points and their projected 2D points were randomly selected to create a 2D–3D correspondence. And, the rest is set to obtain 2D-2D correspondence.

8 is an experimental result of the present application (ie, an experimental result for evaluating the performance of the present device), and is a diagram illustrating a comparative example of RMSEs of a camera position, a camera rotation, and an unknown 3D point. 8 shows the results generated by the four methods.

That is, FIG. 8 shows the results of comparing the estimation errors of the camera position, camera rotation, and unknown feature coordinates by other methods based on actual data according to different reference lines and depths. In FIG. 8 , the upper three diagrams show a case where the depth is fixed to 5 m when the reference line is changed from 0.275 m to 1.1 m. The lower three figures in FIG. 8 show a case in which the reference line is fixed to 1.1 m when the depth is changed from 2 m to 5 m.

Referring to FIG. 8 , as shown in the three lower drawings of FIG. 8 having different depths, it can be confirmed that the estimation accuracy of the camera posture is increased by the proposed method compared to other methods. In particular, it can be confirmed that the proposed method shows the best accuracy when the depth is increased to 4 or 5 m.

Table 3 below summarizes the overall results of the actual data experiments. That is, Table 3 shows the results of comparing estimation errors for different baselines and depths based on actual data.

[Table 3]

Referring to Table 3, it can be seen that the accuracy of the other methods is significantly lowered compared to the simulation results by the proposed method. That is, according to Table 3, it can be confirmed that the proposed method shows better accuracy than other methods.

Since the target object used in this experiment is an ane surface, it may affect the accuracy of most PnP algorithms that are sensitive to planar objects. On the other hand, the proposed method shows relatively little degradation of accuracy for real data. In summary, according to the above experimental results, it can be confirmed that the proposed method shows much better accuracy than other methods when estimating the position and rotation of the camera.

As described above, the present apparatus 10 proposes a novel and versatile approach that can simultaneously estimate the camera pose and coordinates of an unknown 3D point by utilizing 2D-2D and 2D-3D correspondence.

The proposed method of the present apparatus 10 has the advantage of being very versatile because it can be reduced to two existing modes, the PnP method and the triangulation method, depending on the availability of measurements. can

Through the above-described experiment of the present application, the proposed method was verified for both synthetic data and real data. According to the experimental results described above, compared with the three representative methods (MSAC-P3P, EPnP and DLS) recently used in SfM software, the proposed method provides a more accurate estimate of the initially unknown 3D point coordinates and camera posture. It can be demonstrated that it can provide

In addition, according to the experimental results described above, the proposed method provides a more stable estimate even when observing planer objects or remote objects given moderate uncertainties about the coordinates of the 3D reference point. It can be confirmed that the .

In other words, the proposed method is an object (planar) that is an object to be observed (observation object) even in a situation in which an error exists in the coordinates of the 3D reference point (that is, in a situation in which the coordinates of the 3D reference point are inaccurate). An estimate of the three-dimensional coordinates of an object or a remote object) and an estimate of a camera posture may be estimated and provided as more accurate values.

The device 10 can reduce the sensitivity to measurement noise at the 3D reference point by using a hybrid correspondence, and improve the depth accuracy as the reference line becomes longer (i.e., the longer the reference point, the higher the depth accuracy can be ). In addition, the proposed method can minimize the image re-projection error by applying an iterative least square method.

In addition, the proposed method can be reduced to the conventional PnP (Perspective-n-Point) method and triangulation method utilized in the most recent SfM (Structure-from-Motion) approach depending on the availability of measurements.

In addition, the present application has demonstrated that the proposed method is effective in estimating the camera pose and unknown 3D coordinates simultaneously through both synthetic data and real data through the above-described experiments.

As described above, the device 10 estimates the correct camera posture even in a situation in which an error exists in the coordinates of the 3D reference point (that is, the situation in which the coordinates of the 3D reference point is inaccurate), and as well as However, it is possible to simultaneously estimate the camera posture and the 3D coordinates of the target object, which enable simultaneous estimation of the 3D coordinates of the unknown target object.

Conventionally, the PnP method as shown in the lower diagram of FIG. 1 is based on the 3D coordinates (coordinates of the 3D point) of a known (known, known) object (ie, assuming that the exact 3D coordinates of the object are known) the camera's posture technology for estimating

On the other hand, in the conventional triangulation method (triangulation method) as shown in the upper diagram of FIG. 1, when the posture of the camera is known (for example, when the posture of the camera is already known through estimation by the PnP method, etc.), the known camera Based on the pose (i.e., assuming that the pose of the camera is known), an object with unknown 3D coordinates (i.e., an object with unknown 3D coordinates) is viewed through the camera from two viewpoints. Coordinates on the image of the object projected onto the two images of Correspondence) refers to a technique for estimating the 3D coordinates of the object.

According to this, conventionally, when estimating the camera posture or estimating unknown 3D coordinates, two separate techniques (ie, the PnP method and the triangulation method) had to be independently used. That is, conventionally, only camera posture estimation is possible with the PnP method, and only unknown 3D coordinates can be estimated with the triangulation method, and there is no technology for estimating them simultaneously with one estimator.

Illustratively, estimating the pose of the camera and the unknown 3D coordinates together in the prior art is, for example, primarily estimating the pose of the camera based on the known 3D coordinates (coordinates of the known 3D point) using the PnP method, Afterwards, it could be done by estimating unknown 3D coordinates using triangulation secondarily (ie, sequentially performing the PnP method and triangulation method) based on the first estimated camera posture.

However, in the conventional PnP method, a camera posture is estimated under the assumption that known 3D coordinates (ie, 3D reference point) are accurate coordinates. However, in reality, it can be said that it is difficult to consistently provide 3D coordinates (ie, coordinates of a 3D reference point) as accurate coordinates. That is, most conventional methods for estimating the camera posture, including the PnP method, estimate the camera posture under the assumption that the known 3D coordinates considered as the 3D reference point are accurate (that is, assuming that they are accurate 3D coordinates). If there is an error in the known 3D coordinates considered as the reference point (ie, the known 3D coordinates are inaccurate coordinates), there is a problem in that the posture of the camera cannot be accurately estimated.

According to this, in the case of estimating the camera posture and the unknown 3D coordinates together with the prior art, if there is an error in the known 3D coordinates, an error occurs in the camera posture estimated based on this as an example (i.e. , the estimated camera pose has an inaccurate value), and furthermore, there is a problem that an error occurs in 3D coordinates (ie, unknown 3D coordinates) estimated based on the estimated camera pose. That is, in the prior art, as the error of the known 3D coordinates increases, the error of the estimated camera posture also increases, and further, the error of the 3D coordinates (unknown 3D coordinates) estimated based on it increases. have.

That is, in the prior art, as the two techniques are sequentially performed, the error becomes larger and larger as the two techniques are sequentially performed, and the accuracy of both the estimated camera posture and the unknown 3D coordinates is significantly lowered. there is

In order to solve this problem, the present application combines the camera posture estimation technique and the 3D coordinate estimation technique, which have been separately performed in the prior art, into one single process, thereby estimating the camera posture and It is possible to provide the apparatus 10, which is a single estimator that allows the 3D coordinate estimation to be made simultaneously (at once). By providing the apparatus 10 of the present disclosure, even if there is an error in the coordinates of the 3D reference point, it is possible to more accurately estimate the camera posture and the 3D coordinates of the unknown target object.

This device 10 combines two conventional techniques (that is, a PnP-based camera posture estimation technique and a triangulation-based 3D coordinate estimation technique) into one single process, so that there is an error (including noise) in the known 3D coordinates. Even if there is some error in the camera posture estimated based on this, the error is generated by the 2D-2D correspondence technique used in 3D coordinate estimation (that is, a pixel using two projected 2D points commonly projected on two images). In the unit-based technique), it can be offset to some extent, thereby enabling more accurate estimation of the camera's pose and the 3D coordinates of an unknown target object.

That is, in contrast to the known error of 3D coordinates (that is, the coordinates of the 3D reference point), two 2D points that are commonly projected on two images (ie, a point corresponding to a 2D-2D correspondence, a matching 2D point on the two images) , in other words, the common points represented by the same 2D point cloud on two images) are in pixels (that is, expressed in units of pixels), so the error is relatively much smaller than the error of 3D coordinates. In consideration of this point, the present device 10 combines the two techniques described above into one single process, so that even if there is an error in the known 3D coordinates (the coordinates of the reference point), the 2D-2D correspondence, which is a measurement that can offset the error. Since this is utilized, it is possible to provide not only the camera posture but also the 3D coordinates (particularly, the coordinates of the unknown 3D point) to be estimated at the same time more accurately compared to the prior art. The performance evaluation results described above with reference to FIGS. 5, 6, 8, etc. that the present device 10 shows a superior effect compared to the prior art (ie, more accurate camera posture and 3D coordinates are estimated) can be proved through That is, the apparatus 10 may have improved estimation accuracy compared to conventional techniques.

That is, the device 10 uses a known value (that is, the coordinates of a known 3D reference point of a reference object) and a value to which the error can be large (that is, a value that can have a much larger tolerance of error, and the camera Information to be estimated (ie, the estimated camera posture and the estimated 3D position of the camera to be estimated and the 3D to be estimated) while simultaneously optimizing the posture-related information R and t, and the three-dimensional point coordinate information) into the single cost function defined (set) by Equation 22 above coordinates) can be derived, and through this, it is possible to obtain an estimation result with high reliability compared to the prior art (that is, a more accurate estimation value can be derived). As such, it may have a superior effect compared to the prior art.

The apparatus 10 may simultaneously estimate an estimation target camera posture and estimation target 3D coordinates in a direction in which a predefined cost function (single cost function) is minimized.

In addition, in related fields such as cameras/images, conventionally, as an example, estimation based on homogeneous geometry has been made, whereas in the present application, a proposed method based on Euclidean geometry (simultaneous estimation method) It is possible to provide the apparatus 10 that performs

In addition, in providing a plurality of estimation techniques as an example, it may be considered important which technique is used and which geometry or equation is used. In this regard, the apparatus 10 provides a simultaneous estimation method (which means the proposed method) that combines the two techniques described above into one single process based on Euclidean geometry, and also the plate itself used By defining a cost function (ie, objective function) as in Equation 22 with the Iterative Least Square Method, a simultaneous estimation method combining the two techniques into one single process can be provided.

When an input is given (here, the input may mean detection of a change in the camera's posture, for example), by the relationship between H and Z expressed by Equation 22 above in the proposed method, Through the least squares method (least squares method), the estimated camera pose and the estimated target 3D coordinates (coordinates of the 3D point of an unknown object) can be simultaneously estimated.

The apparatus 10 applies the least squares method to the equation defined as Equation 22 until a value that minimizes the error is derived, so that the estimated target camera posture and the estimated target 3D coordinates (of the 3D point of an unknown object) coordinates) can be estimated simultaneously.

When the coordinates of the known 3D reference point of the reference object are given, the apparatus 10 projects on the acquired image corresponding to the coordinates of the known 3D reference point from the acquired image when image acquisition (photography) is made by the camera Since it can be known by calculating the calculated 2D point, it is possible to simultaneously estimate the estimated camera posture and the estimated target 3D coordinates (coordinates of the unknown 3D point of the object) based on the calculated projected 2D point information.

That is, the device 10 receives the coordinates of the known 3D reference point of the reference object, and then when the first image is obtained, a point corresponding to the known 3D reference point projected on the first image by the setting unit 12 ( That is, the coordinates of the first projected 2D point on the first image) may be calculated and set. In this case, the first projected 2D point or its coordinates may be calculated (obtained) using, for example, a conventional feature detection/extraction algorithm as described above. Then, when the second image is obtained by changing (updating) the posture of the camera, the device 10 sets a point corresponding to the known 3D reference point projected on the second image by the setting unit 12 (that is, the second The coordinates of the second projected 2D point on the two images may be calculated and set, and the coordinates of the projected 2D point may also be calculated (obtained) using, for example, a conventional feature detection/extraction algorithm. Thereafter, in response to the change in the camera's posture, the apparatus 10 applies the information of the projected 2D point calculated and set by the setting unit 12 in advance to the cost function set as in Equation 22 to apply the least squares method (least squares method). ), it is possible to simultaneously estimate (obtain) the estimated target camera posture and the estimated target 3D coordinates (coordinates of the 3D point of an unknown object).

In other words, the apparatus 10 relates to a technology capable of simultaneously estimating a camera posture (eg, a position and a direction) and a three-dimensional coordinate of an unknown object. The device 10 is related to Camera Pose Estimation (position/posture estimation of a camera) and Triangulation (three-dimensional coordinate estimation of an object) techniques in the existing Structure-from-Motion (SfM) system, and unlike the existing method, sequentially We propose a new methodology (the proposed method) that combines the two methods to be processed into one system (as a device).

9 is a diagram schematically illustrating a configuration of a simultaneous estimation apparatus 10 (the present apparatus) for simultaneously estimating a camera posture and 3D coordinates of a target object according to an embodiment of the present application.

Referring to FIG. 9 , the apparatus 10 provides a 3D coordinate of a known object (that is, a coordinate of a known 3D reference point of an existing object) and a 2D-3D corresponding point (ie, a 2D coordinate of the object projected onto the image). points corresponding to 2D-3D correspondence), and 2D-2D correspondence points representing the two-dimensional coordinates of an unknown object (ie the target object) projected onto the images (ie, the first image and the second image) from two viewpoints, respectively. (that is, a point corresponding to 2D-2D correspondence) can be used (applied) as an input of the proposed method. As such, the proposed method by the apparatus 10 in which two types of corresponding points (that is, a point corresponding to 2D-3D correspondence and a point corresponding to 2D-2D correspondence) are input is a camera based on an initially set coordinate system. It is possible to simultaneously estimate the relative posture (for example, the position and direction) and the three-dimensional coordinates of the newly observed object (target object). Here, the initially set coordinate system may mean, for example, a coordinate system set based on the N-frame, but is not limited thereto.

The apparatus 10 designs the three-dimensional information acquisition process of the camera posture and object, which was divided into two steps in the existing Structure-from-Motion (SfM) algorithm, with one estimator, and thus has the following four characteristics. (advantages) can be

First, the apparatus 10 relaxes a specific assumption in the existing Perspective-n-Point (PnP) technique, which is a camera pose estimation, and may have improved accuracy compared to the existing algorithm. Here, the specific assumption may mean that 3D information of a known object is accurate.

Second, the apparatus 10 may have a characteristic that the accuracy of depth information among 3D information of an object increases as the distance (baseline, baseline, baseline) between cameras increases compared to the existing algorithm.

Third, scale information is absent in the existing technique of estimating the relative position and posture of a camera by using only 2D-2D corresponding points, whereas the present apparatus 10 directly uses 2D-3D corresponding points to obtain scale information. It can have a characteristic that can be known as .

Fourth, the present device 10 has the feature that it can be freely converted to the conventional triangulation method or the PnP method according to the type of available correspondence point (ie, 2D-2D correspondence or 2D-3D correspondence). can

This device 10 is applied to fields such as virtual reality (VR) & augmented reality (AR), robots and map production, which are fields related to 3D information reconstruction of objects or surrounding environments using cameras. can be applied effectively. In addition, the apparatus 10 can be effectively applied to fields such as navigation, positioning, and robot search, which are fields for research and application development of object position and posture information.

In the existing computer vision and image processing fields, Structure-from-Motion (SfM), a technique for reconstructing 3D information of an object using a camera image, exists. The SfM technique proceeds in the order of camera image acquisition, camera position/position estimation, 3D information estimation of an object, and optimization technique. On the other hand, the present apparatus 10 proposes a new methodology that simultaneously processes the camera position/position estimation and the 3D information estimation process of an object among the existing SfM processing processes, thereby providing estimation results with higher accuracy compared to the prior art. have.

The apparatus 10 can simultaneously acquire (estimate) the position/position of the camera and the 3D information of a new object with relatively high accuracy even when there is an error in the 3D information of a known object compared to the existing technique, and the provided measurement value Depending on the combination of , it may have multi-purpose features that can be converted to the existing PnP technique or the triangulation technique.

In the device 10 , a 2D-3D correspondence point (a 2D-3D correspondence-related point) and a 2D-2D correspondence point (a 2D-2D correspondence-related point) may be utilized (used) as measurement values. Here, the 2D-3D corresponding point represents a combination of the three-dimensional coordinates of the known object and the two-dimensional coordinates of the object projected onto the image plane (the coordinates of the projected 2D point). The 2D-2D correspondence point represents a combination of two-dimensional coordinates when an unknown object is projected into images from different viewpoints (ie, the first image and the second image). The device 10 receives the measurement values of these two combinations (2D-3D correspondence point and 2D-2D correspondence point) as input, and the camera posture (position, direction) at the current time point (eg, the time point corresponding to the second time epoch) and the three-dimensional coordinates of an unknown object can be estimated simultaneously.

While there is a disadvantage in that scale information cannot be restored when only 2D-2D corresponding points are used in the related art, the apparatus 10 can improve this problem by additionally using 2D-3D corresponding points. In addition, the apparatus 10 can estimate the position and posture of the camera in one estimator (single estimator) at the same time, unlike the conventional method of estimating the camera posture, for example, in three steps, In addition, unknown 3D coordinates can be estimated simultaneously. The apparatus 10 may provide a complex technique of not only estimating the position and posture of the camera, but also estimating the 3D coordinates of an unknown object at the same time.

It can be said that scale information is essential in order to express the position and posture of the camera in real 3D space, not virtual reality. Accordingly, the apparatus 10 may express the path of the camera in an actual three-dimensional space by restoring the scale by providing a method proposed differently from the existing ones.

The apparatus 10 can effectively improve the estimation accuracy of the position, posture, and unknown 3D point of the camera compared to the prior art by using not only the 2D-3D corresponding point but also the 2D-2D corresponding point. By using the 2D-3D correspondence point and the 2D-2D correspondence point together, the apparatus 10 can simultaneously estimate the position, posture, and unknown 3D point of the camera at the same time. Also, the apparatus 10 may acquire all information about the camera's position, posture, and image feature point by using both the 2D-3D correspondence point and the 2D-2D correspondence point. The apparatus 10 can estimate the 3D position of the image feature point by using the 2D-2D corresponding point, and at the same time estimate the position and posture of the camera.

The device 10 uses images (ie, first and second images) acquired from a camera without utilizing Global Positioning System (GPS) and Inertial Navigation System (INS) and known feature point coordinates (ie, known coordinates of a reference object). By using the coordinates of the 3D reference point), it is possible to provide (estimate) both the position and posture of the camera and the coordinates of the unknown feature point. That is, the device 10 is a technology capable of estimating the position and posture of the camera and the three-dimensional coordinates of an object using only the camera, and does not require additional sensors such as GPS and INS. It is possible to reduce (reduce) additional cost and weight due to the corresponding sensors (GPS, INS sensors).

The device 10 may be applicable to a single camera as well as a stereo camera. Also, the apparatus 10 may be utilized in various perspective projection-based cameras.

The apparatus 10 may estimate the position and posture of the camera by using the 2D-3D corresponding point. Also, the apparatus 10 may estimate the 3D coordinates of the unknown object by using the 2D-2D correspondence point and the known camera posture and position information. The apparatus 10 may be able to estimate not only the three-dimensional coordinates of an unknown object, but also the position and posture of the camera.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly reviewed.

10 is a flowchart illustrating a simultaneous estimation method for simultaneously estimating a camera posture and 3D coordinates of a target object according to an exemplary embodiment of the present disclosure.

The simultaneous estimation method of simultaneously estimating the camera posture and the 3D coordinates of the target object shown in FIG. 10 may be performed by the apparatus 10 described above. Therefore, even if omitted below, the description of the apparatus 10 may be equally applied to the description of the simultaneous estimation method for simultaneously estimating the camera posture and the 3D coordinates of the target object.

Referring to FIG. 10 , in step S11 , the input unit may receive coordinates of a known 3D reference point of a reference object.

In this case, the known coordinates of the 3D reference point may include the coordinates of the estimated 3D reference point having an error in which the estimation error of the actual 3D point of the reference object is considered. That is, the coordinates of the known 3D reference point considered in step S11 may include, for example, the coordinates of the inaccurate 3D reference point in which an error exists.

Next, in step S12, the setting unit sets the first projected 2D point projected on the first image acquired at the first time epoch through the camera with respect to the known 3D reference point input in step S11, and When the second image is obtained at a second time epoch different from the first time epoch due to the change of posture, a second projected 2D point projected on the second image may be set.

Next, in step S13, the estimator, in response to the change in the posture of the camera, on the two images in which the relation information between the projected 2D point and the 3D point on the image obtained based on the information set in step S12 and the change in the posture of the camera are considered Using a cost function defined in consideration of information on the relative positional relationship with respect to a known 3D reference point of Simultaneous estimation of the estimated target 3D coordinates, which are the coordinates of the unknown 3D point of the target object, can be performed.

Here, the information on the relative positional relationship may include relationship information between the second projected 2D point and the first projected 2D point.

In addition, in step S13, the cost function is an overall state vector defined to include the estimation target camera posture-related state parameter and the estimation target 3D coordinate-related 3D coordinate estimation error, and the total state vector defined to be related to the indirect measurement value in which the depth-related scale factor is considered. It can be defined as the relationship between the indirect measurement vector and the entire observation matrix representing the connection relationship between the indirect measurement and the overall state vector.

In addition, in step S13, the estimator performs simultaneous estimation through the minimization of the cost function, but calculates the minimum error value among the error values of the entire state vector, which is a value of the cost function calculated by multiplying the entire indirect measurement vector and the entire observation matrix. Based on the state vector, the estimated camera posture and the estimated 3D coordinates may be simultaneously estimated.

In addition, in step S13, the cost function may be set to satisfy Equation 22, and since the description of Equation 22 has been described in detail above, the overlapping description will be described below.

In addition, the relation information and the positional relation information considered in step S13 may be information set based on Euclidean geometry.

In the above description, steps S11 to S13 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between the steps may be changed.

The simultaneous estimation method for simultaneously estimating the camera posture and the 3D coordinates of the target object according to an embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the simultaneous estimation method of simultaneously estimating the camera posture and the 3D coordinates of the target object may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

10: Simultaneous estimation device for simultaneously estimating the camera posture and the 3D coordinates of the target object
11: input
12: setting unit
13: estimator

Claims (13)

A simultaneous estimation method for simultaneously estimating a camera posture and 3D coordinates of a target object, comprising:
(a) receiving the coordinates of a known 3D reference point of the reference object;
(b) with respect to the known 3D reference point, setting a first projected 2D point projected on a first image acquired at a first time epoch through a camera, and by changing the posture of the camera, the first time epoch setting a second projected 2D point projected on the second image when a second image is acquired at a second time epoch different from ; and
(c) in response to the change in the posture of the camera, relation information between the projected 2D point and the 3D point on the image obtained based on the information set in step (b), and the change in the posture of the camera on the two images Using a cost function defined in consideration of information on the relative positional relationship with respect to the known 3D reference point, the estimated target camera attitude, which is the changed attitude of the camera corresponding to the second time epoch, and the position corresponding to the second time epoch performing simultaneous estimation of the estimated target 3D coordinates that are the coordinates of the unknown 3D point of the target object in the second image,
The information on the relative positional relationship will include relationship information between the second projected 2D point and the first projected 2D point. According to claim 1,
In step (c), the cost function is
A global state vector defined to include the estimated target camera posture-related state parameter and the estimated target 3D coordinate-related 3D coordinate estimation error, a global indirect measurement vector defined to relate a depth-related scale factor to the considered indirect measurement value, and the indirect measurement value and a relationship between the entire observation matrix representing the connection relationship between the global state vectors and the simultaneous estimation method. 3. The method of claim 2,
Step (c) is,
performing the simultaneous estimation through minimization of the cost function,
Based on a total state vector that calculates a minimum error value among error values of a total state vector that is a value of a cost function calculated by multiplying the entire indirect measurement vector and the entire observation matrix, the estimated camera posture and the estimated target 3D coordinates Simultaneous estimation method for estimating at the same time. 4. The method of claim 3,
In step (c), the cost function is set to satisfy Equation 1 below,
[Equation 1]

here,

is the estimated full state vector with estimation error,

is the estimated total observation matrix,

is the estimated total indirect measurement vector,

Estimate of the entire state vector until the norm of

is updated, the simultaneous estimation method. According to claim 1,
The coordinates of the known 3D reference point in step (a) are,
The simultaneous estimation method, wherein the estimation error of the actual 3D point of the reference object includes the coordinates of the estimated 3D reference point in which the error is considered. According to claim 1,
In the step (c), the information about the relationship information and the location relationship is information set based on Euclidean geometry. A simultaneous estimation device for estimating a camera posture and 3D coordinates of a target object at the same time, comprising:
an input unit for receiving coordinates of a known 3D reference point of a reference object;
With respect to the known 3D reference point, set a first projected 2D point projected on a first image acquired at a first time epoch through a camera, and different from the first time epoch by changing the attitude of the camera a setting unit configured to set a second projected 2D point projected on the second image when a second image is obtained in a second time epoch; and
In response to the change in the posture of the camera, the relationship information between the projected 2D point and the 3D point on the image obtained based on the information set in the setting unit and the known 3D reference point on the two images in which the change in the posture of the camera is considered Using a cost function defined in consideration of information on a relative positional relationship between and an estimator that performs simultaneous estimation of the estimated target 3D coordinates that are coordinates of the unknown 3D point of the target object;
The information about the relative positional relationship includes relation information between a second projected 2D point corresponding to the known 3D reference point projected on the second image and the first projected 2D point projected on the first image which includes, a simultaneous estimation device. 8. The method of claim 7,
The cost function is
A global state vector defined to include the estimated target camera posture-related state parameter and the estimated target 3D coordinate-related 3D coordinate estimation error, a global indirect measurement vector defined to relate a depth-related scale factor to the considered indirect measurement value, and the indirect measurement value and a relationship between the entire observation matrix representing the connection relationship between the global state vectors and the simultaneous estimation apparatus. 9. The method of claim 8,
The estimator is
performing the simultaneous estimation through minimization of the cost function,
Based on a total state vector that calculates a minimum error value among error values of a total state vector that is a value of a cost function calculated by multiplying the entire indirect measurement vector and the entire observation matrix, the estimated camera posture and the estimated target 3D coordinates Simultaneous estimation apparatus for estimating at the same time. 10. The method of claim 9,
The cost function is set to satisfy Equation 2 below,
[Equation 2]

here,

is the estimated full state vector with estimation error,

is the estimated total observation matrix,

is the estimated total indirect measurement vector,

Estimate of the entire state vector until the norm of

is updated, the simultaneous estimation device. 8. The method of claim 7,
The coordinates of the known 3D reference point are,
The simultaneous estimation apparatus of claim 1, wherein the estimation error of the actual 3D point of the reference object includes the coordinates of the estimated 3D reference point in which the error is considered. 8. The method of claim 7,
The information on the relationship information and the information on the position relationship is information set based on Euclidean geometry. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 6 on a computer.

Images