JP2015149675A - Camera parameter estimation apparatus and camera parameter estimation program - Google Patents

Abstract

A camera parameter estimation apparatus capable of improving the estimation accuracy of camera parameters to be estimated is provided. A camera parameter estimation device for estimating camera parameters of a plurality of cameras from videos taken by a plurality of cameras, a feature point detection unit for detecting feature points from the videos, and correspondence between the detected feature points A feature point associating unit that performs attachment, a camera parameter estimation unit that calculates an estimated value of the camera parameter from the associated feature point, and calculates a predicted value of the camera parameter based on the estimated value of the camera parameter that has already been estimated The feature point associating unit for associating using the restriction on the associating of the feature points derived from the predicted values of the camera parameters. [Selection] Figure 1

Description

  The present invention relates to a camera parameter estimation device and a camera parameter estimation program for estimating camera parameters.

  To add a geometric process such as estimating the 3D shape of a scene from an image or video, or synthesizing an image from a virtual viewpoint, the internal parameters (focal length, image center, etc.) Camera calibration processing for estimating camera parameters such as external parameters (position and orientation between cameras) is necessary. Normally, image-based camera calibration is performed using dedicated equipment (such as a chess pattern) before shooting, but this method assumes that the camera is fixed once shooting starts. This method is difficult to apply when using a dynamic camera whose focal length or position information changes.

  On the other hand, a method of performing camera calibration from information existing in an image without using a dedicated device has been proposed (see, for example, Non-Patent Document 1). These methods detect feature points for images taken by multiple cameras (however, each image has an overlapping area), take correspondence between feature points, and perform camera calibration based on them. Do. Unlike the former method, the latter method does not require dedicated equipment, and can be applied to various captured images. Here, the latter approach is called natural feature point based calibration. This natural feature point-based calibration requires correspondence of at least eight or more feature points in a certain image pair, and it is known that the accuracy of correspondence greatly affects the estimation accuracy of camera parameters.

  In natural feature point-based calibration, when the difference in position and orientation between cameras used for shooting increases, the appearance of feature points in the captured image group changes, and feature points that should not be associated with each other There is a case where feature points that are or should be associated with each other may not be associated with each other, and there is a problem that the estimation accuracy of the camera parameter is lowered.

  For this problem, it is known that imposing time series restrictions on feature points is effective when the processing target is video. This is to cope with a decrease in accuracy due to continuous changes in the appearance of feature points by taking over the feature points obtained at a certain time and their correct correspondence to other times. Here, this approach is referred to as an image-based time series constraint.

Noah Snavely, Steven M Seitz, Richard Szeliski, “Photo Tourism: Exploring image collections in 3D”, ACM Transaction on Graphics (Proceedings of SIGGRAPH 2006), 2006

  However, in order to use this image-based time-series constraint, a certain feature point and a corresponding feature point are detected on another image (an image taken by another camera), and a plurality of them are consecutive. It must continue to be detected over time. For this reason, images with sudden changes in appearance due to camera movements or subject changes (for example, content with lighting changes taken on a stage-like video like a hand-held camera) In some cases, this time series constraint may not be applicable.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a camera parameter estimation device and a camera parameter estimation program capable of improving the estimation accuracy of camera parameters to be estimated.

  The present invention is a camera parameter estimation device that estimates camera parameters of a plurality of cameras from videos captured by a plurality of cameras, a feature point detection unit that detects feature points from the videos, and the detected features A feature point associating unit that associates points; a camera parameter estimating unit that calculates an estimated value of the camera parameter from the associated feature points; and a camera based on the estimated value of the camera parameter that has already been estimated. A camera parameter predicting unit that calculates a predicted value of the parameter, and the feature point associating unit performs associating using a restriction on the associating of the feature points derived from the predicted value of the camera parameter And

  According to the present invention, the camera parameter prediction unit uses the camera parameter prediction value immediately before the estimated value of the camera parameter as the prediction value of the camera parameter, and the constraint used in the feature point association unit is an epipolar constraint It is characterized by that.

  In the present invention, the camera parameter prediction unit calculates a probability distribution of camera parameter prediction values based on the estimated values of the camera parameters already estimated, and the feature point association unit appears in the probability distribution. The feature points are associated with each other based on the epipolar constraint determined for each predicted value of the camera parameter and the probability given to the predicted value of the camera parameter.

  The present invention is characterized in that the probability distribution calculated by the camera parameter predicting unit is given a weight that becomes larger as it is immediately before the estimated value of the camera parameter.

  The present invention further includes a sensor for measuring a change in the position and orientation of the camera, and the camera parameter estimation unit calculates in the past when the estimated value of the camera parameter cannot be calculated from the associated feature point. The estimated value of the camera parameter is calculated based on the estimated value of the camera parameter and the degree of change in the position and orientation of the camera measured by the sensor.

  The present invention is a camera parameter estimation program for causing a computer to function as the camera parameter estimation device.

  According to the present invention, it is possible to improve the estimation accuracy of camera parameters to be estimated.

It is a block diagram which shows the structure of 1st Embodiment of this invention. It is a flowchart which shows the processing operation of the camera parameter estimation apparatus shown in FIG. It is a flowchart which shows the processing operation of the camera parameter estimation apparatus in 2nd Embodiment of this invention. It is a flowchart which shows the processing operation of the camera parameter estimation apparatus in 3rd Embodiment of this invention. It is a block diagram which shows the structure of the camera parameter estimation apparatus by 5th Embodiment of this invention. It is a flowchart which shows the processing operation of the camera parameter estimation apparatus in 5th Embodiment of this invention. It is explanatory drawing which shows the example of a multiview video application using only the camera parameter obtained by applying the camera parameter estimation apparatus by this invention.

<First Embodiment>
Hereinafter, a camera parameter estimation apparatus according to a first embodiment of the present invention will be described with reference to the drawings. In the embodiment described below, in aiming to improve accuracy by providing time series constraints in natural feature point based calibration, attention is paid to the continuity of feature point detection and association like image based time series constraints. Rather than providing a constraint, a constraint focusing on the continuity of the estimated camera parameters is provided in the feature point association. First, the value (range) of the camera parameter at the next time (t + 1) is predicted from the camera parameter estimated at a certain time t and the camera parameter estimated until the time t. At time (t + 1), a restriction is imposed on the feature point correspondence using the predicted value, and the camera parameter at time (t + 1) is estimated again under the restriction. It is an outline of this embodiment to repeat this procedure. In this embodiment, this approach is referred to as camera parameter-based time series constraint.

Here, the following notation is defined.
I _i (t): an image taken by camera i at time t. In the present embodiment, the cameras are assumed to be synchronized.
R _i (t): A rotation matrix representing the attitude of the camera i in the world coordinate system at time t.
R _{ii ′} (t): A rotation matrix representing the difference in posture between camera i and camera i ′ at time t. This is obtained from R _i (t), R _{i ′} (t), T _i (t), T _{i ′} (t).
T _i (t): A translation vector representing the position of camera i in the world coordinate system at time t.
T _{ii ′} (t): A translation vector representing the difference between the positions of camera i and camera i ′ at time t. This is obtained from R _i (t), R _{i ′} (t), T _i (t), T _{i ′} (t).
K _i (t): A matrix representing the focal length and the image center of the camera i at time t.
P _i (t): Camera parameters of camera i output by the camera parameter estimation unit at time t (summarized R _i (t), T _i (t), K _i (t)).
P ′ _i (t + 1): The predicted value of the camera parameter of camera i at time (t + 1) output by the camera parameter prediction unit at time t.
f _i ^j (t): feature quantity vector of the j-th feature point on I _i (t).
x _i ^j (t): A coordinate value vector on the image plane of f _i ^j (t).
f _i (t): A set of feature quantity vectors of feature points on I _i (t).
x _i (t): A set of coordinate value vectors of feature points on I _i (t).
M _{ii ′} (t): Matching result of f _i (t), f _{i ′} (t).
E _i (t): A value output from an external sensor (gyro sensor, GPS, etc.) attached to the camera i.

  FIG. 1 is a block diagram showing the configuration of the camera parameter estimation apparatus according to the embodiment. In this figure, reference numeral 1 denotes a multi-view video input unit for inputting multi-view video from outside. Reference numeral 2 denotes a feature point detection unit that detects feature points from the input multi-viewpoint video. Reference numeral 3 denotes a feature point association unit that associates detected feature points. Reference numeral 4 denotes a camera parameter estimation unit that estimates camera parameters from the associated feature points. Reference numeral 5 denotes a camera parameter prediction unit that predicts camera parameters at a predetermined time. Reference numeral 6 denotes a camera parameter output unit that outputs estimated camera parameters. Natural feature point-based calibration is usually composed of a feature point detection unit, a feature point association unit, and a camera parameter estimation unit. A camera parameter prediction unit is added to the natural feature point base calibration, and the predicted camera parameters are converted into feature point association units. This is different from the conventional natural feature point based calibration.

Next, the operation of the camera parameter estimation apparatus shown in FIG. 1 will be described with reference to FIG. First, the multi-view video input unit 1 inputs an image sequence I _i (t) (i = 0... N; n is the number of cameras). Next, the feature point detecting unit 2 performs the feature point detection for the image _I i _(t), we obtain the _f i (t).

Next, if if t = 0, the feature point associating unit 3 applies the feature point group f _i (t), f _{i ′} (t) to each camera pair (i, i ′). Then, feature point correspondence is performed and M _{ii ′} (t) is obtained. On the other hand, if if t> 0, the feature point group f _i (t), f _{i ′} (t) is constrained using camera parameter prediction values P ′ _i (t), P ′ _{i ′} (t). And ask for a response. As a result, M _{ii ′} (t) is obtained.

Next, the camera parameter estimation unit 4 estimates the camera parameter P _i (t) using all M _{ii ′} (t). The camera parameter output unit 6 outputs the value estimated here as the camera parameter P _i (t) at each time. If it cannot be estimated, the camera parameter P _i (t) is not output. Further, when processing at the next time is performed, t = 0 is returned.

Next, the camera parameter prediction unit 5 predicts P ′ _i (t + 1) from the value of P _i (t ′) (t ′ ≦ t). And it transfers to the process of the feature point matching part 3. FIG.

Next, detailed processing operations of the feature point detection unit 2, the feature point association unit 3, the camera parameter estimation unit 4, and the camera parameter prediction unit 5 shown in FIG. 1 will be described. First, the feature point detection unit 2 will be described. The feature point detector 2 detects feature points from the input video. The input of the feature point detection unit 2 is an image I _i (t), (i = 0, 1, t ≧ 0). The output is a set f _i (t) of feature quantity vectors of feature points. The feature point detector 2 detects a feature point from the input image I _i (t) using a feature point detector (Harris, SIFT, etc.), and the feature quantity vector f _ij (t) of each feature point is detected. Output f _i (t) as a set.

Next, the feature point association unit 3 will be described. The feature point associating unit 3 performs associating for a set of feature amount vectors in each image. The input of the feature point association unit 3 is a set of feature vectors f _i (t) (i = 0, 1, t ≧ 0) and a camera parameter predicted value P ′ _i (t). The output is the association vector M _{ii ′} (t) (i ′ = 0, 1, i ≠ i ′). When t = 0, the feature point association unit 3 _converts each feature point f _ij (t) of the image i into an image in the given set of feature vectors f _i (t) and f _{i ′} (t). It calculates which feature point of i ′ corresponds, and outputs the result of the correspondence as M _{ii ′} (t). When t> 0, the camera parameter prediction value P ′ _i (t) given as an input is used to perform the above-described processing after the restriction is set on the association, and the correspondence result is set as M _{ii ′} (t). Output.

Next, the camera parameter estimation unit 4 will be described. The camera parameter estimation unit 4 estimates camera parameters of the camera used for shooting. The input of the camera parameter estimation unit 4 is the feature point coordinates x _i (t) and the association vector M _{ii ′} (t) (i, i ′ = 0, 1, i ≠ i ′, t ≧ 0). The output is a camera parameter P _i (t). The camera parameter estimation unit 4 uses the camera parameter P _i (the camera parameter P _i (t) used for photographing each image from the coordinate vector x _i (t) of the feature point given as input and the association vector M _{ii ′} (t). t) is estimated. However, at least 8 pairs of feature points are required.

Next, the camera parameter prediction unit 5 will be described. The camera parameter prediction unit 5 predicts camera parameters at time (t + 1). The input of the camera parameter prediction unit 5 is a camera parameter P _i (t ′) (i = 0, 1, 0 ≦ t ′ ≦ t), and the output is a camera parameter predicted value P ′ _i (t + 1). is there. The camera parameter prediction unit 5 outputs the predicted value P ′ _i (t + 1) of the value (and range) of the camera parameter at time (t + 1) using the camera parameter P _i (t ′) estimated until time t. .

  The processing operation described above can be implemented by applying the following method. In the present embodiment, the process of the camera parameter prediction unit 5 is added to the procedure of the conventional natural feature point base calibration, and a restriction is imposed on the feature point association unit 3 using the processing. Therefore, here, first, a general method in each procedure of natural feature point base calibration will be described, and how to give constraints in the camera parameter prediction unit 5 and the feature point association unit 3 will be described.

  First, what kind of method is used in each part of the natural feature point based calibration will be described. As the feature point detection unit 2, the SIFT feature amount (for example, “David G. Lowe,“ Distinctive image features from scale-invariant keypoints ”, International Journal of Computer Vision, 60, 2, 2004, pp.91-110 Can be applied). Each feature point on the image is detected using SIFT, and the position, orientation, and feature amount on the image are output. Since the details of the feature amount calculation method are known methods described in the above-mentioned document 1, detailed description thereof is omitted here.

Next, as the feature point associating unit 3, SIFT Matching (for example, “David G. Lowe,“ Distinctive image features from scale-invariant keypoints ”, International Journal of Computer Vision, 60, 2, 2004, pp. 91-110 ”) is applicable. To SIFT feature vector _f ij of each feature point in the image _{I i (t) (t)} , determine the corresponding feature points on the image I _{i '(t).} Whether or not a pair of feature points corresponds is determined by calculating a norm of a difference in each dimension of each feature amount. Since a detailed method is a known method described in the above-mentioned document 1, detailed description is omitted here.

Next, as the camera parameter estimator 4, Structure from Motion (for example, refer to “Richard Hartley and Andrew Zisserman,“ Multiple View Geometry in Computer Vision ”, Cambridge University Press, March 2004, pp. 262-278). And Bundle Adjustment (see, for example, "Bill Triggs, Phillip F. McKauchian, Richard Hartley, Andrew W. Fitzgibbon," Bundle Adjustment-A Modern Synthesis ", Vision Algorithm: Theory and Practice, 2000, pp.298-372). ) Is applicable. From the coordinate vector x _i (t) of the feature point and the correspondence vector M _{ii ′} (t), a basic matrix established between the cameras is obtained, and an internal parameter and an external parameter are obtained by decomposing the matrix (Structure from Motion). Further, in order to increase the accuracy of the obtained camera parameter, nonlinear optimization is performed on the parameter to minimize the reprojection error of the feature point (Bundle Adjustment). This process can be omitted.

  Next, the camera parameter prediction unit 5 and the feature point associating unit 3 which are features of the present embodiment will be described. The case where the camera parameter prediction unit 5 uses the estimated value of the camera parameter at time t as the predicted value of the camera parameter at time (t + 1) and uses epipolar constraints as the constraints in the feature point association unit 3 will be described. Here, for the sake of simplicity, it is assumed that a video from two viewpoints is processed.

The camera parameter prediction unit 5 determines the value of time (t + 1) by the following equation.
P ′ _i (t + 1) = P _i (t) (i = 0, 1, t ≧ 0)

Next, the feature point association unit 3 performs the same process as the natural feature point base calibration described above when t = 0. On the other hand, when t> 0, the basic matrix is calculated using the camera parameters P ′ ₀ (t) and P ′ ₁ (t) predicted by the camera parameter prediction unit 5. The basic matrix F _{ii ′} (i, i ′ = 0, 1, i ≠ i ′) is calculated as follows.
F _{ii ′} = K _{i ′} ^−T [T _{ii ′} × R _{ii ′} ] K _i ⁻¹

When a point in a certain three-dimensional space is projected onto the images I ₀ (t) and I ₁ (t), if the coordinate values are x _i (t) and x _{i ′} (t), the basic matrix F It is known that the following epipolar constraint is satisfied using _{ii ′} .
x _i (t) F _{ii ′} (t) x _{i ′} (t) = 0 (1)

At this time, for the feature quantity vector f _ij (t) of each feature point of the image I _i (t), by substituting the coordinate value on the image plane as x _i (t) into the equation (1), A range in which x _{i ′} (t) exists, that is, a range in which the coordinates of feature points on image I _{i ′} (t) corresponding to f _ij (t) exist is defined as a straight line (epiline) of image I _{i ′} (t). ) Can be limited to above. By limiting the area where appropriate correspondence exists from the whole image to a straight line, it is expected to reduce the number of cases that can be matched because the distance of the feature amount happens to be closer than the feature points that should be matched. it can. As the number of viewpoints that can observe the same feature point increases, the applicable restrictions also increase (for example, triaxial tenor for three viewpoints). Reference 4 “Satoshi Sato,“ Tensor and Multi-View Geometry ”, Information Processing Society of Japan CVIM, 2007, pp: 33-42” is detailed regarding the restrictions according to the number of viewpoints.

From the above, the procedure for obtaining a specific response is as follows. First, with respect to the coordinates _x ij of each feature point in the image _I i (t) (t), given as a candidate feature points (1) image I _{i 'satisfying} the equation (t). Actually, the value of the expression (1) is not always 0 due to the influence of pixel noise or the like, so that x _ij (t) F _{ii ′} (t) and x _i′j (t) on the image List distances below threshold. The feature point group listed here as a candidate is associated with the same method as in the case of t = 0. Thereby, the feature point correspondence which added the camera parameter prediction value as a restriction | limiting is realizable.

  Next, the overall processing operation of the camera parameter estimation apparatus (FIG. 1) in the first embodiment will be described with reference to FIG. FIG. 2 is a flowchart showing the processing operation of the camera parameter estimation apparatus shown in FIG. First, the multi-view video input unit 1 gives a captured multi-view video as an input (step S1). In response to this, the feature point detection unit 2 detects SIFT feature values for the multi-view video (step S2).

  Next, the feature point association unit 3 determines whether or not time = 0 (step S3), and if time = 0, sets all feature points on the image as correspondence candidates (step S4). Then, the feature point associating unit 3 associates each feature point from among the candidates (step S5).

  On the other hand, if time = 0, the feature point association unit 3 calculates a basic matrix from the predicted camera parameters (step S6). Then, the feature point association unit 3 calculates an epi line for each feature point, sets the feature point existing on the epi line as a candidate for correspondence (step S7), and associates each feature point with each of the candidates. (Step S5). The processing in steps S3 to S7 is applied to all camera pairs.

  Next, the camera parameter estimation unit 4 estimates camera parameters from the correspondence of the feature points (step S8). Subsequently, the camera parameter prediction unit 5 sets the estimated camera parameter as a camera parameter prediction value at the next time (step S9).

  As described above, the first embodiment is expected to work most effectively when the frame rate of the camera is high and the camera is hardly moving.

Second Embodiment
Next, a camera parameter estimation device according to a second embodiment of the present invention will be described. Since the configuration of the camera parameter estimation apparatus according to the second embodiment is the same as the configuration shown in FIG. 1, detailed description thereof is omitted here. This embodiment demonstrates the case where the camera parameter estimation part 5 estimates the range of the next time at random. In the present embodiment, it is assumed that images from two viewpoints are processed for simplicity.

In this embodiment, the camera parameter prediction value by the camera parameter prediction unit 5 is not uniquely determined as in the first embodiment, but the range estimated to vary from the camera parameter estimation value by the camera parameter estimation unit 4 to the next time is set. The camera parameter determined in the range is used as the camera parameter predicted value and used in the feature point association unit 3. For example, the rotation matrix R _{ii ′} (t) is expressed by three variables (rotation angles for each of the x-axis, y-axis, and z-axis), and each is θ _k (t) (k = x, y, z), Is moved in a range of ± r _k , the predicted value θ ′ _k (t + 1) is expressed as follows.
θ ′ _k (t + 1) = θ _k (t) + Δθ _k (t), (k = x, y, z) (2)
| Δθ _k (t) | <r _k

  The parameters to be considered this time are considered to be a total of six variables: three variables related to the rotation matrix between the cameras, two variables related to the translation vector, and one variable related to the focal length. However, the scale of the translation vector may be newly considered as a variable. Since the norm of the translation vector is 1, the translation vector has two variables instead of three.

  When this embodiment is implemented, the calculation may be performed for all combinations by dividing the predetermined range into several stages. However, depending on the stage divided from the range, the calculation amount may be enormous. . For this reason, the number of candidates may be specified in advance, and values within a predetermined range may be randomly output according to an appropriate distribution.

This will be specifically described below. First, it is assumed that the camera parameter P (t) is composed of three variables of a rotation matrix, two variables of a translation vector, and one variable of a focal length, and these are p _i (t) (i = 0,..., 5). As each variable is to move in ranges of ± r _i, the predicted value p _{'i (t} + 1) is expressed as follows.
p ′ _i (t + 1) = p _i (t) + Δp _i (t), (i = 0,..., 5)
| Δp _i (t) | <r _i

At this time, when dividing the range ± _{r i} as defined in _{s i} stages, updated values Delta] p _ji for j th candidate _{p 'ji (t + 1)} (t) are expressed as follows.
Δp _ji (t) = j × (2 × r _i / s _i ), (j = 0... (S _i −1))

At this time, since s _i updated values are obtained for each variable, there are s ₀ × s ₁ × s ₂ × s ₃ × s ₄ × s ₅ candidates for P ′ (t + 1). In other words, as described above, the calculation amount may be enormous. Therefore, there is a method of giving randomly as follows.
_Let c _{p be} the number of candidates for P ′ (t + 1) in advance. The updated value Δp ^l _i (t) of each variable p ′ ^l _i (t + 1) for the l-th candidate P ′ ^l (t + 1) is given a random value satisfying | Δp _i (t) | <r _i .

  Next, the overall processing operation of the camera parameter estimation apparatus in the second embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing the processing operation of the camera parameter estimation apparatus in the second embodiment. In FIG. 3, the same reference numerals are given to the same processing operations as the processing operations shown in FIG. First, the multi-view video input unit 1 gives a captured multi-view video as an input (step S1). In response to this, the feature point detection unit 2 detects SIFT feature values for the multi-view video (step S2).

  Next, the feature point association unit 3 determines whether or not time = 0 (step S3), and if time = 0, sets all feature points on the image as correspondence candidates (step S4). Then, the feature point associating unit 3 associates each feature point from among the candidates (step S5).

  On the other hand, if time = 0, the feature point association unit 3 calculates a basic matrix from the predicted camera parameters (step S6). Then, the feature point association unit 3 calculates an epi line for each feature point, sets the feature point existing on the epi line as a candidate for correspondence (step S7), and associates each feature point with each of the candidates. (Step S5). The processing in steps S3 to S7 is applied to all camera pairs.

  Next, the camera parameter estimation unit 4 estimates camera parameters from the correspondence of the feature points (step S8). The camera parameter estimation unit 4 updates each estimated camera parameter by giving a random value (step S10). Then, the camera parameter prediction unit 5 sets the updated value as the camera parameter prediction value at the next time (step S11).

  In this way, the second embodiment is expected to work effectively when the magnitude of movement of the camera in one time is known to some extent.

<Third Embodiment>
Next, a camera parameter estimation device according to a third embodiment of the present invention will be described. Since the configuration of the camera parameter estimation apparatus according to the third embodiment is the same as the configuration shown in FIG. 1, detailed description thereof is omitted here. In the present embodiment, the camera parameter prediction unit 5 calculates the probability distribution of the camera parameter value (camera parameter predicted value) at the next time based on the camera parameter estimated value until time t ′ (<t). A case will be described in which the probability distribution is reflected in the association of feature points in the association unit 3. In the present embodiment, the probability distribution of camera parameter prediction values is calculated using the concept of a Bayesian filter (particularly, a particle filter) as an example. In the present embodiment, it is assumed that images from two viewpoints are processed for simplicity.

  Next, an overall processing operation of the camera parameter estimation apparatus according to the third embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing the processing operation of the camera parameter estimation apparatus in the third embodiment. In FIG. 4, the same reference numerals are given to the same processing operations as the processing operations shown in FIG. First, the multi-view video input unit 1 gives a captured multi-view video as an input (step S1). In response to this, the feature point detection unit 2 detects SIFT feature values for the multi-view video (step S2).

  Next, the feature point association unit 3 determines whether or not time = 0 (step S3), and if time = 0, sets all feature points on the image as correspondence candidates (step S4). Then, the feature point associating unit 3 associates each feature point from among the candidates (step S5).

  On the other hand, if not time = 0, the feature point association unit 3 calculates the likelihood of the predicted camera parameter (step S12). Subsequently, the feature point association unit 3 calculates a basic matrix from the camera parameters with likelihood (step S13). Then, the feature point associating unit 3 calculates an epi-line with likelihood for each feature point, sets the feature point existing on the epi-line as a candidate for correspondence (step S14), and sets each feature point as a respective candidate. Matching is performed considering the likelihood from the inside (step S15). The processes in steps S3 to S5 and S12 to S15 are applied to all camera pairs.

  Next, the camera parameter estimation unit 4 estimates camera parameters from the correspondence of the feature points (step S8). Then, the camera parameter prediction unit 5 predicts the camera parameter at the next time (step S16).

  In the processing operation shown in FIG. 4, since the SIFT feature quantity for the input image at time t is necessary to calculate the likelihood for the camera parameter candidate at time t, the likelihood calculation is performed in step S2. However, likelihood calculation may be performed at the time of prediction in step S16 (time (t-1)). However, in this case, it is necessary to calculate SIFT feature values for the input image at the next time (time t) when predicted. Also, changing the processing order in this way does not affect the calculation results.

  Next, the process of the camera parameter prediction unit 5 in the third embodiment will be described. Since the particle filter is a well-known technique, a detailed description is omitted here, but in order to apply it to an actual problem, it is necessary to define a model to be processed, a state transition function, and a likelihood function. Specific examples thereof will be described below, and further, problem-specific processing handled by the present invention will be described.

  First, since it is camera parameters that are desired to be predicted this time, each camera parameter and its differential value can be considered as a model to be processed. The model at this time changes depending on the definition of a state transition function, which will be described later, but in this embodiment, for simplicity, only the camera parameter is used as a model to be processed. Hereinafter, this model at time t is referred to as a state vector s (t). It is assumed that the camera parameters are three dimensions in the rotation matrix, two dimensions in the translation vector, one dimension in the focal length, and six dimensions in total.

Next, the state transition function assumes a random walk. If the system noise w (t) follows a 6-dimensional normal distribution with an average of 0 and a covariance of Σ, the state transition function D (t) can be defined as follows.
s (t + 1) = D (t) [s (t), w (t)] = s (t) + w (t)

  The random walk state transition function is used when the movement of the object is relatively small or complicated. If the motion of the object can be assumed to be a constant velocity linear motion, a state vector definition and a state transition function that match it are necessary.

  Subsequently, regarding the implementation of the likelihood function, the present embodiment uses “whether or not the correspondence set obtained when the epipolar constraint is applied using the camera parameter is valid” as the plausibility of the predicted state vector. Specifically, the RANSAC process is performed on a corresponding set obtained by applying epipolar constraints using each camera parameter. At this time, epipolar constraints are used as a model used for the RANSAC process. Here, a value obtained by dividing the number of corresponding pairs after the RANSAC process by the number of corresponding pairs before being given is given as a likelihood.

Next, processing of the feature point association unit 3 in the third embodiment will be described. The feature points are associated using the probability distribution obtained above. As a specific procedure, first, an epi-line with likelihood is drawn for each feature point x ₀ ^j (t) by using candidate camera parameters with likelihood. Then, in order to obtain the feature point correspondence, when calculating the distance of the feature amount with respect to the feature point x ₁ ^{j ′} (t) on which x ₀ ^j (t) is on the epi line, the likelihood that the epi line has Is reflected in the distance. That is, a feature point existing on an epiline having a high likelihood can be easily associated with a feature point existing on an epiline having a low likelihood.

  The above is the behavior of the present invention when a probability (likelihood) is given to the predicted camera parameter (a particle filter is used as an example). By assigning the camera parameter probability to the association, it is possible to reflect the dynamics of the camera parameter in the association. For this reason, when processing an image shot by a camera with dynamics, it is expected that camera parameters can be estimated more robustly and with higher accuracy than in the second embodiment.

<Fourth embodiment>
Next, a camera parameter estimation device according to a fourth embodiment of the present invention will be described. Since the configuration of the camera parameter estimation apparatus according to the fourth embodiment is the same as the configuration shown in FIG. 1, detailed description thereof is omitted here. In the present embodiment, an expansion method that uses not only the previous time value but also the previous time and the previous time value in the camera parameter prediction unit 5 will be described. In addition, regarding the feature point associating unit 3, it is assumed that a method of associating based on the probability (likelihood) given to the camera parameter described in the third embodiment is used.

  As a specific implementation method, first, the time k to be used is determined (for example, k = 3 if used for the past three times), and an estimated value at each time is prepared. In the first embodiment, the estimated value at each time is used as it is as the predicted value of the next time. However, when the camera is moving, the time (t−l) (l <k) and l is large. If the estimated value of the time is used, the predicted value is not valid. Therefore, in this embodiment, the estimated value at each time has a weight w (t−l) (where Σlw (t−l) = 1, which is the probability (likelihood) of the camera parameter as described in the third embodiment. It is considered to be equivalent to)). Although details of the weight design at this time are omitted, basically, it is appropriate to decrease the weight as l increases.

  In the present invention, the reliability of the estimated camera parameters is not always the same every time, and there are cases where the estimation accuracy suddenly deteriorates. Since the present embodiment uses not only the previous time but also the past, it is expected to work robustly even in such a case. For the same purpose, it is also conceivable to predict a value one time ahead rather than the next time when making a prediction. Specifically, it can be obtained by using the update formula (2) of the second embodiment once. Of course, as described above, it is appropriate to decrease the weight as l increases. As described above, in the present embodiment, the value of the previous time is used as a constraint, or the value of the next time is predicted and used, the past value is used, and the future value is predicted and used. Is possible.

<Fifth Embodiment>
Next, a camera parameter estimation device according to a fifth embodiment of the present invention will be described. FIG. 5 is a block diagram showing the configuration of the camera parameter estimation apparatus according to the embodiment. In this figure, the same parts as those in the apparatus shown in FIG. The apparatus shown in this figure is different from the apparatus shown in FIG. 1 in that an external sensor data input unit 7 and a camera parameter estimation unit 8 are newly provided. The external sensor data input unit 7 inputs the value of an external sensor such as a gyro sensor or GPS. The camera parameter estimation unit 8 estimates camera parameters at time t. The input of the camera parameter estimation unit 8 is the camera parameter P _i (t−1) and the output value Ei (t) of the external sensor at time (t−1). The output is a camera parameter P _i (t). The camera parameter estimation unit 8 updates the camera parameter using a value Ei (t) output from an external sensor such as a gyro sensor at time t with respect to P _i (t−1), and the camera parameter P _{i at} time t. (T) is output.

Next, the operation of the camera parameter estimation apparatus shown in FIG. 5 will be described with reference to FIG. First, the feature point detection unit 2 inputs an image sequence I _i (t) via the multi-view video input unit 1. Then, the feature point detection unit 2 performs feature point detection on the image I _i (t) to obtain fi (t).

Next, if if t = 0, the feature point associating unit 3 applies the feature point group f _i (t), f _{i ′} (t) to each camera pair (i, i ′). Then, feature point correspondence is performed and M _{ii ′} (t) is obtained. On the other hand, if if t> 0, the feature point group f _i (t), f _{i ′} (t) is constrained using camera parameter prediction values P ′ _i (t), P ′ _{i ′} (t). And ask for a response. As a result, M _{ii ′} (t) is obtained.

Next, the camera parameter estimation unit 4 estimates the camera parameter P _i (t) using all M _{ii ′} (t). The camera parameter output unit 6 outputs the value estimated here as a camera parameter at each time.

Next, when the camera parameter estimation unit 8 inputs E _i (t) via the external sensor data input unit 7 and cannot be estimated by the camera parameter estimation unit 4, E _{i with} respect to P _i (t−1). P _i (t) is estimated using (t) and output as a camera parameter.

Next, the camera parameter prediction unit 5 predicts P ′ _i (t + 1) from the value of P _i (t ′) (t ′ ≦ t). And it transfers to the process of the feature point matching part 3. FIG.

Even if the camera parameter estimation unit 8 does not obtain the camera parameter from the video at time t, if the camera parameter P _i (t−1) at time (t−1) is obtained, P _i (t) is obtained. Desired. Specifically, since a change with respect to each axis of the camera from time (t−1) to time t is obtained as an angle using a gyro sensor or the like, the rotation matrix Ri (t−1) estimated at the previous time is obtained. R _i (t) is obtained by adding to. Furthermore, since the position of the camera at time t can be obtained with GPS, the translation vector T _i (t) is obtained using them. It is assumed that the internal parameter K _i (t) (mainly focal length) at each time is stored as an EXIF tag in the image. These are combined and output as camera parameters Pi (t) at time t.

  Using external sensors in this way is basically when camera parameters cannot be estimated from video (each camera does not have an overlapping area, has an overlapping area, but cannot be calculated because sufficient feature point correspondence cannot be obtained. If it can be calculated from the video, the result is given priority.

  Next, the overall processing operation of the camera parameter estimation apparatus in the fifth embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing the processing operation of the camera parameter estimation apparatus in the fifth embodiment. Here, a case where a gyro sensor and GPS are used as external sensors will be described. First, the multi-view video input unit 1 gives the captured multi-view video as an input to the feature point detection unit 2 (step S21). In response to this, the feature point detection unit 2 detects a feature point (step S22). Then, the feature point association unit 3 associates the feature points (step S23). Subsequently, the camera parameter estimation unit 4 estimates camera parameters (step S24).

  On the other hand, in parallel with this operation, the external sensor data input unit 7 acquires the value of the external sensor (step S25). The camera parameter estimation unit 4 determines whether camera parameters can be estimated (step S26). If the estimation is not possible as a result of the determination, the camera parameter estimation unit 8 estimates the camera parameter at the current time from the camera parameter estimated value at the previous time and the value of the external sensor (step S27). Then, the camera parameter prediction unit 5 predicts camera parameters (step S28).

  As described above, in this embodiment, by combining the camera and the external sensor, the camera parameter can be continuously obtained using the external sensor even when the camera parameter cannot be obtained from the video. The camera parameter prediction unit can be used in a variety of ways, including the methods described above. However, if an external sensor is used, it is expected that the accuracy will vary, or the accuracy will not be very good. When performing the above, it is desirable that the value of the previous time is not used as a constraint as it is in the first embodiment, but that the prediction range is widened to some extent.

<Sixth Embodiment>
Next, a camera parameter estimation device according to a sixth embodiment of the present invention will be described. In the sixth embodiment, an example of a specific application of the camera parameter estimation device according to the present invention will be described. In this embodiment, a scene that can be expected to be most effective by using the camera parameter estimation apparatus according to the present invention and a specific application example thereof will be described. Since the camera parameter estimation apparatus according to the present invention receives a video from a plurality of viewpoints and outputs a camera parameter of each camera, it can be used to generate a free viewpoint video that enables video viewing from various viewpoints.

  In addition, the camera parameter estimation apparatus according to the present invention is a dynamic object when the distance between cameras is large compared to a method using no constraint and a conventional method using an image-based time series constraint. In addition, it can be expected that the effect is great when the shooting camera is a dynamic camera. Therefore, when creating a free viewpoint video by shooting a dynamic subject with a dynamic camera, for example, when preparing multiple cameras on a crane at a live event, etc., and synthesizing a live free viewpoint video from those videos In addition, it is conceivable to synthesize a free viewpoint video by collecting videos taken using a commercially available handheld camera or smartphone for events such as a school performance or athletic meet.

  In order to apply the camera parameter estimation apparatus according to the present invention, it is basically desirable that each camera has an overlapping area with at least one camera other than itself. Therefore, by preparing one camera that reflects the whole at a school or the like, it is possible to assume a situation in which the camera that reflects the school or the event has at least one overlapping area. Video captured by these cameras is collected and synchronized, and the camera parameters are estimated by applying the present invention to each video. Furthermore, even when there is no overlapping area, that is, even when a camera that reflects the whole image is not prepared, as described in the fifth embodiment, if those cameras are provided with an external sensor such as a gyro sensor or GPS, the camera parameters are set. It is also possible to estimate.

<Seventh embodiment>
Next, a camera parameter estimation device according to a seventh embodiment of the present invention is described. Here, an example of a multi-view video application using only camera parameters obtained by applying the camera parameter estimation apparatus according to the present invention will be described. FIG. 7 is an explanatory diagram showing an example of a multi-view video application using only camera parameters obtained by applying the camera parameter estimation device according to the present invention.

  First, in this application, it is assumed that a multi-view video shot from a plurality of cameras is viewed. Even if a multi-view video is usually obtained, it is not clear where each video is taken from, and it is necessary for the viewer to make a decision according to the time. On the other hand, since the camera parameters of each camera that can be estimated by applying the present invention include the position and orientation of each camera, the present invention can grasp their positional relationship. This application presents this positional relationship on a touch panel using a bird's eye view or the like for the camera selection display of FIG. 7, and when the viewer selects a desired viewpoint, the video from that viewpoint is displayed for video presentation. It will be shown on the display. The camera selection display allows viewers to see at a glance from which location each video was shot, and from which location the other camera was shooting at this time. It is expected to save time and effort to determine where the image is taken from, and to assist in changing the viewpoint.

  As described above, in the natural feature point-based calibration for the video, this problem is related to the problem of miscorrespondence of feature points (and the accompanying decrease in camera parameter estimation accuracy) that occurs as the difference in position and angle between cameras increases. The following effects obtained by using the embodiment can be obtained.

  By using camera parameter-based time series constraints and limiting the search range of each feature point, it is possible to reduce miscorrespondence and improve camera parameter estimation accuracy. Unlike image-based time-series constraints, it is not necessary to place constraints on captured images (feature points must be detected continuously in the time direction, etc.), so for all scenes where feature points can be detected Can be applied.

  In addition, since this camera parameter based time series constraint does not impose a time series constraint on the acquired image, when the position of the detected feature point is greatly different in each image at time t and time (t + 1), In other words, it is more effective than image-based time-series restrictions in situations such as a stage or live video where lighting is severely crushed and a video of a moving object such as a person zoomed.

  The camera parameter estimation device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  The present invention can be applied to applications in which it is essential to improve the estimation accuracy of camera parameters to be estimated.

  DESCRIPTION OF SYMBOLS 1 ... Multi viewpoint video input part, 2 ... Feature point detection part, 3 ... Feature point matching part, 4 ... Camera parameter estimation part, 5 ... Camera parameter prediction part, 6. Camera parameter output unit, 7 ... external sensor data input unit, 8 ... camera parameter estimation unit

Claims (6)

A camera parameter estimation device that estimates camera parameters of a plurality of cameras from videos captured by a plurality of cameras,
A feature point detector for detecting feature points from the video;
A feature point associating unit for associating the detected feature points;
A camera parameter estimator that calculates an estimated value of the camera parameter from the associated feature points;
A camera parameter prediction unit that calculates a predicted value of the camera parameter based on the estimated value of the camera parameter that has already been estimated;
The feature point association unit includes:
An apparatus for estimating a camera parameter, characterized in that association is performed using a restriction relating to association of feature points derived from predicted camera parameters. The camera parameter prediction unit sets the camera parameter prediction value immediately before the estimated value of the camera parameter,
The camera parameter estimation apparatus according to claim 1, wherein the restriction used in the feature point association unit is an epipolar restriction. The camera parameter prediction unit calculates a probability distribution of the predicted value of the camera parameter based on the estimated value of the camera parameter that has already been estimated,
The feature point associating unit associates feature points based on an epipolar constraint determined for each predicted value of the camera parameter appearing in the probability distribution and a probability given to the predicted value of the camera parameter. The camera parameter estimation apparatus according to claim 1, wherein:   4. The camera parameter estimation according to claim 3, wherein the probability distribution calculated by the camera parameter prediction unit assigns a weight that increases toward the immediately preceding estimated value of the camera parameter. apparatus. A sensor for measuring a change in the position and orientation of the camera;
When the camera parameter estimation unit cannot calculate the estimated value of the camera parameter from the associated feature points, the estimated value of the camera parameter calculated in the past and the change in the position and orientation of the camera measured by the sensor The camera parameter estimation apparatus according to claim 1, wherein an estimated value of the camera parameter is calculated based on a degree of the camera parameter.   A camera parameter estimation program for causing a computer to function as the camera parameter estimation apparatus according to any one of claims 1 to 5.

Images