JP3347087B2 - 3D structure reconstruction method from 2D video - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for recognizing a translational motion and a rotational motion from a two-dimensional moving image of a video camera or the like, restoring a three-dimensional structure, and knowing the actual size of an object.

[0002]

2. Description of the Related Art A moving image picked up by a video camera or the like contains important information on the movement of an observer and the structure of an object. For example, if data is taken by moving around a building, which is a video camera while walking in a city, data is acquired, and a three-dimensional structure can be restored later by a computer and viewed from any angle. Therefore, restoration of a three-dimensional structure from such a two-dimensional moving image is one of the important issues of computer vision, and if this technology is established, three-dimensional modeling, tracking, passive navigation,
It can be applied to many fields such as robot vision.

[0003] Research in this area can be broadly classified into two types. One is a means for using the correspondence between points on the image obtained at different times, and the other is a means for using the speed (optical flow) on the image. The means of using optical flow compared to the former is
(1) The optical flow can be obtained more easily than the corresponding point on the image, and (2) the speed can be obtained from the optical flow but cannot be obtained from the corresponding point.

Further, as means for restoring an object structure from an optical flow on an image, there are (1) a method using a parallel projection image and (2) a method using a perspective projection image. The former is an approximation of the latter, and this approximation only holds when the object is far from the camera. Therefore, the latter means using the perspective projection image can obtain higher accuracy.

As a means for restoring the structure of an object from an optical flow of a perspective projection image, it has conventionally been necessary to solve a system of nonlinear equations by an iterative method unless special assumptions are made. A special assumption is that when the point to be observed lies on a plane, the movement is only rotation or only translation. Therefore, in general, it is necessary to solve a system of nonlinear equations. However, in this case, there are problems that the uniqueness of the solution is not guaranteed and that a search by an iterative method is required.

[0006] In order to solve these problems, the inventors of the present invention have developed a structure simply by solving a linear equation using an optical flow image obtained by perspective projection from a point performing rigid motion. A method for restoration is proposed (“Reconstruction of 3D motion parameters and structure by linear calculation from optical flow image”, Transactions of the Society of Instrument and Control Engineers, Vo
l. 34, no. 5,438 / 444 (1998)).
According to this method, it is not necessary to solve the nonlinear equation, the uniqueness of the solution is guaranteed, and the accuracy can be easily improved by increasing the number of observation points.

[0007]

However, the above-mentioned three-dimensional structure restoring method has the following problems. (1) As illustrated in FIG. 1, when the depth of the object 1 is relatively small with respect to the distance between the camera 2 and the object 1,
When the motion is small, the translational motion (A) and the rotational motion (B) have a similar optical flow on the image, and it is very difficult to distinguish them. As a result, it is difficult to correctly restore the structure of the object. (2) The size of the object and the translational movement of the camera are obtained only as relative values, and the absolute size is not known. This is because the same result is obtained on an image when a small object moves nearby and moves a little, and when a large object moves far and largely.

The present invention has been made to solve such a problem. That is, the object of the present invention is to eliminate the need to solve nonlinear equations, guarantee the uniqueness of the solution,
Another object of the present invention is to provide a method for restoring a three-dimensional structure from a two-dimensional moving image in which a translational motion and a rotational motion can be easily distinguished and an absolute size of an object can be obtained.

[0009]

The above-mentioned problems are fundamental and cannot be solved only by image processing. But,
These disadvantages can be avoided by supplementing information by adding an acceleration / angular velocity sensor to the camera. The present invention is based on such a new idea.

That is, according to the present invention, a sensor for measuring acceleration and angular velocity is integrally attached to a camera (2) for capturing a moving image of a stationary object (1), and the moving image and the acceleration and angular velocity are integrated. to synchronize the data recording to give al
Moving image so that the angular velocity becomes 0 based on the extracted angular velocity data
Is processed into an image containing only translational motion.
3D structure is restored from the image, and the obtained acceleration data
Calculated from the moving image including only the translational motion
From the ratio s to the speed, this ratio s
The target object is calculated by integrating the size calculated using the travel distance of
A three-dimensional structure restoring method for determining the size of the three-dimensional structure.

An object whose three-dimensional structure (shape) is desired to be restored is stationary, and an image is taken around the object while moving a camera provided with an acceleration / angular velocity sensor, and the image is synchronized with the output of the acceleration / angular velocity sensor. . Next, acceleration
Since angular velocity data can be obtained from the output of the angular velocity sensor,
By performing an operation to cancel the rotation of the moving image, a moving image including only the translational motion can be obtained. As a result, the degree of freedom of the movement of the camera is reduced, so that the movement of the camera and the reconstruction of the target object can be performed from the moving image under a limited situation. As a result, the movement of the camera can be more accurately recognized than before, and the accuracy of the restored value of the three-dimensional structure of the object is improved.

Furthermore, the same unknown scale s, which will be described later, is applied to the values obtained from the moving image for the camera speed and the structure of the object. Therefore, the actual speed obtained from acceleration data, by comparing the speed of the image obtained from a moving image that contains only translational motion can be determined the unknown scale s as the ratio, camera in this ratio s Unit time
The size of the target object can be obtained by integrating the sizes obtained in units of the travel distance of the object.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described specifically. 1. Definition of Coordinate System In the following description, vectors representing positions and velocities will be used. Regarding the vectors, three types of reference coordinate systems are defined to represent element values. The first is a world coordinate system fixed in the world. When a vector is expressed based on this, a B is added to the right shoulder. This is hereinafter referred to as a “reference world coordinate system”. The second is a camera coordinate system that moves with the camera. In this case, C is attached to the right shoulder. Third, a world coordinate system fixed to the world is set so as to overlap the camera coordinate system at each time. In this case, W is attached to the right shoulder. This coordinate system is hereinafter referred to as an “instant-world coordinate system”.

There is only one reference world coordinate system throughout the time when a moving image is taken, but the instant world coordinate system changes at each time as the camera moves. For example, the reference word is a camera - if running in v ^B relative to field coordinate system B _1, the camera coordinate system is so moving together with the camera, a naturally v ^C = 0. Further, the instantaneous word took as to overlap the camera coordinate system of the instant - speed viewed from the field coordinate system W ₂ becomes v ^W = Rv ^B. Here, R is a rotation representing the conversion from B ₁ to W ₂ . Since the values of the elements of the vector do not change even if the reference position is translated, the relationship between the coordinate systems can be established only by rotation. The outputs of the acceleration / angular velocity sensor are acceleration a ^W and angular velocity ω ^W with respect to the instantaneous world coordinate system.

[0015] 2. Method for Reconstructing Object Structure from Optical Flow at One Time FIG. 2 is a diagram showing the relationship between the object and a change in the position of the camera.
The unit vector from the camera center 3 to the point 4 on the object is q ^W , and the vector representing the movement of the camera center is δu ^W
Then, there is a relationship as shown in FIG. 2 between the time t and the time t + δt slightly later than the time t. That is, q ^W (t), q ^W (t +
Since δt) and δu ^W are on the same plane, the scalar triple product becomes 0, so (Equation 1) holds.

(Equation 1)

A unit vector q ^C representing the direction of a point on an object in a camera coordinate system obtained directly from an image obtained by observing the equation (1) with a camera and its time derivative (that is, optical flow) dq ^C / dt are represented by When rewritten using (Equation 2)
Becomes However, v ^W, ω ^W represents speed of the camera, the angular speed. This holds for each point on the object.

(Equation 2)

A vector (Equation 3) composed of v ^W and ω ^W elements is defined.

(Equation 3)

By transforming (Equation 2), an equation Gx = 0 is finally obtained using a matrix G obtained only from observation values (8 or more points are required). By solving this equation, v
^W and ω ^W are obtained. However, the scale of v ^W is unknown. The position of the point on the object is restored using v ^W and ω ^W obtained here. Where v ^{W of} unknown scale
, The scale of the object is also unknown. That is,
There is one unknown (s, which will be described later) for the speed and the entire position of the point, all of which are obtained by multiplying them. In other words, the restored value of the position of the point is obtained in units of the moving distance of the camera per unit time.

3. Method of merging restoration results at different times Since an optical flow is obtained from a moving image at each time, the object structure and the motion of the camera are restored at each time.
However, the restoration value at each time is obtained based on the instantaneous world coordinate system determined to match the camera coordinate system at the observed time, and the scale is obtained based on the camera speed at that time. As a result, even at the same point, the coordinate value differs depending on the time.

However, since the shape of the object itself does not change at each time, the restored values of the object between the times should overlap if the scale, translation, and rotation are appropriately converted. As a result, a scale, translation, and rotation representing the relationship of the coordinate system between the times are obtained. For example, FIG. 3 shows a three-dimensional structure of an object restored at a certain time, and FIG.
Is at time t1, and (B) is at time t2. When the object is stationary, the shape itself does not change at each time, so the restoration value of the object between each time can be overlapped by appropriately performing scale, translation, rotation conversion. .

By performing conversion between respective times using the obtained scale, translation, and rotation, and averaging the superimposed results, the accuracy of the shape of the object is improved. As for the speed, if only scale conversion is performed, v ^W (t) in which the scale ratio between times is in a correct relationship is obtained. If rotation is performed together with scale, the same coordinate system (the world coordinate system used as a reference for superposition) is used. Changes with time at)
^B (t) is obtained. Regarding the angular velocity, the scale is correctly obtained from the image, that is, since ω ^W (t) is known from the image before the fusion, if only rotation is performed, ω
^B (t) is obtained. However, even in this case, the camera speed at each time and the scale applied to the entire position of the restored point remain unknown.

[0022] 4. Method for Solving Problem (1) Using Acceleration / Angular Velocity Sensor The angular velocity is output from the acceleration / angular velocity sensor.
An image containing only translational motion is obtained. As a result, the degree of freedom of the movement of the camera is reduced, so that the movement of the camera and the reconstruction of the target object can be performed from the moving image under a limited situation. In other words, since ω ^W is obtained from the acceleration / angular velocity sensor in (Equation 2), it is sufficient to solve the equation (Equation 4) with H as a matrix obtained from the image and the output of the acceleration / angular velocity sensor.

(Equation 4) As a result, you can see the movement of the camera more accurately than before,
The accuracy of the restoration value of the three-dimensional structure of the object is improved.

[0023] 5. Method of Solving Problem 2 Using Acceleration / Angular Velocity Sensor Consider a method of obtaining an unknown scale by obtaining the speed of a camera using the output of an acceleration / angular speed sensor and comparing the obtained speed with the speed obtained from an image. Since this unknown scale is also the scale of the object, the size of the object is finally obtained. First, a method of obtaining the speed of the camera using the output of the acceleration / angular velocity sensor will be described. This can be theoretically written as Expression (5) using a vector expressed in the reference world coordinate system. Here, v ^B (t ₀ ) is the initial value of the velocity, a ^B is the acceleration, and g ^B is the gravitational acceleration.

(Equation 5)

Since the acceleration / angular velocity sensor is attached to the camera, the sensor output can be obtained with respect to the instantaneous world coordinate system which coincides with the camera coordinate system. Therefore, the expression (5) is modified to obtain the expressions (6) and (7). can get. Where R (t) is
It represents the rotation from the reference world coordinate system to the instantaneous world coordinate system at time t.

(Equation 6)

(Equation 7) This R (t) itself can be calculated as acceleration /
It is determined by integrating the angular velocity of the output of the angular velocity sensor over time.

The velocity obtained from the acceleration / angular velocity sensor based on the above principle is expressed as v ^W _G (t), and the true velocity v ^W
The relationship with (t) is represented by (Equation 8). Here, b (t) represents the unknown initial velocity and the effects of drift.

(Equation 8)

B (t) changes little by little due to sensor noise, but since the value of the change is small, dt / db
(T) ≒ 0. On the other hand, the speed obtained from the image is represented by v ^W _I
If (t) is used, the scale is unknown, so that (Expression 9) is obtained. Therefore, the relationship of (Equation 10) is obtained.

(Equation 9)

(Equation 10)

FIG. 4 is a diagram showing the relationship between the speed obtained from the image and the true speed. As shown in this figure, when b (t) hardly changes over the measurement time, v ^W _I (t)
And a graph plotting v ^W _G (t) are substantially linearly arranged, and the scale s can be determined from the slope of the straight line.

Even if b (t) changes during the measurement time, the change per unit time is small.
Taking the time derivative of (Equation 10) gives (Equation 11), and s is obtained from this relationship.

[Equation 11] Since such Sukeruru s is obtained in the entire speed and restoring the position obtained from the images by the above method, finally, to the ratio s
The distance traveled by the camera per unit time was obtained as a unit.
If the size is integrated , the size of the target object can also be obtained.

As described above, according to the method of the present invention,
A sensor that measures acceleration and angular velocity is integrally attached to a camera that captures a moving image, and data of the moving image and the acceleration and angular velocity are recorded in synchronization. Further, based on the angular velocity data, the moving image is subjected to image processing so that the angular velocity becomes zero, thereby obtaining a moving image including only the translational motion, and the three-dimensional structure is restored from the moving image. Furthermore, the speed obtained from acceleration data, the ratio s between the speed obtained from a moving image that contains only translational motion, this
Is the ratio of camera movement distance per unit time to unit s
The size of the target object is obtained by integrating the sizes obtained by the above .

By synchronizing and capturing the output of the camera and the acceleration / angular velocity sensor, angular velocity data can be obtained from the output of the acceleration / angular velocity sensor. Is obtained. As a result, the degree of freedom of the movement of the camera is reduced, so that the movement of the camera and the reconstruction of the target object can be performed from the moving image under a limited situation. As a result, the movement of the camera can be more accurately recognized than before, and the accuracy of the restored value of the three-dimensional structure of the object is improved. Further, the same unknown scale s is included in the values obtained from the moving image for the camera speed and the structure of the object.
(Equations 9 to 11) are applied, but by comparing the actual speed obtained from the acceleration data with the speed on the image obtained from the moving image including only the translational motion, the unknown scale s is obtained as a ratio. And this ratio s
The distance traveled by the camera per unit time was obtained as a unit.
By integrating the sizes, the size of the object can be obtained.

It should be noted that the present invention is not limited to the above-described embodiment, but can be variously modified without departing from the gist of the present invention.

[0032]

As described above, the method for restoring a three-dimensional structure from a two-dimensional moving image according to the present invention does not require solving a nonlinear equation, guarantees uniqueness of the solution, and facilitates translation and rotation. And the absolute size of the object can be obtained.

[Brief description of the drawings]

FIG. 1 is a diagram showing a relationship between an object and a camera.

FIG. 2 is a relationship diagram between an object and a change in the position of a camera.

FIG. 3 is a diagram schematically showing a restored shape.

FIG. 4 is a relationship diagram between a speed obtained from an image and a true speed.

[Explanation of symbols]

 1 object 2 camera 3 camera center 4 point on object

────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Noboru Noboru 2271-130 Science Park Research and Development Center, Shimo-shi-dami-ji, Moriyama-ku, Nagoya-shi, Aichi Pref. RIKEN Bio-Mimetic Control Research Center (56) Reference References JP-A-9-81790 (JP, A) JP-A-11-306363 (JP, A) JP-A-10-23465 (JP, A) Toshiharu Mukai, Noboru Onishi, "3D Shape Model Creation" Sensor System Using Video Camera and Gyro Sensor, "Proceedings of the 4th Annual Meeting of the Virtual Reality Society of Japan, Japan, Virtual Reality Society of Japan, September 29, 1999, p. 213-216 Toshiharu Mukai, Noboru Onishi, "Reconstruction of 3D Motion Parameters and Structure by Linear Calculation from Optical Flow Image", Transactions of the Society of Instrument and Control Engineers, Japan, Society of Instrument and Control Engineers, May 31, 1998 Date, Vol. 34, No. 5, p. 438-444 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 1/00 315 G01B 11/24 G06T 7/00 G06T 7/20 G06T 7/20 100

Claims (1)

(57) [Claims] 1. A sensor for measuring acceleration and angular velocity is integrally attached to a camera for capturing a moving image of a stationary object, and data of the acceleration and angular velocity are synchronized with the camera. Record and move so that the angular velocity becomes 0 based on the obtained angular velocity data.
The image is processed into a moving image that contains only translational motion.
Of the three-dimensional structure from the moving image of the target, and the speed and translational motion obtained from the obtained acceleration data
From the ratio s to the speed obtained from the moving image containing
The distance traveled by the camera per unit time was obtained as a unit.
A three-dimensional structure restoring method, wherein a size of an object is obtained by integrating the sizes .