JP2002027495A - Three-dimensional image generating system, three- dimensional image generating method and three- dimensional information service system, and program providing medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional image generating system which can generate a three-dimensional image stably even in the outdoors, and a three- dimensional information service system. SOLUTION: A three-dimensional shape can be measured with high accuracy using a plurality of (more than one) portable terminals with camera by preparing bases for installing these portable terminals with camera and calibrating them. An image can be picked up with a stabilized luminance even in the outdoors by a 3D imaging mode where flash light is used while setting predetermined shutter speed and iris or the shutter speed or iris is adjusted automatically through a unit for measuring illuminance. Furthermore, an arrangement for transferring three-dimensional data measured through use of the portable terminal with camera or three-dimensional data stored in a server to other terminal is realized.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a system for generating a so-called three-dimensional image, a three-dimensional image generating method, and a three-dimensional information service system. More specifically, the present invention relates to a three-dimensional image generation system, a three-dimensional image generation method, and a three-dimensional information service system that enable stable 3D shape measurement even in the field by using a camera flash photography or a camera pedestal.

[0002]

2. Description of the Related Art A method of capturing a three-dimensional shape using light, an image (photograph), and the like is roughly classified into an active method (Active visio).
n) and Passive vision. Active methods include (1) measuring the amount of light reflected from the object and the arrival time by emitting laser light or ultrasonic waves and extracting depth information, and (2) special patterns such as slit light. Using a light source, a method of estimating the target shape from image information such as geometric deformation of the target surface pattern, and (3) forming contour lines by moiré fringes by optical processing,
There is a method of obtaining three-dimensional information. On the other hand, as a passive method, monocular stereoscopic vision, in which three-dimensional information is estimated from one image using knowledge about the appearance of an object, light source, illumination, shadow information, and the like, and similar to the human eye, A binocular (or multi-view) stereoscopic vision that reliably estimates the depth information of each pixel based on the principle of triangulation, an optical flow or the like is detected from a moving image scene, and the depth information is estimated from a field of the obtained optical flow. Motion stereoscopic vision and the like are generally known.

Until now, in order to accurately measure three-dimensional shapes of industrial products and the like, a practical measurement system for obtaining a real-time distance image using slit light, and a high-precision calibration of characteristics between a camera and a camera. Some three-dimensional sensing devices (or three-dimensional digitizers), such as a modified two-lens (or multi-lens) stereo system, have been announced.

[0004] In order to measure the three-dimensional shape of the surface of a subject using the same principle as that of detecting the shape and depth of a subject with both eyes, using images observed by a plurality of cameras with different viewpoints, The shape or depth of the target can be measured by associating each pixel between images based on the principle of triangulation. This is generally called "stereo vision" or "stereo three-dimensional image measurement".

[0005] The basic principle of stereo vision is triangulation, the most basic configuration of which is stereo vision with two cameras. That is, the same target is observed as an image by two cameras at different viewpoints, and the difference in the projection position on the image (generally, the displacement of the projection point between the images is referred to as parallax). The three-dimensional coordinate position of the object can be obtained. Therefore, finding the same object at different projection positions between the left and right images is the most important in stereo vision, and is generally called “corresponding pointing”.

On the other hand, the correspondence between the projection point positions between the images is as follows:
It is not at all free and, as shown in FIG. 1, for a point mb on one image, its corresponding point p1 or p2 or p3 will be different in the other image observed from another viewpoint. It exists at m1, m2, or m3 on a certain straight line Lp. Generally, this straight line is called an epipolar line.

Conventionally, methods of “corresponding scoring” that are often used are roughly classified into three: pixel-based matching, area-based matching, and feature-based matching. Pixel-based matching is searching for the correspondence of points in one image as it is in the other image
[References: (1) C. Lawrence Zitnick and Jon A. Webb:
Multi-baseline Stereo Using Surface Extraction, Te
chnical Report, CMU-CS-96-196, (1996)]. Area-based
Matching refers to searching for the correspondence of a point in one image using a local image pattern around the point when searching in the other image [References: (2) Okutomi, Kinide:
Stereo matching using multiple baseline lengths, IEICE Transactions D-II, Vol.J75-D-II, No.8, pp.1317-
1327, (1992), (3) Yokoyama, Miwa, Ashigahara, Koyanatsu, Hayashi,
Later: Stereo Camera System and Its Application, SR
F'97, (1997), (4) Kanade, Kimura: Video rate stereo machine, Journal of the Robotics Society of Japan, Vol.13, No.3, pp.3
22-326, (1995), (5) Kinade, Mosquito Field, Kimura, Kawamura, Yoshida,
Oda: Development of a video rate stereo machine, Journal of the Robotics Society of Japan, Vol.15, No.2, pp.261-267, (1997), (6)
Yamaguchi, Takachi, Iguchi: Stereo matching for stone image measurement using the adaptive window method, Humanities and Computers, Vol.32, No.10, pp.55-60, (1996), (7) Yokoya: Recent Special Issue on Signal Processing in Computers, Recent Topics on Computer Vision, Systems / Control / Information, Vol.38, No.8, pp.436
441, (1994)]. Feature-based matching is to extract features such as shaded edges from images and perform correspondence using only features between images [Ref: (8) HHBa
ker and TOBinford: Depth from edge and intensity
based stereo, In Proc. IJCAI'81, (1981), (9) I Ishiyama, Kakuho, Kawai, Ueshiba, Tomita: Searching for Correspondence Candidates in Segment-Based Stereo, IEICE Technical Report, Vol.96, No.13
6, (1997), (10) WELGrimson: Computational experi
ments with a feature based stereo algorithm, IEEE
Trans. PAMI, Vol.7, No.1, pp.17-34, (1985)].

[0008] A method of obtaining a corresponding point in stereo vision by general Area-based matching will be described. As shown in FIG. 2, in an image Im1 (x, y) observed by a reference camera, a local window W is set around a point of interest, and the window W is used as a template and observed by the other camera. The matching is performed within the set search range while changing the position on the epipolar line of the obtained image Im2 (x, y), and the degree of coincidence is obtained by the following equation.

[0009]

(Equation 1)

In the above equation, Δx and Δy are the amount of movement of the position on the epipolar line. When R (x, y) is the minimum, the degree of coincidence between the windows is the highest, and the movement amounts Δx and Δy of the template at that time are regarded as the disparity of the target point,
The three-dimensional shape or depth of the point of interest can be calculated by a triangulation formula. By repeating such a matching process, three-dimensional shape data corresponding to all pixels can be obtained.

In practice, it is very difficult to accurately determine correspondence between all parts on an image. This is because the correspondence between the images essentially has “ambiguity (Ambiguity)”. In the example of FIG. 2 as well, template matching is performed in a very small area when viewed from the entire image, and a part having the highest matching degree is found. However, there is a possibility that similar patterns exist in various places depending on the scene. high. Further, even if the same target in the scene is viewed, the patterns on the left and right images do not completely match due to geometric distortion when viewed from different viewpoints, differences in camera characteristics, and the like.

[0012]

Various methods have been proposed to solve these problems and to perform more accurate point matching between images. Can be determined more accurately, but not really enough. In particular, image density changes greatly due to illumination light due to changes in the shooting environment, and the positional relationship between cameras cannot always be kept exactly constant. At present, there is no robust three-dimensional shape measurement method or system that can respond to changes and the like.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a stereo image acquisition method independent of a photographing environment, and a robust three-dimensional measurement method and a measurement system capable of coping with a minute change in a camera positional relationship. As a specific application example, using a camera-equipped mobile terminal (two or more) owned by friends, using a camera flash or a camera pedestal at that location, stable 3D shape measurement even in the field To realize a stereo system and a three-dimensional information service system capable of performing the above.

[0014]

SUMMARY OF THE INVENTION The present invention has been made in consideration of the above problems, and a first aspect of the present invention is to execute three-dimensional shape measurement or three-dimensional image generation processing based on a stereo measurement method. A plurality of cameras for capturing images of a measurement target from different viewpoint directions, and a pedestal for setting the plurality of cameras in a fixed positional relationship, wherein at least one of the plurality of cameras is provided. The two cameras are mobile terminal cameras having communication means capable of transmitting captured image data to a server that performs three-dimensional shape measurement or three-dimensional image generation processing. Characterized by having a configuration for transmitting to the server an image captured by a camera and data necessary for three-dimensional shape measurement or three-dimensional image generation processing. In 3-dimensional image generating system that.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the three-dimensional shape measurement
The data required for the two-dimensional image generation processing includes a parameter indicating a positional relationship between the plurality of cameras installed on the pedestal.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the server associates identification data of the pedestal with a parameter indicating a positional relationship of a plurality of cameras installed on the pedestal. It has a table, and is configured to extract a parameter indicating a positional relationship of the camera based on identification data of a pedestal received from a mobile terminal camera having the communication unit, and execute three-dimensional shape measurement or three-dimensional image generation processing. It is characterized by the following.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the pedestal is a data signal for inputting / outputting control signals for setting photographing conditions of a plurality of cameras installed on the pedestal between the cameras. Characterized by having a road.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the pedestal has a data signal path for inputting / outputting a synchronization signal of a plurality of cameras installed on the pedestal between the cameras. Features.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the server executes a process of associating images captured by a plurality of cameras based on an epipolar line area having a set line width. It is characterized by having a configuration.

Further, in one embodiment of the three-dimensional image generation system of the present invention, the server has storage means for storing three-dimensional shape measurement data or three-dimensional image data based on an image received from the portable terminal camera. And
The present invention is characterized in that, based on an authentication process between terminals communicable with a server, an access approval / denial process for data stored in the storage unit is executed.

Furthermore, a second aspect of the present invention is a three-dimensional image generation method for executing a three-dimensional shape measurement or a three-dimensional image generation process based on a stereo measurement method, wherein a plurality of cameras are placed in a fixed positional relationship. An image capturing step of installing the camera on a pedestal to be set and capturing images of an object to be measured from different viewpoint directions with a plurality of cameras; captured image data and three-dimensional shape measurement via at least one of the plurality of cameras; Alternatively, there is provided a data transmission step of transmitting data necessary for the three-dimensional image generation processing, and a data processing step of executing a three-dimensional shape measurement or a three-dimensional image generation processing based on the received data in the server. 3D image generation method.

Further, in one embodiment of the three-dimensional image generation method of the present invention, the data necessary for the three-dimensional shape measurement or the three-dimensional image generation processing includes a positional relationship between the plurality of cameras installed on the pedestal. It is characterized by including the parameters shown.

Further, in one embodiment of the three-dimensional image generating method of the present invention, the server associates identification data of the pedestal with a parameter indicating a positional relationship between a plurality of cameras installed on the pedestal. It has a table, and extracts a parameter indicating a positional relationship of the camera based on identification data of a pedestal received from a mobile terminal camera having the communication means, and performs three-dimensional shape measurement or three-dimensional image generation processing. And

Further, in one embodiment of the three-dimensional image generating method of the present invention, in the image photographing step, a control signal for setting photographing conditions of a plurality of cameras installed on the pedestal is transmitted between the cameras via the pedestal. It is characterized by inputting and outputting with.

Further, in one embodiment of the three-dimensional image generating method according to the present invention, in the image photographing step, a synchronizing signal of a plurality of cameras installed on the base is input and output between the cameras.

Further, in one embodiment of the three-dimensional image generating method according to the present invention, the data processing step performs the associating process between images photographed by a plurality of cameras based on an epipolar line region having a set line width. It is characterized by executing.

Further, in one embodiment of the three-dimensional image generation method according to the present invention, the access permission / denial processing for the data stored in the storage means is executed based on the authentication processing between the terminals capable of communicating with the server. Features.

Further, a third aspect of the present invention is a three-dimensional information service system for providing a three-dimensional shape measurement or a three-dimensional image generation processing service based on a stereo measurement method. A plurality of cameras owned by a user that shoots images from a user, and a pedestal owned by the user that sets the plurality of cameras in a fixed positional relationship. At least one of the plurality of cameras converts captured image data of a user. A mobile terminal camera having communication means capable of transmitting to a contract server, wherein the mobile terminal camera having the communication means is capable of processing captured images captured by the plurality of cameras and three-dimensional shape measurement or three-dimensional image generation processing. The server transmits necessary data to the server, and the server performs three-dimensional shape measurement or three-dimensional image generation processing based on the received data. The three-dimensional information service system characterized by having a configuration for executing certain.

Further, in one embodiment of the three-dimensional information service system of the present invention, the server has storage means for storing three-dimensional shape measurement data or three-dimensional image data based on an image received from the portable terminal camera. Then, based on an authentication process between terminals capable of communicating with the server, an access approval / denial process for data stored in the storage means is executed.

Further, a fourth aspect of the present invention is a program providing medium tangibly providing a computer program for causing a computer system to execute three-dimensional shape measurement or three-dimensional image generation processing based on a stereo measurement method. The computer program is arranged on a pedestal that sets a plurality of cameras in a fixed positional relationship, and an image capturing step of capturing images of a measurement target from different viewpoint directions by the plurality of cameras; and A data transmission step of transmitting photographed image data and data necessary for three-dimensional shape measurement or three-dimensional image generation processing via at least one camera; and three-dimensional shape measurement or three-dimensional shape measurement based on received data in the server. A data processing step of executing a two-dimensional image generation process. In the program providing medium.

A program providing medium according to a fourth aspect of the present invention is a medium for providing a computer program in a computer-readable format to a general-purpose computer system capable of executing various program codes, for example. . The form of the medium is not particularly limited, such as a storage medium such as a CD, an FD, and an MO, and a transmission medium such as a network.

Such a program providing medium defines a structural or functional cooperative relationship between the computer program and the providing medium for realizing the functions of a predetermined computer program on a computer system. Things. In other words, by installing the computer program into the computer system via the providing medium, a cooperative operation is exerted on the computer system, and the same operation and effect as the other aspects of the present invention can be obtained. You can do it.

Still other objects, features and advantages of the present invention are:
This will become apparent from the following detailed description based on the embodiments of the present invention and the accompanying drawings.

[0034]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In a three-dimensional image generating system and a three-dimensional image generating method according to the present invention, a basic principle of a stereo method as a three-dimensional image applied is based on the stereo method of FIG. It is a thing. The configuration of FIG. 1 will be briefly described. FIG. 1 is a conceptual diagram of a general stereoscopic camera arrangement and depth measurement based on the triangular principle as described above. Three points P1, P in three-dimensional space
2. When observing P3 with camera 1 (referred to as a reference camera), it is projected at the same position mb on the image plane, but when observing with camera 2 (referred to as a detection camera), m1, m2, and m3 are observed on the image plane. Projected. Therefore, if the length of the camera lens and the positional relationship between the cameras (referred to as a base line) are known in advance, if the positional displacement (referred to as parallax) between mb and m1, m2, and m3 can be detected, triangulation measurement can be performed. Based on the principle, the distances (called depths) between the camera and P1, P2, P3 can be measured.

FIG. 2 is a conceptual diagram of a method for obtaining a corresponding point by general template matching in order to detect a positional shift (parallax) between mb and m1, m2, and m3.
A search window set by setting a local window W around a point of interest (x, y) in the reference image Im1 and using the window as a template to change the position on the epipolar line of the image Im2 observed by the detection camera Is performed, and the location having the highest matching degree is set as the position of the corresponding point, and the parallax is obtained. However, in practice, it is very difficult to accurately determine correspondence between all parts on an image. The correspondence between images is essentially “ambiguity (Ambiguit
In particular, since the image density changes greatly due to the illumination light due to the change in the shooting environment, and the positional relationship between the cameras cannot always be kept exactly constant, the actual At present, there is no robust three-dimensional shape measurement method or system that can cope with subtle changes in environmental light and camera positional relationship.

The present invention proposes a stereo image acquisition method independent of a photographing environment, and a robust three-dimensional measurement method and measurement system that can cope with minute changes in the camera positional relationship. As a specific application example, a stereo system capable of performing stable 3D shape measurement even outdoors by using a camera-equipped mobile terminal (mobile terminal camera) and using a camera flash or a camera pedestal at that location. And 3D information service system

FIG. 3 is a diagram illustrating a schematic configuration example of a three-dimensional image generation system, a three-dimensional image generation method, and a three-dimensional information service system according to the present invention.

FIG. 3 is a conceptual diagram of a three-dimensional shape measurement system using a camera-equipped portable terminal (portable terminal camera). A plurality of camera-equipped mobile terminals 301 and 302 are installed on a pedestal prepared in advance, and, for example, a person's face is photographed in an arbitrary lighting environment, for example, natural light in the outdoors, or a normal lighting environment in a room. . When a mobile terminal with an arithmetic function is used, if a predetermined three-dimensional shape measurement program is applied using the arithmetic function, the three-dimensional shape of the face can be measured on the spot, but the mobile terminal without the arithmetic function Cannot measure the three-dimensional shape.

Therefore, the portable terminals with cameras 301, 30
From 2, a plurality of captured images are automatically transmitted to a predetermined server 303. Automatic transmission, for example,
By executing the mode setting as the 3D photographing mode in the camera-equipped mobile terminals 301 and 302, the photographing image photographed by each camera and the parameters necessary for the three-dimensional shape measurement can be transmitted to a predetermined server by a transmission processing program. This is realized by adopting a configuration in which the program is executed.

The server 303 automatically performs a three-dimensional shape measurement process using the images sent from the camera-equipped mobile terminals 301 and 302, and converts the obtained three-dimensional shape data of the face to the corresponding data attached to the server. Or send it back to the user's mobile device. The three-dimensional image stored in the server 303 is
A configuration is possible in which access is enabled by user password authentication or the like and three-dimensional measurement data is transmitted to friends via the server 303.

By the way, in order to accurately measure the three-dimensional shape of an object, it is necessary to perform accurate correspondence between images as described with reference to FIGS. Therefore, (1) a robust imaging method that does not depend on the illumination light of the imaging environment, (2) calibration of the positional relationship between the camera parameters and the pedestal, and (3) a robust imaging method that can cope with minute changes in the positional relationship between the cameras. A stereo matching method is required.

FIG. 4 is a diagram illustrating a plurality of examples of a robust photographing method that does not depend on illumination light in some photographing environments. In each example of FIG. 4, two cameras are set at positions where the reference camera and the detection camera are separated by a predetermined distance. Each camera position is set by the base 40 for the form terminal installation,
The position is determined to be a predetermined distance and apart.

FIG. 4A shows that the reference camera 41 has an illuminance measuring device 4.
11, the illuminance measuring device 411 measures the illuminance of ambient light such as sunlight and illumination, and automatically sets the shutter speed of the reference camera 41 and the detection camera 42 based on the measured value. That is, the configuration in FIG. 4A is a shutter speed priority system with a fixed aperture. Note that the illuminance measuring device may be provided in one of the reference camera 41 and the detection camera 42 to set the shutter speed of both, or may be provided separately for the reference camera 41 and the detection camera 42. And the shutter speed may be independently set.

FIG. 4B is a configuration diagram in which the illuminance of the ambient light is measured by the illuminance measuring device 431 in the reference camera 43, and the aperture values of the reference camera 43 and the detection camera 44 are automatically set based on the measured values. Show. That is, the aperture priority method is a fixed shutter speed. In addition, the illuminance measuring device may be provided in one of the reference camera 43 and the detection camera 44 to set the aperture value of both, as in FIG. It is also possible to provide an independent illuminance measuring device and set the aperture value independently.

FIG. 4C shows that a strobe 451 is provided in the reference camera 45 to eliminate the influence of ambient light, and the shutter speed and aperture of the reference camera 45 and the detection camera 46 are fixed. FIG. 3 shows a configuration diagram for capturing a face image. It should be noted that the strobe light may be provided in one of the reference camera 45 and the detection camera 46 as in (a) and (b), or the strobe may be provided in the reference camera 45 and the detection camera 46.
May be provided with independent strobes.

In each of the above-described photographing methods, the reference camera and the detection camera need to operate in synchronization. The synchronization signal is input and output via a reference camera and a base for setting the camera. The pedestal may be separate from the portable terminal or, as will be described later, be configured to be attached to one of the portable terminals as a folded configuration or a drawer configuration.
Various control signals such as the above-described shutter speed, aperture, strobe, synchronization signal, and the like are configured to be input and output by connecting the pedestal and the portable terminal with the connector.

FIG. 5 shows an example of a pedestal structure separate from the camera. FIG. 5 shows two (FIG. 5 (a)) or three (FIG.
FIG. 3B is a conceptual diagram of a pedestal applied to stereo vision using a camera of FIG. In FIG. 5A or 5B, the mobile terminals 1, 501, 503 are used for the reference camera, and the mobile terminals 2, 502, 504 and the mobile terminals 3, 505 are used for the detection camera. The position and angle of each camera are fixed on the pedestal, and the distance between the cameras is set to a predetermined distance so that the subject can be photographed in a specific direction.

The distance between the cameras mounted on the pedestal and the relative rotation angles α and β are determined by the lens length of the camera, the distance to the object to be measured (face), the measurement range, the CCD size, the number of pixels, and the requirements. It is determined by the measurement accuracy to be measured. For example, camera lens length = 8mm, distance from measurement target (face) = 600
mm, measurement range = 300 mm, CCD size = 1/3 "(4.8 mm x 3.6
mm), the number of CCD pixels = 640 × 480, and the measurement accuracy is about ± 1.0 mm, the distance between the pedestals should be about 120 mm, and the relative rotation angle α should be about 8.3 degrees. The exact distance between the pedestals and the relative rotation angles α and β are determined by calibration using a standard known image pattern.

As described above, the position and the direction of each camera can be set to a specific position on the pedestal, and parameters required for three-dimensional shape measurement can be determined based on the set camera position of the pedestal. Becomes Therefore, in the configuration of FIG. 3 described above, the mobile terminal equipped with the camera transmits the identification data of the pedestal to the server 303 together with the captured image, so that the server 303 matches the pedestal identification data registered in the server in advance. The corresponding parameter is extracted from the table in which the parameter data corresponding to each pedestal is associated based on the received identification data, and image processing using the parameter for the captured image can be executed.
In this configuration, the mobile device does not send any parameters,
Only by transmitting the pedestal identification data, the server can execute the measurement of the three-dimensional shape and the processing of generating the three-dimensional image. Note that each parameter of the camera itself is also
By transmitting the camera identification data to the server, it becomes possible to extract parameters corresponding to each camera registered in the server and execute the processing.

FIG. 6 shows a known image pattern used to obtain the camera parameters and the positional relationship between the camera setting bases, and a list of parameters to be obtained. The calibration method is described in, for example,
No. 3548 and Japanese Patent Application Laid-Open No. 11-53549. 6 (a), (b), and (c) show known checker patterns observed by the reference camera at distance 1 (close distance), distance 2 (intermediate distance of the measurement range), and distance 3 (far distance), respectively. It is an image.

Similarly, three checker pattern images at three distances can be observed by the detection camera. Then, by combining the reference image and the reference image observed at each distance, camera internal parameters (focal length, distortioncoefficient, distortion).
center, aspect ratio) and the positional relationship between the pedestals (that is,
Parameters (rotation angles, translational components) representing the positional relationship between the reference camera and the detection camera
Can be determined.

However, the positional relationship between the reference camera and the detection camera slightly changes due to the mechanical accuracy of the pedestal, the variation in the size of the camera body, and the like.
Does not always exist on the epipolar line obtained by the calibration at the time of shipment of the camera.

Therefore, in the present invention, taking into account factors such as the mechanical accuracy of the pedestal and variations in the camera, the epipolar line described above with reference to FIG. 2 is set as an epipolar line (region) having a line width. The corresponding point is searched based on the epipolar line having.

FIG. 7 is a conceptual diagram of a stereo matching method by searching on an epipolar line (region) having a line width. In the conventional matching method, a point of interest (x, y) in the reference image Im1 corresponds to a corresponding point (x ′, y ′) on the detected image.
In order to find, using a window W centered on the point of interest (x, y), using it as a template, perform template matching while moving on the epipolar line of the detected image, and find the point (maximum matching score) x ',
y ') is the corresponding point.

The line width of the epipolar line is determined in consideration of factors such as the mechanical accuracy of the pedestal and the variation of the camera. Using a window (template) centered on the point of interest (x, y) in the reference image Im1, template matching is performed while moving on the epipolar line region of the detected image, and the point (x ′,
y ') was taken as the corresponding point. By such association, a more accurate three-dimensional shape can be obtained.

The line width of the epipolar line is the epipolar line L (x, y) obtained by the calibration at the time of shipment.
The region R (x, y) is set as follows based on Here, it is assumed that the reference camera and the detection camera are installed along the horizontal direction.

[0057]

(Equation 2)

However, K is empirically determined in consideration of factors such as the mechanical accuracy of the pedestal and the variation of the camera. Using a window (template) centered on the point of interest (x, y) in the reference image Im1, template matching is performed while moving on the epipolar line region of the detected image, and the point (x ′, y ')
Was set as the corresponding point. By such association, a more accurate three-dimensional shape can be obtained.

FIG. 8 is a process flow showing a procedure for measuring a three-dimensional shape using a portable terminal with a camera. After capturing a face image as a three-dimensional shape measurement target, the image and the camera and pedestal parameters used at the time of capturing and the identification data of the camera and the pedestal are automatically transmitted to a predetermined server. Then, three-dimensional shape data of the face is obtained by using three-dimensional shape measurement processing software (for example, stereo matching method / software) installed on the server, and stored on the server. The three-dimensional shape data stored in the server can be processed and transferred by a specific user by authenticating a password or the like. Therefore, it becomes possible to transmit and receive the images taken by each of the friends via the server via the portable terminal with the camera which each of the friends has. on the other hand,
In a mobile terminal with an arithmetic function, various processes such as three-dimensional shape measurement and three-dimensional image generation can be performed on the spot without going through a server.

The processing flow of FIG. 8 will be described. In step S801, the camera-equipped mobile terminal 1 and the camera-equipped mobile terminal 2 are installed on a pedestal. Step S802, S
In step 803, an image of a target, for example, a face, is captured by each of the camera-equipped mobile terminals 1 and 2. At this time, the mode is set to the 3D mode. By setting to 3D measurement mode, as described in Fig. 4, use flash light with a fixed shutter speed and aperture, or have a device to measure illuminance and automatically adjust the aperture at a fixed shutter speed Shooting mode is set. Furthermore, in a camera-equipped mobile terminal that does not have an arithmetic function, 3D
According to the setting of the measurement mode, a process of transferring the captured image to a preset server is executed.

In steps S804 and 805, processing as 3D mode processing, that is, a captured image is transferred to a preset server. This processing is not executed when processing is performed in the terminal itself. When performing image transfer,
The pedestal parameters or identification data are also transmitted. The parameters of the pedestal are the inter-camera distance of the pedestal and the relative rotation angles α and β, as described with reference to FIG.
As described above, these are determined in advance by the lens length of the camera, the distance from the measurement target (face), the measurement range, the CCD size, the number of pixels, and the required measurement accuracy.

In step S806, the server
Based on image data and parameters received from each camera, three-dimensional shape measurement processing software installed on the server, such as a stereo matching method
The three-dimensional shape data of a photographing target, for example, a face, is obtained by software and stored on the server (S807).

Step S808 is a step of executing processing / transfer processing of the stored three-dimensional data by authenticating a password or the like. For example, a common key authentication method is used between the camera-equipped mobile terminal and the server.
Alternatively, a mutual authentication process using a public key authentication method is executed, a communication partner is confirmed, access to data stored in the server is permitted, and processing of the stored three-dimensional data is performed. Execute the transfer process. In this case, the communication terminal needs to have an encryption processing unit capable of performing mutual authentication processing. By configuring a cryptographic processing unit capable of mutual authentication processing in the camera-equipped mobile terminal, authentication processing may be performed to receive data from the server, and a cryptographic processing unit capable of mutual authentication processing may be configured. Authentication processing may be performed between the camera-equipped PC and the server to receive data from the server.

FIG. 9 is a diagram showing an embodiment using the above-described three-dimensional measurement method and system and experimental results. Fig. 9 (a),
(b) is a mobile terminal 1 with a camera (reference camera)
And an image observed by the camera-equipped mobile terminal 2 (detection camera). Stereo images were stored in jpg (j peg) format for fast transmission of image data. Using the two images sent to the server and the camera and pedestal parameters, stereo matching is performed, and FIG.
The result of (c) (distance image (x, y, depth) of the face) was obtained.
Further, conversion into xyz space can be performed to obtain three-dimensional data (x, y, z) of the face. FIGS. 9D and 9E are face images when the three-dimensional shape is viewed from a certain viewpoint.

FIG. 10 shows a configuration example of a camera-equipped mobile terminal having a built-in pedestal. FIG. 10A is a configuration example in which a pedestal 1003 is retractably stored in a lower part of a portable terminal 1001 having a camera unit 1002, and FIG. 10B is a portable terminal 100 having a pedestal 1006 having a camera unit 1005.
4 is an example of a configuration that is stored in a foldable manner on the side of No. 4.

FIGS. 10C and 10D show examples of use of FIGS. 10A and 10B, respectively. FIG. 10C shows a configuration in which a pedestal housed in a lower part of the portable terminal so as to be able to be pulled out is pulled out, set at a predetermined distance and separated from another camera. The pedestal has a groove or a connection terminal for positioning the camera, and has a configuration in which the relative positions of the two cameras are determined at a fixed distance and a fixed angle.

A signal path for executing the operations of the two cameras as synchronized processing is formed on the pedestal. For example, when the reference camera 1007 is set on the pedestal 1008, a connection terminal (not shown) of the pedestal 1008 is formed. ) Is connected to the reference camera 1007. Processing signals for the shutter speed, aperture, strobe, etc. described with reference to FIG. 4 are also input and output between the two cameras via a signal path in the base and connector terminals.

FIG. 10D shows a structure in which a pedestal which is foldably stored at the side of the portable terminal is turned down, and a predetermined distance and a distance are set to set another camera. As in (c), the pedestal has a groove or a connection terminal for positioning the camera, the relative position of the two cameras is determined at a fixed distance and a fixed angle, and various signals can be input / output. Have a configuration.

FIG. 11 shows another configuration example of the mobile terminal with a camera. FIG. 11 shows an example in which a camera and a pedestal are configured in a mobile personal computer (PC). FIG. 11 (a)
PC 1101 having pedestal 1103 having camera unit 1102
11B is a configuration example in which the pedestal 1106 is retractably stored in a keyboard portion of a PC 1104 having a camera unit 1105.

FIGS. 11C and 11D show examples of use of FIGS. 11A and 11B, respectively. FIG. 11C shows a configuration in which a pedestal that is foldably housed on the side of a display unit of a PC is tilted, and is separated by a predetermined distance to set a portable terminal as another camera. The pedestal has a groove for positioning the camera, or a connection terminal,
It has a configuration in which the relative positions of the two cameras are determined at a fixed distance and a fixed angle.

A signal path for executing the operations of the two cameras as a synchronized process is formed on the pedestal. For example, when the portable terminal 1107 is set on the pedestal 1108, a connection terminal (not shown) of the pedestal 1108 is formed. ) Is connected to a mobile terminal 1107 as a reference camera. Processing signals for the shutter speed, aperture, strobe, etc. described with reference to FIG. 4 are also transmitted via the signal path in the base and the connector terminals.
Input and output between two cameras.

FIG. 11D shows a configuration in which a pedestal housed so as to be able to be pulled out on the side of the keyboard portion of the PC is pulled out, set at a predetermined distance and separated from another camera.
As in (c), the pedestal has a groove or a connection terminal for positioning the camera, the relative position of the two cameras is determined at a fixed distance and a fixed angle, and various signals can be input / output. Have a configuration.

FIG. 12 shows a three-dimensional image generation processing flow in the configuration of FIG. 10 and the configuration of FIG. FIG.
FIG. 12A is a three-dimensional image generation processing flow in the configuration of FIG. 10, and FIG. 12B is a three-dimensional image generation processing flow in the configuration of FIG.

A description will be given of the processing flow of FIG.
First, in step S1201, the 3D measurement mode in the mobile terminal is set. As described in FIG. 4, the 3D measurement mode uses a flash light with a constant shutter speed and aperture, or has a device for measuring illuminance,
This is a shooting mode in which the aperture is automatically adjusted at a constant shutter speed. In step S1202, the camera of the mobile terminal using the pedestal (attachment) is set as the reference camera, and the other camera is set as the reference camera. In step S1203, the shooting method of the reference camera is set, and the shooting method and parameters of the reference camera are set through the pedestal (attachment). Note that the parameters related to the shooting method mean shutter speed, aperture, synchronized shooting timing, and the like. In step S1204,
Synchronous shooting is performed by two mobile terminals. Step S
In step 1205, the captured image, camera internal parameters, and positional parameters determined by the pedestal are transferred to a device having a 3D processing function, and 3D shape information is obtained. Specifically, the data may be transferred through a wireless network or the like, or may be processed by a mobile terminal having a calculation function.

The processing flow of FIG. 12B will be described. The processing in FIG. 12B is a processing flow based on the premise that a configuration in which a three-dimensional image generation processing program is stored in the PC in the configuration including the PC described in FIG.
First, in step S1211, the 3D
Execute the measurement mode (software). Step S1212
Then, the 3D measurement mode, the photographing method, the parameters, and the like are transferred from the personal computer to the portable terminal through the cable inside the pedestal (attachment) and set. Step S1
In step 213, synchronous photographing is performed by the personal computer with camera and the camera on the portable terminal, and the photographed image on the portable terminal, camera parameters, and the like are transferred to the personal computer through the attachment. In addition, since one relationship between the two cameras is determined by the pedestal (attachment), the parameters are set on the personal computer side. In step S1214, the 3D shape is measured and displayed by the 3D measurement software on the personal computer.

In the above-described embodiment, the configuration has been described in which each of the portable terminal cameras transmits data to the server. However, a plurality of images taken by a plurality of cameras are transmitted to one portable terminal via a base. Alternatively, the configuration may be such that all the captured images are transmitted from the one terminal to the server.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the scope of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0078]

As described above, according to the three-dimensional image generation system, the three-dimensional image generation method, and the three-dimensional information service system of the present invention, the following various effects can be obtained. (1) Set a fixed shutter speed and aperture and execute shooting using flash light, or
Alternatively, it is possible to take an image with a stable luminance value even outdoors by using a 3D photographing mode having a device for measuring illuminance and automatically adjusting a shutter speed or an aperture. (2) A plurality of (two or more) camera-equipped mobile terminals are used, a pedestal on which the camera-equipped mobile terminals are installed is prepared in advance, and calibration is performed, thereby enabling high-accuracy three-dimensional shape measurement. (3) Using a mobile terminal with a camera, three-dimensional images can be shared, such as transferring measured three-dimensional data or three-dimensional shape data stored in a server to another terminal.
(4) To cope with small changes in the positional relationship between cameras,
By improving the conventional method, a stereo matching method using a corresponding point search method on an epipolar line having a certain line width enables high-accuracy three-dimensional measurement. (5) A plurality of users can download the 3D image data stored in the server or transmit the 3D image data stored in the server to a friend via the server by using encrypted password authentication or the like for the 3D shape data stored on the server. 3D information service is possible. (6) By storing a table in which the pedestal identifier and the parameter are associated with each other in the server, the parameter is selected based on the identification data transmitted together with the captured image from the camera-equipped mobile terminal, and the server can accurately select the parameter. It is possible to execute three-dimensional shape measurement and three-dimensional image generation processing.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating camera arrangement and epipolar lines in stereoscopic view.

FIG. 2 is a diagram illustrating a corresponding point search process by template matching.

FIG. 3 is a diagram illustrating an outline of a three-dimensional image generation system according to the present invention.

FIG. 4 is a diagram illustrating each method of the three-dimensional image generation system according to the present invention.

FIG. 5 is a diagram illustrating a configuration example of a pedestal applicable to the three-dimensional image generation system of the present invention.

FIG. 6 is a diagram illustrating camera parameter and pedestal setting position calibration processing applied in the three-dimensional image generation system of the present invention.

FIG. 7 is a diagram illustrating a configuration of an epipolar line having a line width applied in the three-dimensional image generation system of the present invention.

FIG. 8 is a processing flow illustrating a three-dimensional image generation processing procedure in the three-dimensional image generation system of the present invention.

FIG. 9 is a diagram illustrating an example of a three-dimensional image generation process in the three-dimensional image generation system of the present invention.

FIG. 10 is a diagram showing a configuration example and a usage example of a mobile terminal with a camera in the three-dimensional image generation system of the present invention (part 1).
It is.

FIG. 11 is a diagram showing a configuration example and a usage example of a mobile terminal with a camera in the three-dimensional image generation system of the present invention (part 2).
It is.

FIG. 12 is a flowchart illustrating an example of processing by a mobile terminal with a camera in the three-dimensional image generation system of the present invention.

[Description of Signs] 301, 302 Mobile terminal with camera 303 Server 40 Base 41, 43, 45 Reference camera 42, 44, 46 Detection camera 411, 431 Illuminance meter 451 Strobe 501-505 Mobile terminal with camera 1001, 1004.1007 Mobile terminal with camera 1002, 1005 Camera 1003, 1006, 1008 Base 1101, 1104.1107 PC with camera 1102, 1105 Camera 1103, 1106, 1108 Base

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Hideki Koyanatsu 1-14-10 Higashi-Gotanda, Shinagawa-ku, Tokyo F-term in Sony Kihara Laboratory (reference) 2F065 AA04 AA53 BB05 DD00 DD02 DD06 EE05 FF01 FF05 FF09 GG08 HH02 HH13 JJ03 JJ05 JJ09 JJ26 QQ38 RR07 SS02 SS13 UU03 UU05 5B057 CH14 DA07 DB03 DC09 DC32 5C020 AA12 5C061 AA29 AB04

Claims (17)

[Claims] 1. A three-dimensional image generation system for performing three-dimensional shape measurement or three-dimensional image generation processing based on a stereo measurement method, comprising: a plurality of cameras for capturing images of a measurement target from different viewpoint directions; And a pedestal for setting the cameras in a fixed positional relationship, wherein at least one of the plurality of cameras can transmit captured image data to a server that executes a three-dimensional shape measurement or a three-dimensional image generation process The portable terminal camera having communication means, the portable terminal camera having the communication means transmits the images taken by the plurality of cameras and data necessary for three-dimensional shape measurement or three-dimensional image generation processing to the server. A three-dimensional image generation system having a configuration for transmitting. 2. The data required for the three-dimensional shape measurement or three-dimensional image generation processing includes a parameter indicating a positional relationship between the plurality of cameras installed on the pedestal. 3D image generation system. 3. The server includes a table in which identification data of the pedestal is associated with a parameter indicating a positional relationship between a plurality of cameras installed on the pedestal. The three-dimensional image according to claim 1, wherein a parameter indicating a positional relationship of the camera is extracted based on the identification data of the pedestal to be received, and three-dimensional shape measurement or three-dimensional image generation processing is executed. Generation system. 4. The pedestal according to claim 1, wherein the pedestal has a data signal path for inputting and outputting control signals for setting photographing conditions of a plurality of cameras installed on the pedestal between the cameras. 3D image generation system. 5. The three-dimensional image generation system according to claim 1, wherein the pedestal has a data signal path for inputting and outputting a synchronization signal of a plurality of cameras installed on the pedestal between the cameras. . 6. The server according to claim 1, wherein the server executes a process of associating images captured by a plurality of cameras based on an epipolar line area having a set line width. 3D image generation system. 7. The server has storage means for storing three-dimensional shape measurement data or three-dimensional image data based on an image received from the portable terminal camera, and performs authentication processing between terminals capable of communicating with the server. 2. The three-dimensional image generation system according to claim 1, wherein the system is configured to execute access approval / denial processing for data stored in the storage unit. 8. A three-dimensional image generation method for performing three-dimensional shape measurement or three-dimensional image generation processing based on a stereo measurement method, wherein a plurality of cameras are set on a pedestal that sets a fixed positional relationship, and a measurement object is set. An image photographing step of photographing the image with a plurality of cameras from different viewpoint directions, and via at least one of the plurality of cameras,
A data transmission step of transmitting photographed image data and data necessary for three-dimensional shape measurement or three-dimensional image generation processing; and a data processing of executing three-dimensional shape measurement or three-dimensional image generation processing based on received data in the server A method for generating a three-dimensional image, comprising: 9. The data necessary for the three-dimensional shape measurement or the three-dimensional image generation processing includes a parameter indicating a positional relationship between the plurality of cameras installed on the pedestal. 3D image generation method. 10. The portable server having a table in which identification data of the pedestal is associated with a parameter indicating a positional relationship of a plurality of cameras installed on the pedestal. 9. The three-dimensional image generation method according to claim 8, wherein a parameter indicating a positional relationship of the camera is extracted based on the identification data of the pedestal to be received, and three-dimensional shape measurement or three-dimensional image generation processing is executed. 11. The camera according to claim 8, wherein in the image photographing step, control signals for setting photographing conditions of a plurality of cameras installed on the pedestal are input and output between the cameras via the pedestal. 3D image generation method. 12. The three-dimensional image generating method according to claim 8, wherein in the image photographing step, a synchronization signal of a plurality of cameras installed on a base is input and output between the cameras. 13. The data processing method according to claim 8, wherein in the data processing step, an associating process between images captured by a plurality of cameras is performed based on an epipolar line region having a set line width. Dimensional image generation method. 14. The method for generating three-dimensional images according to claim 3, further comprising the step of performing an access approval / denial process for data stored in said storage means based on an authentication process between terminals capable of communicating with a server. Item 9. The three-dimensional image generation method according to Item 8. 15. A three-dimensional information service system for providing a three-dimensional shape measurement or a three-dimensional image generation processing service based on a stereo measurement method, wherein a plurality of cameras owned by a user photograph an image of a measurement target from different viewpoint directions. And a pedestal owned by a user for setting the plurality of cameras in a fixed positional relationship, wherein at least one of the plurality of cameras is capable of transmitting captured image data to a user's contract server. A mobile terminal camera having communication means, the mobile terminal camera having the communication means transmitting images taken by the plurality of cameras and data necessary for three-dimensional shape measurement or three-dimensional image generation processing to the server. The server has a configuration for executing three-dimensional shape measurement or three-dimensional image generation processing based on received data. 3D information service system characterized and. 16. The server has storage means for storing three-dimensional shape measurement data or three-dimensional image data based on an image received from the portable terminal camera, and performs authentication processing between terminals capable of communicating with the server. 16. The three-dimensional information service system according to claim 15, wherein a configuration for performing an access approval / denial process for data stored in said storage means is executed. 17. A program providing medium for tangibly providing a computer program for causing a computer system to execute three-dimensional shape measurement or three-dimensional image generation processing based on a stereo measurement method, wherein the computer program is An image capturing step in which a plurality of cameras are installed on a pedestal that sets a fixed positional relationship, and images of a measurement target are captured by a plurality of cameras from different viewpoint directions, and via at least one of the plurality of cameras. ,
A data transmission step of transmitting photographed image data and data necessary for three-dimensional shape measurement or three-dimensional image generation processing; and a data processing of executing three-dimensional shape measurement or three-dimensional image generation processing based on received data in the server A program providing medium, comprising: a step;