JP5206366B2 - 3D data creation device - Google Patents

Description

  The present invention relates to a three-dimensional data creation device. Industrial application fields of the present invention include, for example, processing model data for receiving (inputting) image data taken by a digital camera or the like and creating a three-dimensional human image such as a chest image or a bronze image, or a CG character The creation of a three-dimensional data for an averter and the construction of a system using the three-dimensional data created by the three-dimensional human image processing data creation device from the reception of the image data.

  Traditionally, to create a bust or bronze statue, take an image of the subject's surroundings and place an order with a sculptor or founder. The sculptor manually works on the sculpture prototype based on the photograph and the actual model. The mold was made from this, and a plaster mold was made based on it, and the founder processed it into a copper image based on the plaster image, and finished it by coloring it. In the conventional method by such a sculptor, there is a problem that it is necessary for the user to be reluctant, and the sculptor takes a long time to create. Also, because it depends on the skill and sense of the sculptor, the artistry is high, but the fidelity (reality) to the model is not necessarily high.

In addition, various methods have been proposed for producing stereoscopic images and relief images (relief images) of human models, and taking images of the model as a subject and producing images based on these photos (photographed images). ing.
As an example of such a method, a model is photographed as a subject, or a parallel line screen (striped pattern) is projected on the model and photographed, and this photographic image is projected on the surface of a material such as clay, In order to trace this, a method of manually engraving and creating an image has been proposed (see, for example, Patent Document 1). In Patent Document 1, many cameras, projectors, and screens depicting a large number of parallel lines are arranged around the object, and a large number of images are obtained when the parallel line screen is projected onto the object. After shooting with the above camera, the photographed photographic image and the parallel line screen are projected onto the original material surface with a projector at a similar position, and the screen on the material and the line on the photograph overlap. Thus, a method for producing a stereoscopic photographic image by processing a material is disclosed.

  However, in the above-described stereoscopic photo image production method, photographing and sculpting work are performed manually, and although it is somewhat automated than the method by the sculptor, it is semi-manual, so it is inefficient and processing accuracy is low. And fidelity (reality).

  As a technique for solving such a problem of creating a three-dimensional image (three-dimensional image), there is a technique for measuring three-dimensional data from an image taken from a surrounding 360 degrees photographed by an automatic photographing machine or the like (for example, see Patent Document 2). ). In Patent Document 2, a large number of cameras are arranged around the object, and the same object is photographed. Then, on the original material surface with a projector similar to the camera, There has been disclosed a method for producing a stereoscopic photographic image by projecting a photographic image taken by a camera and processing the material so that the stripes on the material by the photographic material overlap each other.

  Based on such technology, a service for producing a chest image, a copper image, etc. is also carried out with an automatic cutting machine, a three-dimensional powder printer, or the like. In addition, recently, optical modeling and 3D powder printers have come to be used for prototyping of industrial products and parts and sample preparation.

  Further, there is a software technique for generating three-dimensional data from an image photographed by an automatic photographing machine (see, for example, Non-Patent Document 1).

“Software Technology for Easily Generating 3D Data from Digital Camera Images” SANYO TECHNIC REVIEW COL. 35 NO. 1 JUN. 2003 JP 54-114231 A Japanese Patent Laid-Open No. 01-113744

  However, as disclosed in Patent Document 2, even when three-dimensional data is automatically created from an automatic photographing machine, a 360-degree photographing stand for photographing a large number of images, active scan images, and the like from 360 degrees around or dedicated photographing. Since the camera is required, the user has to go to the place where the dedicated camera is installed to take pictures, which is troublesome. Further, in order to generate three-dimensional data from a photographic image, adjustment work by an expert, data conversion work for bust processing, and the like are necessary.

  A main object of the present invention is to provide a three-dimensional data creation apparatus that automatically creates three-dimensional data from a plurality of images obtained by photographing the same subject at an arbitrary angle.

In order to solve the above-described problems, in the invention described in claim 1, a database including standard model data of a stereoscopic image, image data acquisition means for acquiring image data of a plurality of multi-view images, and three-dimensional data generation A three-dimensional shape data generation program that causes the apparatus to function as a three-dimensional shape data generation unit that generates three-dimensional shape data from the plurality of multi-viewpoint images, and a plurality of image data acquired by the image data acquisition unit, respectively. A three-dimensional shape is obtained by extracting a human region and obtaining corresponding points between the feature portion of the face detected from the extracted human region and the standard model data of the database, and obtaining three-dimensional shape data by the three-dimensional shape data generation program a data generation control unit, and the facial expression change correction means the three-dimensional shape data corrected by the obtaining expression change model data of the face Wherein the expression change correcting means includes a correspondence table data that associates the combination of facial expression types and expression operation unit, the correspondence table data that associates muscle contraction portion expressions operation unit and the face, the face By expressing facial expressions as a combination of facial expression unit movements and converting each table unit movement to the contraction of the facial muscles of the corresponding part, the facial expression data showing emotions is corrected to neutral face data. Or a three-dimensional data creation device , wherein the neutral face data is corrected to facial expression data indicating emotions .

Further, in the invention according to claim 2, the image data acquisition means includes camera information acquisition means for acquiring the camera information when the camera information such as the shooting position and the shooting direction of the acquired image data is known, The standard model data of the three-dimensional image includes standard model data of a person's face and head, and standard model data of facial expression change. The three-dimensional data creation control means includes a facial feature portion detected from the extracted person region, When facial expression data is acquired from corresponding points with the standard model data of the person's face and the facial expression in the facial expression data is compared with the facial expression in the standard model data, and the change in facial expression in the facial expression data is greater than the threshold The facial expression is corrected to a predetermined facial expression based on the facial expression change model data, and if the camera information is known by the camera information acquisition means, the three-dimensional shape is corrected. The 3D shape data of the person is generated by the first 3D shape data generation program included in the data generation program, and the second information included in the 3D shape data generation program when the camera information is unknown. The three-dimensional data creation apparatus according to claim 1, wherein the three-dimensional shape data of the person is generated by the three-dimensional shape data generation program.
According to a third aspect of the present invention, the first three-dimensional shape data generation program uses a three-dimensional data generation apparatus to obtain image data of a plurality of multi-viewpoint images acquired by the image data acquisition unit and the camera. The three-dimensional data creation device according to claim 1, which functions as a three-dimensional shape data generation unit that generates three-dimensional shape data based on camera information acquired by the information acquisition unit.

According to a fourth aspect of the present invention, the second three-dimensional shape data generation program uses a three-dimensional data generation device based on image data of a plurality of multi-viewpoint images acquired by the image data acquisition unit. The three-dimensional data creation device according to claim 1, which functions as a three-dimensional shape data generation unit that generates three-dimensional shape data by a visual volume intersection method.

According to a fifth aspect of the present invention, the second three-dimensional shape data generation program uses a three-dimensional data generation device based on image data of a plurality of multi-viewpoint images acquired by the image data acquisition unit. The three-dimensional data creation device according to claim 1, which functions as a three-dimensional shape data generation unit that generates three-dimensional shape data by a factorization method.

Further, in the invention described in claim 6, the standard model data of the stereoscopic image includes the standard model data of the face and head of a person, the standard model data of facial expressions, the standard model data of clothing and hair, the 3 Dimensional data creation control means extracts head shape and facial features from each image data based on the facial feature detected from the extracted person region, and searches the standard model data of the person's face and head Then, 3D model data most similar to the shape of the head and the facial features is acquired, and the image data of the plurality of multi-viewpoint images are projectively transformed onto the curved surface of the acquired 3D model data to obtain the face and head. The three-dimensional data creation apparatus according to claim 1, wherein the three-dimensional shape data of the part is generated.

  According to the present invention, even if an image is taken at an arbitrary angle (and at an arbitrary time, clothes, facial expression), if there are a plurality of images showing the same person, three-dimensional data is automatically generated based on those images. As usual, you can shoot from a predetermined angle of 360 degrees around, and do not need a dedicated shooting stand or automatic camera, so you can go to the place of the automatic camera and the sculptor. However, it is possible to easily and inexpensively create a realistic bust and figure based on the three-dimensional data that is created simply by sending and transmitting photographs and image data.

  FIG. 1 is a diagram showing a configuration example of a 3D image creation commissioning system 100 using the 3D data creation apparatus of the present invention. The 3D image creation commissioning system 100 is a standard model data such as 3D shape standard model data. 3D data creation device 1 (see FIG. 3) that creates 3D data from image data of a plurality of photographs 10 taken at arbitrary different angles. Using an imaging terminal such as a mobile phone with camera 5 and a digital camera 6 connected to the 3D data creation device 1 via a communication network 4 such as the Internet, or 3D data created by the 3D data creation device 1 3D data generated by a display terminal such as a personal computer 7 that displays a 3D image and the 3D data generation apparatus 1 (in the embodiment, 3D model processing data) 3D image processing apparatus 8 (for example, 3D automatic cutting machine 8-1, stereolithography machine 8-2, or 3D powder printer 8-3) that automatically manufactures a 3D image using ).

  In FIG. 1, a three-dimensional data creation device 1, a standard model database 30, and a three-dimensional image processing device 3 are devices on the side of a contract company, such as an imaging terminal such as a camera-equipped mobile phone 5 and a digital camera 6, a personal computer 7, etc. The display terminal 3D image processing apparatus 8 is a consignor side apparatus.

  Here, the three-dimensional automatic cutting machine 8-1, the stereolithography processing machine 8-2, and the three-dimensional powder printer 8-3 are known three-dimensional image processing apparatuses, and the three-dimensional automatic cutting machine 8-1 is It is a device that automatically cuts metal, solid resin, glass, wood, etc. based on the created three-dimensional data to produce a three-dimensional image such as a breast image. This is a device that automatically produces a 3D image by laser processing UV curable resin based on the data. The 3D powder printer 8-3 uses a mixture of powder and adhesive based on the created 3D data. It is a device that produces a three-dimensional image by laminating using a printing technique.

  In the embodiment, the three-dimensional image processing apparatuses 8-1, 8-2, 8-3,... Have communication control for receiving data from the three-dimensional data creation apparatus 1 via the communication network 7. However, it is configured to connect to a terminal having a communication control function such as a personal computer 7 and the like, and a terminal having a communication control function such as a personal computer 7 from the three-dimensional data creation device 1 via the communication network 7 A three-dimensional image may be automatically produced using the three-dimensional data received.

  Further, as shown in FIG. 1, the three-dimensional data creation device 1 includes a three-dimensional image processing device 3 (for example, a three-dimensional image having substantially the same function as the three-dimensional image processing devices 8-1, 8-2, and 8-3). Connect automatic cutting machine 3-1, stereolithography machine 3-2, or 3D powder printer 3-3) to automatically produce 3D images using the created 3D data You may comprise.

  FIG. 2 is a process chart showing a 3D image contract creation process in the 3D image creation contract system 100 shown in FIG. In FIG. 2, the consignor uses the multi-viewpoint camera image data captured by the imaging terminal 5 or 6 (a plurality of pieces of image data captured from the same person at the same shooting date and time and at the same place at any different angle (FIG. 7B )) Or multiple different image data of the same person taken at different shooting dates and times (see Fig. 7 (a)), delivery date, details of commission, type of data to be commissioned, delivery conditions, etc. Are transmitted to the three-dimensional data creation apparatus 1 via the Internet 4 (process P1), instead of transmitting image data in this process 1, a plurality of photographs taken by the same person at arbitrary different angles or the same A plurality of photographs of people (photographing times may be different) 10 or portraits may be delivered to the consignor by mail or the like.

When receiving the image data, the three-dimensional data creation device 1 registers the received consignment conditions in the memory, performs authentication of the consignor, billing processing, etc., and stores the received image data in a temporary storage memory such as a RAM ( Process P2).
When a photograph or a portrait is received from the contractor by mail or the like in the process P1, the image input processing unit of the three-dimensional data creation device 1 is based on the instruction operation of the operator on the trustee side in this process 2. In step 14, a photograph or the like is read and image data is created, reception registration, entrustee authentication, and billing processing are performed, and the image data is stored in a temporary storage memory such as a RAM.

  Next, the three-dimensional data creation apparatus 1 detects each face area from the multi-viewpoint camera image data, normalizes the size, orientation, rotation, etc. of the face, and then performs “Lucas-Kanade method (gradient method)” , "Kanade-Lucas-Tomasi tracker" etc. to track the movement and difference of feature points, search for corresponding points, analyze the movement of each point, distance depth, camera position, edge motion, etc. The position data and the movement information are obtained, and shape data is obtained from an arbitrary multi-viewpoint image and camera information (see FIG. 12) (process P3).

  In addition, as a process P3, a “3D voxel (stereopixel) space” is divided into a number of 3D grids using “view volume intersection method in projective grid space” or the like, and silhouette images in each image from different viewpoints are 3D voxels. 3D data of the face and head may be generated by backprojecting to space (ie, determining whether each voxel is included in the silhouette when projected on each image) (see FIG. 13).

  Further, as process P3, (measurement matrix) = (movement matrix) × (movement matrix) in (shape matrix) and (shape matrix) are simultaneously obtained from corresponding points in a plurality of images by a “factorization method” or the like. Other methods such as a method of generating face and head 3D data from the (shape matrix) may be used (see FIG. 14).

  The 3D data creation device 1 checks the commission condition registered in the process 2 (process P4), and creates the 3D data on the commissioner's personal computer 5 or mobile phone 6 via the communication network 4 when commissioning the creation of the 3D data. The three-dimensional model data is transmitted and an invoice is issued (process P5).

  If the consignment conditions are creation of 3D model data for processing or 3D image (or prototype) production, 3D model data for processing corresponding to the type of processing equipment is created, and the consignment conditions are 3D model data for processing. In the case of delivery, the processing 3D model data created and transmitted to the contractor's 3D image processing device 8 (or the computer to which it is connected) via the communication network 4 based on the delivery conditions and the invoice is issued To do. If the delivery condition is delivery of a recording medium such as a CD, the created three-dimensional model data is recorded on the CD and an invoice is issued (process P6).

  Further, when the consignment condition is production of a three-dimensional image (or a prototype), the created three-dimensional model data for processing is transmitted to the three-dimensional image processing device 3-1, 3-2, or 3-3 on the trustee side. Then, processing is performed, and when processing is completed, a bill is issued (process 7).

  FIG. 3 is a diagram showing a configuration example of the three-dimensional data creation apparatus of the present invention. The three-dimensional data creation device 1 is composed of a computer having a processing capability similar to that of a personal computer. The three-dimensional data creation device 1 controls the operation of the entire three-dimensional data creation device 1 and, at the time of creating three-dimensional data, is based on a three-dimensional data creation program (described later). CPU 11 for controlling dimensional data creation processing, temporary storage memory 12 such as RAM that temporarily stores programs and data or used as a work area, camera data such as image data, shooting position and direction, etc. via communication network 4 such as the Internet Received from the transmission / reception processing unit 13 that transmits and receives data, the image input processing unit 14 that reads images such as photographs and design images and creates image data, the various programs 21 to 23, the face shape standard model database 30, and the consignor Storage memory that records the commissioned conditions, billing information, created 3D image data, etc. , Keyboard 15 for operation, the display unit 16 and, although not shown, includes a recordable medium recording apparatus in a recording medium such as a CD 3D data created. The image input processing unit 14 and the display unit 16 are not essential.

  Further, the three-dimensional data creation device 1 includes a three-dimensional image processing device 3 such as a three-dimensional image processing device 3-1, 3-2, 3-3, or a three-dimensional image processing device 8-1, 8-2, 8-3. An interface 17 for transmitting the created three-dimensional data to the three-dimensional image processing apparatus 8 such as the above may be provided.

  FIG. 4 is a diagram illustrating a configuration example of the three-dimensional standard model database 30. The three-dimensional standard model database 30 includes three-dimensional standard model data 31 of a face (a person's face and head) and a three-dimensional standard of facial expression change. The model data 32 and the three-dimensional model data 33 such as clothes and hairstyle are stored in a systematic and searchable manner. In the example of FIG. 3, the 3D standard model database 30 is stored in the storage memory 2, but the memory storing the 3D standard model database 30 and the storage memory 2 may be separate.

  The three-dimensional standard model data 31 of the face has a three-dimensional shape derived from face and head feature points obtained from the face area of a plurality of images (arbitrary multi-viewpoint images) obtained by photographing the same person at an arbitrary angle. These are specifications (shape data (for example, a shape matrix made up of normalized coordinates, etc.) constituting feature points and the shape of the model) constituting the shape of the corresponding three-dimensional standard model of the face. Note that the three-dimensional standard model of the face corresponding to the three-dimensional shape of the face or head is not limited to one. The face three-dimensional standard model data 31 is usually associated with a plurality of three-dimensional models for each set of face and head feature points. FIG. 8 shows an example of a three-dimensional standard model of a face.

  The standard model data 32 of facial expression changes includes facial expression and facial expression (for example, opening / closing of eyes, size, position of eyeball, opening / closing of mouth, shape of mouth, swelling of nose, etc.). , Muscular sites that cause magnitudes of laughter, anger, relief, joy, sadness, etc., and an index indicating the degree of contraction). The standard model data 32 of facial expression change is used to correct or complement the facial expression of a human image obtained based on the three-dimensional standard model data 31 when creating the three-dimensional model data. FIG. 9 shows facial expression composed of correspondence table data 32-1 and facial expression (expression data) and AU combination table data 32-2, in which facial expression movements and muscle contraction sites such as facial muscles causing facial expression changes are associated with each other. An example of the standard model data of change is shown.

  The standard model data 33 for clothes, hairstyles, and the like, although not shown, is a systematic registration of three-dimensional data such as various clothes and hairstyles. The standard model data 33 such as clothes and hairstyle is most similar by comparing the registered standard model data with the clothes and hair shapes extracted from the torso and head of the image received (input) at the time of creation of the three-dimensional data. The model data of the clothes and hairstyle to be selected is selected and used to correct or complement the human image obtained based on the three-dimensional standard model data 31.

  FIG. 5 is a diagram showing an example of a program stored in the storage memory 2. In the program storage area 20 of the storage memory 2, as shown in FIG. A communication program for controlling transmission / reception of data (e-mail, image data, control data, etc.) with the terminals 5, 6, 7,... A control program group 21 such as a control program, a trust management program group 22 that performs trust management processing (outsourcing reception processing, authentication processing, billing processing, invoicing issuance processing, etc.) of three-dimensional data from a contractor, photographing at an arbitrary angle A three-dimensional data creation program 23 for creating three-dimensional data from image data of a plurality of images of the same person (hereinafter referred to as “plurality of image data”), three-dimensional model data for processing Creation program 24,3 dimensional data output program 25, etc. are stored.

  Programs such as the control program group 21, the commission management program group 22, and the three-dimensional data creation program 23 may be stored in a memory separate from the storage memory 2 and are resident in the RAM 12. Also good. Further, the commission management program group 22, the 3D data creation program 23, the machining 3D model data creation program 24, the 3D data output program 25, etc., each time the 3D data creation request is made, the communication network 4 is used. It may be downloaded from a server (not shown).

  The three-dimensional data creation program 23 is a program that causes the three-dimensional data creation apparatus 1 to execute an operation corresponding to step P3 in FIG. 2, and is read by a plurality of image data (or image input unit 14) received in the process P2 in FIG. Image data generated from a photograph, a person image, etc.) is pre-processed and the like is detected, and then a feature region of the person's face is detected, and a person region is extracted from each image based on the detected feature The person area extraction program 231 detects a predetermined feature point from the face image of the person area of each extracted image, and associates it with the feature point of the 3D standard model data 31 of the face stored in the standard model database 30 If the difference or change in facial expression is greater than a predetermined value, the feature point correspondence program 232 for performing a facial expression with a predetermined expression (such as a neutral face, a smiling face, a smile) So that facial expressions change in correction program 233 to correct the change in facial expressions, a three-dimensional shape data generating program 234 and the like subprograms for generating a three-dimensional shape data of the person by a predetermined method.

  The machining 3D model data creation program 24 is a program that causes the 3D data creation apparatus 1 to execute an operation corresponding to step P6 in FIG. Three-dimensional model data for processing corresponding to the type of the three-dimensional image processing apparatus is created.

  The three-dimensional data output program 25 is a program that causes the three-dimensional data creation apparatus 1 to execute an operation corresponding to step P5 in FIG. 2, and includes the three-dimensional data created by the machining three-dimensional model data creation program 235, The 3D model data is output by a method designated by the consignor (transmission to the consignor terminal via the communication network 4, recording on the CD, and data output for 3D image processing on the consignor side).

  As a modification of the three-dimensional data creation program 23, the facial expression change correction processing program 233 and the three-dimensional shape data generation program 234 are associated by the feature point association program 232 as shown in FIG. A feature point extraction program 236 that extracts feature points of the subject's head shape and facial appearance from the corresponding points, and the face three-dimensional standard model data 31 are searched, and the subject's head extracted by the feature point extraction program 236 3D model data search program 237 that obtains the most similar 3D model data to the feature points of the face and the three-dimensional shape (curved surface) of the model searched by the 3D model data search program 237 by projective transformation. 3D data generation program 238 for generating 3D data of the head, and standard model of facial expression change It may be replaced into the correction processing program 239 of the expression change for performing the data 32 and clothing, hairstyle standard model data 33 based on facial expression changes and clothing, hair style correction, such as the like (see FIG. 15).

  FIG. 6 is an explanatory diagram of a process of creating three-dimensional shape data based on the present invention. When generating arbitrary 3D images of the same person when generating 3D shape data, when using images of the same person but different dates and places instead of captured images of the same subject at the same date and time The same person may change not only the angle and direction, but also the hairstyle, clothes, makeup, etc. or the expression may be different. Although it is configured to correct to “neutral face” images such as “smile”, it should be corrected in the same way (for example, standard clothes and hairstyles) when the hairstyle, clothes, makeup, etc. have changed. You can also.

  In FIG. 6, (a) shows a plurality of images of the same person (with different shooting date and time and shooting location), and (b) shows the correction processing for the change in facial expression of each image shown in (a). (C) shows an object (person) photographed from multiple viewpoints, that is, from arbitrary different viewpoints (positions) (a plurality of images having the same photographing date and time and photographing location). Further, (d) shows multi-viewpoint image data of the face portion of the target person obtained from each image of (b) or (d), and (e) is generated from the multi-viewpoint image data of (d) 3 An example of dimensional shape data is shown.

  That is, even in the case of a photographed image of the same person as in (a), the facial expression and the like of each image may differ depending on the photographing date and time and the photographing location. After performing the processing (see steps S6, S7, and FIG. 10 in FIG. 7), multi-viewpoint image data as shown in (d) is obtained, and three-dimensional shape data as shown in (e) is generated (step in FIG. 7). S9, see FIGS. 12 to 14). In addition, since changes in facial expressions and the like are small in a plurality of images obtained by photographing the same person at the same shooting date and time and shooting location as shown in (c), there is no correction processing for changes in facial expressions and the like as shown in (b). In addition, multi-viewpoint image data as shown in (d) is obtained to generate three-dimensional shape data as shown in (e).

FIG. 7 is a flowchart showing an embodiment of the three-dimensional data creation operation of the three-dimensional data creation device 1.
The CPU 11 of the three-dimensional data creation device 1 executes the operations shown in steps S2 to S9 in FIG. 7 (functions corresponding to the operations in step P3 in FIG. 2) by the three-dimensional data creation program 23 to perform a three-dimensional model for machining. The data creation program 24 executes operations as shown in steps S10 to S11 in FIG. 7 (functions corresponding to the operations in step P5 in FIG. 2), and the three-dimensional data output program 25 performs steps S10 to S13 in FIG. (The function corresponding to the operation of step P6 in FIG. 2) is executed. That is, the operation of the three-dimensional data creation device 1 is taken out from the storage memory 2 by the CPU 21 under the control of the control program group 21 and stored in the execution program area of the RAM 12, the three-dimensional data creation program 23, This is executed based on the machining three-dimensional model data creation program 24 and the three-dimensional data output program 25.

  When the CPU 11 receives arbitrary multi-viewpoint image data (including camera information) shooting information or continuous moving image data from the imaging terminal of the consignor, the CPU 11 stores the received image data in a predetermined area of the RAM 12. When the 3D data creation consignment data is received from the consignor by mail or the like, each image is read by the image input processing unit 14 to generate each image data, and the generated image data is stored in a predetermined area of the RAM 12. (Step S1).

  Next, the CPU 11 uses the person region extraction program 231 to perform preprocessing such as inclination correction, noise removal, sharpening processing, and normalization that matches the image size of each image data stored in a predetermined region of the RAM 12. (Step S2). Known image processing techniques can be applied to pre-processing such as tilt correction, noise removal, sharpening processing, and normalization.

The CPU 11 detects a human face feature part from each image data stored in a predetermined area of the RAM 12 by the person area extraction program 231 (step S3), and based on the detected feature, the person's face feature part is detected. An area is extracted (step S4).
Extraction of human facial features from multi-viewpoint images using well-known image processing techniques such as “Lucas-Kanade method (gradient method)” and “Kanade-Lucas-Tomasi tracker method” This can be done by tracking point movement and differences and searching for corresponding points.

  Here, when using a plurality of arbitrary images of the same person, the same person is used instead of the photographed image of the same subject at the same date and time. , Not only the angle and direction, but also the hairstyle, clothes, makeup, etc. may change or the expression may differ. The CPU 11 stores the facial features detected from a plurality of different multi-viewpoint images in steps S3 and S4 by the feature point association program 232, and the facial 3D standard model data 31 stored in the standard model database 30. Correspondence with feature points is performed to detect facial expression data (step S5).

  Next, the CPU 11 uses the facial expression change correction processing program 233 to change the facial features and facial expressions in the three-dimensional standard model data 31 of the facial features and facial expressions of the viewpoint images. If it is larger than the predetermined threshold value or if the facial expression change correction function is set to ON, the process proceeds to step S7, and if not, the process proceeds to step S8 (step S6).

  The facial expression change correction processing program 233 performs a facial expression such as a facial expression so that the facial image of a predetermined facial expression (such as a neutral facial expression, a smiling face, or a smile) is obtained with respect to the image of the human face region in the viewpoint image. After performing the change correction process (see FIG. 10), the process proceeds to step S9 (step S7). With the correction processing operation of the facial expression change in step S7, more accurate three-dimensional shape data can be generated by the three-dimensional shape data generation processing performed in step S9 below.

  The CPU 11 checks whether or not camera information such as the shooting positions and shooting directions of a plurality of process-symmetric multi-viewpoint images is known (step S8). If it is known, three-dimensional shape data creation is performed, as shown in FIG. The three-dimensional shape data is generated from a plurality of multi-viewpoint camera images to generate three-dimensional shape data, and the process proceeds to step S10. If the photographing information is not known, the three-dimensional shape by the factorization method as shown in FIG. Three-dimensional shape data is generated by the data creation operation, and the process proceeds to step S10 (step S9). In addition, when imaging information is not known, you may make it produce | generate 3D shape data by 3D shape data creation operation | movement by a visual volume intersection method as shown in FIG.

  Next, the CPU 11 checks the consignment conditions registered in the process 2 of FIG. 2 by the machining three-dimensional model data creation program 24, and the consignment condition is a request for creating a three-dimensional data such as a CG character, 3D model model data for averter, etc. If this is the case, the process proceeds to step S11. If the consignment condition is creation of 3D model data for processing or production of a stereoscopic image (or prototype), the process proceeds to step S12 (step S10).

  If the consignment condition is a request to create 3D data such as a CG character and 3D model data for averter, the network 4 The bill is issued by the bill issuing program included in the trust management program group 22 (step S11).

  When the consignment condition is creation of machining 3D model data or production of a three-dimensional image (or prototype), the CPU 11 uses the machining model data creation program 236 to process the 3D model data for machining according to the type of the machining apparatus. And proceed to step S13 (step S12).

  When the consignment condition is production of a three-dimensional image (or prototype) (to make a three-dimensional image (or prototype) produced by process P6 in FIG. 2), the processing 3D model data created by the contractor 3 The data is transmitted to the three-dimensional image processing device 3-1, 3-2, or 3-3. In the case of delivery of 3D model data for processing (as shown in process P5 in FIG. 2), the 3D image processing device 8 of the consignor (or connection thereof) is connected via the communication network 4 based on the delivery conditions. The processing three-dimensional model data created is transmitted to the computer, and a bill is issued by the bill issuing program included in the trust management program group 22. If the delivery condition is delivery of a recording medium such as a CD, the created three-dimensional model data is recorded on the CD and an invoice is issued (step S13).

  As described above, according to the three-dimensional data creation device of the present invention, even if an image is taken at an arbitrary angle (and at an arbitrary time, clothes, facial expression), if there are a plurality of images showing the same person, those Since 3D data can be automatically generated on the basis of images, it is not necessary to shoot from a predetermined angle of 360 degrees around as in the past, or to use an automatic shooter without the need for a dedicated shooting stand or automatic shooter. Even without going to the place of installation or the sculptor, a realistic bust or figure can be created easily and inexpensively based on the three-dimensional data created simply by sending or sending photographs and image data. It is also possible to order busts and figures by sending images from a personal computer or camera-equipped mobile phone using a communication network such as the Internet or electronic mail.

  FIG. 8 is a diagram showing an example of a three-dimensional standard model of a face and a three-dimensional model after correction of expression changes, and (a) shows an example of a three-dimensional standard model generated from the standard model data 31 of the face. , (B) shows an example of facial expression change model data 32 corrected by the face sphincter and linear muscle models as shown in FIG. 9 in the dimensional standard model shown in (a).

  FIG. 9 is a diagram showing an example of the standard model data 32 of facial expression changes. (A) is a contraction of muscles such as AU number, AU (Action Unit) and facial muscles that cause AU. A facial expression change and muscle contraction correspondence table data 32-1 associated with a part or the like is shown, and (b) shows a combination table data 32-2 of facial expressions (facial expression data) and AU.

FIG. 10 is a flowchart showing an example of correction processing operation of facial expression change by a facial expression change correction processing program, which will be described with reference to FIGS. 6, 8, and 9.
In FIG. 10, the facial expression data detected in step S5 in FIG. 6 is compared with the facial expression data in the facial expression / AU combination table data 32-2 to obtain a combination of AUs (facial expression operation units). That is, the expression of the detected face is expressed in the expression operation unit AU, for example, “surprise” = AU1 (inner eyebrow raised) + AU2 (outer eyebrow raised) + AU5 (upper eyelid raised) + AU26 (mouth opened). (Step S7-1).

  Next, each AU is converted into a corresponding facial muscle contraction using the facial expression change and muscle contraction correspondence table data 32-1. For example, conversion into facial muscle contraction such as AU (i) → inner frontal muscle (0.14) + outer frontal muscle (0.17) + eyelid eyelid muscle (−0.48) (step S7−). 2).

  Then, the 3D face image obtained by contracting the facial expression muscle obtained in step S7-2 corresponding to each facial expression is drawn and reproduced by CG to obtain 3D image data, and the process proceeds to step S8 in FIG. 7 (step S7-). 3).

  FIG. 11 is an explanatory diagram of correction of facial expression and the like by the expression change correction processing program. In the expression change correction processing program 233 (or 235) of FIG. 10, as shown in (a) to (f) of FIG. , “Joy”, “surprise”, “sadness”, “fear”, “disgust”, “anger”, and other expressions of facial expressions showing emotions such as “anger” are corrected to “nothing” shown in (g). The image is a “neutral face” image such as “expression” or “smile”. Conversely, the image of “neutral face” such as “no expression” or “smiling smile” shown in (g) can be corrected to the image of any expression shown in (a) to (f).

<Method for creating three-dimensional shape data from a plurality of camera images>:
In particular, a method for generating three-dimensional shape data from a plurality of photographed images will be described in detail below.

(A): pixel coordinates, camera coordinates, image coordinates;
The three-dimensional space coordinates based on the camera are the “camera coordinates” and the “image coordinates” representing the two-dimensional image are the origin (0, 0) of the camera coordinate system (X, Y, Z). (0, 0) is the camera center on the optical axis, and the camera coordinates are set to (X, Y, Z) when set as an orthonormal coordinate system of the X axis, Y axis parallel to the captured image plane, and the Z axis in the optical axis direction. ) The following expression holds for a point in the three-dimensional space ^T and the image coordinates (x, y) ^T of the two-dimensional image obtained as the perspective projection.
x = 1 × X / Z, y = 1 × Y / Z (where l is the focal length of the camera) (1)

  Here, there is a one-to-one characteristic unique to each camera between “Image coordinates” describing an image in a three-dimensional space by perspective projection and “Pixel coordinates” such as a monitor display screen. There is a mapping relationship. When restoring camera motion and relative positional relationship (three-dimensional shape) from point correspondence in multiple images, point correspondence can be given first by "image coordinates", so it is unified for all cameras In order to deal with the problem, if a one-to-one mapping relationship between the “pixel coordinates” and the “image coordinates” can be obtained (referred to as “camera calibration”), the “pixels (Pixel coordinates)” can be obtained. ) Coordinates "and" image coordinates "can be freely converted.

In perspective projection that expresses a camera model, the problem of obtaining camera motion and 3D shape from point correspondence in multiple images is an inverse problem of the nonlinear mapping, resulting in a nonlinear optimization problem. Since there are problems such as sensitivity to noise, high initial value dependency, and unstable numerical calculation, it is difficult to stably restore a three-dimensional shape.
Therefore, as described in Non-Patent Document 1, external information on the camera position, posture, viewpoint direction, or the like, that is, camera motion information is input, or camera motion information is added by adding an artificial feature such as a reference marker. There are ways to make it easier to find.

  Alternatively, two cameras (or viewpoints) that are orthogonal to each other as much as possible from among a plurality of multi-viewpoint cameras (or viewpoints) as disclosed in Japanese Patent Application Laid-Open No. 2004-220312 and the explanation of the visual volume intersection method described later. ) As a base camera, and using a projection grid space by the base camera, add or restrict correlation information between multiple cameras or viewpoints, for example, a base representing an epipolar geometric relationship between cameras A point (voxel) on the projection grid space is back-projected onto each camera image using a matrix F (Fundamental) matrix or the like, and each point (voxel) exists inside the object in the silhouette image of each camera image. There is a method of restoring the three-dimensional shape data by determining whether it is external or external.

  Alternatively, Takeo Kanade et al., “Restoring Object Shape and Camera Motion by Factorization”, IEICE Transactions, J76-D-II, No.8 (19930825), pp. 1497-1505, Satoshi Fujiki (AIST), “Reconstruction of 3D shape from multiple 2D images using point correspondence -Mathematical factorization method”, Statistical mathematics, Vol. 49, No. 1, pp77- As in 107 and 2001, the perspective projection, which is an ideal camera model, is approximated to an affine projection by “factorization method”, etc., and a plurality of two-dimensional images based on an affine approximate projection such as an orthographic projection model are used. A technique for simultaneously restoring camera motion information and three-dimensional shape information can be used.

  Assuming that a two-dimensional image is obtained by affine approximate projection, the camera motion from point correspondence in a plurality of affine approximate projection images and the three-dimensional shape restoration problem are inverse problems of the linear mapping, so the restoration accuracy is inferior. However, it can be solved more stably in numerical calculation than in the case of nonlinear mapping.

B: point association between multiple images;
First, point features (luminance, color, contour shape, texture, etc.) are associated between a plurality of images. Points that are far away in the three-dimensional space may be projected closer in the two-dimensional image. Since a slight error in a two-dimensional image has a significant effect on recognition and understanding in a three-dimensional space, it is necessary to accurately correspond points between a plurality of images.

For matching feature points between multiple images, methods such as “Lucas-Kanade method (gradient method)” (1981) and “Kanade-Lucas-Tomasi tracker” (Shi and Tomasi, 1994) are used. be able to. It is difficult to correlate images that are separated in time without prior knowledge, but if the correspondence is known at multiple points, the geometric relationship between multiple images taken from the same three-dimensional space (epipolar geometry) The area where other corresponding points can exist can be narrowed down.
For example, for correspondence and tracking between a plurality of frame images, in general, adjacent to each other based on appearance (local image) or similarity (correlation) or difference of image features such as edges (contours) and color histograms. A method (block matching) that searches for the most similar area between the frame images and determines that the object has moved to the searched point is often used.

That is, for each pixel value I (x) at the coordinate x = (x, y), the movement amount (displacement) d = (d _x , _dy ) is sequentially changed, and is expressed by the following equation:
Sum of squared differences (square error) ε (d) = Σ {I _t (x + d) −I _t−1 (x)} ² , or
Sum of absolute differences ε (d) = Σ | I _t (x−d) −I _t−1 (x) |
Cross-correlation γ (d) = Σ {I _t (x−d) −I ^￣ _t } {I _t−1 (x) −I ^￣ _t−1 } / | I _t (x−d) −I ^￣ _t | | I _t-1 (x) −I ^￣ _t-1 | (2)
Etc., and from that,
Difference minimum d ^ = min _d {ε (d)} or similarity (correlation) maximum d ^ = max _d {γ (d)}
... (3)
What is necessary is just to obtain | require the displacement d which becomes. However, if this is fully searched, there is a problem that the amount of calculation increases and the amount of displacement is discrete and not continuous. In the case of matching by rotating or enlarging / reducing a local image, further enormous calculation is required.

・・・（７）
とおくと、A^TAd−A^Tｂ＝0 → A^TAd＝A^Tｂ

・・・（８）
となるので、A^TAが正則なとき、d＾は解を持ち、
d＾＝min_dε＝（A^TA）^-1 A^Tｂ ・・・（９）
となり、全探索しなくても、相違度最小となる変位量を求めることができる。
特徴点検出と上記のような追跡法とを統合した手法は、「Kanade-Lucas-Tomasi（KLT）トラッカー」と呼ばれる。 At this time, in the “Lucas-Kanade method (gradient method)”, the maximum value (or minimum value) is efficiently obtained by climbing (or descending) on the basis of the gradient (slope) around the provisional solution. Can do. That is, the slope of the sum of squared differences (square error, SSD)
ε = Σ {I (x + δx, y + δy, t + δt) −I (x, y, t)} ² (4)
Then, the Taylor expansion of this first term is
I (x + δx, y + δy, t + δt)
= I (x, y, t) + δx {∂I (x, y, t) / δx} + δy {∂I (x, y, t) / δy} + δt {∂I (x, y, t) / δt } + ...
At this time, if the terms after the second order can be ignored as the displacement is minute (can be linearly approximated around x),
As δx = d _x , δy = d _y , δt = 1,
ε = Σ {d _x I _x (x, y, t) + d _y I _y (x, y, t) + I _t (x, y, t)} ²
... (5)
Where I _x (x, y, t) = ∂I (x, y, t) / δx, I _y (x, y, t) = ∂I (x, y, t) / δy, I _t (x, y, t) = ∂I (x, y, t) / δt
The displacement d ^ = min _d ε with the smallest difference is obtained by obtaining d such that ∂ε / ∂d _x = 0 and ∂ε / ∂d _y = 0.
∂ε / ∂d _x = Σ2I _x (x, y, t) {d _x I _x (x, y, t) + d _y I _y (x, y, t) + I _t (x, y, t)} = 0,
_{∂ε / ∂d y = Σ2I y (} x, y, t) {d x I x (x, y, t) + d y I y (x, y, t) + I t (x, y, t)} = 0 (6)
here,

... (7)
A ^T Ad−A ^T b = 0 → A ^T Ad = A ^T b

... (8)
Therefore, when A ^T A is regular, d ^ has a solution,
d ^ = min _d ε = (A ^T A) ⁻¹ A ^T b (9)
Thus, the displacement amount that minimizes the dissimilarity can be obtained without performing a full search.
A method in which feature point detection and the tracking method as described above are integrated is called a “Kanade-Lucas-Tomasi (KLT) tracker”.

C: Three-dimensional shape data generation method (C-1): Method using camera position information and reference markers;
For example, various types of software have been developed for creating three-dimensional shape data of an object by inputting image data taken from multiple viewpoint angles from 360 degrees around the periphery or from a plurality of cameras arranged around the periphery. .
In these, a plurality of multi-viewpoint images taken with a camera whose camera position information is previously taken with a camera placed at a predetermined position and angle are used, or as described in Non-Patent Document 1, for example In addition, the camera position at the time of shooting is automatically calculated from a plurality of camera images obtained by shooting together the reference marker printed on the turntable and the target object placed on the turntable.

  In addition, the 3D information of the shape and texture is automatically calculated from the obtained camera position information and the object image, the 3D information is appropriately corrected by the user, and the 3D data is obtained from the image hand-held by the digital camera. Is generated. Input 10 to 20 or more JPEG-format captured image data from different viewpoints for capturing an object, and execute a series of processes such as camera position calculation and shape information generation to obtain 3D shape data. Create (refer to the description of step S9-1-1 in FIG. 12 described below for the calculation of the camera position, refer to the description of step S9-1-2 for the generation of shape information, and step for the output of three-dimensional shape data) (See description of S9-1-3).

FIG. 12 is a detailed flowchart showing an example of the operation for creating three-dimensional shape data from a plurality of multi-viewpoint camera images in step S9 of FIG.
If the camera information is known in step S8 in FIG. 7, in FIG. 12, the CPU 11 determines the three-dimensional position parameters of the camera in the coordinate system (rotation components α, β, γ and translation components (x, y, z)). The camera position is calculated based on the Hough transform, etc. That is, the reference marker is extracted from the photographed image, and any three of the extracted reference markers and any of the reference markers registered in advance are extracted. A position parameter is calculated by solving simultaneous equations determined by combining the positional relationships of the three points (step S9-1-1).

  Next, the object and background are separated using color information of a predetermined background, and the silhouette (contour) of the target is extracted (step S9-1-2), and the camera position and the two-dimensional contour information of the target are extracted. Based on the above, the 3D shape is reconstructed by voting to the 3D voxel space, and the obtained voxel data is converted to polygon (polygon) data. Etc., and output as three-dimensional shape data, and proceeds to step S10 in FIG. 7 (step S9-1-3). Note that VRML, XVL, IGES, STL, PLY, and the like may be converted into a three-dimensional shape data format that conforms to an output format used in three-dimensional CAD or three-dimensional CG.

  Here, the Hough transform (Hough transform, Hugh transform) is a method often used in problems such as obtaining a straight line from an edge image, and each point (x, y) in the edge image is on a certain straight line. Voting all possible straight lines assuming a point as a curve to the parameter space (ρ, θ) with polar coordinates ρ = x cosθ + y sinθ, and finally (ρ, θ) peaks in the parameter space. Is obtained as a parameter of the straight line to be obtained.

(C-2): visual volume intersection method;
A three-dimensional shape model can also be generated by using the “view volume intersection method” for a plurality of images from multiple viewpoints. In the visual volume intersection method, the silhouette of an object is extracted from images taken by multiple (position) cameras installed in real space, back-projected into space, and the intersection of the silhouettes is calculated. Ask. A procedure for generating a three-dimensional model from a plurality of multi-viewpoint images is shown in the flowchart.

FIG. 13 is a detailed flowchart showing an example of the operation of creating the three-dimensional shape data by the visual volume intersection method in step S9 of FIG.
If the camera information is unknown in step S8 of FIG. 7, in FIG. 13, the CPU 11 first divides the three-dimensional space (voxel space) constituting the shape into cubic lattices (step S9-2-1). By the view volume intersection method, a multi-viewpoint image silhouette image is input, and back projection by orthographic projection is performed on each voxel (step S9-2-2).

  Next, it is determined whether or not the silhouette of the image exists on each voxel. If the silhouette exists, the process proceeds to Step 9-2-4. If the silhouette does not exist, the process proceeds to Step 9-2-4. Proceed (step S9-2-3).

  If the silhouette exists, the voxel is left and the process proceeds to step S9-2-6 (step S9-2-4). If the silhouette does not exist, the voxel is deleted and the process proceeds to step S9-2-6 (step S9-2-6). S9-2-5).

  It is determined whether or not the existence of the silhouette has been checked for all the voxels divided in step S9-2-1. If all the voxels have been checked, the process proceeds to step S9-2-7, and there is a voxel that has not been checked. Returns to step S9-2-2 to check the next voxel (step S9-2-6).

  It is determined whether or not the determination operations in steps S9-2-1 to S9-2-6 have been performed for all the multi-view images. If all the multi-view images have been determined, the process proceeds to step S9-2-8. If there is an image that has not been determined, the process returns to step S9-2-1 to determine the next image (step S9-2-7). Then, the finally existing voxel set is regarded as a three-dimensional shape, and three-dimensional shape data is generated by deleting the voxels inside the three-dimensional shape, and the process proceeds to step S10 in FIG. 7 (step S9-2-8). .

Visual volume intersection method and projective grid space;
The principle of restoring the three-dimensional model based on the “visual volume intersection method” will be briefly described.
The visual volume is a cone whose viewpoint is the apex and whose silhouette is the cross-section of the object, and the “visual volume intersection method” is used to calculate the common part of the visual volume of the object at all viewpoints. This is a technique for restoring the shape of an object.

Arbitrary two of the multiple cameras (or camera images from multiple positions) are the base cameras (or camera images from the base positions) 1 and 2, and central projection from the respective viewpoints of the two base cameras Defines a three-dimensional space.
Here, the three-dimensional space is considered as a projective grid space (PGS), and the voxel A (p, q, r) in the space is a point a ₁ (p, q) is defined as being projected onto a point a ₂ (r, s) on the captured image 2 from the base camera 2.

  When performing the intersection calculation, a geometric relationship between images and a correspondence relationship between camera coordinates and space coordinates are necessary, and these are known by calculating an F (Fundamental) matrix (in the F matrix, 2). It can be determined by the correspondence of 9 or more points between images).

・・・（１０）
a₂のx座標は射影グリッド空間PGSの定義よりrであるから、y座標sも定まる。
３） そして、基底カメラ以外のカメラからの撮影画像（または、基底位置以外からのカメラ画像）ｉに対する投影点の座標x_i,y_iは、次のようにして定まる。
基底カメラ２への投影と同様に、Ｆ_i1を用いて点a₁を画像ｉ上に直線L_i1として投影する。
またＦ_i2を用いて、点a₂を画像ｉ上に直線L_i2として投影する。
４） ２本のエピポーラ線L_i1、L_i2の交点が、画像ｉの投影点の座標である。
５） この処理を、全視点の画像に対して行なう。
このようにして、注目ボクセルに対する全視点の画像の座標値を求めることができ、３次元モデルが復元できる。 The following projection is performed using the F matrix.
1) First, the projection point a ₁ on the image 1 with respect to A (p, q, r) in the space is projected onto a ₁ (p, q) by the definition of the projection grid space PGS.
2) Next, the projection point a ₂ on the image 2, when projected as epipolar line L ₂₁ in the image 2 by using the F matrix F _21, for a ₂ is present on L _21, the straight line L ₂₁ following Can be defined by an expression.

... (10)
Since the x coordinate of a ₂ is r from the definition of the projection grid space PGS, the y coordinate s is also determined.
3) Then, the coordinates x _i and y _i of the projection point with respect to the photographed image from the camera other than the base camera (or the camera image from other than the base position) _i are determined as follows.
Like the projection on the base camera 2 projects as a straight line L _i1 the point a ₁ on the image i with F _i1.
Further, the point a ₂ is projected onto the image i as a straight line L _i2 using F _i2 .
4) The intersection of the two epipolar lines L _i1 and L _i2 is the coordinates of the projection point of the image i.
5) This process is performed on all viewpoint images.
In this way, the coordinate values of the images of all viewpoints for the target voxel can be obtained, and the three-dimensional model can be restored.

  Three-dimensional model restoration by intersection calculation is performed by projecting voxels contained in the space onto one silhouette on the defined projective grid space PGS, deleting all voxels not on the silhouette, and projecting them to the next silhouette image. The process of performing is performed in the order of the base camera 1, the base camera 2, and the other cameras, and only the voxels included in the silhouettes of all the input viewpoint images are regarded as “exist” and the three-dimensional model can be restored.

(C-3): Factorization method:
There is a factorization method as a method of generating three-dimensional shape data from only a plurality of multi-view camera image data and continuous moving image data without using the reference marker as described above.
In general, in order to obtain the three-dimensional shape of an object from any two-dimensional images around the object without limiting the camera position and the viewpoint direction, a huge amount of calculation processing is required and the solution is not good. Become stable. "Factorization" (Tomasi and Kanade (Takeo Kanade), 1992) simplifies the problem by approximating the perspective projection, which is an actual camera model, with an affine projection, and makes numerical calculations faster and faster. The solution can be stabilized. It is also known as an excellent method that can simultaneously restore camera motion and the three-dimensional shape of a target object from a plurality of affine approximate projection images.

FIG. 14 is a detailed flowchart showing an example of the operation for creating three-dimensional shape data by the factorization method in step S9 of FIG.
If the camera information is unknown in step S8 of FIG. 7, in FIG. 14, the CPU 11 firstly, from each image, a line segment, a curve, or a feature point representing the contour outline of the subject person or the feature part of the face. Is extracted (step 9-3-1).

  Next, point features of main points of each image are extracted, and each feature point is associated using the Kanade-Lucas-Tomasi method or the like (step 9-3-2), and each point coordinate (measurement matrix) in the multi-viewpoint image is obtained. ), The camera motion information (motion matrix) and the three-dimensional shape information (shape matrix) of the object are restored by factorization or the like to generate three-dimensional shape data, and the process proceeds to step S10 in FIG. 7 (step S10). 9-3-3).

  Regarding the "factor decomposition method", the above-mentioned literature (Takeo Kanade et al., "Restoring object shape and camera motion by factor decomposition method", IEICE Transactions, J76-D-II, No. 8 (19930825), pp. 1495-1505, Satoshi Fujiki (AIST), “3D shape restoration from multiple 2D images using point correspondence-Mathematical factorization method”, Statistical mathematics, Vol. 49, No. 1, (pp77-107, 2001) and the like are described in detail, so that only the outline will be described below to avoid complication.

It is assumed that the association of point features in a plurality of images has already been obtained and given as image coordinates by the step “B: Point association between a plurality of images” described above. In the affine approximate projection, the mapping by the camera shooting is an affine projection determined by the position and direction of the camera from the object in the three-dimensional space to the two-dimensional image.
When f images and P feature points are given, assuming that FP image coordinates are obtained by F affine projections of P three-dimensional coordinates, the factorization method uses the FP conditions as a matrix. The three-dimensional shape restoration problem from a plurality of two-dimensional images can be expressed in a simple form. That is,
(Measurement matrix) = (motion matrix) × (shape matrix) (11)
Here, a measurement matrix is a 2F × P matrix in which FP image coordinates are arranged, a motion matrix is a 2F × 3 matrix in which F affine projection expression matrices are arranged, and a shape matrix is a three-dimensional coordinate of P feature points. Is a 3 × P matrix. That is, the problem of restoring a three-dimensional shape from a plurality of two-dimensional images can be reduced to factorization of a measurement matrix.

Thus, if the measurement matrix obtained when projected by the affine approximate projection can be estimated from the components of the measurement matrix obtained by the perspective projection, the measurement matrix and the shape of the motion matrix and the shape are obtained (in accordance with the factorization algorithm). The camera motion information and the three-dimensional shape of the object can be restored simply by decomposing the matrix product.
However, since the image coordinates are expressed by orthonormal bases, it is necessary to decompose the image coordinate bases to be orthonormal bases in order to obtain a correct restoration solution. The affine projection model (Mundy and Zisserman, 1992) is a model for an uncalibrated camera, and the conversion from X _fp to x _fp is expressed in the form of an affine projection of the following equation.
x _fp = A _f X _fp + u _f (12)
Here, A _f and u _f are unknown parameters. In the affine projection model, no assumptions are made for A _f , so even if the positional relationship of the target object in the affine space can be known (affine reconstruction), the measurement information such as the length and angle of the target object of the target object can be obtained. It is impossible to know (Euclidean restoration).

（B_fは既知） ・・・（１３） Therefore, a metric affine projection model (MAP model) has been considered as a model for performing Euclidean reconstruction of the target object. In order to perform the Euclidean reconstruction of the target object, it is necessary to know the components of the matrix B _f that is the remainder obtained by deducting the depth parameter λ _{f *} = tZ _{f *} from A _f .

(B _f is known) (13)

（ここで、ｌ：焦点距離） ・・・（１４）
このような仮定を加えたアフィン射影モデルを、「計量アフィン射影」（ＭＡＰ：Metric Affine Projection）モデルと呼ぶ。また、A_fをＭＡＰ行列と呼ぶ。 Furthermore, u _f needs to be known in order to restore the position of the camera.
From the above assumptions,

(Where l is the focal length) (14)
The affine projection model to which such an assumption is added is called a “metric affine projection” (MAP) model. A _f is called a MAP matrix.

（ここで、ｌ：焦点距離） ・・・（１７）
Ｐ個の点特徴の画像がＦ枚得られたとき、複数の２次元画像から、カメラの運動情報と物体の３次元形状情報とを復元する問題は、上式（１７）から｛C_f｝と、｛ｓ^* _p｝を求める問題になる。 Assuming that the solid (target object) is fixed and the camera is moving, to restore the camera motion information and the three-dimensional shape of the object, the coordinate system (world coordinates in which the camera model is fixed to the solid) System). The world coordinate of the camera position in the f-th image is t _f , the orthonormal basis on the f-th image plane is {i _f , j _f }, and the unit vector in the camera optical axis direction is k _f ,
C = (i _f , j _f , k _f ) ^T , a matrix representing the camera orientation in world coordinates (camera base matrix)
Furthermore, the world coordinate s _p of the p feature points, the spatial coordinates in the camera coordinate system of the f image and X _{fp,
s p = t f + C f} T X fp ··· (15)
The expression, relative coordinates s ^* _p = s _p -s from one feature point s _{* *,} expressed in ^{_{t * f = t f -s *}} ,
^{_{s * p = t * f +}} C f T X fp, X fp = C f (s * p -t * f) ··· (16)
When the above MAP model is expressed in the world coordinate system,

(Where l is the focal length) (17)
When F images of P point features are obtained, the problem of restoring camera motion information and object three-dimensional shape information from a plurality of two-dimensional images is as follows: {C _f } And {s ^* _p }.

In the factorization method, a matrix {C _f } (f = 1 to _F ) representing the camera direction is obtained by decomposing a matrix formed from the above FP equations (f = 1 to F, p = 1 to P). ) And world coordinates {s ^* _p } (p = 1 to P) of feature points of the object.

・・・（１８）
と定義すると、
W^*＝M（２F×３行列）・S^*（３×P行列） ・・・（１９）
が成立する。
Mには、カメラ運動に関する未知数｛C_f｝（ｆ＝１〜F）および｛λ^* _f｝（ｆ＝１〜F）のみが、S^*には、３次元形状に関する未知数｛ｓ^* _p｝（ｐ＝１〜P）のみが含まれていることから、計測行列W^*を、運動行列Mと形状行列S^*の積に分解することができれば、カメラの運動情報と物体の３次元形状とが復元できる。 Here, the measurement matrix W ^* , the motion matrix M, and the shape matrix S ^* are

... (18)
Defined as
W ^* = M (2F × 3 matrix) · S ^* (3 × P matrix) (19)
Is established.
M contains only the unknowns {C _f } (f = 1 to F) and {λ ^* _f } (f = 1 to _F ) related to the camera motion, and S ^* represents the unknown {s ^* _p } related to the three-dimensional shape. Since only (p = 1 to P) is included, if the measurement matrix W ^* can be decomposed into the product of the motion matrix M and the shape matrix S ^* , the motion information of the camera and the three-dimensional shape of the object Can be restored.

Here, in the decomposition of W ^* into the product of M and S ^* , since {C _f } is a three-dimensional rotation matrix, the following condition (metric constraint) needs to be satisfied.
M _f M _f ^T = A _f A _f ^T = (1 / λ ^{2 *} _f ) B _f B _f ^T (20)

・・・（２１）
このとき、M、S^*、と暫定解のM＾、S＾^*の間には、
M＝M＾A、 S^*＝A^-1・S＾^*（ただし、A^-1はAの逆行列） ・・・（２２）
の関係を満たす３×３可逆行列Aが存在するので、この暫定的な分解によって、運動と形状がアフィン復元されていることになる。 Factorization algorithm;
In an actual factorization algorithm, the measurement matrix W ^* is decomposed in the order of “Affine restoration” and “Euclid restoration” to restore camera motion information and object three-dimensional shape information.
First, the measurement matrix W ^* is temporarily (provisionally) decomposed into a product of M ^ (2F × 3) and S ^ ^* (3 × P) by specific decomposition (SVD) or the like, and Affine Restore.

... (21)
At this time, between M, S ^* , and M ^, S ^ ^* of the provisional solution,
M = M ^ A, S ^* = A- ¹ · S ^ ^* (where A- ¹ is the inverse matrix of A) (22)
Since there is a 3 × 3 reversible matrix A that satisfies the relationship, the motion and shape are restored to affine by this provisional decomposition.

2) Obtaining the Euclidean restoration solution from the affine restoration solution of the provisional solution results in obtaining the 3 × 3 reversible matrix A. Putting a Q = AA ^T, from the metering constraint, condition to be satisfied by A is
^{_{M ^ f QM ^ f T =}} A f A f T = (1 / λ 2 * f) B f B f T ··· (23)
Here, the unknown quantities in the equation (14) are {λ ^* _f } (f = 1 to F) and Q, and _Bf is known.
_If the singular value decomposition of B _f is B _f = R _f Σ _f D _f , R _f and Σ _f are known,
_{P ^ f = (p ^ f} , q ^ f) T = R f T M ^ f, P f = R f T M f ··· (24)
Then, the constraint condition of the equation (14) is
P ^ _f QP ^ _f ^T = (1 / λ ^{2 *} _f ) Σ ² _f (25)
And become simple. At this time,
_{^{p ^ f T Q p ^ f}} = p 2 f / λ 2 * f, p ^ f T Q q ^ f = 0, q ^ f T Q q ^ f = q 2 f / λ 2 * f,
In other ^{_{words, (p ^ f T Q p}} ^ f) / (p 2 f) = (q ^ f T Q q ^ f) / (q 2 f) = 1 / λ 2 * f, p ^ f T Q q ^ _f = 0, (26)
Therefore, a linear simultaneous equation regarding Q that does not include {λ ^* _f } is obtained as in the following equation.
q ² _f (p ^ _f ^T Q p ^ _f ) −p ² _f (q ^ _f ^T Q q ^ _f ) = 0, p ^ _f ^T Q q ^ _f = 0,
... (27)
By solving these simultaneous equations, Q, which is a 3 × 3 positive symmetric matrix, can be uniquely obtained without the indefiniteness of constant multiplication by {λ ^* _f }.

3) The Cholesky decomposition of the symmetric matrix Q, and the Q = LL ^T,
The general solution of A is A = L ^T U,
The general solutions of the motion matrix M and the shape matrix S ^* are M = M ^ L ^T U, S ^* = U ^T LS ^ ^* (28)
It is obtained by

As described above, from the measurement matrix W ^* (2F × P matrix in which FP image coordinates are arranged) data, the motion matrix M data including the camera motion information (F affine projection expression matrices arranged in 2F) × 3 matrix) and shape matrix S ^* data (3 × P matrix in which the three-dimensional coordinates of P feature points are arranged) including the three-dimensional shape information of the object can be restored simultaneously.

  Details of the factorization algorithm are detailed in the above-mentioned literature. In addition, the “sequential factorization method” that sequentially calculates the factorization method to support real-time processing, and the iterative estimation of the image by the affine approximate projection using the factorization method, the camera motion and the stereoscopic Various improved methods such as the “CH method” (Christy and Horaud, 1996) for restoring the shape with higher accuracy have been proposed, but are omitted here.

In an actual factorization algorithm, the measurement matrix W ^* is decomposed in the order of “Affine restoration” and “Euclid restoration” to restore camera motion information and object three-dimensional shape information.
1) First, the measurement matrix W ^* is temporarily (provisionally) decomposed into the product of M ^ (2F × 3) and S ^ ^* (3 × P) by specific decomposition (SVD). Restore Affine.

The inverse matrix (inverse matrix) is an n-order square matrix A when an n-order square matrix X with AX = XA = I (I is a unit matrix) exists for an n-order square matrix A. Or they say "regular". At this time, X is called an “inverse matrix” of A and is written as A ⁻¹ . A square matrix is a matrix in which the number of row elements matches the number of column elements. Invertible matrix: Or, a regular matrix is related to the normal product of matrices. It is a square matrix with an inverse matrix that is the inverse element.
Also, the symmetric matrix (symmetric matrix), of a square matrix A, means a matrix transposed matrix A ^T of A coincides with A itself.

Cholesky decomposition (Cholesky decomposition): Originally, a positive definite Hermitian matrix A is decomposed into a product of a lower triangular matrix L and a conjugate transpose matrix L ^* of L. In the case of a real symmetric matrix, conjugate transposition is simplified to transposition, which is equivalent to decomposing the symmetric matrix A into A = LL ^T.

(Modification)
FIG. 15 is a flowchart showing a modification of the three-dimensional shape data creation operation of the three-dimensional data creation device 1, and the operations in steps S6 to S in FIG. 7 are replaced with steps S6 ′ to S9 ′, thereby simplifying the operation. In this example, three-dimensional shape data can be created.

  In FIG. 15, the CPU 11 extracts feature points of the shape of the head and the face from the person region of each image extracted in step S4 of FIG. 7 by the feature point extraction program 236 (step S6 ').

  Next, the three-dimensional model data retrieval program 237 retrieves the three-dimensional standard model data 31 of the face, and the three-dimensional model most similar to the subject's head and facial feature points extracted in step S9-4-1. Data is obtained (step S7 ′).

  Next, the three-dimensional shape (curved surface) of the three-dimensional standard model of the face obtained in step 10-4-2 by the three-dimensional data generation program 238 as the face image data of the subject (see FIG. 8A)) Projective transformation is performed on the face to generate three-dimensional shape data of the face (and head) (step S8 ′).

  If the facial expression difference or change in the 3D image composed of the 3D shape data of the face (and head) generated in step S9-4-3 is greater than a predetermined value, the facial expression change correction processing program 239 ( Using standard model data 32 for changing facial expressions and standard model data 33 for clothes, hairstyles, etc. so that a facial image with a predetermined expression (such as a neutral face, a smiling face, a smile) is used. And the process proceeds to step S10 in FIG. 7 (step S9 ′).

  According to the above modification, the image data of the subject's face is projectively transformed to the face of the 3D model data similar to the head and face of the subject, so that the 3D data of the head can be generated. It is possible to process even a device having a low image processing capability or a small device, and the processing time is shortened. For example, even in a gambling vending machine installed in a game center, a 3D data creation device is equipped with a camera, an image is transmitted from a digital camera or a mobile phone, or input by a cable connection. In this way, it is possible to manufacture and sell inexpensive figures by producing and processing 3D figures and busts from face images on the spot.

It is a figure which shows the structural example of the three-dimensional image creation commissioned system using the three-dimensional data creation apparatus of this invention. 3 is a process chart showing a 3D image entrusting / creating process in the 3D image creating entrusting system shown in FIG. 1. It is a figure which shows the structural example of the three-dimensional data creation apparatus of this invention. It is a figure which shows the structural example of a standard model database. It is a figure which shows the example of the program stored in the preservation | save memory. It is explanatory drawing of the three-dimensional shape data creation process based on this invention. It is a flowchart which shows one Example of the three-dimensional breast image data creation operation | movement by a three-dimensional data creation apparatus. It is a figure which shows the example of the 3D standard model of a face, and the 3D model of the facial expression change. It is a figure which shows the structural example of the standard model data of the facial expression change. It is a flowchart which shows one Example of the correction process operation | movement of the change of the expression of the face image by the correction process program of a facial expression change. It is explanatory drawing of correction | amendment of the facial expression etc. by the correction process program of facial expression change. FIG. 8 is a detailed flowchart illustrating an example of an operation for creating three-dimensional shape data from a multi-viewpoint camera image in step S <b> 9 of FIG. 7. It is a detailed flowchart which shows one Example of the operation | movement of three-dimensional shape data preparation by the visual volume intersection method in FIG.7 S9. It is a detailed flowchart which shows one Example of the three-dimensional shape data creation operation | movement by the factorization method in step S9 of FIG. It is a flowchart which shows the modification of the three-dimensional shape data creation operation | movement by a three-dimensional data creation apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 3D data preparation apparatus 2 Database 3, 8 3D image processing apparatus 4 Communication network 11 CPU
22 3D data creation program 31 3D standard model data of face and head 32 3D standard model data of facial expression changes 33 3D standard model data of clothes, hairstyle, etc.

Claims (6)

A database of standard model data of stereoscopic images;
Image data acquisition means for acquiring image data of a plurality of multi-viewpoint images;
A three-dimensional shape data generation program for causing a three-dimensional data creation device to function as a three-dimensional shape data generation means for generating three-dimensional shape data from the plurality of multi-viewpoint images;
Extracting a person area from each of a plurality of pieces of image data acquired by the image data acquisition means, acquiring corresponding points between the feature part of the face detected from the extracted person area and the standard model data of the database, 3D data creation control means for obtaining 3D shape data by a 3D shape data generation program;
Facial expression change correction means for correcting facial expression change model data by correcting the three-dimensional shape data;
With
The facial expression change correction means includes correspondence table data in which combinations of facial expression types and facial expression action units are associated, and correspondence table data in which facial expression motion units and facial muscle contraction sites are associated with each other. By expressing the combination of unit movements and converting the unit movement on each table into the contraction of the facial muscles of the corresponding part, the facial expression data showing emotions is corrected to neutral face data, or neutral Correct facial data to facial expression data showing emotions,
A three-dimensional data creation device characterized by that. The image data acquisition means includes camera information acquisition means for acquiring camera information when camera information such as a shooting position and a shooting direction of image data to be acquired is known,
The standard model data of the stereoscopic image includes standard model data of a person's face and head, standard model data of facial expression changes,
The three-dimensional data creation control means acquires facial expression data from corresponding points between the facial feature portion detected from the extracted person region and the standard model data of the person's face, and the facial expression in the facial expression data and the standard model If the facial expression change in the facial expression data is greater than a threshold value by comparing with the facial expression of the data, the facial facial expression is corrected to a predetermined facial expression based on the facial expression change model data, and the camera information acquisition means If the information is known, the first 3D shape data generation program included in the 3D shape data generation program generates the 3D shape data of the person. If the camera information is unknown, the 3D shape data is generated. Generating the 3D shape data of the person by a second 3D shape data generation program included in the data generation program;
The three-dimensional data creation apparatus according to claim 1.   The first three-dimensional shape data generation program uses a three-dimensional data creation device based on image data of a plurality of multi-view images acquired by the image data acquisition unit and camera information acquired by the camera information acquisition unit. The three-dimensional data creation apparatus according to claim 1, wherein the three-dimensional data creation device functions as three-dimensional shape data generation means for generating three-dimensional shape data.   The second three-dimensional shape data generation program generates a three-dimensional shape data by using a three-dimensional data creation device based on a visual volume intersection method based on image data of a plurality of multi-viewpoint images acquired by the image data acquisition unit. The three-dimensional data creation apparatus according to claim 1, wherein the three-dimensional data creation device functions as a three-dimensional shape data generation unit.   The second three-dimensional shape data generation program generates a three-dimensional shape data by a factorization method based on the image data of a plurality of multi-view images acquired by the image data acquisition unit. The three-dimensional data creation apparatus according to claim 1, which functions as a three-dimensional shape data generation unit. Standard model data of the stereoscopic image includes the standard model data of the face and head of a person, the standard model data of facial expressions, the standard model data of clothing and hair,
The three-dimensional data creation control means extracts a head shape and facial features from each image data based on a facial feature portion detected from the extracted person region, and standard model data of the person's face and head To obtain three-dimensional model data most similar to the shape of the head and facial features, and projectively transform the image data of the plurality of multi-viewpoint images onto the curved surface of the acquired three-dimensional model data. And three-dimensional shape data of the head,
The three-dimensional data creation apparatus according to claim 1 .

Images