JP6207210B2 - Information processing apparatus and method - Google Patents

Description

  The present invention relates to information processing for generating a face model of an arbitrary expression.

  In the field of movie production and entertainment, a technique called performance capture that captures the facial expression of an actor and makes the virtual character look like an actor is actively researched.

  In recent years, a combination of high-resolution image sensors and high-speed computers has led to the introduction of fine irregularities such as pores and wrinkles of a specific person into a computer, and later a method to arbitrarily change the person's facial expression. (See Patent Document 1).

  The facial expression generation method disclosed in Patent Document 1 can faithfully reproduce fine deformations such as pores and wrinkles accompanying rough deformation of the face. In the technique of Patent Document 1, first, rough deformation and fine deformation are acquired by different data acquisition methods.

  Rough deformation data is acquired using a general motion capture technique by attaching a marker to the face of the subject. The rough face deformation at this time is represented by a sparse three-dimensional point group measured using marker points, and a polygon mesh connecting these points.

  On the other hand, fine deformation is obtained by using a photometric stereo and a dense optical flow technique. The fine deformation acquired at this time is represented as a displacement map.

  The displacement map is a technique for defining fine irregularities with a small number of polygons by defining the height of the original shape with a grayscale image or RGB image. In addition, the displacement map has a feature of expressing unevenness by moving the vertex of the 3D model up and down with respect to the actual surface.

  Then, it is necessary to register in the system the maximum facial expression intensity that the subject can express. For this reason, at the time of shooting, the photographer takes a facial expression that expresses the seven basic emotions (joy, sadness, anger, praise, surprise, disgust, fear) to the maximum extent.

  In the facial expression generation device disclosed in Patent Document 1, after data acquisition, the system user makes arbitrary modifications to the CG object of the subject. The facial expression generation device estimates a minute uneven shape variation from a plurality of displacement maps in accordance with the applied deformation, and creates an arbitrary facial expression.

  In the expression generation method disclosed in Patent Document 1, only an imaged person can generate an arbitrary expression including fluctuations in the fine unevenness of the imaged person. However, it is impossible to create an arbitrary expression of another person's face by giving information about the rough deformation and fine deformation (hereinafter referred to as expression variation) of the captured subject to other person's CG object.

US Patent Application Publication No. 2009/0195545 U.S. Pat.No. 7,548,272 JP 2001-076177

Yang Chen, Gerard Medioni `` Object modeling by registration of multiple range images '' Image and Vision Computing, Vol. 10, No. 3, pp. 145-155, April 1992 Thibaut Weise, Sofien Bouaziz, Hao Li, Mark Pauly "Realtime Performance-Based Facial Animation" ACM Transactions on Graphics, Proceedings of the 38th ACM SIGGRAPH 2011, August 2011 J. Ahlberg “CANDIDE-3-an updated parameterized face” Report No. LiTH-ISY-R-2326, Dept. of Electrical Engineering, Linkoping University, Sweden, 2001 Yeongho Seol et al. `` Spacetime Expression Cloning for Blendshapes '' ACM Transaction On Graphics, 31 (2), 14: 1-14: 12, 2012 Z. Deng, P. Y. Chiang, P. Fox, U. Neumann `` Animating Blendshape Faces by Cross-Mapping Motion Capture Data '' ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games 2006 (I3DG) Richard S. Sutton, Andrew G. Barto “Reinforcement Learning: An Introduction”, The MIT Press, 1998 Masatoshi Okutomi, Takeo Kanade “A Multiple-Baseline Stereo” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, No. 4, April 1993 Iain Matthews, Simon Baker “Active Appearance Models Revisited” International Journal of Computer Vision, Vol. 60, No. 2, pp. 135-164, November 2004 Beier T., Neely S. "Feature-based image metamorphosis" Computer Graphics 1992, Vol. 26 (2), pp. 35-42 J. Daugman “Entropy reduction and decorrelation in visual coding by oriented neural receptive fields” Trans. On Biomedical Engineering, Vol. 36, No. 1, 107-114, 1989 T. Ojala, M. Pietikainen, D. Harwood `` Performance evaluation of texture measures with classification based on Kullback discrimination of distributions '' Proceedings of the 12th IAPR International Conference on Pattern Recognition (ICPR 1994), Vol. 1, 582-585 P. Viola, M. Jones “Rapid Object Detection using a Boosted Cascade of Simple Features” IEEE Conference on Computer Vision and Pattern Recognition, 2001 "Computer Graphics", CG-ARTS Association, pp. 146-149, 2004 M Orokku, Kenkichi Fukuroya and Atsushi Okubo, "Principles of 3D computer animation", Modern Science, 94-98, 1997 David J. Fleet, Yair Weiss `` Optical Flow Estimation '' In Paragios et al., Handbook of Mathematical Models in Computer Vision, Springer, 2006

  An object of the present invention is to generate a variation model for generating a face model having an arbitrary expression of another person's face based on an individual's face skeleton model.

  The present invention has the following configuration as one means for achieving the above object.

The information processing according to the present invention generates a skeleton model for converting a general-purpose facial skeleton model into an individual facial skeleton model, and absorbs individual differences in facial expressions using the individual facial skeleton model and the skeleton model. A variation model is calculated, the variation model is classified into predetermined facial expression categories, the classified variation model is registered in a database, and the variation model is calculated as a weighted linear sum of a plurality of target shapes prepared in advance. Then, the weight of the target shape is calculated by a pseudo inverse matrix .

  According to the present invention, it is possible to generate a variation model for generating a face model having an arbitrary expression of another person's face based on an individual's face skeleton model.

The block diagram which shows the structural example of the facial expression production | generation apparatus which performs a front | former stage process. The figure which shows the example of drawing of a general purpose face model. The block diagram which shows the structural example of the facial expression production | generation apparatus which performs a back | latter stage process. The block diagram explaining the process structural example of an Example. The block diagram which shows the structural example of a fluctuation model database production | generation part. The flowchart explaining the process of a fluctuation | variation model database production | generation part. The flowchart explaining the process of an arbitrary facial expression production | generation part. FIG. 6 is a block diagram illustrating a configuration example of a facial expression generation apparatus according to a second embodiment. FIG. 9 is a block diagram illustrating a configuration example of a variation model database generation unit and an arbitrary facial expression generation unit according to the second embodiment. The flowchart explaining the process of a fluctuation | variation model database production | generation part. 9 is a flowchart for explaining processing of an arbitrary facial expression generation unit according to the second embodiment. FIG. 10 is a block diagram illustrating a configuration example of a variation model database generation unit and an arbitrary facial expression generation unit according to the third embodiment.

  Hereinafter, information processing relating to facial expression generation according to an embodiment of the present invention will be described in detail with reference to the drawings.

  In Example 1, an example will be described in which a skeletal model is defined for each individual, a facial expression variation model independent of the individual is created, and a desired facial expression is added to an arbitrary face.

  The processing of the first embodiment is roughly divided into a two-part configuration of the former stage and the latter stage. The first stage of processing calculates a general-purpose variation model of facial expressions that can be applied to all people, excluding individual differences in facial expressions from the face of a specific individual, and stores the general-purpose variation model in a database (DB). In the subsequent processing, a general-purpose variation model is applied to the face of a specific person to create an arbitrary expression on an arbitrary face.

[Calculation of general-purpose variation model]
Hereinafter, a method for calculating a general-purpose variation model of facial expression corresponding to the preceding process and registering the general-purpose variation model in the DB will be described.

  The block diagram of FIG. 1 shows an example of the configuration of a facial expression generation apparatus that executes pre-stage processing.

  The facial expression generation apparatus first prepares a general-purpose face model 100 that is a general-purpose face skeleton model. The general-purpose face model 100 is drawn by general computer graphics (CG) software. A drawing example of the general-purpose face model 100 is shown in FIG. The drawing example shown in FIG. 2 includes a three-dimensional vertex (XYZ coordinates) and a polygon mesh obtained by connecting adjacent vertices.

  In general, the shape of the general-purpose face model 100 does not have a fixed shape that becomes a de facto standard. Therefore, the system user (user) may create the general-purpose face model 100 manually, or create the general-purpose face model 100 that represents the average face shape from the face shapes of multiple persons registered in the DB. You can also.

  However, when creating an average face shape from the DB, of course, it is affected by the personal information (race, gender, age, etc.) of the person registered in the DB. It is desirable to create it according to the characteristics of For example, assuming that a facial expression is added to a Western face using this system, it is desirable that the general-purpose face model 100 is a face model including many features of a Western face.

  Next, the skeleton model 101 that is the face skeleton model of the person A is applied to the general-purpose face model 100 to transform the general-purpose face model 100 into the personal face model 102 of the person A. Specifically, this is a process of moving each vertex of the general-purpose face model 100 to the coordinates specified by the skeleton model 101. Therefore, the skeleton model 101 is represented as a matrix or a set of vectors for moving each vertex of the general-purpose face model 100.

Expression (1) shows a method of transforming the personal face model 102 by applying the skeleton model 101 to the general-purpose face model 100.
Characterized GM = Mc ・ GM… (1)

  Characterized GM (GM is an abbreviation of Generic Model) on the left side of Expression (1) is an individual face model of an expressionless (hereinafter referred to as reference expression) face (hereinafter referred to as an expressionless face) obtained as an input. Characterized GM is obtained by reconstructing each vertex of a depth image obtained as input data with a polygon mesh or the like. The position of each vertex of the expressionless face acquired here may be determined using an average value or median value of a plurality of consecutive frames in a state where there is no expression change. Alternatively, 3D data obtained by using a range scanner or the like for the face of the subject in a relaxed state may be employed.

  The GM on the right side is a matrix that stores all vertex coordinates in the three-dimensional space constituting the general-purpose face model 100. Mc is a skeleton model 101, and is a matrix for moving all the vertices of GM and transforming it into an individual face (in this case, person A). The matrix Mc defined as the skeletal model 101 includes information indicating the position and size of each part such as eyes, mouth, and nose that do not depend on changes in facial expression.

● Skeletal model calculation and personal face model generation There are several methods for calculating the skeletal model 101.

  For example, the technique of Patent Document 1 acquires a three-dimensional shape of a face using an apparatus that can simultaneously acquire an RGB image and a depth image. Then, the position and inclination of the general-purpose face model are adjusted by fitting the general-purpose face model to the general-purpose face model using an ICP (Iterative Closest Point) algorithm. Refer to Non-Patent Document 3 for the general-purpose face model, refer to Non-Patent Document 1 for the ICP (Iterative Closest Point) algorithm, and refer to Non-Patent Document 2 for adjusting the position and inclination of the general-purpose face model. For example, the general-purpose face model is vertex information indicating the shape of a face as described in Non-Patent Document 3, and topology information including these vertices.

  This is because the adjustment of the position and tilt of the face of the subject using the general-purpose face model is based on the assumption that shooting is performed without fixing the position and tilt of the face of the subject. In other words, the position and inclination of the face of the subject can be sufficiently fixed, and when the general-purpose face model is fitted, the process of adjusting the position and inclination is not necessary.

  In general, when applying a shape by the ICP algorithm, alignment is performed for the entire shape with respect to a rigid object. However, when a face is a target, there is a part to which the ICP algorithm cannot be applied because the face is deformed due to a change in facial expression. For example, when the jaw moves, the face shape changes greatly, making it difficult to align with the ICP algorithm.

  Therefore, in this embodiment, the upper half of the face (the periphery of the eyes and the forehead) is regarded as a rigid body without performing alignment on the entire face, and the shape of the shape is determined by point-to-plane association. Make a fit. This is because, in general, the periphery of the eye and the forehead part do not change as much as the chin due to the structure of the face skeleton.

  Then, each vertex of the general-purpose face model is moved so that each vertex of the general-purpose face model matches the shape of the actually acquired face. Naturally, each vertex of the general-purpose face model corresponds to each vertex of the general-purpose face model deformed according to the shape of the face acquired from the input signal. Therefore, the movement amount is calculated for each vertex, and the matrix Mc representing the skeleton model 101 can be obtained.

  Based on the skeleton model 101 obtained as described above, the general-purpose face model 100 is deformed to generate the personal face model 102 of the person A. In calculating the skeleton model 101, it is assumed that the texture to be attached to the individual face model 102 is acquired at the same time.

  Since the personal face model 102 is obtained by simply deforming the general-purpose face model 100 with the skeleton model 101, the personal face model 102 does not include facial expression information. However, a texture representing the skin color of the person A can be attached to the individual face model 102 as necessary.

  The technique of Patent Document 1 uses a sensor that acquires an RGB image and a depth image in real time, but such a special sensor is not necessarily used. By using multiple (two or more) general cameras capable of acquiring RGB signals, it is possible to acquire and reproduce minute uneven shapes such as wrinkles and pores on a person's face. Moreover, you may acquire an individual's high-definition face shape using a commercially available three-dimensional range scanner.

  As a specific method for acquiring the matrix Mc of the skeleton model 101 of a specific individual, first, depth data and texture data are acquired using a range finder or a commercially available camera. At that time, a marker point is attached to the subject's face and the same position of the general-purpose face model 100 (landmark points such as the corner of the eye or the mouth), and the matrix Mc of the skeleton model 101 is described from the movement of the marker point. Can do.

  Alternatively, the general-purpose face model 101 is deformed so that the depth data acquired by the range finder or the like and the matrix GM of the general-purpose face model 101 exactly match each other. At this time, the deformation amount (displacement vector of each vertex) given to the general-purpose face model 101 may be used as the matrix Mc of the skeleton model 101.

Generation of general variation model Next, in order to generate the general variation model 104, a personal face model 103 with an expression is input to the system.

  The shape of the face of the person A is acquired as the personal face model 103 with a facial expression using the same photographing means and method as the acquisition of the shape of the personal face model 102 of the person A. Naturally, the individual face model 103 with an expression is added with a variation of each vertex of the individual face model 102 as an expression with respect to the individual face model 102 as a reference.

  The general-purpose variation model 104 is obtained by absorbing individual differences from the individual face model 103 with an expression and calculating only the variation of the expression. Hereinafter, a method for calculating the amount of change in facial expression will be described.

First, the personal face model 103 with an expression is expressed as shown in Expression (2) when it is Characterized Expression GM.
Characterized Expression GM = Mc ・ Me ・ GM… (2)
Here, Mc is a matrix representing the skeleton model 101 of a specific individual (in this example, person A),
Me is a matrix that represents the general-purpose variation model 104 that shows the variation in facial expressions,
GM is a matrix that represents the shape of the general-purpose face model 100.

  In Equation (2), the matrix Mc of the skeleton model 101 of the person A is the difference between the three-dimensional shape data of the face of the person A obtained by the three-dimensional measurement and the three-dimensional shape of the general-purpose face model 100 as described above. This is a matrix representing the fluctuation component acquired based on the vector.

If the matrix Mc of the skeleton model 101 is expressed as a displacement vector group of each vertex of the matrix GM of the general-purpose face model 100, the matrix Mc is generally a regular matrix. Therefore, it is possible to obtain an inverse matrix of the matrix Mc. Therefore, Me · GM representing a general-purpose face model with a facial expression is calculated by multiplying both sides of Equation (2) by the inverse matrix Mc ⁻¹ of the matrix Mc from the left. That is, the individual difference is absorbed by multiplying the actually measured value by the inverse matrix Mc ⁻¹ of the matrix Mc representing the skeleton model 101.

  What is desired here is a matrix Me representing the general-purpose variation model 104. However, it is mathematically difficult to obtain the matrix Me directly.

  Non-Patent Document 4 prepares a plurality of face shapes called “target shapes” instead of directly calculating the matrix Me representing the general-purpose variation model 104, and adds a plurality of target shapes with a predetermined mixture ratio to a weighted linear Representing a certain intermediate expression by expressing the sum.

  The target shape defines the most deformed shape (vertex group) for each part of the face (left eye, right eye, left eyebrow, right eyebrow, etc.). For example, if the shape of the face (vertex group) when the eyebrows are raised to the maximum is defined as the target shape and each vertex of the general-purpose face model 100 is moved so as to be close to the target shape, the expression that raises the eyebrows is simulated Can be produced.

  In addition, an intermediate facial expression can be created by mixing a plurality of target shapes. For example, prepare target shape 1 with the left and right mouth corners raised and target shape 2 with the outside of the eyebrows lowered, and move each vertex of the general-purpose face model 100 by the weighted linear sum of the target shapes 1 and 2 This makes it possible to create a “smile”. Then, by adjusting the weight used for mixing the target shapes, the influence degree of each target shape can be controlled, and the expression intensity of the facial expression can be changed.

  In this way, the process of mixing a plurality of target shapes with a weighted linear sum is generally called “blend shape” (see Non-Patent Document 5).

  However, there are limits to the expression of facial expressions using blend shapes. A deformation exceeding the maximum deformed shape defined by the target shape or a deformation not included in the shape of the target shape prepared in advance cannot be expressed.

  In the present embodiment, this method is used to prepare about 30 to 40 target shapes for each facial part in advance, and the mixing ratio of these shapes is obtained by matrix calculation. As a result, Me / GM representing a general-purpose face model with a facial expression is obtained indirectly, and the weight w at that time is registered in the variation model database (DB) 105 as a general-purpose variation model 104.

Me / GM representing a general-purpose face model with facial expressions can be expressed by a weighted linear sum obtained by multiplying n target shapes T _k by a coefficient of weight w _k . The target shape T _k stores a coordinate value of a predetermined vertex group.
Me ・ GM = Σ _{k = 1} ⁿ w _k T _k (3)
Where n is the number of target shapes.

  Note that the number n of target shapes may be sufficient to express fluctuations due to facial muscles. In the above description, n is described as 30 to 40, but target shapes exceeding 40 may be prepared. However, since the deformation of the face due to the target shape is calculated by a weighted linear sum of the target shapes, it is desirable that the deformations of the individual target shapes are independent from each other.

  In addition, from the facial expression sample of the photographed subject, the mixture ratio of the target shape is fundamentally decomposed by principal component analysis or the like, and facial expression parameters resulting from the seven basic emotions are calculated. Then, the mixing ratio of the target shape can be automatically calculated from the parameter.

  The mixing ratio (weight) of the target shape may be calculated by obtaining a pseudo inverse matrix or analytically calculated by the Lagrange undetermined multiplier method.

  Here, in order to simplify the description, an example in which the face shape is acquired using the same method has been described. However, depending on the system to be constructed, the face shape may be acquired using another method. This is because if the shape of the personal face model 102 is to be acquired with high accuracy, the shape acquisition device will be enlarged according to the accuracy, and the light source environment at the time of acquisition will often be restricted.

  As an individual face model 103 with a facial expression, it is easily imagined that a device that acquires a shape with high accuracy cannot always be used if an ordinary natural expression is taken. For this reason, it is assumed that the shape acquisition method used for the personal face model 102 and the shape acquisition method of the personal face model 103 with an expression are different. In that case, separately, the measured values of the personal face model 103 with facial expression and the corresponding points of the personal face model 102 are sequentially obtained, so that the conditions described in the method for acquiring the face shape using the same method are made equal. Can do.

  Even if the shape acquisition methods are the same or different, the accuracy of the general-purpose variation model depends on the accuracy of the skeleton model 101, and thus the skeleton model 101 needs to be acquired with high accuracy.

  The weight w is categorized for each facial expression desired to be generated in the subsequent stage and registered in the variation model DB 105. The categories used for classification (hereinafter referred to as facial expression categories) are the seven basic emotions (surprise, joy, anger, fear, sadness, disgust, fear). Furthermore, facial expressions with mixed emotions, facial expressions with different meanings appearing in the peripheral area of the eye and meanings appearing around the mouth, such as smirk, can be prepared as facial expression categories.

  Here, a method of classifying each facial expression category using the weight w calculated as the general-purpose variation model 104 as a parameter and registering it in the variation model DB 105 will be described. Although an example in which the weight w is registered as a parameter of the general-purpose variation model 104 has been described above, the parameter to be registered is not limited thereto.

  Parameters calculated as the general-purpose variation model 104 can be classified into facial expression categories using a general machine learning framework such as a perceptron, a neural network, and a support vector machine.

  An example of the classification method is a method of calculating a target shape weight w for the general-purpose face model 100 with a facial expression set by a system user and using the weight w as a parameter representing the facial expression category for identification.

  When various facial expressions are input as input samples, the facial expression category cluster is updated using k-means clustering, multi-class SVM, or the like. Since various methods are known for updating the cluster, a suitable update method may be selected depending on the system to be constructed.

  In addition, since facial expressions with clear emotions have large variations in facial expressions, clustering is easier than when facial variations are small. On the other hand, since a fine facial expression has a small variation in facial expression, it is difficult to determine whether it is classified into a cluster of an expressionless category or an expression category other than that. In that case, let the several subjects observe the general-purpose face model with facial expressions generated by applying the parameters of the general-purpose variation model 104 to the general-purpose face model 100, and use the reinforcement learning framework based on the classification result by subjective evaluation Update the facial expression category (see Non-Patent Document 6).

  After absorbing individual differences as described above, the general-purpose variation model 104 for facial expressions is calculated as a parameter, and parameters are registered and accumulated in the variation model DB 105 for each predetermined facial expression category.

  The facial expression categories registered in the variation model DB 105 may have a hierarchical structure using attribute information such as age, gender, race, nationality, and the like.

[Add facial expression]
Next, a method for adding an arbitrary expression to an arbitrary face, which corresponds to the subsequent process, will be described.

  The block diagram of FIG. 3 shows an example of the configuration of a facial expression generation apparatus that executes subsequent processing. FIG. 3 shows a configuration obtained by adding the configuration of the post-stage process to the configuration of the pre-stage process shown in FIG. Therefore, the description of the configuration of the pre-processing is omitted.

  The person B is a person who is different from the person A and is not related to the calculation of the general variation model 104. The skeleton model 301 of the person B can be acquired by the same method as the method of acquiring the skeleton model 101 of the person A. The personal face model 302 of the person B is a face model obtained by deforming the general-purpose face model 100 by the skeleton model 301 of the person B, and its shape is theoretically derived from an input signal in a state where there is no expression (reference expression). It is equivalent to the three-dimensional shape that is constructed.

  Here, the general variation model 104 (parameter) calculated from the personal face model 103 with the expression of the person A and accumulated in the variation model DB 105 is applied to the face image of the person B, and the face image of the person B is changed to an arbitrary expression. Deform.

  At this time, the system user is presented with a facial image of the reference facial expression (no expression) of person B via the screen, and at the same time, the representative parameters of the facial expression category registered in the variation model DB 105 are used as the general-purpose facial model 100. A plurality of face images having facial expressions applied to is presented. Hereinafter, a face having a facial expression is referred to as a “facial expression face”.

  The user selects a facial image having a desired facial expression from a plurality of facial expression facial images, and instructs application to the facial image of person B. Alternatively, a list of facial expression categories may be presented to the user, and facial expressions may be added to the face image of the person B based on the facial expression category selected from the list by the user. When the expression category is selected, the representative parameter (the parameter closest to the cluster centroid) registered in the variation model DB 105 is applied to the face image of the person B in order. If the user desires as a result of the application, other parameters of the same expression category are applied to the face image of the person B.

  The person B is involved in the calculation of the general fluctuation model 104, and the general fluctuation model 104 can be generated from the skeleton model 301 of the person B, the personal face model 302 of the person B, and the personal face model 303 with the expression of the person B. In this case, if an appropriate parameter is selected from the variation model DB 105, the facial expression generated from the general-purpose face model 100, the skeleton model 301 of the person B, and the general-purpose variation model 104 matches the actual expression of the person B.

  A processing configuration example of the embodiment will be described with reference to the block diagram of FIG. The facial expression generation apparatus is realized by executing software (programs) acquired from various recording media on a computer device via a network. The computer device may be a general-purpose device such as a personal computer including a CPU, a memory, a storage device, an input / output device, a system bus, a display device, and the like. Alternatively, a dedicated device having hardware optimized for executing software for generating facial expressions may be used.

  In FIG. 4, a database generation unit 400 corresponds to a part that executes the above-described pre-stage processing, and an arbitrary facial expression generation unit 401 corresponds to a part that executes the post-stage processing. The database generation unit 400 includes an input data acquisition unit 402, a variation model database generation unit 403, and a variation model DB 105.

  For example, the input data acquisition unit 402 acquires RGB images from a plurality of commercially available RGB cameras, and acquires depth information from the RGB images using a multi-baseline stereo method or the like (see, for example, Non-Patent Document 7). Then, the acquired RGB image and depth information are input to the variation model database generation unit 403. Various methods of acquiring depth information have been proposed. In addition to the acquisition method using the stereo method, there is a method of acquiring depth information using a depth camera or a range sensor. As a method for acquiring depth information, a suitable means may be selected for each system to be constructed.

Fluctuation Model Database Generation Unit A configuration example of the variation model database generation unit 403 is shown by the block diagram of FIG.

  The skeletal model generation unit 500 determines whether or not the skeleton model of the subject is registered in the skeleton model database (DB) 501 based on the information of the subject (for example, the person A) input from the input data acquisition unit 401. Search for. If a corresponding skeleton model is found as a result of the search, settings for applying the skeleton model to the general-purpose face model are performed. If the corresponding skeleton model is not found, the skeleton model of the subject is generated by the above-described method using the depth information input from the input data acquisition unit 401, and the generated skeleton model is stored in the skeleton model DB501. Register with.

The skeleton model application unit 502 uses the skeleton model registered in the skeleton model DB 501 to absorb individual differences included in the depth information input from the input data acquisition unit 402. In other words, as shown in Equation (2), a generalized face model with a facial expression Me that eliminates individual differences by multiplying the individualized facial model with expression (Characterized Expression GM) by the inverse matrix Mc ^-1 of the skeleton model Mc of a specific individual・ Generate GM.

  The variation model calculation unit 503 calculates a variation model using the weight w of the target shape as a parameter for the general-purpose face model Me / GM with an expression generated by the skeleton model application unit 502. The expression category classification unit 504 classifies the variation model calculated by the variation model calculation unit 503 in accordance with the expression category of the variation model DB 105.

When the variation model is expressed using the weight w of the target shape as a parameter, clustering is performed using the parameter, and a facial expression label is added to the cluster. For example, the expression of a smile is expressed as in equation (4) by a target shape T _A that moves the lower eyelid upward, a target shape T _B that lowers the corner of the eye, a target shape T _C that moves the mouth corner upward, and a weight w. The Note that the target shape T _A , T _B , and T _C store the coordinate values of predefined vertex groups.
Smile = w _A T _A・ w _B T _B・ w _C T _C
For example, (w _A , w _B , w _C ) = (1.0, 0.5, 0.7)

  In this way, the weighting w is calculated for each facial expression category to perform parameter clustering. For clustering, a general clustering method such as k-means may be used. Then, the cluster (expression category) to which the parameter (variation model) calculated by the variation model calculation unit 503 belongs is calculated, and the variation model is registered in the variation model DB 105 according to the calculation result.

  The process of the fluctuation model database generation unit 403 will be described with reference to the flowchart of FIG.

  The skeleton model generation unit 500 acquires input data (image and shape information (depth information) and subject information) obtained by photographing the subject's face from the input data obtaining unit 401 (S601). Then, it is searched whether or not the skeleton model corresponding to the subject is registered in the skeleton model DB 501 (S602). If the skeleton model of the subject is registered in the skeleton model DB 501, the process proceeds to step 605. If the skeleton model of the subject is not registered, the process proceeds to step S603.

  The simplest search method is to register information specific to the subject, such as the subject's name, as the skeleton model label when registering the skeletal model, and match the subject's name with the skeleton model label when searching. Can be considered. In addition, when a faceless facial image of the subject is input, the identification of the subject is performed by matching the shape of the face indicated by the input image with the shape deformed from the general-purpose face model 100 based on the skeleton model. Alternatively, the presence / absence of the subject's skeleton model can also be determined.

  When the skeleton model of the subject is not registered, the skeleton model generation unit 500 generates the skeleton model of the subject (S603), and the skeleton model added with information specific to the subject as a label is the skeleton model DB501. (S604).

  The skeleton model is generated by moving the positions of the general-purpose face model 100 based on the depth information after matching the position of the depth information included in the input data with the position of the general-purpose face model 100. As input data, there is a case where depth information of minute irregularities such as pores and wrinkles can be acquired. In that case, after moving each vertex of the general-purpose face model 100, it is possible to preserve the fine unevenness of the depth information as a displacement map showing the height difference in the normal direction of each mesh (plane) of the general-purpose face model 100 It is. Alternatively, a displacement map may be generated by taking a height difference from each mesh in the Z-axis (depth) direction. The generated displacement map is stored in the skeleton model DB 501 together with the skeleton model.

  Next, the skeleton model application unit 502 reads the skeleton model of the subject from the skeleton model DB 501 (S605). The skeleton model is a vector or matrix for moving each vertex of the general-purpose face model 100. Then, based on the read skeleton model, all the vertices of the general-purpose face model 100 are moved to generate a personal face model of the subject (S606). The personal face model is an expressionless face obtained by deforming the general-purpose face model 100 based on the skeleton model.

  Next, the variation model calculation unit 503 further moves each vertex of the personal face model generated in step S606 based on the depth information included in the input data, and records the movement amount of each vertex as a variation model due to facial expression variation. (S607). Also, the displacement map is acquired again according to the expression at this time.

  Next, the facial expression category classification unit 504 identifies the facial expression category to which the variation model is registered (S608). That is, the label of the corresponding facial expression category is given to the variation model and the displacement map generated in step S607. Then, the variation model and the displacement map are registered in the variation model database 105 (S609).

  Through the above processing, the skeleton model DB 501 and the variation model DB 105 are constructed. That is, by generating a skeleton model and calculating a variation due to facial expression as a variation model from a general-purpose face model to which the skeleton model is applied, a general-purpose variation model that absorbs individual differences can be registered in the variation model DB 105.

Arbitrary Expression Generation Unit The arbitrary expression generation unit 401 uses the variation model DB 105 to virtually add an expression to the personal face model of the person to whom an expression is to be added.

  The arbitrary facial expression generation unit 401 uses the data stored in the skeleton model DB 501 and the variation model DB 105 to perform appropriate facial expression deformation on the face shape of an arbitrary person.

  The personal face model setting unit 405 acquires from the skeleton model DB 501 a skeleton model of the face of the person desired by the system user.

  The expression-equipped face model generation unit 406 generates a personal face model by deforming the general-purpose face model 100 based on the acquired skeleton model. The face model obtained here is always expressionless. Furthermore, an expression based on the general-purpose variation model is added to the generated personal face model.

  The process of the arbitrary facial expression generation unit 401 will be described with reference to the flowchart of FIG.

  The personal face model setting unit 405 acquires a person profile in order to set a personal face model of a person to whom a facial expression is added (S701). The person profile includes information unique to the person, such as a name and a face photograph, and is used to read a skeleton model from the skeleton model DB 501.

  Next, the personal face model setting unit 405 determines whether or not there is a skeleton model (S702), generates a skeleton model (S703), registers a skeleton model (S704), and reads a skeleton model (S705). These processes are the same as the processes of steps S602 to S605 in the variation model database generation unit 403, and detailed description thereof is omitted.

  Next, the personal face model setting unit 405 causes the system user to select a desired facial expression category from the facial expression categories set in the variation model DB 105, and acquires the selection information (S706).

  When a plurality of facial expression samples are registered for each facial expression category, parameters are randomly extracted from the facial expression samples, and facial expressions are added to the general-purpose face model 100. That is, one or more general-purpose face models with facial expressions are displayed for each facial expression category, and the system user is allowed to select a desired general-purpose face model with facial expressions. If there is no facial expression desired by the system user, parameters are extracted from the facial expression sample again to change the general-purpose face model with facial expression to be presented to the system user.

  Next, the expression-equipped face model generation unit 406 applies an expression sample parameter indicated by the selection information to the general-purpose face model 100 to generate a general-purpose face model with an expression (S707). Then, the skeleton model is applied to the general-purpose face model with an expression to add an arbitrary expression to an arbitrary face (S708).

  Steps S707 and S708 need to be executed in this order. If the execution order is changed, the desired deformation cannot be performed. If the skeleton model is being read, the expression adding process can be executed by two matrix operations. Therefore, it is also possible to create a virtual facial expression by adding a facial expression in real time to the face of a person acquired with a video camera.

  As described above, a general-purpose variation model of a facial expression that absorbs individual differences is generated from the input data (a specific individual's image and depth information), and the desired face can be arbitrarily selected using the general-purpose variation model of the desired face. Can create a facial expression.

  In this way, the individual model is absorbed by the skeleton model expression for each individual, the variation due to facial expressions is calculated as a general variation model, and a variation model database that registers the general variation model of facial expressions applicable to everyone is constructed. Can do. Even if the person is changed, by redefining the skeletal model, a desired expression change can be added to the individual face model of the specific person using the general-purpose variation model.

  The information processing according to the second embodiment of the present invention will be described below. Note that the same reference numerals in the second embodiment denote the same parts as in the first embodiment, and a detailed description thereof will be omitted.

  In the first embodiment, a personal face model 102 obtained by deforming the general-purpose face model 100 based on the skeleton model 101 of the person A is created, and a personal face model 103 with a facial expression of the person A (a three-dimensional shape variation model depending on the expression) is acquired. To do. Then, a method for generating a general-purpose variation model 104 applicable to all persons from the personal face model 103 with an expression using the skeleton model 101 and adding an arbitrary expression using the general-purpose variation model 104 has been described.

  In the second embodiment, as in the first embodiment, a method of extracting a variation model of a three-dimensional shape due to a facial expression from depth information included in input data and applying the facial expression variation of the person A to the personal face model of the person B will be described. To do. However, in the second embodiment, the individual difference using the skeleton model is not absorbed, and the facial expression variation of the person A is directly applied to the personal face model of the person B. Of course, in the second embodiment, as in the first embodiment, an object is to construct a variation model database 105 based on facial expressions from the facial expressions of the person A and apply them to the personal face model of the person B to create a desired facial expression.

  A block diagram of FIG. 8 shows a configuration example of the facial expression generation apparatus according to the second embodiment.

  In the first embodiment, an example has been described in which the general variation model 104 is created by absorbing individual differences by calculating the skeleton models of the person A and the person B with high accuracy. In the second embodiment, a variation model is generated from the difference between the expressionless face and the expressional face of the same person.

  The general-purpose variation model 104 according to the first embodiment models the expression variation from the general-purpose face model 100. On the other hand, the variation model 802 of the second embodiment describes the movement of each vertex when the shape changes from the expressionless face model 800 of the person A to the expressional face model 801 in a relative coordinate system. Here, the relative coordinate system is a coordinate system in which, for example, a marker is attached to the face, and a point determined by the marker is used as a reference point (origin), and the marker position after the shape change is expressed by two-dimensional coordinates or three-dimensional coordinates. . Each marker is set with an ID corresponding to the position of the attached face. In other words, in the relative coordinate system variation model (hereinafter referred to as relative coordinate variation model), for each marker ID, the coordinates indicating the marker position of the facial expression face model 801 are described with the marker position of the expressionless face model 800 as the origin. Has been.

  In addition, although the example which sets the origin of a relative coordinate system with the marker was explained, the landmark of the face (the part where the position such as the corner of the eye or the mouth edge can be uniquely specified) is used as the reference point (origin) instead of the marker. It may be adopted.

  Then, by preparing the expressionless face model 804 of the person B corresponding to the reference expression, and applying the relative coordinate variation model 802 to each vertex of the expressionless face model 804 of the person B, the person B having an arbitrary expression A face model 805 with expression is generated.

  Specifically, the three-dimensional shape of the face in a state in which the person A is relaxed (a state in which the expression change does not occur as much as possible) is acquired over a plurality of frames. An average vertex is calculated in consideration of the obtained vertex fluctuations in a plurality of frames, and the face shape constituted by the vertexes is defined as an expressionless face model 800. At this time, it is desired that all the vertices constituting the three-dimensional shape of the face can be matched among a plurality of frames.

  Patent Document 2 discloses an acquisition method that closely matches each vertex of a face among a plurality of frames. The method of Patent Document 2 applies a fluorescent paint instead of a marker on the face of the subject, emits a strobe while controlling the timing, and uses a dense pattern generated by uneven application of the fluorescent paint to obtain a high-precision depth. Get an image. The advantage of this method is that the dense shape of the face can be reproduced by pattern matching with the fine pattern generated by the uneven application of the fluorescent paint applied to the face. Furthermore, the corresponding point search between frames can be accurately performed until the fluorescent paint is removed from the face.

  In the second embodiment, the expressionless face model 800 of the person A is generated using the technique disclosed in Patent Document 2. Then, based on the premise that all vertices can be handled among a plurality of frames, the person A creates some facial expressions and obtains facial expression samples.

  As described above, the face model 801 with a facial expression of the person A is a face model with a facial expression acquired in the state where the expressionless face model 800 of the person A is generated or in the same shooting environment. The technique disclosed in Patent Document 2 makes it difficult to search for corresponding points between frames when the fluorescent paint is dropped from the face.

  The relative coordinate variation model 802 records the variation of all corresponding points of the expressionless face model 800 of the person A and the face model with expression 801 obtained in this way (represented by a movement amount or a vector in 3D space). Is.

  In the variation model database 105, as in the first embodiment, facial expression categories are set in advance. When registering the relative coordinate variation model 802, the user of the system actually confirms the face model 801 with a facial expression of the person A and adds a facial expression label representing an appropriate facial expression category.

  The expressionless face model 804 of the person B is generated in the same manner as the expressionless face model 800 of the person A. Here, in order to apply the relative coordinate variation model 802 generated from the facial expression of the person A to the expressionless facial model 804 of the person B, it is necessary that the correspondence between each vertex of the facial model of the person A and the person B is taken. is there. However, person A and person B differ greatly in face skeleton and surface area, and the way of applying fluorescent paint (unevenness pattern) is different, so that all the vertices of face model of person A and all vertices of face model of person B It is difficult to search for corresponding points between.

  Patent Document 3 discloses a process for obtaining a correspondence between frames of a detailed polygon model.

  The method disclosed in Patent Document 3 for associating CG objects with different shapes reduces the number of polygons from the dense shape of objects A and B (high-level polygon model) by polygon reduction processing, resulting in a sparse shape (low-level) Polygon model). At that time, the process of reducing the number of polygons is stored as metadata. Then, after associating each vertex of the low-level polygon model, each polygon model is restored to a high-level polygon model using the above-described metadata. Through the above processing, the high-definition polygon objects A and B can be associated with each other.

  However, if the polygon reduction process is applied uniformly to the polygons that make up the face shape by the method of Patent Document 3, the information of the parts such as eyes, eyebrows, and mouth that have more information than other parts will be lost. To do. Therefore, without applying the polygon reduction process uniformly, as shown in Non-Patent Document 3, the vertex of the expressionless face model 804 of the person B corresponding to each vertex of the expressionless model 800 of the person A is weighted and reduced. It is necessary to suppress.

  In addition, associating vertices of the low-level polygon model, for example, the system user may manually specify the vertices around the eyes, mouth, and nose, which are important parts of the face. Further, the correspondence between the vertices of the facial feature parts may be taken using a method such as Active Appearance Model (see Non-Patent Document 8).

  In this way, by performing the process of associating the vertices, correspondence between the expressionless face model 800 of the person A and the expressionless face model 804 of the person B is obtained. Then, by applying the relative coordinate variation model 802 calculated from the expression of the person A to each vertex constituting the expressionless face model 804 of the person B, the expression variation of the person A can be applied to the face model of the person B. it can.

  In the second embodiment, instead of using a skeleton model, a relative coordinate variation model 802 in which a shape variation due to a facial expression is modeled using an individual expressionless face model 800 is generated. Taking the expression “smile” as an example, smiles are common to all races and genders, and facial shape fluctuations such as lowering the corners of the eyes and increasing the corners of the eyes can be seen. The shape variation common to everyone can be sufficiently described by a variation model based on the expressionless face model 800 without using a skeleton model.

Variation Model Database Generation Unit A configuration example of the variation model database generation unit 403 and the arbitrary facial expression generation unit 401 of the second embodiment is shown by the block diagram of FIG.

  As in the first embodiment, the input data acquisition unit 402 inputs an RGB image of the face of the subject (for example, a person A) and high-definition depth information.

  The facial expression model generation unit 900 takes an average value or a median value between a plurality of frames in order to remove minute fluctuations at each vertex of the high-definition polygon model input from the input data acquisition unit 402, and performs an expressionless facial model 800. Then, a face model 801 with an expression is generated.

  The relative coordinate variation model calculation unit 901 calculates the amount of movement of each vertex between the expressionless face model 800, which is a high-definition polygon model, and the facial model with expression 801, as a relative coordinate variation model 802.

  The facial expression category classification unit 504 performs processing similar to that in the first embodiment, adds a label of the facial expression category to the relative coordinate variation model 802 calculated by the relative coordinate variation model calculation unit 901, and converts the relative coordinate variation model 802 into the variation model DB 105. Register with. Of course, if necessary, in addition to the expression category label, attribute information such as a person's name is registered in the variation model DB 105 at the same time.

  The corresponding point search unit 902 of the arbitrary facial expression generation unit 401 performs the above-described association processing, and associates all vertices of the expressionless face model 800 of the person A and the expressionless face model 804 of the person B.

  The expression-equipped face model generation unit 406 applies the relative coordinate variation model 802 stored in the variation model database 105 corresponding to the expression category selected by the user of the system, and the person B who has completed the association process An expression variation is added to the expression face model 804.

  The process of the fluctuation model database generation unit 403 will be described with reference to the flowchart of FIG.

  The facial expression model generation unit 900 acquires input data (image and shape information (depth information) and subject information) obtained by photographing the subject's face from the input data obtaining unit 401 (S1001). . Then, it is searched whether or not the subject's expressionless face model 800 exists (whether or not stored in the system memory) (S1002). If the expressionless face model 800 of the subject is present, the process proceeds to step 1004.

  If the expressionless face model 800 of the subject does not exist, the expression face model generation unit 900 generates the expressionless face model 800 (S1003). The generation of the expressionless face model 800 requires input data over a plurality of frames that are continuous in time series. Therefore, the facial expression model generation unit 900 returns the process to step S1001 to acquire input data until sufficient input data (samples) are collected.

  Next, the facial expression model generation unit 900 acquires input data (image and shape information (depth information) and information on the subject) obtained by photographing the subject's face from the input data acquisition unit 401. (S1004). Then, it is searched whether or not a face model with a facial expression of the photographed person exists (whether or not stored in the memory of the system) (S1005). If there is an expressionless face model of the subject, the process proceeds to step 1007.

  When the face model with facial expression of the subject does not exist, the facial expression model generation unit 900 generates a facial model with facial expression (S1006). Generation of a facial model with a facial expression requires input data over a plurality of frames that are continuous in time series. Therefore, the facial expression model generation unit 900 returns the process to step S1004 to acquire input data until sufficient input data (samples) is collected.

  Further, the expressionless face model 800 and the face model with expression 801 of the photographed subject are expected to change slightly due to various factors such as the state of the skin at that time, the mood of the day, and aging. Therefore, in the present embodiment, it is assumed that the expressionless face model 800 and the expression-equipped face model 801 are generated as ad hoc data and are not stored in a storage device or the like. However, depending on the system to be constructed, once the expressionless face model 800 and the expression-equipped face model 801 are generated, the expressionless face model 800 and the expression-equipped face model 801 are stored in the system storage device along with the attribute information of the subject. May be saved.

  Next, the relative coordinate variation model calculation unit 901 calculates the amount of movement of each vertex between the expressionless face model 800 and the facial model with expression 801 as a relative coordinate variation model 802 (S1007).

  Next, the facial expression category classification unit 504 adds a label of the facial expression category to the calculated relative coordinate variation model 802 and registers the relative coordinate variation model 802 in the variation model DB 105 (S1008).

  Processing of the arbitrary facial expression generation unit 401 of the second embodiment will be described with reference to the flowchart of FIG.

  The facial expression model generation unit 900 acquires input data (image and shape information (depth information) and subject information) obtained by photographing the subject's face from the input data obtaining unit 401 (S1101). . Then, it is searched whether or not the subject's expressionless face model 804 exists (whether or not stored in the system memory) (S1102). If the subject's expressionless face model 804 exists, the process proceeds to step 1104.

  If the expressionless face model 804 of the subject does not exist, the expression face model generation unit 900 generates an expressionless face model 804 (S1103). The generation of the expressionless face model 804 requires input data over a plurality of frames that are continuous in time series. Therefore, the facial expression model generation unit 900 returns the process to step S1101 to acquire input data until sufficient input data (samples) is collected.

  Next, the corresponding point search unit 902 determines whether or not the correspondence between the vertices of the expressionless face model 800 and the expressionless face model 804 is obtained (S1104). Processing for searching for corresponding points is performed (S1105).

  When the correspondence between the vertices is taken, or when the corresponding point search ends, the facial model with facial expression generation unit 406 performs the facial expression desired by the system user from the facial expression category set in the variation model DB 105, as in step S706. Let the category be selected. Then, the selection information is acquired (S1106). Then, by applying the relative coordinate variation model 802 corresponding to the selection information to each vertex of the expressionless face model 804 of the person B who can correspond to the vertex, the face model 805 with the expression of the person B who moved each vertex Is generated (S1107).

  As described above, the facial expression generation apparatus according to the second embodiment uses the depth data (depth information) to generate an expressionless face model of each person, and the variation from the expressionless face model caused by the expression change. Are stored in the variation model DB 105 as a relative coordinate variation model 802. Thereby, an expression can be added to an arbitrary face model.

  Note that the facial expression generation apparatus according to the second embodiment can describe a variation model based on facial expressions by two-dimensional image processing without using depth data. For example, the coordinate values of facial feature points can be calculated using Active Appearance Model (AAM) described in Non-Patent Document 8. An expressionless face model and a face model with expression can be generated from the coordinate values of the facial feature points. Using the method described in the second embodiment, a facial expression variation model can be obtained from a two-dimensional image feature amount. Can be generated. Then, using a two-dimensional morphing method (see Non-Patent Document 9), an expression is added to an arbitrary face model.

  Hereinafter, information processing according to the third embodiment of the present invention will be described. Note that the same reference numerals in the third embodiment denote the same parts as in the first and second embodiments, and a detailed description thereof will be omitted.

  In the third embodiment, a method of using a texture variation model in addition to a two-dimensional or three-dimensional variation model will be described. In the following, as Example 3, an example in which a texture variation model is added to the configuration of Example 1 will be described. Of course, an embodiment in which a texture variation model is added to the configuration of the embodiment 2 can also be realized.

  The block diagram of FIG. 12 shows a configuration example of the variation model database generation unit 403 and the arbitrary facial expression generation unit 401 of the third embodiment.

  The texture information calculation unit 1200 performs image processing on the RGB image input from the input data acquisition unit 402 to calculate face texture information. Texture information includes, for example, wrinkles, pores, pupil color, lip color, eyebrow shape, skin color in a uniform light source environment, face landmarks (such as the corners of the eyes, mouth, and eyebrows) Area pattern information (texton, textone) having the following image features. Texton makes it easy to grasp the distinctive features of an image by clustering the image according to a certain rule. Textons are described using various feature quantities such as Gabor features (see Non-Patent Document 10) and LBP features (see Non-Patent Document 11).

  In general, texture information can be calculated by the following two methods.

  In the first method, a face region is specified from a two-dimensional input image by the aforementioned Active Appearance Model (AAM), face detection using Haar-like feature quantities (see Non-Patent Document 12), or the like. Then, after registering the position of the face between frames, the texture variation of the face is acquired. The texture variation model according to this method is a model in which a luminance change or a color change at each coordinate of a two-dimensional image between frames is described as a time series signal.

  The second method obtains the color of each vertex of the face shape restored in three dimensions from the input image, expands the color of each vertex of the three-dimensional face model into texture coordinates, and calculates the texture information (non-patent) Reference 13).

  In the standard CG technique described in Non-Patent Document 13, when mapping a texture to a parametric curved surface such as the surface shape of the face, the point (x, y, z) on the curved surface in the three-dimensional space Generally expressed by parameters (u, v). By changing these parameters in the range of 0 ≦ u ≦ 1 and 0 ≦ v ≦ 1, the three-dimensional coordinate value of the point on the curved surface and the pixel value of the texture can be obtained. Using this texture mapping technique, the parameters (u, v) can be defined as texture coordinates.

  Then, a transformation matrix between the vertex coordinates (x, y, z) and texture coordinates (u, v) of the three-dimensional object is obtained in advance, and the color of each vertex of the face shape restored in three dimensions is determined by the texture coordinates (u, v). It becomes possible to project to. As a method for acquiring texture information, a method suitable for the system to be constructed may be used.

  The texture variation model generation unit 1201 uses a skeleton model to absorb individual differences from the texture information calculated by the texture information calculation unit 1200 and models a texture variation component due to facial expression variation.

  This processing is performed by extracting texture information (features such as texton) from the entire image or for each local region from the texture image developed in the texture coordinates. Assuming that the texture information is extracted from the restored three-dimensional shape, the surrounding areas of the eyes, nose, and mouth of the texture developed in the texture coordinates are mapped to the polygon coordinates uniquely determined in the texture coordinates. This mapping method is generally called parameterized texture mapping or UV mapping (see Non-Patent Document 14). However, this is based on the premise that all the correspondences between the vertices are taken in different three-dimensional face shapes.

  The three-dimensional face shape of the person A that can be acquired from the input data is temporarily transformed into the general-purpose face model 100 using the skeleton model 101 of the person A. Once converted into the general-purpose face model 100, the three-dimensional face shapes of each expression can be associated with each other. Then, the texture information of each associated vertex is developed into texture coordinates to generate a texture image. In the third embodiment, the generated texture image is registered in the variation model DB 105 as a texture variation model.

  The texture variation model may be composed of a single texture image, or the time-series variation of the obtained texture image is calculated using an optical flow estimation process (see Non-Patent Document 15). A model may be constructed.

  When registering the texture variation model in the variation model DB105, the variation model of the texture component on the face (texture variation model) paired with the variation model of the 3D shape component (3D shape variation model) with the expression category label added Register. Note that the 3D shape variation model is represented as a displacement vector of each corresponding vertex or a displacement vector of a normal line in the corresponding polygon. The 3D shape variation model and the texture variation model may be calculated and registered for each local region of the face.

  The expression-equipped face model generation unit 505 randomly selects a 3D shape variation model and a texture variation model registered in the expression category selected by the user of the system and applies them to the general-purpose face model 100. Then, by transforming the general-purpose face model after applying the variation model into the face of the person A or person B registered in the skeleton model DB 501, an arbitrary expression can be added to the arbitrary face model.

  If the 3D shape variation model is defined with the target shape weight w, the texture is blended using the same weight w as the 3D shape variation model and applied to the general-purpose face model 101. Also good. Alternatively, the texture can be switched for each local region (part) of the face.

  Also, the displacement model belonging to multiple facial expression categories, in other words, the weighted linear sum of multiple variation models with different facial expression categories, is used to determine the displacement amount of the 3D shape of each vertex, and the displacement amount is applied to the reference facial expression. A face model having an arbitrary expression may be generated.

  The processing of the third embodiment is substantially the same as the processing of the first embodiment, and detailed description thereof is omitted.

[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (6)

A skeleton model generation means for generating a skeleton model for converting a general-purpose facial skeleton model into a personal facial skeleton model;
Calculating means for calculating a variation model that absorbs individual differences in facial expressions using the individual's facial skeleton model and the skeleton model;
The variation model classified for each predetermined expression category, have a registration means for registering the variation model the classification database,
The information processing apparatus calculates the variation model as a weighted linear sum of a plurality of target shapes prepared in advance, and calculates the weight of the target shape by a pseudo inverse matrix .   The skeleton model generation means generates a skeleton model from a face skeleton model of an individual's reference facial expression, and generates a skeleton model with an expression representing a variation of each vertex of the skeleton model from the face skeleton model with an individual expression. Item 4. The information processing device according to Item 1.   3. The information processing apparatus according to claim 2, wherein the calculation unit calculates the variation model in which individual differences are absorbed from the skeleton model with an expression using a facial skeleton model of the reference facial expression of the individual.   Further, the variation model corresponding to the expression category selected by the user is applied to the skeleton model generated from the facial skeleton model of the reference facial expression of the individual by the skeleton model generation unit to generate the facial model with the expression of the individual The information processing apparatus according to any one of claims 1 to 3, further comprising a face model generation unit that performs the processing. An information processing method for an information processing apparatus having a generation unit, a calculation unit, and a registration unit,
The generating means generates a skeleton model for converting a general-purpose facial skeleton model into a personal facial skeleton model;
The calculation means calculates a variation model that absorbs individual differences in facial expressions using the individual face skeleton model and the skeleton model;
The registration means classifies the variation model for each predetermined facial expression category, registers the classified variation model in a database ,
The information processing method , wherein the calculation means calculates the variation model as a weighted linear sum of a plurality of target shapes prepared in advance, and calculates the weight of the target shape by a pseudo inverse matrix . The program for functioning a computer as each means of the information processing apparatus as described in any one of Claims 1-5 .