JP2007004732A - Image generation device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image generation device for generating a new image on which such a parameter as the shape, dress and motion of a joint object existing in an image is reflected. <P>SOLUTION: This image generation device for generating a new image on which the characteristics of a joint object are reflected from an image obtained by picking up the image of the joint object is provided with an image input part 101 for acquiring an image by picking up the image of the joint object, a parameter calculating part 102 for calculating a first parameter relating to the position of the joint or inter-joint site of the joint object by applying a model having a preliminarily held joint to the joint object in the acquired image, a model conversion part 103 for executing model conversion to estimate a second parameter relating to the shape information of the inter-joint site of the joint object and the position and motion of the joint which is not included in the first parameter by using the first parameter and an image generation part 104 for generating a new image on which the characteristics of the joint object are reflected by using the second parameter. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image generation apparatus and method for generating an image of a joint object including a person or an animal by image processing.

  Many conventional joint object image generation techniques have been implemented in the field of computer graphics. Many of these use a motion capture system to acquire actual human shape and motion data, and generate animation based on the motion data. As a result, various postures and movements of the joint object can be output as an image. There is also a method that can generate animation without using a motion capture system. For example, Patent Document 1 discloses a method of generating an animation of a human body motion by estimating a motion that minimizes a person's muscle motion using dynamics.

For the purpose of motion analysis and the like, an image analysis method for applying a joint model to a joint object existing in an image is disclosed. For example, in Patent Document 2, a model configured for each part of an object is applied to an object present in an image, and as a result of the application, the object is held as a model parameter. In Non-Patent Document 1, a person model composed of 10 body parts is used to apply a person model composed of a combination of cylinders to a person present in the image, and the person present in the image A technique that can track a body part even when it moves is proposed.
Japanese Patent No. 3355113 JP-A-8-214289 Leonid Sigal, Sidharth Bhatia, Stefan Roth, Michael J. Black, Michael Isard, “Tracking Loose-Liberated People”, 2004 IEEE Computer Society Conferencing on Computer Vision and Pattern Recognition, Vol. 1, pp 421-428, 2004

  However, even with the image generation method and image analysis method typified by the above prior art, it is possible to generate an accurate image reflecting parameters such as the body part shape and clothes of the joint object present in the image. There is a problem.

  In the image generation method represented by Patent Document 1, since parameters related to the motion of a joint object are calculated in advance, an animation can be generated from the motion parameters of the joint object. However, parameters relating to clothing and shape are not obtained from joint objects present in the image. For this reason, it is impossible to generate an animation reflecting the shape, color, and movement of a joint object existing in an actual image.

  In addition, in the image analysis methods represented by Patent Document 2 and Non-Patent Document 1, only a human model simplified by a cylinder or a straight line is applied. Therefore, although it is possible to acquire rough parameters such as joint positions of joint objects existing in the image, it is difficult to acquire information on more detailed movement, clothes, and detailed shapes. For example, in Non-Patent Document 1, a method of identifying 10 site positions has been proposed, but in order to perform image generation such as animation generation, more detailed movements related to joint objects existing in the image, Information on clothes, detailed shape, etc. is required. However, with the current technical level, the more models the number of parts increases, the more difficult it is to apply the model, and it is difficult to accurately apply a detailed model sufficient for image generation to the image. Furthermore, as the number of parts of the model increases, the amount of calculation increases because the parameters to be calculated increase. In addition, when a joint or part is hidden by occlusion or the like, there is a problem that it is difficult to obtain position and movement information regarding the hidden joint or part from an image. Occlusion means that a part of a moving object is hidden behind the object and the number of pixels that can be photographed changes.

  Accordingly, the present invention is to solve such problems, and an object of the present invention is to provide an image generation device that generates a new image reflecting information (characteristics) of a joint object existing in an image. .

  That is, the present invention extracts information on the shape, clothing, movement, etc. of the joint object existing in the image from the image of the joint object existing in the image, and exists in the image using the extracted information. An object is to generate a new image reflecting the characteristics of a joint object.

  Specifically, the present invention extracts information on the shape, clothing, movement, etc. of the joint object existing in the image from the image of the joint object existing in the image, and uses the extracted information to temporally Provide a technique for generating spatially interpolated and extrapolated images. In addition, information on the body part shape and movement of the joint object existing in the image is extracted from the image of the joint object existing in the image, and the part constituting the joint object can be clearly seen using the extracted information. A technique for generating images is also provided. Furthermore, from the image of the joint object present in the image, information on the part shape, clothing, and movement of the joint object present in the image is extracted, and using the extracted information, the joint object present in the obtained image is extracted. Also provided is a technique for generating an image of a joint object including a posture and movement different from the posture and movement of the robot.

  In order to achieve the above object, an image generation apparatus according to the present invention is an image generation apparatus that generates a new image reflecting the characteristics of a joint object from an image showing the joint object. By applying an image input means for acquiring an image and a model having a joint held in advance to a joint object in the acquired image, a first parameter related to the position of the joint of the joint object or an inter-joint site is calculated. Using the parameter calculation means for performing the model conversion for estimating the shape information of the inter-articular part of the joint object and the second parameter relating to the position and movement of the joint not included in the first parameter, using the first parameter Model conversion means; and image generation means for generating a new image reflecting the characteristics of the joint object using the second parameter. The features.

  Here, when the image input unit acquires temporally continuous images, the parameter calculation unit uses the image to calculate the first parameter relating to the position and movement of the joint or inter-joint site of the joint object. May be calculated.

  The present invention can be realized not only as such an image generation apparatus but also as an image generation method, a program including the method as a step, a computer-readable recording medium storing the program, and the like.

  Estimating more detailed joint and part position and motion information from the rough joint and part position and motion information obtained by applying the joint model to the joint object existing in the image by the above method. Thus, it is possible to generate a new image reflecting information such as a part shape and movement of a joint object existing in the image.

  Specifically, it is possible to generate an image that is temporally and spatially interpolated and extrapolated from the image of the joint object existing in the image. In addition, it is possible to generate an image in which a part constituting the joint object can be clearly seen from the image of the joint object existing in the image. Furthermore, from the image of the joint object existing in the image, reflecting the shape and clothes of the joint object, the joint object including the posture and movement different from the posture and movement of the joint object existing in the image. An image can be generated. In addition, even when the position and motion information of some joints and parts cannot be obtained from the image due to occlusion etc., the position and motion information of the joints and parts that could not be obtained from the image are estimated. By doing so, it is possible to generate a new image reflecting information such as the shape and movement of the joint object existing in the image.

  Therefore, according to the present invention, a high-accuracy image is generated from the captured real image, and in particular, video complementation that improves the accuracy by complementing the video obtained by shooting with a digital camera, a mobile phone with a camera, a video device, etc. As a device or the like, its practical value is high.

  One embodiment of the present invention is an image generation device that generates a new image reflecting the characteristics of the joint object from an image showing the joint object, and an image input unit that acquires an image obtained by imaging the joint object; Parameter calculating means for calculating a first parameter related to a position of a joint or an inter-joint portion of the joint object by applying a model having a joint held in advance to the joint object in the acquired image; Model conversion means for performing model conversion for estimating a second parameter relating to shape information of an inter-joint part of the joint object, a position and motion of a joint not included in the first parameter, using the parameter, and the second parameter And an image generation means for generating a new image reflecting the characteristics of the joint object. Here, the image generation unit reflects, for example, the position of the joint of the joint object or the inter-joint part, and the color and texture information of the joint part of the joint object as a new image reflecting the characteristics of the joint object. Generated image. As a result, the second parameter having a larger amount of information than the first parameter is estimated, or a parameter relating to information that cannot be obtained only by fitting the joint model to the image due to a cause such as occlusion or the like. By estimating as a parameter, it is possible to generate a new image reflecting information (characteristics) of a joint object existing in the image.

  In a more preferred embodiment of the present invention, the image input unit acquires temporally continuous images, and the parameter calculation unit uses the image to determine the position of a joint or an inter-joint site of the joint object and A first parameter relating to movement is calculated. Thereby, the image generating means generates temporally interpolated and extrapolated images generated based on motion information included in the first parameter or the second parameter with respect to temporally continuous images. Can be generated as the new image.

  According to a more preferred aspect of the present invention, the image generation device further calculates an error between the image generated by the image generation unit and a target image, thereby evaluating the image, and the image Parameter changing means for changing the first parameter based on an evaluation result by the evaluation means, and the model converting means performs the model conversion based on the first parameter changed by the parameter changing means. Features. Thereby, since the parameter can be obtained with higher accuracy by changing the parameter so as to approach the target image, a new faithfully reflecting the characteristics of the joint object existing in the image can be obtained. Image generation is possible. In addition, it is possible to generate a moving image with a higher frame rate from a moving image with a lower frame rate.

  Note that the image generation means may generate an image in which a different color or texture is pasted on each part constituting the joint object as the new image. This makes it possible to grasp the state and movement of each part of the joint object with an image reflecting information (characteristics) of the joint object existing in the image.

  Further, the image generation means generates an image obtained by pasting a texture of an inter-articular part of the joint object on the image of the joint object including a posture or movement different from the posture or movement of the joint object as the new image. May be. As a result, it is possible to generate an image processed into another posture, motion, and shape while reflecting information (characteristics) of the joint object existing in the image.

  According to a more preferred aspect of the present invention, the model conversion unit estimates the parameter value that cannot be extracted when the parameter calculation unit cannot extract a part of the first parameter. It is characterized by performing model conversion for estimating. Thus, information (characteristics) of the joint object existing in the image is reflected by estimating, as the second parameter, a parameter related to information that cannot be obtained only by fitting the joint model to the image due to causes such as occlusion. Thus, a new image can be generated.

  Here, it is preferable that the motion is represented by any one of a motion vector, an acceleration vector, an affine parameter, and an approximate curve parameter. Thus, by obtaining parameters relating to the motion of an object having a joint, it is possible to generate a new image that is temporally interpolated and extrapolated using motion information.

  In addition, the model conversion unit holds correlation information between the first parameter and the second parameter obtained in advance, and performs model conversion for estimating the second parameter by referring to the correlation information. Is preferred. This facilitates the estimation of the second parameter by learning in advance information on the position and movement of the joint of the joint object.

  Examples of the correlation information may include an autocorrelation of the first parameter and a cross-correlation between the first parameter and the second parameter. Alternatively, the correlation information may be calculated from a plurality of sets of cross-correlation information by a weighted linear sum. Furthermore, the correlation information may be obtained in advance for each type of joint object or each joint object. This makes the estimation of the second parameter more accurate.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
First, the first embodiment of the present invention will be described. FIG. 1 is a diagram showing an outline of a processing procedure according to the first embodiment. This image generation apparatus is an apparatus that enables generation of a new image that reflects information (characteristics) related to the position, shape, and movement of a joint object existing in an image. Unit 102, model conversion unit 103, and image generation unit 104.

  The image input unit 101 is an input interface or the like that acquires an image obtained by imaging a joint object with a digital camera, a video apparatus, or the like (that is, an actual image that is not a computer graphic or the like). Here, images arranged in time series may be used.

  The parameter calculation unit 102 is a processing unit that detects the position of a joint of a joint object existing in the image by applying a model having a joint prepared in advance to the image. Here, when the images are arranged in time series and the joint object in the image is moving, the motion of the joint of the joint object and the inter-joint region can also be detected. A configuration example of the parameter calculation unit 102 will be described with reference to FIG. The parameter calculation unit 102 includes a joint object region extraction unit 1021 that extracts a joint object region from the input image, a model fitting unit 1022 that applies a joint model prepared in advance to the extracted joint object region, and a joint model. An inter-joint site position calculation unit 1023 that calculates the position of the inter-joint site from the joint position obtained by fitting. When the joint model is configured in three dimensions, a process of projecting the three-dimensional information into a two-dimensional image space is performed.

  The model conversion unit 103 is a processing unit that performs model conversion to estimate parameters related to the position and movement of joints that are not included in the shape information of joint parts of joint objects from the parameters obtained by the parameter calculation unit 102 The joint position and the movement of the joint object extracted by the parameter calculation unit 102 are input, and the information on the joint position and movement of the joint object including the shape information with higher accuracy is output or obtained from the image by occlusion or the like. Outputs information about joint positions and movements that could not be performed.

  The image generation unit 104 is a processing unit that generates a new image using the joint position estimated by the model conversion unit 103 and information related to the movement information and shape of the inter-joint site. That is, the image generation unit 104 reflects, as a new image reflecting the characteristics of the joint object, for example, the position of the joint or the joint part of the joint object, and the color and texture information of the joint part of the joint object. Generate an image.

  A configuration example of the image generation unit 104 when generating a new image that is temporally interpolated and extrapolated will be described with reference to FIG. The image generation unit 104 includes a pixel movement position calculation unit 1041 that determines a pixel position corresponding to a frame (time) to be generated based on motion information, and an interpolation process that interpolates missing pixels that occur as a result of moving the pixel. And a pixel value determining unit 1043 that determines color information of the moved pixel.

  Note that when no temporally interpolated / extrapolated image is generated, the motion information is unnecessary, and thus the pixel movement position calculation unit 1041 and the interpolation processing unit 1042 are not necessary.

  Next, the image generation method of the joint object by the image generation apparatus of the present embodiment configured as described above will be described in detail with reference to the flowchart of FIG.

First, in S2001, the image input unit 101 receives an input of a captured image.
Next, in S2002, the joint object region extraction unit 1021 of the parameter calculation unit 102 performs background difference processing on the input image to cut out the joint object region. Here, inter-frame difference processing may be performed instead of background difference processing. Further, when the target joint object is a person, the M.M. OrenC. PageorgiouP. SinhaE. Osuna and T. Poggio "Pedestrian Detection using wavelet templates" Proc. of CVPR97pp. The person area may be cut out using 193-1991997 or the like. Furthermore, edge detection processing may be used in combination. In addition, when performing the background difference process, an image as a background without a person is prepared in advance. When a moving image is input, a background sprite can be generated and the generated background sprite image can be used.

  Next, in S2003, the model fitting unit 1022 of the parameter calculation unit 102 uses a joint model 1001 as shown in FIG. 5A, as shown in the model fitting result 1002 in FIG. A model having a joint prepared in advance is applied to the joint object region. Here, Leonid Sigal Sidharth Bhatia, Stefan Roth, Michael J. Black, Michael Isard, “Tracking Loose-Liberated People”, 2004 IEEE Computer Society Conferencing on Computer Vision and Pattern Recognition Vol. Model fitting techniques such as 1, pp 421-428, 2004 can be used. As a result, each part indicated by a black circle in the model fitting result 1002 becomes the joint position detected by the model fitting. That is, three-dimensional joint positions {Xwi (t), Ywi (t), Zwi (t)} and joint angle information can be obtained. Note that the connection angle of the cylinder shown by the joint model 1001 can be used as the joint angle information. When a time series image is input, in addition to the above, model fitting is performed for each input image, and the three-dimensional position information of the black circles indicated by the model fitting result 1002 is obtained in time series, so that the motion information {ΔXwi (T), ΔYwi (t), ΔZwi (t)} can be obtained.

  Further, the inter-joint site position calculation unit 1023 of the parameter calculation unit 102 uses the detected three-dimensional joint position 1002 to determine an intermediate point between adjacent joints as shown in FIG. 5C. Is calculated as a representative position 1003. The representative position 1003 is not necessarily the center position, and the inter-joint site position may be calculated for each inter-joint site.

  Note that FIG. 5 does not limit the number of joints and the number of sites between joints. Further, the center position regarding the part existing at the tip of the joint such as the head, hand, and foot is calculated from the joint position of the neck, wrist, and ankle and the angle information using a specified value. Here, the center position of the head was 15 cm from the neck position, the center position of the hand was 5 cm from the wrist position, and the foot was 9 cm from the ankle position. Of course, each value may be prepared as a database for each body type and sex as in the example shown in FIG.

  In S2004, the model conversion unit 103 estimates output information from input information by using model conversion data describing the relationship between input and output. Specifically, using the joint position and motion of the joint object extracted by the parameter calculation unit 102 as input, the model conversion unit 103 estimates information on the joint position and motion that could not be obtained by occlusion or the like. Output. Furthermore, using the joint position and movement of the joint object extracted by the parameter calculation unit 102 as input, information such as the joint position of the joint object with higher accuracy including shape information is output. Here, an example of generating model conversion data based on correlation information will be described.

Ｘは、入力ベクトルの集合である。Ｎは、データセットの数である。 First, let x be an input vector. x corresponds to the joint position and movement of the joint object extracted by the parameter calculation unit 102.

X is a set of input vectors. N is the number of data sets.

であり、ｍは関節位置や関節間の部位に相当する。 Also,

And m corresponds to the joint position and the part between the joints.

Ｙは、出力ベクトルの集合である。 Next, let y be an output vector. y is a parameter to be estimated, and may be information on the position and movement of a joint that has not been obtained by occlusion or the like, or may be information on the joint position of a joint object having a larger amount of information including shape information.

Y is a set of output vectors.

であり、ｌは、関節位置、関節間部位に加えて、形状を表現するためのマーカ位置等の情報を含む。

In addition to the joint position and the inter-joint site, l includes information such as a marker position for expressing the shape.

Next, the autocorrelation matrix of X is determined as follows.

Further, the cross-correlation matrix between X and Y is determined as follows.

で表すことができる。ここで、Ｃ_x ^*はＣ_xの逆行列、または疑似逆行列である。 Here, if the model transformation matrix is C,

It can be expressed as Here, C _x ^* is an inverse matrix of C _x or a pseudo inverse matrix.

Then, y to be estimated can be expressed by the following equation using a model conversion matrix.

Here, m _x and m _y are the mean vector of x and y.

From equation (8), if the average vector mx of _x , the average vector my of _y , and the model conversion matrix C are held, the model conversion unit 103 generates the output vector y from the newly given input vector x. Can be estimated.

Here, concrete m _x, m _y, method for determining the C will be described.
As X and Y, motion capture data can be used.

  In motion capture, a marker is attached to the joint position of an actual joint object, and the three-dimensional position of the marker can be obtained in time series.

として、位置情報と動き情報を表現できる。 When the input vector x at time t is described with respect to the marker i as follows:

As described above, position information and motion information can be expressed.

  As for the motion information, in addition to the motion vector, the motion vector may be approximated by a function, an affine parameter, or acceleration may be used.

のように、位置情報と動き情報を表現できる。 Similarly, y also describes the marker k.

In this way, position information and motion information can be expressed.

  Here, as for the motion information, in addition to the motion vector, the motion vector may be approximated by a function, an affine parameter, or acceleration may be used.

x and y can be expressed by arranging these vectors in the order of markers.
Further, m _x, m _y are the average of the respective vectors obtained by arranging the marker order.

The model transformation matrix C can be calculated by (Equation 5), (Equation 6), and (Equation 7) with x and y at the same time as a pair. Incidentally, m _x, m _y, for C, may be previously calculated may be calculated each time from the data set.

  Here, a specific example of x and y will be described with reference to FIG. Since it is important that x is relatively easy to detect from the image, it is desirable to use marker data attached to a rough joint position as shown in the marker position example 1101 in FIG. For y, as shown in the marker position example 1102 in FIG. 7B, in addition to the joint position x indicated by a black circle, marker data attached to the inter-joint site indicated by a white circle is also included. be able to. Here, if the positional relationship between the marker indicating the joint position indicated by the black circle and the marker attached to the inter-joint site indicated by the white circle is used, a rough shape can be obtained. Further, as shown in FIG. 7C, polygon data can be used for y as shown in shape data 1103 created by CG or the like. Thereby, the model conversion unit 103 can estimate detailed data including information related to the shape from the data of the joint position that is easy to detect from the image.

  In addition, as an example in which occlusion is likely to occur, a person, an animal, or the like may move horizontally on an image. In such an example, information on either the left half or the right half may not be obtained. In addition, some of the target joint objects may be hidden by other objects, or some of the information on the joint objects may not be obtained.

  In order to prepare for such a situation, as shown in the example of FIG. 8A, x is generated using marker data relating to one half of the marker position example 3201, and FIG. As shown in the example, for y, it is possible to estimate whole body information from half-body information by generating marker data relating to the whole body including markers indicated by white circles in the marker position example 3202.

Note that x, m _x , and C may be generated using a result obtained by fitting a model to an image in advance using a test image or the like. Further, CG joint position data may be used. For y, data including shape data such as polygons from CG may be used. Further, examples of model transformation, for each body type and operation of the joint body, the m _x, m _y, to C may be a plurality of sets prepared, the m _x for each example occlusion is likely to occur, m _y, C A plurality of sets may be prepared.

  In addition, by obtaining correlation information between different postures as in the marker position examples 3301 and 3302 illustrated in FIGS. 9A and 9B, the model conversion unit 103 can obtain the input posture information and It is also possible to output different posture information. Thereby, it is possible to generate an image processed into another posture while reflecting information (characteristics) of the joint object existing in the image. Furthermore, if the correlation information including the motion information is obtained, the model conversion unit 103 can output an operation different from the input operation. As a result, it is possible to generate an image processed into another motion while reflecting information (characteristics) of the joint object existing in the image.

  Further, in the correlation information between different poses as in the marker position examples 3301 and 3302 shown in FIGS. 9A and 9B, in addition to the position information as the elements of the vectors of x and y, (number 9) If the correlation information is obtained from x and y including the motion information including the motion information as in (Equation 10), the model conversion unit 103 outputs an operation different from the input operation. Is also possible. As a result, it is possible to generate an image processed into another motion while reflecting information (characteristics) of the joint object existing in the image. Furthermore, not only the correlation information but also the model conversion unit 103 may learn the relationship between X and Y using a neural network, as will be described later. Further, the relationship between X and Y may be learned using not only the correlation information but also a neural network.

  Here, when a part of the vector y estimated by the model conversion in S2004 is the marker position 1102 in FIG. 7B, three-dimensional shape estimation using a Bezier curved surface or the like is performed using the marker position as a reference point. Of course, it may be approximated using other parametric curved surfaces, or the shape may be expressed by data using polygons as indicated by 1103 in FIG. 7C. The method of curve approximation using a reference point is described in detail in Tsuneya Kurihara, Kenichi Yasushi “3DCG Animation”, Technical Review, P36, 2003, and the like.

  Next, in S2005, the model conversion unit 103 projects the three-dimensional shape estimated in S2004 on an image as post-processing. As a result, the joint object image is divided into meshes. Here, an example in which the camera parameters are known will be described. However, any method capable of projecting a three-dimensional real world coordinate value onto an image may be used, and a method of estimating camera parameters from an image may be used. As a method for projecting a three-dimensional real world coordinate value onto an image, details are described in Xu, Tatsumi, “Three-Dimensional Vision”, page 9, Kyoritsu Shuppan, published in 1998. If the camera parameters can be defined, the three-dimensional shape estimated in S2004 can be projected on the image as shown in FIG. Note that step S2005 may be performed by the image generation unit 104 as preprocessing for image generation.

  Here, a position where a certain point in the three-dimensional shape is projected on the image is defined as (xj, yj). And it is set as the color information (Rj, Gj, Bj) in the pixel of each j position (xj, yj). j is a pixel. Thereby, position and color information on the image corresponding to the estimated three-dimensional position can be obtained. As a result, it is possible to obtain which inter-joint site each mesh region belongs to based on the inter-joint site detected in the parameter calculation S2002. As an effect of this, as shown in FIG. 11, it is possible to generate an image in which each part of the joint object can be clearly seen by color-coding for each region or changing the texture.

  In step S2006, the image generation unit 104 generates an image using parameters related to the joint position, motion, and shape of the joint object estimated in step S2005. Here, the pixel movement position calculation unit 1041 of the image generation unit 104 uses the estimated motion information of the vector y to calculate the motion for each region divided into meshes. As the motion, an affine motion may be used, an average motion vector for each region may be used, or the motion may be approximated by a function. And a pixel is moved using the motion information obtained for every area | region divided into mesh shape, and a new image is produced | generated.

  Here, as shown in FIG. 12, two time-series images I (t) 1201 and I (t + n) 1202 are input, and temporally interpolated between I (t) 1201 and I (t + n) 1202. The case where N images 1203 to be generated are generated will be described, but the number of input images is not specified.

  Assuming that the motion parameter of the region A is Ma and the motion parameter of the region B is Mb for the region partitioned by the mesh, the image 1303 to be interpolated is based on the image of I (t) 1301 and the interpolation processing unit 1042 and The pixel value determination unit 1043 can generate each pixel belonging to the regions A and B by moving by Ma / (N + 1) and Mb / (N + 1).

  For extrapolation, the interpolation processing unit 1042 is based on the motion information generated between I (t) 1301 shown in FIG. 13A and I (t + n) 1302 shown in FIG. The pixel value determination unit 1043 can generate the pixels belonging to the areas A and B by moving Ma / (N + 1) and Mb / (N + 1) from the image of I (t + n) 1302.

Note that the average motion vector u _A (ave) can be used as the motion parameter Ma, and the average motion vector u _B (ave) can be used as the motion parameter Mb of the region B. However, the motion parameter may be not only a motion vector but also an affine parameter, an acceleration vector, or an approximate curve parameter.

  At this time, since the region A and the region B move in different directions, the region A and the region B are separated around the joint position and the region boundary as in the image examples 1304 and 1305 in FIGS. There is a risk of overlapping or overlapping. About this, it is effective to process as follows.

First, in the adjacent regions A and B, the pixel movement position calculation unit 1041 sets the motion parameter of the region A as the motion parameter of the Ma region B as Mb. Next, for each pixel belonging to each region, the shortest distance dist _{j_min} to the pixel at the region boundary is calculated. Here, j is a pixel. Note that dist _{j_min} may be a distance to the center of gravity of the region boundary.

A pixel belonging to region A will be described as an example.
As described below, the pixel movement position calculation unit 1041 determines the motion parameter Ma_j of each pixel. Here, j is a pixel.

で表せる。 Similarly, for pixels belonging to region B,

It can be expressed as

のようにすることも可能である。 A nonlinear function may be used,

It is also possible to do as follows.

  By using the method as described above, the image generation unit 104 can be used on the condition that the parts connected by the joint are not separated and the areas do not overlap as in the interpolated image 1303 of FIG. A new image can be generated. The image generation unit 104 can generate a new image by performing pixel movement based on the joint position so that the parts connected by the joint are not separated.

  In this case, it is sufficient if the condition is such that the motion information of the pixels near the region boundary changes gently.

Of course, the same applies to the extrapolated image 1204.
Further, by moving the pixel, a part of the pixel of the newly generated image may be missing. In this case, interpolation is performed from the neighboring pixel, or the time is reversed from the image at time t + n. It is also effective to generate an image at the time t in the direction and generate an image using an image generated from the forward direction and an image generated from the reverse direction. As an interpolation method, bilinear, bicubic method, morphological processing, or the like can be used.

  Through the above processing, the image generation apparatus according to the present embodiment uses the parameters relating to the movement and joint position obtained by fitting the joint model having joints, and the parameters relating to the shape and clothes obtained from the image. It is possible to generate a new image reflecting information (characteristics) of the joint object existing inside.

  Further, by reproducing the interpolated image 1203, the extrapolated image 1204, and the input image in time order, a moving image with a higher frame rate can be generated from a moving image with a lower frame rate.

(Embodiment 2)
Next, a second embodiment of the present invention will be described. FIG. 14 is a diagram showing an outline of a processing procedure according to the second embodiment. In addition to the first embodiment, this image generation apparatus changes parameters related to the joint position, shape, clothing, and movement of a joint object while evaluating the generated image. It is a device that enables generation of a new image with higher accuracy reflecting information on position, shape, clothes, and movement, and includes an image input unit 101, a parameter calculation unit 102, a model conversion unit 103, an image generation unit 104, An image evaluation unit 201 and a parameter change unit 202 are included. Note that the image input unit 101, the parameter calculation unit 102, and the model conversion unit 103 are the same as those in the first embodiment, and a description thereof will be omitted.

  The image generation unit 104 uses the parameters detected from the input images I (t) 1201 and I (t + n) 1202 based on the input image I (t) 1201 and an image at a time corresponding to I (t + n). Is generated. This is a generated image I ′ (t + n).

  The image evaluation unit 201 calculates an error between the input image I (t + n) 3501 and the generated image I ′ (t + n) 3502 as shown in FIGS. Note that the generated image I ′ (t + n) 3502 is displayed by hatching for each polygon-displayed surface, but actually, the pixel value on the image projected in S2005 may be used.

  Next, the parameter changing unit 202 changes the joint position and the motion parameter, which are parameters detected by the parameter calculating unit 102. Then, the model conversion unit 103, the image generation unit 104, and the image evaluation unit 201 perform each process again according to the changed parameters. Here, the parameter change unit 202 may change the parameter estimated by the model conversion unit 103.

  By repeating these processes, a parameter that reduces the error is determined, and the image generation unit 104 generates a new image using the parameter at this time. Here, it is not always necessary to comprehensively change all the parameters, and the processing can be repeated until the error becomes equal to or less than a threshold value, can be repeated a specified number of times, or can be determined according to the processing time.

  When determining by the processing time, it is necessary to consider the frame rate and the number of newly generated images. For example, when an image sequence of 30 frames / second is generated in real time by newly generating two images between frames for an image input at 10 frames / second, at least 0. It is necessary to generate two images in one second. In this case, it is necessary to generate in 0.05 second per sheet, and such information can be used as a threshold for processing time.

Next, a method for generating an image of a joint object by the image generation apparatus according to the present embodiment configured as described above will be described in detail with reference to the flowchart of FIG. 16. S2101 to S2105 are the same as in the first embodiment. Therefore, the description is omitted.

  In step S <b> 2106, the image generation unit 104 uses the parameters detected from the input images I (t) 1401 and I (t + n) 1402 based on the input image I (t) 1401, and the time corresponding to I (t + n). Generate an image of This is I ′ (t + n). Here, I (t + n) is called a target image.

  A method for generating I ′ (t + n) will be described. Here, a predicted image I ′ (t + n) at time t + n is generated using the parameters detected in S2103 and S2104 based on the input image I (t) by the same method as in the first embodiment.

  In step S <b> 2106, the image evaluation unit 201 calculates an error between the target image I (t + n) 3401 and the predicted image I ′ (t + n) 3402 as an evaluation value. As a method for calculating the evaluation value, the difference between the pixel value of the target image I (t + n) 3501 and the pixel value of the generated image I ′ (t + n) 3502 is calculated. If the overlap between the target image I (t + n) and the generated image I ′ (t + n) 3502 is large and the pixel values are close, it is desirable to determine that the target image is close. Therefore, the following evaluation values can be used.

等を用いることができる。もちろん、上式に限らず、目標画像と予測画像との誤差を評価する計算方法であれば良い。 As a method of calculating the evaluation value,

Etc. can be used. Of course, the calculation method is not limited to the above formula, and any calculation method for evaluating the error between the target image and the predicted image may be used.

  In step S <b> 2107, the image evaluation unit 201 calculates whether the Err value satisfies a preset evaluation value. If the Err value does not satisfy the preset evaluation value, the value closest to the evaluation value and the parameters detected in S2104 and S2105 at that time are stored as a set. On the other hand, if the Err value satisfies the preset evaluation value, the process of S2109 is performed. Of course, all the parameters may be processed comprehensively, and the process of S2109 may be performed after determining the parameter having the best evaluation value. However, when processing time is considered, such as real-time processing, it may be repeated a specified number of times or until a specified processing time is reached. In this case, the parameter closest to the evaluation value is selected in the repeated processing.

  In step S2108, the parameter changing unit 202 changes the result of model fitting detected in step S2103. Then, the process after S2104 is repeated again with the changed parameter as an input.

  At this time, it is also effective to use a hierarchical expression (connection relationship between joint portions) as shown in FIG. Thereby, a hierarchical connection relationship can be obtained for the result of the model fitting. As this effect, for example, the parameters of the torso are determined first, and then using the relationship connected to the torso, such as the left and right upper arms, the left and right thighs, and the head, the joint positions belonging to the upper hierarchy are sequentially By changing the parameters, the parameters can be changed and determined efficiently. That is, it is possible to determine a parameter that effectively reduces the error.

In step S <b> 2109, the image generation unit 104 can perform image generation using the hierarchical expression (connection relationship between joint portions). Assuming that the motion parameter of the joint part in the upper hierarchy is Mb and the motion parameter of the joint part in the lower hierarchy connected to the upper hierarchy by the joint is Ma, (Equation 11) to (Equation 14) are shown. The motion parameter of each pixel is rewritten as the following equation.

  As a result, the movement of the inter-joint site in the upper hierarchy becomes dominant, and thus an image reflecting the structure of the joint object can be generated.

  Furthermore, it is also possible to change the parameters so as to change the inter-joint distance of the joint model prepared in advance.

  In that case, in S2109, the image generation unit 104 generates an image using the parameters finally determined in S2107. The image generation method at this time is the same as that in S2005 in the first embodiment, and it is possible to generate temporally interpolated and extrapolated images, and to clearly see each part constituting the joint object. In addition, it is possible to generate an image in which a different color, texture, or the like is pasted on each part. As shown in the image 3502 in FIG. 15B, the parameters are changed based on the evaluation as shown in (Equation 22) even when there is a partial deviation from the actual target image when generating the image before the image evaluation. By doing so, as shown in the image 3503 after the parameter change, it is possible to generate an image in which the deviation from the target image is minimized. Thereby, a new image with higher accuracy can be generated.

  Through the above processing, in addition to the effects of the first embodiment, by changing the parameters related to the joint position, shape, clothing, and movement of the joint object while evaluating the generated image, the joint of the joint object existing in the image It is possible to generate a new image with higher accuracy reflecting information on the position, shape, clothes, and movement of the image.

(Embodiment 3)
Next, a third embodiment of the present invention will be described. The third embodiment is another operation example in the first and second embodiments. That is, the third embodiment estimates the undetectable parameters by performing model conversion when some of the parameters of the joint model cannot be extracted from the image in the first and second embodiments. This corresponds to an operation example in which image generation is performed using. Here, it demonstrates along FIG. Of course, as in FIG. 14, an image evaluation step for evaluating the generated image based on an error from the target image and a parameter changing step for changing the parameter value based on the evaluation result may be added.

  Since the image input unit 101, the parameter calculation unit 102, and the image generation unit 104 are the same as those in the first embodiment, description thereof will be omitted.

  The parameter calculation unit 102 detects a joint position of a joint object existing in the image by applying a joint model 1001 having a joint prepared in advance to an image as shown in a model fitting result 1002 in FIG. 5B. However, when crossing the image from side to side or when a part of the joint object is temporarily hidden by the shielding object, it may not be possible to obtain a part of information on the joint position of the joint object and its movement. is there.

  The case where a person crosses right and left while walking in the image will be described with reference to FIG. In this case, the part indicated by a black circle in the marker position example 3201 shown in FIG. 8A can be detected from the image, while the marker position example 3202 shown in FIG. A part indicated by a white circle may not be detected from the image because a part of the right half shields a part of the left half during the operation. In this case, it is difficult for the parameter calculation unit 102 to accurately detect all the joint position parameters of the joint model, and even if the shielded joint position is detected, the reliability is low.

Here, to explain using (Equation 2), when some joint positions cannot be obtained by shielding, some of the elements of the vector cannot be obtained as information. In this case, the model conversion unit 103 performs the same calculation as in the first embodiment by putting 0 in the element of X _input representing the joint position that could not be detected from the image as follows.

  Moreover, the parameter which cannot be detected can be estimated by using the following method.

  Taking a person or animal as an example, it may be possible to obtain only information about one half of the joint object existing in the image.

  In such a case, the model conversion unit 103 uses the data of only one half of the set of input vectors of (Equation 1), such as the data indicated by the black circles in the marker position example 3201 in FIG. The set of output vectors of (Equation 3) is used as whole body data such as the black circle and white circle in the marker position example 3202 in FIG. 8B, or the marker position example in FIG. A model transformation matrix of (Expression 5) to (Expression 8) is generated using half body data opposite to the input vector, such as a white circle of 3202. As a result, it is possible to obtain information on the joint position of the whole body or its movement from the input data of one half body. The estimation error is shown in FIG. FIG. 18 shows the left shoulder, left elbow, left wrist, and left knee, which are the joint positions in the left body opposite to the input, based on the input information of the joint positions of the right half body when various operations are performed in time series. The average error when the position of the left ankle is estimated is shown. Since information that cannot be obtained from the image can be obtained with an error of about 5 cm, the accuracy is sufficient for the purpose of image generation. Furthermore, it is also possible to estimate by including shape information in the output vector y.

  As described above, model conversion for estimating whole body data from data of only one half body is particularly effective when, for example, the moving direction of the joint object moves in the horizontal direction on the image. Therefore, the model conversion unit 103 may calculate the moving direction of the object using the optical flow or the like in the image input unit 101, and may select this model conversion method if the moving direction is the horizontal direction. .

In addition, assuming various situations such as the case where joint positions such as elbows and knees cannot be extracted, and the case where an ankle position or the like cannot be detected by other shield 3602 as shown in FIG. By preparing a model conversion matrix according to the situation, the model conversion unit 103 can estimate the joint position, its movement and shape with high accuracy. When S situations are assumed, S model transformation matrices are prepared as in the following equation.

  Through the above processing, in addition to the effects of the first and second embodiments, the model conversion unit 103 estimates parameters that are difficult to detect from the image by occlusion or the like by model conversion. It is possible to generate a new image used.

  Further, if the correlation information including the motion information is obtained in (Equation 21) using the motion information as in (Equation 9) and (Equation 10), the model conversion unit 103 is different from the input operation. It is also possible to output an action. In this case, the correlation information may be obtained so that the input vector x and the output vector y correspond to different actions and postures, respectively. As a result, it is possible to generate an image processed into another posture or motion while reflecting information (characteristics) of the joint object existing in the image.

  As a result, the image generating apparatus according to the present embodiment can generate an image processed into another motion while reflecting information (characteristics) of the joint object existing in the image.

(Embodiment 4)
Next, a fourth embodiment of the present invention will be described. FIG. 20 is a diagram showing an outline of a processing procedure according to the fourth embodiment. In addition to the first to third embodiments, this image generation apparatus is an apparatus that enables generation of an image processed into another shape while reflecting information (characteristics) of a joint object existing in the image. , An image input unit 101, a parameter calculation unit 102, a model conversion unit 103, an image generation unit 104, and a user setting unit 301. Here, although it demonstrates along Embodiment 1, it can utilize in all the embodiments. Note that the image input unit 101, the parameter calculation unit 102, and the model conversion unit 103 are the same as those in the first embodiment, and thus description thereof is omitted.

  Here, when the model conversion unit 103 inputs the joint position and the movement of the joint object extracted by the parameter calculation unit 102 and outputs information such as the joint position of the joint object including the shape information with higher accuracy. Will be described.

  In the user setting unit 301, the user changes some of the parameters including the shape information obtained by the model conversion unit 103. Here, it is also possible to display the screen so as to change parameters such as fattening or thinning.

  The image generation unit 104 generates an image using the parameters including the shape information set by the user setting unit 301 and the parameters obtained by the model conversion unit 103, so that the shape, movement, color, etc. of the input joint object An image with a changed shape is generated after reflecting the information regarding.

  Next, the image generation method of the joint object by the image generation apparatus of the present embodiment configured as described above will be described in detail with reference to the flowchart of FIG.

Since S2201 to S2205 are the same as those in the first embodiment, the description thereof is omitted.
The parameter y estimated by the model conversion unit 103 in S2204 can also obtain the position information of the marker attached to the joint part in addition to the information on the joint position and its movement. In other words, this represents the shape of the joint part.

  In step S <b> 2207, the user setting unit 301 changes parameters related to the shape, such as marker position information attached to the interarticular site, in accordance with an instruction from the user. Taking the case of a person or animal as an example, the size of the belly can be changed by changing the position information of the marker attached on the belly.

  Taking the case of a person or animal as an example, the size of the belly can be changed as shown in FIG. 22 by changing the position information of the marker attached on the belly. FIG. 23 shows an example of a parameter setting screen to be changed by the user. As for the parameters that can be changed by the user, for example, as shown in A to E of FIG. Specifically, as shown in an example 1102 of y marker positions in FIG. 7B, the user setting unit 301 is obtained by setting the marker positions related to the shapes indicated by white circles as controllable inter-joint site positions. The shape of the input image can be changed for each inter-joint site. Then, the user operates the shape parameter control bar 3702 to change the shape for each joint portion. Thus, by displaying on the display device, the user can easily change the shape.

  In step S2206, the image generation unit 104 generates an image using the parameter y estimated in step S2204 and the parameter related to the shape changed in step S2207. As shown in FIG. 22A, the input image 1701 is used as an input, and the parameters relating to the shape are reflected in S2207 to reflect the information of the input image, and as shown in FIG. An output image 1702 in which a part of the shape is processed or changed can be obtained.

  Here, S2207 will be described in detail with reference to FIG. Assume that the shape parameter 1805 in FIG. 24A is obtained in S2204. At this time, the image generation unit 104 generates a Bezier curve having joint positions at both ends as end points and shape parameter 1805 points as control points. In step S2207, the image generation unit 104 changes the shape parameter 1805 as indicated by the arrow in FIG. 24A in accordance with the instruction from the user acquired by the user setting unit 301, that is, the shape parameter control in FIG. When the bar 3702 is operated, a Bezier curve 1806 passing through the joint positions at both ends and the point of the changed shape parameter 1805 is generated, and a new image is generated. At this time, the color and texture information obtained in step S2205 is used as the color information and texture information.

  It should be noted that all the contour points of the region can be used as control points of the Bezier curve, and a parametric method such as spline interpolation can be used instead of the Bezier curve. Also, the shape parameter control bar 3702 in FIG. 23 changes the position of the white dot shown in the marker position example 1102 in FIG. 7B, which corresponds to the shape parameter 1805 in FIG. .

  With the above processing, the image generation apparatus according to the present embodiment can generate an image processed into another shape while reflecting information (characteristics) of a joint object existing in the image.

(Embodiment 5)
Next, a fifth embodiment of the present invention will be described. The fifth embodiment is another operation example in the first to fourth embodiments. That is, the fifth embodiment corresponds to an operation example in which motion information is used in the parameter calculation step and the model conversion step in addition to the processing described in the first to fourth embodiments. Here, Embodiment 1 will be described as an example, but the present invention can be applied to all the embodiments. The input image is preferably a plurality of images arranged in time series. In addition, it is good also as a spatio-temporal image which arranged the image arranged in time series in three dimensions.

  The motion information that can be used by the parameter calculation unit 102 and the model conversion unit 103 in FIG. 1 will be described. The parameter calculation unit 102 can obtain three-dimensional joint position and angle information using a model fitting method. Here, when a time-series image is input, motion information at each time can be obtained in addition to the above. Furthermore, in the model conversion unit 103, in addition to the motion information of the joint position, the motion information of the inter-joint site can be obtained. An example using an acceleration vector in addition to the motion vector will be described below.

For example, when T images arranged in time series are input, the parameter calculation unit 102 calculates the motion vectors {ΔXwi (t), ΔYwi (t), ΔZwi (t) of each joint position or inter-joint site position i. )} Can be obtained. At this time, when three or more time-series images are input, the acceleration vector s can be obtained as in the following equation. Note that v is a motion vector.

  An image generation method using the calculated acceleration vector will be described. In the image generation in S2006 in the first embodiment, an example of generating an image that is temporally interpolated and extrapolated when the joint object in the image is moving at a constant speed has been described. In addition to this, by using the acceleration vector and setting the motion parameter in the first embodiment to (s / (N + 1) + u) / (N + 1), an image that is temporally interpolated and extrapolated taking acceleration into account. Can be generated. Specifically, when the motion of the joint object suddenly increases or stops suddenly, it is possible to generate an interpolated or extrapolated image reflecting the acceleration.

  Next, a case where an Nth order function is fitted instead of a motion vector will be described. When T images arranged in time series are input, the parameter calculation unit 102 can perform fitting with respect to T pieces of joint position information and position information on the image using an Nth order function. As a result, it is possible to generate temporally interpolated and extrapolated images so as to follow the value of the fitted function. Specifically, by fitting with a function, it is possible to express a smoother motion, and thus it is possible to generate a smoother moving image using the interpolated and extrapolated images.

  Next, a case where affine parameters are used instead of motion vectors will be described. When T images arranged in time series are input, the parameter calculation unit 102 and the model conversion unit 103 can estimate affine parameters using T joint position information and position information on the image. Is possible.

Here, an example will be described in which the parameter calculation unit 102 estimates affine parameters using position information on an image. Assuming that the position at time t is (x, y) and that the pixel has moved at time t + 1 is (x'y ',), the affine transformation can be expressed as follows.

  Here, if affine parameters a to f are used in place of Δx, Δy, and Δz in (Equation 9) and (Equation 10), model conversion using affine parameters instead of motion vectors can be performed. As a result, temporally interpolated and extrapolated images using affine parameters instead of motion vectors can be generated as motion parameters. An affine parameter can express a motion including a rotational motion, and is suitable for a rotational motion of an arm or a leg.

  In particular, it is effective for movement including rotational movement. Furthermore, as described with reference to FIG. 17 of the second embodiment, it is also effective to express the joint model hierarchically and combine the affine parameters with it. If expressed hierarchically, it is possible to obtain a hierarchical connection relationship between the joint parts. As this effect, for example, the parameters of the torso are determined first, and then using the relationship connected to the torso, such as the left and right upper arms, the left and right thighs, and the head, the joint positions belonging to the upper hierarchy are sequentially By changing the parameters, the parameters can be changed and determined efficiently. Furthermore, since the movement of the joint part in the upper hierarchy becomes dominant, an image reflecting the structure of the joint object can be generated.

  By the motion parameter determination method described above, in addition to the effects described in the first to fourth embodiments, it is possible to efficiently generate a highly accurate image.

(Embodiment 6)
Next, a sixth embodiment of the present invention will be described. In the sixth embodiment, another model conversion unit (or a more specific example of the model conversion unit 103) that replaces the model conversion unit 103 in the first to fifth embodiments will be described in detail. Here, description will be made along the first embodiment, but the present invention can be applied to all the embodiments.

  In Embodiment 1, the model conversion unit 103 estimates output information from input information by using model conversion data describing the relationship between input and output. Specifically, using the joint position and motion of the joint object extracted by the parameter calculation unit 102 as input, information on the joint position and motion that could not be obtained by occlusion or the like is estimated and output. Furthermore, using the joint position and movement of the joint object extracted by the parameter calculation unit 102 as input, information such as the joint position of the joint object with higher accuracy including shape information is output.

  As a specific example of such a model conversion unit 103, first, a model conversion unit 103a using a neural network shown in FIG. 26 will be described. The model conversion unit 103a includes an input unit 1031 that acquires a parameter output from the parameter calculation unit 102, a neural network 1032 that converts the parameter acquired by the input unit 1031 into an output vector as an input vector, and a neural network 1032 The output unit 1033 outputs the output vector to the image generation unit 104.

  For the neural network 1032, the neural network is learned by the error back propagation method using the input vector x and the output vector y shown in (Equation 1) to (Equation 4). When the learning is completed, the model conversion unit 103a can output y estimated with respect to the input of x. Although the neural network in FIG. 26 has three layers, the number of layers is not limited. Further, the neural network 1032 may be used while learning, instead of learning in advance.

In the learning process, specifically, the combination load w between the hierarchies is changed using a set of the input vector x and the output vector y as shown in FIG.

ここで、ｆ（ｘ）はシグモイド関数、ｎ^lは、階層ｌの素子数である。 Here, z is an output value in each layer, i is an element number of the neural network, and l is a layer number. In addition,

Here, f (x) is a sigmoid function, and n ^l is the number of elements in the hierarchy l.

  By performing iterative learning using the above (Equation 24) to (Equation 26), the connection load w can be obtained. If the combined load w can be obtained, the output vector y can be obtained for the input vector x by the model conversion unit 103a.

  As a result, the joint position and the motion of the joint object extracted by the parameter calculation unit 102 are used as the input vector x, and the model conversion unit 103a estimates the output vector y related to the joint position and the motion that cannot be obtained by occlusion or the like. can do. Furthermore, it is possible to estimate the output vector y related to the joint position of the joint object with higher accuracy including the shape information using the joint position and the motion of the joint object extracted by the parameter calculation unit 102 as the input vector x. . The error back-propagation method is described in detail in Arviv “Neural network and brain theory”, p437, Science, 1992.

Next, a method for expressing the correlation information described in Embodiment 1 as a linear sum of a plurality of correlation information will be described. FIG. 27 is a functional block diagram illustrating a configuration of the model conversion unit 103b that uses a linear sum of a plurality of pieces of correlation information as correlation information. The model conversion unit 103b includes a correlation information storage unit 1035 that holds a plurality of pieces of correlation information, an input unit 1031 that acquires parameters output from the parameter calculation unit 102, and a parameter acquired by the input unit 1031. A conversion unit 1034 that performs conversion using a linear sum of a plurality of pieces of correlation information held in the correlation information storage unit 1035, and an output unit 1033 that outputs parameters obtained by the conversion unit 1034 to the image generation unit 104. . Incidentally, the correlation information storage unit 1035, for each body type and operation of the joint body, m _x described in (Equation 1) through (10), m _y, to C may be a plurality of sets hold the occlusion occurs each easy example, m _x, m _y, C may be a plurality of sets hold.

Here, an example in which P pieces of correlation information are used for each body shape of a joint object will be described with reference to FIG. The correlation information C ^{1 to} C ^{p in} FIG. 25 is an example of generating a body type correlation matrix. For each joint object having each body type, an input vector x and an output vector y are obtained using the three-dimensional position information and motion information obtained from the motion capture data. And the correlation matrix C is produced | generated using each x and y.

(Equation 5) can be rewritten as follows using N ^p sets of input vectors x prepared for each body type.

(Equation 6) can be rewritten as follows using N ^p sets of input vectors x and output vectors y prepared for each body type.

で表すことができる。ここで、Ｃ_x ^p*はＣ_x ^pの逆行列、または疑似逆行列である。 And (Equation 7) is obtained from (Equation 27) and (Equation 28).

It can be expressed as Here, C _x ^{p *} is an inverse matrix of C _x ^p or a pseudo inverse matrix.

Then, y ^p _expected to estimate for each type of joint object can be expressed by the following equation using the model transformation matrix.

The output vector y to be finally estimated can be expressed by a linear sum of P estimation results.

である。α_k，ｍ_x ^k，Ｃ_x ^kは、ＥＭアルゴリズムで推定することができる。 here,

It is. α _k , _mx ^k , and C _x ^k can be estimated by an EM algorithm.

  Regarding the EM algorithm, Nobuo Ueda, “Bayesian Learning I—Basics of Statistical Learning”, IEICE Journal, Vol. 85No. 4pp. 265-271, 2002.

  Thus, the correlation matrix is obtained for each of a plurality of body types, the correlation information storage unit 1035 holds the matrix, and the conversion unit 1034 performs model conversion using the linear sum, so that the model conversion unit 103b is Even for various body types, there is an effect that model conversion can be performed with a linear sum of a plurality of body types.

  As described above, in addition to the effects of the first to fifth embodiments, the body shape variation of the target joint object is robust.

  As described above, the image generation apparatus according to the present invention has been described based on the embodiments and the modifications. However, the present invention is not limited to these forms and examples. Other forms realized by appropriately combining the constituent elements in each embodiment and modification, and forms obtained by subjecting each embodiment to modifications conceived by those skilled in the art are also included in the present invention.

  The correspondence between the claims and the components in the embodiment is as follows. In other words, examples of “image input means”, “parameter calculation means”, “model conversion means”, “image generation means”, “image evaluation means”, and “parameter change means” in the claims are as follows. An image input unit 101, a parameter calculation unit 102, a model conversion unit 103, an image generation unit 104, an image evaluation unit 201, and a parameter change unit 202. However, the constituent elements in the claims are not limited to the corresponding constituent elements in these embodiments, and equivalents thereof are also included.

  The present invention is an image generation apparatus, particularly an apparatus for generating an image of a joint object including a person, an animal, or the like by image processing, for example, parameters relating to the movement of a joint object, the position of a joint, etc. present in the image Images taken by devices that generate new images that reflect the characteristics of joint objects in the image using parameters related to shape, clothing, etc., animation generation devices, digital cameras / mobile phones with video cameras, video devices, etc. It is useful as a video complementing device that improves the accuracy by complementing images, and a device that generates still images and moving images for games, movies, and computer graphics.

The figure which shows the structure of the image generation apparatus in Embodiment 1 of this invention. The figure which shows the detailed constitution of the parameter calculation section The figure which shows the detailed structure of an image generation part. Flow chart showing operation of image generation apparatus Diagram showing joint model Diagram showing an example of a database showing the position of the head, hands, and feet Diagram showing input and output data Diagram showing examples of input and output vectors Diagram showing examples of input and output vectors The figure which shows the example of a projection to the image of three-dimensional information Diagram showing an example of image generation Diagram showing examples of interpolated and extrapolated images Diagram showing image generation method The figure which shows the structure of the image generation apparatus in Embodiment 2 of this invention. The figure explaining operation of an image evaluation part Flow chart showing operation of image generation apparatus Diagram showing an example of hierarchical representation of a joint model The figure which shows the estimation result of the joint position by Embodiment 3 of this invention The figure which shows the example where a part of image is shielded The figure which shows the structure of the image generation apparatus in Embodiment 4 of this invention. Flow chart showing operation of image generation apparatus Diagram showing an example of image generation Figure showing an example of the parameter setting screen Diagram showing image generation method The figure which shows the example of the correlation matrix by Embodiment 6 of this invention The figure which shows another structural example of a model conversion part The figure which shows another structural example of a model conversion part

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Image input part 102 Parameter calculation part 103, 103a, 103b Model conversion part 104 Image generation part 201 Image evaluation part 202 Parameter change part 301 User setting part 1021 Joint object area | region extraction part 1022 Model fitting part 1023 Inter-joint part location calculation part 1031 Input unit 1032 Neural network 1033 Output unit 1034 Conversion unit 1035 Correlation information storage unit 1041 Pixel movement position calculation unit 1042 Interpolation processing unit 1043 Pixel value determination unit

Claims (19)

An image generation device that generates a new image reflecting the characteristics of the joint object from an image obtained by imaging the joint object,
An image input means for acquiring an image obtained by imaging a joint object;
A parameter calculating means for calculating a first parameter related to a position of a joint or an inter-joint portion of the joint object by fitting a model having a joint held in advance to the joint object in the acquired image;
Area dividing means for extracting a second parameter relating to at least one of color and texture information of the inter-articular part of the joint object by performing area division of the image of the joint object based on the first parameter;
Image generating means for generating a new image reflecting the characteristics of the joint object using the first parameter calculated by the parameter calculating means and the second parameter extracted by the area dividing means. A featured image generation apparatus. The image generation means generates an image reflecting the position of the joint or the joint part of the joint object and the color and texture information of the joint part of the joint object as a new image reflecting the characteristics of the joint object. The image generating apparatus according to claim 1. The image input means acquires temporally continuous images,
The image generation apparatus according to claim 1, wherein the parameter calculation unit calculates a first parameter related to a position and a motion of a joint or an inter-joint site of the joint object using the image. The image generation device further includes:
Image evaluation means for evaluating the image by calculating an error between the image generated by the image generation means and the target image;
Parameter changing means for changing the first parameter based on the evaluation result by the image evaluation means,
The region dividing unit performs the region division based on the first parameter changed by the parameter changing unit,
2. The image according to claim 1, wherein the image generation unit generates the image using the first parameter changed by the parameter change unit and the second parameter extracted by the region dividing unit. Generator. The image generation device further uses the first parameter to estimate shape information of the joint part of the joint object, and a third parameter relating to the position and movement of the joint not included in the first parameter. With
2. The image according to claim 1, wherein the image generation unit generates a new image reflecting characteristics of the joint object by using the first parameter, the second parameter, and the third parameter. Generator. The image generation unit generates, as the new image, a temporally interpolated and extrapolated image generated based on motion information included in the first parameter with respect to a temporally continuous image. The image generating apparatus according to claim 3. The image generation apparatus according to claim 1, wherein the image generation unit generates, as the new image, an image in which a different color or texture is pasted on each part constituting the joint object. The image generation means generates, as the new image, an image obtained by pasting a texture of an inter-articular part of the joint object on an image of a joint object including a posture or movement different from the posture or movement of the joint object. The image generation apparatus according to claim 1. The image according to claim 1, wherein the parameter calculation unit includes a joint object region extraction unit that extracts a region of a joint object with respect to the image, and performs the fitting with respect to the extracted region. Generator. The image generation apparatus according to claim 9, wherein the joint object region extraction unit extracts the region by performing edge extraction on the image. The image generating apparatus further includes periodicity detecting means for detecting a period of motion of the joint object existing in the image acquired by the image input means,
The parameter calculating means, the region dividing means, and the image generating means are configured to calculate a first parameter, extract a second parameter, and extract a second parameter, for each time-series image for one period detected by the periodicity detecting means, and The image generating apparatus according to claim 3, wherein the image is generated. The image generation apparatus according to claim 3, wherein the motion is represented by any one of a motion vector, an acceleration vector, an affine parameter, and an approximate curve parameter. The image generating apparatus according to claim 1, wherein the region dividing unit calculates the second parameter using, as an initial value, position information of a joint position or an inter-joint site included in the first parameter. The image generation unit according to claim 1, wherein the image generation unit generates the new image by performing pixel movement based on a joint position so that a portion connected by a joint is not separated. apparatus. 6. The image generation according to claim 5, wherein the model conversion means obtains correlation information between the first parameter and the third parameter in advance and estimates the third parameter based on the correlation information. apparatus. The model conversion unit estimates the third parameter by estimating a parameter value that cannot be extracted when a part of the first parameter cannot be extracted by the parameter calculation unit. 5. The image generating device according to 5. The parameter calculation means calculates the first parameter expressed hierarchically based on the connection relationship between the joint parts of the joint object,
The image generation apparatus according to claim 5, wherein the model conversion unit estimates the third parameter expressed hierarchically based on a connection relation between joint portions of a joint object. An image generation method for generating a new image reflecting characteristics of the joint object from an image obtained by imaging the joint object,
An image input step for acquiring an image obtained by imaging a joint object;
A parameter calculation step of calculating a first parameter related to the position or movement of the joint of the joint object or the inter-joint region by applying a model having a joint held in advance to the joint object in the acquired image;
A region dividing step of extracting a second parameter related to at least one of a color and texture information of an inter-articular portion of the joint object by performing region division of the image of the joint object based on the first parameter;
An image generation step of generating a new image reflecting the characteristics of the joint object using the first parameter calculated by the parameter calculation step and the second parameter extracted by the region division step. A featured image generation method. A program for generating a new image reflecting the characteristics of the joint object from an image obtained by imaging the joint object,
A program causing a computer to execute the steps included in the image generation method according to claim 18.

Images