JP6676562B2 - Image synthesizing apparatus, image synthesizing method, and computer program - Google Patents

Description

  The present invention relates to a technique for generating a free viewpoint video.

  In the free viewpoint video, an image of an arbitrary viewpoint is synthesized using images captured by cameras arranged at a plurality of positions. By such a combining process, it is possible to view a video from any viewpoint. Such free viewpoint video technology has been studied for a long time as the next generation of video media. In the free viewpoint video, by restoring the three-dimensional shape of the subject in the scene, it is possible to generate a video from a viewpoint where a camera is not actually arranged.

  As one of typical researches capable of realizing a high-quality free viewpoint video, there is a research by Collet et al. (See Non-Patent Document 1). This research proposes a series of pipelines for capturing, synthesizing and distributing free viewpoint videos. With this research technology, it is possible to synthesize high-quality free viewpoint videos. However, a large number of cameras and infrared cameras are required. In addition, it is necessary to limit the background to a uniform color in order to extract a subject area. Furthermore, it is necessary to perform calibration specialized for these special environments. Thus, there are very strict constraints on the shooting environment. Therefore, use in an actual scene is difficult.

  As another study, a free viewpoint video synthesizing method under relatively realistic constraints using a distance sensor has been proposed (see Non-Patent Document 2). However, in the technique according to this proposal, the synthesis quality is not sufficiently high. Occlusion and temporal flicker are one of the major factors that degrade the synthetic quality. Since it is necessary to reproduce information that has not been acquired for the occlusion, it is in principle impossible to solve it without using prior information or the like. The flicker in the time direction is caused by the fact that the distance sensor receives the interference of infrared light or the like, and thus the information acquired by the frames varies. This is being solved by filtering distance information. However, although improvements have been made, they have not yet been fully eliminated. Occlusion and flickering in the time direction are problems to be solved because even a slight occurrence causes a great discomfort to the viewer.

A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Evseev, D. Calabrese, H. Hoppe, A. Kirk, S. Sullivan, “High-quality streamable free-viewpoint video,” ACM Transactions on Graphics, 34 (4), 2015. D. Alexiadis, D. Zarpalas, P. Daras, “Fast and smooth 3D reconstruction using multiple RGB-Depth sensors,” in Visual Communications and Image Processing Conference 2014, pp.173-176.

As described above, the conventional free viewpoint video technique has a problem to be solved, and it has not been realized to generate an image of sufficient quality under the constraint conditions that can be used in an actual scene.
In view of the above circumstances, the present invention provides a technique for generating an image in an arbitrary field of view and time with higher quality by using an image obtained without imposing strict constraints such as limiting the background to a uniform color. It is intended to be provided.

  One embodiment of the present invention provides an input unit that receives inputs of a plurality of moving images, a parameter related to a three-dimensional shape of a subject captured in the moving images, and machine learning obtained in advance at each time. Estimating unit for estimating the three-dimensional joint information of the subject based on the result of the above, the estimation result of the three-dimensional joint information in the frame at each time, and parameters related to the three-dimensional shape of the subject, each time An image synthesizing unit that obtains information indicating the three-dimensional shape of the subject in the above and generates an image including an image of the subject in a specified field of view at a specified time based on the information indicating the three-dimensional shape of the subject And an image synthesizing device comprising:

  One embodiment of the present invention is the above-described image synthesizing device, wherein the parameter relating to the three-dimensional shape of the subject is a reference shape information indicating a three-dimensional shape of the subject, and the three-dimensional shape indicated by the reference shape information. , And reference joint information indicating information of each joint.

  One embodiment of the present invention is the image synthesizing apparatus, wherein the frame at each time constituting the moving image is not a target for acquiring joint information of the subject by interpolation based on an image of the frame. And a frame classification unit that determines whether or not the key frame is a key frame, and for the frames other than the key frame, obtains the three-dimensional joint information by an interpolation process using an estimation result of the three-dimensional joint information in the key frame. And a frame interpolation unit that performs the operation.

  One embodiment of the present invention is the above-described image synthesizing apparatus, wherein, for a three-dimensional model representing a three-dimensional shape of a living thing or an object having a joint, an image rendered by a plurality of fields of view, By generating learning data including joint information indicating the position and angle of the joint of the original model, a data generating unit, and performing machine learning using the image and the joint information generated by the data generating unit, A learning unit that acquires a learning result for estimating joint information of a living thing or an object captured in the target image based on a target image that is an image to be processed, and the estimating unit includes: The three-dimensional joint information is estimated based on the result of the machine learning by the learning unit.

  One aspect of the present invention is the above-described image synthesizing device, wherein the data generation unit generates a plurality of scenes having different joint information based on the same three-dimensional model, and generates one or more scenes for each scene. Render an image of the field of view.

  One aspect of the present invention is an input step of receiving an input of a plurality of moving images and a parameter related to a three-dimensional shape of a subject captured in the moving image, and machine learning previously obtained for a frame at each time. Estimating the three-dimensional joint information of the subject based on the result of the estimation, the estimation result of the three-dimensional joint information in the frame at each time, and the parameters related to the three-dimensional shape of the subject, Acquiring the information indicating the three-dimensional shape of the subject in the step of generating an image including an image of the subject in a specified field of view at a specified time based on the information indicating the three-dimensional shape of the subject. And an image combining method comprising:

  One embodiment of the present invention is a computer program for causing a computer to function as the above-described image composition device.

  According to the present invention, an image in an arbitrary field of view and time can be generated with higher quality by using an image obtained without imposing severe restrictions such as limiting the background to a uniform color.

FIG. 1 is a schematic block diagram illustrating a configuration example of an image synthesis device 10 according to an embodiment. FIG. 3 is a diagram illustrating a configuration example of a data generation unit 111. FIG. 3 is a diagram illustrating a configuration example of a learning unit 112. FIG. 3 is a diagram illustrating a specific example of a network constructed by a network construction unit 211. FIG. 3 is a diagram illustrating a configuration example of an input unit 12. FIG. 3 is a diagram illustrating a configuration example of an estimation unit 13. FIG. 3 is a diagram illustrating a configuration example of a frame classification unit 14. FIG. 3 is a diagram illustrating a configuration example of a frame interpolation unit 15. FIG. 3 is a diagram illustrating a configuration example of an image synthesis unit 16. FIG. 3 is a diagram showing a specific example of a flow of preprocessing of the image synthesizing device 10. FIG. 9 is a diagram illustrating a specific example of a process regarding image synthesis.

  FIG. 1 is a schematic block diagram illustrating a configuration example of an image composition device 10 according to the embodiment. The image synthesizing apparatus 10 includes a CPU (Central Processing Unit), a memory, and an auxiliary storage device connected by a bus, and executes an image synthesizing program. By executing the image synthesizing program, the image synthesizing device 10 functions as a device including the learning device 11, the input unit 12, the estimation unit 13, the frame classification unit 14, the frame interpolation unit 15, and the image synthesis unit 16. Note that all or a part of each function of the image synthesizing apparatus 10 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). . The image composition program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built in a computer system. The image composition program may be transmitted via a telecommunication line.

The learning device 11 includes a data generation unit 111 and a learning unit 112.
First, the data generator 111 will be described. The data generation unit 111 generates learning data used by the learning unit 112. The learning data generated by the data generating unit 111 includes image data and joint information. The image data is generated as computer graphics in which a previously generated three-dimensional human model is included in a field of view in a predetermined posture. The data generation unit 111 forms one or more scenes for each three-dimensional person model using one or more three-dimensional person models, and generates computer graphics in one or more fields of view for each scene. To generate a plurality of image data.

  The three-dimensional person model is data having, for example, the position of each joint of the person, the surface shape of the person, and an image (texture image) of the surface of the person. By using a three-dimensional person model, it is possible to generate an image of a person having a desired posture in a desired visual field. Such a three-dimensional person model may be data created in advance by a human hand, data created by artificial intelligence (AI), or a tertiary model such as motion capture. The data may be created using a technique for recording the original shape. The number of joints included in the three-dimensional human model may be appropriately determined according to the accuracy of the estimation process obtained by the estimation unit 13 and the like. For example, the number of joints may be fifteen, a smaller number, or a larger number. For example, when it is necessary to more accurately estimate the movement of the fingertip or the like, it is desirable to increase the number of joints in the corresponding portion. In this case, the number of joints increases.

  The predetermined posture indicates the posture of a person defined by the positions and angles of the joints of the person. By changing the positions and angles of the joints of the three-dimensional human model, a plurality of postures can be obtained from the same three-dimensional human model.

  A scene refers to modeling data of a generated computer graphics space (hereinafter, referred to as “target space”). A scene is defined by a three-dimensional person model and environment information. The environment information is information on an object located in the target space. For example, the environmental information includes the position of the light source located in the target space, the type of the light source, the direction in which the light source emits light, the intensity of the light emitted by the light source, and the objects (walls, furniture, plants, animals, etc.) located in the target space. Etc.) material and position. The scene is different if any one of the posture, the position, and the environment information of the three-dimensional human model is different. The plurality of scenes may be created in advance by human hands or may be created by artificial intelligence.

  As described above, the data generation unit 111 may render computer graphics in a plurality of views for one scene. The field of view is defined, for example, by the position of the viewpoint and the direction of the line of sight. The position of the viewpoint may be defined as the entire periphery of the scene. Further, the field of view may be changed according to information such as the characteristics and use of the scene. For example, if the field of view is predetermined in the image of the actual processing input to the input unit 12 described later, the computer graphics may be generated in the same field of view as the field of view.

  The joint information includes information indicating the position of each joint of the three-dimensional person model (joint position information) and information indicating the angle formed by each joint (joint angle information). The joint position information may be represented, for example, as three-dimensional coordinate values of each joint when a predetermined position of the three-dimensional human model is set as the origin. The joint position information may be represented based on, for example, camera coordinates in the scene. The joint angle information may be represented using, for example, an Euler angle, may be represented using a quaternion, or may be represented in another expression. The joint information is generated for each scene of the three-dimensional person model. Even if the visual field changes, the joint information does not change unless the three-dimensional person model and the scene change.

  The data generation unit 111 generates computer graphics in one or a plurality of views in the same scene (the three-dimensional person model and the scene indicate the same). Computer graphics of one or a plurality of fields of view generated in the same scene are called a scene CG set. The data generation unit 111 outputs data in which the scene CG set is associated with the joint information in the scene as unit learning data. The data generator 111 generates a plurality of such unit learning data. The data generator 111 outputs learning data including a plurality of unit learning data.

  FIG. 2 is a diagram illustrating a configuration example of the data generation unit 111. The three-dimensional person model is represented as “3D model” in FIG. 2 due to the space of the drawing. A plurality of (for example, N types) three-dimensional person models are input to the data generation unit 111. The data generation unit 111 includes a scene generation unit 201 and an image generation unit 202. The scene generation unit 201 generates one or a plurality (for example, M types) of scenes for each input three-dimensional person model. Note that a different number of scenes may be generated for each three-dimensional person model. The image generation unit 202 generates one or a plurality (for example, L types) of computer graphics having a visual field for each scene. The image generation unit 202 generates a plurality of unit learning data based on the generated computer graphics.

  Next, the learning unit 112 will be described. The learning unit 112 performs a learning process based on the plurality of unit learning data generated by the data generation unit 111. The learning unit 112 acquires parameters used for the estimation process performed by the estimation unit 13 by executing machine learning. The estimation process is a process of estimating joint information of a person photographed in a target image from an image to be processed (hereinafter, referred to as a “target image”). The machine learning implemented in the learning unit 112 may be any technology. For example, a technology such as a deep neural network (DNN) or a support vector machine (SVM) may be applied.

  FIG. 3 is a diagram illustrating a configuration example of the learning unit 112. The learning unit 112 receives learning data including a plurality of unit learning data. The learning unit 112 includes a network construction unit 211 and a parameter learning unit 212. Note that the learning unit 112 illustrated in FIG. 3 is only a specific example in a case where DNN is applied. When another machine learning technique is applied to the learning unit 112, the configuration of the learning unit 112 may be changed according to the applied machine learning.

  The network construction unit 211 constructs a network used for learning. For example, when DNN is applied to the learning unit 112, the learning unit 112 constructs a deep neural network that inputs a target image and outputs three-dimensional joint information of a subject. The network constructed by the network construction unit 211 is constructed according to the number of output three-dimensional joint information. For example, in the output layer of the network, the number of dimensions is determined according to the number of required joints.

  FIG. 4 is a diagram illustrating a specific example of a network constructed by the network construction unit 211. The network receives L images included in the unit learning data as input and outputs joint information corresponding to those images. For example, each image (image generated as computer graphics) included in the unit learning data is a 3 × 256 × 256 color image composed of three channels of 256 in the vertical size, 256 in the horizontal size, and RGB. It is. This color image is input for the number of viewpoints. For each image, the number of channels is 36, and a 36 × 5 × 5 convolution is performed using a 5 × 5 kernel. Further, 2 × 2 pooling is performed. At this time, the number of channels generated in the convolutional layer is 36, and the stride width is 2. These outputs are then concatenated vertically by a CONCAT process. Data obtained by convolving a multi-viewpoint image by such CONCAT processing is convolved by the same filter in subsequent processing. By using a network having such a structure, it is possible to extract a spatial feature for obtaining joint information. The structure and processing of the subsequent network follow those of a general network. The number of channels is 72, and 72 × 3 × 3 convolution is performed using a 3 × 3 kernel. Processing such as pooling of 2 × 2 with a stride width of 2 is repeated twice. The results are poured side by side into the FC layer. The FC layer is composed of, for example, three layers. ReLU is used as each activation function. The number of nodes is 512, 1024, and 2048 in order from the upstream. The output layer outputs the xyz coordinate position and the xyz Euler angles for each of the 15 joint positions. Therefore, the output layer has 90 dimensions.

  The network of FIG. 4 described above is only one example. Other values may be used for the size and stride width of each kernel. In addition, any type of activation function may be used. However, as the joint information to be output, it is desirable that the xyz coordinate position and the xyz Euler angle are output for each joint even when the number of joints changes.

  By constructing such a network, a network for estimating three-dimensional joint information of a subject from an image is constructed. In addition, by superimposing images in a plurality of visual fields, it is possible to handle not as joint information on an image but as joint information on a space. Therefore, it is possible to estimate three-dimensional joint information.

  The parameter learning unit 212 obtains network parameters by performing machine learning on the network constructed by the network construction unit 211 using the learning data generated by the data generation unit 111. At this time, with respect to the number of repetitions and the initial parameters, values considered to be optimal may be manually given. In the examples of FIGS. 3 and 4, the parameters obtained by such a learning process are output from the learning unit 112.

Hereinafter, the input unit 12 and the estimation unit 13 will be described.
FIG. 5 is a diagram illustrating a configuration example of the input unit 12. The input unit 12 receives inputs of a plurality of moving images, reference shape information indicating a three-dimensional shape of a predetermined subject, reference joint information indicating three-dimensional joint information of the predetermined subject, and deformation parameters of the predetermined subject. The plurality of moving images are moving images obtained by capturing the same scene at the same time by the cameras arranged at a plurality of positions. For example, a moving image obtained by shooting the field at the same time (for example, 1 hour from 13:00 to 14:00 on the same day) by a plurality of cameras arranged so as to surround a field such as a soccer field is input. . The moving image may be a color moving image, a grayscale moving image, or a binary moving image. Moving image data may be input in real time from each camera, or a moving image recorded on a recording medium such as a hard disk drive (HDD) may be input. The moving images need not be shot at exactly the same time, and a slight time lag may occur before and after.

  The frame separation unit 121 separates the input moving image into a plurality of frame images. In the present embodiment, the image of each frame of the input moving image is processed as the target image. That is, an estimation process using the learning result by the learning device 11 is performed on the image of each frame. It is preferable that each frame of the moving image is provided with a time at which each frame was captured (hereinafter, referred to as “frame time”). If no frame time has been assigned, the input unit 12 may assign a frame time to each frame based on the shooting start time and the playback time of the moving image. In the following description, for simplicity, only one person exists in the moving image, and the person is a subject to be rendered using the reference shape information or the like (hereinafter, referred to as a “subject of interest”). I do. When there are a plurality of objects of interest, the input unit 12 may divide the moving image for each object of interest by performing a process of cutting out a person region in the moving image. By performing the same processing on a moving image relating to each target object as when only one target object is present, the same processing can be performed even when a plurality of target objects are present.

  The frame separating unit 121 associates frames obtained from different moving images with frames that satisfy a predetermined condition indicating that they are images of frames captured at the same time based on the frame time. The predetermined condition may be a condition indicating that a frame of a moving image serving as a reference has the closest frame time among frames obtained from another moving image. A set of frames associated with each frame estimated to have been photographed at the same time is referred to as a same-time frame set. In the following processing, each frame included in the same time frame set may be treated as having been shot at the same time regardless of the actual frame time.

The reference shape information indicates a three-dimensional shape of a subject (subject of interest) predetermined based on a predetermined reference among subjects captured in the input moving image. For example, reference shape information is input for a subject that is likely to be particularly noticed. For example, when a moving image of a soccer match is input, reference shape information of all players (starter players and benched players) participating in the soccer match may be input. The reference shape information may be obtained by performing a three-dimensional shape restoration process on the subject of interest in advance. For example, the reference shape information may be generated based on data obtained by performing measurement using a measuring device such as a distance sensor on the subject of interest. For example, the reference shape information may be generated by using still images captured by cameras at a plurality of positions. The reference shape information may be data (three-dimensional person model data) including, for example, the position of each joint of the person, the surface shape of the person, and an image (texture image) of the person's surface. By using three-dimensional person model data, it is possible to generate an image of a person in a desired posture in a desired visual field. The posture of the subject of interest in the reference shape information may be a posture such as a T pose or an A stance, or may be another posture. In addition, when a defect or noise occurs in the input reference shape information, the input unit 12 uses a method such as Poisson Surface Reconstruction (Reference Document 1) or general spatial filtering to obtain a high quality surface shape. May be performed. By performing such processing, the subsequently restored shape retains a continuous surface. As a result, loss due to a change in the viewpoint position is less likely to occur.
Reference 1: M. Kazhdan, M. Bolitho, H. Hoppe, “Poisson Surface Reconstruction,” Symposium on Geometry Processing 2006, 61-70.

  The reference joint information is three-dimensional joint information of each joint in the posture of the input reference shape information. The three-dimensional joint information indicates a joint position and a joint angle. The position of the joint is represented by, for example, xyz coordinates. The angle of the joint is represented, for example, by an Euler angle about the xyz axis. The reference joint information may be measured by using a measurement technique such as motion capture when generating the reference shape information, or may be obtained by performing an estimation process on an image of the subject of interest. .

  The deformation parameter is a parameter that determines how the shape of the target object in the moving image is deformed when the joint of the target object corresponding to the reference shape information changes. The deformation parameter is, for example, a parameter that defines a change in shape according to the rotation of the joint. The deformation parameter may be obtained in advance by measurement or the like. For example, the deformation parameter may be determined so as to be inversely proportional to the distance between the joint and the vertex of the shape by using a general skinning method. The deformation parameters may be manually defined using software.

  FIG. 6 is a diagram illustrating a configuration example of the estimation unit 13. The target image and the learning result are input to the estimating unit 13. The target image input to the estimation unit 13 is an image of each frame of each moving image input in the input unit 12. The estimating unit 13 estimates the three-dimensional joint information of the subject of interest in the target image using the result of the machine learning performed by the learning device 11. The result of machine learning used by the estimating unit 13 is, for example, a network to which parameters obtained by the learning unit 112 are given. The result of machine learning used by the estimating unit 13 is, for example, a discriminator obtained by machine learning.

  The frame classification unit 14 classifies each of the same-time frame sets into a key frame and a non-key frame using the joint information of the subject of interest in each frame estimated by the estimation unit 13. The frame classifying unit 14 assigns a flag (key frame flag or non-key frame flag) indicating the classification result as to whether the frame is a key frame or not to each of the classified frames. A key frame is a same-time frame set that is not subjected to joint information interpolation processing in a frame interpolation unit 15 described later. The non-key frame is a same-time frame set to be subjected to joint information interpolation processing in a frame interpolation unit 15 described later.

  FIG. 7 is a diagram illustrating a configuration example of the frame classification unit 14. The frame classification unit 14 may, for example, add a key frame flag to the same time frame set at a predetermined cycle, and add a non-key frame flag to another same time frame set. The frame classification unit 14 may add a key frame flag to the same-time frame set in the image that satisfies a predetermined condition. The predetermined condition may be, for example, that the moving speed of the subject of interest in the image has shown an extreme value. The moving speed may be the moving speed of the entire subject of interest or the moving speed of some joints or body parts (for example, arms or face). In this case, the frame classification unit 14 may determine the moving speed of the subject of interest in the image, and add a key frame flag when the moving speed indicates an extreme value. Further, instead of the moving speed, a change rate (change speed) of the angle of a part of the joint of the subject of interest may be used. When any one frame in the same-time frame set satisfies a predetermined condition, the joint information estimation flag assigning unit 122 may assign a key frame flag to the same-time frame set. When a predetermined condition is satisfied in a predetermined number or more of frames in the time frame set, a key frame flag may be given to the same time frame set. A non-key frame flag is assigned to all the same-time frame sets to which no key frame flag has been assigned.

FIG. 8 is a diagram illustrating a configuration example of the frame interpolation unit 15. The frame interpolating unit 15 uses the joint information of the same-time frame set to which a non-key frame flag has been added (hereinafter, referred to as “non-key frame”) as a same-time frame set having a similar frame time and has a key frame flag added thereto. It is obtained by an interpolation process using the joint information of the same time frame set (hereinafter referred to as “key frame”). The frame interpolation unit 15 acquires joint information for all non-key frames by performing interpolation processing. The interpolation processing by the frame interpolation unit 15 may be linear interpolation according to the number of frames between key frames, or some weight may be given to the interpolation coefficient. The moving speed of the subject of interest and the moving speed of each joint between frames may be calculated, and an interpolation coefficient may be obtained based on the calculated moving speed. For example, P is the nearest key frame at a frame time earlier than the non-key frame to be processed, F is the nearest key frame at a frame time later than the non-key frame to be processed, and the angle of the joint to be processed. And the angular velocity are rP and vP in the key frame P, and rF and vF in the key frame F, respectively. Note that a single joint angle is described for simplicity. In this case, considering the change in speed, the angle r of the joint to be processed may be expressed as follows by performing internal division using the speed.

  Next, the image synthesizing unit 16 will be described. The image combining unit 16 receives reference shape information, reference joint information, three-dimensional joint information, and deformation parameters. The input three-dimensional joint information is joint information estimated by the estimating unit 13 or joint information obtained by the interpolation processing of the frame interpolating unit 15. For the key frame, the joint information estimated by the estimation unit 13 is input. For non-key frames, joint information obtained by the interpolation processing of the frame interpolation unit 15 is input. The image synthesizing unit 16 compares the three-dimensional joint information with the reference joint information. The image synthesis unit 16 acquires a deformed shape by deforming the reference shape based on the comparison result and the deformation parameter. The image synthesizing unit 16 performs the above-described processing on the three-dimensional joint information obtained in all the same-time frame sets. The image synthesizing unit 16 synthesizes a free viewpoint video by using the acquired deformed shape.

  FIG. 9 is a diagram illustrating a configuration example of the image synthesis unit 16. Hereinafter, the image combining unit 16 will be described in detail with reference to FIG. 9 as an example. The image synthesis unit 16 includes a joint displacement calculation unit 161, a deformation unit 162, and an image generation unit 163.

  The joint displacement calculator 161 calculates a difference between the three-dimensional joint information and the reference joint information. For example, the joint displacement calculation unit 161 calculates a difference between the three-dimensional coordinates in the three-dimensional joint information and the three-dimensional coordinates in the reference joint information, and obtains a positional shift on the x-axis, the y-axis, and the z-axis. In addition, the joint displacement calculation unit 161 calculates a difference between the three-dimensional angle in the estimated three-dimensional joint information and the three-dimensional angle in the reference joint information, and centers the x-axis, the y-axis, and the z-axis. Get the rotation angle shift.

  The deformation unit 162 deforms the reference shape based on a difference between the three-dimensional joint information and the reference joint information and a deformation parameter. By such a process, the deforming unit 162 deforms the reference shape so that the posture of the subject of interest at the frame time of the same time frame set to be processed becomes the same posture as the subject. The deforming unit 162 outputs information on the shape after being deformed in this way as deformed shape information. By performing such processing in all the same-time frame sets, deformed shape information corresponding to each of the same-time frame sets is obtained. The deforming unit 162 may record the deformed shape information in the storage device in association with each frame time.

  The image generation unit 163 generates a free viewpoint video at the designated visual field and time. The field of view and the time may be specified by, for example, a device that reproduces the free viewpoint video, or may be specified by a person who views the free viewpoint video. The image generation unit 163 acquires deformed shape information at a frame time corresponding to the designated time. The acquired deformed shape information indicates the position and orientation of the subject of interest at that time. The image generation unit 163 renders an image in the designated visual field using the acquired deformed shape information. At this time, the image of the subject of interest is obtained by rendering using the deformed shape information. The image generation unit 163 may render the background image of the subject of interest based on model data indicating the background obtained in advance, or one or a plurality of frames having a near field of view in a corresponding frame set at the same time. Rendering may be performed by performing image processing such as affine transformation using an image. The image generation unit 163 generates a free viewpoint video at the designated visual field and time by combining the background image and the image of the subject of interest. When a moving image is requested, the image generating unit 163 may generate a video at a free viewpoint by repeatedly executing the above processing along the time axis.

  FIG. 10 is a diagram illustrating a specific example of the flow of the pre-processing of the image composition device 10. The pre-processing of the image synthesizing device 10 is executed by the learning device 11. First, the data generation unit 111 acquires a three-dimensional person model including joint information (step S101). Next, the data generation unit 111 generates a scene with a plurality of postures by changing the joint information of the three-dimensional human model (step S102). Next, the data generator 111 renders computer graphics (images) with a plurality of visual fields for each scene (step S103). The data generation unit 111 generates unit learning data by associating a plurality of images generated for each scene with the joint information (step S104). Next, the learning unit 112 performs a learning process based on the plurality of unit learning data (image and joint information) generated by the data generation unit 111 (Step S105). The learning unit 112 sets parameters obtained based on the result of the learning process in the network (step S106).

  FIG. 11 is a diagram illustrating a specific example of a process related to image synthesis. First, the input unit 12 inputs a plurality of moving images in which the target object is reflected, reference shape information indicating the three-dimensional shape of the target object, reference joint information indicating three-dimensional joint information of the target object, and deformation parameters of the target object. Accept (Step S107). Next, the input unit 12 separates the input moving image into frames, and performs processing such as changing the size as needed (step S108). Next, the estimating unit 13 estimates joint information of the subject for each of the same time frame sets (step S109). Next, the frame classification unit 14 classifies each of the same-time frame sets into either a key frame or a non-key frame, and gives a flag according to the classification result (step S110). Next, the frame interpolation unit 15 obtains joint information by interpolation processing in the same-time frame set to which the non-key frame flag has been added (step S111). Next, the image synthesizing unit 16 generates a deformed shape in each of the same-time frame sets (Step S112).

  Thereafter, at the timing of generating the free viewpoint video, the image synthesizing unit 16 renders the image of the subject of interest at the specified time and field of view using the deformed shape acquired in advance (step S113). Then, the image combining unit 16 generates and outputs a combined image by combining the obtained image with the background image (step S114).

  In the learning device 11 configured as described above, the learning data is generated using the computer graphics rendered by the data generation unit 111. Therefore, it is possible to more easily generate a plurality of learning data including three-dimensional joint information and an image. Particularly, machine learning such as deep learning generally requires a large amount of learning data, and thus the learning device 11 described above is effective.

  Further, by changing the joint information of one three-dimensional person model, it is possible to generate an image and joint information based on various joint information from one three-dimensional person model. For example, in the conventional motion capture, even for the same person (three-dimensional person model), it is necessary to measure joint information for each of different postures, which is troublesome. On the other hand, in the learning device 11, even if motion capture is used, once a three-dimensional human model is obtained, learning data of a plurality of postures can be easily obtained by changing joint information. It becomes possible.

  Further, since joint information is defined in the three-dimensional person model, more accurate joint information can be obtained as learning data.

  In the image synthesizing apparatus 10 configured as described above, the following processing is performed to generate a free viewpoint video. Based on a plurality of actually captured moving images, three-dimensional joint information of each joint of the subject of interest at each frame time is estimated. Based on the estimation result of the three-dimensional joint information, the reference shape of the target object obtained in advance is deformed, and the deformed shape information of the target object at each frame time is obtained. Then, when actually generating the free viewpoint video, the video of the target subject is generated by performing rendering in the specified field of view using the deformed shape information of the target subject at the specified time. For this reason, for example, occlusion problems may occur even in a part of a video that is not obtained as a shadow in a plurality of actually captured moving images (for example, a part of a side of a subject of interest or a part below a chin). Can be suppressed.

  In addition, when acquiring the deformed shape information at each frame time, the 3D joint information of the subject of interest is not independently estimated from the moving image at all frame times, but a part of the frame time (key frame flag is added. The estimation result by the estimation unit 13 is adopted only in the same-time frame set of the (set frame time). Then, in the same-time frame set of the remaining frame times (frame times to which the non-key frame flag is added), interpolation is performed based on the estimation result in the same-time frame set to which the key frame flag is added, not from a moving image. Through the processing, three-dimensional joint information is obtained. Therefore, flicker in the time direction hardly occurs at least from the same-time frame set to which the key frame flag is added to the same-time frame set to which the next key frame flag is added. Such processing makes it possible to suppress flicker in the time direction.

(Modification)
The computer graphics generated by the data generation unit 111 may be a color image, a grayscale image, or a binary image.

  Although the target of processing by the above-described image synthesizing device 10 is a person, the target of processing is not necessarily limited to a person. The processing target may be any living thing or object having a joint. For example, animals may be targets for processing. In this case, a learning result of joint information is obtained by using a three-dimensional animal model instead of the three-dimensional person model. An image obtained by photographing an animal to be processed is input to the input unit 12 of the image synthesizing device 10, and joint information of the animal is estimated. For example, a robot may be a processing target. In this case, a learning result of the joint information is obtained by using a three-dimensional robot model instead of the three-dimensional person model. An image of the robot to be processed is input to the input unit 12 of the image synthesizing device 10, and joint information of the robot is estimated.

  The data generation unit 111 may execute a predetermined process (hereinafter, referred to as “image pre-processing”) on the generated computer graphics. The image preprocessing is a process executed for the purpose of further improving the accuracy of the learning process in the learning unit 112 and the estimation process by the estimation unit 13. The image pre-processing may be, for example, a change in size or a cut-out of a region to be processed (for example, a person). However, the image pre-processing performed on the generated computer graphics is common, and the image pre-processing is performed so that all the changed sizes are the same. The image pre-processing performed by the data generation unit 111 is similarly performed on the processing target image input by the input unit 12. In this case, it is desirable that the size of the image after the image preprocessing is executed in the input unit 12 is the same as the size of the computer graphics after the image preprocessing is executed by the data generation unit 111. In other words, it is desirable that the size of the computer graphics used in the learning unit 112 and the size of the image used in the estimation unit 13 are the same.

  The network used by the learning unit 112 is not constructed by the network construction unit 211, but a network constructed in advance may be stored in a storage unit (not shown). In this case, the parameter learning unit 212 executes machine learning by reading the network stored in the storage unit.

  The learning device 11 may be configured as a device different from the image composition device 10. In this case, an image composition system may be constructed. The image composition system includes an image composition device 10 and a learning device 11. In this case, the image composition device 10 includes an input unit 12 and an estimation unit 13. The image synthesizing device 10 acquires data indicating a learning result from the learning device 11 via a network or the like, and executes an estimation process.

  The data generation unit 111 may be provided in a learning data generation device different from the learning device 11. In this case, the learning data generated by the learning data generation device may be provided to the learning device 11 via a network, a storage medium, or the like.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the embodiments and includes a design and the like within a range not departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Image synthesis apparatus, 11 ... Learning apparatus, 111 ... Data generation part, 112 ... Learning part, 12 ... Input part, 13 ... Estimation part, 14 ... Frame classification part, 15 ... Frame interpolation part, 16 ... Image synthesis part, 201: scene generation unit, 202: image generation unit, 211: network construction unit, 212: parameter learning unit, 121: frame separation unit, 161: joint displacement calculation unit, 162: deformation unit, 163: image generation unit

Claims (7)

An input unit that receives input of a plurality of moving images and a parameter related to a three-dimensional shape of a subject captured in the moving image;
An estimating unit that estimates three-dimensional joint information of the subject based on a result of machine learning obtained in advance for a frame at each time;
The information indicating the three-dimensional shape of the subject at each time is obtained based on the estimation result of the three-dimensional joint information in the frame at each time and the parameter related to the three-dimensional shape of the subject, and the three-dimensional shape of the subject An image combining unit that generates an image including an image of the subject in a specified field of view at a specified time based on information indicating
An image synthesis device comprising:   The parameter relating to the three-dimensional shape of the subject includes reference shape information indicating the three-dimensional shape of the subject, and reference joint information indicating information of each joint in the three-dimensional shape indicated by the reference shape information. Item 2. The image synthesizing device according to Item 1. A frame classification unit that determines whether or not a frame at each time configuring the moving image is a key frame that is not a target frame to be obtained by performing interpolation processing on the joint information of the subject based on the image of the frame;
3. The frame interpolating unit that acquires the three-dimensional joint information by performing an interpolation process using an estimation result of the three-dimensional joint information in the key frame for a frame other than the key frame. 4. Image synthesis device. For a three-dimensional model representing a three-dimensional shape of a living thing or an object having a joint, an image rendered by a plurality of visual fields and joint information indicating the position and angle of the joint of the three-dimensional model when rendered are included. A data generator for generating learning data,
By performing machine learning using the image and the joint information generated by the data generation unit, a joint of a living thing or an object captured in the target image based on the target image that is an image to be processed A learning unit that acquires a learning result for estimating information,
The image synthesizing device according to claim 1, wherein the estimating unit estimates three-dimensional joint information based on a result of machine learning performed by the learning unit.   The image synthesizing device according to claim 4, wherein the data generating unit generates a plurality of scenes having different joint information based on the same three-dimensional model, and renders an image of one or a plurality of visual fields for each scene. . An input step of receiving an input of a plurality of moving images and a parameter relating to a three-dimensional shape of a subject captured in the moving image;
An estimation step of estimating three-dimensional joint information of the subject based on a result of machine learning obtained in advance for a frame at each time;
The information indicating the three-dimensional shape of the subject at each time is obtained based on the estimation result of the three-dimensional joint information in the frame at each time and the parameter related to the three-dimensional shape of the subject, and the three-dimensional shape of the subject An image combining step of generating an image including an image of the subject in a specified field of view at a specified time based on information indicating
An image synthesis method comprising:   A computer program for causing a computer to function as the image synthesizing device according to claim 1.

Images