JP2022164390A - Image generation apparatus, image generation method, and program - Google Patents

Abstract

To provide an image generation apparatus, and the like, configured to suppress generation of visual differences or unnecessary configuration to generate a three-dimensional avatar which looks natural.SOLUTION: An image generation apparatus 1 which generates 3D avatars of users 21, 22, ... 2m by applying a 2D image to a 3D template serving as an image template of a 3D model includes: an image acquisition unit 11 which acquires a recorded 3D template and a 2D image; a face image 3D model generation unit 12 which generates a face image 3D model which is a 3D image obtained by adding position information in three-dimensional direction to 2D images of the users 21, 22, ... 2m; an adaptation unit 13 which generates an adapted image by adapting the face image 3D model to the 3D template using the position information of the face image 3D model; and an image correction unit 14 which generates, when predetermined mismatch that causes visual mismatch is generated in the adapted image, a 3D avatar by modifying the position information in three-dimensional direction in at least a part of the adapted image to correct the mismatch.SELECTED DRAWING: Figure 1

Description

There is an application for application of Article 30, Paragraph 2 of the Patent Act Online session of Computer Entertainment Developers Conference 2020 (CEDEC2020) (see URL below). (http://cedec.cesa.or.jp/2020/session/detail/s5efaec5355159.html)

The present invention relates to a technique for creating a three-dimensional image (3D image) by applying a two-dimensional image (2D image) of a creature such as a person to a three-dimensional model using computer graphics.

In recent years, with the development of various image processing technologies, represented by image generation technology using computer graphics, it has become possible to transform a virtual three-dimensional object in three-dimensional space into a three-dimensional image with depth on a two-dimensional image. Techniques for generating 3D images represented as images (hereinafter simply referred to as "3D images") have been developed. Along with this, a technology has been developed for generating a 3D image of a creature represented by a person as an animation and displaying it on a screen as a moving 3D image. As a representative example, avatars (hereinafter simply referred to as "avatars") as characters acting as alter ego of users of computer terminals and communication terminals in virtual spaces such as communities on the Web and virtual cities are created using 3D images. A technique for automatically generating 3D avatars (hereinafter simply referred to as "3D avatars") has been developed and spread.

When generating a 3D image of a living thing such as a person represented by a 3D avatar, it is important to generate an image that resembles a real living thing such as a real person. Therefore, conventionally, a 3D character (hereinafter simply referred to as a "3D character"), which is a character represented by a 3D avatar, etc., is transformed into a 3D avatar, etc. using an image of the face or appearance of a real creature such as a real person. Techniques for generating images of 3D characters are known.

For example, a step of determining a face mesh sequence, a step of deforming the face mesh embedded in the face image, a step of determining displacement values between the vertices of the face mesh and the vertices of the face mesh, and using the face image as a texture to generate the face mesh a facial animation comprising: projecting upward; applying displacement values to vertices of subsequent facial meshes in the facial mesh sequence; and playing the facial mesh sequence to create a facial animation of the facial images. A technique related to a method executed by a device for creating is known (see, for example, Patent Literature 1).

Japanese Patent No. 4872135

Here, when generating a 3D character image using an image of the face or figure of a real creature, an abstract three-dimensional model of the shape of a part of the body such as the head of the creature or the shape of the entire body is used. A 2D image (hereinafter simply referred to as "2D image") having only a two-dimensional spread of a part of the body of a real living creature (for example, a face) in a 3D model (hereinafter simply referred to as "3D model") that forms an image as ) is often performed to generate a 3D character with a shape resembling a real person. When a 3D character is generated by such a method, visual differences and unnecessary visual differences such as unnatural color differences and irregularities appear at the boundary between the part to which the 2D image is applied and the part where the 3D model appears as it is on the surface. configuration, etc. However, in the invention described in Patent Document 1, there is no configuration for suppressing the occurrence of such a visual difference or unnecessary configuration. Therefore, in the invention described in Patent Document 1, there is a problem that it is not possible to prevent the generated 3D character from being configured as something that gives an unnatural impression.

The present invention has been made in view of such problems, and provides an image generation apparatus, an image generation method, and a program that can generate a 3D character that suppresses the occurrence of visual differences and unnecessary configurations and gives a natural impression. The challenge is to provide

In order to solve this problem, the invention according to claim 1 applies a 2D image having two-dimensional direction information to a 3D template as an image template of a 3D model as an image model having three-dimensional direction information. an image generating device for generating a 3D image of a living organism, comprising: image acquisition means for acquiring the recorded 3D template and the 2D image; and adding position information in three-dimensional directions to the 2D image of the living organism. 3D organism model generating means for generating a 3D organism model as a 3D image using the position information of the 3D organism model, and fitting means for generating a matched image by fitting the 3D organism model to the 3D template using the position information of the 3D organism model and correcting the inconsistency by changing the positional information in the three-dimensional direction to at least a part of the matching image when the matching image has a predetermined mismatch that causes a visual discomfort. and image correction means for generating a modified image.

According to a second aspect of the invention, in addition to the configuration of the first aspect, the image correcting means corrects the inconsistency in a portion where color discontinuity occurs in the continuous planes of the corrected image. It is characterized in that a process for correcting the color discontinuity is performed.

The invention according to claim 3 is characterized in that, in addition to the configuration according to claim 1 or 2, the image correction means is configured to: It is characterized by performing processing for correcting unevenness in the Z direction that occurs in the portion.

The invention according to claim 4 is characterized in that, in addition to the configuration according to claim 3, the image correcting means is configured to: It is characterized by performing processing for correcting the position in the Z direction.

The invention according to claim 5 is, in addition to the configuration according to any one of claims 1 to 4, the 3D template is a human head, the 2D image of the creature is a face image of a person, and the The corrected image is a 3D avatar generated using the face image of the person.

The invention according to claim 6 applies a 2D image having two-dimensional direction information to a 3D template as an image template of a 3D model as an image model having three-dimensional direction information to obtain a 3D image of a living organism. An image generation method in an image generation device, comprising: an image acquisition procedure for acquiring the recorded 3D template and the 2D image; A biological 3D model generation procedure for generating a biological 3D model as a 3D image, and a fitting procedure for generating a adapted image by fitting the biological 3D model to the 3D template using the position information of the biological 3D model. and, when a predetermined mismatch that causes a visual discomfort occurs in the matching image, the mismatch is corrected by changing the positional information in the three-dimensional direction to at least a part of the matching image. and an image correction procedure for generating the corrected image.

According to a seventh aspect of the present invention, there is provided a program for causing a computer to function as the image generating apparatus according to any one of the first to fifth aspects.

According to the inventions of claims 1 and 6, in generating a 3D image of a living organism, a 3D template is adapted to a 3D living organism model obtained by adding positional information in three-dimensional directions to a 2D image of the living organism. By generating an image, it is possible to generate a 3D image resembling a real creature that reflects the visual characteristics of the real creature. Further, when a predetermined inconsistency that causes visual discomfort occurs in the matching image, a corrected image in which the inconsistency is corrected by changing the positional information in the three-dimensional direction to at least a part of the matching image. By generating , it is possible to automatically suppress the visual discomfort that occurs when reflecting the visual characteristics of living things on a 3D template, and to generate a 3D image that more closely resembles a real living thing. As a result, a 3D character that gives a natural impression can be generated by suppressing the occurrence of visual differences and unnecessary configurations.

According to the second aspect of the invention, the process of correcting color discontinuity occurring on a continuous plane on a 3D image is appropriately performed, and visual differences due to color discontinuity and generation of unnecessary configurations are prevented. can be suppressed.

According to the third aspect of the invention, by performing processing for correcting unevenness in the Z direction that occurs at the boundary between the 3D template and the 3D biological model adapted to the 3D template, the computational processing of the computer, It is possible to realize a specific configuration that automatically suppresses the occurrence of visual differences and unnecessary configurations.

According to the fourth aspect of the present invention, the 3D organism model is corrected in the Z direction relative to the entire 3D template to which the 3D organism model is fitted, so that the 3D organism model and the In a specific configuration, it is possible to automatically suppress the occurrence of visual differences and unnecessary configurations that may occur at the boundary between the 3D template and the 3D template.

According to the fifth aspect of the invention, when generating a 3D avatar using a 2D image as a face image of a person in the 3D template of the human head, the occurrence of visual differences and unnecessary configurations is suppressed. can.

According to the invention of claim 7, by configuring the present invention as a program, the present invention can be implemented in various computer hardware, suppressing the occurrence of visual differences and unnecessary configurations, and naturally Impressive 3D characters can be generated.

1A and 1B are a system configuration diagram and a functional block diagram of an image generation system and an image generation device according to this embodiment; FIG. 4 is a flow chart showing a processing procedure of the image generating device according to this embodiment; 3A and 3B are diagrams schematically showing the 3D template of the image generating device according to this embodiment when (a) it is displayed from the front and (b) it is displayed from the right side; In this embodiment, the face image 3D model is a diagram schematically showing (a) a state displayed from the front and (b) a state displayed from the right side. In this embodiment, it is a diagram schematically showing a state in which the face image 3D model has undergone a predetermined correction, (a) displayed from the front, and (b) displayed from the right side. In this embodiment, it is a figure which showed typically the state corrected by applying the coordinate information of the face image 3D model to the 3D template. In this embodiment, it is a figure which showed typically the state which deform|transformed the 3D template according to the unevenness|corrugation of the face of the image 3D model. FIG. 4 is a diagram schematically showing a state in which a suitable image is generated from a 3D template in this embodiment; FIG. 10 is a diagram schematically showing a state in which polygon inversion has occurred in a portion of the matching image in this embodiment; In this embodiment, it is a diagram schematically showing a state in which the positions of the eyes of the 3D template and the positions of the eyes of the face image 3D model are transformed so as to match three-dimensionally. In this embodiment, (a) diagrams schematically showing the states before and after the coordinates of the front portion of the face of the 3D template are moved in the Z direction, (b) the inverted portion due to polygon inversion as a result of the movement is a diagram schematically showing a state in which is generated. FIG. 10 is a diagram schematically showing a state in which a reversed portion due to polygon reversal is eliminated in this embodiment; FIG. 4 is a diagram schematically showing (a) a state in which a reversed portion due to polygon inversion is generated in a suitable image, and (b) a state in which a 3D avatar is generated by correcting the reversed portion in this embodiment.

An embodiment of the present invention will be described below with reference to the drawings.

[overall structure]
FIG. 1 is a system configuration diagram and a functional block diagram of an image generation system and an image generation device according to this embodiment.

The image generation system _1A of this embodiment shown in _FIG . . . , 3 n (n≧1) used by 2 _m are configured as a client/server system in which _n terminals 3 ₁ , 3 ₂ , . . .

The details of the image generation device 1 will be described later. _{Users 2 1} , _{2 2} , _. to generate a 3D avatar. Users 2 ₁ , 2 ₂ , . . . 2 _m use the generated 3D avatars as characters in virtual communication spaces such as SNS, virtual malls, and network games formed on the network 4 .

One or a plurality of terminals 3 ₁ , 3 ₂ , . . . 3 _n are _possessed and used by users 2 ₁ , 2 ₂ , . _Terminals 3 ₁ , 3 ₂ , . . _{Terminals 3 1} , _{3 2} , . . . 3 _n are operating means shown in FIG. 33 and display means 32 such as an LCD for displaying characters and images shown in FIG.

In addition, the terminal 3-1 shown in FIG. ₁ is provided with a front-side camera 34 for the user _2-1 to take a so-called "self-portrait" of his or her face. This front-side camera 34 is used to capture a facial photograph 102a as _a 2D image of the user 21, which is used to generate a facial image 3D model 102, which will be described later.

The network 4 is a wired or wireless communication medium used for transmitting and receiving data and signals, such as the Internet, LAN, and intranet, and includes optical fiber cables, metal cables, wireless base stations, routers, switches, modems, and the like.

In addition, for the sake of _simplicity of _explanation , users _{2 1} , _{2 2} , . . . are referred to as terminal 3, respectively.

[Image generation device]
FIG. 1 shows a functional block diagram of an image generation device 1 of this embodiment.

The image generating apparatus 1 of this embodiment is a server computer, and includes a configuration such as a CPU for arithmetic processing, a configuration such as a RAM as a work area for arithmetic processing, a ROM, an EEPROM, a magnetic disk, etc. for storing data and programs. It has a configuration as a region and the like. The image generating apparatus 1 may be one computer device, or may be configured by connecting a plurality of computer devices. Further, the image generating apparatus 1 may be a server system or the like installed in a specific facility (for example, a server room, etc.), or may be configured as a cloud system using a plurality of hardware resources existing on the network 4. good too.

The image generating apparatus 1 of this embodiment transmits and receives data and signals to and from the terminal 3 via the network 4, and automatically generates a 3D avatar based on commands and data input from the user 2 via the terminal 3. It has the ability to generate The image generation device 1 displays on the terminal 3 a virtual facility for the user 3 to use the generated 3D avatar, such as a SNS, a virtual shopping mall, a virtual conference room, etc., and allows the user 2 to use it. may be provided.

The image generating apparatus 1 of this embodiment applies a 2D image having two-dimensional direction information to a 3D template as an image template of a 3D model as an image model having three-dimensional direction information to generate a 3D image of a living organism. Used to generate 3D avatars as images. This 3D avatar is a virtual character generated as a 3D image having depth information in addition to planar information. A 3D avatar moves as an animation in a 3D space (hereinafter simply referred to as "3D space") that is virtually displayed as a 2D image as a virtual space that also has 3D direction information.

In order to generate such a 3D avatar, as shown in FIG. 1, the image generation device 1 includes, as functional means, an image acquisition section 11 as "image acquisition means" and a face image 3D model as "organism 3D model generation means". It comprises a model generator 12 , a matching unit 13 as “matching means”, an image correction unit 14 as “image correction means”, and a database 15 .

The image acquisition unit 11 acquires the 3D template recorded in the database 15 and the 2D image of the user 2 transmitted from the terminal 3 . This 3D template will be described in detail later.

The facial image 3D model generation unit 12 generates a facial image 3D model as a 3D image by adding positional information in three-dimensional directions to a 2D image of the user 3 as a "living 3D model". The 3D model of the face image may be generated by, for example, converting a 2D image into a 3D image using a function using machine learning. This facial image 3D model will be described in detail later.

The matching unit 13 uses the position information of the 3D face image model to generate a matched image by matching the 3D creature model to the 3D template. This matching image will be described in detail later.

The image correcting unit 14 corrects the mismatch by changing the positional information in the three-dimensional direction to at least a part of the matching image when there is a predetermined mismatch that causes a visual discomfort in the matching image. Generate a corrected image. This modified image will be described in detail later.

The database 15 is recording means for recording data, programs, etc. used in the image generating apparatus 1 . For example, the database 15 records data of the 3D template described above, data of k (k≧1) 3D avatars 109 ₁ , 109 ₂ , . . . 109 _k generated. For the sake of simplicity of explanation, the 3D _avatars 109 ₁ , 109 ₂ , .

[Processing procedure]
FIG. 2 is a flow chart showing the processing procedure of the image generation device 1 of this embodiment. A procedure of processing for generating the 3D avatar 109 in the image generation device 1 will be described below based on the flowchart of FIG.

[Step S1: Image Acquisition Procedure]
Consider a case where a user 2 (for example, user 2 ₁ ) of a certain image generation system 1A generates a 3D avatar 109 resembling himself using his own facial photograph.

In this case, the user 2 accesses the image generation device 1 using a certain terminal 3 (for example, the terminal 3 ₁ ), and the user 2 himself (for example, the user 2 ₁ ) creates his own 3D avatar 109 on the image generation device 1. send an order requesting A request to send the 2D image of the user 2 to the image generating device 1 is transmitted from the image generating device 1 to the terminal 3, and the request is displayed on the touch panel display (not shown) of the terminal 3. .

Then, the user 2 (for example, the user 2 ₁ ) uses the terminal 3 (for example, the terminal 3 ₁ ) to transmit a 2D image such as a facial photograph 102a (see FIG. 1), which is a 2D image of his/her own face, to the image generating apparatus 1. Send.

The image acquisition unit 11 of the image generation device 1 acquires a 2D image such as the facial photograph 102a of the user 2 transmitted from the terminal 3 (step S1). Also, the image acquisition unit 11 acquires the 3D template 101 recorded in the database 15 .

3A and 3B schematically show the 3D template 101. FIG. 3A shows the 3D template 101 viewed from the front, and FIG. 3B shows the 3D template viewed from the right side. FIG. 10 is a diagram schematically showing a displayed state, respectively. This 3D template 101 is a 3D template that includes information for indicating the shape of the head of a human body, the color of the skin, and the positions, sizes, shapes, positions, etc. of facial parts such as eyes, nose, mouth, and ears. It has basic information for generating the avatar 109 .

This 3D template 101 is formed by a set of a large number of pixels, and each pixel corresponds to a three-dimensional coordinate axis (for example, horizontal direction (X direction), vertical direction) in the three-dimensional virtual space in which the 3D template 101 exists. (Y direction), height direction (Z direction)) for specifying a position in a virtual three-dimensional space based on three orthogonal axes) for specifying a position in a three-dimensional direction It has three-dimensional coordinates (eg, X-, Y-, and Z-coordinates). By changing the three-dimensional coordinate values, it is possible to change the position and size of the entire 3D template 101, or specific portions or parts of the 3D template 101 (eg, eyes, nose, mouth, etc.). Also, by changing color information (eg, RGB color information, luminance/color difference information, etc.) numerically set for each pixel, the color, brightness, etc. of each pixel can be changed.

[Step S2: Facial image 3D model generation procedure (organism 3D model generation procedure)]
Next, the facial image 3D model generation unit 12 generates a facial image 3D model as a 3D image including information on the three-dimensional direction based on the facial photograph 102a as a 2D image of the user 2 acquired in step S1. do.

The face image 3D model generation unit 12 generates a face image as a 3D image from the face image as a 2D image of the user 2 using, for example, a function using machine learning. Specifically, the data extraction/calculation function of the face image 3D model generation unit 12 extracts information on parts such as the eyes, eyebrows, nose, mouth, and ears from the face photograph 102a of the user 2 as a 2D image. Also, information such as gloss, shadow, color, etc. for each part, contour, and part of the face is acquired, and that information is quantified, and position information such as those parts and the contour of the face, and the size and depth of unevenness, etc. Calculate the information of Based on the calculation result, the face image 3D model generation unit 12 maps each pixel constituting the face image of the 2D image onto the three-dimensional space. Specifically, the numerical values of the three-dimensional coordinates (for example, X coordinate, Y coordinate, Z coordinate) for specifying the position in the three-dimensional space are input to each pixel information constituting the face image as a 2D image. do.

In addition, as a function constituting the face image 3D model generation unit 12, an artificial intelligence function is provided in order to convert a face image as a 2D image into a face image as a 3D image. In advance, correspondence relationships between 2D and 3D images of a large number of face images (how the three-dimensional shape of the face is reflected in the 2D image and the 3D image depending on the size, unevenness, skin color, etc. of each part of the face) etc.) is learned, and based on this learning result, a 2D facial image 102a of the user 2 is generated as a 3D facial image 3D model. For example, by learning about a 2D face image and actual data in the Z coordinate direction, various data including data in the Z coordinate direction are inferred from the 2D face image, and only the 2D image is analyzed using this inferred data. You can obtain the value of the three-dimensional coordinates by inputting However, the face image 3D model generation unit 12 does not have an artificial intelligence function, and converts the 2D face photograph of the user 2 into a 3D face image 3D image using well-known image analysis technology, image recognition technology, or the like. It may be generated as a model.

FIG. 4 shows a conceptual diagram of the face image 3D model 102 generated by the face image 3D model generation unit 12 of this embodiment. FIG. 4(a) schematically shows a state in which the face image 3D model 102 is displayed from the front, and FIG. 4(b) schematically shows a state in which the face image 3D model 102 is displayed from the right side. be. As shown in the figure, the face image 3D model 102 is based on a facial photograph image of a real person's face. It is generated as an image of a three-dimensional shape of the face part having direction (Z direction) information.

Furthermore, the facial image 3D model generation unit 12 performs various adjustments to the facial image 3D model 102 . Specifically, the facial image 3D model adjustment unit performs the following adjustment processes (A) to (D) on the facial image 3D model 102 .

(A) A process of rotating the face image 3D model 102 so that both eyes are horizontal is performed. For example, the facial image 3D model 102 is rotated in the direction indicated by [A2] in FIG. 5 so that the positions of both eyes of the facial image 3D model 102 are aligned with the horizontal position indicated by [A1] in FIG.

(B) A process of rotating the face image 3D model 102 so that the depths of both eyes are equal is performed. For example, the face image 3D model 102 is moved in the direction indicated by [B] in FIG. rotate.

(C) Rotate the face image 3D model 102 so that the chin and forehead do not protrude excessively forward. For example, in the face image 3D model 102 of (a) of FIG. 5, the face image 3D model 102 is rotated so that the Z direction position of [C1] and the Z direction position of [C2] match.

(D) The polygons at both ends of the face image 3D model 102 are truncated. This is because when processing using MP3D or the like is performed in the image generation device 1, the meshes of the sides of the face and the ears may not be generated neatly, so polygons are not reflected in the 3D avatar 109 as the final result. It is done to For example, as shown in [D] in (b) of FIG. 5, processing is performed to remove the polygons of the ear portion of the face image 3D model 102 (see the image of the ear portion in (b) of FIG. 4).

Since the face image 3D model 102 is formed based on data inferred by machine learning, it is possible to create a face image in which the horizontal position of both eyes is shifted, the depth of both eyes is shifted, or the chin or forehead protrudes extremely forward. A 3D model may be generated. Therefore, as described above, the eyes are rotated so that they are horizontal, the eyes are rotated so that the depths of both eyes are equal, and further, processing is performed so that the chin and forehead do not protrude excessively forward.

FIG. 5 shows a conceptual diagram of the face image 3D model 102 adjusted by performing the above processes (A) to (D). (a) of FIG. 5 is a diagram schematically showing a state in which the face image 3D model 102 is displayed from the front (b) is a state in which it is displayed from the right side. The face image 3D model 102 shown in the figure is shown in a state adjusted by the above-described rotation processing (A) to (D) and processing for truncating polygons.

[Step S3: Matching procedure]
After the process of step S2, the matching unit 13 uses the position information of the face image 3D model 102 generated by the face image 3D model generation unit 12 to generate a matched image obtained by fitting the face image 3D model 102 to the 3D template 101. Generate.

In this embodiment, the positional information of the face image 3D model 102 includes, for example, coordinate information on the virtual space of each pixel constituting the face image 3D model 102, eyes, eyebrows, and eyebrows included in the face image 3D model 102. Coordinate information in virtual space for specifying the size and position of parts such as nose, mouth, and ears corresponds to this. The adapting unit 13 applies the virtual space coordinate information of the face image 3D model 102 to the virtual space position information of the corresponding pixels of the 3D template 101, the corresponding facial parts, etc., thereby obtaining a 3D face image. The size and position of the face and parts of the model 102, the state of unevenness, shadows, and the like are applied. In addition, the matching unit 13 also applies information such as the degree of glossiness and color of the face image 3D model 102 to the 3D template 101 .

Specifically, the matching unit 13 first acquires the length L1 in the vertical direction, the length L2 in the horizontal direction, the length L3 in the height direction, etc. at a predetermined position (for example, the position of both eyes) of the 3D face image model 102 . Next, the matching unit 13 acquires the length L11 in the vertical direction, the length L12 in the horizontal direction, and the length L13 in the height direction of the corresponding position (for example, the position of both eyes) of the 3D template 101 . Then, the matching unit 13 sets the vertical length L11 of the 3D template 101 to the vertical length L1 of the face image 3D model 102, and sets the horizontal length L12 of the 3D template 101 to the horizontal length of the face image 3D model 102. By correcting the height direction length L13 of the 3D template 101 to the height direction length L3 of the face image 3D model 102, respectively, the size and shape of the 3D template 101 are changed to the size and shape of the face image 3D model 102. Transform into a size or shape that approximates the shape.

FIG. 6 is a diagram schematically showing a state in which the matching unit 13 of this embodiment corrects the 3D template 101 by applying the coordinate information of the face image 3D model 102. As shown in FIG. The 3D template 101 shown in FIG. 3 has a vertical length L11, a horizontal length L12, and a height L13 of the 3D template 101 shown in FIG. The size and shape of the face portion of the 3D template 101 are approximated to the size and shape of the face image 3D model 102 by correcting the height L2 and the length L3 in the height direction. As a result, a state is created in which the information of the face image 3D model 102 can be easily applied to the 3D template 101 .

Next, the matching unit 13 applies the information of the face image 3D model 102 to the modified 3D template 101 shown in FIG.

Specifically, first, as schematically shown in FIG. 7, the matching unit 13 transforms the unevenness of the face of the 3D template 101 to match the unevenness of the face of the 3D face image model 102 shown in FIG. Specifically, coordinate information in the vertical direction (X direction), horizontal direction (Y direction), and height direction (Z direction) in the virtual space of the pixels of the facial portion of the 3D template is acquired, and the acquired coordinates By applying the information to the corresponding pixels of the 3D template 101 and the coordinate information of the corresponding parts of the face, the unevenness of the face of the 3D template 101 is made close to the unevenness of the face image 3D model.

Specifically, the process of the adapting unit 13 is performed as processes shown in (process 01) to (process 03) below.

(Processing 01) Enlargement/reduction/parallel movement is performed on the face image 3D model 102 of FIG. 4 (or FIG. 5) so that the 3D template 101 shown in FIG. , for a conceptual diagram of a state in which the 3D template 101 and the face image 3D model 102 are superimposed, see FIG.

(Processing 02) Considering a straight line parallel to the Z-axis (horizontal direction in the case of FIGS. 7 and 10) passing through one vertex of the 3D template 101, the intersection with the polygon of the face image 3D model 102 is Explore. When the intersection point is obtained, the three-dimensional coordinates at the intersection point are obtained. This processing is performed for the vertices in the front half of the 3D template 101 .

(Processing 03) Update the vertex coordinates of the corresponding 3D template 101 with the three-dimensional coordinates of the intersection point obtained in (Processing 02).

FIG. 7 schematically shows a 3D template 101 in which the face portion is deformed according to the unevenness of the face image 3D model 102 . In this state, as shown in FIG. 7(b), the shape of the 3D template 101 viewed from the side has changed from the state shown in FIG. 6(b). On the other hand, in this state, the shape of the 3D template 101 seen from the front shown in FIG. 7(a) does not change from the state before correction shown in FIG. 6(a).

The matching unit 13 then updates the texture and texture coordinates of the face of the 3D template 101 . Specifically, the matching unit 13 assigns coordinate information to match the positions and sizes of parts such as the eyes, eyebrows, nose, mouth, and ears of the face image 3D model 102 with the face part of the 3D template 101 . fix it. The matching unit 13 also applies color information, brightness information, gloss information, and the like for each pixel of the face image 3D model 102 or for each part to each pixel of the 3D template 101 . Thereby, a 3D template 101 is generated as a conforming image.

FIG. 8 schematically shows a state in which the matching image 103 is generated from the 3D template 101 by the processing of the matching unit 13. As shown in FIG. As shown in the figure, the face portion of the matching image 103 includes the positions, sizes, shapes, and unevenness of the eyes, eyebrows, nose, mouth, and ears of the face portion of the face image 3D model 102, as well as the skin portion. It is in a state where the state of color, shading, and gloss, including , is applied. That is, the matching image 103 is generated in almost the same state as the completed 3D avatar 109 .

[Step S4: Image Correction Procedure]
Here, the suitable image 103 generated by the processing up to step S3 may undergo so-called polygon inversion or the like. The image correction unit 14 corrects the visual discomfort of the matching image 103 caused by the polygon inversion or the like.

FIG. 9 schematically shows a state in which polygon inversion has occurred in part of the matching image 103 generated by the processing up to step S3. In the matching image 103 shown in FIG. 9, an inverted portion 104 is formed in the vicinity of the right cheek portion indicated by the arrow, in which a part of the polygon protrudes in the front direction.

The generation of the inverted portion 104 by polygon inversion is the relative arrangement of the vertices of the 3D template 101 and the corresponding points on the face image 3D model 102 in the above (Processing 02) and (Processing 03). caused by

For example, as schematically shown in FIG. 10, consider a case where the positions of the eyes 105 of the 3D template 101 and the positions of the eyes 106 of the face image 3D model 102 are three-dimensionally matched.

In this case, as schematically shown in FIG. 11A, by matching the positions of the eyes 105 of the 3D template 101 with the positions of the eyes 106 of the face image 3D model 102, the face of the 3D template 101 is The coordinates of the front portion move downward (downward in FIGS. 11A and 11B) in the Z direction (vertical direction in FIGS. 11A and 11B). As a result, as shown in FIG. 11B, the area 107 where the 3D template 101 is moved in the Z direction (that is, as a result of updating the vertex coordinates in the above (processing 02) and (processing 03), The specific region 107a is the region 107 that has been moved in the Z direction. ), an inverted portion 104 due to polygon inversion is generated as a projecting shape upward in the Z direction (upward in FIGS. 11A and 11B).

In order to remove such a reversed portion 104 of the matching image 103, the image correction unit 14 of this embodiment performs the following processing. This processing is schematically shown in FIG.

Specifically, as schematically shown in FIG. 12, the image correcting unit 14 modifies a group of vertices such as pixels in the area 108 whose vertex coordinates have not been updated in the above (processing 02) and (processing 03) of the suitable image 103. is moved relatively downward in the Z direction (downward in FIG. 12). In FIG. 12, arrows schematically indicate the moving direction of the portion in which the group of vertices of the region 107 is moved.

At this time, how much each vertex of the pixels belonging to the region 108 is translated in the Z direction depends on the region 107 (FIG. 11 ) is calculated for each vertex based on the spatial distribution of the amount of change (see the arrow in FIG. 11) of the vertex group.

Through such processing by the image correction unit 14, the reversed portion 104 of the matching image 103 is removed and a 3D avatar 109 is generated.

FIG. 13(a) schematically shows a suitable image 103 in which a reversed portion 104 is generated, and FIG. 109 are diagrams schematically showing the state where 109 is generated. As shown in FIG. 13(a), the reversed portion 104, which is a projection-like projection (rightward in the drawing) on the cheek due to the polygon reversal, converts the vertices of region 107 to the vertices of region 108. 13A and 13B, it is removed as shown in FIG. 13B. 13(a) where there was a reversed portion 104 is a smoothed image with a natural impression, like the original reversed portion 110 shown in FIG. 13(b).

In the above example, the reversed portion 104 is formed as a projecting shape, but the reversed portion has a difference in color, shade, etc. between the regions 107 and 108 (regardless of the presence or absence of a shape such as a projection). It may also be formed as In this case, the image correction unit 14 generates the 3D avatar 109 as an image with a natural impression by performing processing such as matching the color of the area 107 to the color, shadow, etc. of the area 108 .

The 3D avatar 109 shown in (b) of FIG. 13 is generated with only the head and no hair. , the avatar 109 can also be generated by adding body parts other than the head (for example, the upper body or the whole body). Further, when the inverted portion 104 does not occur in the matching image 103, the process of step S4 can be omitted and the matching image 103 can be used as the avatar 109 as it is.

[Step S5: Generate 3D avatar]
In this manner, the image generation device 1 of this embodiment generates the 3D avatar 109 of the user 2's face. The generated 3D avatar 109 is recorded in the database 15 or transmitted to a terminal (for example, terminal 3 ₁ ) used by the user (for example, user 2 ₁ ) who requested the generation, and is sent to the user (for example, user 2 ₁ ) can be used for communication, games, etc. in the virtual space.

[Effect]
As described above, in this embodiment, when generating the 3D avatar 109 of the user 2, the face image 3D model 102 obtained by adding position information in the three-dimensional direction to the 2D image of the user 2 is adapted to the 3D template 101. By generating the matching image 103 that is similar to the real user 2 , a 3D avatar 109 that resembles the real user 2 and that reflects the visual characteristics of the real user 2 can be generated. In addition, when mismatching of the inverted portion 104 that causes a visual discomfort occurs in the matching image 103, the position information in the Z direction is changed in the region 107, which is at least a part of the matching image 103, to eliminate the mismatch. By generating the modified 3D avatar 109, the visual discomfort that occurs when the visual features of the user 2 are reflected in the 3D template 101 is automatically suppressed, and the real user 2 is better. A similar 3D avatar 109 can be generated. As a result, the 3D avatar 109 that gives a natural impression can be generated by suppressing the occurrence of visual differences and unnecessary configurations.

In this embodiment, processing is appropriately performed to correct color discontinuity between the regions 107 and 108 that occurs on a continuous surface on the matching image 103, and visual differences and unnecessary effects due to color discontinuity are corrected. The occurrence of configuration can be suppressed.

In this embodiment, by performing a process of correcting unevenness in the Z direction in the reversed portion 104 generated at the boundary portion between the 3D template 101 and the face image 3D model 102 adapted to the 3D template 101, the computer can Arithmetic processing can be implemented in a specific configuration to automatically suppress the occurrence of visual differences and unnecessary configurations.

In this embodiment, the 3D template 101 is subjected to processing for correcting the relative Z-direction position of the 3D face image 102 to which the face image 3D model 102 is fitted and the 3D template 101 as a whole. It is possible to implement a specific configuration to automatically prevent visual differences that may occur at the boundary between the face image 3D model 102 and the 3D template 101 and unnecessary configurations.

In this embodiment, when generating a 3D avatar 109 using a 2D image as the face image of the user 2 in the 3D template 101 of the human head, it is possible to suppress the occurrence of visual differences and unnecessary configurations. can.

[remarks]
In this embodiment, a 2D image is input to an inference device included in the trained image 3D model generation unit 12 to obtain a 3D image, and a 3D template is deformed based on the data of the 3D image. An example of applying 3D face image data to a 3D template has been explained. may be generated.

A single 3D template 101 may be used, or a plurality of templates may be prepared for each facial feature, such as each race and gender, and an optimum 3D template may be selected according to the acquired 2D image. 101 may be selected and used. Furthermore, when preparing multiple 3D templates, for example, facial images of Asians and Westerners are learned by machine learning, etc., and facial features (nose height, eyebrows and eyeballs) according to race, gender, and age , etc.) to select an optimum 3D template, or transform the 3D template or face image 3D model so as to be optimum.

In this embodiment, the 3D avatar 109 generated by the image generation device 1 is generated as the face of the user 2, but is not limited to this. , whole upper body, whole body, etc.), and it is also possible to generate a 3D avatar of a person other than the user 2 .

Further, in this embodiment, the image generation device 1 can generate 3D avatars of creatures other than humans, such as pets such as dogs and cats, wild animals, insects, and plants. For example, it is possible to generate 3D avatars of transportation facilities such as automobiles and trains, and buildings.

Further, the image generation apparatus 1 of this embodiment is configured as a server system of a client-server system, and all the functional means exist on the server side. At least part of the means may be configured to exist on the terminal 3 side. Also, the image generating apparatus 1 can be configured in a stand-alone computer that is not connected to the network 4. FIG.

It goes without saying that the above embodiments are examples of the present invention, and the present invention is not limited only to the above embodiments.

1... Image generation device 2, 2 ₁ , 2 ₂ ,... 2 _m ... User (living creature)
11 image acquisition unit (image acquisition means)
12 Face image 3D model generation unit (biological 3D model generation means)
13 ... fitting part (fitting means)
14 image correction unit (image generation means)
101: 3D template 102: 3D face image model (3D biological model)
103 Matching image 104 Inverted portion (predetermined mismatch)
109, 109 ₁ , 109 ₂ , ... 109 _k ... 3D avatar (3D image of creature, modified image, 3D character)

Claims (7)

An image generation device for generating a 3D image of a living organism by applying a 2D image having two-dimensional direction information to a 3D template as an image template of a 3D model as an image model having three-dimensional direction information,
an image acquisition means for acquiring the recorded 3D template and the 2D image;
3D organism model generating means for generating a 3D organism model as a 3D image by adding positional information in three-dimensional directions to the 2D image of the organism;
fitting means for generating a fitted image by fitting the 3D biological model to the 3D template using the position information of the 3D biological model;
When a predetermined mismatch that causes a visual discomfort occurs in the matching image, the mismatch is corrected by changing the positional information in the three-dimensional direction to at least a part of the matching image. and image correction means for generating a corrected image. The image correction means performs a process of correcting the color discontinuity by correcting the color discontinuity at a portion of the corrected image where the color discontinuity occurs. Item 1. The image generation device according to item 1. 3. The image correcting means performs a process of correcting unevenness in the Z direction occurring at a boundary portion between the 3D template and the 3D biological model adapted to the 3D template in the matched image. 3. The image generation device according to 1 or 2. 4. The apparatus according to claim 3, wherein said image correction means corrects a position of said 3D organism model relative to said 3D template to which said 3D organism model is adapted in said Z direction. An image generating device as described. The 3D template is a human head,
the 2D image of the organism is a face image of a person;
5. The image generating apparatus according to claim 1, wherein the modified image is a 3D avatar generated using the facial image of the person. Image generation in an image generation device for generating a 3D image of a living organism by applying a 2D image having two-dimensional direction information to a 3D template as an image template of a 3D model as an image model having three-dimensional direction information a method,
an image acquisition procedure in which the recorded 3D template and the 2D image are acquired;
a biological 3D model generation procedure for generating a biological 3D model as a 3D image by adding positional information in three-dimensional directions to the biological 2D image;
a fitting step in which a fitted image is generated by fitting the 3D biological model to the 3D template using the position information of the 3D biological model;
When a predetermined mismatch that causes a visual discomfort occurs in the matching image, the mismatch is corrected by changing the positional information in the three-dimensional direction to at least a part of the matching image. and an image modification procedure for generating a modified image. A program that causes a computer to function as the image generating apparatus according to any one of claims 1 to 5.

Images