JP7495818B2 - Three-dimensional image processing device and program - Google Patents

Description

The present invention relates to a three-dimensional image processing device and a program for processing a three-dimensional model that represents a three-dimensional image.

Conventional methods for expressing three-dimensional images include a method using multi-view images that use images from multiple different viewpoints, a method using images from one or more viewpoints and a distance image (depth map) from one or more viewpoints, and a method using a combination of three-dimensional shape data and its texture information (hereinafter, the above combinations are referred to as "three-dimensional models").

Three-dimensional shape data can be expressed using primitive models that combine typical geometric shapes such as ellipsoids, polyhedrons, cylinders, cones, and tori; volume models such as voxel data; surface models such as triangular mesh data; point clouds and other point group models; metaball models that combine spherical density distributions in three-dimensional space and define the iso-density curved surface as the surface shape; depth maps that use distance images from a certain viewpoint; elevation maps (DEM: Digital Elevation Maps) that define height values for each point on a two-dimensional plane; or combinations of multiple of these models.

The texture information used may be any one of the following: surface color, transparent color, texture (texture map), surface reflectance characteristics, normal distribution on the surface (bump map), surface roughness, refractive index, transmittance, internal diffusion, and polarization characteristics, or a combination of these.

Three-dimensional models are generated by artificially creating numerical data, or by measuring the shape of real objects using sensors such as image or distance sensors, spotlights, polarizing filters, color filters, spectrophotometers, and computer tomography, or other methods.

For the transmission or storage of 3D images, the 3D model is formatted in an appropriate format and, as necessary, source coding, channel coding, or both are applied (hereinafter referred to as the "encoded 3D model").

In addition, a method of encoding images from different viewpoints using a multi-viewpoint image encoding method may be used. In this case, prediction is performed between images from different viewpoints, and encoding is performed using the encoding method or a similar process. In some cases, a depth map is used in combination with this method to perform encoding.

To view a transmitted or stored encoded 3D model as a 2D image from a desired viewpoint, channel decoding and source decoding are applied to the encoded 3D model as necessary to generate a 3D model, and then the 3D model is rendered from the desired viewpoint. This allows a 2D image from the desired viewpoint to be obtained.

Image (including video) coding methods include a combination of transform, quantization of transform coefficients, entropy coding, and prediction as necessary. Examples include JPEG and JPEG2000 for still images, and MPEG-1, MPEG-2, H.263, MPEG-4, MPEG-4 Part 10 AVC|H.264, MPEG-H HEVC|H.265, MPEG-I VVC, VC-1, VC-2 (Dirac), VC-3, VC-5, VP3, VP6, VP7, VP9, etc. for video.

The various encoding methods mentioned above have the property that the lower the bit rate (the higher the compression rate), the greater the degradation of the image quality of the decoded image. In order to suppress image degradation at low bit rates, there are techniques that provide image restoration means on the receiver side to improve image quality. Known techniques for improving image quality include noise reduction, edge enhancement, and super-resolution image restoration.

For example, a method has been proposed in which the transmitter uses a method that degrades image quality, such as image reduction, in addition to encoding, and the receiver applies super-resolution image restoration to the decoded image (see, for example, Patent Document 1).

In addition, a method has been proposed in which the transmitter side calculates the optimal image restoration method and parameters by referring to either the high-quality image before encoding or a high-quality image generated using an image restoration means with higher performance than the image restoration means on the receiver side, and transmits the optimal method or parameters as auxiliary information from the transmitter side to the receiver side, and the receiver side performs image restoration using the optimal method or parameters (see, for example, Patent Documents 1 and 2).

Patent No. 5419795 Patent No. 6614935

Here, we consider a system in which the transmitter measures the shape of a real object (hereafter referred to as the "subject"), generates a three-dimensional model and transmits it, and the receiver uses the received three-dimensional model to view the subject from a desired viewpoint.

In such systems, when measuring the shape of a subject and generating a 3D model, errors, omissions, noise, etc. may occur in the 3D model.

For example, when using stereo matching with multiple camera images for measurement, subjects with little pattern (or low contrast) may experience erroneous correspondence in the stereo matching, resulting in shape errors.

In addition, especially when there are multiple subjects or when the subject has a concave portion, occlusion can occur between subjects (or between parts of the same subject), which can prevent correct correspondence during stereo matching and can result in holes in the shape.

In addition, if the subject has a confusing pattern, or if there is an object with a confusing pattern nearby, false correspondences may occur during stereo matching, resulting in a breakdown in the shape.

In addition, when using an infrared time-of-flight (TOF) distance sensor for measurement, if the subject has low reflectance to infrared light, the measurement may fail, resulting in errors or holes in the shape.

When a 3D model that has such errors, holes, omissions, noise, or failures (hereinafter referred to as "errors, etc.") is transmitted, the errors, etc. in the 3D model will appear as degradation in the 2D image when viewed. In particular, when the subject is viewed from a viewpoint different from that used for measurement, this degradation tends to be noticeable.

It is preferable to correct errors in 3D models in advance by automatic or manual processing, but in either case, correcting the model to a sufficient quality requires huge computational costs or manual work. For this reason, 3D models cannot be used for live video transmission.

On the other hand, in the technology of Patent Document 1 or Patent Document 2 mentioned above, the degradation that occurs during image encoding and decoding can be corrected by an image restoration means controlled by auxiliary information, making it possible to obtain a decoded image with high image quality.

However, because this technology targets flat (two-dimensional) video images, it cannot be applied directly to processes that correct errors in three-dimensional models. In addition, because this technology corrects degradation caused by video encoding, it cannot be applied to processes that correct degradation of two-dimensional images caused by errors in three-dimensional models.

The present invention has been made to solve the above problems, and its purpose is to provide a 3D image processing device and program that can correct the degradation of the 2D image obtained by rendering when a 3D model is rendered from a desired viewpoint in a system that transmits or stores 3D images.

To solve the above problem, the three-dimensional image processing device of claim 1 is a three-dimensional image processing device that renders a three-dimensional model that represents a three-dimensional image and generates a two-dimensional image, and is characterized by comprising: a rendering unit that uses preset rendering parameters to render the three-dimensional model and generate the two-dimensional image; and a two-dimensional image restoration unit that uses preset restoration parameters to perform restoration processing to restore degradation of the two-dimensional image generated by the rendering unit and generate a restored two-dimensional image.

According to the 3D image processing device of claim 1, even if there is an error in the 3D model, a restored 2D image in which the error has been corrected can be generated.

The three-dimensional image processing device of claim 2 is the three-dimensional image processing device of claim 1, characterized in that the two-dimensional image restoration unit performs the restoration process on the two-dimensional image using the restoration parameters generated by an external device that outputs the three-dimensional model, and generates the restored two-dimensional image.

According to the three-dimensional image processing device of claim 2, even if there is an error in the three-dimensional model, a restored two-dimensional image in which the error has been corrected can be generated using restoration parameters transmitted from or stored in an external device.

The three-dimensional image processing device of claim 3 is the three-dimensional image processing device of claim 2, further comprising a decoding unit, wherein the decoding unit, when an encoded three-dimensional model is generated by encoding the three-dimensional model by the external device, decodes the encoded three-dimensional model to generate a decoded three-dimensional model, and when an encoded restoration parameter is generated by encoding the restoration parameter by the external device, decodes the encoded restoration parameter to generate a decoded restoration parameter, the rendering unit uses a preset rendering parameter to render the three-dimensional model or the decoded three-dimensional model generated by the decoding unit to generate the two-dimensional image, and the two-dimensional image restoration unit performs the restoration process on the two-dimensional image using the restoration parameter or the decoded restoration parameter generated by the decoding unit to generate the restored two-dimensional image.

According to the three-dimensional image processing device of claim 3, an encoded three-dimensional model and/or encoded restoration parameters transmitted or stored from an external device are decoded, and even if there is an error in the three-dimensional model, a two-dimensional image in which the error has been corrected can be generated. In addition, if the encoded three-dimensional model has been generated by lossy encoding, a two-dimensional image in which the degradation of the three-dimensional model caused by the lossy encoding has also been corrected can be generated.

The three-dimensional image processing device of claim 4 is characterized in that, in the three-dimensional image processing device of claim 3, the two-dimensional image restoration unit performs the restoration process on the two-dimensional image using the restoration parameters previously input or the decoding restoration parameters previously generated by the decoding unit to generate the restored two-dimensional image if an input error occurs when the three-dimensional image processing device inputs the restoration parameters or the encoding restoration parameters, and performs the restoration process on the two-dimensional image using the decoding restoration parameters previously generated by the decoding unit to generate the restored two-dimensional image if a decoding error occurs when the decoding unit decodes the encoding restoration parameters.

According to the three-dimensional image processing device of claim 4, when the restoration parameters or the coding restoration parameters are lost due to a transmission or storage malfunction, a two-dimensional image can be generated using the previous restoration parameters or the coding restoration parameters, and the resistance to the transmission or storage malfunction can be improved. In particular, when the error resistance of the code related to the restoration parameters is set lower than the error resistance of the code related to the three-dimensional model, the probability of inputting the three-dimensional model can be improved while suppressing the transmission or storage capacity, and the transmission band (or storage capacity) can be saved.

Furthermore, the three-dimensional image processing device of claim 5 is a three-dimensional image processing device that generates restoration parameters for restoring deterioration of a two-dimensional image generated by rendering a three-dimensional model that represents a three-dimensional image, and includes a rendering unit that inputs the three-dimensional model and uses preset rendering parameters to render the three-dimensional model to generate the two-dimensional image, and a restoration parameter generation unit that performs restoration processing to restore deterioration of the two-dimensional image generated by the rendering unit using provisional restoration parameters, generates a restored two-dimensional image, updates the provisional restoration parameters so that an error between the restored two-dimensional image and a predetermined learning image is reduced, and generates the updated provisional restoration parameters as the restoration parameters when a predetermined termination condition is satisfied, and outputs the three-dimensional model and the restoration parameters generated by the restoration parameter generation unit.

According to the three-dimensional image processing device of claim 5, when an error or the like exists in a three-dimensional model and the two-dimensional image that is the rendering result is deteriorated, restoration parameters can be generated to correct the deterioration after the fact on the decoding side.

The three-dimensional image processing device of claim 6 is the three-dimensional image processing device of claim 5, further comprising a three-dimensional model coding unit that inputs the three-dimensional model and applies source coding, channel coding, or the source coding and the channel coding to the three-dimensional model to generate an encoded three-dimensional model, and a restoration parameter coding unit that applies source coding, channel coding, or the source coding and the channel coding to the restoration parameters generated by the restoration parameter generating unit to generate encoded restoration parameters, and is characterized in that it outputs the encoded three-dimensional model generated by the three-dimensional model coding unit and the encoded restoration parameters generated by the restoration parameter coding unit.

According to the three-dimensional image processing device of claim 6, by compressing information, it is possible to save transmission capacity (or storage capacity), and by adding redundant information to information, it is possible to correct transmission errors (or storage errors).

The three-dimensional image processing device of claim 7 is the three-dimensional image processing device of claim 6, further comprising a three-dimensional model decoding unit that decodes the encoded three-dimensional model generated by the three-dimensional model encoding unit and generates a decoded three-dimensional model, and the rendering unit renders the decoded three-dimensional model generated by the three-dimensional model decoding unit using preset rendering parameters, and generates the two-dimensional image.

According to the three-dimensional image processing device of claim 7, when there is an error in the three-dimensional model and further degradation of the three-dimensional model due to encoding, resulting in degradation of the two-dimensional image that is the rendering result, restoration parameters can be generated to correct the degradation afterwards on the decoding side.

The three-dimensional image processing device of claim 8 is characterized in that, in the three-dimensional image processing device of claim 5, the frequency of outputting the restoration parameters is set lower than the frequency of inputting the three-dimensional model.

According to the three-dimensional image processing device of claim 8, the amount of data of the output restoration parameters is reduced, so that the bit rate or recording capacity required for transmitting or storing the three-dimensional model and the restoration parameters can be reduced.

The 3D image processing device of claim 9 is characterized in that, in the 3D image processing device of claim 6 or 7, the frequency of outputting the encoding and restoration parameters is set lower than the frequency of inputting the 3D model.

According to the three-dimensional image processing device of claim 9, the amount of data of the encoded and restored parameters is reduced, so that the bit rate or recording capacity required for transmitting or storing the encoded three-dimensional model and the encoded and restored parameters can be reduced.

The three-dimensional image processing device of claim 10 is a three-dimensional image processing device according to any one of claims 5 to 9, characterized in that the restoration parameter generation unit inputs the two-dimensional image generated by the rendering unit and the learning image, performs the restoration process to restore the degradation of the two-dimensional image using the provisional restoration parameters, generates a first restored two-dimensional image, performs the degradation process to degrade the first restored two-dimensional image using the degradation parameters, generates a first degraded two-dimensional image, performs the degradation process to degrade the learning image using the degradation parameters, generates a second degraded two-dimensional image, performs the restoration process to restore the degradation of the second degraded two-dimensional image using the provisional restoration parameters, generates a second restored two-dimensional image, updates the provisional restoration parameters and the degradation parameters so that the error between the two-dimensional image and the first degraded two-dimensional image and the error between the learning image and the second restored two-dimensional image are reduced, and generates the updated provisional restoration parameters as the restoration parameters when a predetermined termination condition is satisfied.

According to the three-dimensional image processing device of claim 10, even if the input three-dimensional model and the input learning image are not necessarily acquired from the same subject or under the same conditions, restoration parameters for correcting degradation due to errors in the three-dimensional model can be effectively learned.

Furthermore, the program of claim 11 is characterized in that it causes a computer to function as the three-dimensional image processing device of any one of claims 1 to 4.

The program of claim 12 causes a computer to function as a three-dimensional image processing device according to any one of claims 5 to 10.

As described above, according to the present invention, in a system for transmitting or storing three-dimensional images, when a three-dimensional model is rendered from a desired viewpoint, degradation of the two-dimensional image obtained by rendering can be corrected.

FIG. 2 is a block diagram showing an example of a configuration of a decoding-side 3D image processing device according to the first embodiment. 11 is a flowchart illustrating an example of a process in the decoding-side 3D image processing device according to the first embodiment. FIG. 11 is a block diagram showing an example of a configuration of a decoding-side 3D image processing device according to a second embodiment. FIG. 2 is a block diagram showing an example of a configuration of a three-dimensional image processing device on the encoding side according to the first embodiment. FIG. 11 is a block diagram showing an example of the configuration of a three-dimensional image processing device on the encoding side of Example 2-1. 11 is a flowchart showing an example of processing in a three-dimensional image processing device on the encoding side in Example 2-1. FIG. 11 is a block diagram showing an example of the configuration of a three-dimensional image processing device on the encoding side of Example 2-2. FIG. 11 is a block diagram showing an example of the configuration of a three-dimensional image processing device on the encoding side according to Example 2-3. FIG. 11 is a block diagram showing an example of the configuration of a restoration parameter generating unit in Example 2-3. 13 is a flowchart illustrating an example of a process of a restoration parameter generating unit in the embodiment 2-3.

The following describes in detail the embodiments of the present invention with reference to the drawings. In a system for transmitting or storing three-dimensional images, the present invention performs processing on the two-dimensional image after rendering, rather than directly removing errors from the three-dimensional model, in order to suppress degradation of the two-dimensional image obtained by rendering the three-dimensional model.

[3D image processing device on the decoding side: Example 1]
First, a description will be given of a 3D image processing device (3D image decoding device) on the decoding side of Example 1. The 3D image processing device on the decoding side of Example 1 is an example in which a 2D image generated by rendering a 3D model is subjected to a restoration process using preset (pre-adjusted) restoration parameters to generate a restored 2D image.

FIG. 1 is a block diagram showing an example of the configuration of a 3D image processing device on the decoding side in the first embodiment, and FIG. 2 is a flowchart showing an example of the processing in the 3D image processing device on the decoding side in the first embodiment.

The 3D image processing device 1-1 either directly inputs an encoded 3D model, which is a 3D model that has been encoded, from a corresponding encoding 3D image processing device (external device) or reads it from a memory (not shown). The 3D image processing device 1-1 also inputs parameters for rendering (hereinafter referred to as "rendering parameters"). The 3D image processing device 1-1 then decodes the encoded 3D model and uses the rendering parameters to reconstruct and output a 2D image from the 3D model.

The rendering parameters include, for example, one or more of the parameters indicating the viewpoint position, attitude angle, angle of view, projection method, exposure, and lighting (type, arrangement, and attitude) of the virtual camera. The same applies to the three-dimensional image processing device 1-2 of the second embodiment shown in FIG. 3, which will be described later.

This 3D image processing device 1-1 includes a 3D model decoding unit 11, a rendering unit 12, a restoration parameter storage unit 13, and a 2D image restoration unit 14.

The 3D image processing device 1-1 receives the encoded 3D model and rendering parameters (step S201). The 3D model decoding unit 11 receives the encoded 3D model, decodes the encoded 3D model to generate a decoded 3D model (step S202), and outputs the decoded 3D model to the rendering unit 12.

The 3D model decoding unit 11 pairs with the 3D model encoding unit 31 shown in Figures 4, 5, 7, or 8 described below, and performs inverse processing or approximately inverse processing within a specified error range.

The encoding method used by the 3D model decoding unit 11 and the 3D model encoding unit 31 described later may be any method. For example, a lossless compression method or a lossy compression method may be used as the encoding method.

As a lossless compression method, a source coding method such as run-length coding, differential pulse code modulation (DPCM), Huffman coding, arithmetic coding, or a combination of these can be used. Also, for example, if a three-dimensional model is represented as a two-dimensional map, such as a depth image or an altitude map, it may be regarded as an image and coded using a source coding method based on an image coding method such as JPEG or JPEG2000. Also, if the three-dimensional model changes from moment to moment, it may be coded using a source coding method based on a video coding method such as MPEG-1, MPEG-2, MPEG-4, MPEG-4 Part 10 AVC | H.264, MPEG-H HEVC | H.265, MPEG-I VVC, etc.

The coding method may be channel coding such as error detection coding, error correction coding, or encryption coding, or may be a combination of source coding and channel coding.

By using source coding, channel coding, or both as the coding method, it is possible to compress information and save transmission capacity (storage capacity). In addition, by adding redundant information to information, it is possible to correct transmission errors (storage errors).

The rendering unit 12 inputs the rendering parameters set by the user and also inputs the decoded 3D model from the 3D model decoding unit 11. The rendering unit 12 generates a 2D image in computer graphics (CG) by rendering the decoded 3D model using the rendering parameters (step S203). The rendering unit 12 then outputs the 2D image to the 2D image restoration unit 14.

The restoration parameter storage unit 13 holds parameters (hereinafter referred to as "restoration parameters") that define the operation of the two-dimensional image restoration unit 14. The restoration parameters are set in advance and stored in the restoration parameter storage unit 13. The restoration parameters are, for example, the connection weight coefficient value and bias value of the neural network.

The two-dimensional image restoration unit 14 inputs the two-dimensional image from the rendering unit 12, reads out the restoration parameters from the restoration parameter storage unit 13, and performs restoration processing on the two-dimensional image using the restoration parameters to restore the degradation of the two-dimensional image, thereby generating a restored two-dimensional image (step S204). The two-dimensional image restoration unit 14 then outputs the restored two-dimensional image (step S205).

The two-dimensional image restoration unit 14 can be configured, for example, by a neural network. In this case, the restoration parameters stored in the restoration parameter storage unit 13 are the connection weight coefficient values (and bias values, if necessary) set for each element of the neural network.

For example, the specific values of the restoration parameters are adjusted in advance so that degradation, loss, etc. caused by errors in the 3D model can be corrected or complemented in 2D images that have such degradation, loss, etc.

When the two-dimensional image restoration unit 14 is configured with a neural network, a convolutional neural network can be used as part or all of the network configuration. Such a configuration enables image restoration that takes into account local features of the image, making it possible to faithfully restore the contour shape of the subject, etc. In addition, because the network connections are local, it is possible to reduce the computational costs and hardware scale compared to a fully connected network.

When the two-dimensional image restoration unit 14 is configured with a neural network, an autoencoder can be used as the structure. With such a configuration, it is possible to extract essential information contained in the image and reconstruct the image from the essential information, thereby enabling image restoration with only noise or degradation removed.

In addition, when the two-dimensional image restoration unit 14 is configured with a neural network, a generative adversarial network (GAN) such as CycleGAN can be used as the structure. This can increase the realism of the restored image.

As described above, according to the decoding-side 3D image processing device 1-1 of the first embodiment, the rendering unit 12 uses the rendering parameters to render the decoded 3D model obtained by decoding the encoded 3D model, thereby generating a 2D image.

The two-dimensional image restoration unit 14 performs restoration processing on the two-dimensional image using preset restoration parameters to generate a restored two-dimensional image.

As a result, even if the 2D image obtained by rendering is degraded due to errors in the 3D model and degradation due to encoding, a restored 2D image can be obtained in which the degradation has been corrected. Therefore, in a system for transmitting or storing 3D images, when a 3D model is rendered from a desired viewpoint, degradation of the 2D image obtained by rendering can be corrected.

Note that the 3D image processing device 1-1 on the decoding side in Example 1 shown in FIG. 1 includes a 3D model decoding unit 11, but may not include the 3D model decoding unit 11. In this case, the 3D image processing device 1-1 includes a rendering unit 12, a restoration parameter storage unit 13, and a 2D image restoration unit 14, and directly inputs an unencoded 3D model. In other words, the rendering unit 12 renders the input 3D model using rendering parameters.

As a result, even if the 2D image obtained by rendering the 3D model is degraded due to errors in the 3D model, a restored 2D image can be obtained with the degradation corrected.

In addition, if the decoding-side 3D image processing device 1-1 of the first embodiment shown in FIG. 1 does not include the 3D model decoding unit 11, the corresponding encoding-side 3D image processing device does not need to encode the 3D model. In this case, the encoding-side 3D image processing device outputs the 3D model as is without encoding it.

[3D image processing device on the decoding side: Example 2]
Next, a 3D image processing device (3D image decoding device) on the decoding side of the second embodiment will be described. The 3D image processing device on the decoding side of the second embodiment generates a 2D image by rendering a 3D model. The 3D image processing device performs a restoration process on the 2D image using restoration parameters generated and output by the 3D image processing devices 3-2a, 3-2b, and 3-2c on the encoding side shown in FIG. 5, FIG. 7, or FIG. 8 described later, and generates a restored 2D image.

Figure 3 is a block diagram showing an example of the configuration of a 3D image processing device on the decoding side in Example 2.

This 3D image processing device 1-2 includes a 3D model decoding unit 11, a rendering unit 12, a restoration parameter storage unit 13, a 2D image restoration unit 14, and a restoration parameter decoding unit 15.

Comparing the 3D image processing device 1-1 on the decoding side of the first embodiment shown in FIG. 1 with the 3D image processing device 1-2 on the decoding side of the second embodiment, both 3D image processing devices 1-1 and 1-2 have in common a 3D model decoding unit 11, a rendering unit 12, a restoration parameter storage unit 13, and a 2D image restoration unit 14. On the other hand, the 3D image processing device 1-2 differs from the 3D image processing device 1-1 in that it includes a restoration parameter decoding unit 15 in addition to the configuration of the 3D image processing device 1-1. In FIG. 3, parts that are common to FIG. 1 are given the same reference numerals as in FIG. 1, and detailed descriptions thereof will be omitted.

The 3D image processing device 1-2 inputs the encoded 3D model and the encoded restoration parameters directly from the corresponding encoding side 3D image processing device (external device) or reads them from a memory (not shown). The 3D image processing device 1-2 also inputs rendering parameters.

The restoration parameter decoding unit 15 inputs the encoding restoration parameters generated and output by the encoding-side three-dimensional image processing devices 3-2a, 3-2b, and 3-2c shown in FIG. 5, FIG. 7, or FIG. 8 described later. The restoration parameter decoding unit 15 then generates decoded restoration parameters by decoding the encoding restoration parameters, and stores them in the restoration parameter storage unit 13 as restoration parameters.

The encoded restoration parameters input to the restoration parameter decoding unit 15 are parameters output in a form in which source coding, channel coding, or both have been applied in the encoding-side three-dimensional image processing devices 3-2a, 3-2b, and 3-2c shown in, for example, Figures 5, 7, and 8 described below.

The restoration parameter decoding unit 15 pairs with the restoration parameter encoding unit 34 shown in FIG. 5, FIG. 7, or FIG. 8 described below, and performs mutually inverse processing or approximately inverse processing within a specified error range.

The encoding method used by the restoration parameter decoding unit 15 and the restoration parameter encoding unit 34 described later may be any method, but is preferably a lossless method.

The coding method may be, for example, a source coding method such as run-length coding, differential pulse code modulation (DPCM), Huffman coding, arithmetic coding, or a combination of these. The coding method may also be, for example, a channel coding method such as error detection coding, error correction coding, or encryption coding, or may be a combination of a source coding method and a channel coding method.

By using source coding, channel coding, or both as the coding method, information can be compressed. Also, by adding redundant information to information, it is possible to save transmission capacity (storage capacity) and correct transmission errors (storage errors).

As described above, in the decoding-side 3D image processing device 1-2 of the second embodiment, the rendering unit 12 uses rendering parameters to render a decoded 3D model obtained by decoding the encoded 3D model, thereby generating a 2D image.

The two-dimensional image restoration unit 14 performs restoration processing on the two-dimensional image using the decoded restoration parameters obtained by decoded the encoded restoration parameters, and generates a restored two-dimensional image.

In other words, the 3D image processing device 1-2 generates a restored 2D image from a 2D image using restoration parameters generated and output (transmitted or stored) by the encoding-side 3D image processing devices 3-2a, 3-2b, and 3-2c shown in Figures 5, 7, and 8, which will be described later.

As a result, in a system that transmits or stores 3D images, similar to the 3D image processing device 1-1 on the decoding side in Example 1, it is possible to correct the degradation of the 2D image obtained by rendering when rendering a 3D model from a desired viewpoint.

Note that the 3D image processing device 1-2 on the decoding side in Example 2 shown in FIG. 3 includes a 3D model decoding unit 11 and a restored parameter decoding unit 15, but does not necessarily need to include the 3D model decoding unit 11 and the restored parameter decoding unit 15.

In this case, the 3D image processing device 1-2 includes a rendering unit 12, a restoration parameter storage unit 13, and a 2D image restoration unit 14, and directly inputs the uncoded 3D model and the uncoded restoration parameters. In other words, the rendering unit 12 renders the input 3D model using the rendering parameters, and the 3D image processing device 1-2 stores the input restoration parameters in the restoration parameter storage unit 13.

As a result, even if the 2D image obtained by rendering the 3D model is degraded due to errors in the 3D model, a restored 2D image can be obtained with the degradation corrected.

If the 3D image processing device 1-2 does not have a 3D model decoding unit 11 and a restoration parameter decoding unit 15, the corresponding 3D image processing device on the encoding side does not need to encode the 3D model and restoration parameters. In this case, the 3D image processing device on the encoding side outputs the 3D model and restoration parameters as is without encoding them.

The 3D image processing device 1-2 may also include a rendering unit 12, a restoration parameter storage unit 13, a 2D image restoration unit 14, and a restoration parameter decoding unit 15, and may not include a 3D model decoding unit 11.

The 3D image processing device 1-2 may also include a 3D model decoding unit 11, a rendering unit 12, a restoration parameter storage unit 13, and a 2D image restoration unit 14, but may not include a restoration parameter decoding unit 15.

However, the restoration parameter decoding unit 15 of the 3D image processing device 1-2 may not be able to input the encoding restoration parameters, and may not be able to decode the encoding restoration parameters even if they are input. When such an input error or decoding error occurs, the 2D image restoration unit 14 reads out the restoration parameters previously input and stored by the restoration parameter decoding unit 15 from the restoration parameter storage unit 13, and generates a restored 2D image.

In addition, in a configuration in which the 3D image processing device 1-2 does not include a restoration parameter decoding unit 15 and restoration parameters are directly input, there may be cases in which the restoration parameters cannot be input (if an input error occurs). In this case, the 2D image restoration unit 14 reads out the restoration parameters previously input and stored by the 3D image processing device 1-2 from the restoration parameter storage unit 13, and generates a restored 2D image.

This is assumed to be the case when the corresponding encoding side 3D image processing device does not output the encoding restoration parameters or restoration parameters (first case). It is also assumed to be the case when the corresponding encoding side 3D image processing device outputs the encoding restoration parameters or restoration parameters but they are missing due to an error (second case).

For example, the first case is assumed when the encoded 3D model and the encoding restoration parameters are multiplexed and transmitted. In this case, the 3D image processing device 1-2 on the decoding side uses an identifier or the like indicating that a stream (Elementary Stream) related to the encoding restoration parameters does not exist in the received stream (e.g., a transport stream) and determines that an input error has occurred, assuming that the encoding restoration parameters or restoration parameters have not been output.

The first or second case can be assumed when the encoded 3D model and the encoding and restoration parameters are transmitted over different lines, or when they are transmitted using a transmission line coding method with different error tolerance. In this case, for example, when the corresponding encoding 3D image processing device is not outputting the encoding and restoration parameters, it outputs an empty packet (null packet), and the decoding 3D image processing device 1-2 judges an input error by inputting the empty packet.

In the second case, the 3D image processing device 1-2 on the decoding side determines a missing error when an error is detected by using an error detection code or the like. In addition, the 3D image processing device 1-2 on the decoding side may determine an input error as a timeout when no encoding and restoration parameters are input within a certain time period after inputting the encoded 3D model.

This makes it possible to restore a two-dimensional image using previous restoration parameters if the restoration parameters are lost due to a transmission or storage malfunction, thereby improving resistance to transmission or storage malfunctions.

In particular, when the error tolerance of the code for the restoration parameters is set lower than the error tolerance of the code for the 3D model, it is possible to improve the probability of inputting the 3D model while suppressing the transmission or storage capacity, thereby saving the transmission bandwidth (or storage capacity).

[Encoding side three-dimensional image processing device: Example 1]
Next, a description will be given of a 3D image processing device (3D image encoding device) on the encoding side of Example 1. The 3D image processing device on the encoding side of Example 1 is an example that generates an encoded 3D model by encoding a 3D model, and outputs the encoded 3D model.

FIG. 4 is a block diagram showing an example of the configuration of a 3D image processing device on the encoding side in the first embodiment. This 3D image processing device 3-1 includes a 3D model encoding unit 31. The 3D image processing device 3-1 is an encoding side device corresponding to the case where the 3D image processing device 1-1 on the decoding side in the first embodiment shown in FIG. 1 includes a 3D model decoding unit 11.

The three-dimensional model encoding unit 31 is paired with the three-dimensional model decoding unit 11 shown in FIG. 1, and both operate using the same encoding method. The three-dimensional model encoding unit 31 inputs a three-dimensional model from the three-dimensional modeling unit 2 described below, encodes the three-dimensional model to generate an encoded three-dimensional model, and outputs the encoded three-dimensional model.

The encoded 3D model output by the 3D model encoding unit 31 is either transmitted to the corresponding 3D image processing device 1-1 shown in FIG. 1 or stored in a memory (not shown). The encoded 3D model stored in the memory is read out by the corresponding 3D image processing device 1-1 shown in FIG. 1.

The three-dimensional modeling unit 2 inputs measurement data (images) from multiple viewpoints, for example from a camera (not shown), generates a three-dimensional model from the images from the multiple viewpoints, and outputs the three-dimensional model to the three-dimensional image processing device 3-1.

The three-dimensional modeling unit 2, for example, inputs multiple images from a camera and generates a three-dimensional model from the images from multiple viewpoints. For this generation, for example, a stereo matching method can be applied in which an image from one viewpoint is divided into multiple blocks (partial images), and the disparity is calculated by determining the area on the image from the other viewpoint that maximizes the cross-correlation with the block (or minimizes the error), and depth information is estimated from the disparity. The depth information obtained in this way is associated with the coordinates of one of the images and visualized, thereby obtaining a three-dimensional model from depth images.

The three-dimensional modeling unit 2 may also extract feature points from each of the images from the multiple viewpoints, associate similar feature points with each other, and obtain three-dimensional coordinates from the image coordinate pairs. In this case, a three-dimensional model is obtained as a point cloud consisting of the number of points in the obtained image pairs. For example, SIFT (Scale-Invariant Feature Transform) can be used to extract and associate the feature points.

Furthermore, three or more (e.g., three) nearby points from the point group may be combined to form a group of polygons (e.g., triangles) to obtain a three-dimensional model using a polygon model.

[Encoding side three-dimensional image processing device: Example 2-1]
Next, a three-dimensional image processing device (three-dimensional image coding device) on the coding side of Example 2-1 will be described. The three-dimensional image processing device on the coding side of Example 2-1 is an example in which a three-dimensional model is coded to generate and output an encoded three-dimensional model, a two-dimensional image is generated by rendering the three-dimensional model, and a restoration parameter is generated and output using a learning image and the two-dimensional image.

FIG. 5 is a block diagram showing an example of the configuration of a 3D image processing device on the encoding side in Example 2-1, and FIG. 6 is a flowchart showing an example of the processing in the 3D image processing device on the encoding side in Example 2-1.

This 3D image processing device 3-2a includes a 3D model encoding unit 31, a rendering unit 32, a restoration parameter generating unit 33, and a restoration parameter encoding unit 34. The restoration parameter generating unit 33 includes a restoration parameter storage unit 41, a 2D image restoration unit 42, an error evaluation unit 43, and a restoration parameter update unit 44.

The three-dimensional image processing device 3-2a is an encoding-side device that corresponds to the case where the decoding-side three-dimensional image processing device 1-2 of the second embodiment shown in FIG. 3 is equipped with a three-dimensional model decoding unit 11 and a restored parameter decoding unit 15.

The three-dimensional image processing device 3-2a inputs the three-dimensional model from the three-dimensional modeling unit 2, as well as preset rendering parameters and a learning image (step S601). The three-dimensional modeling unit 2 has already been described in FIG. 4, so a description thereof will be omitted here.

The rendering unit 32 inputs the three-dimensional model and preset rendering parameters, and generates a two-dimensional CG image by rendering the three-dimensional model using the rendering parameters (step S602). The rendering unit 32 then outputs the two-dimensional image to the restoration parameter generating unit 33.

It is preferable that the rendering parameters match the shooting conditions of the learning image. In addition, it is preferable that the rendering unit 32 operates according to an algorithm similar to that of the rendering unit 12 shown in FIG. 3, but it is not necessary that the algorithm is completely the same.

The restoration parameter generating unit 33 inputs the learning image and the two-dimensional image from the rendering unit 32. The restoration parameter generating unit 33 then obtains a restored two-dimensional image from the two-dimensional image using the provisional restoration parameters, and updates the provisional restoration parameters so that the error between the learning image and the restored two-dimensional image is reduced.

When a predetermined termination condition is satisfied, the restoration parameter generating unit 33 outputs the updated provisional restoration parameters as restoration parameters to the restoration parameter encoding unit 34.

Specifically, the restoration parameter storage unit 41 of the restoration parameter generation unit 33 holds provisional parameters that define the operation of the two-dimensional image restoration unit 42. The provisional restoration parameters are, for example, the connection weight coefficient values and bias values of the neural network.

The two-dimensional image restoration unit 42 inputs the two-dimensional image from the rendering unit 32, reads out the provisional restoration parameters from the restoration parameter storage unit 41, and performs restoration processing on the two-dimensional image using the provisional restoration parameters to generate a restored two-dimensional image (step S603). The two-dimensional image restoration unit 42 then outputs the restored two-dimensional image to the error evaluation unit 43.

The two-dimensional image restoration unit 42 is implemented in the same manner as the two-dimensional image restoration unit 14 shown in FIG. 3, but the specific behavior of these units varies depending on the restoration parameters, so the behavior is not necessarily the same. However, when the same restoration parameters are given to the two-dimensional image restoration unit 42 and the two-dimensional image restoration unit 14 shown in FIG. 3, both two-dimensional image restoration units 42 and 14 will operate in the same manner.

For example, if the two-dimensional image restoration unit 42 and the two-dimensional image restoration unit 14 are configured as neural networks, the network connection topology (and activation function) between the elements is assumed to behave in the same way.

The error evaluation unit 43 inputs the learning image and the restored two-dimensional image from the two-dimensional image restoration unit 42, evaluates the error by quantifying the difference between the learning image and the restored two-dimensional image, and calculates an error value (step S604). The error evaluation unit 43 then outputs the error value to the restoration parameter update unit 44.

For example, the error evaluation unit 43 may subtract the pixel value of the learning image from the pixel value of the reconstructed two-dimensional image for each pixel position, and output the error image resulting from the subtraction as the error value. The error evaluation unit 43 may also output the sum of the error images over the entire image as the error value.

Furthermore, the error evaluation unit 43 may sum up the error image for each partial region of the image and output the sum value for each partial region as the error value. In this case, the partial regions may or may not overlap each other.

The error value calculated by the error evaluation unit 43 may be, for example, an absolute error or a squared error. In addition, when the error evaluation unit 43 performs a summation operation when calculating the error value, the error value may be the sum of absolute errors (SAE), the sum of squared errors (SSE), the mean of absolute errors (MAE), the positive square root of the mean of squared errors (RMSE), etc.

Furthermore, the error evaluation unit 43 may calculate the error value based on the structural similarity index (SSIM: Structural SIMilarity) between the training image and the reconstructed two-dimensional image. For example, the error value may be calculated by subtracting the SSIM value from 1.

The error evaluation unit 43 may also calculate an error value based on the cross-entropy or cross-correlation value between the training image and the reconstructed two-dimensional image. The error evaluation unit 43 may also determine an error value to be output by combining (for example, linearly combining) the calculated multiple error values and the structural similarity index, etc.

As the learning image, the image used when the 3D model input to the 3D image processing device 3-2a is generated by the 3D modeling unit 2 may be used. For example, if the 3D model is generated from images from multiple viewpoints, one of the images from the multiple viewpoints may be used as the learning image. In this case, the camera position and angle of view information when the image from the viewpoint was captured can be used as rendering parameters.

In addition, a camera for acquiring learning images may be installed separately from the sensor or camera used in the three-dimensional modeling unit 2, and the images acquired by the learning image acquiring camera may be used as learning images.

The restoration parameter update unit 44 inputs the error value from the error evaluation unit 43, and updates the value of the provisional restoration parameter temporarily stored in the restoration parameter storage unit 41 so as to reduce the error value (step S605). Then, when a predetermined termination condition is satisfied, the restoration parameter update unit 44 outputs the provisional restoration parameter as the restoration parameter to the restoration parameter coding unit 34.

For example, the restoration parameter update unit 44 may change the provisional restoration parameters stored in the restoration parameter storage unit 41 by a predetermined amount or a random value (including a pseudo-random value), and if the error value becomes smaller, adopt the changed provisional restoration parameters, and if the error value does not become smaller, maintain the pre-change provisional restoration parameters.

The restoration parameter update unit 44 may also update the provisional restoration parameters, for example, by gradient descent, based on values (Jacobian values) obtained by partially differentiating the input/output relationship function of the two-dimensional image restoration unit 42 with respect to each of the provisional restoration parameters.

When a neural network is used in the two-dimensional image restoration unit 42 as a method based on partial differentiation, the restoration parameter update unit 44 may update the provisional restoration parameters by the error value backpropagation method.

When a termination condition is met that is defined based on the number of iterations for updating the provisional restoration parameters, the error value input from the error evaluation unit 43, or both, the restoration parameter update unit 44 stops the iterative operation and outputs the provisional restoration parameters last updated (or last updated just before) as restoration parameters to the restoration parameter coding unit 34.

For example, the termination condition may be determined to be satisfied when the error value falls below a predetermined value, and the restoration parameter update unit 44 may output the last updated tentative restoration parameter as the restoration parameter. Alternatively, the termination condition may be determined to be satisfied when the current error value exceeds the previous error value, and the restoration parameter update unit 44 may output the last updated tentative restoration parameter as the restoration parameter.

The termination condition is determined to be satisfied when the difference between the current error value and the previous error value becomes equal to or less than a predetermined value, and the restoration parameter update unit 44 may output the last updated tentative restoration parameter or the tentative restoration parameter updated immediately before the last one as the restoration parameter.

The termination condition may be determined to be satisfied when the number of updates by the restoration parameter update unit 44 reaches a predetermined number of repetitions, and the restoration parameter update unit 44 may output the last updated tentative restoration parameters as the restoration parameters.

The termination condition may also be determined by a logical operation (e.g., a logical sum or a logical product) between a condition on the number of updates by the restoration parameter update unit 44 and a condition based on the error value.

Furthermore, the error evaluation unit 43 may input a verification image different from the learning image instead of the learning image, and the rendering unit 32 may use rendering parameters corresponding to the verification image, and the restoration parameter update unit 44 may determine the termination condition based on the error value output by the error evaluation unit 43 when the rendering unit 32, the two-dimensional image restoration unit 42, and the error evaluation unit 43 are operated.

In this case, the verification image is used only to determine the termination condition, and is not used to update the provisional restoration parameters. Furthermore, the determination of the termination condition associated with the input of the verification image may be performed each time the provisional restoration parameters are updated using the learning image, or may be performed after each update when the restoration parameters are updated multiple times using the learning image.

The three-dimensional model encoding unit 31 is paired with the three-dimensional model decoding unit 11 shown in FIG. 3, and both operate according to the same encoding method. The three-dimensional model encoding unit 31 generates an encoded three-dimensional model by encoding the three-dimensional model, and outputs the encoded three-dimensional model (step S606).

The restoration parameter encoding unit 34 is paired with the restoration parameter decoding unit 15 shown in FIG. 3, and both operate according to the same encoding method. The restoration parameter encoding unit 34 inputs the restoration parameters from the restoration parameter update unit 44, generates encoded restoration parameters by encoding the restoration parameters, and outputs the encoded restoration parameters (step S607).

The encoded 3D model output by the 3D model encoding unit 31 and the encoded restoration parameters output by the restoration parameter encoding unit 34 are transmitted to the corresponding 3D image processing device 1-2 shown in FIG. 3 or stored in a memory (not shown). The encoded 3D model and the encoded restoration parameters stored in the memory are read out by the corresponding 3D image processing device 1-2 shown in FIG. 3.

As described above, in the encoding-side 3D image processing device 3-2a of Example 2-1, the rendering unit 32 generates a 2D image by rendering a 3D model using rendering parameters.

The restoration parameter generating unit 33 uses the provisional restoration parameters to obtain a restored two-dimensional image from the two-dimensional image, and updates the provisional restoration parameters so as to reduce the error between the learning image and the restored two-dimensional image. Then, when a predetermined termination condition is satisfied, the restoration parameter generating unit 33 outputs the updated provisional restoration parameters as the restoration parameters.

Then, the 3D model and the reconstruction parameters are coded and output (transmitted or stored) as the coded 3D model and the coding reconstruction parameters. The 3D image processing device 1-2 on the decoding side of the second embodiment shown in FIG. 3, which receives the coded 3D model and the coding reconstruction parameters output by the 3D image processing device 3-2a, generates a reconstructed 2D image as described above.

As a result, even if the 2D image obtained by rendering the 3D model is degraded due to errors in the 3D model and degradation due to encoding, a restored 2D image can be obtained in which the degradation has been corrected. In other words, restoration parameters can be generated in the 3D image processing device 3-2a on the encoding side so that the decoding side can subsequently correct the degradation of the 2D image.

Therefore, in a system that transmits or stores three-dimensional images, when a three-dimensional model is rendered from a desired viewpoint, the degradation of the two-dimensional image obtained by rendering can be corrected.

Note that although the 3D image processing device 3-2a shown in FIG. 5 includes a 3D model encoding unit 31 and a restoration parameter encoding unit 34, it does not have to include the 3D model encoding unit 31 and the restoration parameter encoding unit 34. In this case, the 3D image processing device 3-2a includes a rendering unit 32 and a restoration parameter generating unit 33, and directly outputs an uncoded 3D model and uncoded restoration parameters. In other words, the 3D image processing device 3-2a outputs the input 3D model as is, and the restoration parameter updating unit 44 of the restoration parameter generating unit 33 outputs the restoration parameters.

The three-dimensional image processing device 3-2a may also include a rendering unit 32, a restoration parameter generating unit 33, and a restoration parameter encoding unit 34, and may not include a three-dimensional model encoding unit 31.

The three-dimensional image processing device 3-2a may also include a three-dimensional model encoding unit 31, a rendering unit 32, and a restoration parameter generating unit 33, but may not include a restoration parameter encoding unit 34.

[Encoding side three-dimensional image processing device: Example 2-2]
Next, a three-dimensional image processing device (three-dimensional image encoding device) on the encoding side of Example 2-2 will be described. The three-dimensional image processing device on the encoding side of Example 2-2 is an example in which a three-dimensional model is encoded to generate and output an encoded three-dimensional model, the encoded three-dimensional model is decoded, the decoded three-dimensional model is rendered to generate a two-dimensional image, and a learning image and the two-dimensional image are used to generate and output restoration parameters.

Figure 7 is a block diagram showing an example of the configuration of a 3D image processing device on the encoding side in Example 2-2.

This 3D image processing device 3-2b includes a 3D model encoding unit 31, a rendering unit 32, a restoration parameter generating unit 33, a restoration parameter encoding unit 34, and a 3D model decoding unit 35. The restoration parameter generating unit 33 includes a restoration parameter storage unit 41, a 2D image restoration unit 42, an error evaluation unit 43, and a restoration parameter update unit 44.

Comparing the encoding side 3D image processing device 3-2a of Example 2-1 shown in FIG. 5 with the encoding side 3D image processing device 3-2b of Example 2-2 shown in FIG. 7, both 3D image processing devices 3-2a and 3-2b have in common the fact that they are equipped with a 3D model encoding unit 31, a rendering unit 32, a restoration parameter generating unit 33, and a restoration parameter encoding unit 34.

On the other hand, the 3D image processing device 3-2b differs from the 3D image processing device 3-2a, which does not have a 3D model decoding unit 35, in that it has a 3D model decoding unit 35. In FIG. 7, parts that are common to FIG. 5 are given the same reference numerals as in FIG. 5, and detailed explanations thereof will be omitted.

The three-dimensional image processing device 3-2b is an encoding-side device that corresponds to the case where the decoding-side three-dimensional image processing device 1-2 of the second embodiment shown in FIG. 3 is equipped with a three-dimensional model decoding unit 11 and a restored parameter decoding unit 15, similar to the three-dimensional image processing device 3-2a shown in FIG. 5.

The 3D model encoding unit 31 outputs the encoded 3D model to the outside and also to the 3D model decoding unit 35.

The 3D model decoding unit 35 inputs the encoded 3D model from the 3D model encoding unit 31, generates a decoded 3D model by decoding the encoded 3D model, and outputs the decoded 3D model to the rendering unit 32.

The three-dimensional model decoding unit 35 performs processing using the same encoding method as the three-dimensional model decoding unit 11 shown in FIG. 3, and they have the same input/output relationship.

When the 3D model encoding unit 31 has a local decoding output, the 3D model decoding unit 35 can be omitted by using the local decoding output instead of the output of the 3D model decoding unit 35. In other words, when the 3D model encoding unit 31 has a local decoding function, it outputs the decoded 3D model generated by local decoding to the rendering unit 32.

In addition, when comparing the operation of the rendering unit 32 in Example 2-1 shown in FIG. 5 with the operation of the rendering unit 32 in Example 2-2 shown in FIG. 7, the former inputs a 3D model that is the original data, whereas the latter inputs a decoded 3D model generated by the encoding and decoding process; otherwise, they are the same.

As described above, in the encoding-side 3D image processing device 3-2b of Example 2-2, the 3D model decoding unit 35 decodes the encoded 3D model in which the 3D model is encoded, and generates a decoded 3D model. The rendering unit 32 renders the decoded 3D model using the rendering parameters, thereby generating a 2D image.

The restoration parameter generating unit 33 uses the provisional restoration parameters to obtain a restored two-dimensional image from the two-dimensional image, and updates the provisional restoration parameters so as to reduce the error between the learning image and the restored two-dimensional image. Then, when a predetermined termination condition is satisfied, the restoration parameter generating unit 33 outputs the updated provisional restoration parameters as the restoration parameters.

Then, the 3D model and the reconstruction parameters are coded and output (transmitted or stored) as an encoded 3D model and encoded reconstruction parameters. The 3D image processing device 1-2 on the decoding side of the second embodiment shown in FIG. 3, which receives the encoded 3D model and encoded reconstruction parameters output by the 3D image processing device 3-2b, generates a reconstructed 2D image as described above.

As a result, similar to the 3D image processing device 3-2a on the encoding side in Example 2-1, restoration parameters can be generated in the 3D image processing device 3-2b on the encoding side so that the decoding side can correct the degradation of the 2D image afterwards.

Therefore, in a system that transmits or stores three-dimensional images, when a three-dimensional model is rendered from a desired viewpoint, the degradation of the two-dimensional image obtained by rendering can be corrected.

In addition, according to the three-dimensional image processing device 3-2b, even if the three-dimensional model encoding unit 31 causes encoding degradation in the three-dimensional model, the restoration parameter generation unit 33 optimizes the restoration parameters, including the encoding degradation. This is particularly effective when the encoding method used in the three-dimensional model encoding unit 31 is lossy. Note that even if the encoding method used in the three-dimensional model encoding unit 31 is lossless, there is no adverse effect on the restoration parameters.

Therefore, suitable restoration parameters can be obtained regardless of the encoding method of the three-dimensional model. If the encoding method used by the three-dimensional model encoding unit 31 is lossy, there will be errors in the three-dimensional model, and furthermore, there will be degradation in the three-dimensional model due to the lossy encoding. Even if the two-dimensional image obtained by rendering the three-dimensional model is degraded, it is possible to generate restoration parameters for obtaining a restored two-dimensional image in which the degradation has been corrected afterwards on the decoding side.

In addition, if the 3D model encoding unit 31 has a local decoding function, the 3D model decoding unit 35 is unnecessary, so there is no increase in computational costs.

Note that although the 3D image processing device 3-2b shown in FIG. 7 includes a restoration parameter encoding unit 34, it does not have to include the restoration parameter encoding unit 34. In this case, the 3D image processing device 3-2b includes a 3D model encoding unit 31, a rendering unit 32, a restoration parameter generating unit 33, and a 3D model decoding unit 35, and directly outputs unencoded restoration parameters. In other words, the restoration parameter updating unit 44 of the restoration parameter generating unit 33 outputs the restoration parameters.

In addition, in the three-dimensional image processing device 3-2b, the three-dimensional model encoding unit 31 outputs the encoded three-dimensional model, but the three-dimensional image processing device 3-2b may output the input three-dimensional model as is.

Furthermore, in the three-dimensional image processing device 3-2a on the encoding side of Example 2-1 shown in FIG. 5 and the three-dimensional image processing device 3-2b on the encoding side of Example 2-2 shown in FIG. 7, the operations associated with the processing of the rendering unit 32, the restoration parameter generating unit 33, and the restoration parameter encoding unit 34 provided in the three-dimensional image processing device 3-2a, and the processing of the three-dimensional model decoding unit 35, the rendering unit 32, the restoration parameter generating unit 33, and the restoration parameter encoding unit 34 provided in the three-dimensional image processing device 3-2b (hereinafter referred to as "restoration parameter updating, etc.") may be executed each time a three-dimensional model is input, or may be executed intermittently.

Furthermore, the operation may be performed at unspecified timing in accordance with user operations on the 3D image processing devices 3-2a and 3-2b, or may be performed only once at a specific or unspecified timing (for example, when the first 3D model is input).

In addition, if the input 3D model is a time-series model and the 3D model encoding unit 31 complies with a video encoding format having a GOP (Group Of Pictures) structure, the operations such as updating the restoration parameters may be performed at intervals of the GOP (for example, at the timing of an intra-picture or intra-slice). The same applies to the 3D image processing device 3-2c shown in FIG. 8, which will be described later.

In addition, in the three-dimensional image processing devices 3-2a and 3-2b, the restoration parameters are output by the restoration parameter update unit 44 provided in the restoration parameter generation unit 33, and the restoration parameter coding unit 34 performs coding and outputs the coded restoration parameters (i.e., the output frequency of the coded restoration parameters) may be the same as the execution frequency of other restoration parameter updates, etc., or the input frequency of the three-dimensional model (hereinafter collectively referred to as the "input frequency of the three-dimensional model"). In addition, the output frequency of the coded restoration parameters may be lower than the input frequency of the three-dimensional model.

By reducing the execution frequency at which the encoding and restoration parameters are output, the restoration parameters can be updated with greater precision due to an increase in the amount of learning, while the amount of data for the encoding and restoration parameters is reduced, allowing the overall amount of code to be reduced. In other words, the bit rate or recording capacity required for transmitting or storing the 3D model and the restoration parameters is reduced. The same applies to the 3D image processing device 3-2c shown in Figure 8, which will be described later.

The restoration parameter update unit 44 provided in the restoration parameter generation unit 33 of the 3D image processing devices 3-2a and 3-2b initializes the provisional restoration parameters with a predetermined numeric sequence or random number sequence (including a pseudo-random number sequence) when the previous or first learning image is input. The same applies to the 3D image processing device 3-2c shown in FIG. 8, which will be described later.

In this case, the restoration parameter update unit 44 may perform the initialization each time a learning image is input, or each time a predetermined number of learning images are input. The restoration parameter update unit 44 may also perform the initialization at unspecified times in accordance with user operations on the 3D image processing devices 3-2a and 3-2b.

The restoration parameter update unit 44 determines restoration parameters so that an error value is small when processing is performed by the 3D model encoding unit 31 or the like using the learning image, 3D model, and rendering parameters input in advance. The restoration parameter update unit 44 may use the restoration parameters determined in advance as a predetermined numerical sequence or the like during the initialization. In this case, transfer learning can be realized using the restoration parameters determined in advance.

[Encoding side three-dimensional image processing device: Example 2-3]
Next, a three-dimensional image processing device (three-dimensional image encoding device) on the encoding side of Example 2-3 will be described. The three-dimensional image processing device on the encoding side of Example 2-3 is an example in which a three-dimensional model is encoded to generate and output an encoded three-dimensional model, the encoded three-dimensional model is decoded, and a two-dimensional image is generated by rendering the decoded three-dimensional model, and restoration parameters are generated from a learning image and a two-dimensional image using CycleGAN and output.

Figure 8 is a block diagram showing an example of the configuration of a 3D image processing device on the encoding side in Example 2-3.

This 3D image processing device 3-2c includes a 3D model encoding unit 31, a rendering unit 32, a restoration parameter generating unit 37, a restoration parameter encoding unit 34, and a 3D model decoding unit 35.

When comparing the encoding side 3D image processing device 3-2b of Example 2-2 shown in FIG. 7 with the encoding side 3D image processing device 3-2c of Example 2-3 shown in FIG. 8, both 3D image processing devices 3-2b and 3-2c have in common the fact that they are equipped with a 3D model encoding unit 31, a rendering unit 32, a restored parameter encoding unit 34, and a 3D model decoding unit 35.

On the other hand, the three-dimensional image processing device 3-2c differs from the three-dimensional image processing device 3-2b equipped with the restoration parameter generating unit 33 in that the three-dimensional image processing device 3-2c is equipped with a restoration parameter generating unit 37 that is different from the restoration parameter generating unit 33. In FIG. 8, parts common to FIG. 7 are given the same reference numerals as in FIG. 7, and detailed explanations thereof will be omitted.

The three-dimensional image processing device 3-2c, like the three-dimensional image processing device 3-2a shown in FIG. 5 and the three-dimensional image processing device 3-2b shown in FIG. 7, is an encoding-side device corresponding to the case where the decoding-side three-dimensional image processing device 1-2 of the second embodiment shown in FIG. 3 is equipped with a three-dimensional model decoding unit 11 and a restored parameter decoding unit 15.

The restoration parameter generating unit 37 inputs the learning image and the two-dimensional image from the rendering unit 32. The restoration parameter generating unit 37 updates the provisional restoration parameters using the neural network of the cycle gun, and when a predetermined termination condition is satisfied, outputs the updated provisional restoration parameters to the restoration parameter encoding unit 34 as restoration parameters.

Figure 9 is a block diagram showing an example of the configuration of the restoration parameter generation unit 37 in Example 2-3, and Figure 10 is a flowchart showing an example of the processing of the restoration parameter generation unit 37.

Comparing the restoration parameter generating unit 33 shown in Figures 5 and 7 with this restoration parameter generating unit 37, both restoration parameter generating units 33 and 37 have in common the point that they update tentative restoration parameters. On the other hand, the restoration parameter generating unit 37 is equipped with a neural network of a cycle gun, and differs from the restoration parameter generating unit 33, which updates only tentative restoration parameters, in that it updates degradation parameters in addition to tentative restoration parameters.

The restoration parameter generation unit 37 includes a restoration parameter storage unit 51, a degradation parameter storage unit 52, a two-dimensional image restoration unit 53, a two-dimensional image degradation unit 54, error evaluation units 55 and 56, a restoration parameter update unit 57, and a degradation parameter update unit 58. Both the two-dimensional image restoration unit 53 and the two-dimensional image degradation unit 54 include a generative adversarial network.

The restoration parameter storage unit 51 holds provisional restoration parameters FP that define the operation of the two-dimensional image restoration unit 53, and the degradation parameter storage unit 52 holds degradation parameters RP that define the operation of the two-dimensional image degradation unit 54. These parameters are the connection weight coefficient values and bias values of the neural network.

The two-dimensional image restoration unit 53 inputs the two-dimensional image A from the rendering unit 32 and reads out the provisional restoration parameters FP from the restoration parameter storage unit 51. The two-dimensional image restoration unit 53 then performs a restoration process on the two-dimensional image A using the provisional restoration parameters FP to generate a first restored two-dimensional image F1 (step S1001). The two-dimensional image restoration unit 53 outputs the first restored two-dimensional image F1 to the two-dimensional image degradation unit 54.

The two-dimensional image degradation unit 54 inputs the first restored two-dimensional image F1 from the two-dimensional image restoration unit 53 and reads out the degradation parameter RP from the degradation parameter storage unit 52. The two-dimensional image degradation unit 54 then performs degradation processing on the first restored two-dimensional image F1 using the degradation parameter RP to generate a first degraded two-dimensional image R1 (step S1002). The two-dimensional image degradation unit 54 outputs the first degraded two-dimensional image R1 to the error evaluation unit 55.

The degradation process in this case is a process in which degradation caused by data errors in the 3D model and encoding by the 3D model encoding unit 31 is simulated for the first restored 2D image F1, and the result is generated as a first degraded 2D image R1.

The error evaluation unit 55 inputs the two-dimensional image A from the rendering unit 32 and the first degraded two-dimensional image R1 from the two-dimensional image degradation unit 54. The error evaluation unit 55 then evaluates the error by quantifying the difference between the two-dimensional image A and the first degraded two-dimensional image R1, and calculates an error value e1 (step S1003). The error evaluation unit 55 outputs the error value e1 to the restoration parameter update unit 57 and the degradation parameter update unit 58.

The two-dimensional image degradation unit 54 inputs the learning image G and reads out the degradation parameter RP from the degradation parameter storage unit 52. The two-dimensional image degradation unit 54 then performs degradation processing on the learning image G using the degradation parameter RP to generate a second degraded two-dimensional image R2 (step S1004). The two-dimensional image degradation unit 54 outputs the second degraded two-dimensional image R2 to the two-dimensional image restoration unit 53.

The degradation process in this case is a process in which degradation of the learning image G caused by data errors in the 3D model and by the encoding by the 3D model encoding unit 31 is simulated, and the result is generated as a second degraded 2D image R2.

The two-dimensional image restoration unit 53 inputs the second degraded two-dimensional image R2 from the two-dimensional image degradation unit 54, and reads out the provisional restoration parameters FP from the restoration parameter storage unit 51. The two-dimensional image restoration unit 53 then performs a restoration process on the second degraded two-dimensional image R2 using the provisional restoration parameters FP, and generates a second restored two-dimensional image F2 (step S1005). The two-dimensional image restoration unit 53 outputs the second restored two-dimensional image F2 to the error evaluation unit 56.

The error evaluation unit 56 inputs the learning image G and the second restored two-dimensional image F2 from the two-dimensional image restoration unit 53. The error evaluation unit 56 then evaluates the error by quantifying the difference between the learning image G and the second restored two-dimensional image F2, and calculates an error value e2 (step S1006). The error evaluation unit 56 outputs the error value e2 to the restoration parameter update unit 57 and the degradation parameter update unit 58.

The restoration parameter update unit 57 inputs the error value e1 from the error evaluation unit 55 and the error value e2 from the error evaluation unit 56. Then, the restoration parameter update unit 57 updates the value of the provisional restoration parameter FP temporarily stored in the restoration parameter storage unit 51, for example, by the error value backpropagation method, so that the error values e1 and e2 become smaller (step S1007). For example, the restoration parameter update unit 57 alternately repeats a process of updating the provisional restoration parameter FP so that the error value e1 becomes smaller and a process of updating the provisional restoration parameter FP so that the error value e2 becomes smaller.

The degradation parameter update unit 58 inputs the error value e1 from the error evaluation unit 55 and the error value e2 from the error evaluation unit 56. Then, the degradation parameter update unit 58 updates the value of the degradation parameter RP temporarily stored in the degradation parameter storage unit 52, for example, by the error value backpropagation method, so that the error values e1 and e2 become smaller (step S1008). For example, the degradation parameter update unit 58 alternately repeats a process of updating the degradation parameter RP so that the error value e1 becomes smaller and a process of updating the degradation parameter RP so that the error value e2 becomes smaller.

The restoration parameter update unit 57 and the degradation parameter update unit 58 stop the update process when a predetermined termination condition is satisfied, similar to the restoration parameter update unit 44 shown in Figs. 5 and 7. The restoration parameter update unit 57 outputs the updated tentative restoration parameter FP as the restoration parameter P to the restoration parameter coding unit 34 (step S1009).

The predetermined termination condition may be the same or different in the restoration parameter update unit 57 and the degradation parameter update unit 58. The predetermined termination condition may be defined, for example, based on both or either one of the error values e1 and e2, or may be satisfied when the error value falls below a predetermined threshold value.

As described above, in the 3D image processing device 3-2c on the encoding side of Example 2-3, the 3D model decoding unit 35 decodes the encoded 3D model in which the 3D model is encoded, and generates a decoded 3D model. The rendering unit 32 renders the decoded 3D model using the rendering parameters, thereby generating a 2D image.

The restoration parameter generation unit 37 uses the neural network of the cycle gun to update the provisional restoration parameters, and when a predetermined termination condition is satisfied, outputs the updated provisional restoration parameters as the restoration parameters.

Then, the 3D model and the reconstruction parameters are coded and output (transmitted or stored) as an encoded 3D model and encoded reconstruction parameters. The 3D image processing device 1-2 on the decoding side of the second embodiment shown in FIG. 3, which receives the encoded 3D model and encoded reconstruction parameters output by the 3D image processing device 3-2c, generates a reconstructed 2D image as described above.

As a result, in a system that transmits or stores 3D images, similar to the 3D image processing devices 3-2a and 3-2b on the encoding side in Examples 2-1 and 2-2, it is possible to correct the degradation of the 2D image obtained by rendering when rendering a 3D model from a desired viewpoint in a system that transmits or stores 3D images.

In addition, according to the three-dimensional image processing device 3-2c, the restoration parameter generating unit 37 uses a cycle gun neural network.

As a result, even if the input 3D model and learning images are not necessarily acquired from the same subject or under the same shooting conditions, it is possible to obtain restoration parameters for correcting data errors in the 3D model and degradation caused by the 3D model encoding unit 31.

In other words, in the three-dimensional image processing devices 3-2a and 3-2b shown in Figures 5 and 7, the three-dimensional model and learning images must be acquired from the same subject and under the same shooting conditions, but in the three-dimensional image processing device 3-2c, there is no such restriction. In other words, in the three-dimensional image processing device 3-2c, it is possible to realize a process for generating restoration parameters that is easy to use.

Note that although the 3D image processing device 3-2c shown in FIG. 8 includes a restoration parameter encoding unit 34, it does not have to include the restoration parameter encoding unit 34. In this case, the 3D image processing device 3-2c includes a 3D model encoding unit 31, a rendering unit 32, a restoration parameter generating unit 37, and a 3D model decoding unit 35, and directly outputs unencoded restoration parameters. In other words, the restoration parameter updating unit 57 of the restoration parameter generating unit 37 outputs the restoration parameters.

In addition, in the three-dimensional image processing device 3-2c, the three-dimensional model encoding unit 31 outputs the encoded three-dimensional model, but the three-dimensional image processing device 3-2c may output the input three-dimensional model as is.

The 3D image processing device 3-2c also includes a 3D model decoding unit 35, but does not necessarily need to include the 3D model decoding unit 35. In this case, the rendering unit 32 directly inputs the 3D model.

The present invention has been described above using Examples 1, 2, and 2-1 to 2-3, but the present invention is not limited to Examples 1 and 2, etc., and can be modified in various ways without departing from the technical concept thereof.

In addition, a normal computer can be used as the hardware configuration of the three-dimensional image processing device 1-1 on the decoding side and the three-dimensional image processing device 3-1 on the encoding side according to the first embodiment of the present invention. The same applies to the three-dimensional image processing device 1-2 on the decoding side according to the second embodiment of the present invention and the three-dimensional image processing devices 3-2a to 3-2c on the encoding side according to the embodiments 2-1 to 2-3.

The three-dimensional image processing devices 1-1, 1-2, 3-1, 3-2a to 3-2c are configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

The functions of the three-dimensional model decoding unit 11, rendering unit 12, restoration parameter storage unit 13, and two-dimensional image restoration unit 14 provided in the three-dimensional image processing device 1-1 are each realized by having the CPU execute a program describing these functions. In addition, the functions of the three-dimensional model decoding unit 11, rendering unit 12, restoration parameter storage unit 13, two-dimensional image restoration unit 14, and restoration parameter decoding unit 15 provided in the three-dimensional image processing device 1-2 are each realized by having the CPU execute a program describing these functions.

Furthermore, the functions of the three-dimensional model encoding unit 31 provided in the three-dimensional image processing device 3-1 are realized by having the CPU execute a program describing the functions. Also, the functions of the three-dimensional model encoding unit 31, rendering unit 32, restoration parameter generating unit 33, and restoration parameter encoding unit 34 provided in the three-dimensional image processing device 3-2a are each realized by having the CPU execute a program describing these functions.

Furthermore, the functions of the three-dimensional model encoding unit 31, rendering unit 32, restoration parameter generating unit 33, restoration parameter encoding unit 34, and three-dimensional model decoding unit 35 provided in the three-dimensional image processing device 3-2b are each realized by having the CPU execute a program that describes these functions.

Furthermore, the functions of the three-dimensional model encoding unit 31, rendering unit 32, restoration parameter generating unit 37, restoration parameter encoding unit 34, and three-dimensional model decoding unit 35 provided in the three-dimensional image processing device 3-2c are each realized by having the CPU execute a program that describes these functions.

These programs are stored in the storage medium and are read and executed by the CPU. In addition, these programs can be distributed by storing them on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memories, and can also be transmitted and received via a network.

1-1, 1-2, 3-1, 3-2a, 3-2b, 3-2c 3D image processing device 2 3D modeling unit 11, 35 3D model decoding unit 12, 32 Rendering unit 13, 41, 51 Restoration parameter storage unit 14, 42, 53 2D image restoration unit 15 Restoration parameter decoding unit 31 3D model encoding unit 33, 36, 37 Restoration parameter generation unit 34 Restoration parameter encoding unit 43, 55, 56 Error evaluation unit 44, 57 Restoration parameter update unit 52 Degradation parameter storage unit 54 2D image degradation unit 58 Degradation parameter update unit A 2D image F1 First restored 2D image F2 Second restored 2D image R1 First degraded 2D image R2 Second degraded 2D image FP Provisional restoration parameter P Restoration parameter RP Degradation parameters e1, e2 Error value

Claims (12)

A three-dimensional image processing device that renders a three-dimensional model representing a three-dimensional image and generates a two-dimensional image,
a rendering unit that renders the three-dimensional model using preset rendering parameters to generate the two-dimensional image;
a two-dimensional image restoration unit that performs a restoration process for restoring deterioration of the two-dimensional image generated by the rendering unit using preset restoration parameters, and generates a restored two-dimensional image;
A three-dimensional image processing device comprising: 2. The three-dimensional image processing device according to claim 1,
The two-dimensional image restoration unit
A three-dimensional image processing apparatus comprising: a processor for processing the two-dimensional image using the restoration parameters generated by an external device that outputs the three-dimensional model; and a processor for processing the two-dimensional image using the restoration parameters generated by an external device that outputs the three-dimensional model; and 3. The three-dimensional image processing device according to claim 2,
Further comprising a decoding unit,
The decoding unit is
When an encoded three-dimensional model is generated by encoding the three-dimensional model by the external device, the encoded three-dimensional model is decoded to generate a decoded three-dimensional model;
When the external device generates encoded restoration parameters by encoding the restoration parameters, the external device decodes the encoded restoration parameters to generate decoded restoration parameters;
The rendering unit is
using preset rendering parameters, the three-dimensional model or the decoded three-dimensional model generated by the decoding unit to generate the two-dimensional image;
The two-dimensional image restoration unit
a decoding unit that generates a restored two-dimensional image by performing the restoration process on the two-dimensional image using the restoration parameters or the decoded restoration parameters generated by the decoding unit. 4. The three-dimensional image processing device according to claim 3,
The two-dimensional image restoration unit
When an input error occurs when the three-dimensional image processing device inputs the restoration parameters or the encoding restoration parameters,
performing the restoration process on the two-dimensional image using the restoration parameters previously input or the decoded restoration parameters previously generated by the decoding unit to generate the restored two-dimensional image;
When a decoding error occurs when the decoding unit decodes the encoding and decoding parameters,
a decoding unit that performs the restoration process on the two-dimensional image by using the decoded restoration parameters previously generated by the decoding unit, thereby generating the restored two-dimensional image. A three-dimensional image processing device generates restoration parameters for restoring degradation of a two-dimensional image generated by rendering a three-dimensional model representing the three-dimensional image, comprising:
a rendering unit that receives the three-dimensional model and renders the three-dimensional model using preset rendering parameters to generate the two-dimensional image;
a restoration parameter generating unit that uses provisional restoration parameters to perform a restoration process for restoring deterioration of the two-dimensional image generated by the rendering unit, generates a restored two-dimensional image, updates the provisional restoration parameters so as to reduce an error between the restored two-dimensional image and a predetermined learning image, and generates the updated provisional restoration parameters as the restoration parameters when a predetermined termination condition is satisfied;
a 3D image processing apparatus which outputs the 3D model and the restoration parameters generated by the restoration parameter generating unit. 6. The three-dimensional image processing device according to claim 5,
a three-dimensional model encoding unit that receives the three-dimensional model, performs source encoding, channel encoding, or both the source encoding and the channel encoding on the three-dimensional model, and generates an encoded three-dimensional model;
a restoration parameter coding unit that performs the information source coding, the channel coding, or the information source coding and the channel coding on the restoration parameters generated by the restoration parameter generating unit to generate encoded restoration parameters,
a 3D image processing apparatus which outputs the encoded 3D model generated by the 3D model encoding unit and the encoded restoration parameters generated by the restoration parameter encoding unit. 7. The three-dimensional image processing device according to claim 6,
Further, a three-dimensional model decoding unit is provided for decoding the encoded three-dimensional model generated by the three-dimensional model encoding unit to generate a decoded three-dimensional model,
The rendering unit is
2. A 3D image processing apparatus comprising: a 3D image processing unit that generates the 2D image by rendering the decoded 3D model generated by the 3D model decoding unit using preset rendering parameters. 6. The three-dimensional image processing device according to claim 5,
2. A three-dimensional image processing apparatus comprising: a processor for processing a three-dimensional image; a processor for processing the three-dimensional image; 8. The three-dimensional image processing device according to claim 6,
2. A three-dimensional image processing apparatus comprising: a processor for processing three-dimensional images, the processor outputting said encoded and restored parameters at a frequency lower than a frequency at which said three-dimensional model is inputted. 10. The three-dimensional image processing device according to claim 5,
The restoration parameter generation unit
inputting the two-dimensional image generated by the rendering unit and the learning image;
performing the restoration process for restoring degradation of the two-dimensional image using the provisional restoration parameters to generate a first restored two-dimensional image; performing a degradation process for degrading the first restored two-dimensional image using degradation parameters to generate a first degraded two-dimensional image;
performing the degradation process for degrading the learning image using the degradation parameters to generate a second degraded two-dimensional image; performing the restoration process for restoring degradation of the second degraded two-dimensional image using the tentative restoration parameters to generate a second restored two-dimensional image;
A three-dimensional image processing device characterized in that the provisional restoration parameters and the degradation parameters are updated so that an error between the two-dimensional image and the first degraded two-dimensional image, and an error between the learning image and the second restored two-dimensional image are reduced, and when a predetermined termination condition is satisfied, the updated provisional restoration parameters are generated as the restoration parameters. A program for causing a computer to function as a three-dimensional image processing device according to any one of claims 1 to 4. A program for causing a computer to function as a three-dimensional image processing device according to any one of claims 5 to 10.

Images