CN114677292A - High-resolution material recovery method based on two image inverse rendering neural network - Google Patents

Abstract

The embodiment of the present disclosure discloses a high-resolution material restoration method based on an inverse rendering neural network of two images. A specific implementation of the method includes: photographing to obtain an initial flash map and an initial guide map; performing segmentation processing on the initial flash map and the initial guide map; block; perform segmentation processing to obtain flash tile set and guide tile set; filter from flash tile set and guide tile set to obtain flash sub-tile set and guide sub-tile set; perform flash sub-tile set Rearrange to obtain a set of heavy lighting main tiles; generate a main material map group; based on each guide sub-tile, main material map group and guiding main tile in the guide sub-tile set, obtain a sub-material map group set; The set of sub-material map groups is spliced to obtain a high-resolution material map group. This implementation achieves the effect of reducing the consumption of video memory.

Description

High-resolution material restoration method based on two-image inverse rendering neural network

technical field

Embodiments of the present disclosure relate to the technical field of high-resolution material restoration, and in particular, to a high-resolution material restoration method based on an inverse rendering neural network for two images.

Background technique

Photorealistic graphics has a wide range of applications in fields such as video games and movies. Its main goal is to use the powerful computing power of computers to simulate the interaction process of lighting and scenes in the real world through algorithms, and finally generate a highly realistic image, in which materials express this interaction process. Material appearance modeling is used to represent the complex lighting effects of the interaction between light, material and geometry. This interaction effect needs to be represented in physics using a high-dimensional mathematical function. For opaque objects, this complex process is defined by a six-dimensional spatially varying bidirectional reflectance distribution function (SVBRDF). The high-dimensional nature of this function makes it an urgent technical difficulty to reconstruct realistic appearances using handheld devices.

The existing method of restoring the material reconstructs a poor appearance, which results in more artifacts in the surface appearance of the rendered plane material. When the high-resolution material is reconstructed, the rendered plane effect is extremely poor and cannot reach human vision. acceptable lighting effects. In addition, the existing method for recovering materials needs to shoot more than two images, which increases the complexity of shooting and requires high video memory.

SUMMARY OF THE INVENTION

This summary of the disclosure serves to introduce concepts in a simplified form that are described in detail in the detailed description that follows. The content section of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

Some embodiments of the present disclosure propose a high-resolution material restoration method based on an inverse rendering neural network of two images to solve one or more of the technical problems mentioned in the above background art section.

Controlling the mobile terminal to capture images to obtain an initial flash map and an initial guide map, wherein the initial flash map is an image taken by the mobile terminal when the flash is turned on, and the initial guide map is the ambient light of the mobile terminal when the flash is not turned on. The above-mentioned initial flash map and the above-mentioned initial guide map are divided into the above-mentioned initial flash map and the above-mentioned initial guide map are divided to obtain the flash map and guide map; the image areas are respectively selected from the above-mentioned flash map and the above-mentioned guide map, as the flash master tile and the guide master tile , wherein the positions of the flash main image block and the guide main image block in the image regions of the flash image and the guide image are matched; the flash image and the guide image are segmented to obtain a flash image block set and a guide tile set; randomly filter flash tiles and guide tiles from the above flash tile set and the above guide tile set as flash sub-tiles and guiding sub-tiles to obtain flash sub-tiles and guiding sub-tiles set; based on the above-mentioned guide sub-tile set, rearrange the above-mentioned flash sub-tile set to obtain a re-illumination main tile set; material map group; based on each guide sub-tile in the above-mentioned guide sub-tile set, the above-mentioned main material map group and the above-mentioned guiding main tile, generate a sub-material map group, and obtain a sub-material map group set; Each sub-material map group in the group set is spliced to obtain a high-resolution material map group.

Most real-world objects are composed of materials with a class of stationary spatial distribution (for example, stationary materials), which exhibit the same or similar materials and textures at different locations, that is, stationary properties. For example, wood, some metals, fabrics, leather, man-made materials, etc. Focus on sufficiently self-similar stationary material samples that have the property of having highly similar small neighborhoods at different locations. Since the same material has different incident light directions and viewpoint directions at different spatial locations, these different reflections can provide clues to missing features and ambiguities, so combining these different reflections can improve the fidelity of surface appearance reconstructions.

While some early methods can recover realistic surface appearance through dense sampling of controllable light sources and viewpoints, this process of recovering materials requires extreme acquisition conditions, long time-consuming and expensive equipment. Other methods can determine the material as accurately as possible by using a consumer-grade camera, but they still require taking a lot of images, adding complexity to the process of capturing images and reconstructing materials. In recent years, deep learning has facilitated the development of texture recovery from a single image. However, for stationary materials, these methods do not take advantage of the properties of having the same material properties in different locations to restore the material to reconstruct a more realistic surface appearance, so they restore poor materials resulting in rendered objects with more artifacts, and they The main purpose of reconstructing low-resolution materials is to reconstruct high-resolution materials. Due to the small receptive field, these training models often produce worse results, resulting in extremely poor rendering effects that are not acceptable to the human eye. surface appearance. Increasing the number of input images can provide more clues for the reconstruction process to resolve missing or ambiguous features, however these methods do not take advantage of stationary properties, resulting in inability to recover reasonable textures from fewer images and increasing the complexity of acquiring images . Moreover, when these methods are directly used to reconstruct high-resolution materials, the increase in the number and resolution of images will greatly increase the consumption of video memory, making it impossible for ordinary users to complete such tasks with computers with general configuration.

The present disclosure proposes a method for determining a high-resolution material from two images using the material self-similarity for stationary materials. In order to reduce the complexity of collecting images, the present disclosure captures two images, improves the fidelity of surface appearance reconstruction, and obtains more information from the two images, so as to support ordinary users to complete the restoration of high-resolution materials. Since only two images need to be taken, the complexity of image acquisition is reduced, high-resolution materials can be determined, and because the network processes low-resolution images, the need for high video memory is avoided, so the present disclosure is suitable for users with ordinary equipment. , and can determine a more plausible material from the two images to reconstruct a realistic surface appearance.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

1 is a flowchart of some embodiments of a high-resolution material restoration method based on two-image inverse rendering neural network according to the present disclosure;

2 is a schematic diagram of an application scenario of a high-resolution material restoration method based on two-image inverse rendering neural network according to some embodiments of the present disclosure;

3 is a schematic diagram of another application scenario of the high-resolution material restoration method based on two-image inverse rendering neural network according to some embodiments of the present disclosure;

FIG. 4 is a schematic diagram of yet another application scenario of the high-resolution material restoration method based on two-image inverse rendering neural network according to some embodiments of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings. The embodiments of this disclosure and features of the embodiments may be combined with each other without conflict.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or a plurality of". multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of these messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

FIG. 1 is a flowchart of some embodiments of a high-resolution material restoration method based on two-image inverse rendering neural network according to the present disclosure. The high-resolution material restoration method based on two-image inverse rendering neural network includes the following steps:

Step 101: Control the mobile terminal to capture images to obtain an initial flash map and an initial guide map.

In some embodiments, the execution subject of the high-resolution material restoration method based on the inverse rendering neural network of two images can control the mobile terminal to capture images through wired connection or wireless connection, and obtain the initial flash map and the initial guide map. The initial flash map is an image captured by the mobile terminal when the flash is turned on, and the initial guide map is an image captured by the mobile terminal under ambient light when the flash is not turned on.

In step 102, the initial flash map and the initial guide map are segmented to obtain the flash map and the guide map.

In some embodiments, the execution body may perform segmentation processing on the initial flash map and the initial guide map to obtain the flash map and the guide map.

Step 103 , respectively selecting image regions from the flash map and the guide map as the flash main image block and the guide main image block.

In some embodiments, the above-mentioned executive body may select image regions from the above-mentioned flash map and the above-mentioned guide map, respectively, as the flash main image block and the guide main image block. Wherein, the positions of the flash main block and the guide main block in the image areas of the flash map and the guide map are matched. For example, the positional coordinates of the flash main tile and the guide main tile in the image area in the flash map and the guide map are the same.

In step 104, the flash map and the guide map are segmented to obtain a set of flash blocks and a set of guide blocks.

In some embodiments, the execution body may perform segmentation processing on the flash map and the guide map to obtain a flash tile set and a guide tile set.

Step 105: Randomly filter flash tiles and guide tiles from the flash tile set and guide tile set as flash sub-tiles and guide sub-tiles to obtain a flash sub-tile set and guide sub-tile set.

In some embodiments, the execution subject may randomly select flash tiles and guide tiles from the flash tile set and the guide tile set as flash sub-tiles and guide sub-tiles to obtain the flash sub-tile set and Bootstrap sub-tile collection.

Step 106 , based on the guide sub-tile set, rearrange the flash sub-tile set to obtain the re-lighting main tile set.

In some embodiments, the above-mentioned execution body may rearrange the above-mentioned flash light sub-tile set based on the above-mentioned guide sub-tile set to obtain a relighting main tile set.

Step 107 , based on the flash map, the set of relighting main tiles, and the main flash tiles, generate a main material map group.

In some embodiments, the execution subject may generate a main material map group based on the flash map, the set of relighting main tiles, and the flash main tile. Among them, the material can refer to the lighting effect of the interaction between the light and the object. The material is a set of material maps with repeated textures, and the texture structure can be random or repeated. The material can also be expressed as a six-dimensional space-varying bidirectional reflection distribution function, and its parametric expression can be expressed as a multi-map (diffuse map: diffuse reflection map, normal map: normal map, roughness map: roughness map, specular map: specular map). Putting these maps and viewpoints and lights into the rendering equation renders the image.

In some optional implementation manners of some embodiments, the above-mentioned information extraction is performed on each development language information included in the above-mentioned development project information set to obtain a general command information set, which may include the following steps:

The first step is to input the above flash map into the pre-trained material generation model to obtain the initial high-resolution material map group.

In the second step, the above-mentioned initial high-resolution material map group is segmented to obtain an initial main material map group. The segmentation process may be an image segmentation process.

In the third step, a main material map group is obtained based on the above-mentioned initial main material map group, the above-mentioned re-lighting main tile set, the above-mentioned flash main tile and the pre-trained auto-encoder.

Optionally, the above-mentioned pre-trained auto-encoder includes an encoder and a decoder; and the above-mentioned auto-encoder based on the above-mentioned initial main material map group, the above-mentioned re-lighting main tile set, the above-mentioned flash main tile and the pre-trained auto-encoder is obtained. The main material graph group, which can include the following steps:

In the first step, the above-mentioned initial main material map group is input to the above-mentioned encoder and the above-mentioned decoder module, and an initial decoded main material map group is obtained. Wherein, the above-mentioned decoder module includes a parameter optimization module and a decoder.

The second step is to perform the following inverse rendering steps based on the above-mentioned initial decoded main material map group and the above-mentioned decoder module:

其中，L_train表示自动编码器使用的损失函数。L_map表示材质贴图的损失值。L_render表示确定出的材质渲染的9张图取对数后的损失值。A joint loss value is generated based on the above-mentioned initial decoded main texture map group, the above-mentioned re-lighting main tile set, and the above-mentioned flash main tile. In response to determining that the joint loss value has converged to a predetermined threshold, the initially decoded set of primary texture maps is used as the set of primary texture maps. In response to determining that the above joint loss value does not converge to the predetermined threshold, the latent vector parameters in the decoder module are adjusted, the adjusted decoder module is used as the decoder module, and the adjusted latent vector parameters are input to the decoder module to obtain the initial value. Decode the main texture map group and perform the above inverse rendering steps again. Among them, in order to obtain the latent space and decode the output texture map from the latent space, the technique uses a fully convolutional autoencoder to model this latent space, which consists of an encoder E() and a decoder. D(), the encoder E() converts the texture into the corresponding latent space, and the decoder D() converts the latent space into the texture: z=E(s), s=D(z). Among them, s represents the material. z characterizes the corresponding latent space. The loss function used by the autoencoder consists of two parts:

where L _train represents the loss function used by the autoencoder. L _map represents the loss value of the material map. L _render represents the loss value after the logarithm of the 9 images rendered by the determined material.

Optionally, the above-mentioned generating a joint loss value based on the above-mentioned initial decoded main material map group, the above-mentioned re-illumination main tile set, and the above-mentioned flash main tile may include the following steps:

The first step is to render the above initially decoded main material map group under the same lighting conditions as each re-illuminated main tile in the above-mentioned re-illuminated main tile set to obtain a first rendered map set.

In the second step, a first rendering loss value is generated based on the above-mentioned first rendering image set and the above-mentioned relighting main tile set.

The third step is to render the above-mentioned initial decoded main material image group under the same lighting conditions as the above-mentioned main flash tile, to obtain a second rendered image.

The fourth step is to generate a second rendering loss value based on the second rendering image and the main flash tile.

The fifth step is to obtain a joint loss value based on the first rendering loss value, the second rendering loss value, the first preset parameter and the second preset parameter. The process of generating joint loss values can be seen in Figure 3.

Optionally, based on the above-mentioned first rendering loss value, the above-mentioned second rendering loss value, the first preset parameter and the second preset parameter, the joint loss value may be obtained through the following steps:

Based on the above-mentioned first rendering loss value, the above-mentioned second rendering loss value, the first preset parameter and the second preset parameter, the following formula is used to obtain the joint loss value:

表示上述第一渲染图集合中第i个第一渲染图。I¹表示上述重光照主图块集合中的重光照主图块。

表示上述重光照主图块集合中第i个重光照主图块。β表示上述第二预设参数。R²表示上述第二渲染图。I²表示上述闪光灯主图块。‖‖表示范数。

的取值为上述第一渲染图集合中第i个第一渲染图的像素值减去上述重光照主图块集合中第i个重光照主图块对应位置的像素值，然后取绝对值并相加，最后除以像素的个数得到的值。‖R²-I²‖的取值为上述第二渲染图的像素值减去上述闪光灯主图块对应位置的像素值，然后取绝对值并相加，最后除以像素的个数得到的值。where L represents the above joint loss value. α represents the above-mentioned first preset parameter. R ¹ represents the first rendering in the first rendering set. i represents the serial number.

Indicates the i-th first rendering in the first rendering set. I ¹ represents the relighted main tile in the above-mentioned relighted main tile set.

Indicates the i-th relighted main tile in the above relighted main tile set. β represents the above-mentioned second preset parameter. R ² represents the second rendering above. I ² represents the above-mentioned flash main tile. ‖‖ represents the norm.

The value is the pixel value of the i-th first rendering image in the above-mentioned first rendering image set minus the pixel value of the corresponding position of the i-th re-illumination main tile in the above-mentioned re-illumination main tile set, and then take the absolute value and add Add, and finally divide by the number of pixels to get the value. The value of ‖R ² -I ² ‖ is the pixel value of the second rendering image minus the pixel value of the corresponding position of the main flash block, then the absolute value is taken and added, and finally divided by the number of pixels. .

Step 108 , based on each of the guide sub-tiles, the main material map group and the guiding main tile in the guide sub-tile set, generate a sub-material map group to obtain a sub-material map group set.

In some embodiments, the above-mentioned execution body may generate a sub-material graph group based on each guiding sub-tile in the above-mentioned guiding sub-tile set, the above-mentioned main material graph group and the above-mentioned guiding main tile, and obtain a sub-material graph group set .

In some optional implementations of some embodiments, the above-mentioned generating a sub-material map group based on each guiding sub-tile in the above-mentioned guiding sub-tile set, the above-mentioned main material map group, and the above-mentioned guiding main tile may include: The following steps:

In the first step, a descriptor is generated based on each pixel in the above-mentioned guiding sub-block, and a descriptor set is obtained.

In the second step, a reference descriptor is generated based on each reference pixel point in the above-mentioned guiding main image block, and a reference descriptor set is obtained.

In the third step, for each descriptor in the above-mentioned descriptor set, a reference descriptor corresponding to the above-mentioned descriptor in the above-mentioned reference descriptor set is determined as a matching descriptor to obtain a matching descriptor set.

The fourth step is to generate a rearrangement map based on the position information of each pixel in the guide sub-block and the position information of the reference pixel corresponding to the matching descriptor set.

The fifth step is to generate a sub-material map group according to the above-mentioned rearrangement mapping table and the above-mentioned guide sub-tiles.

Optionally, generating a descriptor based on each pixel in the above-mentioned guiding sub-block may include the following steps:

The first step is to use Gaussian kernels with a window size of 16, a standard deviation of 4 and a window size of 32 and a standard deviation of 8 to perform Gaussian smoothing on the above-mentioned guide sub-blocks to obtain the first matching sub-block and the second matching sub-block. Sub block.

The second step is to select 33×33, 65×65, and 5×5 red, green, and blue pixel neighbors from the first matching sub-block, the second matching sub-block, and the guiding sub-block, respectively. domain, and select 128, 96, and 32 point pairs from 33×33, 65×65, and 5×5 red, green, and blue pixel neighborhoods, respectively, using a binary robust independent base feature descriptor method. Among them, the binary robust and independent basic feature descriptor method may be point pair selection using Brief G II. BRIEF G II can be a binary robust independent base feature G II.

In the third step, 128-bit, 96-bit and 32-bit descriptors are determined based on the selected 128, 96 and 32 point pairs.

The fourth step is to connect the determined 128-bit, 96-bit and 32-bit descriptors to obtain a 768-bit descriptor.

Optionally, generating a reference descriptor based on each reference pixel in the above-mentioned guiding main image block may include the following steps:

The first step is to use Gaussian kernels with a window size of 16, a standard deviation of 4 and a window size of 32 and a standard deviation of 8, respectively, to perform Gaussian smoothing on the above-mentioned guiding main tiles to obtain the first matching main tile and the second matching. main tile.

The second step is to select 33×33, 65×65, and 5×5 red, green, and blue pixel neighbors from the first matching main tile, the second matching main tile, and the guiding main tile, respectively. domain, and select 128, 96, and 32 point pairs from 33×33, 65×65, and 5×5 red, green, and blue pixel neighborhoods, respectively, using a binary robust independent base feature descriptor method.

In the third step, 128-bit, 96-bit and 32-bit descriptors are determined based on the selected 128, 96 and 32 point pairs.

The fourth step is to connect the determined 128-bit, 96-bit and 32-bit descriptors to obtain a 768-bit descriptor.

Optionally, for each descriptor in the above-mentioned descriptor set, determining a reference descriptor corresponding to the above-mentioned descriptor in the above-mentioned reference descriptor set as a matching descriptor may include the following steps:

The first step is to determine the Hamming distance between the above-mentioned descriptor and each reference descriptor in the above-mentioned reference descriptor set to obtain a Hamming distance set.

The second step is to sort each Hamming distance in the above Hamming distance set in ascending order to obtain a Hamming distance sequence.

In the third step, the reference descriptor corresponding to the smallest Hamming distance in the above Hamming distance sequence is determined as a matching descriptor.

Step 109 , splicing each sub-material graph group in the sub-material graph group set to obtain a high-resolution material graph group.

In some embodiments, the above-mentioned execution body may splicing each sub-material graph group in the sub-material graph group set to obtain a high-resolution material graph group.

Optionally, the above-mentioned executive body may also send the above-mentioned high-resolution material map group to a three-dimensional modeling device, so that the three-dimensional modeling device can construct a three-dimensional model.

FIG. 2 is a schematic diagram of an application scenario of a high-resolution material restoration method based on two-image inverse rendering neural network according to some embodiments of the present disclosure.

In order to reduce the estimation of materials that support high-resolution images, and reduce the consumption of video memory, the complexity of capturing images is reduced. From two images, reconstruct high-quality materials using material properties and the powerful learning ability of deep learning. This technology proposes to use material characteristics, based on the depth inverse rendering framework, to determine from two images (one can be taken with the flash of the mobile phone, that is, the initial flash image, and one can be taken under ambient light, that is, the initial guide image) to determine Process method for high resolution materials. First, in order to exploit structurally repeated or randomly repeated textures, a reflection sample transfer method is introduced to combine multiple observations into a single master tile. Specifically, two images with a resolution of 2048*2048 are cropped, and 256*256 pixels in the same position are randomly cropped from these two images as the main tiles of the flash map and the guide map. At the same time, the flash map and guide map can be divided into 64 tiles of 256*256 pixels respectively, and 32 tiles can be selected as sub tiles. The materials of the main and sub-tiles can be defined as main and sub-materials. The reflection sample transfer method converts the secondary tile into a rough primary tile, which is similar to re-rendering the primary tile in the same direction as the secondary tile's incoming and outgoing light, resulting in a heavily lit primary tile. Second, deep inverse rendering can be used to restore the material of the main tile. To initialize the latent vector of the autoencoder, a convolutional neural network with a single image as input is used to obtain an initial set of high-resolution texture maps from the flash map, and the initial set of primary texture maps is cropped from the initial set of high-resolution texture maps. as the input to the autoencoder. Additionally, a collection of heavy lighting main tiles and flash main tiles can be used as input to the loss function to reconstruct more reasonable main tile materials. The detail can then be reintroduced to the main material using a refinement post-processing step. Finally, all sub-materials can be generated from the main material using the reverse reflection sample transfer method, and when these sub-materials are spliced to obtain a high-resolution material, in order to remove random artifacts in the high-resolution material and edge artifacts in the main tile area, in During the retroreflection sample transfer, the primary material can be mapped to the secondary material using the skip rearrangement method. Refine the high-res material again.

Here is a detailed explanation of why the whole process can reconstruct reasonable high-resolution materials from two images while reducing memory consumption, loss function and jump rearrangement.

1. Reconstruction of high-resolution materials

The complete framework reduces memory consumption and the complexity of capturing images for determining textures for high-resolution images. To this end, we use texture similarity and the powerful learning ability of deep learning to reconstruct high-quality materials from two images. Here we mainly explain the reasons why the framework can effectively solve these problems.

(1) Fully convolutional networks trained on low-resolution image datasets can also process high-resolution images. However, the relatively small receptive field results in poor quality of determined high-resolution materials. For the material, the main tile can be mapped to the sub-tile by using the reverse transfer method, in order to be able to determine the high-resolution material from the two images, the determined main material can be similarly transferred to the sub-material. These sub-materials can then be stitched together to generate high-resolution materials. Finally, refinement is used to generate more reasonably high resolution materials. Therefore, it is critical to determine a reasonable master material to restore the high-resolution material.

(2) A single inference network manages to determine plausible materials from a single image or a fixed number of images. However, in order to support more images as input, these methods must greatly increase the memory consumption to train the network. Furthermore, due to the presence of noise and artifacts in relighted main tiles and these noise and artifacts are irregular, there is no relighted main tile dataset for training, and there is a lack of a large amount of data with texture-similar features. These issues prevent training the inference network to determine the primary material. In addition, slight movement of the flash can change the local lighting conditions, thereby significantly changing the brightness of the patch, and this significant change in the brightness of the patch brings artifacts to the determination results of the inference network. In order to determine a reasonable main material and reduce the consumption of video memory, the depth inverse rendering method is used to determine the main material. Use only a single inference network to determine the initial high-resolution material for a single flash image, and use a small-resolution relighted main tile and the original main tile to determine the main tile material, so the memory consumption does not scale with the image resolution. increase and rise. To be precise, for two images with a resolution of 2560*2560, the video memory required by the network can be 7G.

2. Loss function

It is necessary to determine a more reasonable main material as much as possible, because the main material will affect the quality of the determined high-resolution material. The intuitive approach is to directly implement a differentiable rendering loss L _relit , which renders the predicted material map s under the same lighting conditions as the relighted main tile, and computes these rendering maps R(s, L(i)) and First normal form L1 difference between relighted master tile sets I _i :

L _relit =||R(s, L _i )-I _i ||.

However, the imperfect match between the points of the main tile and the pixels of the sub-tile introduces noise and artifacts to relighting the main tile. These rough relight master tiles can also add noise to certain master materials. While the latent space can regularize the optimization, it is not sufficient to reduce the interference of noise and artifacts. To further determine a more reasonable primary material, another rendering loss, _Lorig , is added, which computes the first normal form between the flash primary tile I _orig and the tile R(s, _Lorig ) rendered with the same lighting conditions _Lorig L1 Difference:

L _orig =||R(s, L _orig )-I _orig ||.

While L _relit 's formulation generates a main tile material with noise and artifacts, it recovers coarse-scale structural information. Based on this rough result, _Lorig 's formulation models the detailed texture constraints to reduce the interference of noise and artifacts, obtaining the best reconstruction results by using a joint loss function:

L=α×L _relit +β×L _orig .

where L represents the above joint loss value. α represents the above-mentioned first preset parameter. β represents the above-mentioned second preset parameter. L _relit represents the L1 difference between the rendering map R(s, L(i)) and the relit master tile set. _Lorig represents the first normal form L1 difference between the flash main tile I _orig and a tile R(s, _Lorig ) rendered using the same lighting condition _Lorig .

The entire loss function graph is shown in Figure 3. The premise of reducing noise and artifacts to determine reasonable fine-scale information is to restore the coarse-scale structure. Therefore, the value of α needs to be set larger than β to perform coarse-to-fine optimization. As a rule of thumb, set α to 6 and β to 1 to balance the size of each loss. Among them, in order to refine the main tile material, a loss function similar to formula L can also be used for refinement, but α and β are set to 1.

3. Jump rearrangement

The inverse transfer method introduces noticeable edge artifacts when the main material is transferred to a high resolution material. Observed that the recovered main tile texture maps have edge pixel values lower or higher than the average pixel value of the entire tile, this difference in pixel values is very noticeable in the diffuse and specular maps, and occasionally in roughness Figure. Specifically, when the average pixel value of the edge of the specular map is slightly lower, in order to ensure that the rendered image is the same as the input image, the average pixel value of the edge of the diffuse map generated by the network will be higher. Define lower or higher pixels as high-low artifacts. A rearrangement-based inverse transfer method maps the main material to sub-materials and stitches these sub-materials together to generate high-resolution materials. However, using the rearrangement method introduces artifacts into the secondary material. Specifically, it can be assumed that the sub-tile contains a part of the main tile. In this part of the area, the rearranged sub-tile is equivalent to copying a part of the main tile. At this time, the main tile in the generated high-resolution material is The edge pixel values of the tile area are either all higher than the surrounding area, or all lower than the surrounding area, so the high-low artifact appears obviously at the edge of the main tile area.

To remove these high and low artifacts (eg, square and random artifacts), the present technique proposes a skip rearrangement method to generate a rearrangement map from primary tiles to secondary tiles. The process of jump rearrangement is shown in Figure 4. First, the central area of the guide map is cropped as the guide main tile (the cropped area can be 216×216 pixels). For each pixel point A of the guide sub-block, find the pixel point B with the most similar material in the guide main block, and then record the position information of these two points, which uses the binary robust independent basic feature description Descriptor (BRIEF descriptor) to judge the similarity of two points (A and B). BRIEF descriptors can be descriptors of feature points. The implementation method of this process is as follows: considering the similarity of multiple scales, use Gaussian kernels with window sizes of 16, 32, and standard deviations of 4 and 8 to perform Gaussian smoothing on the guiding main block to reduce the interference of noise, plus Guide the main block to get three processed guide main blocks P1, P2, P3, for P1, P2, P3, for a pixel, select 33*33, 65*65 and 5*5 pixel neighborhoods to use Brief G II selects 128, 96, and 32 point pairs respectively, calculates 128-bit, 96-bit and 32-bit descriptors, repeats the above operations for the three color channels, and finally connects the descriptors to obtain 768-bit descriptors. The above operations are also performed on each pixel of the guide sub-block to obtain the corresponding descriptor. For a pixel in the guide sub-block, traverse all the pixels in the guide main block, and calculate the corresponding descriptor. Then calculate the Hamming distance of the two point descriptors. The two pixels with the smallest Hamming distance are the best matching points, so the position information of the two points with the smallest Hamming distance is stored. Based on this method, traverse all the pixels of the guide sub-block to find the corresponding best matching point from the guide main block, and record the corresponding position information to generate a rearrangement map. Then, use the rearrangement map to map the main material rearrangement to the secondary material. Figure 4 also shows the rearranged re-illuminated main tile, in which the re-arranged re-illuminated main tile and the re-arranged re-arranged re-illuminated main tile are compared for effects.

For all sub-tiles, use the same method to rearrange the map from the main material to the sub-material. Finally, when stitching all sub-materials, a window median filter is applied to clean up the boundaries between the stitched regions. Since this skip-rearrangement method does not select matching points from the edges of the main tile, this reduces square and random artifacts for the resulting high-resolution materials.

The invention can reduce the complexity of image acquisition, automatically reconstruct reasonable or accurate material maps, when the quality of the maps is very accurate, it can replace professionals to make materials, and when the quality of the maps is slightly poor, it can still provide benchmarks for personnel to create material maps and target clues, so as to assist professionals in making materials, and ultimately reduce the labor cost of material production. In addition, since only two images need to be taken, the complexity of image acquisition is greatly reduced, high-resolution materials can not only be determined, but also the consumption of video memory is very low. Therefore, the present invention is suitable for users with ordinary equipment. From the two images The quality of the determined materials is better, and the reconstructed surface appearance is more realistic, which is more conducive to assisting professionals to make materials.

The above descriptions are merely some preferred embodiments of the present disclosure and illustrations of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned inventive concept, the above-mentioned Other technical solutions formed by any combination of technical features or their equivalent features. For example, a technical solution is formed by replacing the above-mentioned features with the technical features disclosed in the embodiments of the present disclosure (but not limited to) with similar functions.

Claims (10)

1. A high-resolution material recovery method based on two image inverse rendering neural networks comprises the following steps:

controlling a mobile terminal to shoot an image to obtain an initial flash image and an initial guide image, wherein the initial flash image is an image shot by the mobile terminal when a flash is turned on, and the initial guide image is an image shot by the mobile terminal under ambient light when the flash is not turned on;

performing segmentation processing on the initial flash map and the initial guide map to obtain a flash map and a guide map;

Selecting image regions from the flash map and the guide map as a flash master tile and a guide master tile, respectively, wherein the positions of the image regions in the flash map and the guide map of the flash master tile and the guide master tile are matched;

the flash lamp graph and the guide graph are subjected to segmentation processing to obtain a flash lamp graph block set and a guide graph block set;

randomly screening the flash lamp pattern blocks and the guide pattern blocks from the flash lamp pattern block set and the guide pattern block set to serve as flash lamp auxiliary pattern blocks and guide auxiliary pattern blocks, and obtaining a flash lamp auxiliary pattern block set and a guide auxiliary pattern block set;

rearranging the flash lamp secondary image block set based on the guide secondary image block set to obtain a relighting main image block set;

generating a master material map group based on the flash map, the relighting master pattern set, and the flash master pattern;

generating an auxiliary material graph group based on each guide auxiliary graph block in the guide auxiliary graph block set, the main material graph group and the guide main graph block to obtain an auxiliary material graph group set;

and splicing each auxiliary material graph group in the auxiliary material graph group set to obtain a high-resolution material graph group.

2. The method of claim 1, wherein the method further comprises:

and sending the high-resolution material map group to three-dimensional modeling equipment so as to construct a three-dimensional model for the three-dimensional modeling equipment.

3. The method of claim 2, wherein the generating a master material map group based on the flash map, the relighting master tile set, and the flash master tile comprises:

inputting the flash lamp graph into a pre-trained material generation model to obtain an initial high-resolution material graph group;

segmenting the initial high-resolution material map group to obtain an initial main material map group;

and obtaining a main material image group based on the initial main material image group, the relighting main image block set, the flash lamp main image block and a pre-trained automatic encoder.

4. The method of claim 3, wherein the pre-trained auto-encoder comprises an encoder and a decoder; and

obtaining a master material map group based on the initial master material map group, the relighting master pattern set, the flash master pattern block, and a pre-trained auto-encoder, comprising:

inputting the initial main material picture group into the encoder and the decoder module to obtain an initial decoding main material picture group, wherein the decoder module comprises a parameter optimization module and a decoder;

Based on the initial decoding main material graph group and the decoder module, performing the following inverse rendering steps:

generating a joint penalty value based on the initial decoding master texture map group, the relighting master tile set, and the flash master tile;

in response to determining that the joint loss value converges to a predetermined threshold, treat the initial decoded master material map group as a master material map group;

in response to determining that the joint loss value does not converge to a predetermined threshold, adjusting potential vector parameters in a decoder module, taking the adjusted decoder module as the decoder module, inputting the adjusted potential vector parameters to the decoder module to obtain an initial decoded main material map group, and performing the inverse rendering step again.

5. The method of claim 4, wherein said generating a joint penalty value based on said initial decoding master texture map group, said relighting master tile set, and said flash master tile comprises:

rendering the initial decoding main material graph group under the same illumination condition with each relighting main graph block in the relighting main graph block set to obtain a first rendering graph set;

generating a first rendering penalty value based on the first rendering graph set and the relighting master graph set;

Rendering the initial decoding main material graph group under the same illumination condition with the flash lamp main graph block to obtain a second rendering graph;

generating a second rendering penalty value based on the second rendering map and the flash master tile;

and obtaining a joint loss value based on the first rendering loss value, the second rendering loss value, the first preset parameter and the second preset parameter.

6. The method of claim 5, wherein the deriving a joint penalty value based on the first rendering penalty value, the second rendering penalty value, a first preset parameter, and a second preset parameter comprises:

based on the first rendering loss value, the second rendering loss value, the first preset parameter and the second preset parameter, obtaining a joint loss value by using the following formula:

wherein L represents the joint loss value, α represents the first preset parameter, and R¹Representing a first rendering in the first set of renderings, i representing a sequence number,

representing the ith first rendering in the first rendering set, I¹Representing a relighting master tile in the set of relighting master tiles,

represents the ith relighting master pattern block in the relighting master pattern set, beta represents the second preset parameter, R ²Represents the second rendering graph, I²Represents the flash master pattern, | represents a norm,

the value of (a) is the value obtained by subtracting the pixel value of the corresponding position of the ith relighting main block in the relighting main block set from the pixel value of the ith first rendering in the first rendering set, then taking absolute values and adding the absolute values, and finally dividing the absolute values by the number of the pixels, | R²-I²The value of | is obtained by subtracting the pixel value of the corresponding position of the flash light master block from the pixel value of the second rendering block, then taking absolute values and adding the absolute values, and finally dividing the absolute values by the number of pixels.

7. The method of claim 6, wherein the generating a set of secondary material maps based on each of the set of guide secondary tiles, the set of primary material maps, and the guide primary tile comprises:

generating descriptors based on each pixel point in the guide sub-image blocks to obtain descriptor sets;

generating a reference descriptor based on each reference pixel point in the guide master image block to obtain a reference descriptor set;

for each descriptor in the descriptor set, determining a reference descriptor corresponding to the descriptor in the reference descriptor set as a matching descriptor to obtain a matching descriptor set;

Generating a rearrangement mapping table based on the position information of each pixel point in the guide auxiliary image block and the position information of the reference pixel point corresponding to the matching descriptor set;

and generating an auxiliary material graph group according to the rearrangement mapping table and the guide auxiliary image block.

8. The method of claim 7, wherein the generating a descriptor based on each pixel point in the boot sub-tile comprises:

respectively using Gaussian kernels with the window size of 16, the standard deviation of 4, the window size of 32 and the standard deviation of 8 to conduct Gaussian smoothing on the guide sub-image blocks to obtain a first matching sub-image block and a second matching sub-image block;

selecting three pixel neighborhoods 33 x 33, 65 x 65 and 5 x 5 red, green and blue from the first matching sub-tile, the second matching sub-tile and the guiding sub-tile, respectively, and selecting 128, 96 and 32 point pairs from the three pixel neighborhoods 33 x 33, 65 x 65 and 5 x 5 red, green and blue, respectively, using a binary robust independent primitive feature descriptor method;

determining 128-bit, 96-bit, and 32-bit descriptors based on the selected 128, 96, and 32 point pairs;

the determined 128-bit, 96-bit and 32-bit descriptors are concatenated, resulting in a 768-bit descriptor.

9. The method of claim 8, wherein the generating a reference descriptor based on each reference pixel point in the guidance master tile comprises:

respectively using Gaussian kernels with the window size of 16, the standard deviation of 4, the window size of 32 and the standard deviation of 8 to conduct Gaussian smoothing on the guide master image block to obtain a first matching master image block and a second matching master image block;

selecting three 33 x 33, 65 x 65 and 5 x 5 pixel neighborhoods of red, green and blue from the first matching master tile, the second matching master tile and the guide master tile, respectively, and selecting 128, 96 and 32 point pairs from the three 33 x 33, 65 x 65 and 5 x 5 pixel neighborhoods of red, green and blue, respectively, using a binary robust independent primitive descriptor method;

determining 128-bit, 96-bit, and 32-bit descriptors based on the selected 128, 96, and 32 point pairs;

the determined 128-bit, 96-bit and 32-bit descriptors are concatenated, resulting in a 768-bit descriptor.

10. The method of claim 9, wherein the determining, for each descriptor in the set of descriptors, a reference descriptor in the set of reference descriptors that corresponds to the descriptor as a matching descriptor comprises:

Determining Hamming distances between the descriptors and the reference descriptors in the reference descriptor set to obtain a Hamming distance set;

sequencing each Hamming distance in the Hamming distance set from small to large to obtain a Hamming distance sequence;

and determining the reference descriptor corresponding to the minimum Hamming distance in the Hamming distance sequence as a matching descriptor.

Images