CN112634156B - A method for estimating material reflection parameters based on images captured by portable devices - Google Patents

Abstract

The invention discloses a method for estimating material reflection parameters based on images acquired by portable equipment. The method comprises the following steps: shooting a material environment light image and a flash lamp image; estimating the roughness, the specular reflection coefficient, the diffuse reflection coefficient and the normal mapping of the material according to the ambient light map; calibrating the direction, distance and real irradiance from pixel to the camera and the light source; solving a color channel correction vector between the ambient light map and the flash light map; and determining the positions of similar pixel points based on clustering, determining the final clustering class number by adopting a gradual iterative refinement mode, and endowing the points in each class with the same reflection parameters. The method can be used for conveniently estimating the SVBRDF parameters of the material, aiming at the phenomena of serious halation and the like of some material mapping images estimated by the existing neural network, the reflection parameters of the material are estimated again by shooting a material flashlight image and an environment light image, and the reality of rendering is improved.

Description

A method for estimating material reflection parameters based on images captured by portable devices

technical field

The invention relates to material reflection parameter estimation, in particular to a method for estimating material reflection parameters based on images collected by a portable device.

Background technique

In the real world, the appearance properties of objects under different lighting conditions and different viewing angles are determined by the material on the surface of the object. How to restore the material easily and efficiently is an important topic in computer graphics at present, which is widely used in visual effects, electronic business, product design and entertainment. However, digitally obtaining high-quality material appearances from the real world is still a challenging problem, and the biggest challenge is the great complexity of material appearances in the real world. Objects in the real world are generally composed of multiple materials, and the surfaces of the materials have complex geometry, which increases the difficulty of acquisition and modeling.

When the light emitted by the light source hits the surface of the object, part of the light will be absorbed, and another part of the light will be reflected by the surface. BRDF is defined as the ratio between the outgoing luminance L along the ω _o : (θ _o , φ _o ) direction and the incident illuminance E along the ω _i : (θ _i , φ _i ) direction at a certain wavelength at a point on the surface of the object:

Although the BRDF model can effectively represent the reflection characteristics of objects, its parameters do not contain positional information by definition. Therefore, the BRDF model is generally only suitable for representing single-color objects without subtle undulations on the surface, such as metals, opaque plastics, and single-color cloth. However, in the real world, there are very few materials that meet these characteristics, and there are subtle fluctuations and color changes on the surface of many objects. In order to describe such objects, a spatially varying BRDF (SVBRDF) model is proposed:

where p:=(x, y) represents the coordinates of the object surface point.

Therefore, SVBRDF (spatially-varying bi-directional reflectance distribution function) is generally used to describe the reflection characteristics of the object surface. How to acquire and model material SVBRDF has always been an important research topic. The current research focus is on using portable devices such as mobile phones to acquire images, and recovering the reflection characteristics of materials by acquiring a small number of images. Deschaintre et al. proposed a method to automatically extract the and understanding visual cues such as texture, highlights, and shadows so that the appearance properties of materials can be perceived from a single image. Such methods learn a series of priors and constraints by learning from a large dataset to obtain a visually plausible solution. However, a large number of tests show that this method is not robust, especially for materials such as specular materials whose brightness changes drastically with angle changes, the output results have obvious artifacts on the parameter map, and sometimes the roughness and The specular reflection coefficient estimate has a large deviation from the true value.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned defects, the present invention provides a method for estimating material reflection parameters based on images collected by a portable device. The method uses a portable device such as a mobile phone to take a few pictures, so as to solve the phenomenon of artefacts in the texture that is easy to occur in the Deschaintre method, and recreate the method. Estimate the specular parameters and roughness of the material, so that the recovered material image is closer to the real image. This method takes an ambient light image and a flash image to re-estimate the reflection parameters of the material.

The present invention adopts the following technical solutions to realize:

A method for estimating material reflection parameters based on an image collected by a portable device. The SVBRDF of a material is estimated by taking a flash image and an ambient light image, which mainly includes the following steps:

Combined with the auxiliary calibration board, take a picture of the material under ambient light and flash light respectively; input the ambient light map into the material estimation network (such as the method of perceiving the appearance properties of the material from a single image proposed by Deschaintre et al) to estimate Roughness, specular reflection coefficient, diffuse reflection coefficient, and normal four maps of the material; calibrate the direction, distance and true irradiance of the flash map and ambient light map to the camera and light source pixel by pixel; solve the relationship between the ambient light map and the flash map. The color channel correction vector between them; the clustering-based method determines the position of similar pixel points and adopts the method of step-by-step iterative refinement to determine the final number of cluster classes, and assigns the same reflection parameters to the points in each class.

As an improvement of the present invention, a fitting method is used to solve the color channel correction vector between the ambient light image and the flash image. Due to the different light fields, the colors of the ambient light image and the flash image have obvious color casts, and the part that determines the color of the image It is the diffuse reflection part, so a proportional coefficient is added to the three color channels of the diffuse reflection term during fitting, so that the tone of the diffuse reflection part of the ambient light is consistent with the tone of the diffuse reflection part of the flash map.

As another improvement of the present invention, the clustering category is determined in a step-by-step iterative refinement manner. First, the categories in the image are initially divided into fewer categories (such as k=[3:10]), and then iteratively refined step by step, and the result of the parent category is used as the initial value of the subcategory until the conditions for continuous subdivision are no longer satisfied. .

The beneficial effects of the present invention are:

The invention adopts the fitting method to solve the color channel correction vector between the ambient light image and the flash image, so that the color tone of the diffuse reflection part of the ambient light and the diffuse reflection part of the flash image are kept consistent, which plays a role in the white balance of the image; The clustering method of gradual iterative refinement is used to determine the number of pixel categories in the image, and a balance is made between retaining the sense of detail of the image and the robustness of the fitting, solving the problem that the number of clusters in the existing method is difficult to determine. question. Compared with the SVBRDF that only estimates the material based on the neural network method, the method of the present invention can avoid the artifacts caused by the flash light spot in the Deschaintre method, estimate a more reasonable specular reflection coefficient and roughness, and make the material rendering results and real results more accurate. near.

Description of drawings

The specific embodiments of the present invention will be described in further detail below in conjunction with the accompanying drawings;

Fig. 1 is the flow chart of the present invention based on the portable device collecting a small amount of images to estimate the material reflection parameter;

Fig. 2 is the picture that the 4 kinds of materials collected by the present invention are respectively photographed under ambient light and flash;

FIG. 3 is a comparison diagram of the normal, diffuse, roughness and specular maps recovered by the present invention and the existing method on four materials.

FIG. 4 is a comparison diagram of the rendering results of the present invention and the existing method on four materials under the same light source.

Detailed ways

As shown in FIG. 1 , the present invention provides a method for estimating material reflection parameters based on a small amount of images collected by a portable device. The specific steps are as follows:

1. Shooting of material ambient light map and flash map.

1.1 Shoot in the RAW image mode of the camera to ensure that the image does not undergo nonlinear transformation in the ISP process.

1.2 Place the auxiliary calibration paper in the area of the material to be collected, and take an image under ambient light, which is marked as I ₁ . Keep the position of the auxiliary calibration paper unchanged, keep the mobile phone flash turned on, and take an image and mark it as I ₂ . When shooting, the distance between the mobile phone and the material is as close as possible, so that the range of angle changes is large.

2. Input the ambient light map into the material estimation network to estimate the roughness, specular coefficient, diffuse coefficient and normal map of the material.

Input the ambient light image I ₁ into the literature [Deshaintre, Valentin, et al. "Single-imagesvbrdf capture with a rendering-aware deep network." ACM Transactions onGraphics(ToG) 37(4), 1-15, 2018.] In the proposed network, diffuse map, roughness map, specular map and normal map are obtained, denoted as ρ _d , α, ρ _s and n, respectively. Among them, the diffuse reflection map ρ _d and the normal map n are close to the real value, and the two maps are used as the final result to re-estimate ρ _s and α.

3. Pixel-by-pixel calibration of the direction, distance and true irradiance to the camera and light source.

The camera response curve f of the mobile phone is calibrated to estimate the true irradiances E ₁ and E ₂ corresponding to the images I ₁ and I ₂ . The mapping relationship between the ambient light intensity and the camera exposure time to the image pixel brightness is:

I=f(X)=f(EΔt)

Among them, for each pixel point on the image, E represents the irradiance irradiating the pixel point per unit time, that is, the ambient light intensity, X represents the total energy received by the pixel point within the exposure time Δt, and the photometric response function f represents the nonlinear mapping from X to pixel intensity I. f is invertible, and its inverse function is defined as:

g(I)=lnf ^-1 =ln(E)+ln(Δt)

The field of view FOV of the mobile phone is calibrated, which is used to estimate the direction l and distance d from each pixel in the image to the light source, and the direction v to the camera. Since it is impossible for the flash to fall in the exact center of the image, the centroid of the top 5% brightest points on the image is selected as the XY coordinate origin of the flash.

4. Solve for the color channel correction vector between the ambient light map and the flash map.

Due to the difference in the light field, the colors of the ambient light map and the flash map have obvious color casts, and the part that determines the color of the image is the diffuse reflection part, so a proportional coefficient is added to the three color channels of the diffuse reflection term during fitting, and the fitting formula is: :

Among them, [r, _g , b] is the proportional vector of the control color channel; r ² is the distance from the pixel to the light source; L _r is the irradiance received by the camera, and Li is the irradiance emitted by the light source.

In the flash map, ambient light is not explicitly modeled, but is implicitly integrated into the SVBRDF model.

Assuming that the roughness and specular reflection coefficient of all points in the whole image are consistent, use all points in the image to fit the channel coefficients [r, g, b], and the solution process adopts the Levenberg-Marquarelt algorithm.

5. The clustering-based method determines the positions of similar pixels and adopts step-by-step iterative refinement to determine the final number of clusters.

5.1 Use the clustering method to determine the location of similar points. The ambient light image I ₁ is used as the benchmark image for determining the category of pixel points. In order to better measure the distance of the color, the image is transferred from the RGB color space to the CIE 1976L*a*b* color space.

5.2 The K-means clustering method is used, and the centroid initialization method uses K-means++. In order to improve the operation speed, a clustering acceleration algorithm based on triangular inequality is used.

5.3 Determine the clustering category. First, the categories in the image are initially divided into fewer categories k=[3:10], and then iteratively refined step by step, and the parent category result is used as the initial value of the subcategory until the conditions for continued subcategory are no longer satisfied, which is Points in each class are assigned the same reflection parameters. The requirements that can be further subdivided are:

N _i >λ ₁ ·(w·h), λ ₁ ∈(0.05, 0.2)

D _i >λ ₂ ·min(w, h), λ ₂ ∈(0.2, 0.5)

Among them, Ni represents the number of pixels in the _{i-th cluster, D i represents} the average distance from each point in the i-th cluster to the centroid of the spatial distribution of all points in the cluster, w and h are the width and height of the image, respectively, λ ₁ and λ ₂ is the control coefficient.

Example 1

This embodiment mainly compares the existing method (for details, see [Deschaintre, Valentin, et al. "Single-image svbrdf capture with a rendering-aware deep network." ACM Transactions onGraphics (ToG) 37(4), 1-15, 2018 .]) and the estimation results of the method of the present invention in real materials are compared. Figure 2 shows images of four materials collected under ambient light and flash. Fig. 3 is a comparison between the present invention and the existing method. It can be found that the method of the present invention does not have the common artifacts in the existing method, and the specular map and roughness map estimated by the method are more in line with the objective reality. From Fig. 4 Compared with the existing method, the image re-rendered by this method is closer to the actual collected image.

The specific embodiments described above describe in detail the technical solutions and beneficial effects of the present invention. It should be understood that the above are only preferred embodiments of the present invention, and are not intended to limit the present invention. Anything within the scope of the principles of the present invention Any modifications, additions and equivalent substitutions made within the scope of the present invention shall be included in the protection scope of the present invention.

Claims (1)

1. a method for estimating material reflection parameters based on portable equipment collection image, is characterized in that, by taking a flash light map and ambient light map to estimate the SVBRDF of material, the method steps are as follows: (1) Estimate the roughness, specular reflection coefficient, diffuse reflection coefficient and normal map of the material according to the ambient light map; calibrate the direction, distance and true irradiance from the flash map and ambient light map to the camera and light source pixel by pixel; (2) Solve the color channel correction vector between the ambient light map and the flash map; (3) The method based on clustering determines the position of similar pixel points and adopts the method of gradual iterative refinement to determine the final number of clustering classes, and assigns the same reflection parameters to the points in each class; In the described step (2): a fitting method is used to solve the color channel correction vector between the ambient light image and the flash image, and in the fitting process, parameters for controlling the ratio are added to the three color channels of the diffuse reflection item, so that the environment The color tone of the diffuse reflection part of the light map is consistent with that of the diffuse reflection part of the flash map; The step (3) is specifically as follows: at the beginning, only the categories of the pixel points in the image are divided into fewer categories, and then iteratively refines step by step, and the result of the parent category is used as the initial value of the subcategory, until it is no longer satisfied to continue the refinement. The conditions of the points are assigned the same reflection parameters to the points in each category; Among them, the conditions for continued subdivision are: N _i >λ ₁ ·(w·h), λ ₁ ∈(0.05, 0.2) D _i >λ ₂ ·min(w, h), λ ₂ ∈(0.2, 0.5) Among them, Ni represents the number of pixels in the _{i-th cluster, D i represents} the average distance from each point in the i-th cluster to the centroid position of the spatial distribution of all points in the cluster, ω and h are the width and height of the image, λ ₁ and λ ₂ is the control coefficient.

Images