KR20240169499A - Learning method of determining light patterns for three-dimensional scenes, method and apparatus of three-dimensional scenes - Google Patents

Abstract

A method for determining a lighting pattern according to one embodiment comprises: configuring a data set by estimating a first surface normal vector of a three-dimensional object from a first image of the three-dimensional object whose surface normal information is known in advance; generating simulation images in which virtual lighting patterns based on a combination of basis images included in the data set are applied to the three-dimensional object; estimating a second surface normal vector for the three-dimensional object through reconstruction of a photometric stereo technique based on the virtual lighting patterns and simulation images corresponding to the virtual lighting patterns; and training a neural network to determine the lighting pattern based on a difference between the first surface normal vector and the second surface normal vector.

Description

{LEARNING METHOD OF DETERMINING LIGHT PATTERNS FOR THREE-DIMENSIONAL SCENES, METHOD AND APPARATUS OF THREE-DIMENSIONAL SCENES}

The disclosure below relates to a method for determining a lighting pattern for a three-dimensional scene, and a method and device for modeling a three-dimensional scene.

Reconstructing surface normals of real objects can be a crucial task in many applications, such as 3D reconstruction, relighting, and inverse rendering. For example, surface normals can be reconstructed by photometric stereo techniques, which utilize the intensity variation of scene points under different lighting conditions to reconstruct surface normals. However, conventional photometric stereo techniques cannot provide high-quality surface normals because they cannot determine the optimal lighting pattern and cannot handle artifacts caused by specular reflection.

A method for determining an illumination pattern according to one embodiment comprises the steps of: configuring a data set by estimating a first surface normal vector of a three-dimensional object from a first image of the three-dimensional object whose surface normal information is known in advance; generating simulation images in which virtual illumination patterns based on a combination of basis images included in the data set are applied to the three-dimensional object; estimating a second surface normal vector for the three-dimensional object through reconstruction of a photometric stereo technique based on the virtual illumination patterns and simulation images corresponding to the virtual illumination patterns; and training a neural network to determine the illumination pattern based on a difference between the first surface normal vector and the second surface normal vector.

The step of constructing the above data set may include the step of obtaining the first image of the three-dimensional object captured by ground illumination; and the step of estimating the first surface normal vector from the first image using a differentiable rendering technique.

The method for determining the above lighting pattern may further include a step of performing preprocessing to remove a specular component from the first image, and the step of acquiring the first image may include a step of configuring the data set from the preprocessed first image.

The step of performing the above preprocessing may include the step of predicting the positions of grid points displayed on the display device using a mirror; and the step of adjusting the position of the display device to be located at the positions of the grid points to remove a specular reflection component, thereby obtaining the first image.

The step of obtaining the first image by removing the above-described specular reflection component may include the step of optically distinguishing the above-described specular reflection component and the above-described diffuse reflection component from a first image of the three-dimensional object captured by using a polarizing camera; and the step of obtaining an image of the above-described diffuse reflection component from the first image of the three-dimensional object captured by removing the above-described specular reflection component as the first image.

The step of estimating the first surface normal vector may include a step of aligning a second image rendered through the differentiable rendering technique to be the same as the first image by optimizing translation parameters and rotation parameters of the three-dimensional object in a virtual environment; and a step of estimating the first surface normal vector based on the aligned first image and the second image.

The step of generating the above simulation images may include a step of synthesizing the simulation images by simulating the images captured for each of the virtual lighting patterns in a differentiable manner corresponding to the base images.

The step of synthesizing the above simulation images may include the step of synthesizing the simulation images by a weighted sum that multiplies RGB color intensities corresponding to at least a portion of each of the base images for each of the virtual lighting patterns.

The step of predicting the second surface normal vector may include a step of reconstructing a surface normal of the three-dimensional object using a display and a camera.

The step of predicting the second surface normal vector may include the step of reconstructing at least one of the surface normal and the diffuse albedo from simulated images corresponding to the virtual illumination patterns.

The step of reconstructing at least one of the surface normal and the diffuse albedo may include the steps of: setting the diffuse albedo to a maximum intensity among the simulation images; estimating the surface normal using a pseudo-inverse method based on the diffuse albedo set to the maximum intensity; estimating the diffuse albedo using the pseudo-inverse method for each RGB channel of the simulation images; and reconstructing at least one of the surface normal and the diffuse albedo through iterative estimation of the surface normal and the diffuse albedo.

The step of predicting the second surface normal vector may include a step of predicting the second surface normal vector by inputting the virtual lighting patterns and simulation images corresponding to the virtual lighting patterns into a linear system based on a photometric stereo technique.

According to one embodiment, a method for modeling a three-dimensional scene includes the steps of: obtaining lighting patterns corresponding to the three-dimensional target object using a learned neural network; capturing a target scene including the three-dimensional target object by the lighting patterns; and modeling a three-dimensional scene corresponding to the target scene by restoring a surface normal of the three-dimensional target object using a photometric stereo technique based on the lighting patterns.

The step of modeling the above three-dimensional scene may include the step of obtaining a diffuse reflection image corresponding to one of the lighting patterns by performing separation of a diffuse reflection component and a specular reflection component for each frame of the target scene; and the step of applying the photometric stereo technique to the diffuse reflection image to estimate a surface normal vector of the three-dimensional target object.

The above neural network may be trained by a data set configured by estimating a first surface normal vector of a three-dimensional object from a first image of the three-dimensional object whose surface normal information is known in advance.

According to one embodiment, a device for modeling a three-dimensional scene includes a communication interface for receiving lighting patterns corresponding to a three-dimensional target object; a camera for capturing a target scene including the three-dimensional target object by the lighting patterns; and a processor for modeling a three-dimensional scene corresponding to the target scene by restoring a surface normal of the three-dimensional target object through a photometric stereo technique based on the lighting patterns.

The above processor separates the diffuse reflection component and the specular reflection component for each frame of the target scene to obtain a diffuse reflection image corresponding to one of the lighting patterns, and applies the photometric stereo technique to the diffuse reflection image to estimate the surface normal vector of the three-dimensional target object.

The device for modeling the three-dimensional scene may further include a display for displaying at least one of the lighting patterns and the modeled three-dimensional scene.

The device for modeling the above three-dimensional scene may include at least one of a lighting stage, an imaging system including a handheld flash camera, a display camera system, a wearable device including smart glasses, a head mounted device (HMD) including an Augmented Reality (AR) device, a Virtual Reality (VR) device, and a Mixed Reality (MR) device, and a user terminal including a television, a smart phone, a personal computer, a tablet, and a laptop.

Figure 1 is a flowchart illustrating a method for determining a lighting pattern according to one embodiment.
FIG. 2 is a diagram illustrating an overview of a photometric stereo technique according to one embodiment.
FIGS. 3A to 3D are diagrams illustrating a mirror-based calibration and image preprocessing process according to one embodiment.
FIGS. 4A and 4B are diagrams for explaining a method of configuring a data set according to one embodiment.
FIG. 5 is a diagram for explaining a reconstruction method of a photometric stereo technique according to one embodiment.
Figure 6 is a flowchart illustrating a method for modeling a three-dimensional scene according to one embodiment.
FIG. 7 is a diagram showing a surface normal vector of an object restored from a photographed scene using a photometric stereo technique according to one embodiment.
FIG. 8 is a diagram illustrating a learning process for determining a lighting pattern according to one embodiment.
FIG. 9 is a block diagram of a device for modeling a three-dimensional scene according to one embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be implemented in various forms. Accordingly, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present disclosure includes modifications, equivalents, or alternatives included in the technical idea described in the embodiments.

Although the terms first or second may be used to describe various components, such terms should be construed only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, the terms "comprises" or "has" and the like are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but should be understood to not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in that phrase, or all possible combinations of them.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In describing with reference to the attached drawings, identical components are given the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

Fig. 1 is a flow chart illustrating a method for determining a lighting pattern according to one embodiment. In the following embodiments, each step may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each step may be changed, and at least two steps may be performed in parallel. Referring to Fig. 1, a pattern determination device according to one embodiment may output a lighting pattern through steps (110) to (150).

In step (110), the pattern determination device configures a data set by estimating a first surface normal vector of a three-dimensional object from a first image of a three-dimensional object whose surface normal information is known in advance. Here, the three-dimensional object whose surface normal information is known in advance may correspond to a 3D printed object described later, but is not necessarily limited thereto. The first surface normal vector may also be called a 'correct surface normal vector' in that it is for an actually photographed three-dimensional object.

Prior to configuring the data set in step (110), the pattern determination device can perform preprocessing to remove specular reflection components from the first image. During the preprocessing, the pattern determination device can perform image preprocessing for the first image together with calibration. When the preprocessing is performed, the pattern determination device can configure a data set from the preprocessed first image. The pattern determination device can estimate the positions of grid points displayed on the display device using a mirror, for example, by a mirror-based calibration technique, and adjust the position of the display to be located at the positions of the grid points. The pattern determination device can obtain the first image through preprocessing to remove specular reflection components from the image obtained by adjusting the position of the display. The method by which the pattern determination device performs the preprocessing will be described in more detail with reference to FIGS. 3a to 3d below.

The pattern determination device can acquire a first image of a three-dimensional object by ground illumination. The pattern determination device can estimate a first surface normal vector from the first image by a differentiable rendering technique. Here, the 'ground illumination' can be, for example, an OLAT (One-Light-A-Time) pattern in which each light source is turned on one by one at maximum intensity. The OLAT pattern can generally be used when the intensity of each light source is sufficient to provide light energy detected by a camera sensor without significant noise, such as a lighting stage. When the OLAT pattern is extended to a group of adjacent light sources, the light energy can be increased and the measurement noise can be reduced. At this time, a complementary pattern in which half of the lights are turned on and the other half are turned off for each three-dimensional axis can also be used as the ground illumination.

The pattern determination device can align the second image rendered through a differentiable rendering technique to be the same as the first image by determining the translation parameters and rotation parameters of the three-dimensional object in the virtual environment. The pattern determination device can estimate the first surface normal vector based on the aligned first and second images. The pattern determination device can generate a real photometric stereo data set having a known shape using the 3D printed object. The method by which the pattern determination device configures the data set is described in more detail with reference to FIGS. 4A and 4B below.

In step (120), the pattern determination device generates simulation images by applying virtual illumination patterns to a 3D object based on a combination of basis images (e.g., see basis images (430) of FIG. 4a) included in the data set. Since the basis images can generate virtual illumination patterns by combining the basis images, the basis images may also be called 'basis illumination images'. The pattern determination device can synthesize simulation images that simulate images captured for each virtual illumination pattern in a differentiable manner in response to the basis images. Since the pattern determination device can simulate various actual illumination conditions with only light calculations by utilizing the basis images, the optimization of a light and effective illumination pattern can be achieved. The pattern determination device can synthesize simulation images by, for example, a weighted sum that multiplies RGB color intensities corresponding to at least a part of each of the basis images for each virtual illumination pattern. A method of synthesizing simulation images will be described in more detail with reference to FIG. 5 below.

In step (130), the pattern determination device estimates a second surface normal vector for the three-dimensional object through reconstruction of photometric stereo based on virtual lighting patterns and simulated images corresponding to the virtual lighting patterns. The pattern determination device can reconstruct the surface normal by utilizing intensity changes of scene points under various lighting conditions by the photometric stereo technique. The technique of reconstructing the surface normal can be applied to various imaging systems (e.g., the imaging system (310)) including, for example, a lighting stage utilizing numerous point light sources, a handheld flash camera, and a display camera system. The pattern determination device can estimate the second surface normal vector by substituting the virtual lighting patterns and simulated images corresponding to the virtual lighting patterns into a linear system based on the photometric stereo technique.

Photometric stereo technique is a computer vision technique that estimates the surface normal of an object by observing the object under various lighting conditions. It may correspond to a technique that sequentially applies multiple lights or lighting patterns to a target object and extracts the three-dimensional shape of the object using at least three images acquired through a camera. In one embodiment, the 'lighting pattern(s)' may also be called 'display pattern' in that they are provided through a display such as a monitor, for example.

According to the photometric stereo technique, the more the number of lights, the more reliably the 3D shape of the object can be extracted. This can be based on the fact that the amount of light reflected from the surface of the object varies depending on the direction of the surface relative to the light source and the observer. The space of possible surface directions can be limited by measuring the amount of light reflected to the camera. Given enough light sources from various angles, the surface direction can be limited or over-limited to a single direction.

In particular, display photometric stereo techniques using a display as a light source can provide a versatile and accessible system. The display can be equipped with a large number of three-color pixels that can act as programmable point sources. In one embodiment, a differentiable display photometric stereo (DDPS) technique can be used to reconstruct high-quality surface normals using a display and a camera, such as a standard monitor. Instead of relying on handcrafted light patterns, the differentiable display photometric stereo (DDPS) technique can learn the light patterns using a differentiable framework and end-to-end optimization to reconstruct surface normals optimized for the target system. Hereinafter, for the convenience of explanation, the term 'differentiable display photometric stereo technique' will be abbreviated as 'photometric stereo technique'. Therefore, even if not otherwise described below, the term 'photometric stereo technique' can be understood to mean 'differentiable display photometric stereo (DDPS) technique'.

In one embodiment, a differentiable pipeline can be used that combines the formation of a ground-illuminated image with an optimization-based photometric stereo technique. The ground-illuminated model for the formation of a ground-illuminated image can operate by capturing the image by setting individual light sources to their maximum intensity while keeping other light sources off. The differentiable pipeline described above can enable efficient learning of the illumination pattern by allowing the final reconstruction loss to be propagated back to the illumination pattern.

The pattern decision device can optimize the illumination pattern by the reconstruction loss of surface normals using a differentiable framework of image formation and reconstruction by photometric stereo technique. Photometric stereo technique can optimize the illumination pattern for normal reconstruction. Photometric stereo technique can utilize ubiquitous LCD devices and their polarization states for display illumination. Photometric stereo technique can directly apply normal reconstruction loss to illumination learning by using 3D printed objects to narrow the gap between synthetic domain and real domain. The photometric stereo technique is described in more detail with reference to Fig. 2 below.

A pattern determination device can estimate a second surface normal vector by inputting virtual lighting patterns and simulation images corresponding to the virtual lighting patterns into a linear system based on a photometric stereo technique. The pattern determination device can reconstruct at least one of a surface normal and a diffuse albedo from the simulation images corresponding to the virtual lighting patterns. The pattern determination device can set the diffuse albedo to the maximum intensity among the simulation images. The pattern determination device can estimate the surface normal using a pseudo-inverse method based on the diffuse albedo set to the maximum intensity. The pattern determination device can estimate the diffuse albedo using the pseudo-inverse method for each RGB channel of the simulation images. The pattern determination device can reconstruct the surface normal and the diffuse albedo through iterative estimation of the surface normal and the diffuse albedo.

A reconstruction method of the photometric stereo technique and an example of a linear system based on the photometric stereo technique are described in more detail with reference to Fig. 5 below.

In step (140), the pattern determination device trains a neural network to determine an illumination pattern based on the difference between the first surface normal vector estimated in step (110) and the second surface normal vector estimated in step (130). The neural network may be, for example, a Deep Feedforward Network (DFN), a Convolutional Neural Network (CNN), or a Recurrent Neural Network (RNN), but is not necessarily limited thereto.

The pattern determination device can modify the initial illumination pattern by photometric stereo technique, for example, in the following two ways. The pattern determination device can adjust the area of the bright region to ensure proper image intensity capture from various angles. In addition, the pattern determination device can modify the color distribution of the initial illumination pattern by trichromatic photometric stereo to provide various illumination patterns for each color channel. The photometric stereo technique can utilize three-color illumination from various directions by spatially distributing RGB intensities to various regions. At this time, the overall shape of the illumination pattern can be determined at the early stage of the learning process, but is not necessarily limited thereto. The pattern determination device can output the illumination pattern determined by the learned neural network in step (140). At this time, the output illumination pattern can correspond to an illumination pattern with sufficient illumination for all pixels for reconstructing the surface normal.

FIG. 2 is a drawing (200) illustrating an overview of a photometric stereo technique according to one embodiment. In one embodiment, a differentiable display photometric stereo (DDPS) technique may be used, which is designed to achieve high fidelity surface normal reconstruction using a display and a camera.

The differentiable display photometric stereo (DDPS) technique may be composed of a dataset acquisition process (201), a pattern training process (203), and a testing process (205). The dataset acquisition process (201) and the pattern training process (203) may correspond to the illumination pattern determination method described above through Fig. 1. The testing process (205) may correspond to an inference process that models a 3D scene using the learned illumination pattern described through Fig. 6 below.

In the data set acquisition process (201), the pattern determination device can capture a 3D printed object using base lighting, and obtain a first surface normal vector of the object through differentiable rendering based on the captured image.

The pattern determination device can perform calibration and image preprocessing prior to the data set acquisition process (201). In the preprocessing process, the pattern determination device can estimate the positions of grid points displayed on the display using a mirror. The pattern determination device can perform mirror-based calibration to adjust the position of the display to be located at the positions of the grid points. The pattern determination device can acquire the first image by adjusting the position of the display device through calibration to remove the specular reflection component.

In addition, the pattern determination device can separate the specular component from the captured image by utilizing the polarization property. The mirror-based calibration and specular component separation method is described in more detail with reference to FIGS. 3a to 3d below.

In the data set acquisition process (201), the pattern determination device can perform 3D printing for various 3D models, capture base images of the 3D printed object, and obtain an actual surface normal map using the captured base images. At this time, the acquired actual surface normal map may correspond to the first surface normal vector.

The pattern determination device can construct a realistic training data set using 3D printed objects to alleviate the gap between the synthetic domain and the real domain that occurs due to the use of synthetic training data during end-to-end optimization. The pattern determination device can extract a real-world surface normal map (e.g., a first surface normal vector (210)) by fitting the real-world geometry of the 3D model to the captured image. The pattern determination device can effectively reduce the gap between the domains by combining the formation of the basis images (215) and the extraction of the real-world surface normal map (e.g., the first surface normal vector (210)).

Also, in one embodiment, by utilizing the fact that a display such as a monitor emits linear polarization, the linear polarization emitted by the display can be combined with a camera to optically filter out the specular reflection to extract an image ('diffuse reflection image') in which the diffuse reflection component is dominant from the captured base images (215). The pattern determination device can use the diffuse reflection image to satisfy the Lambertian assumption for the photometric stereo technique to more accurately reconstruct the surface normal (230). According to the cosine law of Lambertian, the radiant intensity when viewing a Lambertian surface can be proportional to the cosine of the angle formed with the normal direction of the surface. This can be understood to mean that the radiant intensity is greater when viewed from a side that is coincident with the normal than when viewed obliquely. A 'Lambertian surface' can mean a surface that appears to have the same brightness when viewed from all directions. Since a Lambertian surface reflects the same amount of light in all directions, light from all directions can appear to have the same brightness, regardless of the observer's position. Here, the 'Lambertian assumption' can be that the object has Lambertian surface properties.

Additionally, the pattern determination device can more accurately estimate pixel positions of a display (e.g., a monitor) for a differentiable display photometric stereo (DDPS) system through a mirror-based calibration technique described below.

In the pattern training process (203), the pattern determination device can simulate a new image through rendering (220) for virtual lighting patterns (235) by combining the base images (215) among the data sets acquired in the process (201). At this time, the image for the simulated virtual lighting patterns (235) can be called a 'simulation image' (225).

The pattern determination device can output a surface normal vector ('second surface normal vector') (240) for an object by substituting virtual lighting patterns (235) and simulation images (225) corresponding to the virtual lighting patterns (235) into an image formation formula approximated as a linear system (e.g., see Equation 8). This process may correspond to a photometric stereo technique. The pattern determination device can calculate a difference value (245) between the first surface normal vector (210) corresponding to the correct answer and the output surface normal vector ('second surface normal vector') (240), and backpropagate the difference value (245) to optimize the lighting pattern.

The pattern training process (203) may correspond to a process of training illumination patterns that lead to high-quality normal reconstruction (230) from a training data set. The pattern determination device may provide high-quality reconstruction (230) by optimizing the illumination pattern (235) by utilizing an image formation framework and a photometric stereo technique. According to one embodiment, a series of processes of simulating changes in a scene according to illumination patterns (235) and calculating surface normals of an object using the simulation results may be designed to be differentiable.

In the testing process (205), the pattern determination device can capture (250) an actual scene using the lighting patterns (235) optimized through the pattern training process (203). The pattern determination device can restore (reconstruct (230)) surface normals of objects in the scene (255) captured in the testing process (205) using the photometric stereo technique used in the pattern training process (203).

In the testing process (205), the pattern determination device can perform a test to estimate a surface normal for an object included in an actual scene by using the optimized illumination pattern(s) (235) through the pattern training process (203). At this time, the optimized illumination pattern(s) (235) can be converted and displayed as, for example, an 8-bit RGB pattern. The pattern determination device can capture (250) polarization images from, for example, various repeated illumination patterns (235). The pattern determination device can obtain a diffuse reflection image for the th illumination pattern by performing separation of a diffuse reflection component and a specular reflection component for each frame. The pattern determination device can estimate a surface normal by applying a photometric stereo technique to the diffuse reflection image.

FIGS. 3A to 3D are diagrams for explaining a mirror-based calibration and image preprocessing process according to one embodiment. Referring to FIGS. 3A and 3B, a diagram (320) is illustrated for illustrating an imaging system (310) for mirror-based calibration according to one embodiment and a geometric calibration process based on the mirror (323). Referring to FIG. 3C, a diagram (330) is illustrated for illustrating a process for separating a specular component by utilizing polarization properties according to one embodiment. Referring to FIG. 3D, a diagram (350) is illustrated for explaining a process for obtaining an image from which a specular component is removed by using a polarization image according to one embodiment.

The imaging system (310) may include a display (311) and a camera (313).

The display (311) may be, but is not necessarily limited to, a commercially available large-screen curved LCD (Liquid Crystal Display) monitor that emits linearly polarized light. Each pixel of the display (311) may emit horizontal linearly polarized light in the RGB spectrum in three colors due to the polarization-sensitive optical elements of the LCD. The pixels of the display (311) may be, but are not necessarily limited to, super-pixels of = 9Х16.

The camera (313) may be a polarization camera (313) such as that illustrated in FIG. 3c. The camera (313) may be, for example, a polarization camera having an on-sensor linear polarization filter.

The camera (313) can capture images (I 0° , I 45° , I 90° and I 135° ) (351) of four linear polarization intensities at four different angles (e.g., ₀ °, ₄₅ °, _90° , 135°), for example, as illustrated in the drawing ( ₃₅₀ ) of FIG. 3d . In this case, the linear polarization emitted from the display (311) can interact with the actual scene to generate both specular and diffuse reflection components for the surface point. The specular reflection component tends to maintain the polarization state of the light, whereas the diffuse reflection component can often be unpolarized.

In one embodiment, it can be assumed that the spectral distributions of the camera (313) and the display (311) match. Depending on the embodiment, a spectral blocking filter can be additionally used in front of the camera (313).

Additionally, since one embodiment relies on the assumption of planar geometry of arbitrary vertices of the target scene (e.g., the assumption that the spectral distributions of the camera (313) and the display (311) match), biased estimation may be made for scenes with distinct depth changes. In one embodiment, the biased estimation can be mitigated for scenes with distinct depth changes by using a re-view camera for depth estimation.

Referring to the drawings (320) of FIG. 3a and FIG. 3b, a mirror (323) based geometric calibration process is illustrated. In one embodiment, a mirror (323) based calibration method may be used to estimate the unique parameters of the camera (313) and the position of each pixel of the display (311) with respect to the camera (313).

The pattern determination device can display white pixel grid points (321) on the display (311) to place a mirror (e.g., a flat mirror) (323) in a specific pose (e.g., a specific angle) in front of the camera (313). The pattern determination device can capture an image (324) of the mirror (323) reflecting some of the grid points of the grid pattern (321) to which pixel coordinates of the display (311) are assigned.

More specifically, the pattern determination device can calibrate the three-dimensional position of the mirror (323) using a checkerboard. The pattern determination device can capture an image reflected on the display (311). The pattern determination device can estimate the three-dimensional coordinates of the pixels of the corresponding display (311) using the white pixel grid points (321) displayed on the mirror (323). The pattern determination device can fit a curved plane to the calibrated vertices and sample the pixels of the display (311) from the fitted plane. The pattern determination device can generate multiple pairs of checkerboard images (325) and mirror images (326) reflecting the grid points by repeating this procedure while varying the pose (e.g., angle) of the mirror (323). The pattern determination device can estimate the unique parameters of the camera (313) and the three-dimensional pose of each checkerboard from the checkerboard image (325). The pattern determination device can detect three-dimensional vertices of grid points displayed on each mirror image (326) with the size of a known display (311) and obtain three-dimensional vertices of pixels of the display (311) through calibration. The pattern determination device can obtain pixel position information of the display (311) on the camera coordinate system through mirror-based calibration.

At this time, the pattern determination device can obtain a diffuse reflection image by removing the specular reflection component from the captured image by utilizing the polarization property as in the drawing (330) of Fig. 3c. The pattern determination device can optically distinguish the specular reflection component and the diffuse reflection component from the first image in which the three-dimensional object (331) is captured by using the polarization camera (313). The pattern determination device can analyze the polarization state of the incident light intensity by the polarization camera (313) and separate the diffuse reflection component and the specular reflection component according to the acquisition speed. The pattern determination device can decompose the diffuse reflection image and the specular reflection image by utilizing the linear polarization emitted from the display (e.g., LCD monitor) (311). The pattern determination device can obtain an image of the diffuse reflection component by removing the specular reflection component from the image ('first image') in which the three-dimensional object is captured as an image in which the three-dimensional object is captured. The pattern determination device can effectively reconstruct surface normals by applying the photometric stereo technique only to images in which the diffuse reflection component is dominant (e.g., image (315) or image (357)).

The pattern determination device can effectively reconstruct the surface normal by removing the specular reflection component from the image captured using a display (311) that emits polarized light and a camera (313) capable of capturing various linear polarization information in real time. The pattern determination device can derive the actual three-dimensional coordinates of each pixel of the display (311) through mirror-based calibration.

In one embodiment, when reconstructing surface normals using an image containing both diffuse and specular components, the instability in normal reconstruction caused by the specular component can be alleviated by reconstructing the surface normals from an image in which diffusion is dominant ('diffuse reflection image'). For the reconstruction of normals, the pattern determination device can convert raw images (I 0° , I 45 _° , I 90° and I ₁₃₅ ° ) (351) captured at four different angles (e.g., 0 _° , 45°, _90° , 135°) with four polarization intensity values, as shown in FIG. 3d , into linear Stokes vector elements s ₀ , s ₁ , s ₂ as shown in Equation 1 below.

Images converted into linear Stokes vector elements s ₀ , s ₁ , s ₂ can be referenced to the drawing (353) of Fig. 3d.

The pattern determination device can produce a diffuse reflection component and a specular reflection component S as shown in the mathematical expression 2 below.

Image (315, 355) may correspond to an image of the diffuse reflection component I, and image (317, 357) may correspond to an image of the specular reflection component S. The pattern determination device may apply the image (315, 357) of the diffuse reflection component I to the photometric stereo technique.

As described above, the pattern determination device can perform diffuse-reflective separation using polarized illumination of the display (211) and the imaging system (310).

The pattern determination device can estimate the positions of grid points displayed on the display device using a mirror. The pattern determination device can obtain a first image by adjusting the position of the display device to be located at the positions of the grid points and thereby eliminating the specular reflection component.

FIGS. 4A and 4B are diagrams for explaining a method of configuring a data set according to one embodiment.

Referring to FIG. 4A, a drawing (410) showing a 3D printed object (411) according to one embodiment, a correct image (413) corresponding to a result of rendering the 3D printed object (411), an average image (415) overlaid by a correct silhouette S, a ground-truth 3D model (417) as indicated by a silhouette fitted to the average image (415), and a correct ground-truth normal map (419) obtained from the fitted 3D D model, and ground images (430) of the 3D printed object are illustrated.

Referring to FIG. 4b, a drawing (450) is shown showing a process of estimating a correct surface normal vector ('first surface normal vector') of a corresponding object through differentiable rendering based on an image captured during a data set acquisition process according to one embodiment.

In one embodiment, a data set can be generated using 3D printing using known ground truth geometries. The pattern determination device can 3D print 11 different 3D models using, for example, an FDM-based 3D printer having a print resolution of 0.2 mm. The pattern determination device can use a variety of filaments (e.g., PLA, PLA+, Matte PLA, eSilk-PLA, eMarble-PLA, Gradient Matte PLA, PETG) to provide a variety of appearances in terms of color, scattering, and diffuse/reflective ratios.

The pattern determination device can construct a data set by estimating a first surface normal vector of a three-dimensional object from a first image of a three-dimensional object whose surface normal information is known in advance, such as a 3D printed object (411).

To construct a training scene, the pattern determination device can place some of the 3D printed objects in front of the imaging system. The pattern determination device can generate base images for each scene. (430) can be captured, where j can correspond to the index of the basis illumination at which the j-th superpixel is turned on with maximum intensity in white.

The pattern determination device can align the second image rendered through a differentiable rendering technique to be identical to the first image by optimizing the translation parameters and rotation parameters of the three-dimensional object in the virtual environment. The pattern determination device can estimate the first surface normal vector based on the aligned first and second images.

More specifically, the pattern determination device can overlay the correct silhouette (454) on the average image (453) of the basis images (430). The pattern determination device can extract the silhouette mask using the average image (I _avg ) (453) of the basis images (430) that exhibits a bright appearance for most points of the object scene, such as the average image overlaid by the correct silhouette S.

Given a silhouette mask S, the pattern determination device can align the ground-truth geometry of the 3D printed object in the scene. The pattern determination device can align the pose for estimation of the first surface normal vector.

The pattern decision device can optimize the pose of the ground-truth mesh of the object in the scene by minimizing the silhouette rendering loss compared to the silhouette mask S. The pattern decision device can optimize the pose of the ground-truth mesh of the object by gradient descent. In this case, the silhouette rendering loss may correspond to the difference between the true silhouette (454) of the average image (453) of the camera-captured basis images (430) and the rendered silhouette image (452) obtained from the rendered image (451).

At this time, the silhouette rendering loss is calculated as the mean squared error between the silhouette mask S and the rendered silhouette image (452), and this error can be backpropagated to optimize the location t and rotation r of the object. The pattern determination device can optimize the movement and rotation parameters of the object in the virtual environment through differentiable rendering so that the captured image (e.g., the average image (453) of the camera-captured basis images (430)) and the rendered image (451) become the same.

The above-described optimization process can be expressed, for example, as shown in mathematical expression 3 below.

Here, π can represent a 3D model of a 3D printed object in the scene. can represent a differentiable silhouette rendering function. The pattern determination device can use the calibration parameters of the camera in the virtual camera settings during rendering. The pattern determination device can solve Equation 3 using gradient descent.

When the pattern determination device obtains pose parameters for the 3D models, the pattern determination device can render a pose-aligned normal map (457) using the 3D models (455) at the optimized poses. At this time, the rendered normal map (457) can be used as a ground-truth normal map ( _GT ) for end-to-end optimization. The pattern determination device can generate a pose-aligned rendered image (456) using the rendered normal map (457). The pattern determination device can generate a blended image (458) by blending the pose-aligned rendered image (456) and the average image (I _avg ) (453) of the basis images (430) at a ratio of 1:1. The blended image (458) can be used to visually determine the alignment accuracy of the ground-truth normal map.

FIG. 5 is a diagram for explaining a reconstruction method of a photometric stereo technique according to one embodiment. Referring to FIG. 5, a diagram (500) is illustrated showing a differentiable image formation (510) process and a photometric stereo framework (530) according to one embodiment.

The pattern determination device can generate new simulation images (515) for virtual lighting patterns (513) by combining basis images (430) from the acquired data sets. The pattern determination device can generate the virtual lighting patterns (513) by, for example, an element-wise product operation for the basis images (430). The pattern determination device can estimate a surface normal vector (550) for an object by using a photometric stereo framework (530) by substituting the virtual lighting patterns (513) and the corresponding simulation images (515) into an image formation formula approximated by a linear system, as in, for example, Mathematical Formula 8 below. The photometric stereo framework (530) can correspond to a Differentiable Display Photometric Stereo (DDPS) framework.

The photometric stereo framework (530) can reconstruct surface normals for each pixel by utilizing changes in lighting conditions. The pattern determination unit can calculate the difference between the estimated surface normal vector (550) and the correct normal vector, and backpropagate the calculated difference to optimize the lighting patterns (513).

The pattern decision device can learn the illumination patterns (513) by directly penalizing the reconstruction loss of surface normals through a differentiable framework (530) based on the photometric stereo technique. The photometric stereo framework (530) can utilize the display as an active illumination module capable of generating spatially varying three-color intensity variations.

The pattern decision device can facilitate learning of illumination patterns leading to reconstruction of high-quality surface normals by using a differentiable framework that combines the formation of differentiable basis images (510) and the reconstruction of photometric stereo.

More specifically, the pattern decision unit learns illumination patterns that provide a reconstruction of surface normals using the ground truth normal map ( _GT ) and basis images. (430) A training data set of pairs is available.

For example, different lighting patterns of the dog (513) can be expressed as, where, the th illumination pattern M is a super pixel that is an optimization variable. can be modeled as a lighting pattern M _i having an RGB intensity pattern.

The pattern determination device is an RGB intensity pattern Differentiable image-forming functions that are linked together via automatic differentiation for end-to-end training of and differentiable photometric stereo functions can be used to reconstruct surface normals.

Here, the differentiable image forming function The lighting pattern M and the base images of the training scene Footage captured based on (430) can be simulated. The pattern determination device performs image simulation for K different lighting patterns (513) and simulated captured images. (515) can be obtained.

The pattern decision device is a photometric stereo function Simulated captured video using By processing (515), the surface normal N can be estimated. The pattern determination device is, for example, The surface normal N can be estimated by . Here, ρ represents the albedo, and n can represent the surface normal vector. Here, the albedo (ρ) is can be obtained as follows. Also, the surface normal vector (n) is can be obtained as follows. The surface normal (N) is It can be obtained as follows, Is It can be obtained by the product of light sources l and light source directions i. Also, L represents the output radiant (final color). It can be obtained as follows.

The pattern determination device compares the estimated surface normal (550) with the measured normal N _GT , and the loss according to the comparison result is converted into an illumination pattern through a differentiable flow. It can be backpropagated with the intensity of .

The pattern determination device determines the illumination pattern as in the mathematical expression 4 below, for example. can be optimized.

Here, may correspond to the angular difference between the estimated surface normal (550) and the actual surface normal. In one embodiment, equation (4) may be solved using stochastic gradient descent on a 3D printed data set, for example, using the Adam optimizer.

The pattern decision unit is the basis images of the training samples. Lighting patterns for (430) (513) The captured image is simulated in a differentiable manner as in the mathematical expression 5 below, thereby generating a simulated captured image ( ) can be created.

Here, is a lighting pattern can represent the RGB intensity of the th superpixel. are the underlying images (430) can represent the jth basis image.

The pattern decision device captures each simulated captured image ( ) corresponding to K total illumination patterns (513). ) can be synthesized as in mathematical formula 6 below.

The weighted-sum formulation used in the differentiable image formation (510) according to one embodiment can utilize basis images acquired for actual 3D printed objects based on the light transmission linearity of the ray optics system. Image formation using basis images in the differentiable image formation (510) can enable memory-efficient and effective image formation for synthesizing realistic images.

The pattern determination device has various lighting patterns. Images captured from below or simulated (515) The surface normal N can be reconstructed from the following mathematical expression 7.

Image due to separation of diffuse reflection component and specular reflection component by polarization (515) may contain a dominant diffuse reflection component.

In one embodiment, a trinocular photometric stereo technique can be used, which is independent of the training dataset and has no training parameters, by using optically separated diffuse reflectance images I in photometric stereo. The trinocular photometric stereo technique can help in efficient gradient updates for the illumination pattern during end-to-end learning.

For example, the captured diffuse RGB intensity of a camera pixel can be expressed as , where c is the color channel. may correspond to. For simplicity, In notation, dependency on pixels can be omitted.

Additionally, the illumination vector coming from the center of the jth superpixel of the display can be represented as . The surface normal N can be computed based on the reference-plane assumption that the scene point P corresponding to the camera pixel p lies in a plane 50 cm away from the camera.

The image formation formula approximated by a linear system can be expressed as mathematical expression 8 below.

Here, I represents the captured image, ρ can represent the primary color of the object, or in other words, the albedo, and N can represent the surface normal of the object. can represent the Hadarmard product. M can represent the color (or color intensity) of the pattern. l can represent a matrix for the direction of the light, that is, the incident vector of the light.

The pattern decision unit can estimate the surface normal N by setting the albedo ρ to the maximum intensity among the captured images and solving the linear system using the pseudo inverse matrix by the pseudo-inverse method. Once the surface normal N is estimated, the pattern decision unit can rewrite Equation 8 as Equation 9 below to solve the albedo ρ again.

Here, , is the original vector Wow matrix may correspond to a channel-specific version of .

The pattern decision device is for each channel Using the doctor's calendar for Albedo by channel as can be estimated.

The pattern decision unit can repeat normal estimation and albedo estimation for higher accuracy, and the reconstruction quality can be improved through repeated estimation.

The estimated surface normal (550) may correspond to the reconstruction result of the surface normal and albedo.

FIG. 6 is a flowchart illustrating a method for modeling a three-dimensional scene according to one embodiment, and FIG. 7 is a drawing illustrating a surface normal vector of an object restored from a photographed scene through a photometric stereo technique according to one embodiment.

Referring to FIGS. 6 and 7, a device for modeling a three-dimensional scene according to one embodiment (hereinafter, “modeling device”) can model a three-dimensional scene corresponding to a target scene through steps (610) to (630).

In step (610), the modeling device acquires lighting patterns corresponding to a three-dimensional target object using a learned neural network. At this time, the neural network may be learned by a data set configured by estimating a first surface normal vector of a three-dimensional object from a first image that captures the three-dimensional object whose surface normal information is known in advance. The lighting patterns acquired in step (610) may correspond to lighting patterns optimized through the aforementioned process.

In step (620), the modeling device captures a target scene including a three-dimensional target object as shown in the drawing (710) by the lighting patterns acquired in step (610).

In step (630), the modeling device models a three-dimensional scene corresponding to the target scene by restoring the surface normal of the three-dimensional target object as shown in the drawing (730) below through a photometric stereo technique based on illumination patterns. The three-dimensional scene corresponding to the target scene may include, for example, an estimated diffuse albedo as shown in the drawing (750).

The modeling device can estimate high-quality surface normals by capturing the target scene by the illumination pattern acquired in step (610). The modeling device can estimate surface normals N for the actual target scene by using the illumination pattern optimized through the above-described process.

The modeling device can obtain a diffuse reflection image corresponding to one of the lighting patterns by performing separation of the diffuse reflection component and the specular reflection component for each frame of the target scene.

The modeling device applies a photometric stereo function according to the photometric stereo technique to the diffuse reflection image of the target scene for the optimized illumination pattern. By applying the mathematical expression 10 below, the surface normal (N) of the three-dimensional target object can be estimated.

The modeling device may be, but is not necessarily limited to, at least one of a lighting stage, a handheld flash camera, an imaging system including a display camera system, a wearable device including smart glasses, a head mounted device (HMD) including an Augmented Reality (AR) device, a Virtual Reality (VR) device, and a Mixed Reality (MR) device, a user terminal including a television, a smart phone, a personal computer, a tablet, and a laptop.

FIG. 8 is a diagram illustrating a learning process for determining a lighting pattern according to one embodiment. Referring to FIG. 8, a diagram (800) illustrating a learning process for determining a lighting pattern according to one embodiment is illustrated.

In the data set acquisition process (810), the pattern determination device can capture a real object (815) using base lighting, and obtain a correct surface normal vector (805) of the real object (815) through differentiable rendering based on the captured image.

The pattern determination device can generate basis images based on an image captured of an actual object (815) and obtain the basis images (825) through a capture process (820).

The pattern determination device can perform rendering (830) on the basis images (825) to obtain simulated images (835). In the rendering (830) process, the pattern determination device can perform rendering on the basis images (825) using an illumination pattern (860) learned by a loss function (850) based on the difference between the correct surface normal vector (805) and the surface normal vector (845) estimated by reconstruction according to a differentiable photometric stereo technique.

The pattern determination device can input simulated images (835) and learned illumination patterns (860) into a differentiable photometric stereo framework (530) and estimate surface normal vectors (845) by reconstruction according to a differentiable photometric stereo technique.

FIG. 9 is a block diagram of a device for modeling a three-dimensional scene according to one embodiment. Referring to FIG. 8, a device for modeling a three-dimensional scene ('modeling device') (900) according to one embodiment may include a communication interface (910), a camera (920), a processor (930), a display (950), and a memory (970). The communication interface (910), the camera (920), the processor (930), the display (950), and the memory (970) may be connected to each other through a communication bus (905).

The communication interface (910) receives lighting patterns corresponding to a three-dimensional target object. At this time, the lighting patterns may be output by a neural network learned through the aforementioned process.

The camera (920) captures a target scene including a three-dimensional target object by means of illumination patterns received through the communication interface (910). At this time, the camera (920) may capture stereo images, for example, as the aforementioned polarization camera or stereo camera, but is not necessarily limited thereto. In a modeling device (900) implemented as a smartphone, the camera (920) may include multi-camera sensors having different optical specifications. The cameras of the smartphone have a fixed baseline, but the size and/or resolution of the images captured by each camera (920) may be different.

The processor (930) models a three-dimensional scene corresponding to the target scene captured by the camera (920) by restoring the surface normal of the three-dimensional target object through a photometric stereo technique based on the illumination patterns received through the communication interface (910).

The display (950) can display at least one of the lighting patterns received via the communication interface (910) and the three-dimensional scene modeled by the processor (930).

The memory (970) can store lighting patterns received through the communication interface (910), surface normal vectors of a three-dimensional target object restored by the processor (930), and/or a three-dimensional scene modeled by the processor (930).

In addition, the memory (970) can store various pieces of information generated during the processing in the processor (930) described above. In addition, the memory (970) can store various pieces of data and programs. The memory (970) can include volatile memory or nonvolatile memory. The memory (970) can store various pieces of data by having a large storage medium such as a hard disk.

In addition, the processor (930) can perform an algorithm corresponding to a method or a method described through FIGS. 1 to 7 of the present specification. The processor (930) can execute a program and control the modeling device (900). The program code executed by the processor (930) can be stored in the memory (970).

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the OS. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on them. For example, even if the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included in the scope of the claims described below.

Claims (20)

A step of constructing a data set by estimating a first surface normal vector of a three-dimensional object from a first image of the three-dimensional object whose surface normal information is known in advance;
A step of generating simulation images by applying virtual illumination patterns based on a combination of basis images included in the above data set to the three-dimensional object;
A step of estimating a second surface normal vector for the three-dimensional object through reconstruction of the photometric stereo technique based on the virtual lighting patterns and simulation images corresponding to the virtual lighting patterns;
A step of training a neural network to determine a lighting pattern based on the difference between the first surface normal vector and the second surface normal vector.
A method for determining a lighting pattern, comprising: In the first paragraph,
The steps to construct the above data set are
A step of obtaining the first image of the three-dimensional object by photographing it with ground illumination; and
A step of estimating the first surface normal vector from the first image using a differentiable rendering technique.
A method for determining a lighting pattern, comprising: In the second paragraph,
A step of performing preprocessing to remove the specular component from the first image above.
Including more,
The step of obtaining the first image is
A step of constructing the data set from the first image that has been preprocessed above.
A method for determining a lighting pattern, comprising: In the third paragraph,
The steps for performing the above preprocessing are
A step of predicting the positions of grid points displayed on a display device using a mirror; and
A step of obtaining the first image by adjusting the position of the display device to be placed at the positions of the grid points to remove the specular reflection component.
A method for determining a lighting pattern, comprising: In paragraph 4,
The step of obtaining the first image by removing the above reflection component is
A step of optically distinguishing the specular reflection component and the diffuse reflection component from a first image captured of the three-dimensional object using a polarizing camera; and
A step of obtaining an image of the diffuse reflection component from which the specular reflection component has been removed from the first image that captured the three-dimensional object as the first image.
A method for determining a lighting pattern, comprising: In the second paragraph,
The step of estimating the first surface normal vector is
A step of aligning the second image rendered through the differentiable rendering technique to be the same as the first image by optimizing the movement parameters and rotation parameters of the three-dimensional object in the virtual environment; and
A step of estimating the first surface normal vector based on the aligned first image and the second image.
A method for determining a lighting pattern, comprising: In the first paragraph,
The steps for generating the above simulation images are
A step of synthesizing the simulated images by simulating the captured images for each of the virtual lighting patterns in a differentiable manner in response to the above base images.
A method for determining a lighting pattern, comprising: In Article 7,
The step of synthesizing the above simulation images is
A step of synthesizing the simulation images by a weighted sum that multiplies RGB color intensities corresponding to at least a portion of each of the base images for each of the virtual lighting patterns.
A method for determining a lighting pattern, comprising: In the first paragraph,
The step of predicting the second surface normal vector is
A step of reconstructing the surface normal of the above 3D object using a display and a camera.
A method for determining a lighting pattern, comprising: In the first paragraph,
The step of predicting the second surface normal vector is
A step of reconstructing at least one of the surface normal and diffuse albedo from simulation images corresponding to the virtual lighting patterns.
A method for determining a lighting pattern, comprising: In Article 10,
The step of reconstructing the surface normal and the diffuse albedo is
A step of setting the above diffuse albedo to the maximum intensity among the above simulation images;
A step of estimating the surface normal using a pseudo-inverse method based on the diffuse albedo set to the maximum intensity;
A step of estimating the diffuse albedo using the pseudo-inverse method for each RGB channel of the above simulation images; and
A step of reconstructing the surface normal and the diffuse albedo by iteratively estimating the surface normal and the diffuse albedo.
A method for determining a lighting pattern, comprising: In the first paragraph,
The step of predicting the second surface normal vector is
A step of predicting the second surface normal vector by substituting the virtual lighting patterns and simulation images corresponding to the virtual lighting patterns into a linear system based on the photometric stereo technique.
A method for determining a lighting pattern, comprising: A step of obtaining lighting patterns corresponding to a three-dimensional target object using a learned neural network;
A step of capturing a target scene including a three-dimensional target object by the above lighting patterns; and
A step of modeling a 3D scene corresponding to the target scene by restoring the surface normal of the 3D target object through a photometric stereo technique based on the above lighting patterns.
A method for modeling a three-dimensional scene, comprising: In Article 13
The steps for modeling the above 3D scene are
A step of obtaining a diffuse reflection image corresponding to one of the lighting patterns by performing separation of the diffuse reflection component and the specular reflection component for each frame of the target scene; and
A step of estimating the surface normal vector of the three-dimensional target object by applying the photometric stereo technique to the diffuse reflection image.
A method for modeling a three-dimensional scene, comprising: In Article 12,
The above neural network
A method for modeling a three-dimensional scene, the method being learned by a data set composed by estimating a first surface normal vector of a three-dimensional object from a first image of the three-dimensional object whose surface normal information is known in advance. A computer program stored in a computer-readable storage medium for executing any one of the methods of claims 1 to 15 in combination with hardware. A communication interface for receiving lighting patterns corresponding to a three-dimensional target object;
A camera for capturing a target scene including a three-dimensional target object by the above lighting patterns; and
A processor that models a 3D scene corresponding to the target scene by restoring the surface normal of the 3D target object through a photometric stereo technique based on the above lighting patterns.
A device for modeling a three-dimensional scene, including: In Article 17,
The above processor
A device for modeling a three-dimensional scene, wherein the device separates a diffuse reflection component and a specular reflection component for each frame of the target scene to obtain a diffuse reflection image corresponding to one of the lighting patterns, and applies the photometric stereo technique to the diffuse reflection image to estimate a surface normal vector of the three-dimensional target object. In Article 17,
A display displaying at least one of the above lighting patterns and the modeled three-dimensional scene
A device for modeling a three-dimensional scene, comprising: In Article 17,
The device for modeling the above three-dimensional scene is
A device for modeling a three-dimensional scene, comprising at least one of a lighting stage, an imaging system including a handheld flash camera, a display camera system, a wearable device including smart glasses, a head mounted device (HMD) including an augmented reality (AR) device, a virtual reality (VR) device, and a mixed reality (MR) device, and a user terminal including a television, a smart phone, a personal computer, a tablet, and a laptop.

Images