RU2573767C1 - Three-dimensional scene scanning device with non-lambert lighting effects - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: disclosed is a three-dimensional scene reconstruction device with non-Lambert lighting effects. The device comprises an imaging unit, a control unit, a global illumination mask computing unit, a pattern extraction unit, an adaptive pattern generating unit, an intersection search unit, a projection unit, an information acquisition unit, a pattern association unit, a three-dimensional shape computing unit and a clock frequency unit.

EFFECT: high quality of scanning a scene with non-Lambert lighting effects.

1 dwg

Description

The invention relates to the field of computer technology and can be used in digital systems for obtaining three-dimensional models of physical objects, including: inspection of industrial products, rapid prototyping systems, systems for building three-dimensional models for virtual reality and medicine.

The task of three-dimensional reconstruction is to build a mathematical representation of geometric information about the scene. In most existing approaches to three-dimensional reconstruction, the Lambert law of the reflective power of each scene object is explicitly or implicitly assumed.

In practice, the law of the reflectivity of scene objects is different from the law of Lambert. Of particular interest are objects with a strongly non-Lambertian scattering law: translucent, transparent, mirror, with subsurface scattering effects. Deviation from the Lambert law can be caused by the presence of global lighting effects inside the scene. The brightness of each point in the scene, perceived by the sensor, consists of two components of lighting: direct and global. The direct component of lighting is understood to mean light reflected only once from the scene. The global component of lighting is formed by light undergoing multiple re-reflections within the scene.

The main problem to be solved is the construction of a three-dimensional model of the scene with non-Lambert lighting effects.

The solution to the task of scanning a scene with non-Lambert lighting effects can be achieved by separating the direct and global components of the scene lighting and using structured light patterns that take into account the features of the scanned scene.

Simplified methods of three-dimensional reconstruction of scenes with non-Lambert lighting effects can be divided into the following groups:

1) methods based on coating objects with non-Lambert reflectivity with a substance with a diffuse reflection law;

2) methods based on the use of high-frequency patterns;

3) methods based on post-processing of the received scan data.

The analysis of existing methods of three-dimensional reconstruction shows that the scope of their use is limited, in conditions of a small amount of information about the scene. Methods based on coating objects with non-Lambert reflectivity with a substance with a diffuse reflection law are often not applicable due to the significant effect on the scanned object. Living objects or works of art cannot be coated with diffuse matter without violating their integrity. Methods based on the use of high-frequency patterns do not allow to deal with local and global lighting effects at the same time. Thus, the effect of either local lighting effects (such as subsurface scattering) or global effects (caused by multiple re-reflection of light inside the scene) can be eliminated. Methods based on post-processing of the obtained data impose restrictions on the properties of the scanned scene and fundamentally do not allow to restore the information lost as a result of the influence of non-Lambert effects of lighting.

There is a method of using structured light for three-dimensional reconstruction of objects subject to global illumination effects (Structured light for 3d shape reconstruction subject to global illumination) [Patent USA No. US 8811767 B2 Aug. 19, 2014]. The invention relates to the field of constructing three-dimensional models of physical objects, and in particular to a method and system for scanning scenes subject to global lighting effects. The technical result is the creation of an improved method of scanning a scene using structured light. The specified technical result is achieved due to the fact that information about the depth of the scene is obtained by projecting many patterns of structured light onto the scene. Each of the templates has its own spatial frequency, which is achieved using various encoding functions; get a set of scene images containing one image for each spatial frequency; the depth value for each pixel is calculated by decoding the obtained sets of images; the obtained depth values of each pixel are analyzed and stored if the depth values for the different patterns of structured light match; otherwise, the depth value will be marked as erroneous; analyzes errors associated with various effects of global lighting; Based on the analysis, the properties that a structured light pattern must have in order to be resistant to errors of this kind are determined. These properties affect the spatial frequency of the pattern projected onto the scene; each row and column of pixels of the projector is encoded with a unique spatial or temporal code, in order to form a structured light pattern; for each pixel of the camera sensor, the corresponding row and column of the projector are determined by decoding the measured intensity values; rangefinder information is obtained by triangulating the rays of the camera and the projector.

The features of the analogue method, which coincide with the features of the claimed technical solution, are as follows: allows you to build a three-dimensional model of the scanned scene.

The disadvantages of this method are: the use of several lighting patterns increases the total scanning time; For correct decoding, it is necessary that each pixel of the camera records light from only one pixel of the projector, which is difficult to achieve in practice.

A known method for recognizing the three-dimensional shape of objects [Patent No. 2491503, IPC G01B 11 / 24b IPC G02B 27/22]. The method includes illuminating the surface of an object with optical radiation, receiving and recording the brightness of the reflected optical radiation of the elements of its surface, converting the optical radiation into an electrical signal, followed by its storage and analysis. The method uses illumination of the surface of an object with collimated beams of optical radiation from two mutually perpendicular directions. The shape of the object is determined by the expressions, including the magnitude of the video signals in the image of the surface elements of the object from two mutually orthogonal directions and observation from a direction that coincides with one of the lighting directions; line numbers and line elements for which the third coordinate is measured in the image of the surface of the object, the scanning step of the surface of the object, respectively, along the coordinates OY and OZ.

EFFECT: resistance to differences in types of coverage due to recognition of objects with both diffuse and directionally scattering coatings, and the extension of the information content of the channels of optical and optoelectronic recognition systems.

The features of the analogue method, which coincide with the features of the claimed technical solution, are as follows: allows you to build a three-dimensional model of the scanned scene.

The disadvantages of this method are: the inability to build a highly detailed model of a three-dimensional scene; computational complexity.

A known method of measuring the surface shape of a three-dimensional object [Patent No. 2472108, IPC G01B 9/02, G01B 11/03, G01B 5/012].

The invention relates to measuring technique, namely to profilometry, topography. The method of measuring the surface shape of a three-dimensional object is to use a matrix of probes containing rigidly reflective elements in the upper part, placed in a holder with many parallel guides located in nodes of a two-dimensional grid with known coordinates X _i , Y _i of the contact points of each probe relative to the center of the holder, allowing each probe to move freely along the Z axis perpendicular to the measured surface. Preliminary, the probe matrix is calibrated in height by measuring the coordinates Z _{i of} each probe in the contact position of all the probes with a reference flat smooth surface, the normal to which coincides with the Z direction. When mutual displacement starting three-dimensional object and the cage with probes matrix in the XY plane is measured coordinates X _o, Y _o center ring and the coordinate Z _oi each probe in the contact position of the probes with the object, the surface shape is calculated based on data on the coordinate difference above X _o -X _i , Y _o -Y _i , Z _oi -Z _i and repeat the measurements if necessary. The technical result consists in providing the ability to measure the surface shape of dynamic objects, in improving the accuracy of measuring the height of three-dimensional objects and in reducing scan time.

The features of the analogue method, which coincide with the features of the claimed technical solution, are as follows: allows you to build a three-dimensional model of the scanned scene.

The disadvantages of this method are: the use of a complex optical system leads to high complexity of production and maintenance of the device; the use of rigidly fixed reflective elements makes it impossible to change the working volume of the device.

Closest to the invention is a method and apparatus for measuring the shape of an object using coded structured light (Optical acquisition of object shape from coded structured light) [Patent EU No. EP 2372648 A3]. The method of measuring the shape of an object using coded structured light is based on projecting structured light in the form of a plurality of measurement lines and one reference line onto the objects of the scene; measuring the shape of an object is made at intervals between intersection points; capturing an image of an object with a projection of structured light generated based on template data; the intersection points on the captured image are extracted; information is extracted on the shape of the object at the intervals between the points of intersection of the reference and measuring lines.

The features of the method and device implementing it - the prototype, which coincide with the features of the claimed technical solution, are as follows: allows you to build a three-dimensional model of the scanned scene.

The disadvantages of the known method and device that implements it are: the inability to work with scenes with global lighting effects; the impossibility of working with non-Lambert scenes: mirror, translucent, experiencing the effects of subsurface scattering.

The block diagram of a device that implements the considered algorithm contains: a block for projecting light patterns, a block for generating light patterns, a block for obtaining images of a scene, a block for extracting a pattern from a scene image, a block for associating patterns, a block for searching for overlapping areas, a block for extracting information, and a three-dimensional form calculation unit.

The proposed device for three-dimensional scanning of a scene with non-Lambert lighting effects makes it possible to obtain three-dimensional models of the scene in the form of a point cloud. The device implements the following algorithm.

The first step is to calibrate the camera-projector system. Calibration allows you to take into account the distortions introduced by the optics of the camera and projector. Camera calibration is reduced to obtaining the internal and external parameters of the camera from the available photos or videos captured by it. Camera calibration is often used at the initial stage of solving many tasks of computer vision and especially augmented reality. In addition, camera calibration helps correct distortion in photos and videos.

, а для задания положения 3D-точки в мировых координатах - $${\left[\begin{array}{cccc}{x}_w& y_w& z_w& 1\end{array}\right]}^T$$

. Для проецирования точек трехмерного пространства на плоскость изображения используется матрица, имеющая следующий вид:To represent the 2D coordinates of a point on a plane, a column vector of the form $${\left[\begin{array}{ccc}u& v& \mathrm{one}\end{array}\right]}^T$$

, and to set the position of the 3D point in world coordinates - $${\left[\begin{array}{cccc}{x}_w& y_w& z_w& \mathrm{one}\end{array}\right]}^T$$

. To project points of three-dimensional space on the image plane, a matrix is used that has the following form:

Internal Calibration Parameters

The internal calibration matrix A contains 5 significant parameters. These parameters correspond to the focal length, the angle of the pixels, and the position of the principal point of the optical system. In particular, α _x and α _y correspond to the focal length measured in the width and height of the pixel, u ₀ and v ₀ correspond to the coordinates of the principal point, and y = α _y * tanφ, where φ is the angle of the pixel.

External calibration parameters R, T (where R is a 3 × 1 vector or a 3 × 3 rotation matrix, T is a 3 × 1 transfer vector) are the external calibration parameters that determine the coordinate transformation that transfers the coordinates of scene points from the world coordinate system to the coordinate system, associated with the camera.

и глобальной $$I_g^t$$

компонент. Если бинарный шаблон освещает сцену - прямая компонента освещенности модулируется шаблоном. Если шаблон состоит из равного количества светлых и темных пикселей и имеет высокую пространственную частоту в сравнении с $$I_g^t$$

, вклад глобальной компоненты в яркость пикселя составляет $$\frac{1}{2}{I}_g^t$$

. Таким образом, имеем:In the second step, the optical properties of the scanned scene are determined by separating the direct and global lighting components using high-frequency patterns. Separation of the direct and global lighting components provides important information about the scene materials, as well as improving the efficiency of reconstruction methods based on structured light. The brightness I ^t (ρ) of the pixel ρ∈ (x, y) at time t is a combination of the line $$I_d^t$$

and global $$I_g^t$$

component. If the binary template illuminates the scene, the direct component of the illumination is modulated by the template. If the template consists of an equal number of light and dark pixels and has a high spatial frequency in comparison with $$I_g^t$$

, the contribution of the global component to the pixel brightness is $$\frac{\mathrm{one}}{2}{I}_g^t$$

. Thus, we have:

Here s ^t ∈ {0,1} is the value of the template projected onto the camera pixel x. The position of the camera and the projector in space, as well as their internal calibration parameters, are fixed and do not depend on the geometry of the scene.

In the third step, a set of patterns of structured light is generated, the structure of which depends on the optical properties of the scanned scene. A time coding with variable frequency patterns is used. The binary encoding consists of a sequence of binary images in which each frame represents one of the bit planes of the binary representation of the integer indices of the projector columns.

In the fourth step, the scene is scanned using the generated structured light patterns.

At the fifth step, a three-dimensional model of the scene is constructed by triangulating the rays of the camera and the projector.

The device for three-dimensional scanning of a scene with non-Lambert lighting effects (Fig. 1) contains an image acquisition unit, 1, the first input of which is the information input of the device, the output of which is connected to the input of the global lighting mask calculation unit 3 and the first input of the template extraction unit 4, the output of which the first input of the template association unit 9, the second input of the information receiving unit 8 and the input of the intersection search unit 7, the output is connected to the first input of the information receiving unit 8, the output of which is connected to the network input of the template association unit 9, the output of which is connected to the first input of the three-dimensional shape calculation unit 10, the output of which is connected to the first input of the storage unit 12, the output of which is the information output of the device; the first output of the control unit 2 is connected to the second input of the image acquisition unit 1, the second output of the control unit 2 is connected to the second input of the template extraction unit 4 and the second input of the adaptive template generation unit 5, the third output of the control unit 2 is connected to the second input of the three-dimensional form calculation unit 10 ; the output of the global lighting mask calculation unit 3 is connected to the first input of the adaptive template generation unit 5, the output of which is connected to the second input of the template association unit 9 and the input of the projection unit 6, the output of which is connected to the second input of the storage unit 12; the synchronization of the device is provided by the clock generator 11.

A device for three-dimensional reconstruction of scenes with non-Lambert lighting effects works as follows. Using the control unit 2, a control signal is generated that is transmitted to the second input of the adaptive template generation unit 5, the output of which is used to generate a template for separating the local and global lighting components, which is sent to the projection unit 6. Using the control device 2, a control signal is generated that is sent to the second the input of the image acquisition unit 1, at the output of which an image of the scene with an overlay template is formed to separate the local and global components of the illumination, which is received to the calculation unit of the global lighting mask 3, at the output of which a global lighting mask is generated, which is fed to the first input of the adaptive template generation unit 5. Using the control unit 3, a control signal is generated that is input to the calculation unit of the three-dimensional shape 10, the second input of the extraction unit template 4 and the second input of the adaptive template generation unit 5. At the output of the adaptive template generation unit 5, a sequence of templates is obtained based on the global lighting mask. The generated sequence of patterns is fed to the input of the projection unit 9. The sequence of patterns is projected onto the scene. The input of the image acquisition unit 1 receives a sequence of scene images with superimposed patterns of structured light. From the output of image acquisition 1, the sequence of images of the scene is transmitted to the input of the template extraction unit 4, where the templates superimposed on the scene are extracted. The input of the intersection search block and the second input of the information obtaining unit 8 are provided with the result of the extraction of patterns. In the block of search for intersections 7, an intersection mask is formed, which is fed to the first input of the information obtaining unit 8. In the block of obtaining information 8, distance measuring information about the scene is extracted, which is fed to the third input of the block of association of patterns 9. From the output of the adaptive patterns generation block 5, the sequence of generated of templates is fed to the second input of the template association unit 9. The extracted templates from the output of the template extraction unit 4 are fed to the first input of the template association unit 9, in which GSI postprocessing rangefinder information about the scene. The result of the post-processing is fed to the first input of the three-dimensional form calculation unit 10. The data from the outputs of the projection unit 6 and the three-dimensional form calculation unit 10 are fed to the inputs of the storage unit 12, the output of which is the information output of the device. The synchronization of the device is provided by the clock generator 11.

EFFECT: obtaining a high-quality model of a three-dimensional scene with non-Lambert lighting effects.

Claims (1)

A device for three-dimensional reconstruction of scenes with non-Lambert lighting effects, comprising an image acquisition unit, a projection unit, a pattern extraction unit, an intersection search unit, an information acquisition unit, a pattern association unit, a three-dimensional shape calculation unit, characterized in that the information input of the device is the first input of the acquisition unit Images; the output of the image acquisition unit is connected to the input of the global illumination mask calculation unit and to the first input of the template extraction unit; the first output of the control unit is connected to the second input of the image acquisition unit; the second input of the template extraction unit and the second input of the adaptive template generation unit are connected to the second output of the control unit; to the third output of the control unit is connected a second input of the three-dimensional shape calculation unit; the first input of the adaptive template generation unit is connected to the output of the global illumination mask calculation unit; the output of the template extraction unit is connected to the first input of the template association unit, the input of the intersection search unit and the second input of the information obtaining unit; the output of the intersection search unit is connected to the first input of the information obtaining unit, the output of which is connected to the third input of the template association unit; the output of the adaptive template generation unit is connected to the input of the projection unit and the second input of the template association unit; the output of the template association unit is connected to the first input of the three-dimensional shape calculation unit; the outputs of the projection units and the three-dimensional shape calculation unit are connected to the inputs of the storage unit, the output of which is the information output of the device; synchronization of the device is provided by the clock generator.