JP7245766B2 - 3D model generation method and apparatus - Google Patents

Description

TECHNICAL FIELD The present invention relates to a 3D model generation method and apparatus, and more particularly to a 3D model generation method and apparatus for generating a 3D model of an object from images captured by a plurality of cameras at high speed and with high quality.

As an approach for generating a 3D model of an object from multiple camera images, the visual volume intersection method disclosed in Non-Patent Document 1 is widely known. The visual volume intersection method is a method of generating a 3D model by projecting a binary silhouette image, which is obtained by extracting only the part of the subject from each camera image, onto a 3D space and leaving only the part that is the product set.

A minimum unit that constitutes a 3D model generated based on the visual volume intersection method is called a voxel. A voxel is a small-volume cube with a constant value, and is a regular grid unit for discrete representation of volumetric data.

Non-Patent Document 2 discloses a technique that uses the visual volume intersection method in a free-viewpoint imaging technique or the like. Free-viewpoint video technology is a technology that reconstructs 3D space from images from multiple cameras and enables viewing from angles where there are no cameras. is. However, in a vast area such as a stadium, when generating a 3D model using the normal voxel-based visual volume intersection method, there is a drawback that the calculation time is enormous.

In order to solve such a technical problem, Non-Patent Document 3 discloses a Coarse-to-Fine voxel model generation algorithm as a technique for speeding up the visual volume intersection method. In Non-Patent Document 3, when generating a 3D voxel model by the visual volume intersection method, the model is first generated with a coarse unit voxel size Ma, and a 3D bounding box is obtained by treating a cluster of voxels as one object. After that, we succeeded in greatly reducing the processing time by modeling the inside of each 3D bounding box using the visual volume intersection method with a fine unit voxel size Mb (<Ma). Non-Patent Document 4 discloses a technique for expanding voxels in order to prevent voxel chipping.

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162 (1994). J. Kilner, J. Starck, A. Hilton and O. Grau, "Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events," Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007), Montreal, QC, 2007, pp. 177-184. J. Chen, R. Watanabe, K. Nonaka, T. Konno, H. Sankoh, S. Naito, "A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes", 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS 2019), WeAT17.2, (2019). C. Prock, Andrew & Dyer, Charles. "Towards Real-Time Voxel Coloring.", 1970. C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252 Vol. 2 (1999). Chen, J., Nonaka, K., Sankoh, H., Watanabe, R., Sabirin, H., & Naito, S. Efficient Parallel Connected Component Labeling with a Coarse-to-Fine Strategy. IEEE Access, 2008, 6 , 55731-55740.

Although the Coarse-to-Fine voxel model generation algorithm disclosed in Non-Patent Document 3 can significantly reduce the processing time of the conventional visual volume intersection method, it may degrade the quality of the 3D model.

FIG. 8 is a diagram for explaining the cause of quality deterioration. In Non-Patent Document 3, when estimating a 3D bounding box from coarse voxels, the overlap between the visual volume intersection area and the coarse voxel grid is determined, and only the coarse voxel grid that overlaps the visual volume intersection area is the voxel regarded as the model's domain of existence.

However, since the overlap judgment is performed based on the center coordinates of each voxel grid (indicated by "·" in the figure), there are subordinated parts that are considered to be outside the 3D bounding box even though they are part of the visual volume intersection area. can occur. Such subordinated parts are not modeled in the subsequent fine voxel generation targeting the inside of the 3D bounding box, so they cause defects in the output 3D model, especially near the surface. .

In this regard, according to Non-Patent Document 4, each voxel is dilated, so a part of the visual volume intersection area, which was the outer region of the 3D bounding box in Non-Patent Document 3, is included in the inner region of the 3D bounding box. be able to.

However, in non-patent document 4, when the unit voxel size of coarse voxels increases, the voxel size becomes 0 for small objects such as baseballs, and model generation is not performed at the stage of coarse voxel generation. , there was a concern that the object itself would disappear.

In addition, in Non-Patent Document 3, experiments were conducted on volleyball and judo sequences. Further speeding up is necessary to realize real-time 3D model generation.

On the other hand, if there is room in the processing load, the unit voxel size can be finely designed (=the finer unit voxel size refines the shape of the final model). Speeding up is important in considering the practical use of free-viewpoint video.

The purpose of the present invention is to solve the above technical problems, and to prevent the loss of 3D models and the disappearance of small objects in coarse-to-fine voxel model generation, enabling high-speed and high-quality 3D model generation. An object of the present invention is to provide a model generation method and apparatus.

In order to achieve the above objects, the present invention is characterized by having the following configuration in a 3D model generating apparatus for generating a 3D model of a subject from multi-viewpoint images.

(1) Unit voxel size by means of obtaining a silhouette image for each viewpoint from a multi-view video, means of generating an expanded silhouette image by dilating the outline of the silhouette image, and the visual volume intersection method using the expanded silhouette image. is a means _for calculating a low-resolution voxel model M _L having a first size M ₁ , and a unit voxel size is a first means for calculating a high resolution voxel model _MDH of a second size _M2 smaller than size _M1 ; and means for outputting a 3D model of the object based on the high resolution voxel model _MDH .

(2) means for generating a reduced silhouette image by reducing the outline of _the silhouette image _{; ,} wherein the means for calculating the high resolution voxel model is included in the area of the low resolution voxel model MD _L1 but the low resolution voxel model MD _L2 The high-resolution voxel model MD _H is calculated for the near-surface portion that is not included in the area of .

(3) As for the expansion processing means, the amount of expansion in the expansion processing is adaptively changed according to the size and shape of the silhouette.

(4) As for the means for retraction processing, the retraction amount of retraction processing is adaptively changed according to the size and shape of the silhouette.

According to the present invention, the following effects are achieved.

(1) Since a high-resolution voxel model for a 3D model is calculated for the area of the low-resolution voxel model calculated by the visual volume intersection method using the dilated silhouette image, it can be output without erasing small objects. It becomes possible to suppress the occurrence of defects in the vicinity of the surface of the 3D model.

(2) When calculating the high-resolution voxel model for the 3D model in the area of the low-resolution voxel model, instead of calculating the inside of the 3D bounding box of the low-resolution voxel model, the low-resolution If only the area where the voxel model is generated is targeted for calculation, the calculation range of the high-resolution voxel model can be limited to the area where the low-resolution voxel model is generated, making it possible to speed up the calculation process. Become.

(3) Targeting only the near-surface portion that is included in the region of the low-resolution voxel model calculated using the dilated silhouette image but not included in the region of the low-resolution voxel model calculated using the degenerated silhouette image. Since the high-resolution voxel model for the 3D model is calculated at the same time, the calculation range of the high-resolution voxel model can be limited only to the vicinity of the surface of the 3D model, making it possible to speed up the calculation process.

(4) The amount of expansion when generating an expanded silhouette image by expanding the contour of a silhouette image, and the amount of contraction when generating a reduced silhouette image by contracting the contour of a silhouette image, are determined by determining the amount of silhouette in the silhouette image. Since it is variable according to the size and shape, it is possible to suppress quality deterioration caused by excessive expansion or insufficient expansion, and an increase in the amount of calculation that does not lead to quality improvement.

1 is a functional block diagram of a 3D model generation device according to a first embodiment of the present invention; FIG. It is the figure which showed the expansion example of the silhouette image. FIG. 4 is a diagram showing an example of a 3D bounding box; FIG. 11 is a diagram showing how a visual volumetric intersection area is expanded by using an expanded silhouette image; FIG. 5 is a functional block diagram of a 3D model generation device according to a second embodiment of the present invention; FIG. 10 is a diagram schematically showing a method of generating a 3D model in the second embodiment; FIG. FIG. 10 is a diagram showing how the present invention reduces defects in a near-surface portion of a 3D model. FIG. 10 is a diagram for explaining a technical problem of the Coarse-to-Fine voxel model generation algorithm;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a 3D model generation device 1 according to the first embodiment of the present invention. Here, generation of a 3D model of a subject in a baseball broadcast will be described as an example. .

Such a 3D model generation device 1 can be configured by installing an application (program) that implements each function in a general-purpose computer or server. Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is made into hardware or software.

The silhouette image acquisition unit 101 acquires a silhouette image A used for the visual volume intersection method in units of frames from images (multi-viewpoint images) of a plurality of cameras 2 photographing a plurality of subjects from different viewpoints. In order to form a 3D model by the visual volume intersection method, it is desirable to acquire silhouette images A from three or more cameras 2 .

The silhouette image A is acquired in a binary mask image format in which the subject for which the 3D model is to be generated is represented in white and the other portions are represented in black. Such a silhouette image A can be acquired using the background subtraction method disclosed in Non-Patent Document 5.

The silhouette image processing unit 102 includes at least an expansion processing unit 102a that expands the outline of each silhouette image A to generate an expanded silhouette image _A1 , and the expanded silhouette image _A1 is processed by another processing method to form a silhouette image. Output with image or raw silhouette image A ₀ (=A). In this embodiment, an expanded silhouette image _A1 and a raw silhouette image _A0 are output for each viewpoint.

FIG. 2 is a diagram showing an example of expansion processing by the expansion processing unit 102a of the silhouette processing unit 102. The expanded silhouette image is obtained by expanding the outline of the silhouette image _A0 outward by several pixels uniformly. A ₁ is generated. In this embodiment, the expanded silhouette image _A1 is generated by expanding each pixel of the silhouette image _A0 to a size of 5×5 pixels.

The expansion amount of the silhouette image by the expansion processing unit 102a may be uniform for all the silhouette images. However, when comparing the silhouette images of the multi-view video regarding the same subject, the amount of expansion of the silhouette image with the larger silhouette size in the angle of view, in other words, the expansion amount of the silhouette image with the smaller silhouette size, is It is empirically recognized that defects can be eliminated more accurately by increasing the expansion amount. Therefore, the expansion amount of each silhouette image may be adaptively changed for each subject according to the silhouette size or the distance from the camera that can represent the silhouette size to the subject.

However, in an environment such as a live broadcast of a baseball game where most of the subjects are people such as players and referees, and there is not much difference in the size of the subjects, regardless of whether it is a silhouette image related to multi-view video of the same person, The expansion amount of all silhouette images may be adaptively changed simply based on the silhouette size or the distance between the camera and the subject.

If the amount of dilation when generating the dilated silhouette image _A1 is increased, the missing parts will be reduced, so quality improvement can be expected. The available space may be reduced. Therefore, it is desirable to minimize the expansion amount within a range that can sufficiently suppress defects.

The _low -resolution voxel model calculation unit 103 performs the above-described An expanded visual volume is formed by the visual volume intersection method using each expanded silhouette image _A1 .

The low-resolution voxel model calculation unit 103 further calculates a connected component based on the adjacency relationship of each voxel in this visual volume, and regards the connected area as a model of one subject, so that the unit voxel size computes a low-resolution voxel model _MDL of a first size _M1 . Any existing method can be used for labeling the connected regions. For example, if the labeling method disclosed in Non-Patent Document 6 is adopted, the connected components can be efficiently calculated.

In this embodiment, the first size _M1 is set to 5 cm, a voxel grid with a unit voxel size of 5 cm is arranged in the target range of 3D model generation (the entire baseball field in this embodiment), and each voxel grid Whether or not to form a 3D model is determined based on the visual volume intersection method. The visual volume intersection method acquires the common part of the visual frustum when n silhouette images are projected onto the 3D world coordinates as the visual volume (Visual Hull) VH(I), which is expressed by the following formula. be

In the above equation (1), set I is a set of silhouette images, and Vi is a viewing frustum calculated from silhouette images obtained from the i-th camera. In addition, although the common part of all n silhouette images is usually modeled, the number of silhouette images used for modeling may be changed, such as modeling when n-1 images are common. Reducing the number of silhouette images used for modeling makes it possible to restore the 3D model even if the subject is missing in some of the silhouette images, but it may cause side effects such as increased noise.

The 3D bounding box generator 104 generates a 3D bounding box BB containing each low-resolution voxel model _MDL , as shown in FIG. In this embodiment, as shown in FIG. 4, the dilated silhouette image _A1 is used to generate the visual volume intersection area _E1 . Larger than Area _E2 .

As a result, the two coarse voxel grids _{BB1 and BB2} , which were determined to be the outer regions of the 3D bounding box BB in the conventional technology, are added to the inner region of the 3D bounding box BB. Voxels will be generated.

The high-resolution voxel model calculation unit 105 applies only to the narrow region inside the 3D bounding box BB generated by the 3D bounding box generation unit 104, or only to the voxel grid from which the low-resolution voxel model _MDL is generated. On the other hand, a voxel grid having a second size M ₂ (M ₂ <M ₁ ) with a relatively small unit voxel size is arranged, and high resolution is obtained by the visual volume intersection method using the raw silhouette mask A ₀ Generate a voxel model MD _H.

The 3D model output unit 106 has a function of outputting a 3D model based on the high resolution voxel model MD _H obtained by the high resolution voxel model calculation unit 105 . The high-resolution voxel model _MDH is volume data formed by a large number of voxels, but in general, there are many cases where it is more convenient to handle 3D model data as a polygon model. At this time, it may have a function of converting a voxel model into a polygon model using a method for converting a voxel model into a polygon model, such as the Martin Cube method, and have a function of outputting a 3D model as a polygon model. .

According to this embodiment, since a high-resolution voxel model for a 3D model is calculated for a region of a low-resolution voxel model calculated by the visual volume intersection method using an expanded silhouette image, a small object can be eliminated. Therefore, it is possible to suppress the occurrence of defects in the vicinity of the surface of the output 3D model.

FIG. 5 is a block diagram showing the configuration of the main part of the 3D model generation device 1 according to the second embodiment of the present invention, and FIG. 6 schematically shows the 3D model generation procedure according to the second embodiment. It is a diagram. In FIGS. 5 and 6, the same reference numerals as above denote the same or equivalent parts, so description thereof will be omitted.

In this embodiment, the silhouette image processing unit 102 includes, in addition to the first processing unit 102a, at least a reduction processing unit 102b for generating a reduced silhouette image _A2 by reducing the outline of each silhouette image A. The silhouette image A ₁ and the reduced silhouette image A ₂ are output together with the silhouette image processed by another processing method or the unprocessed silhouette image A ₀ (=A). In this embodiment, an expanded silhouette image A ₁ , a contracted silhouette image A ₂ and an unprocessed silhouette image A ₀ are output for each viewpoint.

The second low-resolution voxel model calculation unit 107 calculates the inner area of the 3D bounding box BB generated by the 3D bounding box generation unit 104 based on the dilated silhouette image _A1 , or the first low-resolution voxel model calculation unit 103. a third size M ₃ (M ₃ >M ₂ ) in which the unit voxel size is relatively larger than the second size M ₂ only for the voxel grid on which the first low-resolution voxel model MD _L1 is generated; A second low-resolution voxel model MD _L2 is generated by the visual volume intersection method using the reduced silhouette image _A2 . The first size _M1 and the third size _M3 may be the same size or may be different sizes.

This process reduces the resulting 3D model itself and is similar to the process of generating a 3D model that is one step smaller. Voxel model calculations can be considered unnecessary.

The high-resolution voxel model calculation unit 105 is a voxel grid in which the first low-resolution voxel model MD _L1 is generated or a region inside its 3D bounding box BB, and the second low-resolution voxel model MD _L2 is arranging a voxel grid with a unit voxel size of a second size _M2 ( _M2 < _M1 , _M2 < _M3 ) only for the near-surface portion of the 3D model, excluding the generated voxel grid; A high-resolution voxel model MD _H is generated by the visual volume intersection method using the raw silhouette mask A ₀ .

According to this embodiment, it is possible to limit the number of fine voxel grids for which the visual volume intersection method needs to be calculated to only the contour region of the 3D model, and the number can be reduced as compared with the first embodiment. Therefore, it is possible to speed up the calculation process.

In the second embodiment, the computational area of the visual volume intersection method using fine voxel grids can be limited to the surface _vicinity of the 3D model. Since additional processing time is required to generate the silhouette image _A2 , the processing time cannot always be shortened compared to the first embodiment. For example, in a scene in which many small-sized objects or elongated objects exist, the processing time may be longer than in the first embodiment.

Therefore, the second embodiment is adopted for a scene in which many objects with simple shapes and large sizes are present, while the first embodiment is adopted for a scene in which many small-sized objects or elongated objects are present. , it is desirable to use each embodiment properly.

Note that the contraction amount when generating the contracted silhouette image _A2 is not limited to _a fixed value. It may be changed adaptively.

In FIG. 7, in the first embodiment, a model is generated with a low-resolution voxel size of 5 cm and a high-resolution voxel size of 2 cm. By expanding it to a size of 5 x 5 pixels, a model without defects can be generated.

Reference Signs List 1 3D model generation device 2 camera 101 silhouette image acquisition unit 102 silhouette image processing unit 102a expansion processing unit 102b retraction processing unit 103 (first) low-resolution voxel model calculation unit , 104... 3D bounding box generation unit, 105... high resolution voxel model calculation unit, 106... 3D model output unit, 107... second low resolution voxel model calculation unit, A, _A0 ... silhouette image (unprocessed), _{A 1 . _ _}

Claims (10)

In a 3D model generation device that generates a 3D model of a subject from multi-view images,
a means for acquiring a silhouette image for each viewpoint from multi-view video;
expansion processing means for expanding a contour of the silhouette image to generate an expanded silhouette image;
low-resolution voxel model calculation means for calculating a low-resolution voxel model having a first unit voxel size by a visual volume intersection method using an expanded silhouette image;
A high-resolution voxel for calculating a high-resolution voxel model having a second size smaller than the first size by a visual volume intersection method using the silhouette image, targeting the region of the low-resolution voxel model. a model computing means;
and means for outputting a 3D model of a subject based on the high resolution voxel model. further comprising means for generating a 3D bounding box for each of said low resolution voxel models;
2. The 3D model generating apparatus according to claim 1, wherein said high resolution voxel model calculation means calculates a second size high resolution voxel model within said 3D bounding box. a reduction process means for generating a reduced silhouette image by reducing the outline of the silhouette image;
a second low-resolution voxel model calculation means for calculating a low-resolution voxel model of a third size, the voxel size of which is larger than the second size, by a visual volume intersection method using the reduced silhouette image;
The high-resolution voxel model calculation means targets a near-surface portion that is included in the region of the low-resolution voxel model with the voxel size of the first size but is not included in the region of the low-resolution voxel model with the third size. 3. The 3D model generation device according to claim 2, wherein the high-resolution voxel model is calculated in . 4. The 3D model generation device according to claim 3, wherein the first size and the third size of the voxel size are the same. 5. The 3D model generating apparatus according to claim 1, wherein said expansion means adaptively changes the amount of expansion of each silhouette image according to the size of the silhouette. 6. The 3D according to claim 5, wherein the expansion processing means adaptively changes the expansion amount of the expansion processing for each silhouette image according to the silhouette size for each multi-viewpoint image of the same subject. model generator. 6. The expansion processing means makes the amount of expansion processing for a silhouette image having a relatively large silhouette size larger than the expansion amount for a silhouette image having a relatively small silhouette size. Or the 3D model generation device according to 6. 8. The 3D model generation device according to claim 1, wherein said dilation processing means adaptively changes the dilation amount of the dilation processing according to the shape of the silhouette image. In a 3D model generation method in which a computer generates a 3D model of a subject from multi-view images,
Acquire a silhouette image for each viewpoint from multi-view video,
expanding the outline of the silhouette image to generate an expanded silhouette image;
Calculate a low-resolution voxel model with a unit voxel size of the first size by the visual volume intersection method using the expanded silhouette image,
Calculating a high-resolution voxel model of a second size whose unit voxel size is smaller than the first size by a visual volume intersection method using the silhouette image, targeting the region of the low-resolution voxel model;
A 3D model generation method, characterized by outputting a 3D model of a subject based on the high resolution voxel model. Acquiring a reduced silhouette image by reducing the outline of the silhouette image,
calculating a low-resolution voxel model of a third size, the voxel size of which is larger than the second size, by a visual volume intersection method using the reduced silhouette image;
A high-resolution voxel model for a near-surface portion that is included in a region of a low-resolution voxel model with a first size of the voxel size but not included in a region of a low- resolution voxel model with a third size of the voxel size 10. The 3D model generation method according to claim 9, wherein the calculation of .

Images