JP7135517B2 - 3D geometric model generation device, 3D model generation method and program - Google Patents

Description

The present invention relates to a three-dimensional shape model generation device, a three-dimensional shape model generation method, and a program for generating a three-dimensional shape model of an object from two or more images of the object taken from different viewpoints. .

There is a three-dimensional restoration method that generates a three-dimensional shape model of an object based on a plurality of two-dimensional captured images (hereinafter referred to as "multi-viewpoint images") of the object taken from different viewpoints.
For example, Non-Patent Document 1 discloses a technique for restoring a three-dimensional shape from multi-viewpoint images. In Non-Patent Document 1, a depth map is generated for each viewpoint (imaging position) by performing stereo matching using multi-viewpoint images. This depth map is a map showing depth information in three-dimensional coordinates for each pixel of each multi-viewpoint image of each viewpoint. Then, the three-dimensional shape of the object is restored by integrating the plurality of created depth maps.
Here, in general, in the method of restoring a three-dimensional shape from multi-viewpoint images, the image quality of the multi-viewpoint images used as input for the method is important because it strongly affects the accuracy and quality of the three-dimensional shape of the object to be restored. is one of the important elements. For example, image blur (image blurring, image blurring, etc.) can be cited as a factor that strongly affects the accuracy and quality of a three-dimensional shape model. If the multi-view image contains a blurred image in which the object to be restored is out of focus, or an image in which the object is blurred due to the movement of the object or the camera during imaging, the restored 3D It is known that shape models tend to be significantly less accurate. Therefore, in the method of restoring the three-dimensional shape of an object from multi-viewpoint images, it is desirable not to use images in which the restoration target is blurred or blurred as multi-viewpoint images.

M. Goesele, B. Curless, S.; M. Seitz, "Multi-View Stereo Revisited", Proc. of the IEEE 2006

However, it is difficult to capture all the multi-viewpoint images used to restore the three-dimensional shape without blurring. For example, there is a case where a multi-viewpoint image is acquired using moving images or images captured by continuous shooting. In this case, since images are captured while moving the object and the camera, expensive equipment and advanced imaging skills are required in order to suppress image blurring. Also, when trying to capture a relatively small object from a close distance, the depth of field becomes shallow, and the background and foreground become blurred. It is difficult. In other words, in the method of restoring the three-dimensional shape of an object from multi-viewpoint images, it is avoided to use multi-viewpoint images captured in such a situation where blurring is likely to occur. This is a factor that hinders the expansion of applications.

The present invention has been made in view of such circumstances, and is a three-dimensional shape model generating apparatus capable of accurately restoring a three-dimensional shape even when using a multi-viewpoint image including an area in which blur has occurred. A three-dimensional model generation method and program are provided.

A three-dimensional geometric model generating device according to the present invention is a three-dimensional geometric model generating device for generating a three-dimensional geometric model of an object from two or more multi-viewpoint images of the object taken from different viewpoints, and a blur map generation unit that generates a blur map representing the amount of blur for each pixel in each image of a multi-view image; and a three-dimensional restoration processing unit that generates a three-dimensional shape model that represents the three-dimensional shape of the object according to the amount of blur for each pixel in the object.

In the three-dimensional shape model generation device of the present invention, the three-dimensional restoration processing unit weights matching scores of stereo pairs used for generating a depth map having depth information for each pixel of each image from the multi-view images. is performed according to the amount of blur for each pixel of the blur map, and a depth map generation unit that generates the depth map; and a three-dimensional point group generation unit that generates a three-dimensional point group representing the three-dimensional shape of the.

In the 3D shape model generation device of the present invention, the 3D restoration processing unit includes a 3D point group generation unit for generating a 3D point group representing the 3D shape of the object, the multi-viewpoint image and the Using the three-dimensional coordinates of the three-dimensional points in the three-dimensional point group, the weighting of the matching score by window matching is performed according to the amount of blur for each pixel in the blur map, and the three-dimensional points in the three-dimensional point group A three-dimensional point optimization unit that optimizes three-dimensional coordinates, and a three-dimensional point that adds new three-dimensional points to the three-dimensional point group according to the three-dimensional coordinates of each three-dimensional point of the three-dimensional point group. and an additional part.

In the 3D geometric model generation device of the present invention, the 3D restoration processing unit converts the blur map a viewpoint selection unit that performs weighting according to the amount of blur for each pixel and selects a viewpoint to be used for stereo matching, wherein the three-dimensional point optimization unit uses the selected viewpoint to determine the It is characterized by performing matching.

In the three-dimensional shape model generation device of the present invention, the three-dimensional restoration processing unit weights matching scores of stereo pairs used for generating a depth map having depth information for each pixel of each image from the multi-view images. is performed according to the amount of blur for each pixel of the blur map, and a depth map generation unit that generates the depth map; A viewpoint to be used for matching processing is selected based on a 3D point group generation unit that generates a 3D point group that represents the 3D shape of and the appropriate score for each viewpoint corresponding to the 3D point in the 3D point group. and a 3D point optimizing unit that performs matching of the 3D points from the selected viewpoint and optimizes the 3D coordinates of the 3D points.

The three-dimensional shape model generation device of the present invention includes: a depth map generation unit in which the three-dimensional restoration processing unit generates a depth map having depth information for each pixel of each image from the multi-view image; a 3D point cloud generation unit that generates a 3D point cloud representing the 3D shape of the object by integrating the plurality of depth maps generated by the unit; a viewpoint selection unit that weights the appropriate score for each viewpoint according to the amount of blur for each pixel of the blur map and selects a viewpoint to be used for matching processing; and a three-dimensional point optimization unit that optimizes the three-dimensional coordinates of the three-dimensional points.

The three-dimensional shape model generation device of the present invention includes: a depth map generation unit in which the three-dimensional restoration processing unit generates a depth map having depth information for each pixel of each image from the multi-view image; a 3D point cloud generation unit that generates a 3D point cloud representing the 3D shape of the object by integrating the plurality of depth maps generated by the unit; a viewpoint selection unit that selects a viewpoint to be used in matching processing based on the appropriate score for each viewpoint; weights the matching score of the three-dimensional point matching using the selected viewpoint; and the blur map and a three-dimensional point optimization unit that optimizes the three-dimensional coordinates of the three-dimensional point according to the amount of blur for each pixel.

A three-dimensional model generation method of the present invention is a three-dimensional model generation method for generating a three-dimensional shape model of an object from two or more multi-viewpoint images of the object taken from different viewpoints, and generating a blur map. A blur map generation process in which a unit generates a blur map representing the amount of blur for each pixel in each image of the multi-view image, and a three-dimensional restoration processing unit uses the multi-view image and the blur map to perform window The method is characterized by including a three-dimensional reconstruction process of weighting the matching score by matching according to the amount of blur for each pixel in the blur map and generating a three-dimensional shape model representing the three-dimensional shape of the object. .

A program of the present invention is a program for causing a computer to operate as a three-dimensional shape model generating device for generating a three-dimensional shape model of an object from two or more multi-viewpoint images of the object taken from different viewpoints, Blur map generation means for generating a blur map representing the amount of blur for each pixel in each image of the multi-view image, weighting the matching score by window matching using the multi-view image and the blur map, A program for operating as three-dimensional reconstruction processing means for generating a three-dimensional shape model representing the three-dimensional shape of the object, according to the amount of blur for each pixel in the blur map.

According to the present invention, it is possible to accurately restore a three-dimensional shape even when using a multi-viewpoint image including a blurred area.

1 is a block diagram showing a configuration example of a one-three-dimensional geometric model generating device 1 according to an embodiment of the present invention; FIG. FIG. 4 is a diagram showing an example of a blur map generated by a blur map generator 107 according to the embodiment of the present invention; FIG. 4 is a flow chart showing an operation example performed by the three-dimensional geometric model generating device 1 according to the embodiment of the present invention; 1 is a diagram showing an example of an image of a three-dimensional shape model generated by the three-dimensional shape model generation device 1 according to the embodiment of the present invention; FIG.

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a configuration example of a three-dimensional geometric model generation device 1 according to an embodiment of the present invention. In FIG. 1, the 3D shape model generation device 1 includes a 3D restoration processing unit 100, a blur map generation unit 107, a multi-viewpoint image input unit 108, a multi-viewpoint image storage unit 109, a 3D point group storage unit 110, a 3D A shape model storage unit 111 and a blur map storage unit 112 are provided.

The three-dimensional restoration processing unit 100 uses two or more multi-viewpoint images and a blur map, weights the matching score by window matching according to the amount of blur for each pixel in the multi-viewpoint images, and calculates the three-dimensional image of the object. Generate a 3D shape model showing the shape. Here, the blur map is a map that indicates blurring and blurring information for each pixel of the multi-viewpoint image. A method for the 3D restoration processing unit 100 to generate a 3D shape model using the blur map will be described later in detail.

The blur map generation unit 107 generates a blur map indicating the intensity of blur for each pixel for each image of the multi-view image. This blur map is a map that shows blurring and blurring information at each pixel point of each multi-viewpoint image. This is the map indicated by . Specifically, when the value of the blur map is close to 0, it indicates that the probability that blur does not occur at that three-dimensional point is low, that is, the probability that blur occurs is high. On the other hand, when the value of the blur map is close to 1, it indicates that there is a high probability that blur does not occur at that 3D point.

The blur map generation unit 107 generates a blur map using, for example, a machine learning technique as one method of generating this blur map. One machine learning technique, also called deep learning, is a convolutional neural network (CNN). For this CNN, it is possible to construct a CNN that generates a blur map from a color image by learning parameters using a learning data set of a blur map corresponding to an input image.

Further, the blur map generation unit 107 may generate a blur map using a frequency analysis technique as another method of generating a blur map. In this case, the blur map generation unit 107 performs Fourier transform on a local region near each pixel in the image to extract high frequency components in the local region. Then, the blur map generation unit 107 determines whether or not the high frequency component in the extracted local region is equal to or greater than a predetermined threshold. This allows the blur map generator 107 to determine whether or not the local area is blurred. That is, the blur map generation unit 107 determines that the local region is in focus when the amount of high frequency components in the local region is equal to or greater than a predetermined threshold, and determines that the amount of high frequency components in the local region is in focus. If it is less than the threshold, it is determined that the local area is out of focus, that is, blurred.
The blur map generation unit 107 writes and stores the generated blur map of the multi-viewpoint image in the blur map storage unit 112 .

FIG. 2 is a diagram showing an example of a blur map generated by the blur map generator 107 according to the embodiment of the present invention. FIG. 2(a) shows an example of a multi-viewpoint image, and FIG. 2(b) shows a blur map corresponding to the multi-viewpoint image of FIG. 2(a). FIG. 2(b) shows the blur map by an image in which the blur map values of 0 to 1 correspond to the brightness of 0 to 255. FIG.
As shown in FIG. 2(a), in the multi-viewpoint image, for example, the background is out of focus, but almost the entire object is in focus.
As shown in FIG. 2(b), in the blur map, the area corresponding to the background is entirely black, indicating that the luminance is low. This indicates that the area corresponding to the background is highly likely to be blurred (that is, out of focus). On the other hand, the area corresponding to the object is shown in white as a whole, indicating that the luminance is high. This indicates that there is a high probability that the region corresponding to the object is not blurred (that is, is in focus).

Returning to FIG. 1, the multi-viewpoint image input unit 108 inputs data of multi-viewpoint images captured from a plurality of different viewpoints captured by an imaging device (not shown), and assigns viewpoint identification information to the multi-viewpoint images. and writes and stores it in the multi-viewpoint image storage unit 109 .
The multi-viewpoint image storage unit 109 stores the data of the multi-viewpoint image input to the multi-viewpoint image input unit 108 in association with the viewpoint identification information.
The 3D point cloud storage unit 110 stores the 3D point cloud data generated by the 3D point cloud generation unit 103 .
The 3D geometric model storage unit 111 stores the data of the 3D geometric model generated by the 3D geometric model generation unit 106 .
The blur map storage unit 112 stores blur map data generated by the blur map generation unit 107 .

Here, a method for the 3D restoration processing unit 100 to generate a 3D shape model using a blur map will be described.
As shown in FIG. 1, the 3D restoration processing unit 100 includes a camera parameter estimation unit 101, a depth map generation unit 102, a 3D point cloud generation unit 103, a viewpoint selection unit 104, a 3D point optimization unit 105, and a 3D A shape model generation unit 106 is provided.

The camera parameter estimation unit 101 estimates camera parameters for each image of the multi-view images. The camera parameters include extrinsic parameters and intrinsic parameters. The extrinsic parameters are the center coordinates of the lens in the world coordinate system, the direction of the optical axis of the lens, etc. The intrinsic parameters are the focal length, image center, image resolution (pixel number) and distortion aberration coefficient.
The camera parameter estimation unit 101 estimates camera parameters by, for example, inputting image information of multi-view images to Structure from Motion (SfM), which extracts feature amounts from images and calculates corresponding points. In SfM, a combination of multi-viewpoint images is matched using input multi-viewpoint images, and camera parameters are estimated such that the reprojection error of the combination of corresponding points in the matched multi-viewpoint images is minimized. do.
In addition, although the case where the camera parameter estimation unit 101 estimates camera parameters using SfM has been described above as an example, the present invention is not limited to this. The camera parameter estimation unit 101 may estimate part or all of the camera parameters by previously calibrating an imaging device that captures multi-viewpoint images.

The depth map generation unit 102 uses each of the multi-viewpoint images and camera parameters to generate a depth map in the world coordinate system corresponding to the multi-viewpoint images of each viewpoint. This depth map is a map showing depth information in three-dimensional coordinates for each pixel of each multi-viewpoint image of each viewpoint.

In this embodiment, the depth map generation unit 102 uses, for example, a patch match stereo method when generating depth maps. In the patch match stereo method, a depth map and a normal vector map of pixels of each multi-viewpoint image of each viewpoint are generated by initializing with random numbers. Then, the depth map generation unit 102 performs spatial propagation of numerical values to other adjacent pixels for each pixel, and spatial distribution of numerical values in multi-viewpoint images of different viewpoints, for the depth map and normal vector map generated by random numbers. Propagation and fine adjustment of the depth information (depth information) of each pixel and the normal vector are repeatedly performed using adjustment values obtained from random numbers. The depth map generation unit 102 generates a depth map and a normal vector map that maximize the matching score between pixels corresponding to the same location in the multi-view images of different viewpoints. Each line vector map (for example, see the description in Japanese Patent Application No. 2016-124820). The depth map generation unit 102 writes and stores each of the generated depth map and normal vector map in the three-dimensional point cloud storage unit 110 .

Here, as the matching score, normalized cross-correlation (NCC) in a local region (patch) of the image, SSD (Sum of Squared Differences) in the local region of the image, or the like is used. In this case, if one image of the matching stereo pair is blurred, the matching score will be low despite the correct depth value, or the matching score will be high despite the incorrect depth value. inconsistencies can occur. For this reason, the matching scores of a stereo pair in which one image is blurred and a stereo pair in which both images are not blurred (that is, are in focus) are treated equally. Otherwise, there is a possibility that the accuracy of the depth value estimated therefrom will be degraded.

Therefore, in the present embodiment, the depth map generator 102 uses the blur map generated by the blur map generator 107 to weight the matching score. Specifically, the depth map generator 102 calculates the matching score S as shown in the following equation (1).

_S =(W1)*NCC(IR, _IC , _mR , _mC , N) (1)

In equation (1), I _R denotes an image at a viewpoint (reference viewpoint) for which a depth map is to be generated. Also, I _C denotes an image at a viewpoint (neighboring viewpoint) that forms a stereo pair with the image at the reference viewpoint. _mR indicates the coordinates of a specific pixel point (pixel of interest) in the reference viewpoint. _mC indicates the coordinates of the projection point at the neighboring viewpoint. N indicates the window size. And W1 indicates a weighting function. Also, NCC(I _R , I _C , m _R , m _C , N) is a local area of _N ×N pixels near pixel point m _R in reference viewpoint image I _R and a projection Represents the value of NCC with a local region of N×N pixels in the vicinity of point _mC . Here, when SSD is used instead of NCC as a matching score, the term of NCC ( _IR , IC, _mR , _mC , N) in _formula (1) is replaced by SSD ( _IR , _IC , _mR , m _C , N). In addition, in the expression (1), "*" is an operator indicating multiplication.

In this embodiment, the depth map generator 102 uses the following equation (2) as the weighting function W1 shown in equation (1).

W1=(1-ABS(B _R (m _R )-B _C (m _C ))) (2)

In equation (2), B _R (m _R ) indicates the value of the blur map of pixel point m _R in the image of the reference viewpoint. B _C (m _C ) indicates the value of the blur map of the projection point m _C in the image of the neighboring viewpoint. ABS(k) is a function indicating the absolute value of real number k.

Also, the depth map generation unit 102 may use the following equation (3) instead of the weighting function W1 shown in the equation (2).

W1=MIN(1-(B _R (m _R )-B _C (m _C )), 1) (3)

In equation (3), MIN(x, y) is a function that indicates the smaller real number between real number x and real number y.

Also, the depth map generator 102 may use the following equation (4) instead of the weighting function W1 shown in equations (2) and (3).

W1=B _R (m _R ) (4)

The depth map generator 102 calculates the matching score shown in Equation (1) using the weight function W1 shown above. The matching score is then weighted using the blur map.
For example, when blurring occurs in one image of a stereo pair to be matched, the value of either B _R (m _R ) or B _C (m _C ) in equation (2) is close to 0. , the other is close to 1. In this case, the weighting function W1 shown in equation (2) has a value smaller than one. In this case, the matching score S shown in equation (1) is a score smaller than the matching score S before weighting.
On the other hand, when blur does not occur in both images of the stereo pair to be matched, both values of B _R (m _R ) and B _C (m _C ) in equation (2) are close to one. In this case, the weighting function W1 shown in equation (2) has a value close to one. In this case, the matching score S shown in equation (1) is almost the same as the matching score S before weighting.
That is, the depth map generation unit 102 calculates the matching score using the weight function W1, so that the matching score S of the stereo pair in which blurring does not occur is equal to the matching score S of the stereo pair in which blurring occurs on one side. Let values higher than S be shown. As a result, stereo pairs in which blurring does not occur are emphasized, and in the depth map, it is possible to suppress the influence due to the occurrence of blurring in the multi-viewpoint image, that is, the decrease in the accuracy of the estimated depth values.

The 3D point cloud generation unit 103 integrates the depth maps and normal vector maps at all viewpoints to generate a 3D point cloud corresponding to the 3D shape of the object. Here, the 3D point group generation unit 103 generates the 3D point group by a predetermined calculation using each of the depth map and the camera parameters of the imaging device. That is, the 3D point group generation unit 103 obtains a 3D point group from each of the depth maps of the multi-viewpoint images of each viewpoint, and converts the 3D coordinates of the 3D points in the 3D point group into the respective camera parameters. Based on this, coordinate transformation is performed to obtain three-dimensional coordinates in the world coordinate system.

Thereby, the 3D point group generation unit 103 integrates the 3D point groups based on the depth maps corresponding to each multi-viewpoint image in the world coordinate system. Then, the three-dimensional point cloud generation unit 103 gives three-dimensional point cloud identification information to the synthesized three-dimensional point cloud, and writes and stores the information in the three-dimensional point cloud table of the three-dimensional point cloud storage unit 110. .

The viewpoint selection unit 104 selects multi-viewpoint images of two or more viewpoints to be used for matching in order to perform matching for each three-dimensional point in the integrated three-dimensional point group.

In this case, if the viewpoint selection unit 104 selects a viewpoint using only the geometrical positional relationship between the 3D point and each viewpoint used for matching, blurring occurs in the vicinity of the projection point of the 3D point. It is possible that even the image with the best viewpoint is selected as the image with the best viewpoint. If the viewpoint selection unit 104 selects a blurred image as a viewpoint to be used for stereo matching, there is a possibility that the precision of stereo matching will drop significantly. That is, there is a possibility that the accuracy of the 3D point calculated using the selected viewpoint will be reduced, or that 3D point will be removed as an erroneous correspondence.

Therefore, in this embodiment, the viewpoint selection unit 104 selects a viewpoint using the blur map generated by the blur map generation unit 107 in addition to the geometric positional relationship between the three-dimensional point and the viewpoint. That is, the viewpoint selection unit 104 weights the appropriate score based on the geometric positional relationship for each viewpoint corresponding to each 3D point in the 3D point cloud according to the amount of blur for each pixel of the blur map, and Select the viewpoint to use for matching.

For example, the viewpoint selection unit 104 obtains the selection cost S _ij (appropriate score) according to the following equation (5), and sequentially selects the viewpoint from the viewpoint with the highest selection cost S _ij for use in three-dimensional point matching. Select as many viewpoints as you need.

S _ij =W2*(V _ij *(C _j ·r _ij )*(−n _i ·r _ij )) (5)

In equation (5), the selection cost S _ij indicates the selection cost when the i-th 3D point in the 3D point group and the j-th viewpoint among the plurality of viewpoints are combined. And W2 indicates a weighting function.
C _j indicates the vector of the optical axis of the j-th viewpoint. n _i indicates the normal vector of the i-th three-dimensional point. r _ij indicates a viewpoint vector from the j-th viewpoint to the i-th three-dimensional point. Also, "·" in the equation (5) is an operator indicating vector inner product, and "*" is an operator indicating simple multiplication. That is, the term (C _j ·r _ij ) in the above equation (5) indicates that the position coordinates of the pixel corresponding to the i-th three-dimensional point of interest are closer to the center (vector C _j and viewpoint vector r This term is calculated so that the selection cost _Sij of the viewpoint corresponding to the multi-viewpoint image (where the angle between Sij and _ij is smaller) becomes higher. The term (-n _i ·r _ij ) is such that the angle between the normal vector n _i corresponding to the i-th three-dimensional point of interest and the viewpoint vector r _ij is smaller and the direction is This term is calculated so that the selection cost _Sij of the viewpoints corresponding to the multi-viewpoint images, which are in directions opposite to each other, increases.

V _ij is a visibility coefficient (parameter) indicating whether or not the i-th 3D point is visible from the j-th viewpoint. where the coefficient V _ij is '1' if the i-th 3D point is visible from the j-th viewpoint, and '0' if the i-th 3D point is not visible from the j-th viewpoint. . That is, the coefficient V _ij sets the selection cost S _ij to 0 when the i-th three-dimensional point cannot be seen from the j-th viewpoint.

In this embodiment, the viewpoint selection unit 104 may use, for example, the weighting function W2 shown in the following equation (6).

W2 _=Bj ( _mij ) (6)

In equation (6), B _j (m _ij ) indicates the value of the blur map of the projection point m _ij at the j-th viewpoint of the i-th three-dimensional point at the j-th viewpoint.

In this manner, the viewpoint selection unit 104 selects a viewpoint based on the selection cost calculated using the weight function W2. As a result, the viewpoint selection unit 104 calculates a low selection cost for an image in which blur occurs in the vicinity of the projection point of the three-dimensional point. Therefore, the viewpoint selection unit 104 can reduce the possibility of selecting an image in which blur occurs near the projection point of the three-dimensional point.

The 3D point optimization unit 105 performs matching of 3D coordinates for each 3D point in the 3D point group using a multi-viewpoint image of each of the selected viewpoints, and performs recalculation of the 3D coordinates. I do. Here, the 3D point optimization unit 105 may use the 3D point cloud generated by the 3D point cloud generation unit 103 as the 3D point cloud to be matched, or the depth map generation unit 102 A three-dimensional point cloud generated by The three-dimensional point group generated by the depth map generation unit 102 here means a three-dimensional point group generated by initializing each of the depth map and normal vector map of pixels of each multi-viewpoint image of each viewpoint with random numbers. It is the original point group.

In this embodiment, three-dimensional point matching processing between multi-viewpoint images is performed using the phase-only correlation method. This phase-only correlation method takes longer processing time than matching by matching score obtained by normalized cross-correlation and SSD, but can obtain three-dimensional coordinates with higher accuracy (for example, Japanese Patent Application No. 2015-141533 See the description in the publication).
In matching by the phase-only correlation method, the three-dimensional point optimization unit 105 uses each of the three-dimensional coordinates and the normal vector at each of the three-dimensional points to determine a local region in each of the multi-viewpoint images of the selected viewpoint. set. Then, the three-dimensional point optimization unit 105 calculates a phase-only correlation function from the image of each local region of the multi-view images, thereby estimating a small translation amount of the image of the local region between the multi-view images. .

The 3D point optimization unit 105 recalculates the 3D coordinates of the 3D point based on the estimated translation amount. The 3D point optimization unit 105 updates the 3D coordinates of the 3D points in the 3D point group using the optimized 3D coordinates of the 3D points.

Also, the 3D point optimization unit 105 may add a new 3D point group to the optimized 3D point group. In this case, the 3D point optimization unit 105 is an example of a '3D point addition unit'. As an example of a method of adding a new 3D point cloud, the 3D coordinates and normal vector of each 3D point in the 3D point cloud after optimization have a small size compared to the size of the entire 3D point cloud. A method of newly adding a three-dimensional coordinate and a normal vector to which random numbers are added can be mentioned. As an example of another method, for each 3D point of the 3D point group after optimization, on a 3D plane set by the normal vector and 3D coordinates of the 3D point, a 3D point There is a method of newly adding a three-dimensional point group that is arranged in a grid pattern at small regular intervals compared to the size of the entire group. Note that the 3D point optimization unit 105 may add a new 3D point group to the 3D point group after optimization, or add a new 3D point group to the 3D point group before optimization. 3D point cloud may be added.

The 3D point optimization unit 105 repeats the process of updating the 3D coordinates of the 3D points in the 3D point group and the process of adding new 3D points to the 3D point group.

In this case, the three-dimensional point optimization unit 105 calculates phase-only correlation functions for a plurality of stereo pairs generated from the multi-view images when performing matching by the phase-only correlation method between the multi-view images, By integrating the phase-only correlation functions of , a phase-only correlation function between multi-viewpoint images is obtained. Here, when blur occurs in one image of a stereo pair, an error occurs in the calculated phase-only correlation function. In other words, when simple averaging is used to integrate multiple phase-only correlation functions, if the multi-view images include images with blurred or blurred objects, the phase-only correlation functions after averaging will be blurred. , and the accuracy of the final calculated 3D coordinates of the 3D point may be reduced.

Therefore, in this embodiment, the 3D point optimization unit 105 uses the blur map generated by the blur map generation unit 107 to calculate the weighted average of the phase-only correlation functions calculated from each stereo pair. Three-dimensional point optimization section 105 then uses the phase-only correlation function calculated by weighted averaging to calculate three-dimensional coordinates. The three-dimensional point optimization unit 105 may calculate a weighted average using, for example, the weight function W1 shown in any one of the above equations (2) to (4).
As a result, the three-dimensional point optimization unit 105 reduces the influence of the phase-only correlation function calculated from the pair in which blur occurs in one image of the stereo pair on the phase-only correlation function after weighted averaging. It is possible to suppress the deterioration of the accuracy of the three-dimensional coordinates due to the occurrence of blurring.
The 3D point optimization unit 105 writes and stores the 3D point group based on the recalculated 3D coordinates in the multi-viewpoint image storage unit 109 .

The three-dimensional shape model generating unit 106 generates a three-dimensional shape model (three-dimensional mesh model) using a three-dimensional point group composed of three-dimensional points whose three-dimensional coordinates are recalculated. The three-dimensional shape model generation unit generates a three-dimensional shape model from the three-dimensional point cloud using, for example, a mesh reconstruction (Poisson Surface Reconstruction) method.
Then, the three-dimensional geometric model generating unit 106 gives geometric model identification information to the generated three-dimensional geometric model, and writes the data of this three-dimensional geometric model to the three-dimensional geometric model storage unit 111 for storage.

FIG. 3 is a flow chart showing an operation example performed by the three-dimensional geometric model generation device 1 according to the embodiment of the present invention. The flow chart shown below is for generating a depth map from a plurality of different multi-viewpoint images, then generating a 3D point group, and based on this 3D point group, generating a 3D shape model. 3 shows the flow of processing of the model generating device 1. FIG.

Step S1:
A multi-viewpoint image input unit 108 inputs a plurality of multi-viewpoint images captured by an imaging device from a plurality of different viewpoints from an external device (not shown), and assigns viewpoint identification information to each of the viewpoints from which the multi-viewpoint images are captured. do. Then, the multi-view image input unit 108 writes each data of the input multi-view image to the multi-view image storage unit 109 to store.

Step S2:
A blur map generation unit 107 generates a blur map for each pixel for each image of the multi-view image. A blur map generation unit 107 generates a blur map of the multi-viewpoint image by inputting the multi-viewpoint image to a CNN that generates a blur map from a color image. Then, the blur map generation unit 107 writes and stores each blur map of the multi-viewpoint image in the blur map storage unit 112 .

Step S3:
The camera parameter estimation unit 101 estimates camera parameters of each multi-view image. The camera parameter estimation unit 101 reads each data of the multi-view images stored in the multi-view image storage unit 109 and inputs each data of the multi-view images to the SfM to estimate the camera parameters of the multi-view images. do. Then, the camera parameter estimation unit 101 outputs the estimated camera parameters of each of the multi-view images to the depth map generation unit 102 .

Step S4:
The depth map generation unit 102 uses each of the multi-viewpoint images and the camera parameters estimated by the camera parameter estimation unit 101 to generate depth maps and normal vector maps corresponding to each viewpoint. At this time, the depth map generation unit 102 uses the weight function W1 to generate the depth map and the normal vector map so as to suppress the influence of blurring in the multi-view image. The depth map generation unit 102 outputs each of the generated depth map and normal vector map to the 3D point group generation unit 103 .

Step S5:
The 3D point group generation unit 103 integrates each of the depth maps and normal vector maps at all viewpoints of the multi-viewpoint image generated by the depth map generation unit 102 to obtain a 3D point cloud corresponding to the 3D shape of the object. Generate a point cloud. The 3D point cloud generating unit 103 gives 3D point cloud identification information to the synthesized 3D point cloud, and writes and stores it in the 3D point cloud table of the 3D point cloud storage unit 110 .

Step S6:
The viewpoint selection unit 104 selects a plurality of, for example, two viewpoints for matching each 3D point in the 3D point group. At this time, the viewpoint selection unit 104 uses the weighting function W2 to select viewpoints so as to suppress the effects of blurring in the multi-viewpoint image. The viewpoint selection unit 104 outputs information indicating the selected viewpoint to the 3D point optimization unit 105 .

Step S7:
The 3D point optimization unit 105 recalculates the 3D coordinates of the 3D points in each of the multi-viewpoint images at a plurality of viewpoints selected using the viewpoints selected by the viewpoint selection unit 104, and converts the 3D points to Optimize. At this time, the 3D point optimization unit 105 uses the weighting function W1 to optimize the 3D points so as to suppress the effects caused by blurring in the multi-viewpoint image. The 3D point optimization unit 105 stores the optimized 3D point cloud data in the 3D point cloud storage unit 110 .

Step S8:
The 3D shape model generation unit 106 generates a 3D shape model using the 3D point cloud data optimized by the 3D point optimization unit 105 . The three-dimensional shape model generation unit 106 generates a three-dimensional mesh model using, for example, a method of mesh reconstruction. The three-dimensional geometric model generation unit 106 stores the generated three-dimensional mesh model in the three-dimensional geometric model storage unit 111 .

FIG. 4 is a diagram showing an example of an image of a three-dimensional shape model generated by the three-dimensional shape model generation device 1 according to the embodiment of the present invention. FIGS. 4A and 4B are diagrams showing examples of multi-viewpoint images captured from different viewpoints. Here, the multi-viewpoint images of FIGS. 4A and 4B include an image in which part of the object is blurred.
Each of FIGS. 4(c) and 4(d) is a diagram showing an example of a conventional three-dimensional mesh model created without weighting with a blur map. Each of FIGS. 4(e) and 4(f) is a diagram showing an example of a three-dimensional mesh model created by weighting with a blur map.

As shown in FIGS. 4(c) and 4(d), when a 3D mesh model is created using a conventional 3D reconstruction method from a multi-viewpoint image including images with blur, Since weighting by the map is not performed, it can be confirmed that a part of the object is lost in the restoration result, or that the object is restored as a noise-like inaccurate shape.

On the other hand, in the present embodiment, as shown in FIGS. 4C and 4D, multi-viewpoint images including blurred images are subjected to weighting using a blur map, and then three-dimensional images are obtained. It can be confirmed that when the original mesh model is created, the restoration area is enlarged and the error is reduced compared to the case where the 3D mesh model is created using the conventional 3D reconstruction method.
As described above, by using the present embodiment, it is possible to restore the three-dimensional shape with high accuracy even when the multi-viewpoint image includes an image in which a part of the target is blurred.

All or part of the three-dimensional geometric model generation device 1 in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" means a medium that dynamically retains a program for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing a part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be implemented using a programmable logic device such as FPGA.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the gist of the present invention.

Reference Signs List 1 3D geometric model generation device 100 3D restoration processing unit 101 camera parameter estimation unit 102 depth map generation unit 103 3D point group generation unit 104 viewpoint selection unit 105 3D point optimization unit 106 Three-dimensional shape model generation unit 107 Blur map generation unit 108 Multi-view image input unit 109 Multi-view image storage unit 110 Three-dimensional point group storage unit 111 Three-dimensional shape model storage unit 112 Blur map storage unit

Claims (9)

A three-dimensional shape model generation device for generating a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by imaging the object from different viewpoints,
a blur map generation unit that generates a blur map representing the amount of blur for each pixel in each image of the multi-view image;
Using the multi-viewpoint image and the blur map, a matching score is weighted by window matching according to the amount of blur for each pixel in the blur map, and a three-dimensional shape model representing the three-dimensional shape of the object is generated. A three-dimensional shape model generation device, comprising: a three-dimensional reconstruction processing unit that generates a three-dimensional shape model; The three-dimensional reconstruction processing unit
weighting the matching score of a stereo pair used for generating a depth map having depth information for each pixel of each image from the multi-view image according to the amount of blur for each pixel of the blur map; a depth map generator to generate;
a three-dimensional point group generating unit that generates a three-dimensional point group representing the three-dimensional shape of the object by integrating the plurality of depth maps generated by the depth map generating unit; Item 1. The three-dimensional shape model generation device according to Item 1. The three-dimensional reconstruction processing unit
a three-dimensional point group generating unit that generates a three-dimensional point group representing the three-dimensional shape of the object;
Using the multi-viewpoint image and the three-dimensional coordinates of the three-dimensional points in the three-dimensional point group, weighting of the matching score by window matching is performed according to the amount of blur for each pixel in the blur map, and the three-dimensional point a 3D point optimizer that optimizes 3D coordinates of 3D points in the group;
2. The three-dimensional point adding unit for adding a new three-dimensional point to the three-dimensional point group according to the three-dimensional coordinates of each three-dimensional point of the three-dimensional point group. 3D shape model generator. The three-dimensional reconstruction processing unit
Appropriate scores based on the geometrical positional relationship of each viewpoint corresponding to each 3D point in the 3D point group are weighted according to the amount of blur for each pixel of the blur map, and a viewpoint to be used for stereo matching is selected. further comprising a viewpoint selection unit for
4. The 3D geometric model generation device according to claim 3, wherein the 3D point optimization unit performs matching of the 3D points using the selected viewpoint. The three-dimensional reconstruction processing unit
weighting the matching score of a stereo pair used for generating a depth map having depth information for each pixel of each image from the multi-view image according to the amount of blur for each pixel of the blur map; a depth map generator to generate;
a three-dimensional point group generating unit that generates a three-dimensional point group representing the three-dimensional shape of the object by integrating the plurality of depth maps generated by the depth map generating unit;
a viewpoint selection unit that selects a viewpoint to be used for matching processing based on the appropriate score for each viewpoint corresponding to the three-dimensional point in the three-dimensional point group;
The three-dimensional shape according to claim 1, further comprising: a three-dimensional point optimization unit that performs matching of the three-dimensional points from the selected viewpoint and optimizes three-dimensional coordinates of the three-dimensional points. model generator. The three-dimensional reconstruction processing unit
a depth map generation unit that generates a depth map having depth information for each pixel of each image from the multi-view image;
a three-dimensional point group generating unit that generates a three-dimensional point group representing the three-dimensional shape of the object by integrating the plurality of depth maps generated by the depth map generating unit;
a viewpoint selection unit that weights the appropriate score for each viewpoint corresponding to the three-dimensional point in the three-dimensional point group according to the amount of blur for each pixel of the blur map, and selects a viewpoint to be used in matching processing;
A three-dimensional point optimization unit that performs matching of the three-dimensional points from the selected viewpoint and optimizes three-dimensional coordinates of the three-dimensional points. A three-dimensional shape model generation device as described. The three-dimensional reconstruction processing unit
a depth map generation unit that generates a depth map having depth information for each pixel of each image from the multi-view image;
a three-dimensional point group generating unit that generates a three-dimensional point group representing the three-dimensional shape of the object by integrating the plurality of depth maps generated by the depth map generating unit;
a viewpoint selection unit that selects a viewpoint to be used for matching processing based on the appropriate score for each viewpoint corresponding to the three-dimensional point in the three-dimensional point group;
weighting the matching score of the matching of the 3D points with the selected viewpoint according to the amount of blur for each pixel of the blur map to optimize the 3D coordinates of the 3D points; 7. The three-dimensional geometric model generation device according to claim 1, further comprising an origin point optimization unit. A three-dimensional model generation method for generating a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by imaging the object from different viewpoints,
a blur map generation process in which a blur map generation unit generates a blur map representing the amount of blur for each pixel in each image of the multi-view image;
A three-dimensional restoration processing unit weights a matching score by window matching using the multi-view image and the blur map according to the amount of blur for each pixel in the blur map, and determines the three-dimensional shape of the object. A three-dimensional shape model generation method, comprising: a three-dimensional restoration processing step for generating a three-dimensional shape model showing A program for causing a computer to operate as a three-dimensional shape model generation device for generating a three-dimensional shape model of an object from two or more multi-viewpoint images obtained by imaging the object from different viewpoints,
said computer,
Blur map generation means for generating a blur map representing the amount of blur for each pixel in each image of the multi-view image;
Using the multi-viewpoint image and the blur map, a matching score is weighted by window matching according to the amount of blur for each pixel in the blur map, and a three-dimensional shape model representing the three-dimensional shape of the object is generated. A program for operating as a three-dimensional reconstruction processing means to generate.

Images