CN104134234B - A fully automatic 3D scene construction method based on a single image - Google Patents

Abstract

The invention discloses a fully automatic method for building a three-dimensional scene model based on a single image, including: a method based on machine learning, using a training image set to train a classifier capable of roughly classifying and labeling input images. The classifier is used to divide the input image into three sub-regions of vertical, ground and sky, and a rough classification and labeling result of the image region is obtained. Based on the "credible" regions in the rough classification and labeling results, the GrabCut image segmentation algorithm is used to correct the rough classification and labeling results and obtain precise boundaries between image geometric regions. On the basis of obtaining the accurate classification and labeling results of the geometric regions of the image and the precise boundaries between the geometric regions of the image, the method of computer graphics is used to model a realistic three-dimensional scene. The fully automatic three-dimensional scene construction based on a single image is realized by adopting the method disclosed in the invention.

Description

A fully automatic 3D scene construction method based on a single image

technical field

The invention relates to the field of image-based modeling, in particular to a fully automatic three-dimensional scene construction method based on a single image.

Background technique

Image-based 3D reconstruction technology can construct realistic 3D graphics from 2D images. Image-based modeling is a new technology that has emerged in recent years. It uses directly captured images and uses as few interactive operations as possible to reconstruct the scene. Its biggest feature is to overcome the shortcomings of traditional geometric modeling and rendering technology, and can realize real-time roaming with photorealistic scenes on a computer with only ordinary computing power. Although the traditional 3D modeling tools are improving day by day, it is still a very time-consuming and labor-intensive task to construct a somewhat complex 3D model. Considering that many 3D models to be constructed can be found or shaped in the real world, 3D scanning technology and image-based modeling technology have become ideal modeling methods in people's minds; Geometric information, and the latter provides a natural way to generate photorealistic synthetic images, so it has quickly become a research hotspot in the field of computer graphics.

Image-based model reconstruction is a frontier issue in computer graphics research. This technology combines theories and methods in many fields such as computer graphics, image processing, and computer vision. It uses the two-dimensional information contained in the image scene to obtain three-dimensional data for model reconstruction, and realizes model reconstruction in the virtual scene. Therefore, It has a good application prospect in computer-aided design and reverse engineering. Image-based modeling technology is based on two-dimensional images for image understanding and final reconstruction of three-dimensional stereo, it is one of the main problems to be solved in computer vision, widely used in robot navigation, fuzzy recognition, virtual reality and building reconstruction and other fields.

The initial research on 3D reconstruction is based on geometric information methods, such as point cloud and so on. In recent years, image-based 3D reconstruction technology has emerged. It uses directly captured photos for reconstruction, which overcomes the calibration problem in traditional geometry-based reconstruction technology. It has great advantages. Therefore, image-based 3D reconstruction technology has become An important subject of research by many scholars.

Most of the current research is on the 3D reconstruction of two or more (sequence) images. Multi-image reconstruction technology needs to perform complicated preprocessing on each image to find the feature points used for matching between images, and feature point matching is a difficult point in image processing. Therefore, the use of multiple images for 3D reconstruction operations There are problems such as high reconstruction cost, complicated operation, large amount of calculation, and it is not suitable for dynamic scene reconstruction.

The main idea of 3D reconstruction based on a single image is to extract the 2D and 3D geometric information of the target such as color, shape, and coplanarity from a single digital image, so as to obtain the spatial 3D information of the target using a small number of known conditions. The three-dimensional reconstruction operation of a single image avoids the trouble of multiple image reconstruction. Its reconstruction process is simple and fast. It only needs to take a digital photo with a suitable angle to obtain the three-dimensional geometric information of the target; Multiple cameras or projectors are required for calibration, which greatly reduces the investment in manpower and material resources; and technically, only one image is preprocessed, without the need for multiple image matching, which avoids the matching difficulties of multiple image reconstruction, greatly Save time and improve efficiency. Therefore, more and more people pay attention to 3D reconstruction with a single image.

Among the current research methods, the 3D reconstruction method of a single image includes an interactive 3D scene construction method and a fully automatic 3D scene construction method. Interactive 3D scene construction methods require user interaction for guidance. Fully automatic 3D scene construction methods generally use machine learning methods to obtain corresponding scene structure classifiers based on image features, and use classifiers to classify and mark image regions. On this basis, the modeling of the 3D scene is carried out. Interactive modeling methods are highly accurate, but require user interaction guidance. The fully automatic 3D scene construction method is a research hotspot in recent years. How to quickly and accurately estimate the image region category and improve the accuracy of automatic reconstruction is the main problem faced by the fully automatic single image 3D reconstruction method.

Contents of the invention

The purpose of the present invention is to provide a fully automatic method for constructing a three-dimensional scene based on a single image, and to construct a three-dimensional scene with a sense of reality on the basis of effectively obtaining the classification and labeling results of the geometric regions of the image and the precise boundaries between the geometric regions in the image. Construct.

The purpose of the present invention is achieved by the following technical solutions: a fully automatic three-dimensional scene construction method based on a single image, comprising the following steps:

Step 1: Use the training image set to obtain a classifier capable of image geometric region division

The classifier of image geometric region division is obtained based on machine learning. First, it is necessary to collect a training image set, then use the training image set to obtain a set of training samples, and finally use the training samples to train the classifier; the training samples are in the training image set Obtained on the Internet, including sample annotation and sample extraction;

The sample annotation refers to the annotation of the geometric area of each image in the training image set, that is, the entire area of each image is divided into multiple geometric sub-areas, and each geometric sub-area should belong to three categories The three categories are: vertical area, ground area and sky area;

After the samples are labeled, it is necessary to extract the sample set that is actually used for training. In order to divide the image area into geometric sub-regions as accurately as possible, a 30*40 rectangular block is used as a sample unit, and each image is divided into a series of 30*40 samples with a certain overlapping area at a step size of 10. rectangular blocks. For each sample rectangular block, extract 1031-dimensional sample features; thus for each training image, one group of training samples (a training sample set) can be obtained, and the training sample sets of all training images form the final training sample set;

The training sample set is extracted, and a supervised training method is used to obtain a classifier that can divide the geometric region of the image, that is, a Support Vector Machine (SVM) classifier is used. The trained classifier model can output a test sample that belongs to Probabilities for the three classes;

Step 2: Use the trained classifier to divide the geometric area of the image input by the user, and obtain the result of rough classification and labeling;

Input an image, first divide the image area into a series of 30*40 sample rectangular blocks with a certain overlapping area with a step size of 10, and extract 1031-dimensional sample features for each sample rectangular block; for each sample rectangular block , the classifier outputs the probabilities that the sample belongs to three categories according to its 1031-dimensional sample features: p(v|P _i ), p(g|P _i ) and p(s|P _i ), where p(v |P _i ) represents the probability that the sample P _i belongs to the vertical area, p(g|P _i ) and p(s|P _i ) represent the probability that the sample P _i belongs to the ground area and the sky area, respectively;

For each 10*10 decision-making unit C _j in the image area, the probability of belonging to the three categories is determined by the categories of the N sample rectangular blocks containing the decision-making unit, and the probability of each decision-making unit C _j belonging to the three categories is calculated for:

Among them, N represents the number of sample rectangular blocks containing the decision-making unit C _j , and P _i represents one of the N rectangular blocks, so as to obtain the probability that the decision-making unit C _j belongs to the three categories; p(v|C _j ) Indicates the probability that the decision-making unit C _j belongs to the vertical area, p(g|C _j ) and p(s|C _j ) represent the probability that the decision-making unit C _j belongs to the ground area and the sky area, respectively;

If and only if the probability p ^* of the decision-making unit C _j belonging to a certain category is greater than 0.5, the decision-making unit is marked as this category, otherwise it is marked as an unknown category;

Step 3: Use the GrabCut image segmentation algorithm to correct the rough classification and labeling results obtained in step 2, and optimize the boundaries between the geometric regions of the image to obtain precise boundaries between the geometric regions of the image

When using the GrabCut-based image segmentation algorithm, the "credible" region in the rough classification result is used as the initial input of GrabCut to automatically optimize the rough labeling result. A set of pixels of a category, that is, the probability of belonging to a certain category is greater than 0.5 and belongs to the top 90% of all pixel sets belonging to this category; the corresponding "credible" area is calculated for each category, and the The "credible" area of a certain category P _* in the image area; the result of the rough classification is corrected by the output of the GrabCut image segmentation algorithm to obtain the precise boundary between the geometric areas in the image;

Step 4. Based on the labeling results output in step 3, the computer graphics method is used to model the 3D scene, so as to provide users with realistic 3D scene roaming.

According to the precise boundary information between the geometric areas in the image, the image area is cropped into different geometric areas; on the basis of setting the camera parameters, the relative depth information is introduced by referring to the ground, so as to restore the important vertices of the geometric areas in the image scene The three-dimensional coordinates; finally use the plane to approximate each geometric sub-area, and place each area in the three-dimensional scene according to the geometric relationship, so as to generate a realistic three-dimensional scene roaming.

In the step 1, the 1031-dimensional sample features include: 1000-dimensional Bag of Visual Words feature, 30-dimensional color feature and 1-dimensional position feature.

In the step 1, the basis function in the SVM classifier is selected as a radial basis function, the model category is selected as a multi-category classifier, and the probability estimation parameter b is set to 1, that is, the trained classifier can output a test sample respectively Probability of belonging to three classes.

The specific implementation of the step 3 is:

The calculation method for the "credible" area of a certain category P _* in the image area in (1) is:

Arrange all the pixels belonging to the category P _* in the rough labeling result in descending order according to the probability that they belong to the category, and remove the pixels with lower probability, and the percentage is k%;

Generate a binary template image M _* corresponding to P _* , M _* is the same size as the original image, and any pixel belonging to the set P _* has a value of 1 at the corresponding pixel position of the template image M _* , otherwise the value is 0;

Detect the connected region in the template image M _* , and fill in the 0-value region with an area smaller than A in the connected region;

The template image M _* is etched with a structuring element of size β. For the corroded pixels that may belong to this category, the set is denoted as , the pixel with a value of 1 in the template image M _* after erosion is regarded as the "credible" pixel of this category, and its set is denoted as ;

The "credible" pixel sets of the three categories (ground, vertical and sky areas) are respectively obtained according to the method of calculating the "credible" area with and the "maybe" pixel set with The calculation parameters are: for the vertical area, k, A, β are 10, 5000, 20 respectively; for the ground area and sky area, k, A, β are 0, 5000, 10 respectively;

(2) The implementation of the GrabCut algorithm to automatically optimize the rough labeling results is as follows:

According to the "credible" pixel set and the "possible" pixel set, each category is separately segmented, and the separate segmentation of a category among the three categories is calculated as follows: the category area is regarded as the foreground, and in addition The two-category regions are considered as background. Specifically, "trustworthy" pixels in this category are regarded as foreground pixels, "trustworthy" pixels in the other two categories are regarded as background pixels, and "likely" pixels in this category are regarded as possible foreground pixels, while The rest of the other pixels are regarded as the possible background; use the above information to initialize the GrabCut segmentation algorithm, respectively establish the mixed Gaussian model of the foreground and the background, and after segmentation, a separate segmentation result with a certain type of area as the foreground can be obtained;

According to the results of the separate segmentation, the labeling results are further optimized. The method is: on the basis of the three-step separate segmentation results, according to the method of separately segmenting a certain area, the foreground and background are separated again with the vertical area as the foreground to obtain the final Image segmentation results;

According to the results of separate segmentation of the sky and the ground, the position of the horizon can be roughly estimated, and the background area in the final image segmentation result is divided into sky and ground areas by using the horizon. The method is: the background area above the horizon is marked as the sky area , the background area below the horizon is marked as the ground area.

Based on the results of the image geometric area labeling, the computer graphics method is used to model the three-dimensional scene, providing users with a realistic three-dimensional scene roaming, including:

According to the labeling result of the image geometric area, the precise boundary between the image geometric areas is obtained, and the boundary of the ground and the erected area is approximated with a polygon to obtain a fitted polygon of the boundary with the Douglas-Peucker algorithm;

Wherein the above-mentioned steps of using the method of computer graphics to carry out the modeling of the three-dimensional scene are:

(1) To model the scene, use the pinhole camera model, the optical axis passes through the image center, the world coordinate system and the camera coordinate system coincide, and the camera field of view is set to 1.431rad;

(2) Use the reference ground plane to obtain the three-dimensional coordinates of the important vertices in the scene. The method is: introduce the reference ground plane, and set the height of the ground plane to -5; obtain the projection matrix according to the above modeling information, under the condition that the ground height is determined , through back projection, calculate the three-dimensional coordinates in the three-dimensional scene corresponding to each pixel of the ground area in the image, especially, the three-dimensional coordinates of the boundary points of the ground area and the vertical area can be obtained;

(3) According to the three-dimensional coordinates of the boundary points of the ground area and the erected area and the fitting polygon of the ground area and the border of the erected area, obtain a series of vertical planes, the method is: the fitting polygon of the border of the ground area and the erected area Each polyline on above is regarded as the intersection line of a certain vertical plane and the area of the ground, and the upper boundary of each vertical plane is determined by the boundary of the vertical area and the sky area in the image annotation result;

(4) For the vertical plane and the ground area, use texture mapping to obtain a realistic three-dimensional scene model; realistic scene roaming includes: changing the viewing angle of the camera, adjusting the focal length and changing the observation position to observe the scene model.

It can be seen from the above-mentioned technical solution provided by the present invention that a support vector machine (SVM) capable of roughly dividing an input image into different geometric sub-regions can be obtained through machine learning-based training. These subregions belong to one of three categories (vertical regions, ground regions, and sky regions). Due to the misclassification and confusion of boundaries between image geometric regions in the results of rough classification and labeling, an image segmentation algorithm is proposed to correct the results of rough classification and labeling to obtain precise boundaries between geometric regions. Using accurate boundary information, the distortion caused by inaccurate boundaries can be avoided in the construction of the 3D scene, thereby generating a realistic 3D scene model.

The advantage of the present invention compared with prior art is:

(1) The present invention combines the advantages of machine learning and image segmentation, corrects the rough classification and labeling results obtained by machine learning with the image segmentation method, and obtains more accurate boundaries between geometric regions, so that it can be used in the construction of three-dimensional scenes Better avoid scene model distortions due to inaccurate boundaries.

(2) On the basis that the accuracy of image classification and labeling obtained by the present invention is comparable to that of the prior art, the technical solution proposed by the present invention is simpler. Existing technologies focus on using more effective image features and building complex classification and labeling models to achieve higher classification and labeling accuracy. In terms of image features, the present invention only uses a small number of effective image features. As far as the classification labeling model is concerned, the present invention only uses a single classifier model. On the basis of obtaining comparable classification labeling accuracy, compared with the prior art that needs to use more image features and needs to build complex classification models, the technical solution proposed by the present invention is simpler and less complex. low and easy to implement.

Description of drawings

Fig. 1 is a system flow diagram of the technical solution of the present invention;

Fig. 2 is a partial image in the training image set used in Embodiment 1 of the present invention;

Fig. 3 is the algorithm flow chart of correcting the rough classification labeling result based on the image segmentation algorithm in the technical solution of the present invention;

Fig. 4 is the comparison diagram of the segmentation results of the GrabCut image segmentation algorithm involved in the technical solution of the present invention under different initialization methods;

Fig. 5 is the precise boundary between the ground area and the vertical area obtained under the "four-step" GrabCut algorithm based on the rough classification and labeling results proposed by the technical solution of the present invention for the input image in Embodiment 1 of the present invention;

FIG. 6 is a result diagram of observing the three-dimensional scene model of the input image under different viewing angles according to Embodiment 1 of the present invention;

Fig. 7 is according to the technical scheme of the present invention, carries out the confusion matrix obtained by classifying and labeling on database 1;

Fig. 8 is according to the technical scheme of the present invention, the comparison of the accuracy of the classification and labeling results obtained in the manner of 6-fold cross-validation on the database 2 and the classification and labeling results of the prior art on the database;

Fig. 9 is a comparison of classification and labeling accuracy obtained by using a support vector machine as a classifier for rough classification and labeling on database 1 and database 2 and the accuracy of labeling result correction using an image segmentation algorithm according to the technical solution of the present invention.

detailed description

The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The scene image described in the embodiment of the present invention is aimed at the outdoor scene image, because the content of the outdoor scene image may be composed of three types of geometric areas: vertical area, ground area and sky area. In general, the content of an outdoor scene image can be composed of the above three types of geometric areas. For example, in several common outdoor scene images shown in Figure 2, the ground area can be grass, roads, etc., and the vertical area can be buildings, trees, etc., the sky area is the sky. Since the present invention not only needs to accurately label the image content, but also needs to construct a 3D scene according to the labeling results, and the existence of a reference ground is assumed in the construction of the 3D scene, so it is suitable for the outdoor environment where the technical solution of the present invention is used to construct the 3D scene. The scene image includes at least a ground area. If only the technical solution of the present invention is used to classify and mark the image content, the scope of application is not limited to the assumption that the image needs to include the ground area.

Since the outdoor scene image content can be composed of three types of geometric regions: vertical region, ground region and sky region. The image features of different geometric regions have some distinguishable features, such as color, the common color of the sky can be blue, and the color of grass is generally green. Based on these observations, the present invention first uses the image data set to train a classifier that can divide the image content according to three geometric regions, that is, the classifier obtained through training can divide the input image into three types according to the local features of the image. Different geometric subregions. The image features used in the embodiment of the present invention include: Dense SIFT (dense scale invariant feature transformation, Dense Scale InvariantFeature Transform) feature, Bag of Visual Words (visual word bag feature), color feature (using LUV or RGB), position feature (takes the normalized height value h). The trained classifier adopts Support Vector Machine SVM (SupportVector Machine), the basis function used in training is selected as radial basis function, the model category is selected as a multi-category classifier, and the probability estimation parameter b is set to 1, that is, the trained model can Output the probability that a test sample belongs to the three categories, and the rest of the settings use the default parameters.

The above-mentioned image features used by the classifier of the image geometric area trained by machine learning are local features. When it discriminates and classifies the geometric area of the image content, although effective classification results can be obtained, the lack of global constraints will lead to some semantic problems. Therefore, the GrabCut image segmentation algorithm is used to further introduce constraints between image regions to optimize and correct the results of rough classification and labeling output by the classifier, so as to obtain image geometry More precise boundaries between regions. Using accurate boundary information, the distortion caused by unclear boundaries can be avoided in the modeling of the 3D scene, thereby generating a realistic 3D scene model.

Embodiment one

FIG. 1 is a system flowchart of a fully automatic single image-based 3D scene modeling method provided by Embodiment 1 of the present invention. The main steps of embodiment one include:

Step 1. Use the training image set to obtain a classifier capable of dividing the image into geometric regions.

Because the classifier for image geometric region division in the embodiment of the present invention is obtained based on machine learning, it is first necessary to collect a training image set, then use the training image set to obtain a set of training samples, and finally use the training samples to train the classifier.

Collections of training image sets are available through internet searches. Since the content of outdoor scene images is ever-changing, the collected training images should be representative and cover as many possible outdoor scenes as possible. Figure 2 shows some images in the training image set used in Embodiment 1. These images are several common outdoor scene images, and they contain at least one of three categories (ground, vertical and sky). The ground area in the three categories can be grass, roads, etc., the vertical area can be buildings, trees, etc., and the sky area is the sky. Of course, if there is only an application for a specific outdoor scene, the training image set can be more targeted. For example, only for outdoor street view images, you can collect street view images of different categories as the training image set.

The training samples are obtained on the training image set, including sample annotation and sample extraction.

Sample labeling refers to labeling the geometric area of each image in the training image set, that is, dividing the entire area of each image into multiple geometric sub-areas, and each geometric sub-area should belong to one of the three categories A sort of. The three categories are: vertical area, ground area and sky area. Since the classifier is trained in a supervised manner in the present invention, the sample labeling needs to be manually marked.

After the samples are labeled, it is necessary to extract the real sample set due to training. The purpose of the present invention is to divide the image area into geometric sub-regions as accurately as possible. Therefore, in the embodiments of the present invention, a 10*10 rectangular block is used as a decision unit, and a 30*40 rectangular block is used as a sample unit. In the embodiment of the present invention, each image is divided into a series of 30*40 sample rectangular blocks with a certain overlapping area at intervals of 10. Taking an image of 800*600 as an example, 58*77=4466 pieces can be obtained Sample rectangular blocks. For each sample rectangular block, 1031-dimensional sample features are extracted, including 1000-dimensional Bag of Visual Words features, 30-dimensional color features, and 1-dimensional position information.

To extract 1000-dimensional Bag of Visual Words features, firstly, it is necessary to pre-extract Dense SIFT features of each training image to form a SIFT feature set, and then use a clustering algorithm to cluster the feature set to obtain 1000 cluster centers of SIFT features. In the embodiment of the present invention, the interval step used in Dense SIFT feature extraction is 4, and the clustering algorithm adopts K-means (K-means) clustering algorithm. For each 30*40 sample rectangular block in a training image, the SIFT feature word frequency histogram of the rectangular area is counted according to the cluster center of the SIFT feature to form a 1000-dimensional Bag of VisualWords feature. The color features used in the embodiment of the present invention adopt 30-dimensional histogram features, and in LUV space, each channel counts 10-dimensional histogram features. The position information used in the embodiment of the present invention is 1-dimensional relative height information, that is, the relative height of each sample rectangular block in the image. For each sample rectangular block, a 1031-dimensional feature is extracted as the feature description of the sample. A set of training samples can be obtained for each training image, and the sample sets of all training images form the final training sample set. In the embodiment of the present invention, only pure training samples are used to train the classifier, that is, the training samples in the rectangular area where the training samples are located all belong to the same category to form the final training sample set (the final training sample set includes three types of training samples). sample).

The training sample set is extracted, and the example of the present invention adopts a supervised training method to obtain a classifier capable of dividing image geometric regions. Specifically, the classifier adopts SVM (Support Vector Machine), the basis function is selected as the radial basis function, the model category is selected as a multi-category classifier, and the probability estimation parameter b is set to 1, that is, the trained model can output The probability that a test sample belongs to each of the three classes.

Step 2. For the image input by the user, use the trained classifier to divide it into geometric regions, and obtain a rough classification and labeling result.

The purpose of step 2 is to roughly label the region category of the input image through the trained classifier. Input an image, first divide the image area into a series of 30*40 sample rectangular blocks with a certain overlapping area with a step size of 10, and extract 1031-dimensional sample features for each sample rectangular block. For each sample rectangular block, the classifier outputs the probabilities that the sample belongs to three categories according to its 1031-dimensional sample features: p(v|P _i ), p(g|P _i ) and p(s|P _i ), where p(v|P _i ) represents the probability that the sample P _i belongs to the vertical region, and p(g|P _i ) and p(s|P _i ) represent the probability that the sample P _i belongs to the ground region and the sky region, respectively.

The purpose of the present invention is to divide the image area into sub-regions as accurately as possible, so in the embodiment of the present invention, a 10*10 rectangular block is used as a decision-making unit (without overlapping each other), and each decision-making unit in the image Contained in multiple sample rectangle blocks. In the embodiment of the present invention, a sample rectangular block of 30*40 is used, and the sampling interval step is 10, so in the internal area of the image, each decision-making unit will be included in 12 sample rectangular blocks. Therefore, for each decision-making unit _Cj , its probability of belonging to the three categories can be jointly determined by the categories of the N sample rectangular blocks containing the decision-making unit. In the embodiment of the present invention, the probability of each decision unit C _j belonging to the three categories is calculated as:

Among them, N represents the number of sample rectangular blocks containing the decision-making unit C _j , and P _i represents one of the N rectangular blocks, so as to obtain the probabilities that the decision-making unit C _j belongs to the three categories. p(v|C _j ) represents the probability that the decision-making unit C _j belongs to the vertical area, and p(g|C _j ) and p(s|C _j ) represent the probability that the decision-making unit C _j belongs to the ground area and the sky area, respectively.

In the embodiment of the present invention, if and only if the probability p ^* of a decision-making unit C _j belonging to a certain category is greater than 0.5, the decision-making unit is marked as the category, otherwise it is marked as the unknown category. The classification and labeling results output by the input image through the classifier (SVM is used in the embodiment) are relatively rough. Some misclassifications mainly occur at the boundaries between geometric regions, and there are also some semantic misclassifications inside regions. In order to correct the results of rough classification and labeling to obtain precise boundaries between geometric regions, thereby facilitating the modeling of realistic 3D scenes, the present invention proposes a correction method based on image segmentation.

Step 3. Use the image segmentation algorithm to correct the rough classification and labeling results obtained in step 2, correct the classification results and optimize the boundaries between the geometric regions of the image.

Aiming at some misclassifications of rough labels and inaccurate boundaries of geometric regions, the present invention proposes a correction method based on the GrabCut image segmentation algorithm. GrabCut is an effective and interactive image segmentation algorithm for separating foreground and background. The Gaussian mixture model of the foreground and background is initialized by some initial annotation information given by the user. Figure 4 shows the results of the GrabCut image segmentation algorithm with the vertical area as the foreground under different initialization methods. (a) in Figure 4 is the vertical area obtained by the GrabCut image segmentation algorithm that only uses a rectangular frame as the segmentation range constraint. Segmentation results, (b) and (c) in Figure 4 are based on the constraints of the rectangular frame, and the foreground information (lines of the vertical area) and background information (lines of the sky and the ground area) are interactively marked by the user as GrabCut The segmentation results of the segmented input, the difference between (b) and (c) in Figure 4 is that (c) has more user interaction, and more foreground and background information is marked. (d) in FIG. 4 is the segmentation result of the vertical region obtained by the fully automatic GrabCut image segmentation algorithm based on the rough image classification and labeling results proposed by the present invention. Since the GrabCut image segmentation algorithm requires user interaction, and the present invention aims to establish a fully automatic 3D scene construction system based on a single image, the GrabCut algorithm cannot be directly used. Notice the rough classification results produced in step 2. Although there are some misclassifications, there are still most of the correctly labeled regions in the image. Therefore, the present invention proposes to use the "credible" region in the rough classification labeling result as the initial input of GrabCut. A "credible" region is defined here as a set of pixels with a high probability of belonging to a certain class, that is, the probability of belonging to a certain class is greater than 0.5 and belongs to the top 90% of all pixels belonging to the class . In an embodiment of the present invention, for each category a corresponding "trusted" region is calculated. Taking an area of one of the three categories as an example, the calculation method of the "trusted" area is as follows:

In the rough labeling result, all the pixel sets belonging to a certain category area are recorded as The pixels in the set are arranged in descending order according to the probability of belonging to the category area. After sorting in descending order, remove the collection The last k% of pixels get a new set P _* . the collection The bottom k% pixels with low probability are regarded as "unreliable" pixels and removed.

Generate a binary template image M _{* corresponding to P* .} M _* is the same size as the original image, and any pixel belonging to the set P _* has a value of 1 at the corresponding pixel position of the template image M _* , otherwise the value is 0.

Detect the connected region in the template image M _* , and cover and fill with 1 value for the 0-valued region with an area smaller than A in the connected region.

The template image M _* is etched with a structuring element of size β. For the pixels that are regarded as "probably" belonging to the category area by the eroded pixels, the set is denoted as . After erosion, the pixels with a value of 1 in the template image M _* are regarded as "credible" pixels of the category area, and their collection is denoted as .

After the above 4 steps, the "credible" pixel set of a certain category area can be obtained and the set of pixels that "likely" belong to the class region . Using the above method, "credible" pixel sets can be obtained for the ground area, vertical area and sky area respectively with and the "maybe" pixel set with In the embodiment of the present invention, (k, A, β) takes (10, 5000, 20) for the vertical area, and (k, A, β) takes (0, 5000, 10) for the ground area and the sky area.

Since GrabCut is an interactive binary segmentation algorithm aiming at separation of foreground and background, three types of regions are involved in the embodiment of the present invention: vertical, ground and sky. Therefore, the technical solution of the present invention proposes a "four-step" GrabCut algorithm based on rough classification and labeling results to automatically optimize the rough labeling results.

when acquired with Afterwards, in the embodiment of the present invention, each category is segmented separately. An individual segmentation of one of the three classes is computed by treating the region of that class as the foreground and the regions of the other two classes as the background. Specifically, "trustworthy" pixels in this category are regarded as foreground pixels, "trustworthy" pixels in the other two categories are regarded as background pixels, and "likely" pixels in this category are regarded as possible foreground pixels, while All other remaining pixels are regarded as possible backgrounds; the above information is used to initialize the GrabCut segmentation algorithm, and the mixed Gaussian models of the foreground and background are respectively established. After segmentation, a separate segmentation result with a certain type of area as the foreground can be obtained.

The result of separate segmentation can correct many misclassifications in the rough classification labeling and the boundaries between regions are more accurate, but there are still some misclassifications between the vertical area and the ground area, and between the vertical area and the sky area. In order to further optimize the labeling results , the fourth step of the "four-step" GrabCut algorithm proposed by the technical solution of the present invention, the foreground area is taken as the vertical area, and the ground and sky areas are regarded as the background area. On the basis of the three-step separate segmentation results, according to the method of separately segmenting the vertical area, the foreground and background are segmented again with the vertical area as the foreground. Figure 3 describes the algorithm flow of using the GrabCut image segmentation algorithm to correct the results of rough classification and labeling of image regions. Compared with the rough classification and labeling results, the classification and labeling results after the GrabCut image segmentation algorithm have corrected many misclassifications of the rough classification and labeling and the boundaries between geometric regions are more precise. The position of the horizon can be roughly estimated by using the results of the separate segmentation of the sky and the ground, so that geometric correction can be performed, that is, the background area in the final image segmentation result is divided into sky and ground areas by using the horizon, and the method is: above the horizon The background area of , is marked as the sky area, and the background area below the horizon is marked as the ground area

Figure 4 shows the segmentation results of the GrabCut segmentation algorithm under different initialization conditions. For more complex backgrounds, the GrabCut algorithm requires a lot of user interaction to obtain better segmentation results. However, the "four-step" GrabCut segmentation algorithm based on the rough classification and labeling results proposed by the technical solution of the present invention can obtain good segmentation results under fully automatic conditions.

Step 4. Based on the labeling results output in step 3, the computer graphics method is used to model the 3D scene, so as to provide users with realistic 3D scene roaming.

The classification and labeling results of the geometric regions of the image obtained in step 3 provide precise boundaries between geometric regions, as shown in Figure 5, the curve ABCDEF (white line) is the boundary between the ground region and the vertical region, which is well distinguished upright objects and the ground. Although the results obtained by the above steps only (geometric labeling area, horizon position and area boundary) cannot accurately restore the 3D scene model, the scene can still be modeled under reasonable assumptions based on the existing information, providing The user roams through a realistic 3D scene.

In the embodiment of the present invention, a pinhole camera model is used, the optical axis passes through the center of the image, and it is assumed that the world coordinate system and the camera coordinate system coincide, and the camera field of view is set to 1.431 rad. Since the height of the reference ground in the model affects the scale of the scene model, the height of the ground plane is set to -5 in the embodiment of the present invention. The projection matrix can be obtained from the above conditions. When the ground height is determined, the 3D coordinates in the 3D scene corresponding to each pixel of the ground area in the image can be calculated through back projection. Since step 3 provides precise boundaries of the ground and vertical areas, the three-dimensional coordinates corresponding to these boundary points can be calculated through back projection. In order to obtain the three-dimensional coordinates of the erected area, the Douglas-Peucker algorithm is first used in the embodiment of the present invention to approximate the boundary between the ground and the erected area with a polygon to obtain a fitted polygon of the boundary. Each polyline on the fitted polygon can be regarded as the intersection line between a certain vertical plane and the ground. Each polyline corresponds to a vertical plane, and the upper boundary of each vertical plane is determined by the boundary of the vertical area and the sky area in the labeling result. Therefore, the geometric model of the scene can be obtained, and a realistic three-dimensional scene can be obtained through texture mapping. Users can change the camera's angle of view, observe the position, adjust the focal length and other operations to roam the scene. Accompanying drawing 6 shows the results of observing the 3D scene model of the input image in Embodiment 1 of the present invention under different viewing angles. (a)(b)(c) in FIG. Observe the resulting plot of the scene model.

According to the technical scheme proposed by the present invention, two recognized database Popup databases (Derek Hoiem, Alexei A. Efros, and Martial Hebert, "Automatic photopop-up," in ACM Transactions on Graphics (TOG).ACM,2005,vol.24,pp.577–584.), referred to as "database 1" and Geometric context database (Derek Hoiem, Alexei A. Efros, and MartialHebert, "Geometric context from a single image," in International Conference of Computer Vision (ICCV).2005, vol.1, pp.654-661.), referred to as "database 2" to evaluate the effectiveness of the technical solution of the present invention. Database 1 contains 144 images, including 82 training images and 62 testing images. Database 2 contains 300 images, divided into 6 parts with 50 images as a part. The standard test method of database 2 adopts 6-fold cross-validation: during the test, one of them is used as the training image set in turn, and the other 5 are used as the test image set. Accompanying drawing 7 is according to the technical scheme of the present invention, trains on the database 1 with 82 training images and obtains the roughly labeled classifier, and carries out the confusion matrix obtained by classifying and labeling 62 test images. The classification labeling accuracy of the present invention corresponding to the confusion matrix is 92%, that is, on the test image set, 92% of the image pixels are correctly classified and labeled, while the baseline of the classification labeling accuracy of the database is 87%. Fig. 8 is a comparison between the accuracy of the classification and labeling results obtained by means of 6-fold cross-validation on the database 2 according to the technical solution of the present invention and the classification and labeling results of the prior art on the database. The data shows that on the standard test database 2, the baseline of classification and labeling accuracy is 86.0%, the best classification and labeling result is 88.9% at present, and the accuracy of the classification and labeling result obtained by the classification method of the present invention is 88.7%. The results show that the classification method of the present invention can obtain classification labeling accuracy comparable to that of the prior art. It should be noted that: in terms of image features, the present invention only uses a small amount of effective image features; in terms of classification and labeling models, the present invention only uses a single classifier model. Therefore, on the basis of obtaining classification and labeling accuracy comparable to that of the prior art, the technical solution proposed by the present invention appears to be more efficient than the prior art that requires the use of more image features and the construction of complex classification models. For simplicity, low complexity, and easy implementation. Fig. 9 is a comparison of classification labeling accuracy obtained by roughly labeling classifiers using support vector machines on database 1 and database 2 and accuracy of labeling result correction using image segmentation algorithms according to the technical solution of the present invention. The data shows that the corrected accuracy of annotation results on database 1 and database 2 is 4.6% and 3.5% higher than the accuracy of rough annotation, respectively. The results show that the accuracy of classification and labeling can be effectively improved by using the image segmentation algorithm proposed by the present invention to correct the rough labeling results.

Through the above description of the implementation manners, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the above-mentioned embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.), including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the method described in the embodiment of the present invention.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in the present invention. Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (3)

1. a kind of full automatic three-dimensional scene construction method based on single image, it is characterised in that comprise the following steps：

Step 1：The grader of image geometry region division can be carried out by being obtained using training image collection；

The grader of image geometry region division is obtained based on machine learning, it is necessary first to collect training image collection, then One group of training sample is obtained using training image collection, finally grader is trained using training sample；The training sample be Obtained in training image collection, including sample mark and sample extraction；

The sample mark refers to the mark that geometric areas is carried out to each width figure inside training image collection, i.e., each width figure The whole region of picture is divided into multiple geometry subregions, and each geometry subregion should be attributed to one kind in three kinds of classifications, this Three kinds of classifications are respectively：Erect region, ground region and sky areas；

, it is necessary to extract the real sample set for being used to train after sample is marked；Image-region is entered in order to reported as precisely as possible The division of row geometry subregion, using 30*40 rectangular block as sample unit, divides every piece image for interval steps with 10 Into the sample rectangular block of 30*40 with certain overlapping region a series of；For each sample rectangular block, the sample of 1031 dimensions is extracted Eigen；One group of training sample, i.e., one training sample set, and the instruction of all training images are obtained for each width training image Practice sample set and form final training sample set；

Training sample set is extracted, the classification of image geometry region division can be carried out by being obtained using the training method for having supervision Device, i.e., using support vector machines (Support Vector Machine) grader, train obtained model to export one Individual test sample is belonging respectively to the other probability of three species；

Step 2：The image that the grader obtained using training is inputted to user carries out the division of geometric areas, obtains rude classification The result of mark；

Piece image is inputted, is first that image-region is divided into a series of 30* with certain overlapping region by interval steps with 10 40 sample rectangular block, the sample characteristics of 1031 dimensions are extracted for each sample rectangular block；For each sample rectangle Block, grader exports the sample and is belonging respectively to the other probability of three species according to the sample characteristics of its 1031 dimension：p(v|P_i)、p(g| P_i) and p (s | P_i), wherein p (v | P_i) represent sample P_iBelong to erect region probability, p (g | P_i) and p (s | P_i) sample is represented respectively This P_iBelong to ground region and the probability of sky areas；

For each decision package C_jIt belongs to the other probability of three species by N number of sample rectangular block comprising the decision package Classification is together decided on, each decision package C_jIt belongs to the other probability calculation of three species：

Wherein N represents to include decision package C_jSample rectangular block number, P_iSome in N number of rectangular block is represented, so as to obtain Decision package C_jIt is belonging respectively to the other probability size of three species；p(v|C_j) represent decision package C_jBelong to the probability for erectting region, p (g|C_j) and p (s | C_j) decision package C is represented respectively_jBelong to ground region and the probability of sky areas；

And if only if decision package C_jBelong to the Probability p of certain classification^*>When 0.5, just mark the decision package for this it is described certain Classification, otherwise labels it as unknown classification；

Step 3：Using based on the rude classification annotation results obtained in GrabCut image segmentation algorithm amendment steps 2, and optimize Border between image geometry region, obtains between image geometry region accurately border；

During using based on GrabCut image segmentation algorithms, the region of " credible " is used as the first of GrabCut using in rude classification result Begin to input and fully automatically optimized rough annotation results；" credible " region is to belong to certain classification with larger possibility Pixel set, that is, the probability for belonging to certain classification is more than and 0.5 and belongs to general in all pixels set for belonging to the category Rate it is larger preceding 90%；" credible " region accordingly is calculated for each classification, is obtained for certain species in image-region Other P_*" credible " region；Based on rude classification mark " credible " region, using the output of GrabCut image segmentation algorithms come The result of rude classification mark is corrected, to obtain in image between geometric areas accurately border；

Step 4, the annotation results exported for step 3, carry out the modeling of three-dimensional scenic using the method for computer graphics, carry For the realistic three-dimensional scenic roaming of user；

According to accurately boundary information between geometric areas in image, image-region is cut into different geometric areas；Setting Determine on the basis of camera parameter, relative depth information is introduced by reference to ground, so as to recover geometric areas in image scene Important summit three-dimensional coordinate；It is final to utilize each geometry subregion of plane approximation, and regional according to geometrical relationship It is placed in three-dimensional scenic, so as to generate realistic three-dimensional scenic roaming；

The step 3 is implemented as：

(11) for certain classification P in image-region_*The computational methods in " credible " region be：

To belonging to category P in rough annotation results_*All pixels according to they belong to the category probability size descending arrange Row, remove the less pixel of probability, and its percentage is k%；

Produce one and P_*Corresponding two-value template image M_*, M_*It is all to belong to set P as artwork size_*In pixel, its In template image M_*Respective pixel positional value be 1, otherwise value be 0；

Detection template image M_*In connected region, for connected region inside exist area be less than A 0 value region, with 1 Value covering filling；

Structural element Erodent Algorithm image M by β of size_*, the picture of the category may be belonged to by being considered as the pixel being corroded Element, its set is designated asThrough excessive erosion rear pattern plate image M_*Intermediate value is 1 pixel, is considered as " credible " pixel of the category, and it collects Conjunction is designated as

Other " credible " set of pixels of three species is obtained according to the method in described calculating " credible " region respectivelyWithAnd " possibility " set of pixelsWithCalculating parameter is respectively：For erectting region, k, A, β takes 10,5000,20 respectively, for Ground region and sky areas, k, A, β take 0,5000,10 respectively；

(12) GrabCut algorithms are fully automatically optimized being embodied as rough annotation results：

Each classification is individually split respectively according to described " credible " set of pixels and " possibility " set of pixels, to certain in three classes The independent segmentation of individual classification, computational methods are：Category region is considered as prospect, two other category regions is considered as background, i.e., The pixel of " credible " in the category is considered as foreground pixel, " credible " pixel of two other classification is considered as background pixel, and will " possibility " pixel in the category is considered as possible prospect, and remaining every other pixel regards possible background as；Utilize Above- mentioned information initializes GrabCut partitioning algorithms, and the mixed Gauss model of foreground and background is set up respectively, can be with after over-segmentation Obtain the independent segmentation result by prospect of certain category regions；

Annotation results are further optimized according to the result of described independent segmentation, method is：In the base of the independent segmentation result of three steps On plinth, the method for erectting region according to independent segmentation, again to erect separation of the region as prospect progress prospect background so as to obtain Obtain image segmentation result finally；

The result individually split according to sky and ground can substantially estimate horizontal position, will be final using horizon Background area in image segmentation result is divided into sky and ground region, and its method is：Background area on horizon Domain is labeled as sky areas, and the background area under horizon is labeled as ground region；

For step 3 export annotation results, using computer graphics method carry out three-dimensional scenic modeling there is provided user Realistic three-dimensional scenic roaming, including：

The annotation results exported according to the step 3, obtain the accurate border between image geometry region, use Douglas- Fitted polygon of the Peucker algorithms to ground and the border polygonal approximation acquisition border for erectting region；

The modeling of three-dimensional scenic is carried out using the method for computer graphics, including：

(21) to described scene modeling, using pinhole camera model, optical axis is sat by picture centre, world coordinate system and camera Mark system overlaps, and camera fields of view is set to 1.431rad；

(22) reference horizontal plane of manufacturing is utilized, the three-dimensional coordinate on important summit in scene is obtained, method is：Introduce reference horizontal plane of manufacturing, ground The height of plane is set to -5；Projection matrix is obtained according to above-mentioned modeling information, under conditions of ground level determination, by anti- Projection, calculates the three-dimensional coordinate in three-dimensional scenic corresponding to each pixel of ground region in image, obtains ground area Domain and the three-dimensional coordinate for erectting zone boundary point；

(23) according to described ground region and the three-dimensional coordinate and ground region and setting regional edge of setting zone boundary point The fitted polygon on boundary, obtains a series of perpendiculars, and method is：By ground region and the fitted polygon of setting zone boundary On each broken line be considered as some perpendicular and ground region intersection, the coboundary of each perpendicular is by described The border that region and sky areas are erect in image labeling result is determined；

(24) to described perpendicular and ground region, realistic three-dimensional scene models are obtained using texture mapping； The scene walkthrough of the sense of reality includes：The visual angle of camera is converted, focus and converts observation position observation model of place.

2. according to the method described in claim 1, it is characterised in that：In the step 1, the sample characteristics of 1031 dimensions include： The position feature of the Bag of Visual Words features of 1000 dimensions, the color characteristic of 30 dimensions and 1 dimension.

3. according to the method described in claim 1, it is characterised in that：In the step 1, the basic function in SVM classifier is elected as RBF, model classification elects multi-class grader as, and probability Estimation parameter b is set to 1, that is, trains obtained grader A test sample can be exported and be belonging respectively to the other probability of three species.