CN110599537A - Mask R-CNN-based unmanned aerial vehicle image building area calculation method and system - Google Patents

Abstract

The present invention provides a method and system for calculating the building area of UAV images based on Mask R-CNN. First, the UAV is used to collect multiple images of pre-selected areas, and the size of each image is consistent; Contains images of buildings; mark the buildings in the remaining images, and use the marked images as a training image set; prepare a complete satellite image dataset; use the Mask R-CNN algorithm, first use the satellite image data The training image set is used to pre-train the initial model to obtain an initial model; the training image set is used to train the initial model to obtain the final segmentation model; the drone is used to collect images of the area to be tested, stitch them into a panorama, perform down-sampling processing, and then crop into Small image; use the segmentation model to process the small image; count the number of pixels contained in each marked building; calculate the area of each building in the image according to the actual situation.

Description

Method and system for calculating building area in UAV images based on Mask R-CNN

technical field

The invention belongs to the field of geographic information science, and in particular relates to a method and system for calculating the area of a building in an unmanned aerial vehicle image based on Mask R-CNN.

Background technique

In recent years, with climate change, geological disasters have occurred more and more frequently. Especially landslides, mudslides, floods, etc. all threaten the safety of human life and property, so preventive measures must be taken in advance. In some areas prone to geological disasters, such as landslide zones, it is very necessary to conduct property assessment, and then take corresponding measures according to the property value. Property values are usually assessed based on the square footage of the home. The traditional method is usually manual operation, which takes a long time and is inefficient. In recent years, remote sensing technology has developed rapidly, so it has become a trend to use remote sensing images to extract buildings. Although remote sensing technology has made great breakthroughs, the resolution of remote sensing images is still too low. Using remote sensing images to extract buildings and calculate the area will bring large errors, which may have a negative impact on house value evaluation. In recent years, UAV technology has developed rapidly, especially in terms of range and payload. There are two reasons to use drone imagery for property appraisals. First of all, by being equipped with a high-definition camera, it is easy to obtain high-resolution images with a resolution of centimeters; second, in some areas prone to geological disasters, the terrain is complex and very dangerous, but the use of drones can be used in these areas. Specify an area to shoot. Therefore, the present invention proposes a new method for automatically calculating the area of buildings based on aerial images taken by drones.

In the traditional image segmentation algorithm, the watershed algorithm is a more popular method. In the segmentation process, it takes the similarity between adjacent pixels as an important reference, so as to obtain images with similar spatial positions and similar gray values. pixels, and then form a closed contour connected with each other. The common operation steps of the watershed algorithm are: grayscale the color image, then obtain the gradient map, and finally, perform the watershed algorithm on the basis of the gradient map to obtain the edge line of the segmented image. In the watershed algorithm, the gradient image is thresholded, and the selection of an appropriate threshold has a great impact on the final segmented image. Therefore, the choice of threshold is the key to the effect of image segmentation. In addition, traditional segmentation algorithms also include clustering and edge detection.

In recent years, deep convolutional neural networks (DCNNs) have become very popular in computer vision applications. DCNNs usually contain many convolutional layers, which can learn deep features from training data. Therefore, deep convolutional neural networks are introduced into the segmentation task and have achieved good results. However, these methods still have limitations. It is difficult to achieve good results with less training data. As a supervised learning method, image segmentation based on deep neural network usually requires a large amount of training data, but in some cases the number of samples is small. Remote sensing images are quite similar to UAV images, and the former can be obtained on some open source websites. Therefore, with the help of the concept of transfer learning, a large number of remote sensing images can be used to pre-train the network, and then a small number of UAVs can be used to take aerial photos The image is fine-tuned to achieve a nice effect.

At present, many deep learning methods have been used for segmentation tasks. These methods can be divided into two categories. The first category is semantic segmentation, such as FCN and deeplab series, and the second category is instance segmentation, such as FCIS and Mask R-CNN. Semantic segmentation refers to classifying each pixel in an image, but it cannot distinguish different objects in the same class. Instance segmentation can be seen as an extension of semantic segmentation. Different from semantic segmentation, instance segmentation will distinguish each instance. Even if it is the same type of target, instance segmentation will still mark each target and distinguish it with different colors and bounding boxes. Instance segmentation can be seen as a combination of object detection and semantic segmentation. Since it is necessary to distinguish adjacent objects in a specified area, instance segmentation is more suitable for calculating the area of buildings in a specified area than semantic segmentation.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method and system for calculating the building area of UAV images based on Mask R-CNN to solve the above-mentioned technical defects. .

A method for calculating the building area of UAV images based on Mask R-CNN, including:

S1. Use drones to collect multiple images of pre-selected areas, and keep the size of each image consistent;

S2. Deleting the images collected in S1, and deleting images that do not contain buildings at all;

S3, mark the remaining images of S2 one by one, mark the buildings in the images, and use the marked images as the training image set;

S4. Prepare a satellite image data set, which contains a plurality of satellite images that have been marked with buildings;

S5. Using the Mask R-CNN algorithm, first use the satellite image dataset for pre-training, and obtain an initial model after the pre-training is completed;

S6. Using the training image set to train the initial model, the model converges after multiple rounds of training, and obtains the final segmentation model;

S7. Using the unmanned aerial vehicle to collect images of the area to be measured, splicing the collected multiple images into a panorama, down-sampling the panorama, and then cutting it into multiple small images of the same size;

S8, use the segmentation model to process all the small images obtained in S7, to mark out the buildings in each small image;

S9, counting the number of pixels contained in each marked building;

S10. According to the actual situation, set the unit pixel area represented by each pixel, and calculate the area of each building in the image.

Further, the image collected in S1 is a three-channel RGB image.

Further, labelme software is used in S3 to label buildings on the image.

Further, the calculation method in S10 is to multiply the unit pixel area by the number of pixels.

The UAV image building area calculation system based on Mask R-CNN includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device for realizing any one based on Mask R-CNN A CNN-based method for building area calculations from drone images.

Compared with the prior art, the advantage of the present invention is that: Mask R-CNN is selected as the segmentation model , and Mask R-CNN has the advantages of simple structure, flexibility, and remarkable segmentation effect. After executing Mask R-CNN, the outline of each building can be obtained and the number of pixels on the outline of each building can be counted, and then the area of each building can be calculated accordingly by using the unit area represented by each pixel. The precision of the result obtained by this method is obviously higher than that of the traditional calculation method.

Description of drawings

The present invention will be further described below in conjunction with accompanying drawing and embodiment, in the accompanying drawing:

Fig. 1 is the flow chart of the method for calculating the building area of the UAV image based on Mask R-CNN in the present invention;

Figure 2 is a qualitative experiment comparison diagram of various algorithms;

Figure 3 is a comparison chart of the quantitative indicators of each algorithm;

Fig. 4 is the rendering of the present invention using Mask R-CNN segmentation;

Fig. 5 is a graph comparing the area calculated by the present invention with the real value.

Detailed ways

In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation manners of the present invention will now be described in detail with reference to the accompanying drawings.

The method for calculating the building area of UAV images based on Mask R-CNN, as shown in Figure 1, includes:

S1. Use drones to collect multiple images of pre-selected areas, each image size is consistent, and the image is a three-channel RGB image;

S2. Deleting the images collected in S1, and deleting images that do not contain buildings at all;

S3, use the labelme software to label the remaining images of S2 one by one, label the buildings in the image, and use the labeled image as a training image set;

S4. Prepare a complete satellite image data set, which contains multiple satellite images that have been marked with buildings;

S5. Using the Mask R-CNN algorithm, first use the satellite image dataset for pre-training, and obtain an initial model after the pre-training is completed;

S6. Using the training image set to train the initial model, the model converges after multiple rounds of training, and obtains the final segmentation model;

S7. Use drones to collect images of the area to be measured, stitch the collected images into a panorama, perform down-sampling processing on the panorama, and then crop it into multiple small images of the same size, the size of which meets the segmentation model processing requirements;

S8, use the segmentation model to process all the small images obtained in S7, to mark out the buildings in each small image;

S9, counting the number of pixels contained in each marked building;

S10. According to the actual situation, set the unit pixel point area represented by each pixel point, and multiply the unit pixel point area by the number of pixel points to calculate the area of each building in the image.

The Mask R-CNN model used in the present invention is a combination of target detection and semantic segmentation, called instance segmentation or target segmentation. Based on the target detection algorithm Faster R-CNN, Mask R-CNN adds a semantic segmentation algorithm full convolutional neural network (FCN) algorithm as a segmentation branch. When the picture passes through Faster R-CNN, many regions of interest (RoI) will be generated, and FCN is applied to each region of interest to realize the classification of pixels. Unlike Faster R-CNN, Mask R-CNN uses RoI Align instead of RoI Pool, which can solve the problem of spatial misalignment and is obviously helpful to improve the quality of segmentation. Furthermore, binary loss is adopted instead of multinomial loss, which can produce accurate binary masking. Another feature of Mask R-CNN is to use a residual network or an improved ResNet network instead of the traditional vgg network to enhance the ability to extract features. ResNet was also proposed by He Kaiming and others and won the ILSVRC2015 competition. Compared with VGGNet, ResNet works better, but has fewer parameters. In general, the structure of ResNet can speed up the training of deep neural networks, and the accuracy is also greatly improved. Mask R-CNN is flexible and can be used for a variety of computer vision tasks, including object detection, image segmentation, and human pose recognition. In the COCO challenge, Mask R-CNN performed better than various previous models. Unlike FCIS proposed by Microsoft, Mask R-CNN is simpler, has better performance, better scalability, and more functions. Mask R-CNN can change different backbone structures, such as Resnet-101 or Resnet-101-FPN Wait. Here, mainly FPN solves the problem of multi-scale detection. In short, Mask R-CNN has three main improvements over Faster R-CNN. First, explore various network structures as the backbone network of Mask R-CNN. Second, use the alignment of interest (RoI Align) instead of pooling of interest (RoI Pool). Third, add the FCN algorithm as a segmentation branch. In our task, Mask R-CNN identifies each object with a bounding box. Each bounding box is then segmented into building and non-building regions.

The qualitative experiments of various algorithms are shown in Figure 2. The first column is the true value, the second column is the FCN segmentation result, the third column is the deeplab segmentation result, the fourth column is the SegNet segmentation result, and the fifth column is the method of the present invention The results of the segmentation; the quantitative index comparison of the above algorithms are shown in Fig. 3.

As shown in Figure 4, it is the result of using Mask R-CNN segmentation. In the figure, seven buildings are selected, namely A, B, C, D, E, F, and G. The area calculated by the present invention and the real value The comparison results are shown in Figure 5, where GT is the real value and the unit is square meters.

In summary, the present invention selects Mask R-CNN as the segmentation model, and adopts the method of neural network learning to train and optimize the model. Mask R-CNN has the advantages of simple structure, flexibility, and remarkable segmentation effect. After executing Mask R-CNN, the outline of each building can be obtained and the number of pixels on the outline of each building can be calculated, and then the area of each building can be calculated accordingly by using the unit area represented by each pixel. The precision of the result obtained by this method is obviously higher than that of the traditional calculation method.

Embodiments of the present invention have been described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementations, and the above-mentioned specific implementations are only illustrative, rather than restrictive, and those of ordinary skill in the art will Under the enlightenment of the present invention, many forms can also be made without departing from the gist of the present invention and the protection scope of the claims, and these all belong to the protection of the present invention.

Claims (5)

1. The method for calculating the building area of unmanned aerial vehicle images based on Mask R-CNN is characterized in that, comprising: S1. Use drones to collect multiple images of pre-selected areas, and keep the size of each image consistent; S2. Deleting the images collected in S1, and deleting images that do not contain buildings at all; S3, mark the remaining images of S2 one by one, mark the buildings in the images, and use the marked images as the training image set; S4. Prepare a satellite image data set, which contains a plurality of satellite images that have been marked with buildings; S5. Using the Mask R-CNN algorithm, first use the satellite image dataset for pre-training, and obtain an initial model after the pre-training is completed; S6. Using the training image set to train the initial model, the model converges after multiple rounds of training, and obtains the final segmentation model; S7. Using the unmanned aerial vehicle to collect images of the area to be measured, splicing the collected multiple images into a panorama, down-sampling the panorama, and then cutting it into multiple small images of the same size; S8, use the segmentation model to process all the small images obtained in S7, to mark out the buildings in each small image; S9, counting the number of pixels contained in each marked building; S10. According to the actual situation, set the unit pixel area represented by each pixel, and calculate the area of each building in the image. 2. the UAV image building area calculation method based on Mask R-CNN according to claim 1, is characterized in that, the image that gathers among the S1 is the RGB image of three channels. 3. the UAV image building area calculation method based on Mask R-CNN according to claim 1, is characterized in that, use labelme software to carry out building annotation to image in S3. 4. the UAV image building area calculation method based on Mask R-CNN according to claim 1, is characterized in that, in S10, calculation method is to multiply with unit pixel point area and pixel point number. 5. The UAV image building area calculation system based on Mask R-CNN is characterized in that it includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device for realizing rights Any method for calculating the building area of UAV images based on Mask R-CNN described in 1-4 is required.