CN114596503A - Road extraction method based on remote sensing satellite image - Google Patents

Abstract

The invention belongs to the field of computer road extraction, and provides a road extraction method based on remote sensing satellite images. The road extraction method based on remote sensing satellite images includes a feature encoder, an iterative feature enhancement sub-network and a multi-task decoder. The iterative feature enhancement sub-network unit includes a semantic-guided feature enhancement module, a direction-aware feature aggregation module and two side branches; the semantic-guided feature enhancement module includes feature selection and feature fusion; the direction-aware feature aggregation module includes the residuals of multiple branches structure, each branch includes a direction-aware deformable convolution and a ReLU unit; the proposed semantic-guided feature enhancement module and direction-aware feature aggregation module and direction-aware deformable convolution make better use of road directions to automatically align convolutions Nuclear receptive field and road area; it solves the connectivity problem of automated road extraction tasks in remote sensing satellite images, and the results output by this method are highly accurate and efficient.

Description

A road extraction method based on remote sensing satellite images

technical field

The invention relates to the field of road extraction in the field of computer vision, in particular to a road extraction method based on remote sensing satellite images.

Background technique

The image semantic segmentation task aims to help the computer understand the object categories and positions contained in the real environment, and identify the content and corresponding positions in the image according to the user-defined object categories. The remote sensing satellite image road extraction task based on semantic segmentation technology aims to predict the binary road segmentation map and locate the road position in the remote sensing satellite image. In recent years, with the great success of deep neural network-based machine learning models in many computer vision tasks such as image classification, object detection, and semantic segmentation, deep learning-based methods have significantly improved on some large public datasets. quality of road extraction from remote sensing satellite images.

However, recent studies have found that the task of road extraction from remote sensing satellite images has more technical challenges than the task of image semantic segmentation. In remote sensing satellite images, the road area is often blocked by clouds or high-rise buildings and their shadows. The appearance of the road area is greatly affected by objective factors such as weather, light intensity and shooting angle, and the road area and the adjacent non-road areas have a high degree of Similar texture features. Therefore, directly applying image semantic segmentation technology to the task of road extraction from remote sensing satellite images usually results in a relatively fragmented road network, which is far behind the requirements of industrial applications. At the same time, researchers try to obtain better road extraction results by improving the neural network structure. Abhishek Chaurasia et al. proposed a new network structure LinkNet to enhance information transfer, increase the information flow between the encoder and the decoder, and input the spatial information directly from the encoder to the decoder. Based on previous results, Lichen Zhou et al. designed a dilated convolution module after the encoder, and the dilated module part contains convolutional convolution in cascade mode and parallel mode. The receptive field of each path is different, so the network can capture multi-scale features for better detection results. Yao Wei et al. used the road boundary as prior information to reduce the bias along the road edge direction. Anil Batra et al. proposed a multi-task learning-based method for road extraction from remote sensing images, StackHourglass, which uses road direction as supervision information and preliminarily proves that road direction information can bring benefits to road extraction tasks. These methods achieve some improvement in road coherence, but do not fully utilize the direction information of the road to assist the road extraction task. The establishment of road topology always follows the guidance of road direction, and reasonable use of road direction can promote more accurate road extraction results. Road directions are inherently related to the road extraction task.

Based on the investigation and analysis of the existing remote sensing image road extraction method StackHourglass, the present invention proposes a road extraction method based on remote sensing satellite images according to the appearance characteristics of remote sensing satellite images and the geometric characteristics of road elements.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to overcome the defect of insufficient utilization of road direction information in the existing method StackHourglass, and to provide a new method for fully utilizing road direction information to assist road extraction. The purpose of the present invention is to give a binary mask representing road pixels by the proposed method under a given single RGB image, and the present invention is suitable for various remote sensing satellite images.

The present invention is a road extraction method based on remote sensing satellite images, which is realized based on a convolutional neural network. The present invention uses road directions as prior knowledge to more accurately infer road areas, especially in occluded and noisy areas, thereby producing more complete and consistent road extraction results. At the same time, the present invention utilizes the task of road extraction to assist the task of road direction prediction. The proposed network consists of three components: a feature encoder, an iterative feature enhancement sub-network and a multi-task decoder.

The technical scheme of the present invention is as follows: a road extraction method (IterNet) based on remote sensing satellite images, comprising a feature encoder, an iterative feature enhancement sub-network and a multi-task decoder; the specific extraction steps are as follows;

Step 1: the remote sensing satellite image is converted into RGB image and input to the feature encoder to carry out feature extraction, and the output resolution is the feature map of the original input image down-sampling 4 times of size, extracts road semantic feature and road direction feature;

Step 2: The features extracted in step 1 are sent to the iterative feature enhancement sub-network; the iterative feature enhancement sub-network includes a plurality of feature enhancement units, each feature enhancement unit includes a semantically guided feature enhancement module (SGFE), a direction-aware feature aggregation module (OAFA) and two side branches;

Input road semantic features and road direction features to the semantic guidance feature enhancement module to obtain enhanced road direction features; the enhanced road direction features are divided into two branches, one branch is used for feature fusion, and the other branch is input to one side The branch obtains the preliminary road direction prediction results, and the side branch consists of two convolutional layers;

The semantic-guided feature enhancement module includes a feature selection part and a feature fusion part; the feature selection part includes a convolution layer, an activation function layer and an hourglass-shaped sub-network;

In the feature selection part, the input road semantic features and road direction features are combined in a cascade manner, and the combined features are sent to the convolution layer and the activation function layer to obtain the single-channel feature confidence;

B=Sigmoid(Ψ ₁ (cat(F _e , F _o ), Θ ₁ )) (1)

Among them, B is the single-channel feature confidence, cat( , ) represents the cascade along the channel direction, Ψ ₁ ( , Θ ₁ ) represents the convolutional layer with parameter Θ ₁ ; F _e represents the road semantic feature; F _o represents the road direction feature;

The generated single-channel feature confidence is used for feature selection in the numerical domains of road semantic features and road direction features, respectively, and input to the hourglass-shaped sub-network; the hourglass-shaped sub-network first downsamples the features after feature selection, and then upsamples them. It is beneficial to expand the receptive field to better capture the global context information;

F _a =Ψ ₂ ((1-B)·F _e +B·F _o ,Θ ₂ ) (2)

Wherein Ψ ₂ ( , Θ ₂ ) represents a convolutional layer with a parameter of Θ ₂ ; F _a is the feature generated through feature selection;

The feature fusion part uses a fusion mask to perform weighted feature fusion of the input road semantic features, road direction features and the results of the feature selection part to obtain enhanced road direction features;

F′ _o =(1+M)·F _o +(1+M)·F _e , (3)

M=Sigmoid(D(E(F _a )))

Among them, D( ) and E( ) represent the encoding part and decoding part in the hourglass-shaped sub-network, respectively, F′ _o represents the enhanced road direction feature, and M represents the fusion mask;

Preliminary road direction prediction results and road semantic features are input to the direction-aware feature aggregation module to generate enhanced road semantic features; the enhanced road semantic features are divided into two branches, one branch is used for feature fusion, and the other branch is input to the other Side branches, which produce preliminary road segmentation results for supervised network learning;

The direction-aware feature aggregation module adjusts the shape and direction of the receptive field according to the preliminary road direction prediction results, focusing on the information area for road extraction; the direction-aware feature aggregation module includes a residual structure of multiple branches, each branch includes a direction-aware Deformable convolution (OD-Conv) and a ReLU unit; different branches use convolution kernels of different sizes and shapes; the features obtained by all branches and the input road semantic features are weighted and combined to output the enhanced road semantic features;

F′ _e = F _e +λ·∑ _i Φ ₀ (F _e , α, θ _i ) (4)

Among them, F _e represents the input road semantic feature, F′ _e represents the enhanced road semantic feature; α represents the road direction obtained by the road direction feature of the side branch, i represents the sequence number of the branch, Φ ₀ represents the parameter θ _i The direction-aware deformable convolutional layer of λ is the hyperparameter of the residual structure, which is used to balance the weight of the aggregated features and the original features;

W represents the parameters of the conventional 2D convolutional layer, and the output of the conventional convolutional layer can be calculated by formula (5).

where X represents the input feature map and Y(p) represents the output value at position p. R denotes the receptive field representing the 2D convolution. Taking the 3×3 convolution kernel as an example, the two-dimensional grid R can be defined as Equation (6).

R={(-1,-1),(-1,0),,...,(0,1),(1,1)} (6)

The convolution kernel of the direction-aware deformable convolution automatically adjusts the receptive field according to the preliminary road direction prediction results; the output of the direction-aware deformable convolution layer is;

where X represents the input feature map; Y(p) represents the output value at position p; R represents the receptive field; α(p) represents the predicted road direction angle at position p, r(p, α) represents the applied coordinate p= Rotation translation at [p ^x , p ^y ]; W(p _n ) weight of convolution kernel receptive field;

Fuse the enhanced direction feature, enhanced road semantic feature, original road semantic feature and original road direction feature, and the fused feature is input to the next feature enhancement unit;

Step 3: The output features of the final sub-network are sent to the multi-task decoder to obtain road extraction results and road direction prediction results.

The convolution kernels of different sizes and shapes are 5×1, 1×5 and 3×3, respectively.

When the r(p, α) is a non-integer, bilinear interpolation is used to sample the input feature map X.

The feature encoder selects ResNet50;

Beneficial effects of the present invention: the present invention solves the connectivity problem of automated road extraction tasks in remote sensing satellite images, and the results output by the method are highly accurate and efficient; the present invention can fully utilize the prior information contained in the road direction , to help improve the result accuracy of the road extraction task; the proposed semantic-guided feature enhancement module can supplement and enhance the road direction information by using the road semantic information; the proposed OD-Conv can better utilize the road direction to automatically align the convolution kernel Feel the wild and road areas.

Description of drawings

FIG. 1 is the overall network structure of the present invention.

FIG. 2 is a schematic diagram of the semantic-guided feature enhancement module SGFE proposed by the present invention.

FIG. 3 is a schematic diagram of the direction-aware deformable convolution proposed by the present invention. The left is the receptive field on the input image, and the right is the proposed direction-aware deformable convolution.

FIG. 4 is a schematic diagram of the direction-aware feature aggregation module OAFA proposed by the present invention.

Fig. 5 is the iterative feature enhancement sub-network proposed by the present invention.

6 is the experimental results of the present invention, from left to right are (a) remote sensing satellite images, (b) road extraction true values, (c) results using only multi-task learning, (d) results using SGFE modules, (e) Results using the OAFA module and (f) results from the method of the present invention.

Detailed ways

The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

FIG. 1 is the overall structure of IterNet proposed by the present invention. IterNet consists of a remote sensing satellite image feature encoder, an iterative feature enhancement sub-network and a multi-task decoder. After the remote sensing satellite images are converted into RGB images, they are sent to the feature encoder proposed by the present invention for feature extraction, and the extracted remote sensing image features are sent to the iterative feature enhancement sub-network. The iterative feature enhancement sub-network consists of multiple feature enhancement units, and each feature enhancement unit includes a SGFE module, an OAFA module and two side branches. In each feature enhancement unit, the input feature extracts the road semantic feature and the road direction feature of the road respectively through the convolution operation, the feature enhancement is performed by the SGFE module and the OAFA module proposed by the present invention, and the fused feature is output and sent to the next feature unit.

Figure 2 is the SGFE module proposed by the present invention, which can use road semantic information to enhance the feature of the road direction. Figure 4 is the OAFA module proposed by the present invention, which can adjust the shape and direction of the OD-Conv receptive field by using the preliminary road direction prediction result , which automatically focuses on the information area for road extraction. Figure 3 shows the OD-Conv proposed by the present invention, which can adaptively focus on the information area extracted by the road, aggregate the information of the road surface and its surrounding environment, and make the road extraction result more accurate. Figure 5 is the iterative feature enhancement sub-network proposed by the present invention. The output of the previous feature enhancement unit is used as the input of the next unit, and the first feature enhancement unit uses the output of the feature encoder as the input feature to enhance the feature. The last feature enhancement unit inputs its enhanced features into a two-branch multi-task decoder, restores the resolution of the image through a series of convolution and upsampling operations, and outputs the final prediction result.

The present invention uses the DeepGlobe data set and the Massachusetts data set to train and test the method. The DeepGlobe dataset includes images from three different regions of Thailand, Indonesia, and India with a total area of 2,220 square kilometers. The dataset contains satellite images of cities and suburbs and the corresponding pixel-level ground-truths. The ground resolution of the remote sensing images is 50cm and the pixel resolution is 1024×1024. The present invention uses 4696 images for training and 1530 images for testing. The Massachusetts dataset consists of 1171 images of Massachusetts. The size of each image is 1500 × 1500 pixels, and the spatial resolution is 1m per pixel. This dataset is collected from aerial imagery and contains satellite images of cities and suburbs and the corresponding pixel-level ground-truths, and the present invention uses the validation set of this dataset for testing.

The present invention is mainly divided into two subtasks of data processing and network training. In terms of data processing, the present invention follows the setting of Anil Batra et al., and uses the true value of the road surface to generate the true value of the road direction prediction task. First, the pixel-level annotation is skeletonized to generate the ground truth of the road centerline, and then the ground truth value of the road centerline is smoothed using the smoothing algorithm proposed by David H Douglas et al. For each section of the road centerline, the present invention generates a series of key points by means of interpolation and magnification, and calculates the direction angle between every two adjacent key points. In order to simplify the direction estimation task, the present invention sets the interval size to 10° to cluster the directions. The present invention expands the skeletonized road direction ground truth by 20 pixels to the outside to generate pixel-level road direction ground truth for the supervision of the road direction prediction task. Finally, the present invention uses the 36-dimensional direction class to annotate the direction of the road surface in the road area, the non-road area is marked as the background class, and the road direction prediction problem is transformed into a pixel-level classification that converts the road direction into 36 types of angles and 1 type of background. question.

For network training, all experiments of the present invention are performed on a server with 16GB RAM and two NVIDIA 2080Ti GPUs (11GB), using the PyTorch framework to implement training and testing. During training, the present invention uses random cropping operations on images of size 512 × 512 to generate images of size 256 × 256 as input to IterNet. In addition, the present invention employs data augmentation operations such as random horizontal flipping, mirroring, and rotation to avoid overfitting. The present invention uses the stochastic gradient descent method to optimize the entire network with a momentum value of 0.9, a batch size of 32, and a total of 160 rounds of training. The initial learning rate is 2e-2, which decreases by a factor of 10 after the 60th epoch, and is set to 5e-4 after the 90th epoch, and decreases by a factor of 10 after the 120th epoch. In the inference process, the image size of Deepglobe dataset is 512×512, and the image size of Massachusetts dataset is 320×320. The images are directly input to the network without further cropping operations.

The comparison method selected in the present invention is StckHourglass. The StckHourglass method simply uses the road direction information to supervise to obtain better road extraction results. For a fair comparison, the StckHourglass method uses its public code and its suggested parameters. settings, and both use the same pretrained network and test on the same test set. From the final experimental results, the IterNet method proposed in the present invention achieves the best performance in all different experimental settings for the indicators IoU, F1, Precision and Recall. The higher the indicators such as IoU, F1, Precision and Recall, the higher the accuracy of the method. The specific experimental results on the Deepglobe dataset are shown in Table 1 below:

Table 1 Specific experimental results on the Deepglobe dataset

The specific experimental results on the Massachusetts dataset are shown in Table 2 below:

Table 2 Specific experimental results on the Massachusetts dataset

Claims (3)

1. A road extraction method based on remote sensing satellite images is characterized by comprising a feature encoder, an iterative feature enhancer network and a multitask decoder; the specific extraction steps are as follows;

step 1: converting the remote sensing satellite image into an RGB image, inputting the RGB image into a feature encoder for feature extraction, outputting a feature map with the resolution being 4 times of the down-sampling size of the original input image, and acquiring road semantic features and road direction features;

step 2: the features extracted in the step 1 are sent into an iterative feature enhancement sub-network; the iterative feature enhancer network comprises a plurality of feature enhancement units, wherein each feature enhancement unit comprises a semantic guide feature enhancement module, a direction perception feature aggregation module and two side branches;

inputting road semantic features and road direction features to a semantic guide feature enhancing module to obtain enhanced road direction features; the enhanced road direction characteristics are divided into two branches, one branch is used for characteristic fusion, the other branch is input into a side branch to obtain a preliminary road direction prediction result, and the side branch is composed of two convolution layers;

the semantic guide feature enhancement module comprises a feature selection part and a feature fusion part; the characteristic selection part comprises a convolution layer, an activation function layer and an hourglass-shaped sub-network;

in the feature selection part, the semantic features and the direction features of the input road are merged in a cascading mode, and the merged features are sent to a convolution layer and an activation function layer to obtain a single-channel feature confidence coefficient;

B＝Sigmoid(Ψ₁(cat(F_e,F_o),Θ₁)) (1)

where B is the confidence of the single-channel feature, cat (·,) represents the cascade in the channel direction, Ψ₁(·,Θ₁) The expression parameter is theta₁The convolutional layer of (1); f_eRepresenting road semantic features; f_oRepresenting road direction characteristics;

the generated single-channel feature confidence coefficient respectively selects the features in the road semantic feature and road direction feature numerical value domains and inputs the feature into an hourglass-shaped sub-network; the hourglass-shaped sub-network firstly carries out down-sampling and then carries out up-sampling on the selected features;

F_a＝Ψ₂((1-B)·F_e+B·F_o,Θ₂) (2)

therein Ψ₂(·,Θ₂) The expression parameter is theta₂The convolutional layer of (1); f_aSelecting the generated features for the passed features;

the feature fusion part performs weighted feature fusion on the input road semantic features, road direction features and the results of the feature selection part by using a fusion mask to obtain enhanced road direction features;

F′_o＝(1+M)·F_o+(1+M)·F_e, (3)

M＝Sigmoid(D(E(F_a)))

wherein D (-) and E (-) represent the encoding and decoding parts, F ', respectively, in an hourglass shaped sub-network'_oRepresents an enhanced road direction feature, M represents a fusion mask;

inputting the preliminary road direction prediction result and road semantic features into a direction perception feature aggregation module to generate reinforced road semantic features; the reinforced road semantic features are divided into two branches, one branch is used for feature fusion, the other branch is input to the other side branch, and a preliminary road segmentation result is generated and used for supervising network learning;

the direction perception characteristic aggregation module adjusts the shape and the direction of a receptive field according to the preliminary road direction prediction result and focuses on an information area for road extraction; the direction perception feature aggregation module comprises a residual error structure of a plurality of branches, and each branch comprises a direction perception deformable convolution and a ReLU unit; different branches adopt convolution kernels with different sizes and shapes; weighted combination of the features obtained by all branches and the input road semantic features is carried out to output the reinforced road semantic features;

F′_e＝F_e+λ·∑_iΦ₀(F_e,α,θ_i) (4)

wherein, F_eRepresenting road semantic features, F'_eRepresenting the reinforced road semantic features; alpha represents the road direction of the side branch obtained from the road direction characteristics, and i represents the pointNumber of branches,. phi₀With the expression parameter theta_iLambda is a hyper-parameter of a residual structure, and is used for balancing the weight of the aggregation characteristic and the original characteristic;

a convolution kernel of the direction perception deformable convolution automatically adjusts the receptive field according to the preliminary road direction prediction result; the direction-sensing deformable convolution layer outputs;

wherein X represents an input feature map; y (p) represents an output value at position p; r represents receptive field; α (p) denotes a predicted road direction angle at a position p, and r (p, α) denotes a direction applied to a coordinate p ═ p [ p ]^x,p^y]Rotational translation of (c); w (p)_n) Weights of the convolution kernel receptive fields;

fusing the enhanced direction features, the enhanced road semantic features, the original road semantic features and the original road direction features, and inputting the fused features into a next feature enhancement unit;

and step 3: and finally, the output characteristics of the sub-network are sent to a multitask decoder to obtain a road extraction result and a road direction prediction result.

2. The method for extracting a road based on remote sensing satellite images as claimed in claim 1, wherein the convolution kernels with different sizes and shapes are respectively 5 x 1, 1 x 5 and 3 x 3.

3. The method for extracting a road based on remote sensing satellite images as claimed in claim 1 or 2, wherein when r (p, α) is a non-integer, bilinear interpolation is adopted to sample the input feature map X.

Images