CN106909924A - A kind of remote sensing image method for quickly retrieving based on depth conspicuousness - Google Patents

Abstract

A kind of remote sensing image method for quickly retrieving based on depth conspicuousness belongs to computer vision field, and in particular to the technology such as deep learning, conspicuousness target detection and image retrieval.The present invention, using depth learning technology, have studied a kind of method for quickly retrieving of remote sensing image with remote sensing image as research object.Multitask conspicuousness target detection model is built using full convolutional neural networks first, the model carries out conspicuousness Detection task and semantic segmentation task simultaneously, in the depth significant characteristics of network pre-training process learning remote sensing image.Then depth network structure is improved, Hash layer trim network is added, study obtains the binary system Hash codes of remote sensing image.Finally comprehensive utilization significant characteristics and Hash codes carry out similarity measurement.The present invention is for realizing that remote sensing image is accurate, efficient retrieval is practical and with significant application value.

Description

A Fast Retrieval Method of Remote Sensing Image Based on Depth Saliency

technical field

The present invention takes remote sensing images as the research object, and uses the latest research results in the field of artificial intelligence - deep learning technology to study a fast retrieval method for remote sensing images. Firstly, the fully convolutional neural network is used to build a multi-task salient target detection model, and the depth salient features of remote sensing images are calculated; then the deep network structure is improved, and the hash layer is added to learn the binary hash code; finally, the salient features and the hash are comprehensively used Xima realizes accurate and fast retrieval of remote sensing images. The invention belongs to the field of computer vision, and specifically relates to technologies such as deep learning, salient target detection and image retrieval.

Background technique

Remote sensing image data, as the basic data in the three major spatial information technologies of geographic information system (Geographic Information System, GIS), global positioning system (Global Positioning System, GPS) and remote sensing mapping technology (remote sensing system, RS), is widely used in the environment Monitoring, resource surveys, land use, urban planning, natural disaster analysis, and military. In recent years, with the development of high-resolution remote sensing satellites, imaging radars, and unmanned aerial vehicle (Unmanned Aerial Vehicle) technologies, remote sensing image data has further presented the characteristics of massive, complex, and high-resolution images, enabling efficient and accurate retrieval of remote sensing images. It has important research significance and application value for promoting the accurate extraction of remote sensing image information and data sharing.

Image retrieval technology has gradually developed from the early text-based image retrieval (Text-Based Image Retrieval, TBIR) to content-based image retrieval (Content-Based Image Retrieval, CBIR) by extracting image features. The image retrieval method based on salient objects can quickly select a few salient regions from complex scenes for priority processing, thereby effectively reducing the complexity of data processing and improving retrieval efficiency. Compared with ordinary image retrieval, the information contained in remote sensing images is complex and changeable, and the target is small and not clearly distinguished from the background. If traditional saliency detection methods are still used, it will be difficult to accurately describe and analyze the salient features of remote sensing images. In recent years, with the latest research results in the field of artificial intelligence - the introduction of deep learning technology, such as: the deep neural network represented by the fully convolutional neural network (Fully Convolutional Neural Network, FCNN), with its unique similar to the human eye The convolution kernel of local perception and the hierarchical cascade structure similar to biological nerves show excellent robustness in the learning of image depth saliency features. Its weight sharing feature also greatly reduces network parameters, and at the same time reduces the risk of overfitting to training data. It is easier to train than other types of deep networks, and can improve the representation accuracy of salient features.

Considering the increasing number of remote sensing images and the limited image semantic description ability, the present invention uses the public large-scale aerial image dataset (AID), Wuhan University remote sensing image dataset (WHU-RS) and Google Earth remote sensing images as data sources , propose a fast remote sensing image retrieval method based on depth saliency. First, build a multi-task salient object detection model based on a Fully Convolutional Neural Network (FCNN), and learn semantic information at different levels of remote sensing images on the pre-trained dataset as deep salient features and convert them into one-dimensional Column vector. Further fine-tune the neural network model, introduce the hash layer and increase the training samples, map the high-dimensional salient features of remote sensing images learned by the model to the low-dimensional space in the form of binary hash codes, and store the salient features separately. Feature vectors and hash codes build a feature database. Extract the salient feature vector and hash code of the remote sensing image to be queried through the trained model, compare the feature database, calculate the hash code Hamming distance (Hamming Distance) and the salient feature vector Euclidean distance (Euclidean Distance) to measure the similarity , to achieve fast retrieval of remote sensing images.

Contents of the invention

Different from the existing remote sensing image retrieval methods, the present invention uses deep learning technology to propose a remote sensing image rapid retrieval method based on depth saliency. First, a fully convolutional neural network (FCNN) is used to build a multi-task deep salient object detection model, which further extends the image-level classification of ordinary convolutional neural networks (CNN) to pixel-level classification. Pre-training the network on a large-scale aerial image dataset (AID), the saliency detection task and the semantic segmentation task share the convolutional layer, comprehensively learn the three-layer semantic information of remote sensing images, effectively remove feature redundancy, and accurately extract deep salient features . Secondly, adding a hash layer to the model, expanding the Wuhan University Remote Sensing Image Dataset (WHU-RS) to fine-tune the neural network, and using the deep neural network to realize the advantages of incremental learning through the stochastic gradient descent algorithm (Stochastic Gradient Descent, SGD), Learn binary hash code point by point to achieve dimensionality reduction of high-dimensional salient features, which can save storage space and improve retrieval efficiency. At the same time, compared with the traditional hash method that needs to input training samples in pairs, the method adopted by the present invention is easier to expand on large-scale data sets. The salient features learned by the neural network pre-training and fine-tuning process are converted into one-dimensional column vectors, and the feature database is constructed together with the binary hash code. Finally, in the image retrieval stage, a coarse-to-fine retrieval strategy is adopted, and binary hash codes and salient features are used to measure Hamming distance and Euclidean distance to achieve fast and accurate retrieval of remote sensing images. The main process of this method is shown in Figure 1, which can be divided into the following three steps: object detection model construction based on depth saliency, neural network pre-training and adding hash layer fine-tuning and multi-level deep retrieval.

(1) Construction of target detection model based on depth saliency

In order to effectively extract the salient area of the image, the present invention will construct a multi-task salient target detection model based on a fully convolutional neural network. The model performs two tasks simultaneously: saliency detection and semantic segmentation. Saliency detection is used for deep feature learning of remote sensing images, and depth saliency is calculated. Semantic segmentation is used to extract semantic information of objects inside images, eliminate background confusion of salient images, and supplement missing parts of salient objects.

(2) Neural network pre-training and adding hash layer fine-tuning

The present invention selects a large-scale aerial image data set (AID) as a standard data set pre-training network. In order to make the salient features learned by the salient object detection model more robust to the retrieval of Chinese remote sensing images, 6050 images were downloaded from Google Earth on the basis of the Wuhan University Remote Sensing Image Dataset (WHU-RS). The WHU-RS data set was expanded to 7000 images for fine-tuning the neural network with different illumination, shooting angles, resolutions and sizes of Chinese remote sensing images.

(3) Multi-level deep search

The invention proposes a retrieval scheme from rough to fine. Rough retrieval utilizes the binary hash code learned by the hash layer, and measures the similarity by Hamming distance. Fine retrieval maps the two-dimensional remote sensing image feature maps generated by the 13th and 15th convolutional layers into a one-dimensional column vector, which is used as a salient feature vector, and the similarity is measured by Euclidean distance. Use ranking-based evaluation criteria to count the precision of the retrieval results (Precision).

1. A remote sensing image fast retrieval method based on depth saliency, characterized in that it comprises the following steps:

Step 1: Construction of object detection model based on deep saliency

Input an RGB image, perform a series of convolution operations through 15 convolutional layers, and then perform the saliency detection task and the superpixel target semantic segmentation task to share the convolutional layer; the first 13 convolutional layers pass through the initial convolutional neural network VGGNet The size of the convolution kernel is 3×3, and the modified linear unit ReLU is used as the activation function after each convolution layer; the maximum pooling operation is performed after the 2nd, 4th, 5th, and 13th convolutional layers; volumes 14 and 15 The convolution kernel sizes of the product layers are 7×7 and 1×1 respectively, and the Dropout layer is connected after the 14th and 15th convolutional layers;

Construct the deconvolution layer by upsampling, initialize its parameters by bilinear interpolation, and update iteratively in the training and learning upsampling function; in the salient target detection task, the output image is normalized to [0,1] through the sigmoid threshold function, Learn salient features; in the semantic segmentation task, use the deconvolution layer to upsample the feature map of the last convolutional layer, and clip the upsampling result so that the output image is the same size as the input image;

Step 2: Neural network pre-training and adding hash layer fine-tuning

Step 2.1: Multi-task salient object detection model pre-training

The FCNN pre-training is carried out through the saliency detection task and the segmentation task together; χ represents the set of N ₁ training images whose width and height are W and Q respectively, Xi is the i-th image, and Y _ijk represents the i-th width and height respectively The pixel-level real segmentation map corresponding to the images of j and k, where i=1...N ₁ , j=1...W, k=1...Q; Z represents the set of N ₂ training images, and Z _n is the nth one Image, n=1...N ₂ , which has a corresponding real binary image M _n with salient objects; θ _s is the parameter of the shared convolution layer, θ _h is the parameter of the segmentation task, θ _f is the parameter of the saliency task; the formula (1) and formula (2) are respectively the cross-entropy cost function J ₁ (χ; θ _s , θ _h ) of the segmentation task and the square Euclidean distance cost function J ₂ (Z; θ _s , θ _f ) of the saliency detection task , the FCNN is trained by minimizing two cost functions:

In formula (1), is the indicator function, h _cjk is the element (j,k) of the c-th confidence segmentation map, c=1...C, h(Xi; θ _s , θ _h ) is the semantic segmentation function, and returns the confidence of C target classes Segmentation map, C is the image category contained in the pre-training data set In the formula (2), f(Z _n ; θ _s , θ _f ) is the salient map output function, and F represents the F-norm operation;

Next, use the stochastic gradient descent SGD method to minimize the above cost function on the basis of regularizing all training samples; since the data set used for pre-training does not have both segmentation and saliency labels, the segmentation task and saliency The task of sex detection is alternately performed; the training process needs to normalize the size of all original images; the learning rate is 0.001±0.01; the momentum parameter is usually [0.9,1.0], and the weight decay factor is usually 0.0005±0.0002; stochastic gradient descent learning The process performs more than 80,000 iterations in total; the detailed pre-training process is as follows:

1) Shared full convolution parameters Initialization based on VGGNet;

2) Randomly initialize the segmentation task parameters through normal distribution and saliency task parameters

3) According to with Using SGD to train the segmentation network, update these two parameters as with

4) According to with Use SGD to train the saliency network, and update the relevant parameters as with

5) According to with Using SGD to train the segmentation network, obtain with

6) According to with Use SGD to train the saliency network, and update the relevant parameters as with

7) Repeat the above steps 3-6 three times to obtain the pre-training final parameters θ _s , θ _h , θ _f ;

Step 2.2: Add a hash layer and fine-tune the network for the target domain

Between the penultimate layer of the pre-trained network and the final task layer, insert a fully connected layer containing s neurons, that is, the hash layer H, which maps high-dimensional features to low-dimensional space and generates binary hash codes Store; the H weight of the hash layer is initialized with the hash value constructed by random projection, the neuron activation function uses the sigmoid function to make the output value between 0 and 1, and the number of neurons is the code length of the target binary code;

The fine-tuning process adjusts the network weight through the back propagation algorithm; the network fine-tuning is to adjust the network weight after the tenth convolutional layer; the data size of the data set used for fine-tuning the network will be reduced by 10% compared with the data set of the pre-trained network- 50%, compared with the pre-trained network parameters, the number of iterations and learning rate of the network parameters in the fine-tuning process is reduced by 1%-10%, and the momentum parameters and weight decay factors remain unchanged;

The detailed fine-tuning process is as follows:

1) Shared full convolution parameters Split Task Parameters and saliency task parameters Obtained through the pre-training process;

2) According to with Using SGD to train the segmentation network, update these two parameters as with

3) According to with Use SGD to train the saliency network, and update the relevant parameters as with

4) According to with Using SGD to train the segmentation network, obtain with

5) According to with Use SGD to train the saliency network, and update the relevant parameters as with

6) Repeat the above steps 3-6 three times to obtain the final parameters θ _s , θ _h , θ _f ;

Step 3: Multi-Level Depth Retrieval

Step 3.1: Coarse retrieval

Step 3.1.1: Generate Binary Hash Code

Input an image I _q to be queried into the fine-tuned neural network, extract the output of the hash layer as the image signature, denoted by Out(H); the binary code is obtained by binarizing the activation value according to the threshold; for each binary bit r =1...s, output binary code according to formula (3):

Among them, s is the number of neurons in the hash layer, and the initial value setting range is [40,100]; Γ={I ₁ ,I ₂ ,…,I _n } means the data set for retrieval containing n images; the corresponding The binary code of each image is expressed as Γ _H ={H ₁ ,H ₂ ,…,H _n }, where i=1…n, H _i ∈{0,1} ^s represents the s-bit binary code generated by s neurons The code value is 0 or 1 respectively;

Step 3.1.2: Hamming distance measures similarity

The Hamming distance between two strings of equal length is the number of different characters in the corresponding positions of the two strings; for a query image I _q and its binary code H _q , if H _q and H _i ∈ Γ If the Hamming distance between _H is less than the set threshold, then define a candidate pool P={I _c1 , I _c2 ,...,I _cm } containing m candidate pictures (candidates), and if the Hamming distance is less than 5, two pictures are considered the images are similar;

Step 3.2: Refined Search

Step 3.2.1: Salient Feature Extraction

The two-dimensional remote sensing image feature maps generated by the 13th and 15th convolutional layers of the neural network for the image I _q to be queried are respectively mapped to one-dimensional vectors for storage; in the subsequent retrieval process, the retrieval results using different feature vectors are compared to determine the final Select the feature map generated by which layer of convolution to extract the salient features of remote sensing images;

Step 3.2.2: Euclidean distance measure similarity

For a query image I _q and a candidate pool P, use the extracted salient feature vectors to select the top k images from the candidate pool P; V _q and denote the feature vectors of the query image q and I _ci respectively; define the Euclidean distance s _i between I _q and the corresponding feature vector of the i-th image in the candidate pool P as the similarity level between them, as shown in formula (4) ;

The smaller the Euclidean distance, the greater the similarity between the two images; each candidate image I _ci is sorted in ascending order according to the similarity with the query image, and the top k images are the retrieval results;

Step 3.3: Evaluation of search results

Use ranking-based evaluation criteria to evaluate the retrieval results; for a query image q and the obtained top k retrieval result images, the precision rate Precision is calculated according to the following formula:

Among them, Precision@k means to set the threshold k, and until the kth correct result is retrieved, the average accuracy rate from the first correct result to the kth correct result; Rel(i) means the query image q and the ranking i Rel(i)∈{0,1}, 1 means that the query image q and the i-th image have the same classification, that is, they are related, and 0 means they are not related.

Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

First of all, compared with the traditional method of manually extracting remote sensing image features, the present invention uses a fully convolutional neural network to build a deep saliency target detection model, selects domestic and foreign remote sensing image databases for training networks, comprehensively analyzes the three-layer semantic information of images, and automatically learns remote sensing Distinctive features of the image. At the same time, the innovative semantic segmentation is added to the fully convolutional neural network to learn the depth saliency of remote sensing images, effectively improving the learned saliency features. Experiments have confirmed that this model can extract salient objects with clearer edges on multi-object detection data sets with more complex scenes, such as the Microsoft COCO data set. The learning ability of deep neural network can be further transferred to the salient feature learning of remote sensing images. Secondly, the present invention introduces a hash layer into the fully convolutional neural network architecture to generate binary hash codes while learning the depth salient features of remote sensing images, which can save storage space and improve subsequent retrieval efficiency. Finally, a coarse-to-fine retrieval strategy is adopted in image retrieval, and binary hash codes and salient features are used to measure the similarity. Experiments have confirmed that adding a hash layer to the AlexNet neural network, and adopting a multi-level retrieval strategy from coarse to fine, in the retrieval of 2.5 million ordinary images of different categories, the accuracy rate of the top K similar images returned by statistics, namely The topK precision rate, when K is 1000, the topK precision rate can reach 88% on average, and the retrieval time is about 1s. Therefore, transferring this method to the retrieval of remote sensing images is feasible and has important application value for realizing accurate and efficient retrieval of remote sensing images.

Description of drawings

Fig. 1 Flowchart of fast remote sensing image retrieval method based on depth saliency;

Figure 2 Architecture diagram of target detection model based on depth saliency;

Fig. 3 adds the neural network architecture diagram of the hash layer;

Figure 4 is a diagram of the multi-level retrieval process.

detailed description

According to the above description, the following is a specific implementation process, but the protection scope of this patent is not limited to this implementation process.

Step 1: Construction of object detection model based on deep saliency

The salient area is subjectively understood as the area where the human eye focuses attention, and is closely related to the Human Visual System (HVS). Objectively speaking, it refers to a certain feature of the image, and there is a sub-area with the most obvious feature. Therefore, the crux of the saliency detection problem lies in feature learning and extraction. In view of the powerful function of deep learning in this aspect, the present invention uses the fully convolutional neural network for the salient detection problem, and proposes a multi-task salient target detection model based on the fully convolutional neural network. The model performs two tasks simultaneously: a saliency detection task and a semantic segmentation task. The saliency detection task is used to learn the depth features of remote sensing images and calculate the depth saliency. The semantic segmentation task is used to extract the semantic information of the internal objects in the image, eliminate the background confusion of the saliency map, and supplement the missing parts of the saliency target.

The fully convolutional neural network architecture proposed by the present invention is implemented based on the mainstream open source deep learning framework Caffe, and the specific model structure is shown in Figure 2. Input an RGB image, and perform a series of convolution operations through 15 convolutional layers (Conv). The saliency detection task and the superpixel target semantic segmentation task share the convolutional layer. The first 13 convolutional layers are initialized by the convolutional neural network VGGNet, and the convolution kernel size is 3×3. After each convolutional layer, the Rectified Linear Unit (ReLU) is used as the activation function to speed up the convergence speed. After the 2nd, 4th, 5th, and 13th convolutional layers, the maximum pooling (MaxPooling) operation is performed to reduce the feature dimension and reduce the amount of calculation while ensuring the invariance of the features. The convolution kernel sizes of the 14th and 15th convolutional layers are 7×7 and 1×1 respectively, and each convolutional layer is connected to the Dropout layer to solve the potential over-fitting phenomenon of the complex neural network structure, that is, the model over-learning training data The noise and details of the model lead to high error rate and poor generalization ability in the actual test. The deconvolution layer is constructed by upsampling, its parameters are initialized by bilinear interpolation, and iteratively updated in the training and learning upsampling function. In the salient target detection task, the output image is normalized to [0,1] through the sigmoid threshold function to learn salient features. In the semantic segmentation task, the deconvolution layer is used to upsample the feature map of the last convolutional layer, and the upsampling result is cropped (Crop), so that the output image is the same size as the input image, so that each pixel is A prediction is produced while preserving the spatial information in the original input image.

Step 2: Neural network pre-training and adding hash layer fine-tuning

The present invention uses the publicly available large-scale aerial image data set (AID) for pre-training of neural networks, aiming at better learning semantic features of different levels of remote sensing images. Introducing the hash layer and using the expanded Wuhan University Remote Sensing Image Dataset (WHU-RS) to further fine-tune the network can not only map the high-dimensional features learned by the neural network to low-dimensional, shorten the retrieval time, but also make the neural network learned features are more robust.

Step 2.1: Multi-task salient object detection model pre-training

Step 2.1.1: Build a pre-training dataset

In the pre-training stage, the public large-scale aerial image dataset (AID) is selected as the standard data set for pre-training. AID contains 30 categories and 10,000 aerial images, all of which are selected from Google Earth and annotated by professional remote sensing technical personnel. The images of each category are taken from different countries and regions, and taken by different remote sensing detectors at different times. The image size is 600×600 pixels, and the resolution ranges from 0.5m/pixel to 8m/pixel. Compared with other datasets, the intra-class gap of this dataset is small, and the inter-class gap is relatively large. It is currently the largest dataset in aerial image datasets.

Step 2.1.2: Salient object detection model pre-training

FCNN pre-training is carried out through the saliency detection task and the segmentation task together. χ represents _a collection of N1 training images whose width and height are respectively W and Q, Xi is the i-th image, and Y _ijk represents the pixel-level real segmentation map corresponding to the i-th image whose width and height are j and k respectively, where i=1...N ₁ , j=1...W, k=1...Q. Z represents a set of N ₂ training images, and Z _n is the nth image, n=1...N ₂ , which has a corresponding real binary image M _n with salient objects. θ _s is the parameter of the shared convolutional layer, θ _h is the parameter of the segmentation task, and θ _f is the parameter of the saliency task. Formula (1) and formula (2) are respectively the cross-entropy cost function J ₁ (χ; θ _s , θ _h ) of the segmentation task and the square Euclidean distance cost function J ₂ (Z; θ _s , θ _f ), the FCNN is trained by minimizing two cost functions:

In formula (1), is the indicator function, h _cjk is the element (j,k) of the c-th confidence segmentation map, c=1...C, h(Xi; θ _s , θ _h ) is the semantic segmentation function, and returns the confidence of C target classes Segmentation map, C is the image category contained in the pre-training data set, and C is 30 in the present invention; in formula (2), f(Z _n ; θ _s , θ _f ) is a saliency map output function, and F represents the F-norm operation.

Next, the above cost function is minimized using stochastic gradient descent (SGD) with regularization over all training samples. Since the dataset used for pre-training does not have both segmentation and saliency annotations, the segmentation task and saliency detection task are alternated. Since the training process needs to normalize all original image sizes, the present invention resets the original image size to 500×500 pixels for pre-training. The learning rate is a necessary parameter of the SGD learning method, which determines the speed of weight update. If it is set too large, the cost function will oscillate, and the result will exceed the optimal value. If it is too small, the convergence speed will be too slow. Generally, a smaller learning rate is preferred. rate, such as 0.001±0.01 to keep the system stable. The momentum parameter and the weight decay factor can improve the training adaptability, the momentum parameter is usually [0.9,1.0], and the weight decay factor is usually 0.0005±0.0002. Through experimental observation, the present invention sets the learning rate to 10 ^-10 , the momentum parameter to 0.99, and the weight decay factor to the default value of 0.0005 in the Caffe framework. The stochastic gradient descent (SGD) learning process was accelerated by an NVIDIA GTX 1080 GPU device for a total of 80,000 iterations. The detailed pre-training process is as follows:

1) Shared full convolution parameters Initialization based on VGGNet;

2) Randomly initialize the segmentation task parameters through normal distribution and saliency task parameters

3) According to with Using SGD to train the segmentation network, update these two parameters as with

4) According to with Use SGD to train the saliency network, and update the relevant parameters as with

5) According to with Using SGD to train the segmentation network, obtain with

6) According to with Use SGD to train the saliency network, and update the relevant parameters as with

7) Repeat the above steps 3-6 three times to obtain the final pre-training parameters θ _s , θ _h , θ _f .

Step 2.2: Add a hash layer and fine-tune the network for the target domain

Step 2.2.1: Construct a Chinese remote sensing imagery dataset for fine-tuning the network

The expanded Wuhan University Remote Sensing Image Dataset (WHU-RS) was selected for neural network fine-tuning. The original WHU-RS dataset contains 19 scene classifications, a total of 950 remote sensing images with different resolutions, and the image size is 600×600 pixels. All images are taken from Google Earth. Combined with China's topography and geomorphology, the original data set is reconstructed and expanded to 7000 remote sensing images as a sample library, each category contains more than 200 images. The illumination, shooting angle, resolution, and size of the newly added sample images are all different, which is conducive to the learning of more robust salient features by the neural network.

Step 2.2.2: Join the hash layer to fine-tune the network

The feature vectors generated by deep neural networks have high dimensionality, which is very time-consuming in large-scale image retrieval. Because the binary hash codes of similar images are similar, the present invention inserts a fully connected layer containing s neurons between the penultimate layer of the pre-trained network and the final task layer, that is, the hash layer H, Map high-dimensional features to low-dimensional space, and generate binary hash codes for storage. The network structure is shown in Figure 3. The H weight of the hash layer is initialized with the hash value constructed by random projection, the neuron activation function uses the sigmoid function to make the output value between 0 and 1, the threshold is set to 0.5 according to experience, and the number of neurons is the code length of the target binary code . The hash layer not only provides the feature abstraction of the previous layer, but also serves as a bridge connecting intermediate and high-level image semantic features.

The fine-tuning process adjusts the network weights through the Back Propagation algorithm. Network fine-tuning can be performed for the entire network or a portion of the network. Since the features learned by the low-level network structure are more general, and in order to avoid overfitting, the present invention uses the expanded WHU-RS data set to focus on adjusting the high-level network, that is, the network weight after the tenth convolutional layer. Usually, the data size of the data set used for fine-tuning the network will be reduced by 10%-50% compared with the pre-training data set. In the present invention, the fine-tuning network data set contains 7000 images, which is significantly smaller than the pre-training data set containing 10000 images. For the data set, compared with the pre-trained network parameters, the network parameters in the fine-tuning process should be appropriately reduced, and the number of iterations and learning rate can be reduced by 1%-10%. In the present invention, the number of iterations is reduced to 8000 during the fine-tuning process, the learning rate is reduced by 1% to 10 ^-12 , and the momentum parameter and weight decay factor remain unchanged, ie respectively set to 0.99 and 0.0005.

The detailed fine-tuning process is as follows:

1) Shared full convolution parameters Split Task Parameters and saliency task parameters Obtained through the pre-training process;

2) According to with Using SGD to train the segmentation network, update these two parameters as with

3) According to with Use SGD to train the saliency network, and update the relevant parameters as with

4) According to with Using SGD to train the segmentation network, obtain with

5) According to with Use SGD to train the saliency network, and update the relevant parameters as with

6) Repeat steps 3-6 above three times to obtain the final parameters θ _s , θ _h , θ _f .

Step 3: Multi-Level Depth Retrieval

The shallow part of the deep convolutional neural network learns the underlying visual features, while the deep part captures image semantic information. Therefore, the present invention adopts a coarse-to-fine retrieval strategy to realize fast and accurate image retrieval. The process of feature extraction and retrieval is shown in Figure 4.

Step 3.1: Coarse retrieval

First, a series of proposals with similar high-level semantic features are retrieved, i.e., have similar binary activation values at the hashing layer, and then a similar image ranking is further generated according to the similarity measure.

Step 3.1.1: Generate Binary Hash Code

Input an image I _q to be queried into the fine-tuned neural network, and extract the output of the hash layer as the image signature, denoted by Out(H). The binary code is obtained by binarizing the activation value according to the threshold. For each binary bit r=1...s, output binary code according to formula (3):

Among them, s is the number of neurons in the hash layer. If the number is too large, overfitting will occur. The recommended initial value setting range is [40,100]. The specific value is adjusted according to the actual training data. In the present invention, s is set to 48. Γ={I ₁ , I ₂ ,...,I _n } represents a data set for retrieval including n images. The corresponding binary code of each image is expressed as Γ _H ={H ₁ ,H ₂ ,…,H _n }, where i=1…n, H _i ∈ {0,1} ^s represents s generated by s neurons Bit binary code value is 0 or 1 respectively.

Step 3.1.2: Hamming distance measures similarity

The Hamming distance between two strings of equal length is the number of different characters in the corresponding positions of the two strings. For an image to be queried I _q and its binary code H _q , if the Hamming distance between H _q and H _i ∈ Γ _H is smaller than the set threshold, define a candidate that contains m candidate pictures (candidates) Pool P={I _c1 , I _c2 ,...,I _cm }, in general, two images can be considered similar if the Hamming distance is less than 5.

Step 3.2: Refined Search

Step 3.2.1: Salient Feature Extraction

Since different convolutional layers of deep convolutional networks learn different levels of semantic features of different images, among them, the features learned by middle and high-level convolutional layers are more suitable for image retrieval tasks. Therefore, the two-dimensional remote sensing image feature maps generated by the image I _q to be queried through the 13th and 15th convolutional layers of the neural network are respectively mapped into one-dimensional vectors for storage. In the subsequent retrieval process, the retrieval results using different feature vectors were compared to determine which layer of convolution generated feature maps was finally selected to extract the salient features of remote sensing images.

Step 3.2.2: Euclidean distance measure similarity

For a query image I _q and a candidate pool P, use the extracted salient feature vectors to pick out the top k images from the candidate pool P. V _q and denote the feature vectors of the query image q and I _ci , respectively. Define the Euclidean distance s _i between I _q and the corresponding feature vector of the i-th image in the candidate pool P as the similarity level between them, as shown in formula (4).

The smaller the Euclidean distance, the greater the similarity between two images. Each candidate image I _ci is sorted in ascending order according to the similarity with the query image, and the top k images are the retrieval results.

Step 3.3: Evaluation of search results

The present invention uses ranking-based evaluation criteria to evaluate retrieval results. For a query image q and the obtained top k retrieval result images, the precision rate (Precision) is calculated according to the following formula:

Among them, Precision@k indicates that the threshold k is set according to actual needs, and the average accuracy rate from the first correct result to the kth correct result is retrieved until the kth correct result is retrieved; Rel(i) indicates the query image q and The correlation of the i-th image, Rel(i)∈{0,1}, 1 means that the query image q and the i-th image have the same classification, that is, they are related, and 0 means they are not related.

Claims (1)

1. A remote sensing image fast retrieval method based on depth saliency, characterized in that it comprises the following steps: Step 1: Construction of object detection model based on deep saliency Input an RGB image, perform a series of convolution operations through 15 convolutional layers, and then perform the saliency detection task and the superpixel target semantic segmentation task to share the convolutional layer; the first 13 convolutional layers pass through the initial convolutional neural network VGGNet The size of the convolution kernel is 3×3, and the modified linear unit ReLU is used as the activation function after each convolution layer; the maximum pooling operation is performed after the 2nd, 4th, 5th, and 13th convolutional layers; volumes 14 and 15 The convolution kernel sizes of the product layers are 7×7 and 1×1 respectively, and the Dropout layer is connected after the 14th and 15th convolutional layers; Construct the deconvolution layer by upsampling, initialize its parameters by bilinear interpolation, and update iteratively in the training and learning upsampling function; in the salient target detection task, the output image is normalized to [0,1] through the sigmoid threshold function, Learn salient features; in the semantic segmentation task, use the deconvolution layer to upsample the feature map of the last convolutional layer, and clip the upsampling result so that the output image is the same size as the input image; Step 2: Neural network pre-training and adding hash layer fine-tuning Step 2.1: Multi-task salient object detection model pre-training The FCNN pre-training is carried out through the saliency detection task and the segmentation task together; χ represents the set of N ₁ training images whose width and height are W and Q respectively, Xi is the i-th image, and Y _ijk represents the i-th width and height respectively The pixel-level real segmentation map corresponding to the images of j and k, where i=1...N ₁ , j=1...W, k=1...Q; Z represents the set of N ₂ training images, and Z _n is the nth one Image, n=1...N ₂ , it has a corresponding real binary image M _n with salient objects; θ _s is the shared convolutional layer parameter, θ _h is the segmentation task parameter, θ _f is the saliency task parameter; the formula (1) and formula (2) respectively represent the cross-entropy cost function J ₁ (χ; θ _s , θ _h ) of the segmentation task and the square Euclidean distance cost function J ₂ (Z; θ _s , θ _f ) of the saliency detection task , the FCNN is trained by minimizing two cost functions: ${\mathrm{<span\; class="notranslate">\; J</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$ In formula (1), is the indicator function, h _cjk is the element (j,k) of the c-th confidence segmentation map, c=1...C, h(Xi; θ _s , θ _h ) is the semantic segmentation function, and returns the confidence of C target classes Segmentation map, C is the image category contained in the pre-training data set In the formula (2), f(Z _n ; θ _s , θ _f ) is the salient map output function, and F represents the F-norm operation; Next, use the stochastic gradient descent SGD method to minimize the above cost function on the basis of regularizing all training samples; since the data set used for pre-training does not have both segmentation and saliency labels, the segmentation task and saliency The task of sex detection is alternately performed; the training process needs to normalize the size of all original images; the learning rate is 0.001±0.01; the momentum parameter is usually [0.9,1.0], and the weight decay factor is usually 0.0005±0.0002; stochastic gradient descent learning The process performs more than 80,000 iterations in total; the detailed pre-training process is as follows: 1) Shared full convolution parameters Initialization based on VGGNet; 2) Randomly initialize the segmentation task parameters through normal distribution and saliency task parameters 3) According to with Using SGD to train the segmentation network, update these two parameters as with 4) According to with Use SGD to train the saliency network, and update the relevant parameters as with 5) According to with Using SGD to train the segmentation network, obtain with 6) According to with Use SGD to train the saliency network, and update the relevant parameters as with 7) Repeat the above steps 3-6 three times to obtain the pre-training final parameters θ _s , θ _h , θ _f ; Step 2.2: Add a hash layer and fine-tune the network for the target domain Between the penultimate layer of the pre-trained network and the final task layer, insert a fully connected layer containing s neurons, that is, the hash layer H, which maps high-dimensional features to low-dimensional space and generates binary hash codes Store; the H weight of the hash layer is initialized with the hash value constructed by random projection, the neuron activation function uses the sigmoid function to make the output value between 0 and 1, and the number of neurons is the code length of the target binary code; The fine-tuning process adjusts the network weight through the back propagation algorithm; the network fine-tuning is to adjust the network weight after the tenth convolutional layer; the data size of the data set used for fine-tuning the network will be reduced by 10% compared with the data set of the pre-trained network- 50%, compared with the pre-trained network parameters, the number of iterations and learning rate of the network parameters in the fine-tuning process is reduced by 1%-10%, and the momentum parameters and weight decay factors remain unchanged; The detailed fine-tuning process is as follows: 1) Shared full convolution parameters Split Task Parameters and saliency task parameters Obtained through the pre-training process; 2) According to with Using SGD to train the segmentation network, update these two parameters as with 3) According to with Use SGD to train the saliency network, and update the relevant parameters as with 4) According to with Using SGD to train the segmentation network, obtain with 5) According to with Use SGD to train the saliency network, and update the relevant parameters as with 6) Repeat the above steps 3-6 three times to obtain the final parameters θ _s , θ _h , θ _f ; Step 3: Multi-Level Depth Retrieval Step 3.1: Coarse retrieval Step 3.1.1: Generate Binary Hash Code Input an image I _q to be queried into the fine-tuned neural network, extract the output of the hash layer as the image signature, denoted by Out(H); the binary code is obtained by binarizing the activation value according to the threshold; for each binary bit r =1...s, output binary code according to formula (3): ${\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& {\mathrm{<span\; class="notranslate">\; out</span>}}^{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; 0.5</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; w</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; e</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$ Among them, s is the number of neurons in the hash layer, and the initial value setting range is [40,100]; Γ={I ₁ ,I ₂ ,…,I _n } means the data set for retrieval containing n images; the corresponding The binary code of each image is expressed as Γ _H ={H ₁ ,H ₂ ,…,H _n }, where i=1…n, H _i ∈{0,1} ^s represents the s-bit binary code generated by s neurons The code value is 0 or 1 respectively; Step 3.1.2: Hamming distance measures similarity The Hamming distance between two strings of equal length is the number of different characters in the corresponding positions of the two strings; for a query image I _q and its binary code H _q , if H _q and H _i ∈ Γ If the Hamming distance between _H is less than the set threshold, then define a candidate pool P={I _c1 , I _c2 ,...,I _cm } containing m candidate pictures (candidates), and if the Hamming distance is less than 5, two pictures are considered the images are similar; Step 3.2: Refined Search Step 3.2.1: Salient Feature Extraction The two-dimensional remote sensing image feature maps generated by the 13th and 15th convolutional layers of the neural network for the image I _q to be queried are respectively mapped to one-dimensional vectors for storage; in the subsequent retrieval process, the retrieval results using different feature vectors are compared to determine the final Select the feature map generated by which layer of convolution to extract the salient features of remote sensing images; Step 3.2.2: Euclidean distance measure similarity For a query image I _q and a candidate pool P, use the extracted salient feature vectors to select the top k images from the candidate pool P; V _q and denote the feature vectors of the query image q and I _ci respectively; define the Euclidean distance s _i between I _q and the corresponding feature vector of the i-th image in the candidate pool P as the similarity level between them, as shown in formula (4) ; ${\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; P</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$ The smaller the Euclidean distance, the greater the similarity between the two images; each candidate image I _ci is sorted in ascending order according to the similarity with the query image, and the top k images are the retrieval results; Step 3.3: Evaluation of search results Use ranking-based evaluation criteria to evaluate the retrieval results; for a query image q and the obtained top k retrieval result images, the precision rate Precision is calculated according to the following formula: $\mathrm{<span\; class="notranslate">\; PR</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; c</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; @</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\frac{{<span\; class="notranslate">\; \&Sigma;</span>}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}\mathrm{<span\; class="notranslate">\; Re</span>}\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$ Among them, Precision@k means to set the threshold k, and until the kth correct result is retrieved, the average accuracy rate from the first correct result to the kth correct result; Rel(i) means the query image q and the ranking i Rel(i)∈{0,1}, 1 means that the query image q and the i-th image have the same classification, that is, they are related, and 0 means they are not related.