Disclosure of Invention
The present application has been made to solve the above-mentioned technical problems. The embodiment of the application provides a geographic information remote sensing mapping system and a method thereof, which acquire a remote sensing multispectral image to be processed; and excavating spectral features of the remote sensing spectral image on the ground features by adopting an artificial intelligence technology based on deep learning, and extracting ground feature implicit feature information in the remote sensing spectral image based on the spectral features of the ground features so as to divide all pixels of the image into a plurality of categories according to properties. Therefore, the identification and classification of the ground object can be accurately carried out, so that the remote sensing spectrum image can be accurately divided into a plurality of ground object category subregions with different category pixel properties.
In a first aspect, there is provided a geographic information remote sensing mapping system comprising:
the multispectral image acquisition module is used for acquiring a remote sensing multispectral image to be processed;
the direction gradient histogram conversion module is used for calculating a direction gradient histogram of the remote sensing multispectral image to be processed;
The multichannel aggregation module is used for aggregating the remote sensing multispectral image to be processed and the direction gradient histogram along the channel dimension to obtain a multichannel image;
the image blocking module is used for carrying out image blocking processing on the multichannel image to obtain a sequence of image blocks;
The multi-scale feature extraction module is used for respectively passing the sequence of the image block through a double-branch structure comprising a first convolution neural network model and a second convolution neural network model to obtain a sequence of an image block feature matrix, wherein the first convolution neural network model and the second convolution neural network model use two-dimensional convolution kernels with different scales;
The global feature association module is used for arranging the sequences of the image block feature matrixes into a multi-channel global feature matrix; and
The semantic segmentation module is used for carrying out image semantic segmentation on the multi-channel global feature matrix to obtain an image semantic segmentation result.
In the above geographic information remote sensing mapping system, the multi-scale feature extraction module includes: the first scale image feature extraction unit is used for performing depth convolution coding on each image block in the sequence of the image blocks by using a first convolution neural network model of the double-branch structure so as to obtain a first branch feature matrix; the second scale image feature extraction unit is used for performing depth convolution coding on each image block in the sequence of the image blocks by using a second convolution neural network model of the double-branch structure so as to obtain a second branch feature matrix; and a multi-scale feature fusion unit, configured to fuse the first branch feature matrix and the second branch feature matrix to obtain the image block feature matrix.
In the above geographic information remote sensing mapping system, the first scale image feature extraction unit is configured to: and respectively carrying out convolution processing based on a two-dimensional convolution kernel, pooling processing along a channel dimension and nonlinear activation processing on input data in forward transfer of layers by using each layer of the first convolution neural network model of the double-branch structure to output the first branch feature matrix by the last layer of the first convolution neural network model of the double-branch structure, wherein the input of the first layer of the first convolution neural network model of the double-branch structure is each image block in the sequence of the image blocks.
In the above geographic information remote sensing mapping system, the second scale image feature extraction unit is configured to: and respectively carrying out convolution processing based on a two-dimensional convolution kernel, pooling processing along a channel dimension and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model of the double-branch structure to output the second branch characteristic matrix by the last layer of the second convolution neural network model of the double-branch structure, wherein the input of the first layer of the second convolution neural network model of the double-branch structure is each image block in the sequence of the image blocks.
In the above geographic information remote sensing mapping system, the multi-scale feature fusion unit is configured to: adopting convolution dictionary contrast response learning to fuse the first branch feature matrix and the second branch feature matrix by the following optimization formula so as to obtain the image block feature matrix; wherein, the optimization formula is:
Wherein M 1 and M 2 are the first and second branch feature matrices, respectively, and II F represents the Frobenius norm of the matrix, Representing the addition of the matrix,Representing matrix multiplication, M c represents the image block feature matrix, and (-) T represents the transpose of the matrix.
In the above geographic information remote sensing mapping system, the global feature association module is configured to: arranging the sequence of the image block feature matrix into a multi-channel global feature matrix according to the following arrangement formula; wherein, the arrangement formula is:
Mc=Concat[M1,M2…Mn]
Wherein M 1,M2…Mn represents the sequence of the image block feature matrix, concat [. Cndot. ] represents a cascading function, and M c represents the multi-channel global feature matrix.
In the geographic information remote sensing mapping system, the image semantic segmentation result is that the to-be-processed remote sensing multispectral image is segmented into a plurality of ground object category subregions with different category pixel properties.
In a second aspect, a method for remote sensing and mapping geographic information is provided, which includes:
acquiring a remote sensing multispectral image to be processed;
calculating a directional gradient histogram of the remote sensing multispectral image to be processed;
The remote sensing multispectral image to be processed and the direction gradient histogram are aggregated along the channel dimension to obtain a multichannel image;
performing image blocking processing on the multichannel image to obtain a sequence of image blocks;
The sequence of the image block is respectively passed through a double-branch structure comprising a first convolution neural network model and a second convolution neural network model to obtain a sequence of an image block characteristic matrix, wherein the first convolution neural network model and the second convolution neural network model use two-dimensional convolution kernels with different scales;
arranging the sequence of the image block feature matrix into a multi-channel global feature matrix; and
Image semantic segmentation is carried out on the multi-channel global feature matrix to obtain an image semantic segmentation result
In the above method for remote sensing mapping of geographic information, the steps of passing the sequence of image blocks through a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model to obtain a sequence of image block feature matrices, wherein the first convolutional neural network model and the second convolutional neural network model use two-dimensional convolution kernels with different scales, and include: performing depth convolution coding on each image block in the sequence of image blocks by using a first convolution neural network model of the double-branch structure to obtain a first branch feature matrix; performing depth convolution coding on each image block in the sequence of image blocks by using a second convolution neural network model of the double-branch structure to obtain a second branch feature matrix; and fusing the first branch feature matrix and the second branch feature matrix to obtain the image block feature matrix.
In the above-mentioned geographic information remote sensing mapping method, performing depth convolution encoding on each image block in the sequence of image blocks by using the first convolution neural network model of the dual-branch structure to obtain a first branch feature matrix, including: and respectively carrying out convolution processing based on a two-dimensional convolution kernel, pooling processing along a channel dimension and nonlinear activation processing on input data in forward transfer of layers by using each layer of the first convolution neural network model of the double-branch structure to output the first branch feature matrix by the last layer of the first convolution neural network model of the double-branch structure, wherein the input of the first layer of the first convolution neural network model of the double-branch structure is each image block in the sequence of the image blocks.
Compared with the prior art, the geographic information remote sensing mapping system and the method thereof provided by the application acquire the remote sensing multispectral image to be processed; and excavating spectral features of the remote sensing spectral image on the ground features by adopting an artificial intelligence technology based on deep learning, and extracting ground feature implicit feature information in the remote sensing spectral image based on the spectral features of the ground features so as to divide all pixels of the image into a plurality of categories according to properties. Therefore, the identification and classification of the ground object can be accurately carried out, so that the remote sensing spectrum image can be accurately divided into a plurality of ground object category subregions with different category pixel properties.
Detailed Description
The following description of the technical solutions according to the embodiments of the present application will be given with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Unless defined otherwise, all technical and scientific terms used in the embodiments of the application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to limit the scope of the present application.
In describing embodiments of the present application, unless otherwise indicated and limited thereto, the term "connected" should be construed broadly, for example, it may be an electrical connection, or may be a communication between two elements, or may be a direct connection, or may be an indirect connection via an intermediate medium, and it will be understood by those skilled in the art that the specific meaning of the term may be interpreted according to circumstances.
It should be noted that, the term "first\second\third" related to the embodiment of the present application is merely to distinguish similar objects, and does not represent a specific order for the objects, it is to be understood that "first\second\third" may interchange a specific order or sequence where allowed. It is to be understood that the "first\second\third" distinguishing objects may be interchanged where appropriate such that embodiments of the application described herein may be practiced in sequences other than those illustrated or described herein.
As described above, at present, when classifying remote sensing images, there are several main problems in supervision classification due to the existence of problems such as homozygotic spectrum and foreign matter homozygotic spectrum: the mixing and separating problem of water body and mountain shadow; the classification boundary between the unused area and the urban public transportation area is not clear; the river can be misjudged as urban public transport land; lin De and arable land confusion; mountain shadows can be misjudged as urban public transportation land. Accordingly, an optimized geographic information remote sensing mapping system is desired.
Accordingly, considering that the characteristics of the target ground object in the remote sensing image are reflected on the remote sensing image when the remote sensing image is classified by actually performing the remote sensing mapping of the geographic information, in order to identify the attributes such as the type, the property, the spatial position, the shape, the size and the like of the ground object based on the ground object characteristics on the remote sensing image, the different implicit characteristic information of each ground object in the remote sensing image of the geographic information needs to be accurately analyzed and identified, so that an accurate classification result of the remote sensing image is obtained. However, because the remote sensing image has more information, the traditional method can not accurately and effectively capture and identify the features of each ground object in the remote sensing image, so that the classification accuracy is reduced. Therefore, in the technical scheme of the application, based on the spectrum characteristics of the ground object, the implicit characteristic information of the ground object in the remote sensing spectrum image is extracted, and all pixels of the image are classified into a plurality of categories according to the properties. In the process, the difficulty is how to fully and accurately express the implicit characteristic distribution information about the ground object in the remote sensing spectrum image, so as to accurately identify and classify the ground object, and accurately divide the remote sensing spectrum image into a plurality of ground object category subregions with different category pixel properties.
In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. In addition, deep learning and neural networks have also shown levels approaching and even exceeding humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like.
The development of deep learning and neural networks provides a new solution idea and scheme for mining implicit characteristic distribution information about ground features in remote sensing spectrum images.
Specifically, in the technical scheme of the application, firstly, a remote sensing multispectral image to be processed is acquired. Then, considering that the implicit characteristic distribution information about various ground objects in the to-be-processed remote sensing multispectral image is presented at the image brightness, color and texture ends in the to-be-processed remote sensing multispectral image, in order to accurately express the characteristics of various ground objects in the to-be-processed remote sensing multispectral image, in the technical scheme of the application, the direction gradient histogram of the to-be-processed remote sensing multispectral image is calculated, and then the direction gradient histogram and the multichannel image obtained by aggregating the to-be-processed remote sensing multispectral image along the channel dimension are taken as input data, so that the characteristic expression capability of the various ground objects is further improved.
It should be understood that the directional gradient histogram is a method for describing local brightness, color and texture characteristics of an image, the algorithm image divides the image into small-sized cell spaces, calculates gradients of pixels in the cells, generates cells (HOG, histogram of Oriented Gradient) according to gradient distribution, and then counts the HOG distribution of the cells in a larger-sized block space to generate a block space HOG, describing local texture information. Therefore, the multi-channel image data obtained by aggregation of the direction gradient histogram and the remote sensing multi-spectrum image to be processed is used as input data, so that the feature extractor can more effectively and easily extract the related features of the brightness, color and texture information about various ground features in the image, and the category features about the ground features are often reflected on the brightness, color and texture information level of the image.
Further, in consideration of the fact that in the classification process of actually performing the category detection of the ground feature, due to the fact that the implicit features of the image about the ground feature are small-scale feature information relative to the multi-channel image, capturing and extracting of the effective features of the ground feature in the image is difficult, and therefore the expression capability of the multi-channel image about the feature of the ground feature is reduced. Therefore, in the technical scheme of the application, the multi-channel image is further subjected to image blocking processing to obtain a sequence of image blocks. Accordingly, in a specific example of the present application, the multi-channel image may be subjected to uniform image blocking processing, so as to subsequently improve the implicit feature expression capability of the various features, so as to obtain the sequence of image blocks, where each image block in the sequence of image blocks has the same size. It should be understood that after the image blocking processing, the scale of each image block in the sequence of image blocks is reduced compared with the original image, so that the distribution information of the hidden features of the features with small scale in the multi-channel image is not a small-size object in the image block any more, so as to improve the precise expression of the hidden features of the features of various types, and further improve the classification precision of the features of various types.
Then, feature mining of the sequence of image blocks is performed using a convolutional neural network model having excellent performance in terms of implicit feature extraction of images, particularly considering that, when the various types of features are identified and classified, since implicit feature information about the various types of features in the respective image blocks of the sequence of image blocks has not only large-scale features but also small-scale features. Therefore, in order to improve the expression capability of the hidden features of the various ground objects, so as to accurately classify the various ground objects, in the technical scheme of the application, the hidden feature extraction of each image block is further performed by using a convolutional neural network model with different feature receptive fields. That is, specifically, in the technical solution of the present application, the sequence of the image block is respectively passed through a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model to obtain the sequence of the image block feature matrix. It should be noted that, here, the first convolutional neural network model and the second convolutional neural network model use two-dimensional convolutional kernels with different scales, so as to extract multi-scale implicit feature distribution information about the various features in each image block.
Then, considering that the multi-scale implicit characteristic distribution information about various ground objects in each image block has an association relationship, in the technical scheme of the application, the sequence of the image block characteristic matrix is further arranged into a multi-channel global characteristic matrix, so that the multi-scale implicit characteristic information about various ground objects in each image block is fused, namely, the ground object category characteristic information about the properties of different category pixels in the remote sensing multi-spectrum image to be processed is fused.
Further, after the feature information of the ground object category about the pixel properties of different categories in the remote sensing multispectral image to be processed is obtained, image semantic segmentation is carried out on the multichannel global feature matrix so as to obtain an image semantic segmentation result. Specifically, in the technical scheme of the application, the image semantic segmentation result is that the remote sensing multispectral image to be processed is segmented into a plurality of ground object category subregions with different category pixel properties, so as to accurately identify and classify the ground objects.
In particular, in the technical solution of the present application, when the sequence of image blocks is obtained by respectively using a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model to obtain the sequence of image block feature matrices, each image block feature matrix in the sequence of image block feature matrices is obtained by fusing a first branch feature matrix obtained by the first convolutional neural network model and a second branch feature matrix obtained by the second convolutional neural network model, and in order to improve the fusion effect of the first branch feature matrix and the second branch feature matrix, in the technical solution of the present application, a convolutional dictionary contrast response learning is preferably adopted to perform feature fusion, where the feature fusion is expressed as follows:
Wherein M 1 and M 2 are the first and second branch feature matrices, respectively, and iij F represents the Frobenius norm of the matrix.
That is, based on the neighborhood operator attribute of the convolution kernel representation of the convolutional neural network of the first branch feature matrix M 1 and the second branch feature matrix M 2, the feature expression of the intrinsic prior structure is performed on the n-level (n-hop) neighbors of the feature values of the first branch feature matrix M 1 and the second branch feature matrix M 2 through the spatial structured convolutional dictionary contrast learning based on the point-plus-feature flow between the corresponding features, and the prior knowledge of the low-rank fusion representation is used as the feature response reference of the high-dimensional feature distribution, so that the interpretable response among the features is learned, and therefore, the feature expression effect of the feature matrix M c of the image block obtained after the fusion is improved, and the accuracy of the image semantic segmentation result of the multi-channel global feature matrix obtained through the sequence reconstruction of the feature matrix of the image block is improved. Therefore, the identification and classification of the ground object can be accurately carried out, so that the remote sensing spectrum image can be accurately divided into a plurality of ground object type subregions with different type pixel properties.
Fig. 1 is an application scenario diagram of a geographic information remote sensing mapping system according to an embodiment of the present application. As shown in fig. 1, in the application scenario, first, a remote sensing multispectral image to be processed (e.g., C as illustrated in fig. 1) is acquired; the acquired remote sensing multispectral image to be processed is then input into a server (e.g., S as illustrated in fig. 1) deployed with a geographic information remote sensing mapping algorithm, wherein the server is capable of processing the remote sensing multispectral image to be processed based on the geographic information remote sensing mapping algorithm to perform image semantic segmentation on the multi-channel global feature matrix to obtain an image semantic segmentation result.
Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described in detail with reference to the accompanying drawings.
In one embodiment of the present application, FIG. 2 is a block diagram of a geographic information remote sensing mapping system in accordance with an embodiment of the present application. As shown in fig. 2, a geographic information remote sensing mapping system 100 according to an embodiment of the present application includes: the multispectral image acquisition module 110 is used for acquiring a remote sensing multispectral image to be processed; the directional gradient histogram conversion module 120 is configured to calculate a directional gradient histogram of the remote sensing multispectral image to be processed; a multi-channel aggregation module 130, configured to aggregate the remote sensing multi-spectral image to be processed and the direction gradient histogram along a channel dimension to obtain a multi-channel image; an image blocking module 140, configured to perform image blocking processing on the multi-channel image to obtain a sequence of image blocks; a multi-scale feature extraction module 150, configured to pass the sequence of image blocks through a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model, respectively, to obtain a sequence of image block feature matrices, where the first convolutional neural network model and the second convolutional neural network model use two-dimensional convolution kernels with different scales; a global feature association module 160, configured to arrange the sequence of image block feature matrices into a multi-channel global feature matrix; and a semantic segmentation module 170, configured to perform image semantic segmentation on the multi-channel global feature matrix to obtain an image semantic segmentation result.
Specifically, in the embodiment of the present application, the multispectral image acquisition module 110 is configured to acquire a remote sensing multispectral image to be processed. As described above, at present, when classifying remote sensing images, there are several main problems in supervision classification due to the existence of problems such as homozygotic spectrum and foreign matter homozygotic spectrum: the mixing and separating problem of water body and mountain shadow; the classification boundary between the unused area and the urban public transportation area is not clear; the river can be misjudged as urban public transport land; lin De and arable land confusion; mountain shadows can be misjudged as urban public transportation land. Accordingly, an optimized geographic information remote sensing mapping system is desired.
Accordingly, considering that the characteristics of the target ground object in the remote sensing image are reflected on the remote sensing image when the remote sensing image is classified by actually performing the remote sensing mapping of the geographic information, in order to identify the attributes such as the type, the property, the spatial position, the shape, the size and the like of the ground object based on the ground object characteristics on the remote sensing image, the different implicit characteristic information of each ground object in the remote sensing image of the geographic information needs to be accurately analyzed and identified, so that an accurate classification result of the remote sensing image is obtained. However, because the remote sensing image has more information, the traditional method can not accurately and effectively capture and identify the features of each ground object in the remote sensing image, so that the classification accuracy is reduced. Therefore, in the technical scheme of the application, based on the spectrum characteristics of the ground object, the implicit characteristic information of the ground object in the remote sensing spectrum image is extracted, and all pixels of the image are classified into a plurality of categories according to the properties. In the process, the difficulty is how to fully and accurately express the implicit characteristic distribution information about the ground object in the remote sensing spectrum image, so as to accurately identify and classify the ground object, and accurately divide the remote sensing spectrum image into a plurality of ground object category subregions with different category pixel properties.
In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. In addition, deep learning and neural networks have also shown levels approaching and even exceeding humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like.
The development of deep learning and neural networks provides a new solution idea and scheme for mining implicit characteristic distribution information about ground features in remote sensing spectrum images.
Specifically, in the technical scheme of the application, firstly, a remote sensing multispectral image to be processed is acquired.
Specifically, in the embodiment of the present application, the directional gradient histogram conversion module 120 is configured to calculate a directional gradient histogram of the remote sensing multispectral image to be processed. Then, considering that the implicit characteristic distribution information about various ground objects in the to-be-processed remote sensing multispectral image is presented at the image brightness, color and texture ends in the to-be-processed remote sensing multispectral image, in order to accurately express the characteristics of various ground objects in the to-be-processed remote sensing multispectral image, in the technical scheme of the application, the direction gradient histogram of the to-be-processed remote sensing multispectral image is calculated, and then the direction gradient histogram and the multichannel image obtained by aggregating the to-be-processed remote sensing multispectral image along the channel dimension are taken as input data, so that the characteristic expression capability of the various ground objects is further improved.
Specifically, in the embodiment of the present application, the multi-channel aggregation module 130 is configured to aggregate the remote sensing multi-spectral image to be processed and the direction gradient histogram along a channel dimension to obtain a multi-channel image. It should be understood that the directional gradient histogram is a method for describing local brightness, color and texture characteristics of an image, the algorithm image divides the image into small-sized cell spaces, calculates gradients of pixels in the cells, generates cells (HOG, histogram of Oriented Gradient) according to gradient distribution, and then counts the HOG distribution of the cells in a larger-sized block space to generate a block space HOG, describing local texture information. Therefore, the multi-channel image data obtained by aggregation of the direction gradient histogram and the remote sensing multi-spectrum image to be processed is used as input data, so that the feature extractor can more effectively and easily extract the related features of the brightness, color and texture information about various ground features in the image, and the category features about the ground features are often reflected on the brightness, color and texture information level of the image.
Specifically, in the embodiment of the present application, the image blocking module 140 is configured to perform image blocking processing on the multi-channel image to obtain a sequence of image blocks. Further, in consideration of the fact that in the classification process of actually performing the category detection of the ground feature, due to the fact that the implicit features of the image about the ground feature are small-scale feature information relative to the multi-channel image, capturing and extracting of the effective features of the ground feature in the image is difficult, and therefore the expression capability of the multi-channel image about the feature of the ground feature is reduced. Therefore, in the technical scheme of the application, the multi-channel image is further subjected to image blocking processing to obtain a sequence of image blocks.
Accordingly, in a specific example of the present application, the multi-channel image may be subjected to uniform image blocking processing, so as to subsequently improve the implicit feature expression capability of the various features, so as to obtain the sequence of image blocks, where each image block in the sequence of image blocks has the same size. It should be understood that after the image blocking processing, the scale of each image block in the sequence of image blocks is reduced compared with the original image, so that the distribution information of the hidden features of the features with small scale in the multi-channel image is not a small-size object in the image block any more, so as to improve the precise expression of the hidden features of the features of various types, and further improve the classification precision of the features of various types.
Specifically, in the embodiment of the present application, the multi-scale feature extraction module 150 is configured to pass the sequence of image blocks through a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model, so as to obtain a sequence of image block feature matrices, where the first convolutional neural network model and the second convolutional neural network model use two-dimensional convolution kernels with different scales. Then, feature mining of the sequence of image blocks is performed using a convolutional neural network model having excellent performance in terms of implicit feature extraction of images, particularly considering that, when the various types of features are identified and classified, since implicit feature information about the various types of features in the respective image blocks of the sequence of image blocks has not only large-scale features but also small-scale features.
Therefore, in order to improve the expression capability of the hidden features of the various ground objects, so as to accurately classify the various ground objects, in the technical scheme of the application, the hidden feature extraction of each image block is further performed by using a convolutional neural network model with different feature receptive fields. That is, specifically, in the technical solution of the present application, the sequence of the image block is respectively passed through a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model to obtain the sequence of the image block feature matrix. It should be noted that, here, the first convolutional neural network model and the second convolutional neural network model use two-dimensional convolutional kernels with different scales, so as to extract multi-scale implicit feature distribution information about the various features in each image block.
Fig. 3 is a block diagram of the multi-scale feature extraction module in the geographic information remote sensing mapping system according to an embodiment of the present application, as shown in fig. 3, the multi-scale feature extraction module 150 includes: a first scale image feature extraction unit 151, configured to perform depth convolution encoding on each image block in the sequence of image blocks by using a first convolution neural network model of the dual-branch structure to obtain a first branch feature matrix; a second scale image feature extraction unit 152, configured to use a second convolutional neural network model of the dual-branch structure to perform depth convolutional encoding on each image block in the sequence of image blocks to obtain a second branch feature matrix; and a multi-scale feature fusion unit 153, configured to fuse the first branch feature matrix and the second branch feature matrix to obtain the image block feature matrix.
It should be noted that the multi-scale neighborhood feature extraction module is essentially a deep neural network model based on deep learning, which is capable of fitting any function by a predetermined training strategy and has a higher feature extraction generalization capability compared to the conventional feature engineering.
The multi-scale neighborhood feature extraction module comprises a plurality of parallel one-dimensional convolution layers, wherein in the process of feature extraction by the multi-scale neighborhood feature extraction module, the plurality of parallel one-dimensional convolution layers respectively carry out one-dimensional convolution coding on input data by one-dimensional convolution check with different scales so as to capture local implicit features of a sequence.
Wherein, the first scale image feature extraction unit 151 is configured to: and respectively carrying out convolution processing based on a two-dimensional convolution kernel, pooling processing along a channel dimension and nonlinear activation processing on input data in forward transfer of layers by using each layer of the first convolution neural network model of the double-branch structure to output the first branch feature matrix by the last layer of the first convolution neural network model of the double-branch structure, wherein the input of the first layer of the first convolution neural network model of the double-branch structure is each image block in the sequence of the image blocks.
Further, the second scale image feature extraction unit 152 is configured to: and respectively carrying out convolution processing based on a two-dimensional convolution kernel, pooling processing along a channel dimension and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model of the double-branch structure to output the second branch characteristic matrix by the last layer of the second convolution neural network model of the double-branch structure, wherein the input of the first layer of the second convolution neural network model of the double-branch structure is each image block in the sequence of the image blocks.
In particular, in the technical solution of the present application, when the sequence of image blocks is obtained by respectively using a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model to obtain the sequence of image block feature matrices, each image block feature matrix in the sequence of image block feature matrices is obtained by fusing a first branch feature matrix obtained by the first convolutional neural network model and a second branch feature matrix obtained by the second convolutional neural network model, and in order to improve the fusion effect of the first branch feature matrix and the second branch feature matrix, in the technical solution of the present application, a convolutional dictionary contrast response learning is preferably adopted to perform feature fusion, where the feature fusion is expressed as follows: adopting convolution dictionary contrast response learning to fuse the first branch feature matrix and the second branch feature matrix by the following optimization formula so as to obtain the image block feature matrix; wherein, the optimization formula is:
Wherein M 1 and M 2 are the first and second branch feature matrices, respectively, and II F represents the Frobenius norm of the matrix, Representing the addition of the matrix,Representing matrix multiplication, M c represents the image block feature matrix, and (-) T represents the transpose of the matrix.
That is, based on the neighborhood operator attribute of the convolution kernel representation of the convolutional neural network of the first branch feature matrix M 1 and the second branch feature matrix M 2, the feature expression of the intrinsic prior structure is performed on the n-level (n-hop) neighbors of the feature values of the first branch feature matrix M 1 and the second branch feature matrix M 2 through the spatial structured convolutional dictionary contrast learning based on the point-plus-feature flow between the corresponding features, and the prior knowledge of the low-rank fusion representation is used as the feature response reference of the high-dimensional feature distribution, so that the interpretable response among the features is learned, and therefore, the feature expression effect of the feature matrix M c of the image block obtained after the fusion is improved, and the accuracy of the image semantic segmentation result of the multi-channel global feature matrix obtained through the sequence reconstruction of the feature matrix of the image block is improved. Therefore, the identification and classification of the ground object can be accurately carried out, so that the remote sensing spectrum image can be accurately divided into a plurality of ground object type subregions with different type pixel properties.
Specifically, in the embodiment of the present application, the global feature association module 160 is configured to arrange the sequence of the image block feature matrices into a multi-channel global feature matrix. Then, considering that the multi-scale implicit characteristic distribution information about various ground objects in each image block has an association relationship, in the technical scheme of the application, the sequence of the image block characteristic matrix is further arranged into a multi-channel global characteristic matrix, so that the multi-scale implicit characteristic information about various ground objects in each image block is fused, namely, the ground object category characteristic information about the properties of different category pixels in the remote sensing multi-spectrum image to be processed is fused.
Wherein, the global feature association module 160 is configured to: arranging the sequence of the image block feature matrix into a multi-channel global feature matrix according to the following arrangement formula; wherein, the arrangement formula is:
Mc=Concat[M1,M2…Mn]
Wherein M 1,M2…Mn represents the sequence of the image block feature matrix, concat [. Cndot. ] represents a cascading function, and M c represents the multi-channel global feature matrix.
Specifically, in the embodiment of the present application, the semantic segmentation module 170 is configured to perform image semantic segmentation on the multi-channel global feature matrix to obtain an image semantic segmentation result. Further, after the feature information of the ground object category about the pixel properties of different categories in the remote sensing multispectral image to be processed is obtained, image semantic segmentation is carried out on the multichannel global feature matrix so as to obtain an image semantic segmentation result. Specifically, in the technical scheme of the application, the image semantic segmentation result is that the remote sensing multispectral image to be processed is segmented into a plurality of ground object category subregions with different category pixel properties, so as to accurately identify and classify the ground objects.
In a specific example of the application, the image semantic segmentation result is to segment the remote sensing multispectral image to be processed into a plurality of ground object category subregions with different category pixel properties.
In summary, a geographic information remote sensing mapping system 100 according to an embodiment of the present application is illustrated, which acquires a remote sensing multispectral image to be processed; and excavating spectral features of the remote sensing spectral image on the ground features by adopting an artificial intelligence technology based on deep learning, and extracting ground feature implicit feature information in the remote sensing spectral image based on the spectral features of the ground features so as to divide all pixels of the image into a plurality of categories according to properties. Therefore, the identification and classification of the ground object can be accurately carried out, so that the remote sensing spectrum image can be accurately divided into a plurality of ground object category subregions with different category pixel properties.
As described above, the geographic information remote sensing mapping system 100 according to the embodiment of the present application may be implemented in various terminal devices, for example, a server for geographic information remote sensing mapping, or the like. In one example, the geographic information remote sensing mapping system 100 according to embodiments of the present application may be integrated into the terminal device as a software module and/or hardware module. For example, the geographic information telemetry system 100 may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, the geographic information telemetry system 100 may also be one of a number of hardware modules of the terminal device.
Alternatively, in another example, the geographic information telemetry system 100 and the terminal device may be separate devices, and the geographic information telemetry system 100 may be connected to the terminal device via a wired and/or wireless network and communicate the interactive information in a agreed data format.
In one embodiment of the present application, fig. 4 is a flowchart of a method for remote sensing of geographic information according to an embodiment of the present application. As shown in fig. 4, a method for remote sensing and mapping geographic information according to an embodiment of the present application includes: 210, acquiring a remote sensing multispectral image to be processed; 220, calculating a direction gradient histogram of the remote sensing multispectral image to be processed; 230, aggregating the remote sensing multispectral image to be processed and the direction gradient histogram along a channel dimension to obtain a multichannel image; 240, performing image blocking processing on the multi-channel image to obtain a sequence of image blocks; 250, passing the sequence of image blocks through a dual-branch structure comprising a first convolutional neural network model and a second convolutional neural network model to obtain a sequence of image block feature matrices, wherein the first convolutional neural network model and the second convolutional neural network model use two-dimensional convolutional kernels with different scales; 260, arranging the sequence of the image block feature matrix into a multi-channel global feature matrix; and 270, performing image semantic segmentation on the multi-channel global feature matrix to obtain an image semantic segmentation result.
Fig. 5 is a schematic diagram of a system architecture of a geographic information remote sensing mapping method according to an embodiment of the application. As shown in fig. 5, in the system architecture of the remote sensing mapping method of geographic information, firstly, a remote sensing multispectral image to be processed is obtained; then, calculating a direction gradient histogram of the remote sensing multispectral image to be processed; then, the remote sensing multispectral image to be processed and the direction gradient histogram are aggregated along the channel dimension to obtain a multichannel image; then, carrying out image blocking processing on the multichannel image to obtain a sequence of image blocks; then, the sequence of the image block is respectively passed through a double-branch structure comprising a first convolution neural network model and a second convolution neural network model to obtain a sequence of an image block characteristic matrix, wherein the first convolution neural network model and the second convolution neural network model use two-dimensional convolution kernels with different scales; then, arranging the sequence of the image block feature matrix into a multi-channel global feature matrix; finally, performing image semantic segmentation on the multi-channel global feature matrix to obtain an image semantic segmentation result
In a specific example, in the above-mentioned geographic information remote sensing mapping method, the steps of passing the sequence of image blocks through a dual-branch structure including a first convolutional neural network model and a second convolutional neural network model to obtain a sequence of image block feature matrices, where the first convolutional neural network model and the second convolutional neural network model use two-dimensional convolution kernels with different scales, include: performing depth convolution coding on each image block in the sequence of image blocks by using a first convolution neural network model of the double-branch structure to obtain a first branch feature matrix; performing depth convolution coding on each image block in the sequence of image blocks by using a second convolution neural network model of the double-branch structure to obtain a second branch feature matrix; and fusing the first branch feature matrix and the second branch feature matrix to obtain the image block feature matrix.
In a specific example, in the above-mentioned remote sensing mapping method of geographic information, performing depth convolution encoding on each image block in the sequence of image blocks by using the first convolution neural network model of the dual-branch structure to obtain a first branch feature matrix includes: and respectively carrying out convolution processing based on a two-dimensional convolution kernel, pooling processing along a channel dimension and nonlinear activation processing on input data in forward transfer of layers by using each layer of the first convolution neural network model of the double-branch structure to output the first branch feature matrix by the last layer of the first convolution neural network model of the double-branch structure, wherein the input of the first layer of the first convolution neural network model of the double-branch structure is each image block in the sequence of the image blocks.
In a specific example, in the above-mentioned remote sensing mapping method of geographic information, performing depth convolution encoding on each image block in the sequence of image blocks by using the second convolution neural network model of the dual-branch structure to obtain a second branch feature matrix, including: and respectively carrying out convolution processing based on a two-dimensional convolution kernel, pooling processing along a channel dimension and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model of the double-branch structure to output the second branch characteristic matrix by the last layer of the second convolution neural network model of the double-branch structure, wherein the input of the first layer of the second convolution neural network model of the double-branch structure is each image block in the sequence of the image blocks.
In a specific example, in the above method for remote sensing mapping geographic information, fusing the first branch feature matrix and the second branch feature matrix to obtain the image block feature matrix includes: adopting convolution dictionary contrast response learning to fuse the first branch feature matrix and the second branch feature matrix by the following optimization formula so as to obtain the image block feature matrix; wherein, the optimization formula is:
Wherein M 1 and M 2 are the first and second branch feature matrices, respectively, and II F represents the Frobenius norm of the matrix, Representing the addition of the matrix,Representing matrix multiplication, M c represents the image block feature matrix, and (-) T represents the transpose of the matrix.
In a specific example, in the above method for remote sensing mapping geographic information, the arranging the sequence of the image block feature matrices into a multi-channel global feature matrix includes: arranging the sequence of the image block feature matrix into a multi-channel global feature matrix according to the following arrangement formula; wherein, the arrangement formula is:
Mc=Concat[M1,M2…Mn]
Wherein M 1,M2…Mn represents the sequence of the image block feature matrix, concat [. Cndot. ] represents a cascading function, and M c represents the multi-channel global feature matrix.
In a specific example, in the above-mentioned geographical information remote sensing mapping method, the image semantic segmentation result is to segment the remote sensing multispectral image to be processed into a plurality of ground object category subregions with different category pixel properties.
It will be appreciated by those skilled in the art that the specific operation of the various steps in the above described method of remote geographical information mapping has been described in detail above with reference to the remote geographical information mapping system of fig. 1 to 3, and thus, duplicate descriptions thereof will be omitted.
The present application also provides a computer program product comprising instructions which, when executed, cause an apparatus to perform operations corresponding to the above-described method.
In one embodiment of the present application, there is also provided a computer-readable storage medium storing a computer program for executing the above-described method.
It should be appreciated that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the forms of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects may be utilized. Furthermore, the computer program product may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Methods, systems, and computer program products of embodiments of the present application are described in the flow diagrams and/or block diagrams. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The basic principles of the present application have been described above in connection with specific embodiments, but it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are merely examples and not intended to be limiting, and these advantages, benefits, effects, etc. are not to be construed as necessarily possessed by the various embodiments of the application. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, as the application is not necessarily limited to practice with the above described specific details.
The block diagrams of the devices, apparatuses, devices, systems referred to in the present application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
It is also noted that in the apparatus, devices and methods of the present application, the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent aspects of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or terminal device that comprises the element.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the application to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.