CN116258971B - Multi-source fused forestry remote sensing image intelligent interpretation method - Google Patents
Multi-source fused forestry remote sensing image intelligent interpretation method Download PDFInfo
- Publication number
- CN116258971B CN116258971B CN202310543222.7A CN202310543222A CN116258971B CN 116258971 B CN116258971 B CN 116258971B CN 202310543222 A CN202310543222 A CN 202310543222A CN 116258971 B CN116258971 B CN 116258971B
- Authority
- CN
- China
- Prior art keywords
- feature
- remote sensing
- image
- features
- fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 230000004927 fusion Effects 0.000 claims abstract description 110
- 238000012937 correction Methods 0.000 claims abstract description 35
- 230000002457 bidirectional effect Effects 0.000 claims abstract description 21
- 239000013598 vector Substances 0.000 claims description 24
- 238000010586 diagram Methods 0.000 claims description 12
- 238000013528 artificial neural network Methods 0.000 claims description 9
- 238000011176 pooling Methods 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000012545 processing Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 235000019580 granularity Nutrition 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000002679 ablation Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003703 image analysis method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/188—Vegetation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an intelligent interpretation method of a multisource fused forestry remote sensing image, and relates to the technical field of remote sensing image processing; the method comprises the following steps: s10, inputting the acquired forestry remote sensing image into a multi-source image fusion module, and obtaining multi-source and multi-granularity forestry remote sensing image characteristics after the multi-source image fusion module carries out characteristic correction and characteristic fusion on the forestry remote sensing image; s20, inputting forestry remote sensing image features into a multi-source multi-granularity feature fusion module, wherein the multi-source multi-granularity feature fusion module is a top-down and bottom-up bidirectional fusion backbone network, and feature bidirectional fusion is realized through the bidirectional fusion backbone network; s30, the features subjected to the feature bidirectional fusion are sent into a forestry remote sensing image interpretation module for interpretation; the beneficial effects of the invention are as follows: the problem that small target features in forestry remote sensing images are difficult to extract is solved.
Description
Technical Field
The invention relates to the technical field of remote sensing image processing, in particular to an intelligent interpretation method of a multisource fusion forestry remote sensing image.
Background
The great progress of earth observation technology provides continuous remote sensing image data for observing and understanding earth surface changes. How to fully utilize mass remote sensing images and effectively analyze and understand the mass remote sensing images becomes a hot spot and difficult problem which are urgently needed to be researched at present. The remote sensing image scene classification technology (RSISC) is one of the important contents of remote sensing image understanding, and the main task of the RSISC is to add predefined label information to slices in a large-scale remote sensing image, such as airports, ports, farmlands, residential areas and the like. The predefined scene category labels are usually determined according to the ground range function covered by the image, so that the same type of remote sensing image scene usually comprises multiple types of ground objects. Different from pixel-level information and target-level information, the remote sensing image scene classification is a semantic-level remote sensing image analysis method, and has wide and important application value in aspects of forest and farmland coverage investigation, geological disaster monitoring, target detection and identification, urban environment planning and evaluation and the like.
At present, most of remote sensing data processing software built-in classification methods only utilize remote sensing image data, an interpretation method is constructed from the structure of an image to interpret, the accuracy is difficult to further improve, and the problem that small target features in forestry remote sensing images are difficult to extract is commonly existed.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a multisource fusion forestry remote sensing image intelligent interpretation method.
The technical scheme adopted for solving the technical problems is as follows: in the intelligent interpretation method of the forestry remote sensing image with multisource fusion, the improvement is that the method comprises the following steps:
s10, inputting the acquired forestry remote sensing image into a multi-source image fusion module, and obtaining multi-source and multi-granularity forestry remote sensing image characteristics after the multi-source image fusion module carries out characteristic correction and characteristic fusion on the forestry remote sensing image;
s20, inputting forestry remote sensing image features into a multi-source multi-granularity feature fusion module, wherein the multi-source multi-granularity feature fusion module is a top-down and bottom-up bidirectional fusion backbone network, and feature bidirectional fusion is realized through the bidirectional fusion backbone network;
s30, the features subjected to the feature bidirectional fusion are sent into a forestry remote sensing image interpretation module for interpretation.
Further, in step S10, the multi-source image fusion module includes two parallel neural networks, extracts features from the forestry remote sensing RGB image features and the other mode image features through the neural networks, and performs feature correction and feature fusion on the features.
Further, the other mode image feature is a full color image, a hyperspectral image or a multispectral image.
Further, the feature correction of the feature includes the steps of:
s101, after forestry remote sensing RGB image features and other modal image features are input, splicing the two inputs, and then carrying out global pooling so as to keep more information;
s102, obtaining feature mapping by using the multi-layer neural network mlp and further dividing the feature mapping into two channel attention vectorsAnd->Multiplying the result with the original input, wherein RGB backbone features and attention vector +.>Multiplying and adding the multiplied and added with the other mode image characteristic according to elements, and adding the other mode image characteristic and the attention vector +.>After multiplication, the obtained product is added with forestry remote sensing RGB image characteristics according to elements, and the formula is as follows:
；
and->Input feature maps respectively representing the RGB trunk feature and the other mode image feature, where i represents the connection of feature graphs from both modes, and (2)>Representing a global average pooling operation;
；
wherein the method comprises the steps ofFor sigmoid function, +.>Representing MLP network, < >>Dividing the result into->Andchannel weight vectors respectively representing the RGB trunk feature and the other mode image feature;
；
；
and->Respectively representing a characteristic diagram of the RGB trunk characteristic and the characteristic of another mode image after the input of the characteristic of the channel is corrected;
s103, the multi-source image fusion module comprises a forestry remote sensing characteristic correction module, and after entering the forestry remote sensing characteristic correction module, two spatial attention vectors are obtained through 1X1 convolution with the two output channels being 1, a similarity matrix is obtained through softmax, and the spatial attention vectors are separated intoAnd->Multiplying the obtained product by the output in the step S102, adding the multiplied product to obtain forestry remote sensing RGB image characteristics with corrected characteristics and the image characteristics of the other mode, and outputting the obtained forestry remote sensing RGB image characteristics and the image characteristics of the other mode:
；
；
；
；
wherein the method comprises the steps ofConv1×1 (·) table for RELU functionThe illustration is by 1x1 convolution,/->Dividing the result into->And->Spatial weight vectors respectively representing the RGB trunk feature and the other mode image feature; />Andand respectively representing the characteristic diagrams of the RGB trunk characteristic and the characteristic of the other mode image after the input of the characteristic is subjected to spatial characteristic correction.
Further, the feature fusion of the features comprises the following steps:
s104, the multi-source image fusion module comprises a forestry remote sensing feature fusion module, and the forestry remote sensing feature fusion module receives forestry remote sensing RGB image features and other modal image features which are output by the forestry remote sensing feature correction module;
s105, performing element-based difference operation on the received forestry remote sensing RGB image characteristics and the other mode image characteristics, and obtaining an attention vector after the obtained characteristics pass through a global pooling layer and a sigmoid functionAfter the characteristic is multiplied by elements with the input forestry remote sensing RGB image characteristic, the characteristic fusion operation of the layer is finished after the characteristic is added by elements with the input other mode image characteristic, and the formula is as follows:
；
the output of the forestry remote sensing feature fusion module is:
。
further, in the step S10, multiple-source and multiple-granularity forestry remote sensing image features C2, C3, C4 and C5 are obtained, and the forestry remote sensing image features C2, C3, C4 and C5 are features with multiple scales output by different layers;
in the step S20, the implementation of feature bidirectional fusion by the multi-source multi-granularity feature fusion module includes the following steps:
s201, carrying out 1X1 convolution on the feature C5 to obtain a feature P5;
s202, up-sampling the features C4, C3 and C2 after 1X1 convolution, and adding units with the features of the next layer after 1X1 convolution from top to bottom to sequentially obtain features P4, P3 and P2;
s203, the feature P2 is subjected to 3X3 convolution to obtain a feature N2; then, the features P2, P3 and P4 are subjected to 3X3 convolution and 3X3 convolution with a step length of 2, downsampling is carried out on the features, unit addition is carried out on the output of the upper layer from bottom to top, and the features N3, N4 and N5 are sequentially obtained after each 3X3 convolution.
Further, the feature C3 is subjected to unit addition on the feature P4 subjected to 1X1 convolution and up-sampling to obtain P3, and the feature N2 is subjected to down-sampling through a convolution of 3X3 with a step length of 2;
then, carrying out feature fusion on the feature P3 and the feature N2 after downsampling in a unit adding mode; then using convolution of 3X3 to increase the characteristic capacity of the characteristics after characteristic fusion;
non-linearizing the feature using a ReLU activation function to obtain feature N3:
；
。
further, in step S30, the features N2, N3, N4, N5 after multi-source and multi-granularity fusion are sent into the forestry remote sensing image interpretation modules, and different forestry remote sensing image interpretation modules are selected according to different forestry remote sensing image interpretation tasks.
The beneficial effects of the invention are as follows: the forestry remote sensing characteristic correction module corrects and calibrates noise of each mode by utilizing information complementation of different modes; the forestry remote sensing feature fusion module can effectively highlight the difference of features by subtracting pixel by pixel, so that the fusion of different modal features can obtain a better effect in the training process; in addition, a multi-source multi-granularity feature fusion module is provided, and the problem that small target features in forestry remote sensing images are difficult to extract is solved.
Drawings
Fig. 1 is a flow chart of a multi-source fused forestry remote sensing image intelligent interpretation method of the invention.
Fig. 2 is a schematic structural diagram of a remote sensing characteristic correction module in the forest industry.
Fig. 3 is a schematic structural diagram of a remote sensing feature fusion module in the forest industry.
Fig. 4 is a schematic structural diagram of a multi-source multi-granularity feature fusion module according to the present invention.
Detailed Description
The invention will be further described with reference to the drawings and examples.
The conception, specific structure, and technical effects produced by the present invention will be clearly and completely described below with reference to the embodiments and the drawings to fully understand the objects, features, and effects of the present invention. It is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments, and that other embodiments obtained by those skilled in the art without inventive effort are within the scope of the present invention based on the embodiments of the present invention. In addition, all the coupling/connection relationships referred to in the patent are not direct connection of the single-finger members, but rather, it means that a better coupling structure can be formed by adding or subtracting coupling aids depending on the specific implementation. The technical features in the invention can be interactively combined on the premise of no contradiction and conflict.
The invention provides a multisource fusion forestry remote sensing image intelligent interpretation method, which is shown in combination with fig. 1, namely a flow diagram of the multisource fusion forestry remote sensing image intelligent interpretation method, and mainly comprises a multisource image fusion step, a multisource and multiscale feature fusion step and a forestry remote sensing image translation step.
With continued reference to fig. 1 to 3, the method for intelligently interpreting the forestry remote sensing images by multi-source fusion according to the present invention includes the following steps:
s10, inputting the acquired forestry remote sensing image into a multi-source image fusion module, and obtaining the characteristics of the forestry remote sensing image with multiple sources and multiple granularities after the multi-source image fusion module carries out characteristic correction and characteristic fusion on the forestry remote sensing image.
In this embodiment, the multi-source image fusion module includes two parallel neural networks, extracts features from forestry remote sensing RGB image features and another mode image feature (referred to as a mode in fig. 2 and 3) respectively through the neural networks, and performs feature correction and feature fusion on the features. Wherein the other mode image feature is a full-color image, a hyperspectral image or a multispectral image; in this embodiment, the other modality image feature is a full color image.
In the present invention, for the feature correction, the present invention provides a specific embodiment, and the multi-source image fusion module includes a forestry remote sensing feature correction module, as shown in fig. 2, that is, a schematic structural diagram of the forestry remote sensing feature correction module (in fig. 2, simply referred to as a feature correction module).
Forestry remote sensing images typically present some noise, and although features from different modalities have their specific noise measurements and corrections, their semantic information is lost. Since the information from different sensing modes is usually complementary, the invention proposes a forestry remote sensing feature correction module according to this idea. The forestry remote sensing characteristic correction module is used as one of the most important modules in the multisource fusion forestry remote sensing image, and the main body of the forestry remote sensing characteristic correction module is divided into two parts, namely 1, channel characteristic correction; 2. and correcting the spatial characteristics. In this embodiment, the feature correction of the feature includes the following steps:
s101, after forestry remote sensing RGB image features and other modal image features are input, splicing the two inputs, and then carrying out global pooling so as to keep more information;
s102, obtaining information by using a multi-layer neural network mpl, and dividing a channel attention vector into two partsAndmultiplying the result with the original input, wherein RGB backbone features and attention vector +.>After multiplication, the two kinds of image features are added according to elements, and the main feature and the attention vector of the other kind of image features are added>After multiplication, the obtained product is added with forestry remote sensing RGB image characteristics according to elements, and the formula is as follows:
；
and->Input feature maps respectively representing the RGB trunk feature and the other mode image feature, where i represents the connection of feature graphs from both modes, and (2)>Representing a global average pooling operation;
；
wherein the method comprises the steps ofFor sigmoid function, +.>Representing MLP network, < >>Dividing the result into->And->Channel weight vectors respectively representing the RGB trunk feature and the other mode image feature;
；
；
and->Respectively representing a characteristic diagram of the RGB trunk characteristic and the characteristic of another mode image after the input of the characteristic of the channel is corrected;
s103, the multi-source image fusion module comprises a forestry remote sensing characteristic correction module, and after entering the forestry remote sensing characteristic correction module, two spatial attention vectors are obtained through 1X1 convolution with the two output channels being 1, a similarity matrix is obtained through softmax, and the spatial attention vectors are separated intoAnd->Multiplying the obtained product by the output in the step S102 and adding the multiplied product to obtain forestry remote sensing RGB image characteristics and characteristics after characteristic correctionAnother mode image feature is output:
；
；
；
；
wherein the method comprises the steps ofConv1×1 (·) represents the convolution of 1×1,/-for RELU function>The results are divided intoAnd->Spatial weight vectors respectively representing the RGB trunk feature and the other mode image feature; />Andand respectively representing the characteristic diagrams of the RGB trunk characteristic and the characteristic of the other mode image after the input of the characteristic is subjected to spatial characteristic correction.
The forestry remote sensing characteristic correction module filters and calibrates the characteristic of each mode by utilizing the complementation of the characteristic information of different modes, reduces the loss of semantic information and simultaneously processes the noise of different modes, thereby realizing better multi-mode characteristic extraction and information interaction.
Further, for the above feature fusion, the multi-source image fusion module includes a forestry remote sensing feature fusion module, and the structure diagram of the forestry remote sensing feature fusion module is shown with reference to fig. 3 (the feature fusion module is abbreviated as "feature fusion module" in fig. 3). In a forestry remote sensing feature fusion module, obtaining complementary information of adjacent feature nodes, and then fusing reconstruction information from an RGB layer and another mode; based on this idea, the invention proposes a simple and efficient feature fusion module.
In this embodiment, feature fusion of features includes the following steps:
s104, the multi-source image fusion module comprises a forestry remote sensing feature fusion module, and the forestry remote sensing feature fusion module receives forestry remote sensing RGB image features and other modal image features which are output by the forestry remote sensing feature correction module;
s105, performing element-based difference operation on the received forestry remote sensing RGB image characteristics and the other mode image characteristics, and obtaining an attention vector after the obtained characteristics pass through a global pooling layer and a sigmoid functionAfter the characteristic is multiplied by elements with the input forestry remote sensing RGB image characteristic, the characteristic fusion operation of the layer is finished after the characteristic is added by elements with the input other mode image characteristic, and the formula is as follows:
；
the output of the forestry remote sensing feature fusion module is:
。
in the ablation experiment, the difference between the elements is changed into element-by-element addition, and the effect is obviously reduced. Compared with the pixel-by-pixel addition, the pixel-by-pixel subtraction can effectively highlight the difference of the features, so that the fusion of the features of different modes can obtain better effect in the training process.
S20, inputting forestry remote sensing image features into a multi-source multi-granularity feature fusion module, wherein the multi-source multi-granularity feature fusion module is a top-down and bottom-up bidirectional fusion backbone network, and feature bidirectional fusion is realized through the bidirectional fusion backbone network;
higher-level neurons are sensitive to overall features, while lower-level neurons are more easily activated by local features, indicating that a top-down path can propagate semantically strong features. The model further enhances the capacity of the whole feature hierarchy by propagating low-level features, and for this purpose, the invention establishes a path for transverse connection from the low level to the top level to realize feature bidirectional fusion.
In this embodiment, in the step S10, the forestry remote sensing image features C2, C3, C4, and C5 with multiple sources and multiple granularities are obtained, and the forestry remote sensing image features C2, C3, C4, and C5 are features with multiple scales output by different layers.
For the multi-source multi-granularity feature fusion module to realize feature bidirectional fusion, in combination with the illustration of fig. 4, the present invention provides a specific embodiment, and in step S20, the multi-source multi-granularity feature fusion module to realize feature bidirectional fusion includes the following steps:
s201, carrying out 1X1 convolution on the feature C5 to obtain a feature P5;
s202, up-sampling the features C4, C3 and C2 after 1X1 convolution, and adding units with the features of the next layer after 1X1 convolution from top to bottom to sequentially obtain features P4, P3 and P2;
s203, the feature P2 is subjected to 3X3 convolution to obtain a feature N2; then, the features P2, P3 and P4 are subjected to 3X3 convolution and 3X3 convolution with a step length of 2, downsampling is carried out on the features, unit addition is carried out on the output of the upper layer from bottom to top, and the features N3, N4 and N5 are sequentially obtained after each 3X3 convolution.
In this embodiment, the calculation of the features N2 to N3 is taken as an example: the feature C3 is subjected to unit addition on the feature P4 subjected to 1X1 convolution and up-sampling to obtain P3, and the feature N2 is subjected to down-sampling through a convolution of 3X3 with a step length of 2; then, carrying out feature fusion on the feature P3 and the feature N2 after downsampling in a unit adding mode; then using convolution of 3X3 to increase the characteristic capacity of the characteristics after characteristic fusion; non-linearizing the feature using a ReLU activation function to obtain feature N3:
；
。
s30, the features subjected to the feature bidirectional fusion are sent into a forestry remote sensing image interpretation module for interpretation.
In step S30, the features N2, N3, N4, N5 after multi-source and multi-granularity fusion are sent into the forestry remote sensing image interpretation modules, and different forestry remote sensing image interpretation modules are selected according to different forestry remote sensing image interpretation tasks. For example, the forestry image classification task uses a full link layer, the forestry image target detection task uses a Head such as a fast RCNN, and the forestry image division task uses a Head such as an FCN.
Based on the method, the invention provides a multi-source image fusion module, wherein the forestry remote sensing characteristic correction module corrects and calibrates noise of each mode by utilizing information complementation of different modes; the forestry remote sensing feature fusion module can effectively highlight the difference of features by subtracting pixel by pixel, so that the fusion of different modal features can obtain a better effect in the training process. In addition, a multi-source multi-granularity feature fusion module is provided, which can better enable features of different scales to have stronger semantic information, and further improve forestry remote sensing image features; the problem that small target features in forestry remote sensing images are difficult to extract is solved.
While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are included in the scope of the present invention as defined in the appended claims.
Claims (5)
1. A multisource fusion forestry remote sensing image intelligent interpretation method is characterized by comprising the following steps:
s10, inputting the acquired forestry remote sensing image into a multi-source image fusion module, and obtaining multi-source and multi-granularity forestry remote sensing image characteristics after the multi-source image fusion module carries out characteristic correction and characteristic fusion on the forestry remote sensing image;
in the step S10, the multi-source image fusion module includes two parallel neural networks, extracts features from forestry remote sensing RGB image features and another mode image feature through the neural networks, and performs feature correction and feature fusion on the features;
the feature correction of the feature comprises the following steps:
s101, after forestry remote sensing RGB image features and other modal image features are input, splicing the two inputs, and then carrying out global pooling so as to keep more information;
s102, obtaining feature mapping by using the multi-layer neural network mlp and further dividing the feature mapping into two channel attention vectorsAnd->Multiplying the result with the original input, wherein RGB backbone features and attention vector +.>Multiplying and adding the multiplied and added with the other mode image characteristic according to elements, and adding the other mode image characteristic and the attention vector +.>After multiplication, the obtained product is added with forestry remote sensing RGB image characteristics according to elements, and the formula is as follows:
；
and->Input feature maps respectively representing the RGB trunk feature and the other mode image feature, where i represents the connection of feature graphs from both modes, and (2)>Representing a global average pooling operation;
；
wherein the method comprises the steps ofFor sigmoid function, +.>Representing MLP network, < >>Dividing the result into->And->Channel weight vectors respectively representing the RGB trunk feature and the other mode image feature;
；
；
and->Respectively representing a characteristic diagram of the RGB trunk characteristic and the characteristic of another mode image after the input of the characteristic of the channel is corrected;
s103, the multi-source image fusion module comprises a forestry remote sensing characteristic correction module, and after entering the forestry remote sensing characteristic correction module, two spatial attention vectors are obtained through 1X1 convolution with the two output channels being 1, a similarity matrix is obtained through softmax, and the spatial attention vectors are separated intoAnd->Multiplying the obtained product by the output in the step S102, adding the multiplied product to obtain forestry remote sensing RGB image characteristics with corrected characteristics and the image characteristics of the other mode, and outputting the obtained forestry remote sensing RGB image characteristics and the image characteristics of the other mode:
；
；
；
；
wherein the method comprises the steps ofConv1×1 (·) represents a 1× pass for the RELU function1 convolution->The results are divided intoAnd->Spatial weight vectors respectively representing the RGB trunk feature and the other mode image feature; />Andrespectively representing a characteristic diagram of the RGB trunk characteristic and the characteristic of another mode image after spatial characteristic correction;
the feature fusion of the features comprises the following steps:
s104, the multi-source image fusion module comprises a forestry remote sensing feature fusion module, and the forestry remote sensing feature fusion module receives forestry remote sensing RGB image features and other modal image features which are output by the forestry remote sensing feature correction module;
s105, performing element-based difference operation on the received forestry remote sensing RGB image characteristics and the other mode image characteristics, and obtaining an attention vector after the obtained characteristics pass through a global pooling layer and a sigmoid functionAfter the characteristic is multiplied by elements with the input forestry remote sensing RGB image characteristic, the characteristic fusion operation of the layer is finished after the characteristic is added by elements with the input other mode image characteristic, and the formula is as follows:
；
the output of the forestry remote sensing feature fusion module is:
；
s20, inputting forestry remote sensing image features into a multi-source multi-granularity feature fusion module, wherein the multi-source multi-granularity feature fusion module is a top-down and bottom-up bidirectional fusion backbone network, and feature bidirectional fusion is realized through the bidirectional fusion backbone network;
s30, the features subjected to the feature bidirectional fusion are sent into a forestry remote sensing image interpretation module for interpretation.
2. A multi-source fused forestry remote sensing image intelligent interpretation method as claimed in claim 1, wherein the other mode image feature is a full color image, a hyperspectral image or a multispectral image.
3. The intelligent interpretation method of the multi-source fused forestry remote sensing image according to claim 1, wherein in the step S10, multi-source multi-granularity forestry remote sensing image features C2, C3, C4 and C5 are obtained, and the forestry remote sensing image features C2, C3, C4 and C5 are features with multiple scales output by different layers;
in the step S20, the implementation of feature bidirectional fusion by the multi-source multi-granularity feature fusion module includes the following steps:
s201, carrying out 1X1 convolution on the feature C5 to obtain a feature P5;
s202, up-sampling the features C4, C3 and C2 after 1X1 convolution, and adding units with the features of the next layer after 1X1 convolution from top to bottom to sequentially obtain features P4, P3 and P2;
s203, the feature P2 is subjected to 3X3 convolution to obtain a feature N2; then, the features P2, P3 and P4 are subjected to 3X3 convolution and 3X3 convolution with a step length of 2, downsampling is carried out on the features, unit addition is carried out on the output of the upper layer from bottom to top, and the features N3, N4 and N5 are sequentially obtained after each 3X3 convolution.
4. The intelligent interpretation method of the forestry remote sensing image with multisource fusion according to claim 3, wherein the feature C3 is obtained by adding a unit to the feature P4 after 1X1 convolution and up-sampling, and the feature N2 is downsampled through a convolution of 3X3 with a step length of 2;
then, carrying out feature fusion on the feature P3 and the feature N2 after downsampling in a unit adding mode; then using convolution of 3X3 to increase the characteristic capacity of the characteristics after characteristic fusion; non-linearizing the feature using a ReLU activation function to obtain feature N3:
；
。
5. the intelligent interpretation method of multi-source fused forestry remote sensing images as claimed in claim 4, wherein in the step S30, the multi-source multi-granularity fused features N2, N3, N4, N5 are sent into the forestry remote sensing image interpretation modules, and different forestry remote sensing image interpretation modules are selected according to different forestry remote sensing image interpretation tasks.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310543222.7A CN116258971B (en) | 2023-05-15 | 2023-05-15 | Multi-source fused forestry remote sensing image intelligent interpretation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310543222.7A CN116258971B (en) | 2023-05-15 | 2023-05-15 | Multi-source fused forestry remote sensing image intelligent interpretation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116258971A CN116258971A (en) | 2023-06-13 |
| CN116258971B true CN116258971B (en) | 2023-08-08 |
Family
ID=86681100
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310543222.7A Active CN116258971B (en) | 2023-05-15 | 2023-05-15 | Multi-source fused forestry remote sensing image intelligent interpretation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116258971B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117387634B (en) * | 2023-12-13 | 2024-02-27 | 江西啄木蜂科技有限公司 | Color-changing wood forest zone unmanned aerial vehicle path multi-target planning method based on user preference |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110197182A (en) * | 2019-06-11 | 2019-09-03 | 中国电子科技集团公司第五十四研究所 | Remote sensing image semantic segmentation method based on contextual information and attention mechanism |
| CN112800964A (en) * | 2021-01-27 | 2021-05-14 | 中国人民解放军战略支援部队信息工程大学 | Remote sensing image target detection method and system based on multi-module fusion |
| CN113469094A (en) * | 2021-07-13 | 2021-10-01 | 上海中科辰新卫星技术有限公司 | Multi-mode remote sensing data depth fusion-based earth surface coverage classification method |
| CN114936993A (en) * | 2022-05-13 | 2022-08-23 | 常熟理工学院 | High-resolution and pixel relation attention-enhancing strong fusion remote sensing image segmentation method |
| CN115170979A (en) * | 2022-06-30 | 2022-10-11 | 国家能源投资集团有限责任公司 | Mining area fine land classification method based on multi-source data fusion |
| US11521377B1 (en) * | 2021-10-26 | 2022-12-06 | Nanjing University Of Information Sci. & Tech. | Landslide recognition method based on laplacian pyramid remote sensing image fusion |
| CN116012722A (en) * | 2022-09-08 | 2023-04-25 | 中国人民解放军战略支援部队信息工程大学 | A Scene Classification Method for Remote Sensing Images |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10068171B2 (en) * | 2015-11-12 | 2018-09-04 | Conduent Business Services, Llc | Multi-layer fusion in a convolutional neural network for image classification |
| US11830167B2 (en) * | 2021-06-21 | 2023-11-28 | Ping An Technology (Shenzhen) Co., Ltd. | System and method for super-resolution image processing in remote sensing |
-
2023
- 2023-05-15 CN CN202310543222.7A patent/CN116258971B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110197182A (en) * | 2019-06-11 | 2019-09-03 | 中国电子科技集团公司第五十四研究所 | Remote sensing image semantic segmentation method based on contextual information and attention mechanism |
| CN112800964A (en) * | 2021-01-27 | 2021-05-14 | 中国人民解放军战略支援部队信息工程大学 | Remote sensing image target detection method and system based on multi-module fusion |
| CN113469094A (en) * | 2021-07-13 | 2021-10-01 | 上海中科辰新卫星技术有限公司 | Multi-mode remote sensing data depth fusion-based earth surface coverage classification method |
| US11521377B1 (en) * | 2021-10-26 | 2022-12-06 | Nanjing University Of Information Sci. & Tech. | Landslide recognition method based on laplacian pyramid remote sensing image fusion |
| CN114936993A (en) * | 2022-05-13 | 2022-08-23 | 常熟理工学院 | High-resolution and pixel relation attention-enhancing strong fusion remote sensing image segmentation method |
| CN115170979A (en) * | 2022-06-30 | 2022-10-11 | 国家能源投资集团有限责任公司 | Mining area fine land classification method based on multi-source data fusion |
| CN116012722A (en) * | 2022-09-08 | 2023-04-25 | 中国人民解放军战略支援部队信息工程大学 | A Scene Classification Method for Remote Sensing Images |
Non-Patent Citations (1)
| Title |
|---|
| Remote sensing image fusion techniques based on statistical model;Peng-bo Wang et al.;Proceedings of the IET International Conference on Information Science and Control Engineering 2012 (ICISCE 2012);第1-5页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116258971A (en) | 2023-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Qingyun et al. | Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery | |
| Chen et al. | Deep Siamese multi-scale convolutional network for change detection in multi-temporal VHR images | |
| Wang et al. | Spatiotemporal fusion of remote sensing image based on deep learning | |
| Dibs et al. | Multi-fusion algorithms for detecting land surface pattern changes using multi-high spatial resolution images and remote sensing analysis | |
| CN108764063B (en) | Remote sensing image time-sensitive target identification system and method based on characteristic pyramid | |
| Mo et al. | Attribute filter based infrared and visible image fusion | |
| CN111738110A (en) | Vehicle target detection method in remote sensing images based on multi-scale attention mechanism | |
| CN113901900B (en) | Unsupervised change detection method and system for remote sensing images of the same or different sources | |
| CN112434745B (en) | Occlusion target detection and identification method based on multi-source cognitive fusion | |
| CN112801158A (en) | Deep learning small target detection method and device based on cascade fusion and attention mechanism | |
| Zhao et al. | Airborne multispectral LiDAR point cloud classification with a feature reasoning-based graph convolution network | |
| CN113610070A (en) | Landslide disaster identification method based on multi-source data fusion | |
| Zuo et al. | HF-FCN: Hierarchically fused fully convolutional network for robust building extraction | |
| CN114862871B (en) | A method for extracting wheat planting areas from remote sensing images based on SE-UNet deep learning network | |
| CN116258971B (en) | Multi-source fused forestry remote sensing image intelligent interpretation method | |
| CN114612901B (en) | Image change recognition method, device, equipment and storage medium | |
| CN116935240A (en) | Surface coverage classification system and method for multi-scale perception pyramid | |
| CN112767351B (en) | Substation equipment defect detection method based on sensitive position dependence analysis | |
| CN117576483A (en) | Multi-source data fusion feature classification method based on multi-scale convolutional autoencoder | |
| CN115810123A (en) | Small target pest detection method based on attention mechanism and improved feature fusion | |
| CN114943902A (en) | Urban vegetation unmanned aerial vehicle remote sensing classification method based on multi-scale feature perception network | |
| CN115457396A (en) | Surface target ground object detection method based on remote sensing image | |
| CN112036437A (en) | Rice seedling detection model based on improved YOLOV3 network and method thereof | |
| Lu et al. | Citrus green fruit detection via improved feature network extraction | |
| CN117274800A (en) | Hyperspectral remote sensing image anomaly detection method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |