CN107958216A - Based on semi-supervised multi-modal deep learning sorting technique - Google Patents

Abstract

The present invention proposes to use deep learning to classify while considering the rich multimodal information of samples and the classification contribution differences of each modality, and uses a semi-supervised method to solve the problem of insufficient samples. The data are sent to the deep neural network separately, and a semi-supervised method is used to utilize a large number of unlabeled samples, and the deep neural network based on self-encoding is used for feature learning, and all marked and unlabeled data are sent to the self-encoding deep network for learning. Design similar networks for different modalities, obtain their respective initialization parameters through self-encoding reconstruction, and obtain the hidden attribute classification of marked samples through clustering. For unlabeled data, first calculate through multi-objective deep network Its deep features, then find approximate labeled samples based on cluster labels, and finally predict the labels of unlabeled samples based on the label information of labeled samples.

Description

Semi-supervised multimodal deep learning classification method

technical field

The present invention relates to a semi-supervised method based on semi-supervised multi-modal deep learning to solve the problem of insufficient samples while considering the rich multi-modal information of samples and the differences in the classification contributions of each modality while using deep learning to classify. Learn how to classify.

Background technique

Hyperspectral remote sensing images have multiple modal information such as texture, spatial correlation, and spectrum. The fusion of so many aspects of feature information enables hyperspectral remote sensing to detect more ground object information, which greatly improves human understanding of the objective world. cognitive ability.

At present, more researchers directly use deep models to fuse part of the modal information, and fewer researchers construct deep network architectures for the differences in the classification contribution of modalities. In addition, the classification performance of images mainly depends on a large number of training samples, but in practice there are often insufficient labeled samples. There are still relatively few studies on how to maintain the generalization ability of deep learning on the basis of having fewer labeled samples.

In the known technology, there are many deep models, such as SAE, DBN, DCNN and so on. In 2015, Li et al. proposed a SAE-based deep architecture to combine low-level features. Chen et al. proposed to use another new deep model DBN for classification. The DBN model can extract more robust features and obtain better classification accuracy than other models.

In terms of deep model DCNN, DCNN can effectively extract spatial information by using local connections, and reduce training parameters by sharing weights. Zhao et al. first compress spectral features by local discriminative embedding method, learn spatial features by CNN, and then put the processed spectral features and spatial features into a multi-classifier. Chen et al. proposed an end-to-end DCNN framework for feature learning and hyperspectral image classification. A.Romero et al. proposed to use unsupervised convolutional neural network to analyze remote sensing images. This method builds a DCNN model through an unsupervised greedy layered pre-training method. Aiming at the overfitting problem caused by few training samples and high dimensionality, Chen et al. used L2 regularization and dropout to solve it. Li et al. proposed the idea of using CNN for pixel-wise feature classification to compensate for the lack of data.

Contents of the invention

To sum up, most of the existing known technology researches directly use the deep model to fuse part of the modal information, and few researchers construct a deep network architecture for the difference in the classification contribution of the modal. The disadvantage of the known technology is that many deep models, such as SAE and DBN, adopt a fully connected mode between different layers, which requires training a large number of parameters, but in practice, there is often a lack of a large number of available labeled training samples. In addition, SAE and DBN only represent spatial information in the form of vectors during the training phase, so they cannot extract spatial information more effectively.

For the DANN model, although it uses local connections, it can effectively extract spatial information and reduce the number of parameters by sharing weights. However, it requires many hidden layers to learn more complex data features, and may still need to learn a lot of parameters, especially when the number of training samples is insufficient, which often leads to overfitting problems.

In contrast, the existing known technologies cannot comprehensively consider how to maintain the generalization ability of deep learning on the basis of having fewer labeled samples and how to combine the multimodal information of samples with fewer labeled samples to construct Deep Learning Architecture.

The present invention is proposed in order to overcome the defects in the prior art. The present invention combines the multi-modal information of the sample's spatial domain features, texture features, and spectral features and uses the classification contribution differences of each modal information to construct a deep network architecture. In addition, it also considers the problem of a small number of labeled samples, using semi-supervised A method to predict a large number of unlabeled samples from a small number of labeled samples.

The cost is classification accuracy.

The technical solution adopted in the present invention is: a classification method based on semi-supervised multimodal deep learning, comprising the following steps:

(1) The hyperspectral image is divided into spectral mode, spatial texture mode and spatial correlation mode, and the corresponding deep convolutional neural network is designed for each mode;

(2) Design a fusion classifier F that fuses the three modalities according to the differences in the classification contributions of the modalities; each modal itself has its own hidden attributes, so different modal data can be clustered to obtain its Intrinsic attribute category, each modality corresponds to a latent attribute classifier, which are spectral latent attribute classifier S, texture latent attribute classifier T, and correlation latent attribute classifier R.

(3) For all labeled samples, use the clustering method to obtain their hidden attribute classification;

(4) After clustering, each sample has two labels, and the corresponding relationship between these two categories can be obtained;

(5) Carry out cluster analysis on unlabeled samples, first calculate its deep features through multi-target deep network, then find approximate labeled samples based on cluster labels, and finally predict the label of unlabeled samples according to the label information of labeled samples ;

The step (5) needs to set the confidence standard of the predicted label. If it exceeds the standard, no prediction is made. When a certain batch of unlabeled samples is predicted to have the label information, the classification contribution difference of each modality is adaptively adjusted. .

Beneficial effects of the present invention:

The present invention relates to a semi-supervised multi-modal deep learning classification method, which comprehensively considers the multi-modal information of samples and the contribution difference of each modal classification, and combines it with a semi-supervised method to solve the problem of few labeled samples. Improve the generalization ability of deep models.

A semi-supervised multi-modal deep learning classification method based on semi-supervised methods to solve the problem of insufficient samples while considering the rich multi-modal information of samples and the differences in the classification contributions of each modality while using deep learning for classification. . This method takes into account the fusion of multi-modal information of texture features, spatial features, and spectral features of the pixel, and also considers the contribution of each modal information to the deep network architecture. The contribution difference of the classification and the prediction of unlabeled samples are combined to improve the generalization ability of the deep model, and finally to improve the accuracy of hyperspectral image classification.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings.

The above objects, advantages and features of the present invention will become more apparent by referring to the following detailed description of the adopted embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a schematic diagram of the overall process structure of the present invention;

Fig. 2 is the specific practical steps of the semi-supervised multimodal deep learning framework of the embodiment of the present invention;

Fig. 3 is the specific practical steps of the semi-supervised multimodal deep learning framework of the embodiment of the present invention;

Fig. 4 is the concrete flowchart of the semi-autonomous network structure parameter initialization based on the embodiment of the present invention;

Fig. 5 is a specific flow chart of the multi-objective-based collaborative prediction method for unlabeled samples according to the embodiment of the present invention.

Detailed ways

Specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Fig. 1 shows the system structure of the embodiment of the present invention. 102 is the input of hyperspectral image samples and label information, 103 represents our preprocessing of received samples, 104 represents our grouping of processed samples, and 105 is sending the grouped samples to semi-supervised multiple Learning in the modal deep learning framework, 106 represents combining the results of each deep learning framework to make a decision process, and 107 represents obtaining the final classification result.

Fig. 2 and Fig. 3 show the specific practical steps of the semi-supervised multi-modal deep learning framework of the embodiment of the present invention. In this scheme, different modal data of hyperspectral images are fed into the deep neural network, and a semi-supervised method is used to utilize a large number of unlabeled samples. 201 to 203 indicate that the hyperspectral image is divided into spectral mode, spatial texture mode and spatial correlation mode, and the corresponding deep convolutional neural network is designed for each mode respectively 204 to 206, and the deep network adopts AlexNet Architecture, the AlexNet architecture has five convolutional layers and two fully connected layers. The difference degree of classification contribution according to the modalities is shown in 304 , and 306 is the fusion classifier F designed to fuse the three modalities. Each modality itself has its own hidden attributes. For example, similar textures can be classified as a category. Therefore, different modality data can be clustered to obtain their intrinsic attribute categories. Each modality corresponds to an implicit attribute. Classifiers, respectively 305 is a spectral hidden attribute classifier S, 307 is a texture hidden attribute classifier T, and 308 is a correlation hidden attribute classifier R. The deep neural network uses two classification targets of sample category and hidden attribute category to learn abstract features. The specific multi-category target learning strategy (shown in 207 ) is shown in FIG. 3 . Taking the spectral modality 301 as an example, it demonstrates the learning strategy of multi-category targets, in which 306 is the fusion classifier F, which takes the sample category as the target, and the spectral hidden attribute classifier S, as shown in 305, uses the spectral latent With the attribute as the target, by fusing two classification targets, the hidden attribute category based on the sample category can be learned, as shown in 309 .

Fig. 4 shows specific practical steps of initialization of network structure parameters based on autoencoder according to the embodiment of the present invention. Hyperspectral remote sensing images usually only have a small amount of high-quality label information, and it is difficult to use supervised methods to learn suitable network initialization parameters. In order to learn relatively reasonable initial network parameters, a deep neural network based on self-encoding will be used for feature learning. 402 is to send all labeled and unlabeled data into the self-encoding deep network for learning. This deep network architecture includes five layers of convolutional layers and four layers of encoding and self-encoding layers. 403 is to design similar networks for different modalities. , 404 is to obtain the respective initialization parameters through self-encoding reconstruction. For a certain modality data v of the hyperspectral image, the corresponding output is 405 is by minimizing the mean square reconstruction error To train the network, where M and N are the number of labeled samples and unlabeled samples, respectively, and r represents the modality. 406 is where we optimize the reconstruction network using a standard backpropagation algorithm based on stochastic gradient descent. 407 is when the network of each modality reaches convergence, the parameters of the convolution layer in the network structure based on self-encoding are used as the parameters of the convolution layer of the corresponding mode in the corresponding semi-supervised deep network structure.

FIG. 5 shows specific practical steps of the multi-objective-based collaborative prediction method for unlabeled samples according to the embodiment of the present invention. For a certain modality data of the hyperspectrum, 503 is for all labeled samples, using the clustering method to obtain its hidden attribute classification. After clustering, each sample has two labels: the original category label c and the clustering The hidden attribute category label c' after the class, 504 is for the labeled sample, and the corresponding relationship between the two categories can be obtained. For unlabeled samples, the purpose of clustering is: according to the clustering results of unlabeled samples (unknown c labels) (obtaining c' labels), in labeled samples according to c' labels to find similar c labels. In order to better balance the category label c and the hidden attribute category label c', in the 505 categories, we target both for deep model training. Based on the consideration of multi-modal data, multi-modal fusion is adopted for the classification target of category label c, and the fusion strategy adopts the difference degree of classification contribution mentioned above. The objective function of multi-target deep model learning is: Here r represents the modality, and w ^r represents the classification contribution of modality r. After the network training is completed based on the multi-objective loss function, 506 is for the unlabeled data, first calculate its deep features through the multi-objective deep network, then 507 is to find approximate marked samples based on the cluster label, and 508 is based on the label of the marked sample Information predicts the labeling of unlabeled samples. It is necessary to set the confidence standard of the prediction label. If it exceeds the standard, no prediction is made. 509 means that when a certain batch of unlabeled samples is predicted with label information, each modality can be adaptively adjusted. The classification contribution difference degree of , 510 means that when the network training tends to be stable and no new samples meeting the prediction standard appear, the training can be stopped. After the respective classification results are obtained through the deep model, the final classification result is determined through decision combination.

Although the present invention has been illustrated in conjunction with the preferred embodiments thereof, those skilled in the art will understand that various modifications, substitutions and alterations can be made to the present invention without departing from the spirit and scope of the invention. Accordingly, the invention should not be limited by the above-described embodiments, but by the appended claims and their equivalents.

Claims (2)

1. based on semi-supervised multimodal deep learning classification method, it is characterized in that, comprising the steps: (1) The hyperspectral image is divided into spectral mode, spatial texture mode and spatial correlation mode, and the corresponding deep convolutional neural network is designed for each mode; (2) Design a fusion classifier F that fuses the three modalities according to the differences in the classification contributions of the modalities; each modal itself has its own hidden attributes, so different modal data can be clustered to obtain its Intrinsic attribute category, each modality corresponds to a latent attribute classifier, which are spectral latent attribute classifier S, texture latent attribute classifier T, and correlation latent attribute classifier R. (3) For all labeled samples, use the clustering method to obtain their hidden attribute classification; (4) After clustering, each sample has two labels, and the corresponding relationship between these two categories can be obtained; (5) Carry out cluster analysis on unlabeled samples, first calculate its deep features through multi-target deep network, then find approximate labeled samples based on cluster labels, and finally predict the label of unlabeled samples according to the label information of labeled samples . 2. the multimodal deep learning classification method based on semi-supervised according to claim 1, it is characterized in that, described step (5) needs to set the confidence standard of prediction mark, if exceed standard then do not predict, when certain batch After the unlabeled samples are predicted with labeled information, the classification contribution difference of each modality is adaptively adjusted.