CN106845525A - A kind of depth confidence network image bracket protocol based on bottom fusion feature - Google Patents

Abstract

The present invention provides a deep belief network image classification protocol based on underlying fusion features. First, the color, texture and shape features in the sample image are extracted to form a weight matrix for multi-feature fusion; and then the weight matrix is normalized. ; Finally, the normalized weight matrix is used as the original data to train and test the deep belief network, and the classification result is obtained. The deep belief network is a deep structure composed of multiple restricted Boltzmann machine model RBMs connected together and a BP network; each layer of RBM is connected, and the output of the previous layer of RBM is used as the input of the next layer of RBM. The output of the last layer of RBM is used as the input of the BP network to form the entire deep belief network; the original data is input into the first layer of RBM, and the output of the BP network is the classification result. The invention has good expansibility, multi-feature fusion improves classification accuracy, and deep belief network improves classification efficiency.

Description

A Deep Belief Network Image Classification Protocol Based on Underlying Fusion Features

technical field

The invention relates to a classification algorithm of natural images, in particular to a deep belief network image classification protocol based on underlying fusion features, which is applicable to the technical fields of image information, image classification and retrieval.

Background technique

With the rapid development of digital technology, information technology and multimedia technology, digital images have become an indispensable part of people's daily life, and the number of images is growing at an alarming rate. Faced with more and more image information, image classification And retrieval has become the focus of research. Traditional classification and retrieval methods based on text and annotation have some disadvantages: time-consuming and labor-intensive; the rapid increase of digital images makes it almost impossible to annotate all images; annotators have a great subjective influence. This limits the development of text-based and annotation-based image classification and retrieval. Subsequently, a large number of researches on content-based image classification and retrieval (Content Based Image Retrieval-CBIR) were carried out. This technology overcomes the shortcomings of manual annotation and can realize automatic and intelligent classification, retrieval and management. The current difficulties of image classification are mainly reflected in two aspects: (1) the selection and extraction of features; (2) the selection and learning of classifiers.

Feature selection and extraction are the basis of image classification. There are two types of image features, one is low-level visual features, including color, shape and texture features, SIFT (Scale Invariant Feature Transform) features, etc.; the other is middle-level semantic features, mainly including semantic features, regional semantic concept features, BOW features, etc.

In the classifier method, most of the current classification learning algorithms are mostly shallow structure algorithms, including common support vector machines (SVM), Booting and Logistic Regre-ssion, etc. The typical process of SVM application is to first extract the local features of the image and form a feature code, then use the histogram of the feature words formed by the local features of each image as a feature, and finally train the model through SVM, the limitation of which is limited samples In the case of complex functions and computing units, the ability to express complex functions is limited, and its generalization ability for complex classification problems is subject to certain restrictions. The BP algorithm is a typical algorithm for traditional multi-layer network training, but in fact, this training method is far from ideal for a network that only contains a few layers. Deep learning forms a more abstract high-level representation (attribute category or feature) by combining low-level features to discover distributed feature representations of data. The combination of high-dimensional image descriptors and linear classifiers is currently the most commonly used image classification method.

Lecun Y et al. (Lecun Y, Bottou L, Bengio Y, et al. Gradlet-based learning applied to document recognition [J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324) proposed a layer-by-layer non- The concept of Deep Belief Networks (DBNs) for supervised learning processes. DBN is a deep neural network structure composed of multi-layer Restricted Boltzmann Machines (Restricted Boltzmann Mechines-RBM), which solves the problem of traditional BP algorithm training multi-layer neural network. As a deep learning network, DBN essentially regards the learning structure as a network. The core idea of deep learning is as follows: (1) Unsupervised learning is used for pre-training of each layer of network; Supervised learning only trains one layer, and uses its training results as the input of its higher layer; (3) Use supervised learning to adjust all layers, that is, stack multiple layers, and the output of the previous layer is used as the input of the next layer. In this way, hierarchical expression of input information can be realized. Deep belief network training can be divided into two stages, the first stage is unsupervised feature learning; the second stage is supervised network parameter fine-tuning and classification. At present, the deep belief network has been successfully applied in handwritten font recognition, speech recognition and other fields, and achieved good results.

Subsequently, a large number of scholars conducted related research and improved the DBN algorithm, such as Sun Jinguang, etc. A DBN of numerical attributes is proposed, and comparative verification is carried out on multiple data sets of UCI to prove its effectiveness. Fu Yan et al. (Fu Yanyanming et al. Image classification based on multi-features and improved SVM integration [J] Computer Engineering 2011 37(21): 196-199) think that the existing image classification methods cannot make full use of the relationship between the single features of the image. Complementary advantages and characteristics, resulting in inaccurate classification. It uses principal component analysis to transform the extracted features, and uses the integrated classifier of support vector machine for classification. Simulation experiments show that multi-features have better image classification accuracy than single features. and faster classification speed.

Contents of the invention

The purpose of the present invention is to overcome the problems of low classification accuracy and low efficiency of existing single feature descriptors and shallow structure classification algorithms, and provide a deep belief network image classification protocol with high classification accuracy and high efficiency.

In order to solve the above technical problems, the technical solution of the present invention is to provide a deep belief network image classification protocol based on the underlying fusion features, characterized in that the steps are:

Step 1: Extract the color, texture and shape features in the sample image to form a weight matrix for multi-feature fusion;

Step 2: performing normalization processing on the weight matrix;

Step 3: Use the normalized weight matrix as the original data to train and test the deep belief network to obtain the classification result.

Preferably, the deep belief network is a deep structure composed of a plurality of restricted Boltzmann machine models RBM connected together and a BP neural network; each layer of RBM is connected, and the output of the previous layer of RBM is used as the rear The input of one layer of RBM and the output of the last layer of RBM are used as the input of the BP neural network to form the entire deep confidence network; the original data is input into the first layer of RBM, and the output of the BP neural network is the classification result.

Preferably, the RBM performs layer-by-layer feature extraction on the original data, from specific to abstract, so that the input obtained by the neural network becomes a feature vector that is easier to classify. At the same time, the deep structure composed of multi-layer RBM makes the process of feature extraction The error or redundant information in the neural network is weakened layer by layer, and finally the model reaches the overall optimum in the reverse adjustment process of the BP neural network.

Preferably, in step 3, 80% to 90% of the original data are randomly selected as the training set, and the rest are used as the test set.

Preferably, the training set is used to train the deep confidence network to obtain the weight and bias parameters of the deep confidence network; the depth confidence network determined by the parameters obtained in the training process is used to test the test set, and the error evaluation is performed to obtain classification results.

The protocol provided by the invention is based on the underlying image features and the deep belief network DBN, focusing on the underlying characteristics of the image. On the premise of detailed and comprehensive analysis of the underlying characteristics of the image, the color, texture and shape features in the sample image are extracted to form a multi-feature fusion The weight matrix and the feature matrix are normalized, and the four-layer DBN classifier constructed is used for training and classification. Using the Corel library and testing by training weights, the actual implementation effect of the algorithm shows that the average classification accuracy rate of the algorithm is much higher than that of the classification algorithm using a single feature and other mainstream classification algorithms.

Compared with the prior art, the deep belief network image classification protocol based on the underlying fusion features provided by the present invention has the following beneficial effects:

1. Good scalability. The agreement does not set an upper limit on the number of images recognized at the same time. In theory, as long as the comprehensive characteristics of each image are properly selected and fused, they can be recognized. Therefore, the scalability of the protocol is better.

2. The fusion of multiple features. Multi-feature fusion can improve the problem that a single feature distinguishes an image and easily causes the loss of the salient features of the image, thereby reducing the classification accuracy.

3. DBN classifier. Deep learning combines low-level features to form more abstract high-level representations or features to discover the feature representation of the data distribution and improve the efficiency of classification.

Description of drawings

Figure 1 is a schematic diagram of the RBM network structure;

Fig. 2 is the structural representation of DBN classifier;

Fig. 3 is a flow chart of a deep belief network image classification protocol based on underlying fusion features;

Figure 4 is a Corel diagram;

Figure 5 is a schematic diagram of the classification accuracy of each group in the experiment;

Figure 6 shows the misclassification rates of 10 types of images.

detailed description

Below in conjunction with specific embodiment, further illustrate the present invention. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

The present invention provides a deep belief network image classification protocol IBFCP based on underlying fusion features. The IBFCP is based on multiple features, so that it can provide a higher degree of recognition, and the higher the degree of recognition, the easier the classification.

1. Feature expression

The extraction and expression of image features is the basis of image classification technology. Generally speaking, the features of content-based image retrieval are mainly visual features, mainly including color, texture, and shape features.

1) Color characteristics

Color feature is the most important and widely used visual feature in content-based image retrieval, mainly because it is easy to extract, has the advantages of rotation invariance, scale invariance, and translation invariance, and is not sensitive to changes in viewing angles. . At present, the color features that are widely used mainly include color histogram, color moment (first-order moment, second-order moment and third-order moment), color correlation map, color information entropy, etc. These features can be extracted in different color spaces (such as RGB, HSV space, etc.).

2) Texture features

Texture feature is a visual feature that does not depend on color or brightness and reflects homogeneous phenomena in images. Texture features contain important information about the organization and arrangement of the surface structure of objects and their connection with the surrounding environment. Texture features have been widely used in content-based image classification, and users can classify images by texture feature similarity.

The texture features commonly used in image classification mainly include Tamura texture features, autoregressive texture models, directional features, wavelet transforms, and co-occurrence matrices.

3) Shape features

The shape of objects and regions is another important feature in image classification and retrieval. It is different from low-level features such as color or texture, and the expression of shape features is based on the division of objects or regions in the image. Since current techniques cannot achieve accurate and robust automatic image segmentation, shape features in image classification can only be applied together with other features. On the other hand, since people are not subjectively sensitive to the transformation, rotation and scaling of object shapes, suitable shape features must be independent of transformation, rotation and scaling, which also brings difficulty to the calculation of shape similarity.

The shape features used in image classification mainly include Hu invariant moments, edge orientation histograms, Fourier descriptors, Z moments, and orientation gradient histograms.

2. Deep Belief Network DBN

Deep learning forms a more abstract high-level representation or feature by combining low-level features to discover the feature representation of the data distribution. DBN is an extension of the deep architecture, which is extended from the RBM model. DBN is a probabilistic generative model containing multiple hidden layers (the number of hidden layers is greater than 2), and can effectively represent and train nonlinear data. The current layer captures highly correlated associations from the hidden units of the previous layer, building a joint distribution between observations and labels. The core idea of DBN is to extract and abstract the input data from the bottom up to each layer of restricted Boltzmann machine, and retain important information as much as possible.

1) Restricted Boltzmann machine model RBM

RBM is an energy model of unsupervised learning, which includes two layers of structure (input layer) and hidden layer (output layer), a random neural network model with symmetrical connection and no self-feedback, fully connected between layers, and no connection within the layer . If the display units are divided into 2 classes (only 0 or 1), the RBM can be represented by a joint probability distribution.

The RBM network structure is shown in Figure 1, where v is the visible layer, which is used to represent the observation data; h is the hidden layer, which can be regarded as some feature extractors; W is the connection weight between the two layers. The hidden layer unit and the display layer unit of RBM can be any exponential family unit (that is, given the hidden layer unit/display layer unit, the distribution of the display layer unit/hidden layer unit can be any exponential family distribution), such as softmax unit, Gaussian units, Poisson units, etc.

In Figure 1, the RBM network structure has m visible layer nodes and n hidden layer nodes, m and n are both positive integers, and each visible layer node is only related to n hidden layer nodes, and is independent of other visible layer nodes Yes, the state of the visible layer node is only affected by n hidden layer nodes, and each hidden layer node is also only affected by n visible layer nodes. This feature makes the training of RBM easier.

In 2002, Hinton proposed the contrastive divergence (Constrastive Divergence-CD) algorithm, and then improved it, and introduced the CD algorithm into the RBM model in 2006, which solved the difficulty of obtaining the expectation of the joint distribution in the RBM model The problem of precise value improves the effect and efficiency of training. Because of its convenience, ease of use, and high flexibility, it is widely used in feature extraction, classification, noise reduction, and dimensionality reduction. The RBM model consists of a visible layer v and a hidden layer h. The retrieved information exchanged by users is converted into the visible layer v, and the visible layer and the hidden layer are connected through a symmetrical weight layer W. The energy function defined by RBM is:

Among them, θ={w _mn , b _m , c _n } are the parameters of RBM, all of which are real numbers; w _mn represents the connection weight of n between the visible layer unit m and the hidden layer unit; b _m represents the bias of the visible layer unit m set; c _n represents the bias of hidden layer unit n. When the parameters are determined, based on the energy function of formula (1), the joint probability distribution of v and h can be obtained:

Among them, Z(θ) is the normalization factor (also called partition function):

The probability distribution P(v; θ) for the observed data v corresponds to the marginal distribution of P(v, h; θ), also known as the likelihood function. The marginal distribution (joint distribution) of the corresponding data can be defined as:

Similarly, there are:

2) Deep Belief Network Model

What DBN needs to learn during the training process is the joint probability distribution, and in the field of machine learning, the meaning represented by the joint probability is the generative model of the object. In 2006, Hinton proposed the DBN model, which is a deep structure composed of multiple RBM models overlapping each other and a BP neural network. The training process mainly includes two aspects: (1) Using RBM structure training, screening data Feature information; (2) Connect each layer of RBM, pass the network in the last layer, use the RBM output as the input of the BP neural network, and use the data for supervised training to form the entire deep structure. DBN extracts features layer by layer from the original input, from concrete to abstract, so that the input obtained by the neural network becomes a feature vector that is easier to classify. The remaining information is weakened layer by layer, and finally the model reaches the overall optimum in the reverse adjustment process of the BP neural network. Compared with the traditional neural network, the advantage of the deep structure of DBN is that it overcomes the shortcomings of the traditional neural network, such as long training time, easy to fall into local optimum, and slow processing of large data when the deep structure increases. A DBN can be thought of as a neural network with trained initial weights. The existing work has proved the following three rules: (1) the number of top-level units exceeds the threshold, and the accuracy remains stable at a certain level; (2) the number of layers increases, and the computing performance tends to decline; (3) RBM training increases with iteration As the number of times increases, the performance also improves accordingly.

In a deep trust network composed of m RBMs, the nth (n<m) RBM model starts after the n-1 RBM model is trained, and the input of P(h _n ; h _n-1 , w _n ) is the output P(h _n-1 ; h _n-2 , w _n-1 ) from the n-1th RBM model. At the same time, its output P(h _n+1 ; h _n , w _n+1 ) is It constitutes the input of the n+1th RBM model, and n is a positive integer greater than 1. Hinton believes that for a typical DBN with 1 hidden layer, the relationship between the visible data v and the hidden vector h can be expressed in probability as follows:

Among them, h ¹ , h ² , h ¹ , h ^k are all implicit vectors;

3. DBN classification method based on multi-feature fusion

1) Fusion of multiple features

For complex images, it is generally difficult for a feature to have sufficient recognition. Obviously, a combination of multiple features can provide a higher degree of recognition, and the higher the degree of recognition, the easier it is to classify.

Some of the salient features of a class of images are expressed in local feature points, some are expressed in color features, and some are expressed in texture features or shape features. Using a single feature to classify all images can easily cause the loss of salient features of a class of scene images and reduce the classification accuracy. Not only that, but the salient features of different images of the same type of scene are also different. If only one feature is used to classify images, it is easy to lose the salient features of a single image, resulting in a decrease in classification accuracy. Multi-feature fusion can improve this situation and further improve the classification accuracy.

Aiming at the characteristics of complex objects in color images, this application extracts three types of features of color, texture and shape for each image, including 9 color moments, 6 tamura features, 20 gray-level co-occurrence matrices, and 7 Hu different Variable moments, 16 edge histograms, 48 features in total. Then, the algorithm of multi-feature fusion is used to combine features in image classification, so as to avoid the problem of single feature and improve the classification accuracy.

2) Build a DBN classifier

DBN adopts a deep structure composed of four RBMs, and its structure is 48-90-90-90-10. The first layer of RBM regards the input as the display layer, with a total of 48 nodes, corresponding to 48 features of the image, and the hidden layer (output layer) of the RBM is used as the display layer of the second layer of RBM (a total of 90 nodes); the second layer The hidden layer (output layer) of the RBM is used as the visible layer of the third layer RBM (90 nodes in total); the hidden layer (output layer) of the third layer RBM is used as the visible layer of the fourth layer (90 nodes in total); the fourth layer The hidden layer (output layer) of the layer RBM will be the output of the DBN, which includes 10 units, that is, the image is divided into 10 categories. The 4th layer adds the sigma function as the final result output layer.

The formula for the Sigma function is:

The structure of the DBN classifier is shown in Figure 2.

3) IBFCP protocol implementation process

The deep belief network image classification protocol based on the underlying fusion features uses the Corel 1K database, randomly selects 90% of it as the training set, and the remaining 10% as the test set. The algorithm flow is shown in Figure 3.

Specific steps are as follows:

(1) Feature expression and fusion: 3 types of feature information of color, texture and shape are extracted for each image, a total of 48 features are formed to form a 48-dimensional feature vector, and a feature set of 100048 is formed for 1000 images.

(2) Normalization processing: In order to make the subsequent actual operation more accurate and ensure the scale consistency of each data, the feature vector must be normalized, and all the data after normalization are in [0, 1] Between, its normalization formula is:

X _i , X _min and X _max are the smallest and largest eigenvectors in the feature set, respectively;

(3) Data classification: randomly select 900 (90%) from the feature set as the training set, and the remaining 100 (10%) as the test set.

(4) Training process: A 4-layer DBN structure is used for training. Proposed by Hinton GE in "Training Products of Experts by Minimizing Contrastive Divergence" (Hinton G, Osindero S, Teh Y. Afast learning algorithm for deep belief nets [J]. Neural Computation, 2006, 18(7): 1527-1554) The fast learning algorithm of contrastive divergence is used for learning.

(5) Test process: use the weight and bias obtained in the DBN training process to test the test set,

According to the distribution of RBM, the difference between the samples obtained after Gibbs sampling and the original data is evaluated for error, and the classification result is obtained.

(6) Experiment and result analysis

In order to verify the above algorithm, the software simulation environment of this experiment is Matlab2013a installed under Win8.1, and the computer hardware configuration is Intel(R) Core(TM) 2Duo E8400, 3.0GHz CPU, 4GB memory, 320GB hard disk.

a.Experimental data

Corel Image Library is one of the commonly used image classification and image retrieval libraries. It has 2 types, namely Corel10K and Corel 1K. The images are all 256x384 px or 384x256 px jpg images. Corel 10K includes 10,000 images, there are 100 categories of images, and each category has 100 images. Corel 1K has 10 types of diagrams, 100 diagrams for each type.

For the same literature (Hinton G E. Training Products of Experts by Minimizing Contrastive Divergence [J], Neural computation, 2002, 14 (8): 1771-1800. and Rao M B, Kavitieval C H.A New Feature Set for Content Based Image Retrieval [J] ].Information Communication and Embedded Systems, 2013, 1(1): 84-89.) for comparison, this paper uses the same image library as Corel 1K library. The 10 categories are flowers, horses, dinosaurs, elephants, buildings, beaches, buses, people, food, and mountains. The categories are 1 to 10, with 100 images for each category, and a total of 1000 images. Figure 4 shows these 10 categories, one for each category.

b. Data packet

The entire image database is divided into two parts, one part is used as a training set, and the other part is used as a test set. The training set is 90% of the total number of samples; the test set is 10% of the total number of samples. The classification process adopts random classification. The random classification results are shown in Table 1.

Table 1 Random classification results

c. Experimental results

Each time, 9 groups are selected as the training set, and the other group is used as the test set to obtain a set of results. Perform 10 times to ensure that each sample can be used as a test set for experiments. Through 10 experiments, the correct rate of 10 groups of experiments is obtained.

Figure 5 shows the classification accuracy of each group in the 10 experiments. From the classification accuracy rate of each group, the accuracy rate of the 10th group is the last, which is 92%. Group 7 had the lowest correct rate of 79%, with an average correct rate of 85.1%.

According to the statistical results of 10 experiments, the misclassification of each type of image is calculated, as shown in Table 2. Each row in Table 2 represents the classification of a class of images (100 in total), a _ij (i=1, 2,..., 10; production 1, 2,..., 10) indicates the classification of the i-th class of images Time is divided into the number of the jth class. The total in column j indicates the number of 1000 images classified into class j (should be 100 images per class). The last column indicates the classification accuracy for such images. It can be seen from Table 2 that among the 10 categories of images, the classification accuracy rate of each category is different, and the classification accuracy rate of the group of dinosaurs is the highest, which is 100%, and all of them are classified correctly. There are 4 categories of "people", "beach", "building" and "elephant" with a correct rate lower than 80%.

Table 2 Experimental classification results

Figure 6 shows the misclassification rates for 10 categories of images. The misclassification rate is the number of images misclassified into this class divided by the total number of images classified into the cost class. For example, for each "person", the number of misclassified images into this class is 17, and the total number of images classified into this class is 89. The fraction is 17/89×100%=19.1%. The misclassification rates of the 10 categories of images are shown in Figure 6. It can be seen from Figure 6 that the misclassification rates of the three types of images of "building", "elephant" and "mountain" are relatively high, all exceeding 20%. The 3 classes of "car", "dinosaur" and "flower" have a lower misclassification rate.

d. Method comparison

1) Single feature and algorithm result of the present invention

Table 3 lists the classification results of common features, mainly including grayscale histogram, color histogram, grayscale co-occurrence matrix, color co-occurrence matrix and the results of the algorithm of the present invention. Among them, the classification size of the first 5 methods is 16.

Table 3 Comparison of the classification accuracy rate of a single feature and the algorithm in this paper

It can be seen from Table 3 that the average classification accuracy rate of a single feature is not more than 70%, while the result of the feature fusion algorithm of the present invention reaches 85.1%, and the classification effect is better.

The present invention proposes a new DBN image classification algorithm, which first extracts general features such as color, texture, and shape from the original image, and then uses these features as original data to carry out deep belief network training. Through the multi-feature fusion of color, texture and shape, it solves the problem of low classification rate of single feature and existing algorithms, and uses 4-layer DBN network for training to overcome single feature and support vector machine SVM, Boosting and other shallow structure algorithms The disadvantage of poor classification effect, but also avoid the slow speed of direct training from the pixel level. The algorithm has an average classification accuracy rate of nearly 86%, which is higher than the classification algorithm using a single feature and other mainstream classification algorithms.

Claims (5)

1. a kind of depth confidence network image bracket protocol based on bottom fusion feature, it is characterised in that step is：

Step 1：Color, texture and shape facility in extraction sample image, constitute the weight matrix of multiple features fusion；

Step 2：The weight matrix is normalized；

Step 3：Depth confidence network is trained and tested as initial data by the use of the weight matrix after normalized, Draw classification results.

2. a kind of depth confidence network image bracket protocol based on bottom fusion feature as claimed in claim 1, its feature It is：The depth confidence network is the structure and a BP god linked together by the limited Boltzmann machine model RBM of multiple Through the depth structure that network is constituted；By each layer RBM connections, the output of preceding layer RBM as later layer RBM input, last Layer RBM outputs constitute entire depth confidence network as the input of BP neural network；The initial data is input into ground floor RBM, The output of BP neural network is classification results.

3. a kind of depth confidence network image bracket protocol based on bottom fusion feature as claimed in claim 2, its feature It is：Initial data is carried out the RBM feature extraction successively, from specific to abstract so that the input that neutral net is obtained As a characteristic vector more easily classified, meanwhile, the depth structure of multilayer RBM compositions is caused in characteristic extraction procedure Mistake or redundancy successively weakened, and finally model is reached entirety during the reverse adjustment of BP neural network It is optimal.

4. a kind of depth confidence network image classification based on bottom fusion feature as described in any one of claims 1 to 3 is assisted View, it is characterised in that：In the step 3, from initial data, 80%~90% is randomly choosed as training set, remaining conduct Test set.

5. a kind of depth confidence network image bracket protocol based on bottom fusion feature as claimed in claim 4, its feature It is：Depth confidence network is trained using the training set, obtains the weight and offset parameter of depth confidence network；Adopt The depth confidence network of the parameter determination obtained with training process is tested test set, and carries out error evaluation, is drawn point Class result.