CN110110845B - Learning method based on parallel multi-level width neural network - Google Patents

Abstract

The present invention discloses a learning method based on a parallel multi-level wide neural network, comprising the following steps: obtaining a verification set and constructing a base classifier; training and verifying each level of the parallel M-level wide neural network to obtain the trained parallel M-level wide neural network and the verification output corresponding to each level of the wide neural network; obtaining the decision threshold of each level of the wide neural network by statistical calculation; and testing the verified parallel multi-level wide neural network by a test set. The neural network of the present invention has a multi-level structure, each level learns for different parts of the data, and can achieve parallel training and testing. Each level uses a wide neural network to learn features in the width direction; multiple wide neural networks are reconnected as base classifiers in the width direction to achieve classifier integration in two width directions; incremental learning of the network is achieved by adding a new level of wide neural network; and parallel testing can be achieved.

Description

A learning method based on parallel multi-level wide neural network

technical field

The invention belongs to the technical field of artificial intelligence and machine learning, and in particular relates to a learning method based on a parallel multi-level wide neural network.

Background technique

With the great success of deep learning network-based learning models in the fields of large-scale image processing and machine vision, the complexity of learning models has also increased rapidly. They require a large amount of high-dimensional data to train, which greatly increases the demand for computing resources and computing time. In addition, the actual data is often not homogeneous, some samples are very easy to classify, but there are also many samples that are difficult to classify. Most classification errors occur when the input samples are difficult to classify, such as unbalanced distribution of samples, abnormally obtained samples, and samples that are close to the classification boundary or linearly inseparable.

In the existing deep learning model, simple samples and complex samples are processed in the same way, which reduces the efficiency of computing resources. At the same time, existing deep learning networks such as convolutional neural networks often have many layers, and all samples have to go through all network layers, which is very time-consuming when generalizing or testing the network. However, the early parallel multi-level self-organizing network only receives nonlinearly transformed samples rejected by the previous level at each level, and these samples are transformed to other spaces that are easy to classify, so as to classify again. However, the problem of how to adjust and allocate computing resources for high-dimensional data for data samples of different difficulties to improve the speed and efficiency of learning and classification has not been well resolved.

SUMMARY OF THE INVENTION

In view of the above defects, the present invention provides a learning method based on a parallel multi-level wide neural network. The neural network of the present invention has a multi-level structure, each level learns for different parts of the data, and can realize parallel training and test. Each level uses a wide neural network for feature learning in the width direction; through the reconnection of multiple wide neural networks as the base classifiers in the width direction, the integration of two classifiers in the width direction is realized; by adding a new level of The breadth neural network realizes the incremental learning of the network; and can realize parallel testing, which greatly shortens the learning and classification time of complex samples and improves the efficiency of network operation.

In order to achieve the above object, the present invention adopts the following technical solutions to solve it.

(2) A learning method based on a parallel multi-level wide neural network, the parallel multi-level wide neural network includes a multi-level wide neural network, wherein each level of the wide neural network includes an input layer, a hidden layer, a decision layer and an output layer connected in sequence layer, the decision layer is used to determine whether each test sample is output by the current stage, and the learning method includes the following steps:

Step 1, obtain the original training sample set, build a parallel M-level width neural network Net ₁ ,...Net _m ,...,Net _M (m=1, 2..., M), each level of width neural network is used as the base classifier of the corresponding level ; By performing M data transformations on the original training sample set, M corresponding verification sets x _{v_1} , ... x _{v_m} , ... x _{v_M are obtained} ;

Among them, the total number of samples in the original training sample set is N _tr .

Step 2, using the original training sample set and M verification sets x _{v_1} , ... x _{v_m} , ... x _{v_M} to train and verify each level of the parallel M-level width neural network respectively, and obtain the parallel M-level width neural network after training and The verification output y _{v_m} (m=1, 2..., M) corresponding to each level of width neural network; the minimum error method is used to obtain the label y _{v_ind_m} corresponding to each verification output y _{v_m} , and then the parallel M-level width neural network after training is obtained The correct classification sample set y _{vc_m} and the wrong classification sample set y _{vw_m} of the validation set of the neural network of each level width;

Step 3: Statistical calculation is respectively performed on the correct classification sample set y _{vc_m} and the wrong classification sample set y _{vw_m} of the verification set of each level width neural network of the parallel M-level width neural network after training, corresponding to each level width neural network after training. The decision threshold T _m of the network; the decision threshold T _m of the width neural network of each level is used as the decision basis of the corresponding level width neural network, and the parallel M level width neural network determined by the decision threshold is obtained;

Step 4: Obtain a test set, and take the test set as the input data of the parallel M-level width neural network determined by the decision threshold and input it to the neural network of each level determined by the decision threshold for testing, and obtain the width of each level determined by the decision threshold. Output: Obtain the error vector of the neural network with the width of each level, and judge the output of the neural network with the width of each level determined by the decision threshold, so as to obtain the label y _{test_ind_m} corresponding to the test output of the neural network with the width of each level determined by the decision threshold.

The characteristics and further improvement of the technical solution of the present invention are:

(1) In step 1, the data is transformed to compress or deform the samples in the original sample set through elastic transformation (Elastic); or the data transformation is to rotate the samples in the original sample set through affine transformation (Affine). , flip, zoom in or zoom out.

(2) In step 2, the original training sample set and M verification sets x _{v_1} , ... x _{v_m} , ... x _{v_M} are used to train and verify each level of the parallel M-level width neural network respectively, which includes the following sub-steps :

In sub-step 2.1, the original training sample set is used as the input sample of the first-level wide neural network Net ₁ , and the first-level wide neural network Net ₁ is trained to obtain the trained first-level wide neural network.

In sub-step 2.2, the first validation set x _{v_1} is used to validate the first-level width neural network after training, and a misclassified sample set y _{vw_1} of the validation set of the first-level width neural network is obtained.

Sub-step 2.3, use the misclassified sample set y _{vw_1} of the first-level width neural network as the input sample A _{v_1} of the second-level width neural network; then randomly select the training sample set A _{v_2} from the original training sample set, so that the total input sample set The number of samples in {A _{v_1} +A _{v_2} } is equal to the number of samples in the original training sample set, and the total input sample set {A _{v_1} +A _{v_2} } is used as the input sample of the 2-stage wide neural network.

Sub-step 2.4, use the total input sample set {A _{v_1} +A _{v_2} } to train the second-level width neural network to obtain the second-level width neural network after training; use the second validation set x _{v_2} to train the second-level width neural network after training. The second-level width neural network is used for verification, and the misclassified sample set y _{vw_2} of the verification set of the second-level width neural network is obtained.

By analogy, the third-level to M-th level width neural networks are trained respectively, and the trained parallel M-level width neural network and the corresponding verification output y _{v_m} of each level width neural network (m=1, 2..., M) are obtained. .

(3) In step 2, the minimum error method is:

First, set the total number of categories of the original training sample set as C, and construct a reference matrix R _j (1≤j≤C).

Wherein, the elements of the jth row of the reference matrix R _j are all 1, the other elements are all 0, and the dimension of each reference matrix R _j is C×N _tr .

Secondly, according to the verification output y _{v_m} of the neural network of each stage width after training, the error vector between the verification output y _{v_m} and the reference matrix R _j of the corresponding stage is obtained:

J _{v_mj} =||softmax(y _{v_m} )-R _j || ₂ , 1≤j≤C;

The dimension of J _{v_mj} is 1×N _tr ; the dimension of y _{v_m} is C×N _tr .

Finally, take the minimum value of the error vector J _{v_mj} between the verification output y _{v_m} and the reference matrix R _j of the corresponding level, and obtain the class label y _{v_ind_m} corresponding to the neural network of each level width after training:

Among them, the dimension of y _{v_ind_m} is 1×N _tr .

(4) In step 3, the statistical calculation includes the following substeps:

Sub-step 3.1, set the correct classification sample set and misclassified sample set of the mth-level width neural network of the trained parallel M-level width neural network are: y _{vc_m} and y _{vw_m} , the correct classification sample set and the wrong classification sample set The total number of samples are: N _{vc_m} and N _{vw_m} , and N _{vc_m} +N _{vw_m} =N _tr , then the errors of the correctly classified sample set and the wrongly classified sample set are:

e _{vc_m} =||softmax(y _{vc_m} )-t _{vc_m} || ₂ ;

e _{vw_m} =||softmax(y _{vw_m} )-t _{vw_m} || ₂ ;

Among them, t _{vc_m} is the true label corresponding to the correctly classified sample y _{vc_m} in the m-level width neural network, and t _{vw_m} is the true label corresponding to the misclassified sample y _{vw_m} in the m-level width neural network.

Sub-step 3.2, according to the correctly classified sample set y _{vc_m} and the wrongly classified sample set y _{vw_m} , calculate the mean and variance of the correctly classified sample set y _{vc_m} as μ _c and σ _c respectively; the mean and variance of the wrongly classified sample set y _{vw_m} are: u _w and σ _w ; then the Gaussian distributions corresponding to the correctly classified sample set y _{vc_m} and the wrongly classified sample set y _{vw_m} are:

The Gaussian probability density functions corresponding to the correctly classified sample set y _{vc_m} and the incorrectly classified sample set y _{vw_m} are:

In sub-step 3.3, according to the error e _{vw_m} and the variance σ _w of the misclassified sample set y _{vw_m} , the decision threshold T _m =min(e _{vw_m} )-ασ _w of the m-level width neural network is obtained.

where α is a constant to give a margin so that all misclassified samples y _{vw_m} are rejected at the current stage.

(5) In step 4, the acquisition of the test set is: acquiring the original test sample set x _test ; through M times of data expansion, correspondingly acquiring M groups of test sample sets x _{test_1} , . . . , x _{test_m} , . . , x _{test_M} , which is the test set.

Further, the data expansion is: perform N _testD times of the data transformation on each sample in the original test sample set x _test respectively, and correspondingly obtain N _testD test sample sets, as the parallel M level determined by the decision threshold. The test set x _{test_m} of the m-th-level wide neural network.

Among them, the total number of test samples in the original test sample set X _test is N _{test_saples} .

(6) In step 4, the described acquisition of the error vector of each level of width neural network includes the following sub-steps:

Sub-step 4.1, the M groups of test sample sets x _{test_1} , x _{test_2} , ..., x _{test_M} are respectively input in parallel to the parallel M-level width neural network determined by the decision threshold, corresponding to the N of each level width neural network determined by the decision threshold. _testD outputs y _{test_m_d} , (d=1, 2...N _testD ).

Sub-step 4.2, calculate the average value of the N _testD outputs y _{test_m_d} of the neural network with the width of each level determined by the decision threshold, (d=1, 2...N _testD ), and obtain the test output of the neural network with the width of each level determined by the decision threshold

Sub-step 4.3, set the total number of categories in the test set as C, construct a reference matrix R _j (1≤j≤C); obtain the error vector between the verification output y _{v_m} and the reference matrix R _j of the corresponding level:

J _{test_mj} =||softmax(y _{test_m} )-R _j || ₂ , 1≤j≤C;

Among them, the elements of the jth row of the reference matrix R _j are all 1, and the other elements are 0. The dimension of each reference matrix R _j is C×N _{test_samples} ; The dimension of J _{test_mj} is 1×N _{test_samples} , y _{v_m} The dimension of _{test_samples} is C×N.

(7) The output of each level width neural network determined by the decision threshold is judged as:

When the minimum error of the current stage width neural network is less than or equal to the current stage decision threshold, it is judged that the current stage is the correct classification output stage of the output:

min(J _{test_mj} )≦T _m .

When the minimum error of the current-level width neural network is greater than the current-level decision threshold, it is judged that the current level cannot correctly classify the output, and the output is transferred to the next-level width neural network for testing, and so on until the output is found to be correct Classification output stage:

min(J _{test_mj} )>T _m .

(8) In step 4, the label y _{test_ind_m} corresponding to the test output of the neural network of each level width determined by the obtained decision threshold is:

where the dimension of y _{test_ind_m} is 1×N _{test_samples} .

Compared with the prior art, the beneficial effects of the present invention are:

(1) The neural network of the present invention has a multi-level base classifier, and each level is used to learn different parts of the data set. According to the complexity of the problem and the data set, the structure of the neural network can be adaptively determined to realize computing resources. Optimization.

(2) The neural network of the present invention has the advantage of incremental learning. When new training data is available, the judgment of the current neural network is realized. According to the judgment result, it is determined whether the newly added training data can be correctly classified. Correct separation, new samples are learned by adding a new width radial basis function as a new stage of the neural network without retraining the entire network.

(3) The neural network of the present invention can be tested in parallel during the test, that is, the test data is given to all levels of the network at the same time, and the decision threshold of each level obtained during the training process determines where each test sample is ultimately One-level neural network output, parallel testing process greatly reduces the waiting time when actually using the network.

(4) The neural network of the present invention can be used as a general learning framework with strong flexibility, and each level of the neural network can use a BP neural network, a convolutional neural network or other types of classifiers according to actual needs.

Description of drawings

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

1 is a schematic diagram of a parallel multistage neural network of the present invention and a schematic diagram of a training and testing process; wherein, FIG. 1(a) is a schematic diagram of a parallel multistage width neural network of the present invention; FIG. 1(b) is a schematic diagram of the present invention Schematic diagram of the training and verification process of the parallel multi-level width neural network; Fig. 1(c) is the principle diagram of the testing process of the parallel multi-level width neural network of the present invention.

FIG. 2 is a structural diagram of the parallel multi-stage wide neural network of the present invention.

Fig. 3(a) is the error distribution diagram of the validation set of the parallel multi-stage width neural network of the present invention on one of the stages; Fig. 3(b) is the Gaussian probability density function of the statistical parameters in Fig. 3(a).

FIG. 4 is a comparison diagram of the test result of the parallel 26-level wide neural network on the MNIST data set in the embodiment of the present invention and the classification result of the existing learning model.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the examples, but those skilled in the art will understand that the following examples are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention.

Using the MNIST handwriting dataset, each image in this dataset is an 8-bit grayscale handwritten digit 0-9 image, the image size is 28×28, there are 10 categories in total, there are 60,000 original training sample sets, and 10,000 images are used as the test set , which is one of the important general image datasets for training and testing of new learning models. For this data set, referring to FIG. 1 and FIG. 2 , in this embodiment, the width radial basis function network is used as the base classifier, that is, each level of the parallel multi-level width neural network adopts the width radial basis function network, and the parallel width radial basis function network is selected. The number of stages of the neural network is 26.

(1) Obtain the validation set and build the base classifier.

First, perform 26 elastic transformations on the image samples in the original training sample set of N _tr = 60000 to obtain M = 26 verification sets x _{v_1} , x _{v_2} , . . . , x _{v_26} . In this embodiment, in order to ensure sufficient There are many misclassified samples of the validation set, and each validation set includes N _val = 10 data sets transformed from the original training set. Among them, the number of samples in each validation set is N _val =10 times of the original training sample set.

Secondly, a wide radial basis function network is used as the base classifier to design a parallel multi-level wide neural network; M=26 wide radial basis function networks are connected together to form a parallel multi-level wide neural network Net ₁ , Net ₂ , . ..Net _M ; each base classifier acts as a stage, focusing on a different part of the dataset.

Finally, construct the width radial basis function network. The specific process is as follows:

的径向基函数网络，该径向基函数网络的中心为随机取自原始训练样本集的一个子集，标准差取值为常数。采用滑动窗口获取原始训练样本集中每个图像样本的多组局部特征图像，从而获得多组局部特征矩阵，将多组局部特征矩阵作为高斯基函数的输入数据，得到多个径向基函数网络即为宽度径向基函数网络。The construction includes N _0k = 1000 Gaussian base functions as

The radial basis function network of , the center of the radial basis function network is randomly selected from a subset of the original training sample set, and the standard deviation is constant. The sliding window is used to obtain multiple sets of local feature images of each image sample in the original training sample set, thereby obtaining multiple sets of local feature matrices, and using multiple sets of local feature matrices as the input data of the Gaussian basis function to obtain multiple radial basis function networks. is the width radial basis function network.

2) Train and verify each stage of the parallel M-level width neural network, and obtain the parallel M-level width neural network after training and the verification output y _{v_m} (m=1, 2..., M) corresponding to each level width neural network.

The first-level BRF network is trained using the original training sample set. After training, the misclassified training samples are sent to the second-level BRF network as part of the second training set for training. Level 2 network. The verification set obtained in step (1) is used to verify the training network of the current level, and at the same time, more misclassified samples are provided as part of the training set of the next level. As shown in Figure 1 (a) and (b), specifically, the following sub-steps are included:

In sub-step 2.1, the original training sample set is used as the input sample of the first-level wide neural network Net ₁ , and the first-level wide neural network Net ₁ is trained to obtain the first-level wide neural network after training.

In sub-step 2.2, the first verification set x _{v_1} is used to verify the trained first-level wide neural network, and a misclassified sample set y _{vw_1} of the first-level wide neural network is obtained.

Sub-step 2.3, use the misclassified sample set y _{vw_1} of the first-level width neural network as the input sample A _{v_1} of the second-level width neural network; then randomly select the training sample set A _{v_2} from the original training sample set, so that the total input sample set The number of samples in {A _{v_1} +A _{v_2} } is equal to the number of samples in the original training sample set, and the total input sample set {A _{v_1} +A _{v_2} } is used as the input sample of the 2-stage wide neural network.

Sub-step 2.4, use the total input sample set {A _{v_1} +A _{v_2} } to train the second-level width neural network to obtain the second-level width neural network after training; use the second validation set x _{v_2} to train the second-level width neural network after training. The class width neural network is used for verification, and the misclassified sample set y _{vw_2} of the second class width neural network is obtained.

Repeat sub-steps 2.3 and 2.4 to train the 3rd to Mth-level width neural networks respectively, and obtain the parallel M-level width neural network after training and the corresponding verification output y _{v_m} of each level width neural network (m=1, 2 ..., M).

The specific training and verification process of the above-mentioned width radial basis function network is as follows:

的径向基函数网络，输出记为：The image samples in the original training sample set are used as input data, and the image size is M ₁ ×M ₂ =28×28. The size of the sliding window is r=13×13, the initial position of the sliding window is set at the upper left corner of each image sample, the sliding step is selected as 1 pixel, the sliding window slides from left to right, and from top to bottom, and the The 3-dimensional image block of 60000 image samples in the sliding window is stretched into a matrix x _k ∈ R ^r×N , that is, each local feature image is formed into a corresponding original matrix by pixel, and the second to The last column is sequentially arranged to the first column to form a column vector; N column vectors are sequentially arranged to form a local feature matrix x _k (1≤k≤K) of a set of training image samples, each column of the local feature matrix x _k represents a sample. Then input the local feature matrix x _k to include N _0k =1000 Gaussian base functions as

The radial basis function network of , the output is recorded as:

为包含N＝60000个元素的列向量。in,

is a column vector containing N=60000 elements.

Each sliding of the sliding window corresponds to a radial basis function network. After the final sliding, K=(M ₁ -m+1)(M ₂ -m+1)=(28-13+1)×(28- 13+1)=256 radial basis function networks.

采用索引s_k将降序向量a_k中每个待处理图像的局部特定位置对应的原始位置进行标记，得到排序的输出数据Φ′_k＝sort(Φ_k，s_k)。For each radial basis function network, sorting and downsampling are introduced to the output of the Gaussian basis function. For each radial basis function network, sorting and down-sampling are introduced to its Gaussian basis function output data _Φk after nonlinear transformation. Sum up each column of the output data Φk of the width radial basis function network to obtain a row _vector , each element of the row vector is the sum of the pixels at the local specific position of each image to be processed, for each to-be-processed image. The sum of the pixels in the local specific position of the processed image is sorted in descending order to obtain a descending vector

The index _sk is used to mark the original position corresponding to the local specific position of each image to be processed in the descending vector _ak to obtain sorted output data Φ′ _k =sort(Φ _k , _sk ).

Downsampling the sorted output data, setting the downsampling interval N _kS =20, and the number of sampled outputs is:

采样输出为Φ_ks＝subsample(Φ′_k，N_kS)，则高斯基函数的输出为Φ＝[Φ_1S，Φ_2S，…，Φ_KS]。Then the output number of the total width radial basis function network is

The sampling output is Φ _ks =subsample(Φ′ _k , N _kS ), then the output of the Gaussian basis function is Φ=[Φ _1S , Φ _2S , . . . , Φ _KS ].

Set the desired output as D=[D ₁ , D ₂ , ..., D _C ]; perform linear layer connection on the output of the Gaussian basis function of the width radial basis function network, then the weight of the linear layer is: W=[W ₁ , W ₂ , ..., W _C ];

where C=10 is the total number of categories of the original sample.

具体公式为：Obtain the class output Y=[Y ₁ , Y ₂ , _.

The specific formula is:

Compute the least-mean-square estimate of the weights of the linear layers via the pseudo-inverse matrix of the Gaussian basis function output Φ of the wide radial basis function network

where Φ ⁺ is the pseudo-inverse matrix of the Gaussian basis function output Φ of the width radial basis function network.

Finally, the category output of the width radial basis function network is calculated as:

Then obtain the width radial basis function network after training, use the corresponding validation set to verify the width radial basis function network after each level of training, and obtain the validation output y _{v_m} corresponding to the width radial basis function network of each level after training ( m=1,2...,M).

Through the obtained verification output y _{v_m} (m=1, 2..., M), the category label y _{v_ind_m} corresponding to each verification output y _{v_m} is further obtained, and the specific steps are as follows:

First, set the total number of categories of the original training sample set as C, and construct a reference matrix R _j (1≤j≤C).

Wherein, the elements of the jth row of the reference matrix R _j are all 1, the other elements are all 0, and the dimension of each reference matrix R _j is C×N _tr .

Secondly, according to the verification output y _{v_m} of the neural network of each stage width after training, the error vector between the verification output y _{v_m} and the reference matrix R _j of the corresponding stage is obtained:

J _{v_mj} =||softmax(y _{v_m} )-R _j || ₂ , 1≤j≤C;

The dimension of J _{v_mj} is 1×N _tr ; the dimension of y _{v_m} is C×N _tr .

Finally, take the minimum value of the error vector J _{v_mj} between the verification output y _{v_m} and the reference matrix R _j of the corresponding level, and obtain the class label y _{v_ind_m} corresponding to the neural network of each level width after training:

Among them, the dimension of y _{v_ind_m} is 1×N _tr .

Comparing the class label y _{v_ind_m} corresponding to the width neural network of each level after training with the verification output y _{v_m} of each level, the correct classification sample set y _{vc_m} and the wrong classification sample set y _{vw_m} of each level width neural network can be obtained.

(3) The decision threshold T _m of the width neural network of each stage is obtained by statistical calculation

The difficult part of this network is the determination of the decision threshold of each level, which is used to determine which level of network output each sample should be during testing. After training and testing, statistical calculations are performed on the correctly classified sample set and the misclassified sample set, respectively. Assuming that in the m-level width neural network, the correct classification sample set and the wrong classification sample set are: y _{vc_m} and y _{vw_m} respectively, the total number of samples of the correct classification sample set and the wrong classification sample set are: N _{vc_m} and N _{vw_m} , and N _{vc_m} +N _{vw_m} =N _tr .

In the above verification process, in order to ensure that there are enough misclassified sample sets in the final sample, each verification set can contain N _val verification sample sets obtained by transforming the original training sample set through data, that is, each verification set can contain N _val sets of validation samples, that is to say, the number of samples in each validation set is N _val times the original training samples.

The error of the two types of sample sets is calculated by the following formula:

e _{vc_m} =||softmax(y _{vc_m} )-t _{vc_m} || ₂ ;

e _{vw_m} =||softmax(y _{vw_m} )-t _{vw_m} || ₂ ;

Among them, t _{vc_m} and t _{vw_m} are the true labels corresponding to the correctly classified samples y _{vc_m} and the misclassified samples y _{vw_m} in the m level. Assuming that the mean and variance of the two types of sample statistics for correct classification and misclassification are: μ _c , u _w , σ _c , σ _w , the corresponding two Gaussian distributions are:

The Gaussian probability density functions are:

At the first level of the parallel multi-level wide neural network, its validation set error distribution and its probability density function are shown in Figure 3(a) and (b), then the decision threshold of the m-level wide neural network is:

T _m =min(e _{vw_m} )-ασ _w ;

where α is a constant to give a margin so that all misclassified samples y _{vw_m} are rejected at the current stage.

(4) Test the parallel multi-level wide neural network determined by the decision threshold through the test set

As shown in Figure 1(c), the specific test process is:

First, obtain the test set, the specific process is: obtain the original test sample set X _test ; through M data expansion, correspondingly obtain M groups of test sample sets x _{test_1} , . . . , x _{test_m} , . . . , x _{test_M} , namely Test set; among them, the total number of test samples in the original test sample set x _test is N _{test_samples} .

The above-mentioned data expansion is: perform N _testD data transformations on each sample in the original test sample set X _test respectively, and correspondingly obtain N _testD test sample sets, which are used as the first step of the parallel M-level width neural network determined by the decision threshold. A test set x _{test_m} for a class-m wide neural network.

The above test set acquisition method can obtain the stability of the test in the subsequent test process.

Secondly, the M groups of test sample sets x _{test_1} , ..., x _{test_m} , ..., x _{test_M are} input in parallel to the parallel M-level width neural network determined by the decision threshold, and the test set is tested, that is, each test set corresponds to Input the input to the neural network with the width of each level determined by the decision threshold for testing, and correspondingly obtain the output of N _testD test sample sets of the neural network with the width of each level determined by the decision threshold; take the average of the outputs of the N _testD test sample sets to obtain the decision threshold. Determine the test output of the neural network of width per stage

Again, set the total number of categories in the test set as C, and construct the reference matrix R _j (1≤j≤C); obtain the error vector between the verification output y _{v_m} and the reference matrix R _j of the corresponding level:

J _{test_mj} =||softmax(y _{test_m} )-R _j || ₂ , 1≤j≤C;

Among them, the elements of the jth row of the reference matrix R _j are all 1, and the other elements are 0. The dimension of each reference matrix R _j is C×N _{test_samples} ; The dimension of J _{test_mj} is 1×N _{test_samples} , y _{v_m} The dimension of _{test_samples} is C×N.

Finally, judge the output of each level width neural network determined by the decision threshold, specifically: when the minimum error of the current level width neural network is less than or equal to the current level decision threshold, that is, min(J _{test_m} j)≤T _m , then it is judged as The current stage is the correct classification output stage for this output.

其中，y_{test_ind_m}的维数为1×N_{test_samples}。When the minimum error of the current-level width neural network is greater than the current-level decision threshold, that is, min(J _{test_mj} )>T _m , it is judged that the current level cannot correctly classify the output, and the output is transferred to the next-level width neural network for processing. Test, and so on, until the output finds the correct classification output stage. Then get the label corresponding to the test output of the neural network of each level width determined by the decision threshold

where the dimension of y _{test_ind_m} is 1×N _{test_samples} .

If the test sample cannot be output in the first 25 levels, it will be output directly in the last 26 level.

Finally, the output L _test of the test set in the entire network can be obtained; wherein, both the correctly classified samples and the wrongly classified samples can be calculated statistically, and then the accuracy of the sample classification of the parallel multi-level wide neural network of the present invention can be obtained.

Comparative ratio

The same original training sample set, validation set and test set as in the above-mentioned embodiment are used, and random forest (RF), multi-layer perceptron (MP), traditional radial basis function network (RBF), support vector machine (SVM) are used respectively. , Breadth Learning System (BLS), Conditional Deep Learning Model (CDL), Deep Belief Network (DBL), Convolutional Neural Network LeNet-5, Deep Boltzmann Machine (DBM) and Deep Stochastic Deep Forest (gc) as base classification Figure 4 shows the accuracy of data classification obtained by various learning methods.

As can be seen from Figure 4, compared to the current mainstream learning models: random forest (RF), multi-layer perceptron (MP), traditional radial basis function network (RBF), support vector machine (SVM), breadth learning system (BLS), Conditional Deep Learning Models (CDL), Deep Belief Networks (DBL), Convolutional Neural Networks LeNet-5, Deep Boltzmann Machines (DBM), and Deep Stochastic Deep Forests (gc forest), the parallel of the present invention The accuracy of the classification result of the multi-level wide neural network (PMWNN) has very high competitiveness. The final classification accuracy of the method of the present invention is 99.10%, and WRBF is a wide radial basis function network. Compared with the deep random deep forest learning model, the neural network of the method of the present invention has a multi-level base neural network, each level is used to learn different parts of the data set samples, and can adaptively determine the neural network according to the complexity of the problem and the data set. The structure of the network realizes the optimization of computing resources; at the same time, the neural network of the present invention can be tested in parallel during the test, that is, the test data is given to all levels of the network at the same time, and the decision threshold of each level obtained in the training process is obtained. To decide which level of neural network each test sample is finally output by, the parallel testing process greatly reduces the waiting time when actually using the network.

In addition, the parallel multi-level wide neural network of the present invention can realize incremental learning, that is, when new data comes, a new wide radial basis function network can be added to learn new characteristics without retraining the entire parallel multi-level neural network Wide neural network, this means that the proposed network can learn new knowledge without forgetting old knowledge. The new training data is input to the current M-level network. If there are misclassified samples, then they and the original training set with data augmentation are used to build a new training data set, train the new width radial basis function network, and use the new The validation set is validated, and the decision threshold is calculated to build the M+1-th level network. Ultimately, the new parallel multi-level wide neural network will consist of M+1 level wide radial basis function networks. At the same time, the parallel multi-level width neural network designed by the present invention can be tested in parallel during testing, all test samples will be sent to the width radial basis function network of all levels, and the decision threshold determines which width radial basis function network is allocated. Give the corresponding test sample. This process does not need to wait for the network output of other stages, so that it can be parallelized during testing and speed up the testing process.

Each level of width neural network in the parallel multi-level width neural network of the present invention may be a width radial basis function network, BP neural network, convolutional neural network or other classifiers, and each level of the multi-level width neural network The types of classifiers can be different.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these changes and modifications of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these changes and modifications.

Claims (9)

1. A learning method based on a parallel multi-level width neural network comprises a multi-level width neural network, wherein each level of width neural network comprises an input layer, a hidden layer and an output layer which are connected in sequence, and the learning method is characterized by comprising the following steps:

step 1, obtaining an original training sample set, and constructing a parallel M-level width neural network Net₁，…Net_m，…，Net_MM is 1, 2 …, M, each level width neural network is used as a base classifier of the corresponding level; performing M times of data transformation on an original training sample set to correspondingly obtain M verification sets x_{v_1}，…x_{v_m}，…x_{v_M}；

Wherein, the total number of samples of the original training sample set is N_trEach training sample is an image sample to be learned; the parallel M-level width neural network Net is constructed₁，…Net_m，…，Net_MThe specific process is as follows:

designing a parallel multi-level width neural network by adopting a width radial basis function network as a basis classifier; m width radial basis function networks are connected together to form a parallel multi-level width neural network Net₁，Net₂，...Net_M(ii) a Each base classifier is used as a first level;

the specific process for constructing the wide radial basis function network comprises the following steps:

construction includes N_0kA Gaussian base function of

The center of the radial basis function network is a subset randomly taken from an original training sample set, and the value of the standard deviation is a constant; acquiring multiple groups of local characteristic images of each image sample to be learned in an original training sample set by adopting a sliding window so as to obtain multiple groups of local characteristic matrixes, and taking the multiple groups of local characteristic matrixes as input data of a Gaussian basis function to obtain multiple radial basis function networks, namely a wide radial basis function network;

step 2, adopting an original training sample set and M verification sets x_{v_1}，…x_{v_m}，…x_{v_M}Respectively training and verifying each level of the parallel M-level width neural network to obtain the trained parallel M-level width neural network and the verification output y corresponding to each level of the width neural network_{v_m}M is 1, 2 …, M; obtaining each verification output y by adopting a minimum error method_{v_m}Corresponding label y_{v_ind_m}Further obtaining a correct classification sample set y of the verification set of each level of width neural network of the trained parallel M level width neural network_{vc_m}And misclassification sample set y_{vw_m}；

The original training sample set and M verification sets x are adopted_{v_1}，…x_{v_m}，…x_{v_M}Training and verifying each stage of the parallel M-stage width neural network respectively, comprising the following sub-steps:

substep 2.1, using the original training sample set as the level 1 width neural networkNet₁For the 1 st order width neural network Net₁Training to obtain a trained first-level width neural network;

substep 2.2, using a first verification set x_{v_1}Verifying the trained 1 st-level width neural network to obtain an error classification sample set y of a verification set of the 1 st-level width neural network_{vw_1}；

Substep 2.3, misclassification sample set y of first-level width neural network_{vw_1}Input samples A as a level 2 wide neural network_{v_1}(ii) a Then randomly extracting a training sample set A from the original training sample set_{v_2}Let the total input sample set { A }_{v_1}+A_{v_2}The number of samples in (A) is equal to the number of samples in the original training sample set, and the total input sample set (A) is_{v_1}+A_{v_2}Taking the samples as input samples of a 2 nd-level width neural network;

substep 2.4, use the total input sample set { A }_{v_1}+A_{v_2}Training the 2 nd-level width neural network to obtain a trained 2 nd-level width neural network; using a second verification set x_{v_2}Verifying the trained 2 nd-level width neural network to obtain an error classification sample set y of a verification set of the 2 nd-level width neural network_{vw_2}；

And analogizing in sequence, respectively training the neural networks with the widths from the 3 rd level to the M th level to obtain the trained parallel neural networks with the widths of the M levels and the corresponding verification output y of the neural networks with the widths of each level_{v_m}；

The training process of each stage of the width neural network comprises the following steps:

(a) using image samples to be learned in an original training sample set as input data, setting the initial position of a sliding window at the upper left corner of each image sample to be learned, selecting a sliding step length of 1 pixel, sliding the sliding window from left to right in sequence from top to bottom, and stretching 3-dimensional image blocks of all the image samples in the sliding window into a matrix x_kRespectively forming corresponding original matrixes by each local characteristic image according to pixels, and sequentially arranging the 2 nd to the last columns of each original matrix to the 1 st column to form a column vector;arranging N column vectors in sequence to form a local feature matrix x of a group of training image samples_kK is more than or equal to 1 and less than or equal to K, and a local feature matrix x_kEach column of (a) represents one image sample to be learned;

(b) matrix x of local features_kInput to the device including N_0kA Gaussian base function of

The output is noted as:

wherein,

a column vector containing N elements;

sliding the sliding window to correspond to one radial basis function network every time, and obtaining K radial basis function networks after sliding is finished;

(c) for each radial basis function network, the output data phi of the Gaussian basis function subjected to nonlinear transformation_kIntroducing sequencing and downsampling:

output data phi for wide radial basis function networks_kEach row of the image to be learned is summed to obtain a row vector, each element of the row vector is the sum of the pixels of the local specific position of each image to be learned, the sums of the pixels of the local specific position of each image to be learned are arranged in a descending order to obtain a descending order vector

Using an index s_kVector a in descending order_kMarking an original position corresponding to the local specific position of each image to be learned to obtain sequenced output data phi'_k＝sort(Φ_k，s_k)；

Downsampling the sorted output data, and setting a downsampling interval N_kSThen the sampling output is phi_kS＝subsample(Φ′_k，N_kS) The output of the Gaussian base function is [ phi ] - [ phi ]_1S，Φ_2S，…，Φ_KS]；

(d) Setting the desired output to D ═ D₁，D₂，…，D_C](ii) a And performing linear layer connection on the output of the Gaussian basis function of the wide radial basis function network, wherein the weight of the linear layer is as follows: w ═ W₁，W₂，…，W_C]；

Wherein C is the total number of categories of the original sample;

obtaining class output Y ═ Y of the wide radial basis function network₁，Y₂，…，Y_C]Phi W; in particular, a least mean square estimate of the weights of the linear layer is calculated by minimizing the square error

The concrete formula is as follows:

least mean square estimation of weights of linear layers by pseudo-inverse matrix of gaussian basis function output phi of wide radial basis function network

Wherein phi⁺Outputting a pseudo-inverse matrix of phi for the Gaussian basis function of the wide radial basis function network;

finally, the class output of the wide radial basis function network obtained by calculation is as follows:

further obtaining a trained width radial basis function network, and completing the training process of each level of width neural network;

step 3, correctly classifying a sample set y of a verification set of each level of width neural network of the trained parallel M level width neural network_{vc_m}And misclassification sample set y_{vw_m}Respectively carrying out statistical calculation to correspondingly obtain the decision threshold T of the trained neural network with each level of width_m(ii) a Decision threshold T of neural network with each level width_mThe decision basis is used as the decision basis of the neural network with the corresponding level width, and the parallel neural network with the M level width determined by the decision threshold is obtained;

step 4, obtaining a test set, taking the test set as input data of the parallel M-level width neural network determined by the decision threshold, and inputting the input data to each level of width neural network determined by the decision threshold in parallel to test to obtain the output of each level of width neural network determined by the decision threshold; obtaining an error vector of each level of width neural network, and judging the output of each level of width neural network determined by the decision threshold, thereby obtaining a label y corresponding to the test output of each level of width neural network determined by the decision threshold_{test_ind_m}。

2. The parallel multi-level width neural network-based learning method of claim 1, wherein in step 1, the data transformation compresses or deforms the samples in the original sample set by elastic transformation; or the data transformation rotates, flips, zooms in, or zooms out samples in the original sample set through affine transformation.

3. The learning method based on the parallel multi-level width neural network as claimed in claim 1, wherein in step 2, the minimum error method is:

firstly, setting the total class number of an original training sample set as C, and constructing a reference matrix R_j，1≤j≤C；

Wherein, the reference matrix R_jEach reference matrix R having 1 for the element in the jth row and 0 for the remaining elements_jOf dimension C × N_tr；

Second, it is used forAccording to the verification output y of the trained neural network with each stage width_{v_m}Obtaining verification output y_{v_m}Reference matrix R corresponding to the stage_jError vector between:

J_{v_mj}＝||softmax(y_{v_m})-R_j||₂，1≤j≤C；

wherein | | | purple hair₂Representing the 2 norm of the matrix, softmax () being a normalized exponential function; j. the design is a square_{v_mj}Dimension of 1 × N_tr；y_{v_m}Of dimension C × N_tr；

Finally, output y to verification_{v_m}Reference matrix R corresponding to the stage_jError vector J therebetween_{v_mj}Calculating the minimum value to obtain the class label y corresponding to the trained neural network with each level of width_{v_ind_m}：

Wherein, y_{v_ind_m}Dimension of 1 × N_tr。

4. The parallel multi-level width neural network-based learning method of claim 1, wherein in step 3, the statistical calculation comprises the following sub-steps:

and 3.1, setting a correct classification sample set and an incorrect classification sample set of the mth level width neural network of the trained parallel M level width neural network as follows: y is_{vc_m}And y_{vw_m}The total number of samples in the correctly classified sample set and the incorrectly classified sample set is respectively as follows: n is a radical of_{vc_m}And N_{vw_m}And N is_{vc_m}+N_{vw_m}＝N_trThen, the errors of the correctly classified sample set and the incorrectly classified sample set are respectively:

e_{vc_m}＝||softmax(y_{vc_m})-t_{vc_m}||₂；

e_{vw_m}＝||softmax(y_{vw_m})-t_{vw_m}||₂；

wherein, t_{vc_m}Is an m-level width neural networkIn correctly classifying the sample y_{vc_m}Corresponding real label, t_{vw_m}Is a misclassified sample y in an m-level wide neural network_{vw_m}A corresponding real label;

substep 3.2, sample set y is classified correctly_{vc_m}And misclassification sample set y_{vw_m}Respectively calculate the correct classification sample set y_{vc_m}Respectively, mean and variance of_cAnd σ_c(ii) a Misclassification sample set y_{vw_m}The mean and variance of (a) are respectively: u. of_wAnd σ_w(ii) a Then the sample set y is correctly classified_{vc_m}And misclassification sample set y_{vw_m}The corresponding gaussian distributions are:

correctly classifying sample set y_{vc_m}And misclassification sample set y_{vw_m}The corresponding gaussian probability density functions are:

substep 3.3, sorting the sample set y according to errors_{vw_m}Error e of_{vw_m}Sum variance σ_wObtaining a decision threshold T of the neural network with m-level width_m＝min(e_{vw_m})-ασ_w；

Wherein α is a constant to give a margin to allow all misclassified samples y_{vw_m}Is rejected at the current stage.

5. Parallel multi-level-width neural network-based science according to claim 2The learning method is characterized in that in step 4, the test set acquisition is as follows: obtaining an original test sample set x_test(ii) a Correspondingly acquiring M groups of test sample sets x through M times of data expansion_{test_1}，...，x_{test_m}，...，x_{test_M}I.e. a test set.

6. The parallel multi-level-width neural network-based learning method of claim 5, wherein the data is augmented as: for the original test sample set x_testIs performed for each sample in N_testDConverting the data to obtain N_testDTest sample set as test set x of M-th order width neural network of parallel M-order width neural network determined by decision threshold_{test_m}；

Wherein, the original test sample set x_testTotal number of middle test samples is N_{test_samples}。

7. The parallel multi-level width neural network-based learning method of claim 1, wherein in step 4, the obtaining the error vector of each level of width neural network comprises the following sub-steps:

substep 4.1, set M groups of test samples x_{test_1}，x_{test_2}，...，x_{test_M}Respectively parallelly inputting the data to parallel M-level width neural networks determined by the decision threshold, and correspondingly obtaining N of each-level width neural network determined by the decision threshold_testDAn output y_{test_m_d}，d＝1，2…N_testD；

Substep 4.2, N for each level of width neural network determined for decision threshold_testDAn output y_{test_m_d}，d＝1，2…N_testDCalculating the average value to obtain the test output of each level of width neural network determined by the decision threshold

Substep 4.3, setting the total class number of the test set as C, and constructing a reference matrix R_j，1≤j is less than or equal to C; obtaining verification output y_{v_m}Reference matrix R corresponding to the stage_jError vector between:

J_{test_mj}＝||softmax(y_{test_m})-R_j||₂，1≤j≤C；

wherein, the reference matrix R_jEach reference matrix R having 1 for the element in the jth row and 0 for the remaining elements_jOf dimension C × N_{test_samples}；J_{test_mj}Dimension of 1 × N_{test_samples}，y_{v_m}Of dimension C × N_{test_samples}。

8. The learning method based on the parallel multi-level width neural network of claim 7, wherein the output of each level of width neural network determined by the decision threshold is determined as:

when the minimum error of the current stage width neural network is less than or equal to the current stage decision threshold, judging that the current stage is the correct classification output stage of the output:

min(J_{test_mj})≤T_m；

when the minimum error of the current stage width neural network is larger than the decision threshold of the current stage, judging that the current stage can not correctly classify the output, transferring the output to the next stage width neural network for testing, and repeating the steps until the output finds the correctly classified output stage:

min(J_{test_mj})＞T_m。

9. the learning method based on the parallel multi-level width neural network of claim 8, wherein in step 4, the label y corresponding to the test output of each level of width neural network determined by the decision threshold is obtained_{test_ind_m}Comprises the following steps:

wherein, y_{test_ind_m}Dimension of 1 × N_{test_samples}。

Images