CN110738242B - Bayes structure learning method and device of deep neural network - Google Patents

Abstract

Embodiments of the present invention provide a Bayesian structure learning method and device for a deep neural network. The method includes constructing a deep neural network including a plurality of learning units with the same internal structure, each learning unit contains several hidden layers, and between the hidden layers includes a plurality of computing units, the network structure is the relative weight of each computing unit, and the parameters are used. The network structure is modeled by changing the variational distribution; the training subset is extracted, and the network structure is sampled by the reparameterization process; . In the embodiment of the present invention, a deep neural network including a plurality of learning units with the same internal structure is constructed, and the relative weights of the calculation units among the hidden layers in the learning unit are trained through a training set, so as to obtain an optimized network structure, This brings a comprehensive improvement to the prediction performance and prediction uncertainty of deep neural networks.

Description

A Bayesian Structure Learning Method and Device for Deep Neural Networks

technical field

The present invention relates to the technical field of data processing, in particular to a Bayesian structure learning method and device of a deep neural network.

Background technique

Bayesian deep learning aims to provide accurate and reliable uncertainty assessments for flexible and efficient deep neural networks. Traditionally, Bayesian networks introduce uncertainty in the network weights, which often prevents the model from being prone to overfitting, and also brings uncertainty to the model using efficient predictions. However, there are problems with introducing uncertainty on network weights. First, the prior distribution of the artificially set weights is often unreliable, which can easily lead to problems such as over-pruning, which greatly limits the model fitting ability; This is because of the complex dependencies in the variational distribution. Recently, particle-based variational inference techniques have also been used to optimize Bayesian networks, but they also suffer from problems such as particle collapse and degradation.

Therefore, current Bayesian networks cannot provide accurate and reliable prediction performance in practical applications.

SUMMARY OF THE INVENTION

Due to the above problems in the existing methods, embodiments of the present invention provide a Bayesian structure learning method and device for a deep neural network.

In a first aspect, an embodiment of the present invention provides a Bayesian structure learning method for a deep neural network, including:

Constructing a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, the learning unit includes a preset number of hidden layers, and every two hidden layers includes a plurality of computing units, defining the network structure is the relative weight of each computing unit, and adopts a parameterized variational distribution to model the network structure;

Randomly extract a training subset from a preset training set, and use a reparameterization process to sample the network structure of the learning unit;

Calculate the evidence lower bound ELBO of the deep neural network according to the sampled network structure;

If the change of the lower bound of the evidence exceeds a preset loss threshold, the network structure and network weight are optimized according to a preset optimization method, and a training subset is randomly extracted from the training set again to continue the optimization of the learning unit. The network structure is trained;

If the change of the lower bound of the evidence does not exceed the preset loss threshold, it is determined that the training ends.

Further, the re-parameterization process is used to sample the network structure of the learning unit; specifically, it includes:

According to the preset adaptive coefficient, the network structure of the learning unit is sampled using a reparameterization process.

Further, according to the sampled network structure, the evidence lower bound of the deep neural network is obtained; specifically, it includes:

According to the sampled network structure, the output results corresponding to the marked samples in the training subset are calculated, and the error of the deep neural network and the difference between the network variational distribution and the preset prior distribution are calculated. log density difference;

A weighted summation of the error of the deep neural network and the log density difference is performed to obtain an evidence lower bound for the deep neural network.

Further, in the construction of a deep neural network, the deep neural network includes at least one learning unit with the same internal structure; specifically, it includes:

Constructing a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, and inserting a preset down-sampling layer and/or an up-sampling layer between predetermined learning units; wherein, the down-sampling layer includes : batch regularization layer, linear rectification layer, convolutional layer and pooling layer, the upsampling layer is constructed by deconvolution layer.

Further, the input layer of the deep neural network is a convolutional layer used for preprocessing, and the output layer is a linear fully connected layer.

Further, the variational distribution is a concrete distribution.

In a second aspect, an embodiment of the present invention provides a Bayesian structure learning device for a deep neural network, including:

a construction unit for constructing a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, the learning unit includes a preset number of hidden layers, and every two hidden layers includes a plurality of computations unit, the network structure is defined as the relative weight of each computing unit, and the parameterized variational distribution is used to model the network structure;

A training unit for randomly extracting a training subset from a preset training set, and sampling the network structure of the learning unit by a reparameterization process;

an error calculation unit, configured to obtain the evidence lower bound ELBO of the deep neural network according to the sampled network structure;

The judgment unit is configured to optimize the network structure and network weight according to a preset optimization method if the change of the lower bound of the evidence exceeds a preset loss threshold, and randomly extract a training subset from the training set again to continue the analysis. The network structure of the learning unit is trained; if the change of the lower bound of the evidence does not exceed a preset loss threshold, it is determined that the training is over.

Further, the training unit is specifically configured to randomly extract a training subset from a preset training set, and use a reparameterization process to sample the network structure of the learning unit according to a preset adaptability coefficient.

In a third aspect, an embodiment of the present invention also provides an electronic device, including:

processors, memories, communication interfaces and communication buses; wherein,

The processor, the memory, and the communication interface communicate with each other through the communication bus;

The communication interface is used for information transmission between communication devices of the electronic device;

The memory stores computer program instructions executable by the processor, and the processor invokes the program instructions to perform the following methods:

Constructing a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, the learning unit includes a preset number of hidden layers, and every two hidden layers includes a plurality of computing units, defining the network structure is the relative weight of each computing unit, and adopts a parameterized variational distribution to model the network structure;

Randomly extract a training subset from a preset training set, and use a reparameterization process to sample the network structure of the learning unit;

Calculate the evidence lower bound ELBO of the deep neural network according to the sampled network structure;

If the change of the lower bound of the evidence exceeds a preset loss threshold, optimize the distribution and network weight of the network structure according to a preset optimization method, and randomly extract a training subset from the training set again to continue the learning The network structure of the unit is trained;

If the change of the lower bound of the evidence does not exceed the preset loss threshold, it is determined that the training ends.

In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following method is implemented:

Constructing a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, the learning unit includes a preset number of hidden layers, and every two hidden layers includes a plurality of computing units, defining the network structure is the relative weight of each computing unit, and adopts a parameterized variational distribution to model the network structure;

Randomly extract a training subset from a preset training set, and use a reparameterization process to sample the network structure of the learning unit;

Calculate the evidence lower bound ELBO of the deep neural network according to the sampled network structure;

If the change of the lower bound of the evidence exceeds a preset loss threshold, optimize the distribution and network weight of the network structure according to a preset optimization method, and randomly extract a training subset from the training set again to continue the learning The network structure of the unit is trained;

If the change of the lower bound of the evidence does not exceed the preset loss threshold, it is determined that the training ends.

The Bayesian structure learning method and device for a deep neural network provided by the embodiments of the present invention, by constructing a deep neural network including a plurality of learning units with the same internal structure, and using a training set for each hidden layer in the learning unit. The relative weights of the computing units are trained to obtain an optimized network structure of the learning unit, thereby bringing about an overall improvement in the prediction performance and prediction uncertainty of the deep neural network.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flowchart of a Bayesian structure learning method of a deep neural network according to an embodiment of the present invention;

2 is a schematic structural diagram of a Bayesian structure learning device for a deep neural network according to an embodiment of the present invention;

FIG. 3 illustrates a schematic diagram of the physical structure of an electronic device.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

FIG. 1 is a flowchart of a Bayesian structure learning method for a deep neural network according to an embodiment of the present invention. As shown in FIG. 1 , the method includes:

Step S01, constructing a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, the learning unit includes a preset number of hidden layers, and between every two hidden layers includes a plurality of computing units, The network structure is defined as the relative weight of each computing unit, and a parameterized variational distribution is used to model the network structure.

A deep neural network is constructed according to actual needs. The deep neural network includes: an input layer, an output layer, and at least one repeatedly stacked learning unit between the input layer and the output layer, and each learning unit is connected in series in sequence, with the same network structure.

The learning unit includes a preset number of K hidden layers, and any two hidden layers include multiple computing units, such as a fully connected unit, a convolution unit, and a pooling unit. The network structure of the learning unit α={α ^{(i, j)} |1≤i<j≤K}, where α ^{(i, j)} includes the distance between the hidden layer of the i-th layer and the hidden layer of the j-th layer and each Calculate the relative weights corresponding to the units. The calculation from the feature representation of the i-th hidden layer to the feature representation of the j-th hidden layer is: according to the α ^{(i, j)} , the weighted sum of the outputs of the calculation units between the two.

The relative weight corresponding to each computing unit obeys the classification distribution. In order to facilitate training based on the optimization method, it can be set as a learnable, continuous and parameterized variational distribution to construct the network structure.

Further, the variational distribution is a concrete distribution.

The variational distribution can be set according to actual needs, and the embodiment of the present invention only provides an example of the concrete distribution: concrete distribution, but for the sake of simplicity, the concrete distribution is used as an example in the following embodiments. for example.

Step S02 , randomly extract a training subset from a preset training set, and use a reparameterization process to sample the network structure of the learning unit.

Step S03: Obtain the evidence lower bound ELBO of the deep neural network according to the sampled network structure.

Step S04, if the change of the lower bound of the evidence exceeds the preset loss threshold, optimize the distribution of the network structure and the network weight according to the preset optimization method, and randomly extract training subsets from the training set again to continue the analysis. The network structure of the learning unit is trained.

Step S05, if the change of the lower bound of the evidence does not exceed the preset loss threshold, it is determined that the training is over.

A training set containing N samples is preset, and each sample (x _n , y _n ) includes input data x _n and a corresponding pre-labeled output result _yn .

According to the training set, the deep neural network is trained, and after the training is completed, the optimized network structure of the learning unit is finally obtained, and the optimized network structure includes the corresponding calculation units between each two hidden layers. The relative weight of the optimization. The specific training process is as follows:

使用重参数化技术从变分分布中随机采样出所述学习单元的网络结构α，所述深度神经网络中的网络权重和偏置参数表示为w，所述深度神经网络根据所述采样的网络结构得到的预测概率

f是一个以x_n为输入以w为参数、以α为学习单元的网络结构的深度神经网络对应的函数。1. Randomly select a subset of size k in the training set

Using the reparameterization technique to randomly sample the network structure α of the learning unit from the variational distribution, the network weight and bias parameters in the deep neural network are represented as w, and the deep neural network is based on the sampled network. Predicted probabilities obtained by the structure

f is a function corresponding to a deep neural network with x _n as the input, w as the parameter, and α as the network structure of the learning unit.

2. Obtain the current evidence lower bound (Evidence Lower Bound, ELBO) of the deep neural network according to the sampled network structure.

3. Compare the current evidence lower bound with the evidence lower bound obtained after the last training is completed to obtain the change of the evidence lower bound. If the change of the lower bound of the evidence obtained after this training exceeds the preset loss threshold, it is determined that the training is not completed, and the current network structure and network weight are further optimized according to the preset optimization method. Then, re-select a subset from the training set to continue the training process of 1-3. Until the change of the lower bound of the evidence is less than or equal to the loss threshold, it is determined that the training is over.

It can be seen from the above training process that the training process of the embodiment of the present invention is only to train the connection relationship between the hidden layers inside the learning unit. Therefore, for the K-layer hidden layer structure, the network structure α of the learning unit has a total of K (K-1)/2 variables, each of which represents the relative weight of computing units between a pair of hidden layers.

In the embodiment of the present invention, a deep neural network including a plurality of learning units with the same internal structure is constructed, and the relative weights of the calculation units between the hidden layers in the learning unit are trained through a training set, so as to obtain the learning unit's relative weight. The optimized network structure brings a comprehensive improvement to the prediction performance and prediction uncertainty of the deep neural network.

Based on the above embodiment, further, in step S02, a reparameterization process is used to sample the network structure of the learning unit; specifically, it includes:

According to the preset adaptive coefficient, the network structure of the learning unit is sampled using a reparameterization process.

其中∈＝{∈^(i，j)}一组服从各个维度独立的Gumbel变量，τ是正实数表示温度。In order to prevent the problem that the training is difficult to converge during the training process, when constructing the learning unit, a preset adaptive coefficient β={β ^{(i, j)} } is added to the reparameterization process to adjust the sampling variance. The resulting specific reparameterization process is

where ∈={∈ ^(i,j) } a set of Gumbel variables obeying the independence of each dimension, and τ is a positive real number representing temperature.

The specific process is as follows:

a. Randomly sample a set of independent variables ∈ from the Gumbel distribution;

b. Multiply the variable obtained in (a) by the adaptive coefficient β to obtain the scaled variable;

c. Add the variable obtained in (b) to the parameter θ of the concrete distribution and divide by the temperature coefficient τ;

d. Input the result obtained in (c) into the softmax transform to obtain the sampled network structure α=g(θ, β, ∈).

The embodiment of the present invention avoids the problem of difficulty in training convergence during the training process by adjusting the adaptive coefficient for the reparameterization process, thereby bringing an overall improvement to the prediction performance and prediction uncertainty of the deep neural network.

Based on the above embodiment, further, the step S03 specifically includes:

Step S031, according to the sampled network structure, calculate the output results corresponding to the marked samples in the training subset, and calculate the error of the deep neural network, as well as the variational distribution of the network and a preset prior. The log density difference in the distribution.

Step S032 , weighting and summing the error of the deep neural network and the logarithmic density difference to obtain the evidence lower bound of the deep neural network.

同时，基于当前采样后的网络结构α，计算它在变分分布与预设的先验分布中的对数密度的差值，两个分布都是适应性的concrete分布，对数密度为：After each sampled network structure of the learning unit is obtained according to the extracted training subset by using the reparameterization process, the prediction obtained by each sample in the training subset through the current deep neural network is obtained according to the sampled network structure. results, and calculate the error of the prediction result. There are many calculation methods for the error. Here we only take cross-entropy as an example to illustrate: cross-entropy

At the same time, based on the current sampled network structure α, the difference between its logarithmic density in the variational distribution and the preset prior distribution is calculated. Both distributions are adaptive concrete distributions, and the logarithmic density is:

Therefore, the easy-to-obtain difference is recorded as KL;

其中

表示整个训练集对应的KL距离分配到该子集所对应的权重，该损失即为变分推理的近似误差，利用预设优化方法，例如，梯度下降的方法，来解优化问题min_θ，wELBO，以此来进一步优化所述网络结构。Weighting the difference between the cross entropy and the logarithmic density, the overall loss of the deep neural network is obtained as

in

Indicates that the KL distance corresponding to the entire training set is assigned to the weight corresponding to the subset, and the loss is the approximate error of variational inference. Use a preset optimization method, such as gradient descent, to solve the optimization problem min _{θ, w} ELBO to further optimize the network structure.

In the embodiment of the present invention, in each training, the difference between the cross-entropy and the logarithmic density is calculated, and weighted to obtain the evidence lower bound of the deep neural network, and the network structure is optimized according to the change of the evidence lower bound, so that the depth The prediction performance and prediction uncertainty of the neural network brought an overall improvement.

Based on the above embodiment, further, the step S01 specifically includes:

Constructing a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, and inserting a preset down-sampling layer and/or an up-sampling layer between predetermined learning units; wherein, the down-sampling layer includes : batch regularization layer, linear rectification layer, convolutional layer and pooling layer, the upsampling layer is constructed by deconvolution layer.

The deep neural network constructed in the embodiment of the present invention includes a plurality of learning units connected in series, and a preset down-sampling layer and/or an up-sampling layer is inserted between some predetermined learning units according to actual application requirements. Among them, in the picture classification, only the downsampling layer needs to be inserted, and for semantic segmentation, both need to be inserted. Among them, the downsampling layer is generally composed of batch regularization-linear rectification-convolution-pooling, and the upsampling is generally constructed by deconvolution.

Further, the input layer of the deep neural network is a convolutional layer used for preprocessing, and the output layer is a linear fully connected layer.

The convolution computing unit used adopts the operation order of batch regularization-linear rectification-convolution-batch regularization;

Integrate simultaneous, independent, and same-type computing units into a group operation to improve computing efficiency, mainly by organizing multiple convolutions into a group convolution to improve computing efficiency.

The embodiment of the present invention increases the depth by inserting a downsampling layer and/or an upsampling layer between predetermined learning units, adding a preprocessing convolution layer to the input layer, and inserting a linear fully connected layer at the end of the network. performance of neural networks.

The deep neural network constructed in the above embodiment can be applied to various scenarios, for example:

Image classification:

1) Use the network prediction result and the marked cross entropy as the error in the model training for training;

2) During the test, for a set of test samples, 100 network structures are randomly sampled from the distribution of the learned network structure, and based on these structures, the model gives 100 sets of predicted probabilities;

3) Take the mean value of these predicted probabilities to obtain the predicted probability;

4) Take the category with the largest predicted probability in 3) as the classification of the picture.

Semantic segmentation:

1) Use the sum of the prediction results of all pixels and the marked cross-entropy as the error in the model training for training;

2) During the test, for a set of test samples, 100 network structures are randomly sampled from the distribution of the learned network structure. Based on these structures, the model gives 100 groups of pixel-level prediction probabilities;

3) Average these predicted probabilities to obtain pixel-level predicted probabilities;

4) Take the largest category of the result obtained in 3) on each pixel as the segmentation result of the pixel.

Detecting adversarial examples:

1) For a set of adversarial samples, randomly sample 30 network structures from the learned network structure distribution;

2) Based on these structures, the model gives 30 groups of predicted probabilities;

3) Take the average of these predicted probabilities to obtain the final predicted probability of the model;

4) Calculate the entropy of the predicted probability obtained in 3) as an index for detection;

5) If the entropy obtained in 4) is significantly larger than the predicted corresponding entropy of the normal sample, it means that the adversarial sample is detected.

Detection field migration:

1) For a group of samples sampled from different fields with the training data, randomly sample 100 network structures from the learned structure distribution;

2) Based on these structures, the model gives 100 groups of predicted probabilities;

3) Take the average of these predicted probabilities to obtain the final predicted probability of the model;

4) Calculate the entropy of the predicted probability obtained in 3) as an index for detection;

5) If the entropy obtained in 4) is significantly larger than the predicted corresponding entropy of the normal sample, it means that domain migration is detected.

Using the deep Bayesian structure network in the following example to test on the natural image classification datasets CIFAR-10 and CIFAR-100, the trained models achieved classification error rates of 4.98% and 22.50%, respectively, which are significantly better than the most advanced models. Advanced deep neural networks ResNet and DenseNet. Applying the present invention to the CamVid semantic segmentation task, under the same training conditions as the powerful FC-DenseNet method, an average IoU that is 2.3 higher than that of FC-DenseNet can be achieved, and at the same time, it is comparable to the world-leading method. s level. In addition, the model trained by the present invention can detect adversarial samples and domain transfer through the uncertainty of prediction, and in the test, the invention shows that the prediction uncertainty is significantly better than that of the traditional Bayesian neural network. To sum up, the present invention introduces uncertainty into the network structure of the deep network by using the network structure learning space proposed in the neural network architecture search, performs Bayesian modeling, and uses the random variational inference method for learning, and alleviates the The problems of difficult design priors and difficult inference posteriors of Bayesian deep networks with weight uncertainty have brought a comprehensive improvement to the prediction performance and prediction uncertainty of the network model, and significantly improved the performance of Bayesian neural networks in pictures. Performance on tasks such as classification, semantic segmentation, detecting adversarial examples, and domain transfer.

FIG. 2 is a schematic structural diagram of a Bayesian structure learning device for a deep neural network according to an embodiment of the present invention. As shown in FIG. 2 , the device includes: a construction unit 10 , a training unit 11 , an error calculation unit 12 and a judgment unit 13 , wherein ,

The construction unit 10 is used to construct a deep neural network, the deep neural network includes at least one learning unit with the same internal structure, the learning unit includes a preset number of hidden layers, and every two hidden layers include complex numbers. A computing unit, the network structure is defined as the relative weight of each computing unit, and a parameterized variational distribution is used to model the network structure; the training unit 11 is used to randomly extract a training subset from a preset training set, And adopt the reparameterization process to sample the network structure of the learning unit; the error calculation unit 12 is used to obtain the evidence lower bound ELBO of the deep neural network according to the sampled network structure; the judgment unit 13 is used for if If the change of the lower bound of the evidence exceeds the preset loss threshold, the network structure and network weight are optimized according to the preset optimization method, and a training subset is randomly extracted from the training set again to continue the network analysis of the learning unit. The structure is trained; if the change of the lower bound of the evidence does not exceed the preset loss threshold, it is determined that the training is over.

The construction unit 10 constructs a deep neural network according to actual needs, and the deep neural network includes: an input layer, an output layer, and at least one repeatedly stacked learning unit between the input layer and the output layer, and each learning unit is connected in series in sequence, have the same network structure.

The learning unit includes a preset number of K hidden layers, and any two hidden layers include multiple computing units, such as a fully connected unit, a convolution unit, and a pooling unit. The network structure of the learning unit α={α ^(i,j) |1≤i<j≤K}, where α ^(i,j) includes the distance between the hidden layer of the i-th layer and the hidden layer of the j-th layer and each Calculate the relative weights corresponding to the units. The calculation from the feature representation of the i-th hidden layer to the feature representation of the j-th hidden layer is: according to the α ^(i,j) , the weighted sum of the outputs of the calculation units between the two.

The relative weights corresponding to each computing unit are subject to classification distribution, and can be set as a learnable, continuous, and parameterized variational distribution in order to facilitate training based on the optimization method.

Further, the variational distribution is a concrete distribution.

The variational distribution can be set according to actual needs, and the embodiment of the present invention only provides an example of the concrete distribution: concrete distribution, but for the sake of simplicity, the concrete distribution is used as an example in the following embodiments. for example.

The training unit 11 presets a training set including N samples, and each sample (x _n , y _n ) includes input data x _n and a corresponding pre-labeled output result _yn .

According to the training set, the deep neural network constructed by the construction unit 10 is trained, and after the training is completed, the optimized network structure of the learning unit is finally obtained, and the optimized network structure includes every two The relative weight of optimization corresponding to each computing unit between two hidden layers. The specific training process is as follows:

使用重参数化技术从变分分布中随机采样出所述学习单元的网络结构α，所述深度神经网络中的网络权重和偏置参数表示为w，所述深度神经网络根据所述采样的网络结构得到的预测概率

f是一个以x_n为输入以w为参数、以α为学习单元的网络结构的深度神经网络对应的函数。1. The training unit 11 randomly selects a subset of size k in the training set

Using the reparameterization technique to randomly sample the network structure α of the learning unit from the variational distribution, the network weight and bias parameters in the deep neural network are represented as w, and the deep neural network is based on the sampled network. Predicted probabilities obtained by the structure

f is a function corresponding to a deep neural network with x _n as the input, w as the parameter, and α as the network structure of the learning unit.

2. The error calculation unit 12 obtains the current evidence lower bound (Evidence Lower Bound, ELBO) of the deep neural network according to the sampled network structure.

3. The judging unit 13 compares the current lower bound of evidence with the lower bound of evidence obtained after the last training, so as to obtain the change of the lower bound of evidence. If the change of the lower bound of the evidence obtained after this training exceeds the preset loss threshold, it is determined that the training is not completed, and the sampled network structure and network parameters are further optimized according to the preset optimization method. Then, re-select a subset from the training set to continue the training process of 1-3. Until the change of the lower bound of the evidence is less than or equal to the loss threshold, it is determined that the training is over.

It can be seen from the above training process that the training process of the embodiment of the present invention is only to train the connection relationship between the hidden layers inside the learning unit. Therefore, for the K-layer hidden layer structure, the network structure α of the learning unit has a total of K (K-1)/2 variables, each of which represents the relative weight of computing units between a pair of hidden layers.

The apparatus provided in the embodiment of the present invention is used to execute the foregoing method, and its function refers to the foregoing method embodiment for details, and the specific method flow is not repeated here.

In the embodiment of the present invention, a deep neural network including a plurality of learning units with the same internal structure is constructed, and the relative weights of the calculation units between the hidden layers in the learning unit are trained through a training set, so as to obtain the learning unit's relative weight. The optimized network structure brings a comprehensive improvement to the prediction performance and prediction uncertainty of the deep neural network.

Based on the above embodiment, further, the training unit is specifically configured to randomly extract a training subset from a preset training set, and use a reparameterization process to sample the network structure of the learning unit according to a preset adaptability coefficient .

其中∈＝{∈^(i，j)}一组服从各个维度独立的Gumbel变量，τ是正实数表示温度。In order to prevent the problem that the training is difficult to converge during the training process, when constructing the learning unit, the constructing unit adds a preset adaptive coefficient β={β ^{(i, j)} } to the reparameterization process to adjust the sampling Variance. Therefore, the specific reparameterization process obtained by the training unit is:

where ∈={∈ ^(i,j) } a set of Gumbel variables obeying the independence of each dimension, and τ is a positive real number representing temperature.

The specific process is as follows:

a. Randomly sample a set of independent variables ∈ from the Gumbel distribution;

b. Multiply the variable obtained in (a) by the adaptive coefficient β to obtain the scaled variable;

c. Add the variable obtained in (b) to the parameter θ of the concrete distribution and divide by the temperature coefficient τ;

d. Input the result obtained in (c) into the softmax transform to obtain the sampled network structure α=g(θ, β, ∈).

The apparatus provided in the embodiment of the present invention is used to execute the foregoing method, and its function refers to the foregoing method embodiment for details, and the specific method flow is not repeated here.

The embodiment of the present invention avoids the problem of difficulty in training convergence during the training process by adjusting the adaptive coefficient for the reparameterization process, thereby bringing an overall improvement to the prediction performance and prediction uncertainty of the deep neural network.

FIG. 3 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 3 , the electronic device may include: a processor (processor) 301, a communication interface (Communications Interface) 303, a memory (memory) 302, and a communication bus 304, The processor 301 , the communication interface 303 , and the memory 302 communicate with each other through the communication bus 304 . The processor 301 may invoke logic instructions in the memory 302 to perform the above-described method.

Further, an embodiment of the present invention discloses a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer During execution, the computer can execute the methods provided by the foregoing method embodiments.

Further, an embodiment of the present invention provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the methods provided by the foregoing method embodiments. method.

Those skilled in the art can understand that: in addition, the above-mentioned logic instructions in the memory 302 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (7)

1. A Bayesian structure learning method for a deep neural network is characterized by comprising the following steps:

constructing a deep neural network, wherein the deep neural network comprises at least one learning unit with the same internal structure, the learning unit comprises a preset number of hidden layers, a plurality of computing units are arranged between every two hidden layers, a network structure is defined as the relative weight of each computing unit, and the network structure is modeled by adopting parameterized variational distribution;

randomly extracting a training subset from a preset training set, and sampling a network structure of the learning unit by adopting a re-parameterization process;

according to the sampled network structure, calculating an evidence lower bound ELBO of the deep neural network;

if the change of the evidence lower bound exceeds a preset loss threshold value, optimizing the network structure and the network weight according to a preset optimization method, and randomly extracting a training subset from the training set again to continue training the network structure of the learning unit;

if the change of the evidence lower bound does not exceed a preset loss threshold, judging that the training is finished;

when the deep neural network is used for picture classification, then:

training by using the cross entropy of the network prediction result and the label as an error in model training;

during testing, for a group of test samples, randomly sampling 100 network structures from the network structure distribution obtained by learning, and based on the structures, giving 100 groups of prediction probabilities by a model;

averaging the prediction probabilities to obtain the prediction probability;

taking the category with the maximum prediction probability as the classification of the pictures;

when the deep neural network is used for semantic segmentation, then:

training by using the sum of the prediction results of all pixels and the cross entropy of the labels as an error in model training;

during testing, for a group of test samples, randomly sampling 100 network structures from the network structure distribution obtained by learning, and based on the structures, giving 100 groups of pixel-level prediction probabilities by a model;

averaging the prediction probabilities to obtain pixel-level prediction probabilities;

taking the class with the maximum pixel-level prediction probability on each pixel as the segmentation result of the pixel;

when the deep neural network is used to detect challenge samples, then:

for a group of confrontation samples, randomly sampling 30 network structures from the network structure distribution obtained by learning;

based on these structures, the model gives 30 sets of prediction probabilities;

averaging the prediction probabilities to obtain the final prediction probability of the model;

calculating the entropy of the final prediction probability of the obtained model as a detection index;

if the obtained entropy is obviously larger than the predicted corresponding entropy of the normal sample, the confrontation sample is detected;

when the deep neural network is used to detect a domain migration, then:

for a group of samples which are sampled from different fields from the training data, randomly sampling 100 network structures from the structure distribution obtained by learning;

based on these structures, the model gives 100 sets of prediction probabilities;

averaging the prediction probabilities to obtain the final prediction probability of the model;

calculating the entropy of the obtained prediction probability as a detection index;

if the entropy obtained in the step (b) is obviously larger than the entropy corresponding to the prediction of the normal sample, the field migration is detected;

the network structure for sampling the learning unit by adopting the reparameterization process specifically comprises the following steps:

sampling the network structure of the learning unit by adopting a re-parameterization process according to a preset adaptability coefficient;

when a learning unit is constructed, a preset adaptability coefficient beta is added to a heavy parameterization process^(i，j)Adjusting the variance of the samples; the specific reparameterization process thus obtained is

Wherein ∈ { [ epsilon ]^(i，j)A set of Gumbel variables subject to dimensional independence, where τ is a positive real number representing temperature;

the specific process is as follows:

a. randomly sampling a group of independent variables belonging to the same group from Gumbel distribution;

b. multiplying the variable obtained in (a) by an adaptive coefficient beta to obtain a scaled variable;

c. adding the variable obtained in (b) to a parameter θ of the constret distribution, and then dividing by the temperature coefficient τ;

d. inputting the result obtained in the step (c) into softmax transformation, and obtaining a sampled network structure alpha which is g (theta, beta and epsilon);

the calculating of the evidence lower bound ELBO of the deep neural network according to the sampled network structure specifically includes:

according to the sampled network structure, calculating output results corresponding to the labeled samples in the training subset, and calculating the error of the deep neural network and the difference value of the logarithmic density in the network variation distribution and the preset prior distribution;

and carrying out weighted summation on the error of the deep neural network and the logarithmic density difference value to obtain the evidence lower bound of the deep neural network.

2. The Bayesian structure learning method for the deep neural network as recited in claim 1, wherein the deep neural network is constructed and comprises at least one learning unit with the same internal structure; the method specifically comprises the following steps:

constructing a deep neural network, wherein the deep neural network comprises at least one learning unit with the same internal structure, and a preset down-sampling layer and/or an up-sampling layer are/is inserted between predetermined learning units; wherein the downsampling layer comprises: the device comprises a batch regularization layer, a linear rectification layer, a convolution layer and a pooling layer, wherein the up-sampling layer is constructed by a deconvolution layer.

3. The Bayesian structure learning method for the deep neural network as recited in claim 2, wherein an input layer of the deep neural network is a convolutional layer for preprocessing, and an output layer is a linear fully-connected layer.

4. The Bayesian structure learning method for a deep neural network as recited in claim 3, wherein the variation distribution is a concoret distribution.

5. A Bayesian structure learning device for a deep neural network, comprising:

the deep neural network comprises at least one learning unit with the same internal structure, the learning unit comprises a preset number of hidden layers, a plurality of computing units are arranged between every two hidden layers, the network structure is defined as the relative weight of each computing unit, and the parameterized variational distribution is adopted to model the network structure;

the training unit is used for randomly extracting a training subset from a preset training set and sampling the network structure of the learning unit by adopting a re-parameterization process;

the error calculation unit is used for obtaining an evidence lower bound ELBO of the deep neural network according to the sampled network structure;

the judging unit is used for optimizing the distribution and the network weight of the network structure according to a preset optimization method if the change of the evidence lower bound exceeds a preset loss threshold value, and randomly extracting a training subset from the training set again to continue training the network structure of the learning unit; if the change of the evidence lower bound does not exceed a preset loss threshold, judging that the training is finished;

when the deep neural network is used for picture classification, then:

training by using the cross entropy of the network prediction result and the label as an error in model training;

during testing, for a group of test samples, randomly sampling 100 network structures from the network structure distribution obtained by learning, and based on the structures, giving 100 groups of prediction probabilities by a model;

averaging the prediction probabilities to obtain the prediction probability;

taking the category with the maximum prediction probability as the classification of the pictures;

when the deep neural network is used for semantic segmentation, then:

training by using the sum of the prediction results of all pixels and the cross entropy of the labels as an error in model training;

during testing, for a group of test samples, randomly sampling 100 network structures from the network structure distribution obtained by learning, and based on the structures, giving 100 groups of pixel-level prediction probabilities by a model;

averaging the prediction probabilities to obtain pixel-level prediction probabilities;

taking the class with the maximum pixel-level prediction probability on each pixel as the segmentation result of the pixel;

when the deep neural network is used to detect challenge samples, then:

for a group of confrontation samples, randomly sampling 30 network structures from the network structure distribution obtained by learning;

based on these structures, the model gives 30 sets of prediction probabilities;

averaging the prediction probabilities to obtain the final prediction probability of the model;

calculating the entropy of the final prediction probability of the obtained model as a detection index;

if the obtained entropy is obviously larger than the predicted corresponding entropy of the normal sample, the confrontation sample is detected;

when the deep neural network is used to detect a domain migration, then:

for a group of samples which are sampled from different fields from the training data, randomly sampling 100 network structures from the structure distribution obtained by learning;

based on these structures, the model gives 100 sets of prediction probabilities;

averaging the prediction probabilities to obtain the final prediction probability of the model;

calculating the entropy of the obtained prediction probability as a detection index;

if the obtained entropy is obviously larger than the entropy corresponding to the prediction of the normal sample, the field migration is detected;

the network structure for sampling the learning unit by adopting the reparameterization process specifically comprises the following steps:

sampling the network structure of the learning unit by adopting a re-parameterization process according to a preset adaptability coefficient;

when a learning unit is constructed, a preset adaptability coefficient beta is added to a heavy parameterization process^(i，j)Adjusting the variance of the samples; the specific reparameterization process thus obtained is

Wherein ∈ { [ epsilon ]^(i，j)A set of Gumbel variables subject to dimensional independence, where τ is a positive real number representing temperature;

the specific process is as follows:

e. randomly sampling a group of independent variables belonging to the same group from Gumbel distribution;

f. multiplying the variable obtained in (a) by an adaptive coefficient beta to obtain a scaled variable;

g. adding the variable obtained in (b) to a parameter θ of the constret distribution, and then dividing by the temperature coefficient τ;

h. inputting the result obtained in the step (c) into softmax transformation, and obtaining a sampled network structure alpha which is g (theta, beta and epsilon);

the calculating of the evidence lower bound ELBO of the deep neural network according to the sampled network structure specifically includes:

according to the sampled network structure, calculating output results corresponding to the labeled samples in the training subset, and calculating the error of the deep neural network and the difference value of the logarithmic density in the network variation distribution and the preset prior distribution;

and carrying out weighted summation on the error of the deep neural network and the logarithmic density difference value to obtain the evidence lower bound of the deep neural network.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the bayesian structure learning method of the deep neural network of any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the bayesian structure learning method of a deep neural network according to any one of claims 1 to 4.

Images