CN108898213A - A kind of adaptive activation primitive parameter adjusting method towards deep neural network - Google Patents

Abstract

A kind of adaptive activation function parameter adjustment method for deep neural network, described method comprises the following steps: step 1, at first carry out mathematical definition to adaptive activation function parameter adjustment method; Step 2, carry out adaptive activation function based on MNIST data set and other classic activation functions to compare and analyze the experimental results. The network used has three hidden layers, each hidden layer has 50 neurons, and the gradient descent algorithm is used to iterate for 100 cycles. The learning rate is set to 0.01, and the minimum batch The number is 100; step 3, after the optimal activation function version is obtained in step 2, it is applied to the detection of specific bladder cancer cells. In the process of continuous training of the network, the present invention finds the optimal activation function suitable for the network by continuously adjusting its own shape, improves the performance of the network, reduces the overall number of learnable parameters of the adaptive activation function in the network, and accelerates the learning rate of the network , to improve the generalization of the network.

Description

An Adaptive Activation Function Parameter Adjustment Method for Deep Neural Networks

technical field

The invention belongs to the field of adaptive activation functions, and designs an adaptive activation function parameter adjustment method oriented to a deep neural network. Specifically, the adaptive activation function controls its own shape by adding learnable parameters. At the same time, these learning parameters can be updated with the progress of network training through the back propagation algorithm, reducing the overall number of learnable parameters of the adaptive activation function in the network. .

Background technique

Nowadays, machine learning is widely used in social life, while traditional machine learning mostly uses shallow structures, such as Gaussian Mixture Model (GMM), Conditional Random Field (CRF), Support Vector Machine (SVM), etc. The expressive ability of the function is limited, and the extraction of the original input signal features is relatively elementary. Its generalization ability for complex classification problems is limited to a certain extent, and it is difficult to solve some complex natural signal processing problems, such as human speech and natural image recognition. Therefore, deep learning greatly promotes the development of machine learning by simulating the brain for learning. The biggest feature of deep learning is to transform the original data into higher-level and more abstract feature expressions through some simple but nonlinear models. A deep nonlinear network structure, which realizes the approximation of complex functions, and the ability to learn the essential characteristics of data sets from a small number of sample sets. Practice has proved that deep learning is good at discovering complex structures in high-dimensional data, and is widely used in research fields such as computer vision, speech recognition, and natural language processing.

With the application of deep learning in various fields, more and more researches focus on the innovation and optimization of deep learning algorithms. These include optimization of classifiers and loss functions, gradient descent optimization based on backpropagation, optimization of network weight parameter initialization, and optimization of artificial neural networks, among which the optimization of artificial neural networks is an important part of deep learning algorithm innovation. Artificial neural networks will have different network structures and number of neurons depending on the task. In these networks, people usually use the same activation function, such as Sigmoid, Tanh, Relu. The adaptive activation function proposed in recent years makes network neurons present different shapes, but with the expansion of network size and the increase of neurons, the learnable parameters used to adjust the shape of these neurons show a linear growth, and the network Learning efficiency is greatly reduced. It can be seen that the basic structure of artificial neural network can be regarded as formed by the interconnection of some neurons, and the activation function plays a pivotal role in it.

The main function of the activation function in the artificial neural network is to provide the nonlinear expression ability of the network. If the neurons in a neural network are only linear operations, then the network can only express a simple linear mapping, even if the depth and width of the network are increased, it is still a linear mapping, and it is difficult to effectively model nonlinearly distributed data in the actual environment. After adding the nonlinear activation function, the deep neural network has the ability of hierarchical nonlinear mapping learning. The invention mainly improves the activation function, optimizes the connection between neurons in the network, thereby further improving the performance of the network.

Contents of the invention

In order to reduce the overall number of learnable parameters of the adaptive activation function in the network, speed up the learning rate of the network, and improve the generalization ability of the network, the present invention proposes an adaptive activation function parameter adjustment method for deep neural networks. In the process, by constantly adjusting its own shape to find the optimal activation function suitable for the network, improve the performance of the network.

The technical solution adopted by the present invention to solve its technical problems is:

A kind of adaptive activation function parameter adjustment method for deep neural network, described method comprises the following steps:

Step 1, first mathematically define the parameter adjustment method of the adaptive activation function, the process is as follows:

Assuming that the number of adjustable parameters of the adaptive activation function is N, then the adaptive activation function is defined as:

f _(x) = f(a*x+c)

Among them, a and c are learnable parameters used to control the shape of the activation function. The so-called neural network is regarded as a combination of many individual neurons, and the output of the neural network is defined as a composite of weights, deviations and learnable neuron parameters. function, the function is as follows:

h _(w,b,a,c) = h(f(a*x+c))

Where h represents the output of the neural network, w and b represent the weights and biases of the network; at the same time, this function is also regarded as using the same set of learnable parameters for all neurons in the neural network. A broader definition is: Each neuron in a neural network uses different adjustable parameters, as follows:

where fn represents each neuron in a layer of the network, each layer of neurons using the same adjustable parameters is defined as follows:

Use the backbroadcasting algorithm to train the adaptive activation function in the neural network, where the learnable parameters are optimized along with the weights and biases as the network is trained, and the parameters {a1,...,n,b1,...,n} according to The chain rule of derivation is updated as follows:

where ai∈{a1,…,n,b1,…,n}, L represents the cost function, This term can be obtained from the latter layer through backpropagation. The weighting term ∑Xi can be used in all positions of the feature map or neural network layer. For the variables shared in one layer, the gradient ai can be obtained by the following formula, ∑ i is used to sum over all channels or neurons in a layer, the formula is as follows:

Step 2. Based on the MNIST dataset, compare and analyze the experimental results of the adaptive activation function and other classic activation functions to obtain the final activation function version. The process is as follows:

The network used has three hidden layers, each hidden layer has 50 neurons, iteratively uses the gradient descent algorithm at any time for 100 cycles, the learning rate is set to 0.01, and the minimum number of batches is 100.

Further, in the step 2, the comparison activation functions used include the traditional Sigmoid function, the traditional ReLU activation function, the unified version of the adaptive activation function, the respective versions of the adaptive activation function and the hierarchical versions of the adaptive activation function.

Step 3, after obtaining the optimal activation function version in step 2, apply it to the detection of specific bladder cancer cells, the process is as follows:

3.1. Create a data set for bladder cancer;

3.2. Select the combination algorithm and model to initialize the parameters;

3.3. Compare and analyze the experimental results of the optimal activation function and the traditional activation function.

Further, in the step 2, the comparison activation functions used include the traditional Sigmoid function, the traditional ReLU activation function, the unified version of the adaptive activation function, the respective versions of the adaptive activation function and the hierarchical versions of the adaptive activation function.

Furthermore, in the above 3.1, the bladder cancer cell data set is made into pascal_voc2007 format, mainly using the generated xml file to save the label information of the cells.

In the above 3.2, select the Faster R-CNN algorithm, use the vgg16 model to initialize the network parameters, and use the vgg16 pre-trained model to initialize the network parameters.

In the above 3.3, the traditional activation function in the Faster R-CNN algorithm is replaced with the optimal activation function version generated in step 3.2, and finally the experimental results are analyzed and compared.

The beneficial effects of the present invention are mainly manifested in: the effectiveness of the adaptive activation function parameter adjustment method is proved theoretically and experimentally, the optimal activation function is provided for the network, the gradient dispersion and other problems existing in the traditional activation function are avoided, and the fitting ability of the network is improved .

Description of drawings

Fig. 1 is the convergence curve diagram of activation function AS of the present invention;

Fig. 2 is an adjustment diagram of the learnable parameters of the AS activation function of the present invention;

Fig. 3 is original sigmoid activation function and final AS activation function figure of the present invention;

Fig. 4 is a graph of experimental comparison results between the final AS activation function of the present invention and other activation functions.

Figure 5 is a diagram of the Sigmoid activation function.

Figure 6 is a diagram of the Tanh activation function.

Figure 7 is a diagram of the ReLU activation function.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

Referring to Fig. 1～Fig. 7, a kind of adaptive activation function parameter adjustment method for deep neural network, described method comprises the following steps:

Step 1, first mathematically define the parameter adjustment method of the adaptive activation function, the process is as follows:

Assuming that the number of adjustable parameters of the adaptive activation function is N, then the adaptive activation function is defined as:

f _(x) = f(a*x+c)

Among them, a and c are learnable parameters used to control the shape of the activation function. The so-called neural network is regarded as a combination of many individual neurons, and the output of the neural network is defined as a composite of weights, deviations and learnable neuron parameters. function, the function is as follows:

h _(w,b,a,c) = h(f(a*x+c))

Where h represents the output of the neural network, w and b represent the weights and biases of the network; at the same time, this function is also regarded as using the same set of learnable parameters for all neurons in the neural network. A broader definition is: Each neuron in a neural network uses different adjustable parameters, as follows:

where fn represents each neuron in a layer of the network, each layer of neurons using the same adjustable parameters is defined as follows:

Use the backbroadcasting algorithm to train the adaptive activation function in the neural network, where the learnable parameters are optimized along with the weights and biases as the network is trained, and the parameters {a1,...,n,b1,...,n} according to The chain rule of derivation is updated as follows:

where ai∈{a1,…,n,b1,…,n}, L represents the cost function, This term can be obtained from the latter layer through backpropagation. The weighting term ∑Xi can be used in all positions of the feature map or neural network layer. For the variables shared in one layer, the gradient ai can be obtained by the following formula, ∑ i is used to sum over all channels or neurons in a layer, the formula is as follows:

Step 2. Based on the MNIST dataset, compare and analyze the experimental results of the adaptive activation function and other classic activation functions. The process is as follows:

The network used has three hidden layers, each hidden layer has 50 neurons, iteratively uses the gradient descent algorithm at any time for 100 cycles, the learning rate is set to 0.01, and the minimum number of batches is 100.

Further, in the step 2, the comparison activation functions used include the traditional Sigmoid function, the traditional ReLU activation function, the unified version of the adaptive activation function, the respective versions of the adaptive activation function and the hierarchical versions of the adaptive activation function.

Based on the MNIST data set, the present invention adds learnable parameters to the sigmoid classic activation function to make it an adaptive activation function: Adaptive Sigmoid (AS), and then performs each version of the adaptive activation function with the two classic activation functions of Sigmoid and ReLU Comparison of test results.

MNIST is a handwritten digit recognition data set, known as the fruit fly of deep learning experiments, which contains 60,000 pictures as training data and 10,000 pictures as test set. In the MNIST dataset, each grayscale image represents a number from 0 to 9. The size of the picture is 28*28, and the handwritten numbers will appear in the middle of the picture. The activation function AS is defined as:

f＝b ₀ *sigmoid(a ₀ *x+a ₁ )+b ₁

a0, a1, b0, b1 are four learnable parameters that control the shape of the function and can be trained along with the network weights and biases.

The present invention mainly adds learnable parameters in the sigmoid classic activation function to make it an adaptive activation function AS. The mathematical definition of the function is as follows:

Assume that the number of adjustable parameters of the adaptive activation function is N, where N=2 is assumed. Then the adaptive activation function can be defined as:

f _(x) = f(a*x+c)

Where a and c are learnable parameters used to control the shape of the activation function. The so-called neural network can be regarded as a combination of many individual neurons. The output of the neural network is defined as a composite function of weights, deviations and learnable neuron parameters. The function is as follows:

h _(w,b,a,c) = h(f(a*x+c))

where h represents the output of the neural network, and w and b represent the weights and biases of the network. At the same time, this function can also be seen as using the same set of learnable parameters for all neurons in the neural network. A more general definition is that each neuron in a neural network uses different adjustable parameters, as follows:

where fn represents each neuron in one layer of the network. Each layer of neurons is defined using the same adjustable parameters as follows:

The present invention uses a reverse rebroadcasting algorithm to train an adaptive activation function in a neural network, wherein learnable parameters along with weights and biases are optimized along with network training. The parameters {a1,...,n,b1,...,n} can be updated according to the chain derivation rule, and the update is as follows:

where ai∈{a1,…,n,b1,…,n}, L represents the cost function. This term can be obtained by backpropagation from the latter layer, and the weighting term ΣXi can be used in all positions of the feature map or neural network layer. For the variables shared in one layer, the gradient ai can be obtained by the following formula, ∑i is used to sum all channels or neurons in one layer, the formula is as follows:

In step 3, the method of adaptive activation function is applied to deep learning, and the present invention applies the optimal activation function obtained in step 2 to the detection of bladder cancer cells. The process is as follows:

3.1. Make the data set. The present invention makes the bladder cancer cell data set into the pascal_voc2007 format, mainly using the generated xml file to save the label information of the cells.

3.2. Select the appropriate algorithm and model to initialize the parameters. The present invention selects the Faster R-CNN algorithm and uses the vgg16 model to initialize network parameters, mainly using the vgg16 pre-training model to initialize network parameters to reduce training time and reduce the risk of underfitting or overfitting.

3.3. Compare and analyze the experimental results of the optimal activation function and the traditional activation function. It mainly uses the optimal activation function version generated in step 2 to replace the traditional activation function in the Faster R-CNN algorithm, and finally analyzes and compares the experimental results.

Finally, in the method proposed by the present invention, the same adjustable activation function is used in the entire network, no matter how many neurons are used in the neural network, the total number of parameters added is the number of parameters that can be learned by the adaptive activation function (these parameters are used to control the shape of the function). The entire network uses the same adjustable activation function, just like the polynomial superposition of composite functions, which enhances the nonlinearity of the network, improves the fitting ability of the network, and speeds up the learning speed of the network.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

As shown in Figure 1, the network used has three hidden layers, each hidden layer has 50 neurons, and iteratively uses the gradient descent algorithm at any time for 100 cycles. The learning rate is set to 0.01 and the minimum batch size is 100. The comparative activation functions used include the traditional Sigmoid function, the traditional ReLU activation function, the unified version of the adaptive activation function, the respective versions of the adaptive activation function, and the hierarchical version of the adaptive activation function. Figure 1 shows the convergence curves of ReLU, Sigmoid and three versions based on the adaptive activation function AS. "relu_train" indicates the classification error rate on the training set using the Relu activation function. "relu_test" indicates the classification error rate on the test set using the Relu activation function. "AUsigmoid" represents the unified version (Unified Version, UV) of the adaptive activation function AS, that is, each neuron uses the same activation function. "ALsigmoid" represents the respective version of the adaptive activation function (Individual Version, IV), that is, each neuron uses the activation function that it thinks is good. "AIsigmoid" means a layered version (Layer Version, LV), that is, all neurons in each layer use the same activation function, but the activation function between each layer is not necessarily the same.

The expression of the traditional Sigmoid activation function is as follows:

Refer to Figure 5 for the Sigmoid activation function diagram.

The Sigmoid activation function is a common activation method because it has a good explanation for the activation frequency of neurons: from no activation at all to full saturation activation at 1 at the maximum boundary. But now the Sigmoid function is rarely used. An important reason is that the Sigmoid function is saturated and the gradient disappears. This is because the Sigmoid neuron has a bad characteristic, that is, when the activation value of the neuron is close to 0 or 1, it will saturate. When in these areas, the function gradient is almost 0, which leads to the backpropagation When , this (local) gradient will be multiplied by the gradient of the entire loss function with respect to the output of the gate unit, and the result of the multiplication will also approach zero, which effectively terminates the gradient, causing almost no signal to pass through the neuron to The weight goes to the data again, leading to the final gradient dispersion problem.

Another classic activation function is: Tanh nonlinear function, the expression is as follows:

Refer to Figure 6 for the Tanh activation function diagram.

It can be seen from the figure that, compared with the Sigmoid function, although Tanh compresses the real value to [-1,1], it also has the same saturation problem as Sigmoid. But unlike the Sigmoid neuron, its output is zero-centered. In practice, the Tanh nonlinear function is more popular than the Sigmoid nonlinear function. It can be said that a Tanh neuron is a simply enlarged Sigmoid neuron.

Compared with the first two classic activation functions, ReLU is now a widely used activation function. Its mathematical formula is as follows:

f(x)=max(0,x)

Refer to Figure 7 for the ReLU activation function diagram.

Compared with the Sigmoid and Tanh functions, ReLU has a huge acceleration effect on the convergence of gradient descent, which is due to its linear, non-saturated formula. When the input of the ReLU activation function is a positive number, there is no gradient saturation problem; when the input is a negative number, the ReLU is not activated at all, which means that once a negative number is input, the ReLU will be paralyzed. For example, when a large gradient is backpropagated through a ReLU neuron, it may cause the gradient to update to a special state where the neuron is likely not to be reactivated by any other computation node. If this happens, then the gradients backpropagated through this neuron will all become zero from now on. In other words, this ReLU unit will be irreversibly paralyzed during training, which leads to the loss of data diversity.

It can be seen from Figure 1 that on the MNIST training set, using the unified version of the activation function AS achieves a lower classification error rate than using the Relu activation function. Compared with using the original sigmoid activation function, the network has a stronger fitting ability.

Figure 2 shows the parameter adjustment process diagram of the unified version of the adaptive activation function. The initial settings of the learnable parameters of the adaptive activation function are: a0=1.0, a1=0.0, b0=1.0, b1=0.0, after training iterations , its final parameters become: a0=3.87, a1=0.07, b0=5.89, b1=-0.51, basically there is not much change.

As shown in Figure 3, the final unified version of the adaptive activation function has a larger value range than the traditional Sigmoid activation function, which largely solves the gradient dispersion problem of the traditional Sigmoid activation function, so the classification accuracy is increased. .

As shown in Figure 4, the present invention uses the final adaptive activation function version (RAS) and other several activation functions to compare the experimental results, and the final adaptive activation function formula is:

f=5.89*sigmoid(3.87*x+0.07)-0.51

From the comparison chart of the experimental results, it can be seen that the unified adaptive activation function can achieve the best experimental results. In the experiment of bladder cancer cell detection, the detection results and speed of using the unified adaptive activation function are faster than the traditional activation function. Well, it further proves that each network can be trained to get its own most suitable activation function.

Claims (5)

1. an adaptive activation function parameter adjustment method for deep neural network, it is characterized in that, described method comprises the following steps: Step 1, first mathematically define the parameter adjustment method of the adaptive activation function, the process is as follows: Assuming that the number of adjustable parameters of the adaptive activation function is N, then the adaptive activation function is defined as: f _(x) = f(a*x+c) Among them, a and c are learnable parameters used to control the shape of the activation function. The so-called neural network is regarded as a combination of many individual neurons, and the output of the neural network is defined as a composite of weights, deviations and learnable neuron parameters. function, the function is as follows: h _(w,b,a,c) = h(f(a*x+c)) Where h represents the output of the neural network, w and b represent the weights and biases of the network; at the same time, this function is also regarded as using the same set of learnable parameters for all neurons in the neural network. A broader definition is: Each neuron in a neural network uses different adjustable parameters, as follows: where fn represents each neuron in a layer of the network, each layer of neurons using the same adjustable parameters is defined as follows: Use the backbroadcasting algorithm to train the adaptive activation function in the neural network, where the learnable parameters are optimized along with the weights and biases as the network is trained, and the parameters {a1,...,n,b1,...,n} according to The chain rule of derivation is updated as follows: where ai∈{a1,…,n,b1,…,n}, L represents the cost function, This term can be obtained from the latter layer through backpropagation. The weighting term ∑Xi can be used in all positions of the feature map or neural network layer. For the variables shared in one layer, the gradient ai can be obtained by the following formula, ∑ i is used to sum over all channels or neurons in a layer, the formula is as follows: Step 2. Based on the MNIST dataset, compare and analyze the experimental results of the adaptive activation function and other classic activation functions. The process is as follows: The network used has three hidden layers, each hidden layer has 50 neurons, iteratively uses the gradient descent algorithm at any time for 100 cycles, the learning rate is set to 0.01, and the minimum number of batches is 100. Step 3, after obtaining the optimal activation function version in step 2, apply it to the detection of specific bladder cancer cells, the process is as follows: 3.1. Create a data set for bladder cancer; 3.2. Select the algorithm and model to initialize the parameters; 3.3. Compare and analyze the experimental results of the optimal activation function and the traditional activation function. 2. the method for a kind of depth neural network adaptive activation function parameter adjustment as claimed in claim 1, is characterized in that, in described step 2, the contrast activation function that uses has traditional Sigmoid function, traditional ReLU activation function, self-adaptive A unified version of the activation function, individual versions of the adaptive activation function, and a layered version of the adaptive activation function. 3. The method for adjusting a deep neural network adaptive activation function parameter as claimed in claim 1 or 2, is characterized in that, in said 3.1, the bladder cancer cell data set is made into pascal_voc2007 format, mainly using the generated The xml file saves the label information of the cells. 4. the method for a kind of depth neural network adaptive activation function parameter adjustment as claimed in claim 1 or 2, is characterized in that, in described 3.2, select Faster R-CNN algorithm, utilize vgg16 model to carry out the initialization of network parameter, Use the vgg16 pre-trained model to initialize the network parameters. 5. the method for a kind of depth neural network self-adaptive activation function parameter adjustment as claimed in claim 4, is characterized in that, in described 3.3, utilize the optimal activation function version that generates in step 3.2 to replace in Faster R-CNN algorithm The traditional activation function, and finally analyze and compare the experimental results.