CN112069876A - Handwriting recognition method based on adaptive differential gradient optimization - Google Patents

Abstract

The invention discloses a handwriting recognition method based on adaptive differential gradient optimization. In the BP neural network parameter optimization algorithm for handwriting recognition, the invention integrates and deforms the commonly used gradient descent algorithm again by combining the traditional control theory idea; then, a differential link is added in a conventional gradient descent algorithm for advanced correction, and the future change trend of an error signal is forecasted through the change rate of the error, so that the precision is improved; and finally, the learning rate is adaptively adjusted by utilizing the stored average value of past square gradients of exponential decay, so that the training rate is accelerated. The method provided by the invention introduces a differential link, can effectively improve the training speed, and forecasts the future change trend of the error signal through the change rate of the error. And the learning rate can be adjusted in a self-adaptive mode, namely when the training is close to the optimal value, the learning rate is reduced due to the fact that the accumulated past square gradient is increased, and the learning rate is prevented from being too large to skip the optimal point.

Description

A Handwriting Recognition Method Based on Adaptive Band Differential Gradient Optimization

technical field

The invention belongs to the field of artificial intelligence, and relates to network training and optimization in deep learning and intelligent computing, in particular to a handwriting recognition method based on adaptive band differential gradient optimization.

Background technique

Artificial intelligence is a technology that studies and develops the simulation, extension and expansion of human intelligence. Its main research contents can be summarized into four aspects: machine perception, machine thinking, machine behavior and machine learning. Machine learning is the use of computer, probability theory, statistics and other knowledge to allow computers to learn new knowledge and new skills by inputting data into computer programs. intelligent. Deep learning is a broader machine learning approach based on learning features, which attempts to learn at multiple levels, where higher level concepts are defined from lower level concepts that help define Many higher level concepts.

With the deepening of research, deep learning technology has been applied to hundreds of practical problems, beyond the connotation of traditional multi-layer neural networks. Optical Character Recognition (OCR) is a widely used field. Its main task is to analyze and recognize image files of text data to obtain text and layout information.

Convolutional neural networks are a special class of neural networks for data processing inspired by the structure of the visual system and proposed by biologists Hubel and Wiesel in 1962. Through experiments on cats, they found that the information processing of the human visual system is hierarchical. The primary visual cortex extracts edge features, the intermediate visual cortex extracts shapes or objects, and the higher visual cortex obtains feature combinations. Inspired by this, Lecun et al. proposed Convolutional Neural Networks in 1989. Since then, convolutional neural networks have been widely used in image processing, handwriting recognition and other fields, and many improved models have been derived.

The optimization of neural networks is a very important problem in the field of deep learning and intelligent computing, especially the learning and correction of neural network weights. Most learning algorithms are based on iterative update methods, and the commonly used gradient-based optimization algorithms are the largest The difficulty is how to choose the appropriate learning rate and gradient to speed up the training rate and convergence rate.

At present, the commonly used optimization method for handwriting recognition methods is the gradient descent algorithm with momentum, which mainly has 1) poor convergence rate and slow convergence rate. 2) The weight update lags the actual gradient change. 3) Learning rate for the problem of singleness of different parameters. In view of this, the present invention proposes an adaptive gradient optimization method with differential gradient after reintegrating the conventional gradient descent algorithm, which can effectively overcome the above problems and improve the convergence rate and convergence rate of training, thereby improving the recognition method of handwriting recognition. Rate.

SUMMARY OF THE INVENTION

Aiming at the deficiencies in the prior art, the present invention provides a handwriting recognition method based on adaptive band differential gradient optimization.

1) Preprocessing of model parameters: offline training of multi-layer convolutional neural network models.

2) Preprocessing the scanned samples: processing the handwritten character samples and locating the characters in the image, cutting or segmenting, normalizing, binarizing, smoothing, removing dryness, thinning and generating samples. The training sample set and the test sample set are constructed based on the data set.

3) Input the sample set and extract the sample set features: find out the essential features in the samples that can effectively distinguish different types, and through feature extraction, find the most useful information and greatly reduce the amount of data.

4) Network establishment and optimization: The extracted features are input into the single-hidden layer BP neural network system for training, and the system is optimized by the adaptive band differential gradient optimization method.

5) Use the resulting neural network to identify unknown handwriting samples.

Among them, the traditional gradient descent algorithm has a relatively slow convergence rate and poor convergence, which often leads to inaccurate training. In order to further speed up the training rate and avoid training falling into local extreme points, the adaptive gradient optimization method with differential is adopted.

The adaptive band differential gradient optimization method can be roughly divided into three parts:

The first part: According to the traditional control theory, deform several common gradient descent methods;

The second part: Introduce the differential link in the traditional gradient optimization to speed up the BP neural network training;

The third part: optimize the learning rate of the gradient descent method, so that the learning rate can be adjusted adaptively, and the optimization rate and convergence rate can be improved;

The specific technical solutions are as follows:

An adaptive band differential gradient optimization method, comprising the following steps:

Step 1: According to the traditional control theory, the deformation gradient descent method:

The PID controller is composed of proportional unit P, integral unit I and differential unit D. It mainly uses current, historical and advanced information to feedback the system. The relationship between its input and output is:

Where: u(t) represents the output, e(t) represents the error, and K _p , T _i and T _d represent the gains of P, I and D, respectively.

In neural networks, the update rules for stochastic gradient descent are as follows:

为梯度，将其等同为PID控制中的误差e(t)，可以将随机梯度下降法看作比例环节。Where: w _t+1 is the network weight at time t+1, w _t is the network weight at time t, α represents the learning rate,

is the gradient, which is equivalent to the error e(t) in PID control, and the stochastic gradient descent method can be regarded as a proportional link.

In order to speed up the target convergence, a stochastic gradient descent method with a momentum term is proposed, and its expression is as follows:

where r is the momentum coefficient.

After merging the above equations, the cumulative summation method is used to obtain the following gradient update rules:

Another gradient descent method is the Nesterov accelerated gradient method. The update rules are as follows:

Like the stochastic gradient descent method with a momentum term, the following update rules are obtained after transformation:

表示网络权重。in,

represents the network weight.

Comparing the above two rewritten algorithms with the PID control input and output formulas, it can be seen that both gradients use historical information and current information to update, which is equivalent to the proportional integral link.

Step 2: Introduce the differential link in the traditional gradient optimization to speed up the training of deep neural network:

It can be seen from the rewritten formulas of the two algorithms of the stochastic gradient descent method with the momentum term and the Nesterov accelerated gradient method above that the momentum term is the accumulation of historical gradients. With the existence of gradient accumulation, the weights will not change in time, namely:

Among them, σ% represents the overshoot, w _max represents the maximum value of the weight, and w ^* represents the optimal value of the weight.

In the principle of automatic control, the overshoot mainly reflects the stability in the dynamic performance of the system; the larger the overshoot, the worse the reflection stability, that is, the larger the amplitude ratio of the maximum output deviation from the steady state value; the steady state value usually It is the normal working state or equilibrium state of the system; obviously, the farther the system deviates from the equilibrium state, the more unfavorable it is for the normal operation of the system, so it is usually better to have a smaller overshoot;

Differential control is to predict the future change trend of the error signal through the rate of change of the error. By providing a leading control effect, the differential control can make the controlled process tend to be stable;

Therefore, the unstable trend produced by integral control can be counteracted;

In view of this, in order to eliminate the unstable trend brought about by the accumulation of historical gradients, the change of gradient is added, that is, the differential term:

Among them, D _t represents the differential term at time t, D _t-1 represents the differential term at time t-1, T _d represents the gain of the differential term, J _t represents the loss function at time t, and w _t is the network weight at time t;

By adding a differential term, leading information is introduced into the gradient descent method, which can effectively reduce overshoot, that is, respond to instantaneous changes; the improved gradient descent method is updated as follows:

where r is the momentum coefficient, Δw _t is the momentum term at time t, Δw _t+1 is the momentum term at time t+1, D _t+1 is the cumulative differential term at time t+1, and D _t is the cumulative differential term at time t , w _t+1 is the network weight at time t+1, and α is the learning rate.

Step 3: Optimize the learning rate of the gradient descent method, so that the learning rate can be adjusted adaptively, and the optimization rate and convergence rate can be improved:

Stores an exponentially decaying mean v _t+1 of past squared gradients, and keeps an exponentially decaying mean m _t+1 of past gradients, similar to momentum:

Among them, m _t+1 and v _t+1 represent the first-order moment (mean) and the second-order moment (no central variance) of the gradient, respectively; when Δw _t+1 and v _t+1 are initialized as vectors with a value of 0, They are biased towards zero, especially at the initial time step, when the decay rate is small, and these biases can be counteracted by computing bias-corrected first- and second-order moment estimates as follows:

Then the learning rate is:

Then, by using these methods to update the parameters, get the final update rule:

为t+1时刻的梯度二阶矩，

为t+1时刻的梯度一阶矩。where η is a hyperparameter, ε prevents minima where the denominator is zero,

is the second-order moment of the gradient at time t+1,

is the first moment of the gradient at time t+1.

In the present invention, in the BP neural network parameter optimization algorithm for handwriting recognition, in order to better speed up the training rate and improve the convergence rate, the commonly used gradient descent algorithm is reintegrated and deformed by combining the traditional control theory; A differential link is added to the conventional gradient descent algorithm to perform advanced correction, and the future change trend of the error signal is predicted through the rate of change of the error, thereby improving the accuracy; finally, the average value of the past square gradients of the stored exponential decay is used to adaptively adjust the learning rate. thereby speeding up the training rate. The adaptive gradient optimization method with differential gradient in the present invention can effectively improve the training rate by introducing the differential link, predict the future change trend of the error signal through the rate of change of the error, and the learning rate can be adjusted adaptively, that is, when the training is close to the optimal value, it is Since the accumulated past squared gradient increases and the learning rate decreases, avoid skipping the optimal point if the learning rate is too large.

Description of drawings

Fig. 1 is the overall scheme block diagram of the adaptive band differential gradient optimization method provided by the present invention;

Fig. 2 is the loss curve of simulation training in handwriting recognition;

Fig. 3 is the verification loss curve in the handwriting recognition simulation training;

Fig. 4 is the training accuracy curve in the handwriting recognition simulation;

Figure 5 shows the verification accuracy curve in the handwriting recognition simulation.

Detailed ways

The present invention specifically includes the following steps:

Step 1. Preprocessing of model parameters: offline training of single hidden layer BP neural network structure, in which the number of input nodes is 13, that is, each sample has 13 feature expressions.

Step 2. Preprocess the scanned sample:

Preprocessing is a necessary stage before recognition. It is the preprocessing of scanned handwritten character samples and the positioning, cutting or segmentation, normalization, binarization, smoothing, de-drying, thinning and character positioning of scanned images. Generating samples; positioning and cutting is to process these image samples through algorithms, search for the positioning marks on the paper image, and then read the image at the specified position according to the size of the square. Finally, a training sample set and a test sample set are generated.

Step 3. Input the sample set, extract the features of the sample set, perform classification training and optimize the network:

The sample sets are input into the pre-trained network respectively. The purpose of feature extraction is to find out the essential features in the original data that can effectively distinguish different types. Through feature extraction, the most useful information can be found, and the amount of data can be greatly reduced. Network training reduces the burden and reduces the time cost and calculation amount of learning samples. The main extraction features of the present invention include digit length, width of each part, stroke length, shape, angle and curvature of each branch, coarse grid feature, stroke density feature and the like. The specific method is to divide the image samples into 13 regions, that is, each sample has 13 feature expressions.

Step 4. Network establishment and optimization:

In step 3, it is mentioned that feature extraction 13-dimensional feature expression, that is, in the established BP neural network training, the number of input nodes is 13. The single hidden layer BP neural network structure is selected for training, but the traditional BP algorithm has a slow convergence rate and poor convergence, which often leads to inaccurate training. In order to further speed up the training rate and avoid the training falling into local extreme points, the adaptive gradient optimization method with differential gradient is adopted. The proposed adaptive gradient gradient optimization method is described as follows:

Stores an exponentially decaying mean v _t+1 of past squared gradients, and keeps an exponentially decaying mean m _t+1 of past gradients, similar to momentum:

where m _t+1 and v _t+1 represent estimates of the first and second moment of the gradient, respectively; when m _t+1 and v _t+1 are initialized to vectors with a value of 0, they are biased towards zero , especially in the initial time step, when the decay rate is small, these biases can be offset by computing bias-corrected first- and second-order moment estimates as follows:

A differential term correction is introduced, namely:

Then, by using these methods to update the parameters, get the final update rule:

Step 5. Use the resulting neural network to identify unknown handwriting samples.

Figure 2 and Figure 3 are the training loss curve and the verification loss curve respectively. From the two figures, it can be seen that the adaptive gradient optimization method with differential gradient proposed in this paper has better convergence and generalization than the traditional gradient descent algorithm with momentum. Strong chemical ability.

Figure 4 and Figure 5 are the training accuracy curve and the verification accuracy curve respectively. From the two figures, it can be clearly seen that the handwriting recognition method based on adaptive band differential gradient optimization has high accuracy and fast convergence rate.

The above-mentioned specific embodiments are used to explain the present invention, are only preferred embodiments of the present invention, rather than limit the present invention, within the spirit of the present invention and the protection scope of the claims, any modification, equivalent replacement, Improvements and the like all fall within the protection scope of the present invention.

Claims (1)

1. Based on the adaptive handwriting recognition method with differential gradient optimization, the steps of the method are as follows: 1) Preprocessing of model parameters: offline training of multi-layer convolutional neural network models; 2) Preprocessing the scanned samples: processing the handwritten character samples and positioning, cutting or segmenting, normalizing, binarizing, smoothing, de-drying, refining and generating samples in images; based on data Set up a training sample set and a test sample set; 3) Input the sample set and extract the sample set features: find out the essential features in the samples that can effectively distinguish different types, and find the most useful information through feature extraction; 4) Network establishment and optimization: The extracted features are input into the single hidden layer BP neural network system for training, and the adaptive belt differential gradient optimization method is used to optimize; 5) using the obtained neural network to identify the unknown handwriting sample; The adaptive band differential gradient optimization method is to introduce a differential term and an adaptive mechanism into the traditional momentum stochastic gradient descent method, specifically: Step 1: Introduce the differential link in the gradient descent method with momentum to speed up the training of deep neural network: In order to eliminate the unstable trend caused by the accumulation of historical gradients, the change of gradient is added, and the differential term is introduced:

Among them, D _t represents the differential term at time t, D _t-1 represents the differential term at time t-1, T _d represents the gain of the differential term, J _t represents the loss function at time t, and w _t is the network weight at time t; By adding a differential term, advanced information is introduced into the gradient descent method, and the improved gradient descent method is updated as follows:

where r is the momentum coefficient, Δw _t is the momentum term at time t, Δw _t+1 is the momentum term at time t+1, D _t+1 is the cumulative differential term at time t+1, and D _t is the cumulative differential term at time t , w _t+1 is the network weight at time t+1, α is the learning rate; Step 2: Optimize the learning rate of the gradient descent method, so that the learning rate can be adjusted adaptively: Optimize the gradient descent learning rate so that it can be adjusted adaptively, namely:

Finally, get the update rule:

where η is a hyperparameter, ε prevents minima where the denominator is zero,

is the second-order moment of the gradient at time t+1,

is the first moment of the gradient at time t+1.

Images