CN115086174B - Communication optimization method based on random event triggered stochastic gradient descent algorithm - Google Patents

Abstract

The invention provides a communication optimization method based on a random event triggering random gradient descent algorithm, which is suitable for a distributed machine learning training process, and aims at the high-volume communication problem existing in the current distributed machine learning training. The L2 norm is used for measuring the distance between two gradients, the probability of the gradient is converted into a [0,1] interval by an exponential function, and a Bernoulli variable is defined to indicate the probability of each triggering result. In practical application, random numbers can be uniformly generated in the [0,1] interval to serve as a threshold value for describing practical probability comparison. The method is suitable for enhancing the generalization performance of the method in practical application when the data of enterprises are polluted.

Description

Communication optimization method based on random event triggered stochastic gradient descent algorithm

Technical field

The invention belongs to the field of machine learning technology, and specifically relates to a communication optimization method based on a stochastic gradient descent algorithm triggered by random events.

Background technique

Gradient Descent is the process of minimizing a function by following the gradient of the cost function. This process involves the recognition of cost forms and their derivative forms that allow us to move in a given direction from a known given point. For example, move downward toward the minimum value. In machine learning, we can use stochastic gradient descent to minimize the error in the training model, that is, one evaluation and update is completed at each iteration. This optimization algorithm works by making a prediction for each training instance it sees and repeating the process a certain number of times. This process can be used to find the coefficients of the model that lead to the smallest error on the training data. Stochastic gradient descent (SGD), also known as incremental gradient descent, is an iterative method for optimizing a differentiable objective function. This method iteratively updates the weights and bias terms by computing the gradient of the loss function on mini-batches of data. SGD far surpasses naive gradient descent on highly non-convex loss surfaces, and this simple hill-climbing technique has dominated modern non-convex optimization.

In the era of big data, massive data brings rich information, which inevitably leads to a significant increase in communication and computing costs. In this case, even the SGD method has extremely heavy communication; on the other hand, there is often some interference or noise in the data, which will interfere with the convergence of the algorithm. How to design a suitable SGD variant method in the presence of these two problems, which can effectively reduce the required communication while ensuring algorithm convergence, is an urgent technical problem that needs to be solved in this field.

Contents of the invention

In view of the shortcomings of the existing technology, the problem to be solved by the present invention is how to reduce the heavy communication rate caused by distributed machine learning training in the presence of noise. Through the event triggering mechanism, the number of communication rounds is reduced and the high burden caused by communication is solved.

To this end, the present invention proposes a communication optimization method based on a random event-triggered stochastic gradient descent algorithm, including:

Step 1: Collect local data as a sample data set and build a cloud-edge distributed machine learning framework;

Step 2: The child node constructs a stochastic gradient descent algorithm model based on random event triggering and trains it;

Step 3: The cloud center server performs gradient summary and updates model parameters;

Step 4: Use the trained stochastic gradient descent algorithm model to predict data.

The specific expression of step 1 is:

Step 1-1: Determine the public cloud center servers of M data providers (hereinafter referred to as sub-nodes), and each sub-node uses its own local data as a sample data set;

Step 1-2: Select the machine learning model structure, determine the loss function, and build a cloud-edge distributed machine learning framework. The objective function f _m of each sub-node is to minimize the loss on the local data set:

In the formula, N _m is the number of local data sets, is the loss function of each sample data;

Then the global goal is to minimize the average loss of all M child nodes:

The cloud center server sends model initialization to each child node and starts training;

The specific expression of step 2 is:

Step 2-1: At each iteration time k, each child node m randomly selects a data sample to calculate the gradient according to the current model parameter ω _k ;

Step 2-2: Construct trigger conditions based on random events:

In the formula, Represents the stochastic gradient at the kth moment, /> Indicates the gradient of the last transmission of child node m based on k time, and τ _m is the delay factor of node m;

Step 2-3: Determine whether node m transmits random gradient at time k according to the constructed trigger condition. The specific expression is:

Step 2-3-1: Calculate the probability of whether node m transmits the current gradient at time k according to formula (4)

in, is the probability of no transmission at time k,/> is the probability of transmission;

Step 2-3-2: Generate a random number uniformly in the interval [0, 1] If currently transmitting/> The probability is greater than/> Then/> Sent to the cloud center node, if currently transmitting/> The probability is less than/> then give up the transmission;

The specific expression of step 3 is:

Step 3-1: The cloud center server updates the model parameters based on the information transmitted by the node; the specific expression is:

Step 3-1-1: The cloud center server resets the delay factor τ _m of the node that is transmitting to 1, and sets the delay factor of the node that is not transmitting to τ _m +1;

Step 3-1-2: The cloud center server performs gradient descent using the new gradient of the node that is transmitting and the old gradient of the node that is not transmitting, and obtains the model parameter ω _{k+1 at time k+1} ;

In the formula, α _k represents the learning step factor;

Step 3-2: The cloud center server sends the updated model parameters ω _k+1 to all child nodes, and loops from step 1-1 to step 3-2 until the training is completed.

The beneficial effects of the present invention are:

1) In the actual distributed machine learning training process, especially when multiple companies work together to train models. This method defines the criterion for the difference between two gradients. The L2 norm represents the distance between two vectors in space. The smaller it is, the closer the two vectors are in space. During gradient descent, it can be considered that for the global function, two similar gradients will bring about a similar decrease in function value. Taking advantage of this feature, the number of communications can be reduced through the trigger conditions designed by this method, so that the communication overhead during the training process is reduced.

2) Ability to select transmission randomly. When the difference norm of the two gradients is small, the algorithm will transmit with a smaller probability, but it does not absolutely not transmit. On the contrary, when the difference norm of the two gradients is large, the algorithm will transmit with a larger probability. There is a probability of transmission, but it is not certain that it will be transmitted. Therefore, in this case, due to the uncertainty caused by additional noise, the child nodes still have the opportunity to adaptively choose whether to upload the latest gradient to correct the impact of the noise. It is suitable for when enterprise data is contaminated, and can enhance the generalization performance of this method in practical applications.

Description of the drawings

Figure 1 is a schematic diagram of the cloud-edge distributed machine learning framework in the present invention;

Figure 2 is a schematic diagram of the event triggering function in the present invention;

Figure 3 is a flow chart of the communication optimization method based on the stochastic gradient descent algorithm triggered by random events in the present invention;

Figure 4 is a result graph on the Iris data set in the present invention, where (a) is the result graph when acr=0.05, σ ² =0, (b) is the result graph when acr = 0.178, σ ² =0.05 The result diagram, (c) is the result diagram when acr=0.452, σ ² =0.1, the horizontal axis coordinate Epoch represents the number of iteration rounds, loss is the loss function value of training, Acc is the accuracy on the test set, and the communication rate is each The communication rate at a moment, acr is the average communication rate of all communication rounds, σ ² is the variance of gradient noise;

Figure 5 is a result graph on the Covtype data set in the present invention, where (a) is the result graph when acr=0.256, σ ² =0, (b) is the result graph when acr = 0.308, σ ² =0.1 The result graph, (c) is the result graph when acr=0.615, σ ² =0.4;

Figure 6 is a result diagram of the present invention on the MNIST data set, in which (a) is the loss function value of training, (b) is the accuracy on the test set, and cr is the average communication rate of all communication rounds;

Figure 7 is a result diagram on the Cifar10 data set in the present invention, where (a) is the loss function value of training, (b) is the accuracy on the test set;

Figure 8 shows the results compared with deterministic random triggering on the actual data set. The abscissa AverageCommunication Rate is the average communication rate, Average Loss is the average loss function value, SET-SGD is the random event trigger, and DET-SGD is the deterministic event. trigger.

Detailed ways

The invention will be further described below in conjunction with the accompanying drawings and specific implementation examples. The method of the present invention is suitable for machine learning training, and is particularly effective on distributed machine learning systems with noise, aiming to reduce unnecessary communications. The method includes the design of a random triggering mechanism, the probability distribution of Bernoulli variables, and the use of event triggering to reduce the number of transmissions and significantly reduce communication overhead. First, use the L2 norm to measure the distance between the two gradients, then use the exponential function to convert it into the probability of the [0, 1] interval, and define a Bernoulli variable to indicate the probability of each trigger result. In practical applications, random numbers uniformly generated in the [0,1] interval can be used as thresholds to characterize actual probability comparisons. The present invention realizes randomness in the triggering mechanism, has smoother properties, and can reduce the impact of noise; in terms of numerical simulation, it greatly reduces the number of communications during the training process and saves bandwidth while ensuring convergence. resource.

As shown in Figure 3, first design a random event triggering mechanism, abandon the original fixed threshold of standard event triggering, and describe the result of each event triggering by probability. In distributed machine learning, there is a cloud central parameter server and M training nodes. Taking node m as an example, at the k-th moment, the node accepts the latest model parameters ω _k sent by the cloud parameter server at this time, and then generates a random gradient with noise Then convert the stochastic gradient/> With the latest transmission gradient based on time k/> Bring it into the random trigger condition and get k time transmission/> The probability. After all M nodes have completed this operation, the central server uses the received gradients to update uniformly, and in this cycle, the training of the stochastic gradient descent algorithm is completed.

The present invention proposes a communication optimization method based on random event-triggered stochastic gradient descent algorithm, which specifically includes the following steps:

The technical solution of the present invention is introduced by taking a scenario involving two data owners (i.e., Bank A and Bank B, hereinafter referred to as child nodes) as an example (this architecture can be extended to scenarios involving multiple data owners).

Step 1: Collect local data as a sample data set and build a cloud-edge distributed machine learning framework; the specific expression is:

Step 1-1: A and B want to jointly train a machine learning model to predict the loan amount available for new users. Their business systems respectively have the information of their respective savers (salaries, loan status, etc.) to construct data sets. Each child node uses its own local data as a training data set,

Step 1-2: Due to data privacy protection and security considerations, data cannot be exchanged directly, so they jointly selected a public cloud center server, selected the machine learning model structure (such as neural network, logistic regression, etc.), and determined the loss function ( Cross entropy loss, error mean square value loss, etc.), together form a cloud-edge distributed machine learning framework, as shown in Figure 1. In Figure 1, Cloud Server is the cloud center server, Worker is the child node, and Dataset is each child node. data set, ω _k is the parameter of the model at time k, Indicates transmitting the gradient at the latest moment, and none means not transmitting.

Among them, the objective function f _m of each child node is to minimize the loss on the local data set:

In the formula, N _m is the number of local data sets, is the loss function of each sample data;

Then the global goal is to minimize the average loss of all M child nodes:

The cloud center server sends model initialization to each byte point and starts training;

Step 2: The child node constructs a stochastic gradient descent algorithm model triggered by random events and conducts training; the specific expression is:

Step 2-1: At each iteration time k, each child node m randomly selects a data sample to calculate the gradient based on the current model parameter ω _k .

Step 2-2: Taking node m as an example, the random gradient at the k-th moment is It represents the gradient of the last transmission of node m based on time k, and τ _m is the delay factor of node m. Will/> with/> Bring it into the designed random trigger conditions, as shown in Figure 2, and get k time transmission/> The probability. The trigger condition based on random events is expressed as:

When the distance between two gradients is closer, their L2 norm is smaller, and the probability of transmission is also smaller;

Step 2-3: Determine whether node m transmits random gradient at time k according to the constructed trigger condition. The specific expression is:

Step 2-3-1: Definition is the probability of no transmission at time k,/> is the probability of transmission, then whether node m transmits the current latest gradient at time k can be determined by the following Bernoulli random variable/> The distribution expression of:

Step 2-3-2: Generate a random number uniformly in the interval [0, 1] If currently transmitting/> The probability is greater than/> Then/> Sent to the cloud center node, if currently transmitting/> The probability is less than/> then give up the transmission;

Step 3: The cloud center server performs gradient summary and updates model parameters; the specific expression is:

Step 3-1: Update the model parameters according to the information transmitted by the node; the specific expression is:

Step 3-1-1: The cloud center node resets the delay factor τ _m of the node that is transmitting to 1, and sets the delay factor of the node that is not transmitting to τ _m +1;

Step 3-1-2: The cloud center server performs gradient descent using the new gradient of the node that is transmitting and the old gradient of the node that is not transmitting, and obtains the model parameter ω _{k+1 at time k+1} ;

In the formula, α _k represents the step size during model training;

Step 3-2: The cloud center server sends the updated model parameters ω _k+1 to all child nodes, and loops from step 1-1 to step 3-2 until the training is completed;

Step 4: Use the trained stochastic gradient descent algorithm model to predict data; the specific expression is:

Step 4-1: After the model training is completed, check the communication cost of the trained model and perform actual deployment;

Step 4-2: When there is data about a new customer, use it as input to the model and finally get its loan amount.

In order to verify the effectiveness of the method of the present invention, experiments were first conducted under a linear model and on a relatively small database. Iris is a small dataset containing 150 samples with four features divided into three types. 99 samples (labeled setosa and versicolour) were used for binary classification, and the results are shown in Figure 4.

Covtype has 54 features and the values vary greatly, so we redefine all features as [0,1] and use 30000 samples (labeled Douglas-fir and Krummholz) for training and the other 7877 samples for testing Set, the results are shown in Figure 5.

It can be seen that all curves converge to zero regardless of the amount of noise. In fact, the noise needs to be controlled within a specific range, within which the algorithm can reduce the impact of the noise to a certain extent. Note that the communication rate also tends to decrease. At the beginning of training, because the distance from the optimal point is far away and the gradient is relatively large, the difference between the two gradients is also relatively large, which results in a higher trigger frequency. As the number of iterations increases, the gradient gradually decreases to zero. At this time, the difference between the two gradients becomes smaller, so the trigger frequency can also be reduced. In other words, this method ensures convergence through a larger number of early triggers and saves later communication traffic. This phenomenon also explains why larger variance leads to larger communication rates, especially in the later stages.

We built the LeNet5 model on the dataset MNIST and the CifarNet model on the dataset Cifar10 to show the versatility of the method of the present invention. The communication rate (Cr for short) is the ratio of the actual number of transmissions to the number of transmissions that should be made.

As shown in Figure 6, a communication rate of 10% can achieve similar performance to standard SGD on MNIST. Grayscale handwritten digits are a simple data sample that do not require many parameters to characterize. Therefore, for complex parameters, a few transfers will not affect the overall results.

In Figure 7, the performance of the method of the present invention on Cifar10 is shown. In this simulation we can see the trade-off between convergence and communication rate. It only requires 50% of the communication rate to achieve similar performance to standard SGD.

In short, the stochastic gradient descent algorithm SGD proposed by the present invention can greatly reduce the communication burden, and can converge to approximately the same accuracy as the SGD baseline under appropriate hyperparameters.

Finally, on the actual data set, deterministic event-triggered SGD (DET-SGD) was used to test the inventive method under different average communication rates to verify the superiority of the inventive method. All averages are calculated from the results of 30 iteration rounds. The results show that the method of the present invention is significantly better than DET-SGD, especially at low communication rates, which proves that the method of the present invention has better robustness in a noisy environment. The results are shown in Figure 8.

Claims (1)

1. A communication optimization method based on a random event-triggered random gradient descent algorithm, comprising:

step 1: collecting local data as a sample data set, and building a cloud-edge distributed machine learning framework;

step 2: the child nodes construct a random gradient descent algorithm model triggered on the basis of random events and train the model;

step 3: the cloud center server performs gradient summarization and updates model parameters;

step 4: carrying out data prediction by using the trained random gradient descent algorithm model;

the step 1 comprises the following steps:

step 1-1: determining a cloud center server common to M data providers, wherein each child node takes own local data as a sample data set;

step 1-2: selecting a machine learning model structure, determining a loss function, and constructing a cloud-edge distributed machine learning framework, wherein the objective function f of each child node _m Is to minimize loss on the local dataset:

wherein N is _m Is the number of local data sets and,is a loss function for each sample data;

the global goal is to minimize the average loss of all M child nodes:

the cloud center server sends the model initialization to each child node and starts training;

the step 2 comprises the following steps:

step 2-1: at each iteration instant k, each child node m is based on the current model parameters ω _k Randomly selecting a data sample to calculate the gradient;

step 2-2: constructing trigger conditions based on random events:

in the method, in the process of the invention,represents the random gradient at time k, +.>Representing the last transmitted gradient of child node m with k time as reference, τ _m Is the delay factor of node m;

step 2-3: judging whether the node m transmits a random gradient at the moment k according to the constructed triggering condition

The step 2-3 comprises the following steps:

step 2-3-1: calculating the probability of whether node m transmits the current gradient at time k according to equation (4)

Wherein, for the probability that no transmission is at time k, +.>Probability of transmission;

step 2-3-2: at [0,1]Generating a random number uniformly in intervalIf the current transmission +.>The probability of (2) is greater than +.>Will beSending to the cloud center node, if the current transmission is +.>The probability of (2) is less than +.>The transmission is aborted;

the step 3 comprises the following steps:

step 3-1: the cloud center server updates model parameters according to the information transmitted by the nodes;

step 3-2: the cloud center server updates the model parameter omega _k+1 Transmitting to all child nodes, and circulating steps 1-1 to 13-2 until training is completed;

the step 3-1 comprises the following steps:

step 3-1-1: delay factor τ for the node that the cloud center server will transmit _m Reset to 1, set the delay factor of the node not transmitting to τ _m +1；

Step 3-1-2: the cloud center server uses the new gradient of the node for transmission and the old gradient of the node for no transmission to carry out gradient descent to obtain a model parameter omega at the moment k+1 _k+1 ；

Wherein alpha is _k Representing the step size factor of the learning.