CN113315716B - Training method and equipment of congestion control model and congestion control method and equipment - Google Patents

Abstract

The present disclosure provides a congestion control model training method and equipment, and a congestion control method and equipment. The congestion control method includes: obtaining the current first network state information and the current application's preference for network transmission performance; inputting the obtained first network state information and the preference into the congestion control model to obtain the predicted usage that needs to be executed Actions to resize the congestion window; perform predicted actions to reset the congestion window.

Description

Congestion control model training method and device, congestion control method and device

technical field

The present disclosure generally relates to the field of communication technologies, and more specifically, relates to a congestion control model training method and equipment, and a congestion control method and equipment.

Background technique

In recent years, in order to solve the problem of network congestion and improve network performance, many congestion control protocols have been proposed, including heuristic protocols and learning-based protocols.

The learning-based congestion control protocols PCC and PCC Vivace learn the relationship between rate control behavior and observed performance in an online manner. To avoid the hard mapping between collected states and actions in traditional TCP variants, they choose the optimal sending rate by employing online learning techniques that continuously try to modify the sending rate in a small range to approach better utility function performance. While PCC and PCC Vivace are able to achieve good performance. Learning-based congestion control protocols learn congestion control strategies by interacting with the environment, and this strategy can choose appropriate actions to control the sending rate or congestion window according to the state of the network. However, learning-based congestion control protocols drive performance through pre-designed rewards or objective functions, which are fixed, and when new applications emerge, these protocols cannot meet the performance requirements of these applications, so the objective needs to be redesigned function and retrain a new model.

Contents of the invention

Exemplary embodiments of the present disclosure aim to provide a congestion control model training method and device, and a congestion control method and device, so as to solve at least the above-mentioned problems in the related art, and may not solve any of the above-mentioned problems.

According to the first aspect of an embodiment of the present disclosure, a method for training a congestion control model is provided, including: initializing the communication network environment used in the current training round; The network status information is input into the congestion control model to obtain the predicted action to adjust the size of the congestion window; execute the predicted action to reset the congestion window, and control the sender to send data to the receiver under the currently set congestion window package; when the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before executing the action, the first network state information after executing the action, and the preference, calculate The loss function of the congestion control model; by adjusting the model parameters of the congestion control model according to the loss function, the congestion control model is trained, and it is determined whether to end this training round, wherein, when it is determined not to end this In the first training round, return to the step of inputting the current training round's preference for network transmission performance and the current first network state information into the congestion control model to obtain the predicted action for adjusting the size of the congestion window.

Optionally, the first network state information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and sending rate; wherein, the delay, packet acknowledgment rate, and sending rate are based on the receiving end feedback The ACK message is OK.

Optionally, the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

Optionally, the step of determining whether to end the current training round includes: determining whether to end the current training round according to changes in the state information of the second network.

Optionally, according to the change of the second network state information, the step of determining whether to end the current training round includes: when the second network state information after performing the action satisfies the first preset condition, determining that the action is An action of victory; when the second network state information after executing the action satisfies the second preset condition, the action is determined to be a failed action; when the consecutive times of the action of victory reaches the first preset number of times, the determination ends In this training round: when the number of consecutive failed actions reaches the second preset number of times, it is determined to end the current training round; when the total number of executed actions reaches the third preset number of times, it is determined to end the current training round.

Optionally, the second network status information includes: throughput and delay; the first preset condition is: the throughput is 90%-110% of the bandwidth, and the delay is ≤0.7×timeout threshold; the second preset condition is: throughput The volume is between 50%-70% of the bandwidth and the latency is ≥0.7×timeout threshold.

Optionally, the method further includes: initializing the size of the congestion window; wherein, the step of initializing the size of the congestion window includes: estimating the bandwidth of the communication network, and determining the initial size of the congestion window based on the estimated bandwidth.

Optionally, the step of estimating the bandwidth of the communication network includes: determining the total number of ACK messages fed back by the receiving end for the N data packets sent by the sending end; dividing the total number by an average value obtained by N to determine The bandwidth of the communications network.

Optionally, when the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before performing the action, the first network state information after performing the action, and the preference , the step of calculating the loss function of the congestion control model includes: when the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before performing the action, and the first network state information after performing the action The first network state information, the reward function of the action, and the preference calculate the loss function of the congestion control model.

Optionally, the reward function of the action is calculated based on the preference and third network state information after performing the action; wherein the third network state information includes at least one of the following items: Packet rate, throughput, and latency.

Optionally, the congestion control model is constructed based on a reinforcement learning algorithm; wherein, a value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preferences for network transmission performance.

Optionally, the action predicted by the congestion control model, with probability ∈ is an action randomly selected from the action set, and with probability 1-∈ is an optimal action obtained by using a value function.

Optionally, the loss function of the congestion control model is calculated based on: a loss function L ^S (θ) for making the value function closer to the maximum reward function, and an auxiliary loss function L ^T (θ).

Optionally, the loss function of the congestion control model is expressed as: (1-ε) L ^S (θ)+ε L ^T (θ); wherein, ε is a trade-off index, and the later in a training round is used For the predicted action, the greater the value of ε when calculating the loss function of the congestion control model for this action, 0≤ε≤1.

Optionally, the objective function of the congestion control model is a compound objective function related to the following items: reward function, value function, first network state information after performing an action, first network state information before performing an action, action, this The preference of the network transmission performance for each training round, and the best preference in the current network environment.

Optionally, the method further includes: when it is determined to end the current training round, determining whether to end the training process of the congestion control model; when it is determined not to end the training process of the congestion control model, returning to perform initialization of the current training The steps of the communication network environment used by the round to proceed to the next training round.

According to the second aspect of the embodiments of the present disclosure, there is provided a congestion control method, including: obtaining the current first network state information and the preference of the current application for network transmission performance; inputting the obtained first network state information and the preference into To the congestion control model, get the predicted action to adjust the size of the congestion window; execute the predicted action to reset the congestion window.

Optionally, the first network state information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and sending rate; wherein, the delay, packet acknowledgment rate, and sending rate are based on the receiving end feedback The ACK message is OK.

Optionally, the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

Optionally, the method further includes: initializing the size of the congestion window; wherein, the step of initializing the size of the congestion window includes: estimating the bandwidth of the communication network, and determining the initial size of the congestion window based on the estimated bandwidth.

Optionally, the step of estimating the bandwidth of the communication network includes: determining the total number of ACK messages fed back by the receiving end for the N data packets sent; bandwidth.

Optionally, the congestion control model is constructed based on a reinforcement learning algorithm; wherein, a value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preferences for network transmission performance.

Optionally, the congestion control model is obtained by training using the training method described above.

According to a third aspect of an embodiment of the present disclosure, there is provided a congestion control model training device, including: an environment initialization unit configured to initialize the communication network environment used in the current training round; a prediction unit configured to The round's preference for network transmission performance and the current first network state information are input into the congestion control model to obtain the predicted action for adjusting the size of the congestion window; the congestion window setting unit is configured to perform the predicted action to Reset the congestion window, and control the sending end to send data packets to the receiving end under the currently set congestion window; the loss function calculation unit is configured to perform according to the action when the sending end receives the ACK message fed back by the receiving end The first network state information before the action, the first network state information after the action is executed, and the preference, calculating a loss function of the congestion control model; the training unit is configured to pass according to the loss function Adjusting the model parameters of the congestion control model to train the congestion control model; the round end determination unit is configured to determine whether to end this training round, wherein, when it is determined not to end this training round, the prediction unit will In this training round, the preference for network transmission performance and the current first network state information are input into the congestion control model to obtain the predicted action to adjust the size of the congestion window.

Optionally, the first network state information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and sending rate; wherein, the delay, packet acknowledgment rate, and sending rate are based on the receiving end feedback The ACK message is OK.

Optionally, the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

Optionally, the round end determining unit is configured to determine whether to end the current training round according to the change of the second network state information.

Optionally, the round end determining unit is configured to determine that the action is a winning action when the state information of the second network after performing the action satisfies the first preset condition; when the second network after performing the action When the status information satisfies the second preset condition, it is determined that the action is a failed action; when the continuous number of victorious actions reaches the first preset number of times, it is determined to end this training round; When the second preset number of times is reached, it is determined to end the current training round; when the total number of executed actions reaches the third preset number of times, it is determined to end the current training round.

Optionally, the second network status information includes: throughput and delay; the first preset condition is: the throughput is 90%-110% of the bandwidth, and the delay is ≤0.7×timeout threshold; the second preset condition is: throughput The volume is between 50%-70% of the bandwidth and the latency is ≥0.7×timeout threshold.

Optionally, the device further includes: a window initialization unit configured to initialize the size of the congestion window; wherein the window initialization unit is configured to estimate the bandwidth of the communication network, and determine the congestion window based on the estimated bandwidth the initial size of .

Optionally, the window initialization unit is configured to determine the total number of ACK messages fed back by the receiving end for the N data packets sent by the sending end; and determine the communication network based on the average value obtained by dividing the total number by N bandwidth.

Optionally, the loss function calculation unit is configured to, when the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before the action is executed, and the first network status information after the action is executed. The state information, the reward function of the actions, and the preferences compute the loss function of the congestion control model.

Optionally, the reward function of the action is calculated based on the preference and third network state information after performing the action; wherein the third network state information includes at least one of the following items: Packet rate, throughput, and latency.

Optionally, the congestion control model is constructed based on a reinforcement learning algorithm; wherein, a value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preferences for network transmission performance.

Optionally, the action predicted by the congestion control model, with probability ∈ is an action randomly selected from the action set, and with probability 1-∈ is an optimal action obtained by using a value function.

Optionally, the loss function of the congestion control model is calculated based on: a loss function L ^S (θ) for making the value function closer to the maximum reward function, and an auxiliary loss function L ^T (θ).

Optionally, the loss function of the congestion control model is expressed as: (1-ε) L ^S (θ)+ε L ^T (θ); wherein, ε is a trade-off index, and the later in a training round is used For the predicted action, the greater the value of ε when calculating the loss function of the congestion control model for this action, 0≤ε≤1.

Optionally, the objective function of the congestion control model is a compound objective function related to the following items: reward function, value function, first network state information after performing an action, first network state information before performing an action, action, this The preference of the network transmission performance for each training round, and the best preference in the current network environment.

Optionally, the device further includes: a training end determination unit configured to determine whether to end the training process of the congestion control model when it is determined to end the current training round, wherein, when it is determined not to end the congestion control model During the training process, the environment initialization unit initializes the communication network environment used in the current training round to enter the next training round.

According to a fourth aspect of an embodiment of the present disclosure, there is provided a congestion control device, including: an acquisition unit configured to acquire current first network state information and a current application's preference for network transmission performance; a prediction unit configured to The obtained first network state information and the preference are input into the congestion control model to obtain the predicted action for adjusting the size of the congestion window; the congestion window setting unit is configured to execute the predicted action to reset the congestion window.

Optionally, the first network state information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and sending rate; wherein, the delay, packet acknowledgment rate, and sending rate are based on the receiving end feedback The ACK message is OK.

Optionally, the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

Optionally, the device further includes: a window initialization unit configured to initialize the size of the congestion window; wherein the window initialization unit is configured to estimate the bandwidth of the communication network, and determine the initial size of the congestion window based on the estimated bandwidth size.

Optionally, the window initialization unit is configured to determine the total number of ACK messages fed back by the receiving end for the sent N data packets; and determine the bandwidth of the communication network according to an average value obtained by dividing the total number by N.

Optionally, the congestion control model is constructed based on a reinforcement learning algorithm; wherein, a value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preferences for network transmission performance.

Optionally, the congestion control model is obtained through training using the training device described above.

According to a fifth aspect of the embodiments of the present disclosure, there is provided an electronic device, including: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions are executed by the at least one processor When , the at least one processor is prompted to execute the above-mentioned method for training a congestion control model and/or the above-mentioned congestion control method.

According to a sixth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the above-mentioned congestion A method of training a control model and/or a method of congestion control as described above.

According to a seventh aspect of the embodiments of the present disclosure, there is provided a computer program product, including computer instructions, and when the computer instructions are executed by at least one processor, the above-mentioned congestion control model training method and/or the above-mentioned congestion control method.

The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

According to the congestion control model of the exemplary embodiment of the present disclosure, the best congestion control strategy can be selected according to the application's preference for transmission performance, so that the transmission performance requirements of different applications can be met without redesigning the objective function and training model; The congestion control method according to the exemplary embodiment of the present disclosure is suitable for performing congestion control on various types of applications, and can achieve a trade-off between throughput, delay and packet loss, and can meet the transmission performance requirements of different types of applications;

The multi-objective reinforcement learning network of the congestion control model of the exemplary embodiment of the present disclosure is able to optimize over the entire preference space of congestion control, which enables the trained model to generate an optimal policy for any given preference, which fundamentally Changing the objective function or utility function of the existing protocol is a fixed design, which has great advantages in meeting different types of applications;

By setting different initial congestion window values for different network bandwidths, network convergence can be effectively accelerated;

By making a method of ending the training round according to changes in the network environment, the training round can be ended at an appropriate time according to changes in network bandwidth utilization, delay, and throughput, thereby improving the training efficiency of the model;

In addition, a win-lose-tie method is proposed to interrupt the training rounds to improve the training quality of the congestion control model, which solves the problem of spurious training interruptions in the application of multi-objective reinforcement learning to the congestion control problem.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings here are incorporated into the specification and constitute a part of the specification, show embodiments consistent with the disclosure, and are used together with the description to explain the principle of the disclosure, and do not constitute an improper limitation of the disclosure.

FIG. 1 shows a schematic diagram of an implementation scenario of a congestion control method and device according to an exemplary embodiment of the present disclosure;

FIG. 2 shows a flowchart of a method for training a congestion control model according to an exemplary embodiment of the present disclosure;

FIG. 3 shows a flowchart of a congestion control method according to an exemplary embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of a congestion control model training method and a congestion control method according to an exemplary embodiment of the present disclosure;

FIG. 5 shows a structural block diagram of a training device for a congestion control model according to an exemplary embodiment of the present disclosure;

FIG. 6 shows a structural block diagram of a congestion control device according to an exemplary embodiment of the present disclosure;

FIG. 7 shows a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed ways

In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

It should be noted that the terms "first" and "second" in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

What needs to be explained here is that "at least one of several items" appearing in this disclosure all means to include "any one of the several items", "a combination of any of the several items", The three categories of "the whole of the several items" are juxtaposed. For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "execute at least one of step 1 and step 2", which means the following three parallel situations: (1) execute step 1; (2) execute step 2; (3) execute step 1 and step 2.

With the rapid development of mobile Internet technology and the increasing number of terminals, many terminal devices are installed with different types of applications, for example, including delay-sensitive applications and throughput-sensitive applications, for delay-sensitive applications such as VoIP Or cloud gaming, which requires a transmission delay as low as a few milliseconds, these applications may not benefit from higher bandwidth; while for throughput-sensitive applications, such as video streaming or file sharing applications, high bandwidth is usually required to obtain more bandwidth. good performance. Depending on the type of application and its requirements on network transmission performance (eg, high throughput, low latency, and low packet loss), congestion control methods may need to follow completely different strategies. As shown in Figure 1, if an application is a file transfer application that is sensitive to throughput, throughput is very important, then the application has high requirements for file transfer throughput and relatively low requirements for delay; if One application is a delay-sensitive real-time streaming application, for which minimizing delay is crucial, it requires low transmission delay to reduce video stuttering, and there will be relatively low packet loss requirements, while for The throughput requirements are relatively low. As the most important protocol in the network transport layer, the computer network congestion control protocol needs to provide high-quality network services for applications with different network performance requirements, that is, the transport layer must not only adapt to changing network conditions, but also adapt to different application requirements , so as to meet the different needs of users and improve the quality of user experience.

This disclosure considers that the objective function of the existing learning-based congestion control model is fixed, and it is difficult to readjust and retrain the model according to different requirements of different applications, and proposes a multi-objective congestion control method, which can be trained well The same congestion control model can meet the transmission performance requirements of different applications. This method uses multi-objective reinforcement learning and preference, and can be applied to different types of applications. Specifically, the application's preference for network transmission performance, current network status information Input to the congestion control model, the congestion control model can give the best action to adjust the size of the congestion window according to the performance requirements of the application.

It should be understood that the congestion control method and/or congestion control device according to the present disclosure may not only be applied to the above-mentioned scenarios, but may also be applied to other suitable scenarios, which are not limited in the present disclosure.

Fig. 2 shows a flowchart of a method for training a congestion control model according to an exemplary embodiment of the present disclosure.

Referring to FIG. 2, in step S101, the communication network environment used in the current training round is initialized.

The communication network environment is used for data transmission between the sending end and the receiving end used in this training round.

In addition, as an example, before the start of the training round, the sending end and the receiving end used in the training round can also be initialized, and the initialized sending end and the receiving end can shake hands so that in the training round, The sending end is controlled to continuously send data packets to the receiving end in the communication network environment, and the receiving end sends a response ACK message to the sending end, and the current network status can be monitored by analyzing the confirmation message of the data packet from the receiving end.

In step S102, the current training round's preference for network transmission performance and the current first network state information S are input into the congestion control model to obtain the predicted action a for adjusting the size of the congestion window that needs to be executed.

As an example, the first network status information may include at least one of the following items: the size of the congestion window, the delay Delay, the packet acknowledgment rate ACK_rate, and the sending rate Sending_rate. For example, the delay, packet acknowledgment rate, and transmission rate may be determined based on ACK messages (ie, acknowledgment messages) fed back by the receiving end. For example, the sending end can be controlled to send data packets to the receiving end under the currently set congestion window size. When the sending end receives the ACK message fed back by the receiving end, it can determine the current delay, packet confirmation rate, and sending rate based on the ACK message. , so as to obtain the current first network state information.

As an example, the preference for network transmission performance may include a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

As an example, a sample may be selected from the preference set as the preference for network transmission performance in this training round. For example, one sample in the preference set can be: the preference for throughput is 0.7, the preference for packet loss rate is 0.2, and the preference for delay is 0.1; another sample in the preference set can be: the preference for throughput The preference is 0.5, the preference for packet loss rate is 0.1, and the preference for delay is 0.4. It should be understood that the preference set may be set according to various requirements of different types of applications on network transmission performance.

In step S103, perform a predicted action to reset the congestion window, and control the sending end to send data packets to the receiving end under the currently set congestion window.

As an example, the action of predicting the current congestion window size may be performed to obtain and set the size of the congestion window that needs to be set.

As an example, the action predicted by the congestion control model can be an action in the action set. As an example, the action set can be {*0.5,-50,-10.0,+0.0,+10.0,*2.0,+50}, for example, *0.5 means the current congestion window size *0.5, as the size of the congestion window to be set; 10.0 means the current congestion window size + 10.0, as the size of the congestion window to be set; -10.0 means the current congestion window size After window size -10.0, it is the size of the congestion window that needs to be set.

In step S104, when the sending end receives the ACK message fed back by the receiving end, according to the action a, the first network state information before the action is executed (that is, input to the congestion control model to predict the first The network state information S), the first network state information S' after the action is executed, and the preference for network transmission performance in this training round, calculate the loss function of the congestion control model.

Here, the first network state information S′ after the above action is performed is the first network state information determined based on the ACK message fed back by the receiving end for the data packet sent to the receiving end in step S103.

As an example, according to the action a, the first network state information S before performing the action, the first network state information S' after performing the action, the reward function r of the action, and the current training round For the preference of network transmission performance, calculate the loss function of the congestion control model.

The reward function of an action is the Reward function used to measure the benefits of actions predicted by the congestion control model. As an example, the reward function of the action may be calculated based on the current training round's preference for network transmission performance and the third network state information after the action is executed. For example, the third network state information may include at least one of the following items: packet loss rate, throughput, and delay. It should be understood that the third network state information after performing the above actions is the third network state information determined based on the ACK message fed back by the receiving end for the data packet sent to the receiving end in step S103.

As an example, the reward function for an action can be determined based on the following triplet [L(Throughput(t)), L(Loss_rate(t)), L(Delay(t))], for example, the reward function for an action can be this The weighted sum of the three quantities, and the weight of each quantity is related to the preference of the network transmission performance in this training round. Among them, t represents time; L(x) represents the activation function, for example, L(x)=(-10) ^-x +1; Throughput(t) can represent the result after normalizing the throughput, for example, The throughput can be normalized by dividing the throughput by the bandwidth; Delay(t) can represent the result of normalizing the delay, for example, by dividing the delay by the timeout threshold timeout, to The delay is normalized; Loss_rate(t) can represent the packet loss rate itself, that is, the packet loss rate does not need to be normalized.

中。相应地，作为示例，可在每个训练回合开始前，初始化重放缓存

As an example, in step S103, perform the action predicted by the congestion control model to reset the congestion window, control the sending end to send data packets to the receiving end under the newly set congestion window, and wait for the acknowledgment message ACKs returned by the receiving end. In step S104, after receiving the ACK, the reward function r of this Step of the training round can be obtained by calculating rtt, comparing the number of packets sent and the number of confirmation messages, etc., and the observed current first network state information S' (i.e. , the first network status information after the action is executed). As an example, the state S before this step, the action a of this step, the reward r obtained after the action is executed, and the new state S' transferred to can be saved in the replay cache in the form of a vector

middle. Accordingly, as an example, the replay cache can be initialized before the start of each training round

In addition, it should be understood that the first network state information after the action in this step can be used as the first network state information before the action in the next step.

As an example, the congestion control model may be constructed based on a reinforcement learning algorithm DQN. As an example, when the preference for network transmission performance includes the degree of preference for multiple network transmission performances, the congestion control model is a multi-objective model. As an example, the congestion control model may be a multi-objective reinforcement learning model.

As an example, the value function (Q function) in the reinforcement learning algorithm may be a value function about actions, first network state information, and preferences for network transmission performance.

As an example, the congestion control model can use the ∈-greedily strategy to sample an action as the predicted action _at , and can use the formula (1) to sample an action. Specifically, the sampled action has a probability of ∈ from the action set A An action chosen at random with probability 1-∈ is the optimal action obtained using the Q-function,

Among them, A represents the action set, ω represents the preference of the network transmission performance in the training round, θ represents the parameters of the Q function, and _st represents the current first network state information.

As an example, the congestion control model may be a multi-objective model. In order to mathematically express the congestion control problem, it is assumed that there are multiple objectives, and each objective may be expressed in the form of an objective function to achieve different objective functions m _i ( O) Overall maximization:

stg _i (O)≤0,i=1,...,a _g

Wherein, m _i (O) represents the objective function of the i-th objective, and i=1,...,m _f , g _i (O) represent the constraint functions of the congestion control problem.

As an example, the objective function of the congestion control model may be a composite objective function related to the following items: reward function, value function, first network state information after performing an action, first network state information before performing an action, action, this The preference of the network transmission performance for each training round, and the best preference in the current network environment.

As an example, the objective function of the congestion control model may be a composite objective function TQ(s, a, ω), and the objective function TQ(s, a, ω) may be expressed as:

r()表示奖励函数，γ表示权重系数，Q()表示值函数，s′表示执行动作后的第一网络状态信息，s表示执行动作前的第一网络状态信息，a表示动作，ω表示本个训练回合对网络传输性能的偏好，ω′表示在当前网络环境下的最佳偏好，

表示动作集合，Ω表示偏好集合。in,

r() represents the reward function, γ represents the weight coefficient, Q() represents the value function, s′ represents the first network state information after the execution of the action, s represents the first network state information before the execution of the action, a represents the action, and ω represents The preference for network transmission performance in this training round, ω′ represents the best preference in the current network environment,

represents the action set, and Ω represents the preference set.

As an example, the loss function of the congestion control model may be calculated based on: a loss function L ^S (θ) for making the value function closer to the maximum reward function, and an auxiliary loss function L ^T (θ).

Here, the auxiliary loss function is proposed considering that the optimal boundary contains a large number of discrete solutions, which causes the curve of the loss function to become unsmooth.

As an example, the loss function of the congestion control model can be expressed as: (1-ε) L ^S (θ) + ε L ^T (θ); where ε is a trade-off index, and the later in a training round is used For the predicted action, the greater the value of ε when calculating the loss function of the congestion control model for this action, 0≤ε≤1.

In other words, in each training round, the initial value of ε is 0, and ε gradually increases from 0 to 1 with the increase of the number of steps, so that the loss function migrates from L ^S (θ) to ^LT (θ).

As an example, the loss function L ^S (θ) can be expressed as:

As an example, the auxiliary loss function L ^T (θ) can be expressed as:

r表示奖励函数，γ表示权重系数，θ表示模型参数，θ_k表示第K步模型的参数，Q()表示值函数，s′表示执行动作后的第一网络状态信息，s表示执行动作前的第一网络状态信息，a表示动作，ω表示本个训练回合对网络传输性能的偏好，ω′表示在当前网络环境下的最佳偏好。in,

r represents the reward function, γ represents the weight coefficient, θ represents the model parameters, θ _k represents the parameters of the K-th step model, Q() represents the value function, s′ represents the first network state information after the execution of the action, and s represents the first network state information before the execution of the action. The first network state information of , a represents the action, ω represents the preference of the network transmission performance in this training round, and ω′ represents the best preference in the current network environment.

In step S105, the congestion control model is trained by adjusting model parameters of the congestion control model according to the loss function.

As an example, the parameter θ of the Q function of the congestion control model may be adjusted according to the loss function.

表示参数θ的梯度量，As an example, stochastic gradient descent on the parameters θ of the Q-function can be performed using equation (3) to update the Q-function of the model, where,

Indicates the gradient amount of the parameter θ,

In step S106, it is determined whether to end the current training round, wherein, if it is determined not to end the current training round, return to step S102.

In addition, as an example, the method for training a congestion control model according to an exemplary embodiment of the present disclosure may further include: when determining to end the current training round, determining whether to end the training process of the congestion control model; During the training process of the congestion control model, return to step S101 to enter the next training round, that is, prepare for the next training round. When it is determined to end the training process of the congestion control model, the training of the congestion control model is stopped, that is, the training of the congestion control model has been completed. For example, whether to end the training process of the congestion control model may be determined according to the prediction effect of the congestion control model or the total training time. In addition, it should be understood that the initial communication network environments of different training rounds may be the same or different, and the preferences of different training rounds for network transmission performance may be the same or different.

As an example, it may be determined whether to end the current training round according to a change situation of the second network state information.

As an example, when the second network status information after the action is executed satisfies the first preset condition, the action can be determined as a victory action; when the second network status information after the action is executed satisfies the second preset condition condition, determine that the action is a failed action; when the continuous number of Win _Num of the winning action reaches the first preset number of times, it is determined to end this training round; when the continuous number of Lose _Num of the failed action reaches the second preset When the total number of times of execution reaches the third preset number of times, it is determined to end the current training round. For example, the first preset number M can be set to 50, the second preset number L can be set to 50, and the third preset number X can be set to 200.

As an example, the second network state information may include: throughput and delay.

As an example, the first preset condition may be: the throughput is at 90%-110% of the bandwidth, and the delay≤0.7×timeout threshold; the second preset condition may be: the throughput is at 50%-70% of the bandwidth, and Latency ≥ 0.7 x timeout threshold.

This disclosure considers that determining the end of a training round with a fixed number of steps or a fixed time will lead to a pseudo-interruption state and introduce a considerable deviation in training, and proposes a method for dynamically ending a training round, which is determined by the winner or loser. Inspired by the idea of draw and draw games, decisions are made according to changes in the network environment. This method can end the round at an appropriate time according to changes in network bandwidth utilization, delay and throughput, thereby improving the training efficiency of the model. The present disclosure proposes a method that enables the most appropriate way to end a training session. A training round is a complete training process of the reinforcement learning algorithm. In this process, a series of actions are sequentially selected at each moment according to the network state and congestion control strategy. The length of the training round has an effect on whether the best model will be learned. big impact. As an example, in each training round, the initial values of Win _Num and Lose _Num are both 0, whenever a predicted action is determined as a winning action, the value of Win _Num is +1, and the value of Lose _Num Reset to 0, whenever a predicted action is determined to be a failed action, the value of Lose _Num will be +1, and the value of Win _Num will be reset to 0; if the number of consecutive victories of the training round Win _Num ≥ M , then stop the training round with victory, if the number of consecutive failures Loss _Num ≥ L, stop the training round with failure, if the current training round has gone through X steps, then end the training round with a tie, in other words, in the current Win _Num has not reached M and Lose _Num has not reached L before the training round has taken X steps.

In addition, this disclosure considers that the initial congestion window (init-cwnd, that is, the size of the congestion window set when the sender starts sending data) has a significant impact on the model convergence speed, but init-cwnd is usually set to a fixed value in related technologies , therefore, there is a problem that fast convergence cannot be achieved due to different link capacities in different network scenarios. The disclosure considers that the congestion control method cannot achieve fast convergence in different network scenarios due to the large differences in different links, and proposes to dynamically design the initial congestion window according to the link bandwidth, thereby improving the convergence speed of the congestion control method.

As an example, the method for training a congestion control model according to an exemplary embodiment of the present disclosure may further include: initializing the size of the congestion window; wherein, the step of initializing the size of the congestion window may include: estimating the bandwidth of the communication network, and based on The estimated bandwidth determines the initial size of the congestion window.

As an example, the step of estimating the bandwidth of the communication network may include: determining the total number Num _ack of ACK messages fed back by the receiving end for the N data packets sent by the sending end; dividing the total number Num _ack by N to obtain The average value Num _ave determines the bandwidth bw _i of said communication network.

As an example, the estimated network bandwidth bw _i corresponding to Num _ave can be found from the predefined bandwidth combination Bandwidth by formula (4):

表示一组在公式(5)中定义的关于接收速率的特征函数，(c_j-1,c_j)表示一组接收速率，in,

Represents a set of characteristic functions about the reception rate defined in formula (5), (c _j-1 ,c _j ) represents a set of reception rates,

As an example, the initial size _Winit-cwnd of the congestion window can be determined based on the estimated bandwidth bw _i by formula (6), where b is a coefficient, for example, b can be set to 2.5 according to experimental data to achieve better The fitting rate and learning effect of ,

_Winit-cwnd = b*bw _i (6)

As an example, the size of the congestion window may be initialized before the current training round, so that _Winit-cwnd is used when the sender sends the data packet for the first time in the current training round. As another example, the size of the congestion window may be initialized based on the network status information of the first N steps of the current training round.

It should be understood that if the communication network environments of multiple training rounds are the same, the multiple training rounds may share the same _Winit-cwnd .

Fig. 3 shows a flowchart of a congestion control method according to an exemplary embodiment of the present disclosure.

Referring to FIG. 3 , in step S201 , the current first network state information and the current application's preference for network transmission performance are obtained.

Here, the application is an application that uses the congestion control method for data transmission.

In step S202, input the obtained first network state information and the preference into the congestion control model to obtain the predicted action for adjusting the size of the congestion window that needs to be executed.

In step S203, a predicted action is performed to reset the congestion window. It should be understood that the congestion control method according to the exemplary embodiment of the present disclosure may be repeatedly executed to adjust the size of the congestion window in real time according to the network status.

As an example, the first network state information may include at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and transmission rate. For example, the delay, packet acknowledgment rate, and transmission rate may be determined based on the ACK message fed back by the receiving end.

As an example, the preference for network transmission performance may include a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

As an example, the congestion control model may be constructed based on a reinforcement learning algorithm; wherein, a value function in the reinforcement learning algorithm may be a value function about actions, first network state information, and preferences for network transmission performance.

As an example, the congestion control model may be trained by using the training method described in the above exemplary embodiments.

As an example, the congestion control method according to an exemplary embodiment of the present disclosure may further include: initializing the size of the congestion window; wherein, the step of initializing the size of the congestion window may include: estimating the bandwidth of the communication network, and based on the estimated bandwidth, Determines the initial size of the congestion window. As an example, the size of the congestion window may be initialized when the application starts data transmission with the receiving end.

As an example, the step of estimating the bandwidth of the communication network may include: determining the total number of ACK messages fed back by the receiving end for the N data packets sent; bandwidth.

The specific processing in the congestion control method according to the exemplary embodiment of the present disclosure has been described in detail in the above embodiment of the related congestion control model training method, and will not be described in detail here.

Fig. 4 shows a schematic diagram of a congestion control model training method and a congestion control method according to an exemplary embodiment of the present disclosure.

As shown in Figure 4, when training the congestion control model, when starting a training round, the sending end can send data to the receiving end based on the initial congestion window size set by the bandwidth, so as to improve the convergence speed of the network; and, the network environment can be used The training data generated interactively is used to train the congestion control model using the multi-objective reinforcement learning DQN network; in addition, the training round interruption algorithm can also be used to solve the false interruption problem of the training round. By using a well-trained congestion control model to implement the congestion control method, an optimal congestion control strategy can be generated for any specified preference, thereby meeting the needs of different types of applications.

According to the training method of the congestion control model according to the exemplary embodiment of the present disclosure, the performance of different network indicators can be improved according to different preference settings without resetting the objective function or reward function, so as to meet the transmission performance requirements of different types of applications , reducing the training time and training cost of the congestion control model;

Moreover, in order to improve the convergence of the model and solve the problem that the convergence speed of the current congestion control model is relatively slow, a method for dynamically initializing the congestion window is also proposed according to an exemplary embodiment of the present disclosure, and different initialization parameters are set for different network bandwidths. The size of the congestion window, which can improve the convergence speed of the congestion control model in different network environments;

Moreover, in order to solve the problem of false training interruptions in the application of multi-objective reinforcement learning to congestion control problems, an exemplary embodiment of the present disclosure also proposes a win-lose-tie interruption round algorithm to improve the training quality of the congestion control model , so that multi-objective reinforcement learning can be applied to congestion control problems and improve the training efficiency of the algorithm;

In addition, the congestion control method according to the exemplary embodiment of the present disclosure has relatively excellent experimental performance. This method achieves a trade-off between high throughput and low latency, and accordingly exhibits excellent congestion control capabilities in different network environments of 12Mbps and 50Mbps. For different cellular network scenarios, by setting different preferences, the congestion control method according to the exemplary embodiments of the present disclosure can meet the transmission performance requirements of different types of applications, and achieve an optimal trade-off between different performance indicators, that is, You can set different preferences to meet the performance requirements of different types of applications in dynamic network scenarios.

Fig. 5 shows a structural block diagram of a training device for a congestion control model according to an exemplary embodiment of the present disclosure.

As shown in FIG. 5 , the training device 10 of the congestion control model according to an exemplary embodiment of the present disclosure includes: an environment initialization unit 101, a prediction unit 102, a congestion window setting unit 103, a loss function calculation unit 104, a training unit 105, and a round The determination unit 106 is ended.

Specifically, the environment initialization unit 101 is configured to initialize the communication network environment used in the current training session.

The predicting unit 102 is configured to input the current training round's preference for network transmission performance and the current first network state information into the congestion control model to obtain the predicted action for adjusting the size of the congestion window that needs to be executed.

The congestion window setting unit 103 is configured to perform predicted actions to reset the congestion window, and control the sending end to send data packets to the receiving end under the currently set congestion window.

The loss function calculation unit 104 is configured to, when the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before executing the action, the first network state information after executing the action, As well as the preference, a loss function of the congestion control model is computed.

The training unit 105 is configured to train the congestion control model by adjusting model parameters of the congestion control model according to the loss function.

The round end determination unit 106 is configured to determine whether to end the current training round, wherein, when it is determined not to end the current training round, the prediction unit 102 will use the current training round's preference for network transmission performance and the current first network state The information is fed into the congestion control model to get the predicted actions that need to be performed to adjust the size of the congestion window.

As an example, the first network state information may include at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and transmission rate; wherein, the delay, packet acknowledgment rate, and transmission rate are based on the receiving end feedback The ACK message is OK.

As an example, the preference for network transmission performance may include a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

As an example, the round end determining unit 106 may be configured to determine whether to end the current training round according to the change of the second network state information.

As an example, the round end determination unit 106 may be configured to determine that the action is a winning action when the second network status information after the action is executed satisfies the first preset condition; When the network state information satisfies the second preset condition, it is determined that the action is a failed action; when the continuous number of victorious actions reaches the first preset number of times, it is determined to end this training round; when the continuous number of failed actions reaches When the second preset number of times, it is determined to end the current training round; when the total number of times of performing actions reaches the third preset number of times, it is determined to end the current training round.

As an example, the second network status information may include: throughput and delay; the first preset condition is: the throughput is 90%-110% of the bandwidth, and the delay is ≤0.7×timeout threshold; the second preset condition is: throughput The volume is between 50%-70% of the bandwidth and the latency is ≥0.7×timeout threshold.

As an example, the training device 10 of the congestion control model may further include: a window initialization unit (not shown), the window initialization unit is configured to initialize the size of the congestion window; wherein, the window initialization unit is configured to estimate the communication network bandwidth, and based on the estimated bandwidth, determine the initial size of the congestion window.

As an example, the window initialization unit may be configured to determine the total number of ACK messages fed back by the receiving end for the N data packets sent by the sending end; and determine the communication network based on the average value obtained by dividing the total number by N bandwidth.

As an example, the loss function calculation unit 104 may be configured to, when the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before the action is executed, and the first network state information after the action is executed. The network state information, the reward function of the action, and the preference calculate the loss function of the congestion control model.

As an example, the reward function of the action may be calculated based on the preference and third network state information after performing the action; wherein the third network state information includes at least one of the following items: Packet rate, throughput, and latency.

As an example, the congestion control model may be constructed based on a reinforcement learning algorithm; wherein, a value function in the reinforcement learning algorithm is a value function about an action, first network state information, and a preference for network transmission performance.

As an example, the action predicted by the congestion control model, with a probability of ∈ is an action randomly selected from the action set, and with a probability of 1-∈ is an optimal action obtained by using a value function.

As an example, the loss function of the congestion control model may be calculated based on: a loss function L ^S (θ) for making the value function closer to the maximum reward function, and an auxiliary loss function L ^T (θ).

As an example, the loss function of the congestion control model can be expressed as: (1-ε) L ^S (θ) + ε L ^T (θ); where ε is a trade-off index, and the later in a training round is used For the predicted action, the greater the value of ε when calculating the loss function of the congestion control model for this action, 0≤ε≤1.

和/或As an example, the loss function L ^S (θ) can be expressed as:

and / or

The auxiliary loss function L ^T (θ) is expressed as:

r表示奖励函数，γ表示权重系数，θ表示模型参数，θ_k表示第K步模型的参数，Q()表示值函数，s′表示执行动作后的第一网络状态信息，s表示执行动作前的第一网络状态信息，a表示动作，ω表示本个训练回合对网络传输性能的偏好，ω′表示在当前网络环境下的最佳偏好。in,

r represents the reward function, γ represents the weight coefficient, θ represents the model parameters, θ _k represents the parameters of the K-th step model, Q() represents the value function, s′ represents the first network state information after the execution of the action, and s represents the first network state information before the execution of the action. The first network state information of , a represents the action, ω represents the preference of the network transmission performance in this training round, and ω′ represents the best preference in the current network environment.

As an example, the objective function of the congestion control model may be a composite objective function related to the following items: reward function, value function, first network state information after performing an action, first network state information before performing an action, action, this The preference of the network transmission performance for each training round, and the best preference in the current network environment.

As an example, the objective function of the congestion control model may be a composite objective function TQ(s, a, ω), and the objective function TQ(s, a, ω) is expressed as:

r()表示奖励函数，γ表示权重系数，Q()表示值函数，s′表示执行动作后的第一网络状态信息，s表示执行动作前的第一网络状态信息，a表示动作，ω表示本个训练回合对网络传输性能的偏好，ω′表示在当前网络环境下的最佳偏好，

表示动作集合，Ω表示偏好集合。in,

r() represents the reward function, γ represents the weight coefficient, Q() represents the value function, s′ represents the first network state information after the execution of the action, s represents the first network state information before the execution of the action, a represents the action, and ω represents The preference for network transmission performance in this training round, ω′ represents the best preference in the current network environment,

represents the action set, and Ω represents the preference set.

As an example, the congestion control model training device 10 according to an exemplary embodiment of the present disclosure may further include: a training end determination unit (not shown), the training end determination unit is configured to determine whether to End the training process of the congestion control model, wherein, when it is determined not to end the training process of the congestion control model, the environment initialization unit 101 initializes the communication network environment used in the current training round to enter the next training round.

Fig. 6 shows a structural block diagram of a congestion control device according to an exemplary embodiment of the present disclosure.

As shown in FIG. 6 , the congestion control device 20 according to the exemplary embodiment of the present disclosure includes: an acquisition unit 201 , a prediction unit 202 , and a congestion window setting unit 203 .

Specifically, the acquiring unit 201 is configured to acquire the current first network status information and the current application's preference for network transmission performance.

The predicting unit 202 is configured to input the obtained first network state information and the preference into the congestion control model, and obtain the predicted action for adjusting the size of the congestion window that needs to be executed.

The congestion window setting unit 203 is configured to perform predicted actions to reset the congestion window.

As an example, the first network state information may include at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and transmission rate; wherein, the delay, packet acknowledgment rate, and transmission rate are based on the receiving end feedback The ACK message is OK.

As an example, the preference for network transmission performance may include a degree of preference for at least one of the following items: throughput, packet loss rate, and delay.

As an example, the congestion control device 20 may further include: a window initialization unit (not shown), the window initialization unit is configured to initialize the size of the congestion window; wherein, the window initialization unit is configured to estimate the bandwidth of the communication network, and based on the estimated The estimated bandwidth determines the initial size of the congestion window.

As an example, the window initialization unit may be configured to determine the total number of ACK messages fed back by the receiving end for the sent N data packets; and determine the bandwidth of the communication network according to an average value obtained by dividing the total number by N.

As an example, the congestion control model may be constructed based on a reinforcement learning algorithm; wherein, a value function in the reinforcement learning algorithm is a value function about an action, first network state information, and a preference for network transmission performance.

As an example, the congestion control model may be trained using the training device 10 of the above exemplary embodiment.

With regard to the device in the above embodiments, the specific manner in which each unit executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

In addition, it should be understood that each unit in the training device 10 and the congestion control device 20 of the congestion control model according to the exemplary embodiment of the present disclosure may be implemented as hardware components and/or software components. Those skilled in the art may implement each unit, for example, by using a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC) according to the defined processing performed by each unit.

FIG. 7 shows a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. 7, the electronic device 30 includes: at least one memory 301 and at least one processor 302, the at least one memory 301 stores a set of computer-executable instructions, when the set of computer-executable instructions is executed by at least one processor 302 , executing the congestion control model training method and/or the congestion control method as described in the foregoing exemplary embodiments.

As an example, the electronic device 30 may be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above-mentioned set of instructions. Here, the electronic device 30 is not necessarily a single electronic device, but may also be any assembly of devices or circuits capable of individually or jointly executing the above-mentioned instructions (or instruction sets). Electronic device 30 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote (eg, via wireless transmission).

In electronic device 30, processor 302 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a special purpose processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 302 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

The processor 302 can execute instructions or codes stored in the memory 301, wherein the memory 301 can also store data. Instructions and data may also be sent and received over the network via the network interface device, which may employ any known transmission protocol.

The memory 301 can be integrated with the processor 302, for example, RAM or flash memory is arranged in an integrated circuit microprocessor or the like. Additionally, storage 301 may comprise a separate device, such as an external disk drive, storage array, or any other storage device usable by the database system. Memory 301 and processor 302 may be operatively coupled, or may communicate with each other, such as through an I/O port, network connection, etc., such that processor 302 can read files stored in the memory.

In addition, the electronic device 30 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 30 may be connected to each other via a bus and/or a network.

According to an exemplary embodiment of the present disclosure, there may also be provided a computer-readable storage medium storing instructions, wherein, when the instructions are executed by at least one processor, at least one processor is prompted to execute the method described in the above-mentioned exemplary embodiments. A congestion control model training method and/or a congestion control method. Examples of computer readable storage media herein include: Read Only Memory (ROM), Random Access Programmable Read Only Memory (PROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Random Access Memory (RAM) , Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Flash Memory, Non-volatile Memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM , DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or Optical Memory, Hard Disk Drive (HDD), Solid State Hard disks (SSD), memory cards (such as MultiMediaCards, Secure Digital (SD) or Extreme Digital (XD) cards), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other means configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and to provide said computer program and any associated data, data files and data structures to the processor or the computer to enable the processor or the computer to execute the computer program. The computer program in the above-mentioned computer-readable storage medium can run in an environment deployed in computer equipment such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data and data files and data structures are distributed over network-connected computer systems so that the computer programs and any associated data, data files and data structures are stored, accessed and executed in a distributed fashion by one or more processors or computers.

According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor to complete the method for training a congestion control model as described in the above exemplary embodiments and/or or congestion control methods.

Other embodiments of the present disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (42)

1. A training method for a congestion control model, comprising: Initialize the communication network environment used in the current training round; Select a sample from the preference set as the preference for network transmission performance in this training round; Input this training round's preference for network transmission performance and the current first network state information into the congestion control model to obtain the predicted action to adjust the size of the congestion window; Execute predicted actions to reset the congestion window, and control the sender to send packets to the receiver under the currently set congestion window; When the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before executing the action, the first network state information after executing the action, the reward function of the action, and The preference is to calculate the loss function of the congestion control model, wherein the reward function of the action is calculated based on the preference of the network transmission performance in this training round and the third network state information after the action is executed ; By adjusting the model parameters of the congestion control model according to the loss function, the congestion control model is trained, and it is determined whether to end this training round, wherein, when it is determined not to end this training round, return to execute this A training round's preference for network transmission performance and the current first network state information are input into the congestion control model to obtain the predicted action steps for adjusting the size of the congestion window; When determining to end this training round, determine whether to end the training process of the congestion control model; When it is determined not to end the training process of the congestion control model, return to the steps of initializing the communication network environment used in the current training round and selecting a sample from the preference set as the preference of the network transmission performance for this training round, to enter the next training round, Among them, different training rounds have different preferences for network transmission performance. 2. The method according to claim 1, wherein the first network status information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and sending rate; Wherein, the delay, the packet acknowledgment rate, and the sending rate are determined based on the ACK message fed back by the receiving end. 3. The method according to claim 1, wherein the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay. 4. The method according to claim 1, wherein the step of determining whether to end this training round comprises: According to the change of the second network state information, it is determined whether to end the current training round. 5. The method according to claim 4, wherein, according to the variation of the second network state information, the step of determining whether to end this training round comprises: When the second network status information after performing the action satisfies the first preset condition, determine the action as a winning action; When the second network status information after performing the action satisfies a second preset condition, determine that the action is a failed action; When the number of consecutive victorious actions reaches the first preset number of times, it is determined to end this training round; When the continuous number of failed actions reaches the second preset number of times, it is determined to end this training round; When the total number of times of performing actions reaches the third preset number of times, it is determined to end the current training round. 6. The method according to claim 5, wherein the second network state information comprises: throughput and delay; The first preset condition is: the throughput is 90%-110% of the bandwidth, and the delay is ≤0.7×timeout threshold; The second preset condition is: the throughput is 50%-70% of the bandwidth, and the delay is greater than or equal to 0.7×timeout threshold. 7. The method according to claim 1, further comprising: initializing the size of the congestion window; Wherein, the step of initializing the size of the congestion window includes: estimating the bandwidth of the communication network, and determining the initial size of the congestion window based on the estimated bandwidth. 8. The method according to claim 7, wherein the step of estimating the bandwidth of the communication network comprises: Determine the total number of ACK messages fed back by the receiving end for the N data packets sent by the sending end; The bandwidth of the communication network is determined according to an average value obtained by dividing the total number by N. 9. The method according to claim 1, wherein the third network status information includes at least one of the following items: packet loss rate, throughput, and delay. 10. The method according to claim 1, wherein the congestion control model is constructed based on a reinforcement learning algorithm; Wherein, the value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preference for network transmission performance. 11. The method according to claim 10, wherein the action predicted by the congestion control model has a probability of ∈ is an action randomly selected from the action set, and the probability of 1-∈ is obtained using a value function optimal action. 12. The method according to claim 10, wherein the loss function of the congestion control model is based on: in order to make the value function closer to the loss function L ^S (θ) of the maximum reward function, and an auxiliary loss function L ^T (θ) calculated. 13. The method according to claim 12, wherein the loss function of the congestion control model is expressed as: (1ε) L ^S (θ)+ε L ^T (θ); Wherein, ε is a trade-off index, the later the predicted action in a training round, the greater the value of ε when calculating the loss function of the congestion control model for this action, 0≤ε≤1. 14. The method according to claim 10, wherein the objective function of the congestion control model is a composite objective function about the following items: a reward function, a value function, the first network state information after performing an action, and performing an action The previous first network state information, action, preference for network transmission performance in this training round, and the best preference in the current network environment. 15. A congestion control method, comprising: Obtain the current first network state information and the current application's preference for network transmission performance; inputting the obtained first network state information and the preference into the congestion control model, and obtaining the predicted action for adjusting the size of the congestion window; Execute the predicted action to reset the congestion window, Wherein, the congestion control model is applicable to different types of applications, and different types of applications have different preferences for network transmission performance, Wherein, the congestion control model is obtained by training using the training method described in any one of claims 1 to 14. 16. The method according to claim 15, wherein the first network status information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and transmission rate; Wherein, the delay, the packet acknowledgment rate, and the sending rate are determined based on the ACK message fed back by the receiving end. 17. The method according to claim 15, wherein the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay. 18. The method according to claim 15, further comprising: initializing the size of the congestion window; Wherein, the step of initializing the size of the congestion window includes: estimating the bandwidth of the communication network, and determining the initial size of the congestion window based on the estimated bandwidth. 19. The method according to claim 18, wherein the step of estimating the bandwidth of the communication network comprises: Determine the total number of ACK messages fed back by the receiving end for the N data packets sent; The bandwidth of the communication network is determined according to an average value obtained by dividing the total number by N. 20. The method according to claim 15, wherein the congestion control model is constructed based on a reinforcement learning algorithm; Wherein, the value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preference for network transmission performance. 21. A training device for a congestion control model, comprising: The environment initialization unit is configured to initialize the communication network environment used in the current training round, and select a sample from the preference set as the preference for network transmission performance in this training round; The prediction unit is configured to input the current training round's preference for network transmission performance and the current first network state information into the congestion control model, and obtain the predicted action for adjusting the size of the congestion window that needs to be executed; The congestion window setting unit is configured to perform predicted actions to reset the congestion window, and control the sending end to send data packets to the receiving end under the currently set congestion window; The loss function calculation unit is configured to, when the sending end receives the ACK message fed back by the receiving end, according to the action, the first network state information before performing the action, the first network state information after performing the action, The reward function of the action and the preference calculate the loss function of the congestion control model, wherein the reward function of the action is based on the preference of the network transmission performance in this training round and the The third network status information is calculated; a training unit configured to train the congestion control model by adjusting model parameters of the congestion control model according to the loss function; The round end determination unit is configured to determine whether to end this training round, wherein, when it is determined not to end this training round, the prediction unit will use this training round's preference for network transmission performance and the current first network state information Input to the congestion control model to obtain the predicted action to adjust the size of the congestion window; The training end determination unit is configured to determine whether to end the training process of the congestion control model when it is determined to end the current training round, wherein, when it is determined not to end the training process of the congestion control model, the environment initialization unit initializes the current The communication network environment used in the training round, and select a sample from the preference set as the preference for network transmission performance in this training round to enter the next training round, Among them, different training rounds have different preferences for network transmission performance. 22. The device according to claim 21, wherein the first network status information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and sending rate; Wherein, the delay, the packet acknowledgment rate, and the sending rate are determined based on the ACK message fed back by the receiving end. 23. The device according to claim 21, wherein the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay. 24. The device according to claim 21, wherein the round end determination unit is configured to determine whether to end the current training round according to the change of the second network state information. 25. The device according to claim 24, wherein the round end determining unit is configured to determine that the action is a winning action when the second network status information after the action is executed satisfies a first preset condition ; When the second network state information after performing the action satisfies the second preset condition, determine that the action is a failed action; when the number of consecutive victorious actions reaches the first preset number, determine to end the training A round; when the consecutive number of failed actions reaches the second preset number of times, it is determined to end the current training round; when the total number of executed actions reaches the third preset number of times, it is determined to end the current training round. 26. The device according to claim 25, wherein the second network status information includes: throughput and delay; The first preset condition is: the throughput is 90%-110% of the bandwidth, and the delay is ≤0.7×timeout threshold; The second preset condition is: the throughput is 50%-70% of the bandwidth, and the delay is greater than or equal to 0.7×timeout threshold. 27. The device of claim 21, further comprising: a window initialization unit configured to initialize the size of the congestion window; Wherein, the window initialization unit is configured to estimate the bandwidth of the communication network, and determine the initial size of the congestion window based on the estimated bandwidth. 28. The device according to claim 27, wherein the window initialization unit is configured to determine the total number of ACK messages fed back by the receiving end for the N data packets sent by the sending end; and divide the total number by N The resulting average value determines the bandwidth of the communication network. 29. The device according to claim 21, wherein the third network status information includes at least one of the following items: packet loss rate, throughput, and delay. 30. The device according to claim 21, wherein the congestion control model is constructed based on a reinforcement learning algorithm; Wherein, the value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preference for network transmission performance. 31. The device according to claim 30, wherein the action predicted by the congestion control model has a probability of ∈ is an action randomly selected from the action set, and the probability of 1-∈ is obtained using a value function optimal action. 32. The device according to claim 30, wherein the loss function of the congestion control model is based on: a loss function L ^S (θ) in order to make the value function closer to the maximum reward function, and an auxiliary loss function L ^T (θ) calculated. 33. The device according to claim 32, wherein the loss function of the congestion control model is expressed as: (1ε) L ^S (θ)+ε L ^T (θ); Wherein, ε is a trade-off index, the later the predicted action in a training round, the greater the value of ε when calculating the loss function of the congestion control model for this action, 0≤ε≤1. 34. The device according to claim 30, wherein the objective function of the congestion control model is a composite objective function related to the following items: a reward function, a value function, the first network state information after performing an action, and performing an action The previous first network state information, action, preference for network transmission performance in this training round, and the best preference in the current network environment. 35. A congestion control device, comprising: an acquisition unit configured to acquire current first network status information and current application preference for network transmission performance; A predicting unit configured to input the obtained first network state information and the preference into the congestion control model, and obtain the predicted action for adjusting the size of the congestion window that needs to be executed; a congestion window setting unit configured to perform predicted actions to reset the congestion window, Wherein, the congestion control model is applicable to different types of applications, and different types of applications have different preferences for network transmission performance, Wherein, the congestion control model is trained by using the training device according to any one of claims 21 to 34. 36. The device according to claim 35, wherein the first network state information includes at least one of the following items: the size of the congestion window, delay, packet acknowledgment rate, and transmission rate; Wherein, the delay, the packet acknowledgment rate, and the sending rate are determined based on the ACK message fed back by the receiving end. 37. The device according to claim 35, wherein the preference for network transmission performance includes a degree of preference for at least one of the following items: throughput, packet loss rate, and delay. 38. The device of claim 35, further comprising: a window initialization unit configured to initialize the size of the congestion window; Wherein, the window initialization unit is configured to estimate the bandwidth of the communication network, and determine the initial size of the congestion window based on the estimated bandwidth. 39. The device according to claim 38, wherein the window initialization unit is configured to determine the total number of ACK messages fed back by the receiving end for the N data packets sent; and divide the total number by N to obtain The average value determines the bandwidth of the communication network. 40. The device according to claim 35, wherein the congestion control model is constructed based on a reinforcement learning algorithm; Wherein, the value function in the reinforcement learning algorithm is a value function about actions, first network state information, and preference for network transmission performance. 41. An electronic device, characterized by comprising: at least one processor; at least one memory storing computer-executable instructions, Wherein, when the computer-executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the congestion control model training method according to any one of claims 1 to 14 and/or as described in The congestion control method according to any one of claims 15 to 20. 42. A computer-readable storage medium, characterized in that, when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute any one of claims 1 to 14. The training method of the congestion control model described in claim 1 and/or the congestion control method described in any one of claims 15 to 20.

Images