CN116321293A - Edge Computing Offloading and Resource Allocation Method Based on Multi-agent Reinforcement Learning - Google Patents

Abstract

The invention discloses an edge computing offloading and resource allocation method based on multi-agent reinforcement learning, including: constructing a mobile edge computing offloading and resource allocation model based on complex scenarios of multiple mobile users and multiple edge servers, and setting optimization problems based on system overhead Objective function and constraints; Model the optimization problem as a Markov decision process, and set the state space, action space and reward function in deep reinforcement learning; use the method based on multi-agent deep reinforcement learning to find the best Optimize the unloading strategy and resource allocation strategy, and optimize the objective function, and at the same time use the NoisyNet method to add Gaussian noise to the output layer of the Actor network, improve the network model exploration efficiency, and improve the optimization effect; the method disclosed in the present invention integrates multi-agent DDPG and NoisyNet can effectively reduce the total system overhead, improve the optimization effect, and improve the overall experience of users in the system.

Description

Edge Computing Offloading and Resource Allocation Method Based on Multi-agent Reinforcement Learning

technical field

The invention relates to the technical fields of wireless communication network and mobile edge computing, in particular to an edge computing unloading and resource allocation method based on multi-agent reinforcement learning.

Background technique

In recent years, with the rapid development of the Internet of Things and 5G, low-latency, high-speed Internet of Everything has become possible, and a series of emerging interactive applications have emerged, such as virtual reality, augmented display, face recognition, intelligent services, and autonomous driving. wait. Due to the limitations of the computing of IoT devices or mobile smart devices, some computing tasks have to be offloaded to cloud servers with sufficient computing power, which promotes the development of mobile cloud computing; however, the server of cloud computing is centralized, and its computing Resources and bandwidth are limited. When faced with a large number of network access devices, it is easy to cause data processing to be untimely, difficult to meet the needs of all user equipment, and even easy to cause system failure, and the distance from users is long, resulting in high propagation delay. It is difficult to meet the effective processing of tasks with high latency requirements.

The proposal of mobile edge computing (MEC) can effectively make up for these shortcomings. In the MEC system, the MEC server can provide more powerful computing capabilities than the local device. Although it is not as good as the cloud server, it is closer to the device. At the same time, the distributed MEC server The structure prevents the traffic of the core network from being congested; mobile edge computing handles the time delay generated by mobile devices by harvesting a large amount of idle computing power and storage space distributed on the edge of the network and using these resources on mobile devices. Sensitive or computationally complex tasks. In the related issues of MEC, the decision-making of computing offloading and the allocation of resources are the key technologies to determine whether MEC can play a good role. Therefore, the research on computing offloading and resource allocation is an urgent requirement to improve the performance of MEC, which has very important research significance. .

In the current research work, edge computing offloading scenarios are mainly divided into three types: single-user single-edge server, multi-user single-edge server, and multi-user multi-edge server; The goal is to minimize the delay or minimize the weighted sum of the two, and set up an optimization problem and solve it with constraints such as the energy of the user terminal, computing resources, computing resources of the edge server, and the maximum allowable delay of the task, so as to obtain the optimal strategy; However, this kind of optimization problem is usually NP-hard, especially when the network scale is large, even through heuristic algorithms such as genetic algorithm and particle swarm algorithm, it still takes a long time to obtain the optimal strategy; in addition, the dynamics of the network Changes require the central node to continuously solve complex optimization problems, and it is difficult to adaptively track network dynamic changes.

In recent years, with the rapid development of artificial intelligence technology, reinforcement learning algorithms have received widespread attention; the agent and the environment continuously interact to obtain rewards to guide behaviors, so that the agent can make better action decisions over time and obtain greater The reward is approximately optimal; since reinforcement learning evaluates actions and corrects action choices based on feedback, it does not need to rely on prior knowledge and can adaptively track environmental changes, which is suitable for solving decision-making optimization problems in more complex scenarios ; Therefore, edge computing offloading and resource allocation decision optimization can be performed with the help of reinforcement learning algorithms to minimize system overhead and improve user experience. The present invention improves on the traditional reinforcement learning algorithm DDPG to adapt to more complex scenarios.

Contents of the invention

The purpose of this section is to outline some aspects of embodiments of the invention and briefly describe some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and titles of this application, to avoid obscuring the purpose of this section, abstract and titles, and such simplifications or omissions should not be used to limit the scope of the invention.

In view of the above problems, the present invention has been proposed.

Therefore, the technical problem solved by the present invention is: the traditional reinforcement learning method is expensive in mobile edge computing offloading and resource allocation scenarios, and the model is difficult to quickly converge in complex scenarios.

In order to solve the above technical problems, the present invention provides the following technical solution: a method for edge computing offloading and resource allocation based on multi-agent reinforcement learning, including: constructing a mobile edge computing offloading and resource allocation model according to complex scenarios with multiple mobile users and multiple edge servers , and set the parameters of the model; based on the system overhead of mobile edge computing offloading and resource allocation calculated by the model, set the objective function and constraint conditions of the optimization problem; model the optimization problem as a Markov decision process , and set the state space, action space and reward function in deep reinforcement learning; use the method based on multi-agent deep reinforcement learning to find the optimal unloading strategy and resource allocation strategy for each mobile user, and optimize the objective function, At the same time, the NoisyNet method is used to add Gaussian noise to the output layer of the Actor network to improve the exploration efficiency of the network model and improve the optimization effect.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, wherein: the mobile edge computing offloading and resource allocation model includes a system model, a task model, a mobility model, and a computing Model;

The system model includes M edge servers and N mobile user equipments, the edge servers are deployed next to wireless access points, each wireless access point independently covers a cell, and the mobile user equipment can pass through the cell's wireless The access point offloads computing tasks to the edge server of the cell, requests computing resources, and connects and transmits data between the wireless access points through the base station;

其中，L_n表示任务的数据量，X_n表示计算任务所需的CPU循环数，/>

表示完成任务所需的最大允许时延；The task model includes that each mobile user equipment randomly generates a computing task at each moment, and the attributes of the generated computing task are represented by a triplet A _n , namely

Among them, L _n represents the amount of data of the task, X _n represents the number of CPU cycles required to calculate the task, />

Indicates the maximum allowable delay required to complete the task;

The mobility model includes modeling user mobility using discrete random hops, and using an average dwell time to represent the strength between hops;

的计算包括，The probability density function of the average dwell time

The calculation includes,

表示用户实际驻留时间。Among them, β _n represents the average residence time of mobile user n,

Indicates the actual dwell time of the user.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, it also includes,

The calculation model includes the total overhead of the mobile user equipment under different offloading decisions and resource allocation strategies, and the total overhead includes delay, energy consumption and resource cost;

The calculation of the mobile user equipment total cost Q _n includes,

表示没有发生任务迁移时移动用户设备的总开销，/>

表示发生任务迁移时移动用户设备的总开销，ω₁表示时延系数，ω₂表示能耗系数，ω₃表示资源成本系数，/>

表示本地计算和边缘计算的最大时延，/>

表示本地计算和边缘计算加上额外迁移时延的最大时延，E_n表示本地计算和边缘计算的能耗，f_m→n表示边缘服务器m给用户n分配的计算资源，P_βn表示事件发生的概率，T_mn表示边缘处理任务的总时延；in,

Indicates the total overhead of moving user equipment when no task migration occurs, />

Indicates the total overhead of mobile user equipment when task migration occurs, ω ₁ represents the delay coefficient, ω ₂ represents the energy consumption coefficient, ω ₃ represents the resource cost coefficient, />

Indicates the maximum latency of local computing and edge computing, />

Indicates the maximum delay of local computing and edge computing plus additional migration delay, E _n represents the energy consumption of local computing and edge computing, f _m→n represents the computing resources allocated by edge server m to user n, P _βn represents the occurrence of events The probability of , T _mn represents the total delay of edge processing tasks;

The performance of the system is measured by using the overhead expectation, and the calculation of the expected total overhead of the mobile user equipment includes,

表示时延、能耗以及计算资源的平均开销。in,

Indicates the average overhead of latency, energy consumption, and computing resources.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, wherein: the calculation of the objective function of the optimization problem includes,

Among them, γ _n represents the proportion of task unloading, I(x _n ) represents the indicator function, and x _n represents the user's initial location index.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, wherein: the setting of the constraints of the optimization problem includes,

表示每个边缘服务器的计算资源总量，T_n表示任务处理的总时延，/>

表示任务的最大允许时延，C1表示用户初始位置范围的约束，C2表示任务卸载比例的约束，C3表示保证边缘服务器分配给移动用户的计算资源是非负的，C4表示保证分配给每个任务的计算资源总和不会超过边缘服务器的全部计算资源，C5表示规定任务的最大允许时延。Wherein, U _m represents the user equipment set that can access the wireless access point on the side of the edge server,

Indicates the total amount of computing resources of each edge server, T _n indicates the total delay of task processing, />

Indicates the maximum allowable delay of the task, C1 indicates the constraint of the user's initial location range, C2 indicates the constraint of the task unloading ratio, C3 indicates that the computing resources allocated by the edge server to the mobile user are non-negative, and C4 indicates that the allocation of each task is guaranteed The sum of computing resources will not exceed all computing resources of the edge server, and C5 indicates the maximum allowable delay of the specified task.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, wherein: the setting of the state space includes,

The state space s _n of the mobile user equipment n is expressed as,

表示用户移动性属性，

表示边缘服务器的剩余资源。Among them, A _n represents the task information, h _nm represents the channel gain,

represents the user mobility attribute,

Indicates the remaining resources of the edge server.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, wherein: the setting of the action space includes,

The action space a _n of the mobile user equipment n is expressed as,

a _n ＝{γ _n ，f _m→n }

Among them, γ _n represents the unloading strategy to be optimized, and f _m→n represents the resource allocation strategy to be optimized.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, wherein: the setting of the reward function includes,

The reward function r _n of the mobile user equipment n is expressed as,

表示用户完全在本地处理任务的开销，P_n表示任务超时惩罚因子。in,

Represents the overhead of the user completely processing the task locally, and P _n represents the task timeout penalty factor.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, the optimization problem is solved by using the mobile edge computing offloading and resource allocation method based on multi-agent deep reinforcement learning The steps include,

Consider each mobile user device as an agent, each agent includes Actor network and Critic network, set the learning rate of Actor network to 0.0004, and the learning rate of Critic network to 0.004, and the network adopts soft update method for each step update , the soft update coefficient τ is set to 0.01;

Using the deterministic policy gradient method to obtain each mobile user's computing offloading strategy and resource allocation strategy, using the idea of centralized training and distributed execution, each mobile user's critic network collects the state observation information and action information of all mobile users for training , the Actor network generates action strategies for each mobile user separately to adapt to complex dynamic environments;

The NoisyNet method is used to introduce a Gaussian noise variable ε into the last layer of the Actor network, that is, the output layer, so that the output layer becomes a noise layer, and the network is used to learn to generate weights and deviations, thereby improving the efficiency of model exploration;

The calculation of the generation of the weights and biases includes,

ω＝μ ^ω +σ ^ω ⊙ε ^ω

b＝μ ^b +σ ^b ⊙ε ^b

Among them, ω represents the weight generated by the noise layer, b represents the deviation generated by the noise layer, μ ^ω , σ ^ω , μ ^b , σ ^b represent the parameters that the noise layer needs to learn, ε ^ω , ε ^b represent Gaussian noise variables;

Introduce an experience pool that stores all agent states, actions, rewards, and next states. The size of the experience pool is set to 64. When network training is required, a small batch of training data is randomly selected from the experience pool for training. This reduces the dependence and correlation between samples;

Utilize the gradient descent method and the gradient ascent method to update the Critic network and Actor network parameters, and perform iterative optimization, so that the output of the Critic network is constantly approaching the real cost value, and the Actor network outputs continuously optimized unloading and resource allocation strategies, thereby reducing System overhead.

As a preferred solution of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning in the present invention, wherein: the calculation of the iterative optimization includes,

表示Critic在线网络的损失函数，/>

表示Actor在线网络的策略梯度，I表示样本数量，/>

表示第n个智能体在线网络的评价值，/>

表示第n个智能体目标网络输出的评价值，/>

表示N个智能体的状态集合，/>

表示N个智能体的下一状态集合，/>

表示第n个智能体的动作，/>

表示第n个智能体下一状态输出的动作，/>

表示Critic在线网络的参数，/>

表示Critic目标网络的参数，/>

表示智能体开销相关的奖励，γ表示折扣因子，/>

表示Actor在线网络的输出的动作，/>

表示Actor在线网络中输出层的噪声，/>

表示Actor目标网络的参数，/>

表示Actor在线网络的参数。in,

Represents the loss function of the Critic online network, />

Indicates the strategy gradient of Actor's online network, I indicates the number of samples, />

Indicates the evaluation value of the nth agent online network, />

Indicates the evaluation value output by the nth agent target network, />

Represents the state set of N agents, />

Indicates the next state set of N agents, />

Indicates the action of the nth agent, />

Indicates the action of the next state output of the nth agent, />

Indicates the parameters of the Critic online network, />

Indicates the parameters of the Critic target network, />

Represents the agent's cost-related reward, γ represents the discount factor, />

Indicates the action of the output of the Actor's online network, />

Indicates the noise of the output layer in the Actor online network, />

Indicates the parameters of the Actor's target network, />

Represents the parameters of the Actor's online network.

Beneficial effects of the present invention: the present invention proposes a computing offloading and resource allocation method based on multi-agent deep reinforcement learning, which adopts centralized training and distributed execution techniques, and regards each user as an agent, Each agent can use each other's state observation information to train the Critic and Actor networks and make decisions separately, forming a cooperative relationship between agents. Compared with traditional reinforcement learning methods, it is more suitable for dynamic and complex environments. The mobility allows users to flexibly make unloading decisions and resource allocation decisions, reducing the overall overhead of the system; and the present invention uses NoisyNet to introduce parameterized noise in the output layer of the Actor network, which further improves the ability to explore computing offloading and resource allocation strategies in complex scenarios. Ability and efficiency; In addition, compared with DDPG in the same scene, the Noise-MADDPG method disclosed in the present invention has improved training stability, convergence speed, and optimization effect, and has high use value and promotion value.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort. in:

Fig. 1 is the overall flowchart of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning provided by the present invention;

Fig. 2 is a mobile edge computing scene diagram of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning provided by the present invention;

Fig. 3 is a frame diagram based on multi-agent deep reinforcement learning of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning provided by the present invention;

Fig. 4 is a relationship diagram of the number of training rounds and the average reward of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning provided by the present invention;

Fig. 5 is a relationship diagram of total system overhead and number of users of the edge computing offloading and resource allocation method based on multi-agent reinforcement learning provided by the present invention.

Detailed ways

In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Example. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative efforts shall fall within the protection scope of the present invention.

In the following description, a lot of specific details are set forth in order to fully understand the present invention, but the present invention can also be implemented in other ways different from those described here, and those skilled in the art can do it without departing from the meaning of the present invention. By analogy, the present invention is therefore not limited to the specific examples disclosed below.

Second, "one embodiment" or "an embodiment" referred to herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. "In one embodiment" appearing in different places in this specification does not all refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

The present invention is described in detail in conjunction with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the cross-sectional view showing the device structure will not be partially enlarged according to the general scale, and the schematic diagram is only an example, which should not limit the present invention. scope of protection. In addition, the three-dimensional space dimensions of length, width and depth should be included in actual production.

At the same time, in the description of the present invention, it should be noted that the orientation or positional relationship indicated by "upper, lower, inner and outer" in the terms is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention. The invention and the simplified description do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and thus should not be construed as limiting the present invention. In addition, the terms "first, second or third" are used for descriptive purposes only, and should not be construed as indicating or implying relative importance.

Unless otherwise specified and limited in the present invention, the term "installation, connection, connection" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integrated connection; it can also be a mechanical connection, an electrical connection or a direct connection. A connection can also be an indirect connection through an intermediary, or it can be an internal communication between two elements. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations.

Example 1

Referring to Figures 1 to 3, an embodiment of the present invention provides an edge computing offloading and resource allocation method based on multi-agent reinforcement learning, including:

S1: Construct a mobile edge computing offloading and resource allocation model according to the complex scenario of multiple mobile users and multiple edge servers, and set the parameters of the model. It should be noted:

Mobile edge computing offloading and resource allocation models include system model, task model, mobility model and computing model;

Specifically, the system model includes, as shown in Figure 2, the system adopts the common coverage mode of the base station and the wireless access point, the system includes M edge servers and N mobile user equipment, and M={1,2,... ,M},N={1,2,...,N} is a collection of all edge servers and mobile user equipment, the edge server is deployed next to the wireless access point, each wireless access point independently covers a cell, and the mobile The user equipment can offload computing tasks to the edge server of the cell through the wireless access point of the cell, request computing resources, and the wireless access points connect and transmit data through the base station;

其中，L_n表示任务的数据量，X_n表示计算任务所需的CPU循环数，/>

表示完成任务所需的最大允许时延；Specifically, the task model includes that each mobile user equipment randomly generates a computing task at each moment, and the attributes of the generated computing task are represented by a triplet A _n , namely

Among them, L _n represents the amount of data of the task, X _n represents the number of CPU cycles required to calculate the task, />

Indicates the maximum allowable delay required to complete the task;

Specifically, the mobility model includes modeling user mobility using discrete random hops, and using the average dwell time to represent the strength between hops;

It should be noted that the dwell time indicates the time that the mobile user stays in the area provided by the designated wireless access point. It is an important performance indicator for planning network resources and improving service quality. Once the mobile user leaves the area served by the current wireless access point , if the task has not been processed yet, the task will be migrated due to cell switching;

的计算包括，It should be stated that the probability density function of the average dwell time

The calculation includes,

表示用户实际驻留时间；Among them, β _n represents the average residence time of mobile user n,

Indicates the actual residence time of the user;

Specifically, the computing model includes a local computing model and an edge computing model;

① Local computing model

可表示为，Let f _n represent the local computing power of user device n, that is, the local CPU speed per second, the unit is CPU cycls/s, the task is divisible, the user offloads part of the task to the selected edge server for execution, and the unloading ratio is γ _n , then for task A _n , its local computation delay

can be expressed as,

Specifically, the energy consumption of tasks calculated locally is calculated using a widely recognized calculation energy consumption model per CPU revolution E=κf ² , where κ represents a conversion coefficient of energy consumption, which is related to the hardware structure, and f represents computing power , then the local energy consumption of task A _n can be expressed as,

Therefore, the total overhead of local computation can be expressed as,

Among them, ω ₁ represents the weight coefficient of delay, and ω ₂ represents the weight coefficient of energy consumption. If ω ₁ >ω ₂ , it means that the task is more sensitive to delay; if ω ₁ <ω ₂ , it means that the task itself is short of energy. In the application, the appropriate weight coefficient can be determined according to the actual task requirements;

②Edge Computing Model

2)边缘服务器的任务计算时延/>

3)任务结果从边缘服务器回传到用户的下行传输时延

假设任务处理结果比输入数据小的多，下行速率比上行速率大的多，那么边缘服务器上的任务结果回传到用户设备的时延可以忽略；When some tasks are uploaded to the edge server deployed on the side of the wireless access point for processing, the processing delay is mainly composed of three parts: 1) The uplink transmission delay of tasks uploaded from the user to the edge server

2) The task calculation delay of the edge server/>

3) The downlink transmission delay of the task result from the edge server to the user

Assuming that the task processing result is much smaller than the input data, and the downlink rate is much larger than the uplink rate, then the delay of the task result on the edge server back to the user equipment can be ignored;

Each edge server can serve multiple mobile users at the same time. When a mobile user establishes a transmission connection with the wireless access point on the side of the edge server, the uplink signal-to-interference-noise ratio (SINR) between the user and the wireless access point must be Satisfy the constraint SINR _nm ≥ ^{SINR th} , where SINR ^th is the threshold to ensure the transmission quality, so the set of user equipment that can access the wireless access point on the side of the edge server can be expressed as U _m ={n∈N|SINR _nm ≥SINR ^th };

Assuming that the uplink adopts the OFDMA access method, the channels under the same wireless access point are orthogonal to each other, no interference will be caused between user equipment, all user equipment are equally allocated channel bandwidth, and different wireless access points share channel resources , the user equipment transmits task data at the maximum power, and the channel gain from the user equipment to the wireless access point on the side of the edge server is h _nm , which is related to factors such as transmission distance and small-scale fading, then SINR _nm can be expressed as,

Among them, σ ² represents the variance of Gaussian white noise, _{and In} represents the interference between user equipment n and the wireless access point, including intersymbol interference, interchannel interference, etc.;

The transfer rate can be expressed as,

Wherein, B represents sub-channel bandwidth;

Therefore, the transmission delay of the task from the device to the edge server can be expressed as,

因此，分配给每个任务的计算资源应满足以下约束：Assuming that the total amount of computing resources of each edge server is F ^max , the computing resources allocated by users offloaded to the edge server for processing are expressed as

Therefore, the computing resources allocated to each task should satisfy the following constraints:

Therefore, the calculation delay of the edge server can be expressed as,

The total latency of edge processing tasks is,

The energy consumption generated by task transmission can be expressed as:

Finally, the total overhead of offloading part of task A _n to the edge server on the side of the wireless access point for processing is expressed as,

Among them, in addition to delay and energy consumption, overhead also includes the resource cost of occupying the edge server ω ₃ f _m→n , ω ₃ is expressed as the resource cost coefficient, and the range is ω ₃ ∈ [0,1].

Furthermore, user mobility involves task migration, resulting in increased delay and increased system overhead. The present invention optimizes the unloading strategy and resource allocation strategy through the disclosed method to reduce the task migration delay, thereby reducing the overall system overhead. Specifically, it can be divided into two situations:

① Before the mobile user equipment leaves the coverage area of the original wireless access point, if the offloading task can be completed on the original edge server, the task migration will not occur. The calculation of the total cost of the user equipment at this time includes:

假设迁移时延只与数据量多少有关，此时用户设备总开销的计算包括，②Before the mobile user equipment leaves the coverage area of the original wireless access point, if the offloading task cannot be completed on the original edge server, task migration will occur, and the execution results on the original edge server will be migrated to another edge server through the base station At the wireless access point where the wireless access point is located, the wireless access point will send the execution result back to the mobile user equipment, and additional migration delay will be generated at this time

Assuming that the migration delay is only related to the amount of data, the calculation of the total cost of the user equipment includes,

Furthermore, the calculation of the total cost Q _n of the mobile user equipment includes,

表示没有发生任务迁移时移动用户设备的总开销，/>

表示发生任务迁移时移动用户设备的总开销，ω₁表示时延系数，ω₂表示能耗系数，ω₃表示资源成本系数，/>

表示本地计算和边缘计算的最大时延，/>

表示本地计算和边缘计算加上额外迁移时延的最大时延，E_n表示本地计算和边缘计算的能耗，f_m→n表示边缘服务器m给用户n分配的计算资源，/>

表示事件发生的概率，T_mn表示边缘处理任务的总时延；in,

Indicates the total overhead of moving user equipment when no task migration occurs, />

Indicates the total overhead of mobile user equipment when task migration occurs, ω ₁ represents the delay coefficient, ω ₂ represents the energy consumption coefficient, ω ₃ represents the resource cost coefficient, />

Indicates the maximum latency of local computing and edge computing, />

Indicates the maximum delay of local computing and edge computing plus additional migration delay, E _n represents the energy consumption of local computing and edge computing, f _m→n represents the computing resources allocated by edge server m to user n, />

Indicates the probability of event occurrence, and T _mn indicates the total delay of edge processing tasks;

Using overhead expectations to measure the performance of the system, the calculation of the expected total overhead of mobile user equipment includes,

表示时延、能耗以及计算资源的平均开销。in,

Indicates the average overhead of latency, energy consumption, and computing resources.

S2: System overhead of mobile edge computing offloading and resource allocation based on model computing, setting the objective function and constraints of the optimization problem. It should be noted:

The computation of the objective function for an optimization problem involves,

Among them, γ _n represents the task unloading ratio, I(x _n ) represents the indicator function, and x _n represents the user’s initial position index;

It should be noted that if x _n >0, then I(x _n )=1;

Further, the setting of the constraint conditions of the optimization problem includes,

表示每个边缘服务器的计算资源总量，T_n表示任务处理的总时延，/>

表示任务的最大允许时延，C1表示用户初始位置范围的约束，C2表示任务卸载比例的约束，C3表示保证边缘服务器分配给移动用户的计算资源是非负的，C4表示保证分配给每个任务的计算资源总和不会超过边缘服务器的全部计算资源，C5表示规定任务的最大允许时延。Wherein, U _m represents the user equipment set that can access the wireless access point on the side of the edge server,

Indicates the total amount of computing resources of each edge server, T _n indicates the total delay of task processing, />

Indicates the maximum allowable delay of the task, C1 indicates the constraint of the user's initial location range, C2 indicates the constraint of the task unloading ratio, C3 indicates that the computing resources allocated by the edge server to the mobile user are non-negative, and C4 indicates that the allocation of each task is guaranteed The sum of computing resources will not exceed all computing resources of the edge server, and C5 indicates the maximum allowable delay of the specified task.

S3: Model the optimization problem as a Markov decision process, and set the state space, action space and reward function in deep reinforcement learning. It should be noted:

① Set the state space s _n of mobile user equipment n as,

表示用户移动性属性，

表示边缘服务器的剩余资源；Among them, A _n represents the task information, h _nm represents the channel gain,

represents the user mobility attribute,

Indicates the remaining resources of the edge server;

It should be noted that the state space of all mobile user equipments is expressed as s={s ₁ , s ₂ ,...,s _N };

② Set the action space a _n of mobile user equipment n as,

a _n ＝{γ _n ，f _m→n }

Among them, γ _n represents the unloading strategy to be optimized, and f _m→n represents the resource allocation strategy to be optimized;

It should be noted that the action space of all mobile user equipments is expressed as a={a ₁ ,a ₂ ,...,a _N };

③ Set the reward function r _n of mobile user equipment n as,

表示用户完全在本地处理任务的开销，P_n表示任务超时惩罚因子；in,

Indicates the overhead of the user completely processing the task locally, and P _n indicates the task timeout penalty factor;

It should be noted that the reward function of the entire system, that is, all users, is expressed as r=Σ _n∈N r _n .

S4: Use the method based on multi-agent deep reinforcement learning to find the optimal unloading strategy and resource allocation strategy for each mobile user, and optimize the objective function. At the same time, use the NoisyNet method to add Gaussian noise to the output layer of the Actor network to improve the network performance. Improve the efficiency of model exploration and improve the optimization effect.

It should be noted:

The steps of solving the optimization problem using the mobile edge computing offloading and resource allocation method based on multi-agent deep reinforcement learning include,

① Treat each mobile user device as an agent, each agent includes actor network and critic network, both of which include online neural network and target neural network, adopt similar network structure, including input layer, two FC layers , two BN layers and an output layer, the number of neurons of the two FC layers is set to 500 and 300 respectively, the BN layer is used to normalize the output of the FC layer, to prevent the output of the FC layer from entering the saturation range of the activation function, and the middle layer uses Relu Activation function, the output layer of the Actor network uses Tanh to activate the output, the output layer of the Critic network uses Relu to activate the output, set the learning rate of the Actor network to 0.0004, and the learning rate of the Critic network to 0.004, and the network adopts a soft update method for each step Update, the soft update coefficient τ is set to 0.01;

②Using the deterministic policy gradient method to obtain the computing offloading strategy and resource allocation strategy of each mobile user, using the idea of centralized training and distributed execution, the Critic network of each mobile user collects the state observation information and action information of all mobile users Training, the Actor network generates action strategies for each mobile user to adapt to complex dynamic environments;

③Use the NoisyNet method to introduce Gaussian noise variable ε into the last layer of the Actor network, that is, the output layer, so that the output layer becomes a noise layer, and learn to generate weights and deviations through the network, thereby improving the efficiency of model exploration;

It should be noted that the calculations for the generation of weights and biases include,

ω＝μ ^ω +σ ^ω ⊙ε ^ω

b＝μ ^b +σ ^b ⊙ε ^b

Among them, ω represents the weight generated by the noise layer, b represents the deviation generated by the noise layer, μ ^ω , σ ^ω , μ ^b , σ ^b represent the parameters that the noise layer needs to learn, and adjust the size during training along with other training parameters of the Actor network , ε ^ω , ε ^b represent Gaussian noise variables;

④ Introduce an experience pool that stores all agent states, actions, rewards, and next states. The size of the experience pool is set to 64. When network training is required, a small batch of training data is randomly selected from the experience pool for training, thereby reducing Dependencies and correlations between samples;

⑤ Utilize the gradient descent method and the gradient ascent method to update the critic network and actor network parameters, and perform iterative optimization, so that the output of the critic network is constantly approaching the real cost value, and the actor network outputs continuously optimized unloading and resource allocation strategies, thereby reducing system overhead;

It should be noted that the calculation of iterative optimization includes,

表示Critic在线网络的损失函数，/>

表示Actor在线网络的策略梯度，I表示样本数量，/>

表示第n个智能体在线网络的评价值，/>

表示第n个智能体目标网络输出的评价值，/>

表示N个智能体的状态集合，/>

表示N个智能体的下一状态集合，/>

表示第n个智能体的动作，/>

表示第n个智能体下一状态输出的动作，/>

表示Critic在线网络的参数，/>

表示Critic目标网络的参数，/>

表示智能体开销相关的奖励，γ表示折扣因子，/>

表示Actor在线网络的输出的动作，/>

表示Actor在线网络中输出层的噪声，/>

表示Actor目标网络的参数，/>

表示Actor在线网络的参数。in,

Represents the loss function of the Critic online network, />

Indicates the strategy gradient of Actor's online network, I indicates the number of samples, />

Indicates the evaluation value of the nth agent online network, />

Indicates the evaluation value output by the nth agent target network, />

Represents the state set of N agents, />

Indicates the next state set of N agents, />

Indicates the action of the nth agent, />

Indicates the action of the next state output of the nth agent, />

Indicates the parameters of the Critic online network, />

Indicates the parameters of the Critic target network, />

Represents the agent's cost-related reward, γ represents the discount factor, />

Indicates the action of the output of the Actor's online network, />

Indicates the noise of the output layer in the Actor online network, />

Indicates the parameters of the Actor's target network, />

Represents the parameters of the Actor's online network.

It should be noted that the present invention proposes a computing offloading and resource allocation method based on multi-agent deep reinforcement learning. This method adopts the technique of centralized training and distributed execution, and regards each user as an agent. Agents can use each other's state observation information to train Critic and Actor networks and make decisions separately, forming a cooperative relationship between agents. Compared with traditional reinforcement learning methods, it is more suitable for dynamic and complex environments. It can flexibly make unloading decisions and resource allocation decisions for users, reducing the overall overhead of the system; and the present invention uses NoisyNet to introduce parameterized noise in the output layer of the Actor network, further improving the ability and ability to explore computing offloading and resource allocation strategies in complex scenarios. Efficiency; In addition, compared with DDPG in the same scene, the Noise-MADDPG method disclosed in the present invention has improved training stability, convergence speed, and optimization effect, and has high use value and promotion value.

Example 2

4-5 is the second embodiment of the present invention. This embodiment is different from the first embodiment in that it provides a verification test of edge computing offloading and resource allocation methods based on multi-agent reinforcement learning. The technical effect adopted in the method is verified and explained.

This embodiment is implemented based on the Python 3.7 programming language and the PyTorch 1.13.1 deep reinforcement learning framework, using Pycharm as the IDE;

信道带宽B＝1Mhz，信道增益h_nm的范围是[0.1,0.9]，用户设备计算能力f_n的范围是[800,900]Mhz，发射功率p_n的范围是[20,30]W，平均驻留时间β_n是服从均值为0.4，方差为1.2的高斯分布；基站总计算能力/>

的范围是[14000,16000]Mhz，时延系数ω₁＝0.8，能耗系数ω₂＝0.2，资源成本系数ω₃＝0.8。Set the key parameters of the experimental simulation: the number of users N=15, the number of edge servers M=5, the range of task data L _n is [0.3,0.5]Mhz, the range of CPU cycles X _n required for the calculation task is [900, 1100] MCycles, the maximum allowable delay of the task

Channel bandwidth B=1Mhz, the range of channel gain h _nm is [0.1,0.9], the range of user equipment computing capability f _n is [800,900]Mhz, the range of transmit power p _n is [20,30]W, the average resident Time β _n follows a Gaussian distribution with a mean of 0.4 and a variance of 1.2; the total computing power of the base station/>

The range is [14000, 16000]Mhz, the delay coefficient ω ₁ =0.8, the energy consumption coefficient ω ₂ =0.2, and the resource cost coefficient ω ₃ =0.8.

As shown in Figure 4, under the parameters set in this embodiment, with the increase of the number of training rounds, the Noise-MADDPG method adopted by the method provided by the present invention reaches a state of convergence, and compared with the DDPG method, the training effect is significantly improved.

Change the number of users N to obtain the relationship diagram between the total system overhead and the number of users as shown in Figure 5, respectively take the number of users N=10, 15, 20, 25, and further verify the superiority of the method provided by the present invention compared to the DDPG method And compared to the effectiveness of all local computing and full offload computing.

Therefore, compared with the DDPG method in the same scene, the Noise-MADDPG method disclosed in the present invention has improved training stability, convergence speed, and optimization effect, and has high use value and promotion value.

It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention without limitation, although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be carried out Modifications or equivalent replacements without departing from the spirit and scope of the technical solution of the present invention shall be covered by the claims of the present invention.

Claims (10)

1. The edge computing unloading and resource allocation method based on multi-agent reinforcement learning is characterized by comprising the following steps:

constructing a mobile edge computing unloading and resource allocation model according to complex scenes of multiple mobile users and a multi-edge server, and setting parameters of the model;

calculating the system overhead of unloading and resource allocation based on the mobile edge calculated by the model, and setting an objective function and constraint conditions of the optimization problem;

modeling the optimization problem into a Markov decision process, and setting a state space, an action space and a reward function in deep reinforcement learning;

and searching an optimal unloading strategy and a resource allocation strategy for each mobile user by adopting a multi-agent deep reinforcement learning-based method, optimizing the objective function, and simultaneously adding Gaussian noise to an output layer of an Actor network by adopting a NoisyNet method, so that the network model exploration efficiency is improved, and the optimization effect is improved.

2. The multi-agent reinforcement learning-based edge computing offloading and resource allocation method of claim 1, wherein: the mobile edge computing unloading and resource allocation model comprises a system model, a task model, a mobility model and a computing model;

the system model comprises M edge servers and N mobile user equipment, wherein the edge servers are deployed beside wireless access points, each wireless access point independently covers a cell, the mobile user equipment can offload calculation tasks to the edge servers of the cell through the wireless access point of the cell to request calculation resources, and the wireless access points are connected and transmit data through a base station;

the task model comprises that each mobile user equipment randomly generates a computing task at each moment, and the generated computing task attribute uses a triplet A _n Representation, i.e.

wherein ,L_n Representing the data volume of a task, X _n Represents the number of CPU cycles required for the calculation task, < >>

Representing a maximum allowable delay required to complete a task;

the mobility model comprises modeling user mobility by adopting discrete random hops, and representing the intensity between hops by using average residence time;

probability density function of the average residence time

The calculation of (c) includes the steps of,

wherein ,β_n Representing the average residence time of the mobile user n,

indicating the actual residence time of the user.

3. The multi-agent reinforcement learning-based edge computing offloading and resource allocation method of claim 2, wherein: also included is a method of manufacturing a semiconductor device,

the calculation model comprises the total cost of the mobile user equipment under different unloading decisions and resource allocation strategies, wherein the total cost comprises time delay, energy consumption and resource cost;

the mobile user equipment overhead Q _n The calculation of (c) includes the steps of,

wherein ,

indicating the total overhead of the mobile user equipment when no task migration has occurred,/->

Representing the total overhead, ω, of the mobile user equipment when task migration occurs ₁ Representing the delay coefficient, omega ₂ Representing the energy consumption coefficient omega ₃ Representing resource cost coefficients, < >>

Maximum delay representing local computation and edge computation,/-for>

Representing the maximum delay of local computation and edge computation plus additional migration delay, E _n Representing the energy consumption of local computation and edge computation, f _m→n Representing the computing resources allocated by edge server m to user n,

representing the probability of occurrence of an event, T _mn Representing total latency of edge processing tasks；

The performance of the system is measured by using the overhead expectations, the calculation of which includes,

wherein ,

representing the average overhead of latency, energy consumption, and computing resources.

4. The multi-agent reinforcement learning-based edge computing offloading and resource allocation method of claim 3, wherein: the calculation of the objective function of the optimization problem includes,

wherein ,γ_n Representing task offload ratio, I (x _n ) Representing an indication function, x _n Representing the initial position index of the user.

5. The multi-agent reinforcement learning-based edge computing offloading and resource allocation method of claim 4, wherein: the setting of constraints for the optimization problem includes,

wherein ,U_m Representing a set of user equipments having access to a wireless access point on the side of an edge server,

representing the total amount of computing resources per edge server, T _n Representing the total delay of task processing, +.>

Representing maximum allowable time delay of a task, C1 representing a constraint of a user initial position range, C2 representing a constraint of a task offloading ratio, C3 representing a constraint of ensuring that computing resources allocated to a mobile user by an edge server are non-negative, C4 representing a constraint of ensuring that a sum of computing resources allocated to each task does not exceed all computing resources of the edge server, and C5 representing a maximum allowable time delay of a prescribed task.

6. The multi-agent reinforcement learning-based edge computing offloading and resource allocation method of claim 5, wherein: the setting of the state space comprises that,

the state space s of the mobile user equipment n _n It is indicated that the number of the elements is,

wherein ,A_n Representing task information, h _nm Indicating the gain of the channel,

representing user mobility attributes, ++>

Representing the remaining resources of the edge server.

7. The multi-agent reinforcement learning-based edge computing offloading and resource allocation method of claim 6, wherein: the setting of the action space includes,

the action space a of the mobile user equipment n _n It is indicated that the number of the elements is,

a _n ＝{γ _n ，f _m→n }

wherein ,γ_n Representing an offloading policy to be optimized, f _m→n Representing the resource allocation policy to be optimized.

8. The multi-agent reinforcement learning based edge computing offload and resource allocation method of claim 7, wherein: the setting of the bonus function includes,

the reward function r of the mobile user equipment n _n It is indicated that the number of the elements is,

wherein ,

representing the overhead of a user processing tasks entirely locally, P _n Representing a task timeout penalty factor.

9. The multi-agent reinforcement learning-based edge computing offloading and resource allocation method of claim 8, wherein: the step of solving the optimization problem by adopting the multi-agent deep reinforcement learning-based mobile edge computing unloading and resource allocation method comprises,

each mobile user equipment is regarded as an agent, each agent comprises an Actor network and a Critic network, the learning rate of the Actor network is set to be 0.0004, the learning rate of the Critic network is set to be 0.004, the network adopts a soft update mode to update each step, and the soft update coefficient tau is set to be 0.01;

acquiring a calculation unloading strategy and a resource allocation strategy of each mobile user by adopting a deterministic strategy gradient method, and collecting state observation information and action information of all mobile users by using a Critic network of each mobile user to train by utilizing a centralized training distributed execution idea, wherein an Actor network respectively generates the action strategy for each mobile user so as to adapt to a complex dynamic environment;

introducing Gaussian noise variable epsilon to the last layer of an Actor network, namely an output layer, by adopting a NoisyNet method, so that the output layer is changed into a noise layer, and learning through the network to generate weight and deviation, thereby improving the model exploration efficiency;

the calculation of the generation of the weights and deviations includes,

ω＝μ ^ω +σ ^ω ⊙ε ^ω

b＝μ ^b +σ ^b ⊙ε ^b

wherein ω represents the weight of noise layer generation, b represents the deviation of noise layer generation, μ ^ω 、σ ^ω 、μ ^b 、σ ^b Parameters epsilon representing the noise floor to be learned ^ω 、ε ^b Representing gaussian noise variation;

introducing an experience pool for storing all the states, actions, rewards and next states of the intelligent agents, wherein the size of the experience pool is set to 64, and when network training is required, small batches of training data are randomly extracted from the experience pool for training, so that dependence and correlation among samples are reduced;

and updating Critic network and Actor network parameters by using a gradient descent method and a gradient ascent method, and performing iterative optimization to ensure that the output of the Critic network is continuously close to a real overhead value, and the Actor network outputs continuously optimized unloading and resource allocation strategies, thereby reducing the system overhead.

10. The multi-agent reinforcement learning based edge computing offloading and resource allocation method of claim 9, wherein: the calculation of the iterative optimization includes,

wherein ,

representing the loss function of Critic on-line network, < ->

Representing the policy gradient of the Actor on-line network, I represents the number of samples, +.>

Representing the evaluation value of the n-th agent on-line network,/->

Evaluation value representing n-th agent target network output,/->

State set representing N agents, +.>

Representing the next state set of N agents, -/-, for example>

Representing the actions of the nth agent, +.>

Action representing the next state output of the nth agent,/->

Parameters representing Critic on-line network, +.>

Parameters representing Critic target network, +.>

Representing an agent overhead related reward, gamma representing a discount factor,/->

Action representing the output of the Actor on-line network, < >>

Noise representing the output layer of an Actor in-line network, < >>

Parameters representing the target network of the Actor +.>

Representing the parameters of the Actor online network.

Images