CN103309748B - Adaptive scheduling host system and scheduling method of GPU virtual resources in cloud game - Google Patents

Abstract

A host system for self-adaptive scheduling of GPU virtual resources in a cloud game, comprising a scheduling control module, an agent module respectively connected to the scheduling control module, an image application programming interface analysis module, and a list of virtual machines, the agent module It consists of a scheduling module and a monitoring module. All modules are deployed in the host machine; the scheduling control module is responsible for the communication of information between other modules: the scheduling control module obtains the information of all running virtual machines from the virtual machine list and sends them to required modules. The present invention realizes the scientific management of GPU resources in the virtual machine, improves the utilization rate of GPU resources, and allocates enough GPU resources to all running virtual machines through a fair allocation method based on the fairness-based QoS adaptive scheduling method, and ensures that the QoS requirements are met. while maximizing overall GPU utilization.

Description

Adaptive scheduling host system and scheduling method for GPU virtual resources in cloud games

technical field

The invention relates to a resource self-adaptive scheduling host system in the field of GPU virtualization, which is applied to a cloud game platform, and is specifically a scheduling system that intervenes in an operating system image API.

Background technique

The gradual maturity of GPU virtualization technology has promoted the development of cloud game applications, so cloud game applications in cloud services are popular. However, due to the poor performance of the current default GPU resource sharing mechanism, the user experience of cloud games is inevitably damaged by some real-time uncertain factors, such as rendering complex game scenes.

In video stream quality analysis, FPS (Frames Per Second) is defined as the number of frames per second, which measures the amount of information used to save and display dynamic video, that is, the number of frames of animation or video.

In the user experience of cloud games, QoS requirements refer to the minimum FPS and maximum delay that guarantee the normal use of users.

The main factors affecting the running of the game in the virtual environment are: CPU running time, GPU execution game synchronous rendering time, unpredictable game scene changes and intervention between virtual machines. These factors cause the FPS of the game to fail to meet the QoS requirements, affecting user experience.

Currently existing virtualization solutions have poor resource sharing mechanisms, mainly reflected in low resource utilization and failure to meet service level agreement (QoS) requirements. Most of the resource scheduling mechanisms in virtual machines now use the first-in-first-out method, which causes some virtual machines to fail to meet the QoS requirements, and running multiple virtual machines on a single server will bring performance fluctuations to game applications in each virtual machine. Especially on cloud gaming platforms.

After searching the prior art, it was found that Carnegie Mellon University in the United States raised this problem in the GPU accelerated window system. IBM Tokyo Research Institute has developed an automatic resource scheduling system to accelerate the operation of stencil applications in general-purpose GPU clusters. The University of North Carolina at Chapel Hill proposed two approaches to integrate GPUs into soft real-time multiprocessor systems to improve overall system performance. Stony Brook University of New York proposed GERM, which aims to provide a fair GPU resource allocation algorithm, and also uses a fixed frame rate method such as vertical synchronization to prevent the game from relying too much on hardware resources. But they all have the following disadvantages, GERM does not consider the QoS requirement and the fixed frame rate method cannot effectively use hardware resources. TimeGraph cannot guarantee that all virtual machines meet the QoS requirements, especially when the important load is less.

Contents of the invention

Aiming at the deficiencies of existing GPU resource scheduling methods, the present invention proposes a system and method for adaptively scheduling GPU virtual resources on a host machine in a cloud game. By using GPU semi-virtualization and library injection technology, a Introduce lightweight scheduling control, using green software technology that does not need to change the image driver of the host machine, the operating system of the guest machine and the application. The scheduling algorithm alleviates the impact of real-time uncertain factors and ensures high utilization of system resources. Specifically, the QoS adaptive scheduling algorithm based on fairness not only ensures that each virtual machine meets the basic requirements of QoS, but also redistributes GPU resources, and allocates GPU resources of virtual machines with higher FPS to those that do not meet QoS Therefore, the fairness-based QoS adaptive scheduling algorithm not only meets the QoS requirements, but also achieves fair scheduling and significantly improves GPU utilization.

Technical solution of the present invention is as follows:

A host system for self-adaptive scheduling of GPU virtual resources in a cloud game, which is characterized in that it includes a scheduling control module, an agent module connected to the scheduling control module, an image application programming interface analysis module, and a virtual machine list. The agent module described above is composed of a scheduling module and a monitoring module, and all modules are deployed in the host machine;

The dispatching control module is responsible for the communication of information between other modules: the dispatching control module obtains the information of all running virtual machines from the virtual machine list, and sends them to the required modules;

Another function of the scheduling control module is to automatically adjust the parameter FPS of the scheduled scheduling method to make the scheduling method run well;

The monitoring module obtains the real-time running information of the virtual machine from the scheduling control module and sends it to the scheduling module,

The scheduling module receives the command from the upper-layer GPU command distributor, and processes the received command according to the real-time running information of the virtual machine sent by the monitoring module, and finally sends it to the lower-level host GPU to drive;

The agent module and the scheduling control module decide together how many GPU resources need to be allocated to the virtual machine;

The image application programming interface analysis module is responsible for calculating the overhead caused by the execution of GPU commands generated by the image application programming interface, and sending the result to the scheduling control module, so that other modules that need the information can obtain it.

The virtual machine list saves all the running virtual machines in the host system. Each virtual machine can be indexed by the system. When a new virtual machine is added and starts running, the system will automatically add it to the virtual machine list. And reschedule the GPU resources again.

The scheduling method of using the GPU virtual resources in the cloud game to adaptively schedule the host computer system is characterized in that the method includes the following steps:

① The user sets the number of frames per second achieved in the scheduling control module, the virtual machine list records the real-time information of the virtual machine running, and the scheduling control module obtains the information of all running virtual machines from the virtual machine list;

②The image application programming interface analysis module calculates the overhead caused by the GPU command operation, and sends it to the scheduling control module;

③ The monitoring module obtains the real-time information of the virtual machine operation recorded in the virtual machine list through the scheduling control module, and obtains the overhead caused by the GPU command operation calculated by the image application programming interface analysis module through the scheduling control module, and sends them together to Scheduling module;

④ The scheduling module receives the commands from the GPU command distributor, and calculates the running time of the sleep function according to the real-time information of the virtual machine running obtained from the monitoring module and the overhead caused by the GPU command running, processes the received commands, and finally sends to the underlying host GPU driver;

The concrete steps of described step ④ are as follows:

When the real-time information of the virtual machine operation obtained by the scheduling module from the monitoring module shows that the FPS is greater than 30fps, the scheduling module inserts a sleep function before calling the rendering picture function, and then calls the rendering picture function to render the picture;

When the real-time information of the virtual machine operation obtained by the scheduling module from the monitoring module shows that the FPS is less than 30fps, the scheduling module allocates more GPU resources for the virtual machine, and then calls the rendering screen function to render the screen;

When the real-time information of virtual machine operation obtained by the scheduling module from the monitoring module shows that the FPS is equal to 30fps, the scheduling module directly calls the rendering screen function to render the screen.

Principle of the present invention is as follows:

(1) System architecture based on paravirtualization (2) Adaptive scheduling algorithm. Among them: the system architecture based on paravirtualization is a lightweight scheduling module based on the host machine library implantation technology, and there is no need to modify the image API of the host machine, the operating system and application programs of the client machine; among them, the QoS based on fairness The adaptive scheduling algorithm meets the minimum FPS and maximum delay of cloud game user experience, and emphasizes the fairness of resource scheduling.

When the virtual machine is running, the calculation time of the Present call is very stable, and even if there is a fluctuation, it will gradually change smoothly. Therefore, we believe that the agent of each virtual machine can predict the calculation amount of the Present through the historical information of the respective virtual machine. This algorithm uses The arithmetic mean of the past 20 consecutive Present call times is used as the prediction of the next frame. In order to make the frame length of each virtual machine stable at the set value FPS, each frame is extended by delaying its last Present call. The effect of delay is achieved by inserting a Sleep call before each Present call, and the amount of sleep is given by the formula. We found that when there is a lot of competition for GPU resources, the amount of time for Present will change significantly, and we can get more accurate GPU calculation time by inserting the flush command before the Sleep call.

Realization of fairness: 1. The QoS adaptive scheduling algorithm based on fairness will maintain a virtual machine list, which records in detail the current running virtual machine's occupation of GPU resources and the FPS of each virtual machine running the game. 3. The image API analyzer analyzes the running image information of the game in the virtual machine in real time to calculate its FPS and predict the running cost. 4. On the one hand, the scheduling control module writes the image information of the game running, such as FPS, into the virtual machine list through the image API analyzer; on the other hand, it decides to release those with too high FPS GPU resources of virtual machines and reallocate them to virtual machines below the QoS standard. 5. The scheduling module is responsible for the release and reallocation of GPU resources mentioned in 4. On the one hand, the GPU resources are released by inserting Sleep() to delay the frame length, and on the other hand, the idle GPU resources obtained by inserting Sleep are allocated to those low A virtual machine based on Qos. 6. During the above-mentioned release and reallocation process, the virtual machines with too high FPS are responsible for the reallocation of GPU resources in equal proportion. The virtual machine with the most GPU resources occupies the first place in GPU utilization, so this reallocation algorithm of GPU resources is relatively fair.

Compared with the prior art, the beneficial effect of the present invention is that the scientific management of GPU resources in the virtual machine is realized, and the utilization rate of GPU resources is improved, and the fairness-based QoS adaptive scheduling method provides the The virtual machine allocates sufficient GPU resources and guarantees compliance with QoS requirements, while maximizing overall GPU utilization.

Description of drawings

FIG. 1 is a schematic diagram of the architecture of a virtual resource scheduling system in the prior art.

FIG. 2 is a schematic diagram of a host system for adaptively scheduling GPU virtual resources in a cloud game according to the present invention.

Figure 3 Frame composition and frame delay.

Fig. 4 is a flow chart of a host system scheduling method for adaptive scheduling of GPU virtual resources.

Figure 5. The influence of different parameters of the control system on performance, (a) and (b) are two different test procedures.

Figure 6. GPU resource reallocation between two virtual machines.

Fig. 7 Performance evaluation results of the fairness-based QoS adaptive scheduling method.

Detailed ways

The following is a detailed description of the embodiments of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and provides detailed implementation methods and specific operation processes. The applicable platform of the present invention is not limited to the following embodiments.

Figure 1 is a schematic diagram of the architecture of the virtual resource scheduling system in the prior art, and a schematic diagram of the GPU virtual resource adaptive scheduling host system in the cloud game of the present invention. All the modules of the system of the present invention are deployed in the host machine, which is a media The scheduling system based on the GPU and the GPU application programming interface of the host machine includes a scheduling control module 1, an agent module connected respectively to the scheduling control module 1, an image application programming interface analysis module 4 and a virtual machine list 5, the described The agent module is composed of scheduling module 2 and monitoring module 3 .

Scheduling control module 1 obtains the feedback of performance information from all running virtual machines and automatically adjusts the parameters of the scheduling method that have been set to make the scheduling method run well; each running virtual machine has a scheduling module 2 and a monitoring module The agent module composed of 3, the monitoring module sends the real-time performance information of the corresponding virtual machine to the scheduling control module 1, and the scheduling module receives commands from the driver and schedules GPU computing tasks. The agent module usually also decides how many GPU resources need to be allocated to the virtual machine together with the scheduling control module 1 . The image application programming interface analysis module 4 calculates the overhead caused by the execution of GPU commands, and these commands are generated by the image API, such as the Present command. The virtual machine list includes all running virtual machines in the host system, and each virtual machine can be indexed by the system. When a new virtual machine starts running, the host system will automatically add it to the virtual machine list and then Reschedule GPU resources. The paravirtualization-based system architecture described here employs Type II virtualization. When the client application wakes up the standard GPU rendering API, the client GPU computing library prepares the corresponding GPU cache in the main memory and issues GPU command packets, which are pushed into the virtual GPU I/O queue and processed by the host in turn , and finally the dispatch layer sends commands to the host driver asynchronously. Direct memory access is used to move caches from guest memory to GPU memory.

As shown in Figure 2, the definition of frame delay in this method: the time difference between the return values of two adjacent Presents is defined as the frame delay. Each frame consists of four parts: GPU rendering, Present(), Sleep(), target calculation, and image painting. Due to the changes in GPU rendering, target calculation, and image painting, the lengths of different frames are different. As shown in the pseudocode of the algorithm in Figure 4, this algorithm is to adjust the time of Sleep(), one of the components of the frame, to stabilize the size of each frame, and finally maintain each frame at a similar level. Specifically, the QoS adaptive scheduling algorithm based on fairness is divided into two aspects, namely, the realization of scheduling adaptability and the realization of scheduling fairness. The logic flow chart of the method program is shown in FIG. 3 .

The realization of the adaptive scheduling part: 1. First, the user sets the FPS (frames per second) that the system wants to achieve in the scheduling control module. The virtual machine list records the real-time information of the virtual machine operation in detail. The virtual machine running information in the machine list. 2. The image application programming interface analysis module calculates the overhead caused by the running of GPU commands (such as rendering screen function commands) and sends it to the scheduling control module. 3. The monitoring module obtains the real-time running information of the virtual machine and the GPU command overhead from the scheduling control module, and sends them to the scheduling module together. On the one hand, the scheduling module receives commands from the GPU command distributor from the upper layer and calculates the running time of the sleep function by combining the virtual machine running information obtained from the monitoring module and the overhead caused by the GPU command running, and processes the received commands. Finally, it is sent to the underlying host GPU driver: if the information shows that the FPS is greater than 30, a sleep function is inserted before calling the rendering screen function, so as to achieve the purpose of extending the delayed frame, and finally the rendering screen function is called to render the screen. If it is less than 30, the scheduling module allocates more GPU resources for the virtual machine (FPS less than 30), and then calls the rendering screen function to render the screen. If it is equal to 30, directly call the render screen function to render the screen.

When the virtual machine is running, the calculation time of the Present call is very stable, and even if there is a fluctuation, it will gradually change smoothly. Therefore, we believe that the agent of each virtual machine can predict the calculation amount of the Present through the historical information of the respective virtual machine. This algorithm uses The arithmetic mean of the past 20 consecutive Present call times is used as the prediction of the next frame. In order to make the frame length of each virtual machine stable at the set value FPS, each frame is extended by delaying its last Present call. The effect of delay is achieved by inserting a Sleep call before each Present call, and the amount of sleep is given by the formula. We found that when there is a lot of competition for GPU resources, the amount of time for Present will change significantly, and we can get more accurate GPU calculation time by inserting the flush command before the Sleep call.

An example of the present invention is briefly described here. Assuming that the standard FPS is 30, and a total of 3 virtual machines V1, V2, and V3 are running on the system, we assume that the ratios of historical GPU usage and FPS are t1, t2, and t3, respectively. If V1 runs at 20FPS at the beginning, V2 runs at 40FPS, and V3 runs at 60FPS, then the scheduling driver module first receives the performance parameters transmitted from the monitor, and once it detects that V1 does not meet the QoS requirements, QoS self-adaptation based on fairness The scheduling algorithm starts to work, and first calculates the GPU resources that should be allocated to V1, 10t1. The other two virtual machines V2 and V3 should prepare to release the resources of 5t1 respectively

The specific operating platform of this embodiment is the CPU of i7-2600K 3.4GHz, 16GB RAM, the video memory of AMD HD6750 graphics card and 2GB, host operating system and sub-operating system all adopt Windows7x64, the version of VMware is 4.0, each host operating system It is dual-core and has 2GB of RAM, and the screen resolution is 1280*720 (high-definition quality). In order to simplify the performance comparison, this embodiment disables the swap space and GPU accelerated window system on the host machine.

This embodiment adopts two types of loads, the first is "ideal model game" and the other is "realistic model game". The code names of the three games are A, B, and C respectively. First, we tested and evaluated the effect of the control system using PI control. Figure 5 shows the performance of the control system at kp=0.5, kp=0.25, kp=0.1 and ki=0.5, ki=0.1, ki=0.05 From the performance, it can be concluded that the QoS adaptive scheduling algorithm improves the performance of the system.

Analysis of Sleep effect and GPU reallocation, the system developed a library injection technology to insert the sleep function when calling the GPU API, we evaluated the effect of the sleep function on controlling FPS and GPU resource utilization, in this test, There is only one virtual machine running in the control system, the image display is set to 1920*1200 resolution, the initial sleep time is adjusted to 300ms per frame, and then decreases by 1ms per second. As shown in Figure 6, the obtained experimental data shows that the GPU and CPU resources of one virtual machine can be obtained by other virtual machine resources, that is, the system architecture and algorithm can effectively schedule GPU resources in multiple virtual machines.

Finally, test and evaluate the scheduling algorithm performance of the system. Fairness-based QoS adaptive scheduling algorithm, first, C has the highest FPS while B's FPS is lower than 30 due to GPU resource competition. Then the FSA scheduling strategy starts to play a role. In the second second, the system detects that the FPS of B is about 28 and the other two loads are greater than 30. The system releases the GPU resources from A and C and reassigns them to B, so at the third second, B's FPS increases to 33 and the other two loads decrease only a little, and C has the highest FPS and A's FPS has not decreased below the set standard. It can be seen from Figure 7 that for GPU resource utilization, the maximum is 99.1%, the minimum is 85.2%, and the average is 92.7%. Although there is still a small amount of waste of GPU resources, the fairness-based QoS adaptive scheduling algorithm Basically, the GPU utilization can be maintained at a very high level most of the time.

Through testing, the GPU virtual resource adaptive scheduling host system and adaptive scheduling method in the cloud game of the present invention realize the scientific management of GPU resources in the virtual machine and improve the utilization rate of GPU resources. Among them, the fairness-based QoS adaptive scheduling algorithm allocates enough GPU resources to all running virtual machines in a fair way and ensures compliance with QoS requirements, while maximizing the overall GPU utilization. Tests show that the algorithm achieves the corresponding goals under various loads, and limits the overhead to 5-10%.

Claims (3)

1. the GPU virtual resource adaptive scheduling host machine system in a cloud game, it is characterized in that, the proxy module comprising dispatching control module (1) and be connected respectively with this dispatching control module (1), image application program DLL analysis module (4) and virtual machine list (5), described proxy module is made up of scheduler module (2) and monitoring modular (3), all modules are all deployed in host

Described dispatching control module (1) is responsible for the communication of information between other each modules: dispatching control module (1) obtains the information of the virtual machine of all operations from virtual machine list (5), and sends to the module of needs;

Described monitoring modular (3) obtains the operation real time information of virtual machine from dispatching control module (1) and it is sent to scheduler module (2),

Described scheduler module (2) receives the order from upper strata GPU order distributor, and the operation real time information of the virtual machine sent according to monitoring modular (3), the order received is processed, is finally sent to lower floor main frame GPU and drives;

Described proxy module determines to need how many GPU resource are distributed to virtual machine together with dispatching control module (1);

Described image application program DLL analysis module (4) is responsible for calculating the GPU order produced by image application program DLL and is run the expense brought, and result is sent to dispatching control module (1), thus other modules needing this information are obtained

Virtual machine list (5) saves the virtual machines run all in described host machine system, each virtual machine can by system index, when there being new virtual machine add and bring into operation, system will automatically add it in virtual machine list (5) and again reschedule GPU resource.

2. utilize the dispatching method of the GPU virtual resource adaptive scheduling host machine system in the cloud game described in claim 1, it is characterized in that, the method comprises the steps:

1. user arranges the number of pictures per second (FPS) reached in dispatching control module, have recorded the real time information that virtual machine runs in virtual machine list, dispatching control module obtains the information of the virtual machine of all operations from virtual machine list (5);

2. image application program DLL analysis module (4) calculates GPU order and runs the expense brought, and is sent to dispatching control module (1);

3. monitoring modular (3) obtains the real time information of the virtual machine operation of recording in virtual machine list by dispatching control module (1), and run by the GPU order that dispatching control module (1) acquisition image application program DLL analysis module calculates the expense brought, and they are together sent to scheduler module (2);

4. scheduler module (2) receives the order from GPU order distributor, and the real time information run according to the virtual machine that obtains from monitoring modular and GPU order run the running time of the overhead computational sleep function brought, the order received is processed, is finally sent to lower floor main frame GPU and drives.

3. dispatching method according to claim 2, is characterized in that, described step concrete steps are 4. as follows:

The transmission frame number per second shown from the real time information that the virtual machine that monitoring modular obtains runs when scheduler module (2) is greater than 30fps, then scheduler module inserts sleep function before calling rendered picture function, then calls rendered picture function rendered picture;

The transmission frame number per second of the real time information display that the virtual machine obtained from monitoring modular when scheduler module (2) runs is less than 30fps, then scheduler module distributes more GPU resource for this virtual machine, and then calls rendered picture function rendered picture;

The transmission frame number per second shown from the real time information that the virtual machine that monitoring modular obtains runs when scheduler module (2) equals 30fps, then scheduler module directly calls rendered picture function rendered picture.