CN116991553A - A virtual GPU allocation method and system in a container cloud environment based on API interception and forwarding - Google Patents

Abstract

The invention relates to a virtual GPU allocation method and system in a container cloud environment based on API interception and forwarding. The method includes: abstracting the physical GPU in the cloud environment into fine-grained virtual GPU resources in the video memory dimension and computing dimension; using a declarative approach to allocate a virtual GPU with a specified amount of computing resources and a specified amount of video memory resources to the container; through customization The scheduler schedules the container to the appropriate working node and binds it to the appropriate GPU device; inserts custom CUDA, OpenCL, HIP and other API interception libraries to intercept relevant API calls when the container calls CUDA and other APIs; The intercepted APIs related to computing resources and video memory resources implement resource quota algorithms, and then forward and call real CUDA and other APIs; when the elastic allocation mode is enabled, the computing resources of the virtual GPU of the container in the idle phase are recycled and allocated to other appropriate Containers are used.

Description

A virtual GPU allocation method in a container cloud environment based on API interception and forwarding and system

Technical field

The invention belongs to the field of software technology, and specifically relates to a virtual GPU allocation method and system in a container cloud environment based on API interception and forwarding.

Background technique

With the continuous development of cloud native technology, more and more high-performance computing tasks using GPUs are deployed on GPU clusters in the cloud environment in the form of containers, and these high-performance computing tasks use GPU resources on the cloud. Compute acceleration. According to public data, Microsoft's cloud cluster contains 2,490 GPU resources, and Alibaba's single cluster contains 6,500 GPU resources. Current mainstream cloud service providers (such as Amazon, Google, Alibaba, Huawei, etc.) have integrated container orchestration frameworks such as Kubernetes into their infrastructure to support container clouds. Currently, many companies and communities have explored ways to use GPUs in containers and deploy high-performance computing tasks in container clouds. However, most of the current solutions allow containers to exclusively occupy GPU resources. For example, Kubernetes uses device plug-ins to allocate the entire GPU to a single container. In this case where the container monopolizes GPU resources, running high-performance computing tasks inevitably leads to insufficient utilization of GPU resources. Some solutions that allow containers to share GPUs do not take into account resource recycling and reallocation of containers when GPU computing is idle, which also leads to a waste of GPU resources. Therefore, in a multi-tenant environment, how to share GPU resources between different containers is very important.

The solution to efficiently share GPUs between containers is to allocate virtual GPUs to different containers. Each virtual GPU has a specified computing resource quota and video memory resource quota. This process is also called GPU virtualization. The current mainstream GPU virtualization methods are performed at different levels of the GPU software and hardware call stack.

GPU virtualization is performed at the application software layer. For example, AntMan (XiaoW, Ren S, Li Y, et al. AntMan: Dynamic Scaling on GPU Clusters for Deep Learning [C]//OSDI.2020:533-548.) studied and published by the Alibaba research team is in the deep learning framework. The layer implements GPU virtualization. By modifying deep learning frameworks such as PyTorch and Tensorflow, AntMan realizes GPU sharing for deep learning tasks at the framework layer, and allocates and limits the amount of computing resources and video memory resources for different deep learning tasks. AntMan takes the operators of the deep learning computational graph model as the granularity and limits the usage of task GPU computing resources by limiting the operation of the operators. AntMan maintains a video memory resource pool and limits the usage of task GPU video memory resources through the video memory resource pool. The disadvantage of GPU virtualization at the application software layer is that it only supports high-performance computing tasks in specific application scenarios and is highly bound to application software such as deep learning frameworks.

GPU virtualization is performed at API layers such as CUDA, OpenCL, and HIP. For example, the latest research results KubeShare (YehT A, Chen H H, Chou J. Kubeshare: A framework to manage gpus as first-class and shared resources in container cloud [C]//Proceedings of the 29th international symposium on high-performance parallel and distributed computing. 2020:173-184.) Implement GPU virtualization at the CUDA driver-level API layer. The main idea of KubeShare is to allocate a specified amount of GPU computing resources and video memory resources to a high-performance computing task, intercept the task's API calls related to the computing resources and video memory resources at the CUDA driver-level API layer, and decide whether to forward and call the real API. , thereby achieving the resource limitation effect. The advantage of GPU virtualization at the API layer is that it is independent of application software and is universal; the disadvantage is that it is less secure and requires frequent updates to adapt to the latest API.

GPU virtualization is performed at the GPU driver layer. This method implements more virtualization functions, such as dynamic migration and other features, and is more secure without the need to frequently change the GPU acceleration software library. However, due to the low level of virtualization, the implementation of virtualization is relatively complex, and due to the GPU manufacturer's strict closed-source policy on driver code, virtualization at the GPU driver layer is more difficult and usually requires the use of reverse engineering and other technologies.

GPU virtualization is performed at the hardware level. For example, NVIDIA's Tesla A100 uses hardware-level virtualization. Tesla A100 divides the computing unit SM into 7 parts equally and the video memory resources into 8 parts evenly. Task processes running on the Tesla A100 will have exclusive access to their own hardware resources (such as on-chip cross-connect ports, L2 cache banks, memory controllers, and DRAM address buses). GPU virtualization at the hardware level allows applications to directly access the physical GPU device instead of forwarding through API or driver forwarding. The advantage of virtualizing the GPU at the hardware level is that users do not need to change the GPU's acceleration software library and GPU driver, and get near-native performance due to hardware support, and the application is completely safe when executing; the disadvantage is that virtualization is controlled by the GPU hardware It is more expensive to implement it through the provider, and because it does not go through high-level layers such as the operating system, it is impossible to customize some policies. It is difficult to implement resource scheduling algorithms, checkpoint functions, and real-time like the GPU virtualization method based on API interception and forwarding. Features such as migration and fault tolerance.

Contents of the invention

In order to overcome the shortcomings of existing technical solutions, the purpose of the present invention is to provide a virtual GPU allocation method and system in a container cloud environment based on API interception and forwarding. First, the physical GPU in the cloud environment is abstracted into fine-grained virtual GPU resources in the memory dimension and computing dimension, which are allocated to containers; then the logic of high-performance computing tasks using computing resources and video memory resources on the GPU is analyzed, and performance modeling is performed. , implement a virtualization strategy based on API interception and forwarding to achieve the effect of resource quota; finally provide a working mode of elastic allocation of computing resources, recycle the computing resources of the virtual GPU of the container in the idle stage, and allocate them to other suitable containers for use, improving the cluster GPU resource utilization.

The technical solution of the present invention is:

A virtual GPU allocation method in a container cloud environment based on API interception and forwarding, including the following steps:

Abstract the physical GPU in the cloud environment into fine-grained virtual GPU resources in the memory dimension and computing dimension;

Use a declarative approach to allocate a virtual GPU with a specified amount of computing resources and a specified amount of video memory resources to the container;

Through the custom scheduler, the container is scheduled to the appropriate working node and bound to the appropriate GPU device;

Insert custom API interception libraries such as CUDA, OpenCL, HIP, etc., intercept relevant API calls when the container calls CUDA and other APIs, implement resource quota algorithms on the intercepted APIs related to computing resources and video memory resources, and then forward and call the real CUDA etc API;

When the elastic allocation mode is enabled, the computing resources of the virtual GPU of the container in the idle phase are recovered and allocated to other appropriate containers.

Further, the physical GPU in the cloud environment is abstracted into fine-grained virtual GPU resources in the memory dimension and computing dimension, including:

Use the detection interface provided by the GPU manufacturer to detect the physical GPU information on the node;

The execution time of the physical GPU is evenly divided into 100 parts. Each execution time corresponds to a granular virtual GPU computing resource and also corresponds to the GPU utilization;

The video memory of the physical GPU is divided into several parts according to the size of 1MB, and each part of the video memory corresponds to a granular virtual GPU video memory resource;

Expand and implement the device plug-in function provided by Kubernetes, and register and report fine-grained virtual GPU computing resources and virtual GPU memory resources.

Further, the use of a declarative approach to allocate a virtual GPU with a specified amount of computing resources and a specified amount of video memory resources to the container includes: the user submits a high-performance computing task Pod using GPU resources to the Kubernetes cluster through a YAML description file, and specifies the container The required number of virtual GPU memory resources and virtual GPU computing resources.

Further, the custom scheduler is used to schedule the container to the appropriate working node and bind it to the appropriate GPU device, including:

The Pod scheduler monitors and schedules Pods that require virtual GPUs through the watch mechanism of Kubernetes;

After the monitor listens to the submitted Pod, it puts the Pod into the scheduler task queue;

The notifier listens to CRD and obtains the usage of GPU resources in the cluster at all times;

When the available GPU resources meet the conditions, the notifier notifies the scheduler that the Pod at the head of the task queue is dequeued;

The dequeued Pod executes the scheduling and binding process. Through filtering, scoring, binding, etc., the Pod is scheduled to the appropriate node and GPU and bound;

The Pod scheduler internally sets common scheduling algorithms (first adaptation algorithm, best adaptation algorithm, worst adaptation algorithm, etc.) for users to choose in different scenarios. Users can also expand the interface through reserved scheduling algorithms to implement customized scheduling. algorithm.

Further, the custom API interception libraries such as CUDA, OpenCL, and HIP are inserted to intercept relevant API calls when the container calls APIs such as CUDA, implement a resource limit algorithm on the intercepted APIs related to computing resources and video memory resources, and then forward them. And call real CUDA and other APIs, including:

Compile a dynamic link library containing all APIs of native CUDA, OpenCL, HIP and other acceleration platforms;

Through the hooking mechanism of the Linux operating system, set the container environment variable LD_LIBRARY_PATH to the directory of API interception libraries such as CUDA to intercept the APIs they call;

Intercept API calls related to computing resources and video memory resources, and execute resource quota algorithms;

For the limit of computing resources, the monitoring-based GPU computing resource limit algorithm is used;

For the quota of video memory resources, a quota-based GPU video memory resource quota algorithm is used;

By using system calls such as dlopen and dlsym of the Linux operating system, the address of the real CUDA and other APIs is obtained and the relevant APIs are called to achieve the effect of forwarding the API.

Further, the monitoring-based GPU computing resource quota algorithm includes:

Monitor the GPU utilization of the container in real time through the relevant interfaces of the GPU manufacturer;

Quantify the execution time of computing units such as CUDA kernel functions and the available time of the GPU;

Run in producer-consumer mode, dynamically adjust the rate at which high-performance computing tasks in the container start computing units such as CUDA kernel functions.

Further, the quota-based GPU memory resource limit algorithm includes:

Maintain a video memory quota for each container and record the current video memory size used by the container. The video memory quota value is 0 when the container is created;

When a process in the container calls APIs related to applying for or releasing video memory, it uses the current quota size to determine whether to forward the call to the real API.

Further, when the elastic allocation mode is enabled, the computing resources of the virtual GPU of the container in the idle phase are recovered and allocated to other suitable containers for use, including:

Divide tasks into two categories, one is Resource Sensitive Tasks (RST), and the other is Resource Insensitive Tasks (RIT);

An algorithm based on sliding window sampling is used to determine whether the high-performance computing task is in the "idle phase";

Divide computing resources on the GPU into fixed computing resources and elastic computing resources, and reallocate the recovered elastic computing resources;

The value of fixed computing resources is equal to the sum of the computing resources of all RSTs that are not elastically increased;

The value of elastic computing resources is equal to the sum of total resources minus fixed resources minus the computing resources of RIT in a restricted state;

When there is no RIT in the GPU that is not in a restricted state, elastic computing resources are evenly allocated to RST;

When there are RITs in the GPU that are not in a restricted state, elastic computing resources are evenly allocated to the RITs.

A virtual GPU allocation system in a container cloud environment based on API interception and forwarding using the above method, the system includes:

The Pod scheduler is used to select appropriate nodes and GPUs for high-performance computing task Pods submitted by users, and bind and execute them;

The device plug-in module is used to abstract virtual GPU resources and publish them to Kubelet, and apply for and allocate virtual GPUs with specified computing resources and video memory resource quotas for high-performance computing task Pods;

Pod resource manager is used to interact with API interception libraries such as CUDA, maintain resource quotas throughout the life cycle of Pods, and achieve elastic computing resource allocation;

The API interception library module is used to intercept and forward CUDA and other APIs, execute resource quota algorithms, and achieve the effect of virtual GPU computing resource and video memory resource limits.

The advantages of the present invention compared with the prior art are:

In view of the characteristics of the GPU software and hardware call stack for high-performance computing tasks, the present invention proposes a method of virtualizing the GPU in the open source API layer of acceleration platforms such as CUDA, OpenCL, and HIP, thus avoiding the impact of the closed-source strategy of the GPU manufacturer.

The present invention proposes a corresponding GPU resource quota algorithm based on the principle of using GPU computing resources and video memory resources for high-performance computing tasks, which has the advantages of accurate quota and low overhead.

In view of the idle phenomenon of computing resources when high-performance computing tasks are running, the present invention proposes a computing resource allocation strategy based on an adaptive elastic allocation algorithm, thereby reducing the waste of GPU computing resources.

In view of the deployment characteristics in the current cloud environment, the present invention designs and develops a corresponding software system based on Kubernetes, which has the characteristics of compatibility with native Kubernetes, convenient deployment, and good interactivity.

Description of the drawings

Figure 1 is a schematic system structure diagram of the virtual GPU allocation system of the present invention.

Figure 2 is a schematic flowchart of the execution of the Pod scheduler of the present invention.

Detailed ways

The technical solution of the present invention will be further described below with reference to the accompanying drawings of the present invention. The described embodiments are some of the embodiments of the present invention, but do not represent all of the embodiments.

For those skilled in the art, some well-known technologies may not be described in detail.

A virtual GPU allocation method in a container cloud environment based on API interception and forwarding of the present invention includes the following steps:

Abstract the physical GPU in the cloud environment into fine-grained virtual GPU resources in the memory dimension and computing dimension;

Use a declarative approach to allocate a virtual GPU with a specified amount of computing resources and a specified amount of video memory resources to the container;

Through the custom scheduler, the container is scheduled to the appropriate working node and bound to the appropriate GPU device;

Insert custom CUDA, OpenCL, HIP and other API interception libraries to intercept related API calls when the container calls CUDA and other APIs;

Implement the resource quota algorithm on the intercepted APIs related to computing resources and video memory resources, and then forward and call the real CUDA and other APIs;

When the elastic allocation mode is enabled, the computing resources of the virtual GPU of the container in the idle phase are recovered and allocated to other appropriate containers.

A virtual GPU allocation system in a container cloud environment based on API interception and forwarding of the present invention consists of four parts: a Pod scheduler, a device plug-in module, a Pod resource manager, and an API interception library module, wherein:

Pod scheduler: By running a series of scheduling algorithms, it selects appropriate nodes for high-performance computing tasks Pod submitted by users to the Kubernetes cluster (Pod is the smallest computing unit for Kubernetes cluster management and scheduling, representing a group of containers in a running state) and GPU for binding and execution. The Pod scheduler implements the scheduling function through the watch (monitoring) mechanism. Common scheduling algorithms (first adaptation algorithm, best adaptation algorithm, worst adaptation algorithm, etc.) are set up internally for users to choose in different scenarios. Users can also pre-set the scheduling function. The reserved scheduling algorithm extension interface can implement customized scheduling algorithms.

Device plug-in module: By extending the framework provided by Kubernetes to support custom resources, it abstracts virtual GPU resources and publishes them to Kubelet, and applies for and allocates specified computing resources and video memory resource quotas to high-performance computing task Pods. Virtual GPU. The device plug-in is deployed on each worker node through DaemonSet and registered with Kubelet through the gRPC service. When running, the device plug-in starts a gRPC service under its own socket to provide support to Kubelet, including monitoring and reporting physical GPU resources, and allocating virtual GPUs to high-performance computing task Pods.

Pod Resource Manager: Responsible for interacting with API interception libraries such as CUDA, maintaining resource quotas throughout the Pod's life cycle, and achieving elastic computing resource allocation. When a Pod starts, the Pod Resource Manager will obtain the Pod container's process ID, virtual GPU resource allocation, and other information, and then use gRPC to communicate with API interception libraries such as CUDA to complete some initialization work. At runtime, the Pod Resource Manager manages the allocation of virtual GPUs for high-performance computing task Pods throughout their life cycle, continuously monitors the resource usage of Pods, and implements elastic shrinkage and allocation strategies for computing resources.

API interception library module: By customizing API interception libraries such as CUDA, OpenCL, HIP, etc., it intercepts and forwards CUDA and other APIs, and executes resource quota algorithms to achieve the effect of virtual GPU computing resource and video memory resource quota. The API interception library module uses the hook mechanism provided by the Linux operating system to implement interception, and uses the relevant system calls of Linux to obtain the real CUDA and other API addresses for forwarding. The API interception library module interacts with the Pod Resource Manager through gRPC, obtains the virtual GPU resource quota allocated by the container, and implements the GPU resource limit.

For example, the virtual GPU allocation in the container cloud environment means that the user submits a high-performance computing task Pod using the GPU to the Kubernetes cluster in a declarative manner, and specifies a custom amount of GPU memory resources and computing resource quota for the Pod. , the system binds Pod scheduling to the appropriate physical GPU for execution, and limits GPU memory resources and computing resources according to the corresponding quota, so that different Pods are isolated from each other.

For example, in the declarative approach, the format of the declarative Pod description file is YAML, and the specific standards are as follows:

The Pod description file is used to provide information such as the name of the Pod, the name of the Pod scheduler, the name of the container image, the container startup parameters, and the resource share required by the container. An example of a Pod description file is as follows. The angle brackets after the field identify <field type: field content>, where str is a string type.

Among them, the content of the field apiVersion is the API version number of Kubernetes, which is used to specify the version of the current corresponding API server. It is used to ensure the upper and lower compatibility of API servers between different versions; the content of the field kind is the type of resource, which is the resource type supported by this system. It is a Pod; the content of the field metadata→name is the name of the Pod, which is unique; the content of the field spec→schedulerName is the name of the Pod scheduler used by the cluster; the content of the field spec→containers→image is the image name and version of the container; The content of the field spec→containers→name is the name of the container; the content of the field spec→containers→command is the startup parameters of the container; the content of the field spec→containers→resources→limits→doslab.io/gpu-memory is allocated for the container GPU memory resource quota, in MB; the content of the field spec→containers→resources→limits→doslab.io/gpu-core is the GPU computing resource quota allocated for the container. Each physical GPU is divided into 100 computing resources. , corresponding to the GPU utilization.

The following describes the preferred implementation of the key processes of the present invention:

(1) Pod submission and scheduling process

When the system is deployed, the Pod scheduler runs on the Master node of the cluster. When there are Pods using virtual GPUs, the scheduling process will be executed. Users submit high-performance computing task Pods using GPUs to the Kubernetes cluster through declarative description files YAML, and need to specify the amount of GPU memory resources and computing resources required in the container. Kubernetes will parse and generate Pod instance objects in the cluster, which will be captured by the watch mechanism of the Pod scheduler and execute the scheduling logic.

The Pod scheduler puts the captured high-performance computing task Pod into the scheduler task queue. When the GPU resources meet the conditions, the notifier notifies the dequeue and enters the scheduling and binding process. The usage of GPU resources in the cluster comes from the CRD (Custom Resource Definitions) reported by the device plug-in module. First enter the filtering stage, which selects all nodes that meet Pod scheduling requirements (such as CPU, memory and other resource requirements) for subsequent scheduling work. Then enter the scoring stage, and select the most suitable node for the Pod and the GPU device on the node based on the corresponding scheduling algorithm and resource allocation. Finally, bind the corresponding node to the Pod, and write the GPU ID information into the Annotation field of the Pod.

(2) Abstraction and allocation process of virtual GPU

Device plug-ins are a framework provided by Kubernetes to support custom resources. When the system is deployed, the device plug-in module runs on each working node in the form of DaemonSet, abstracts the physical GPU on the node into fine-grained virtual GPU resources, and reports them to Kubelet; at runtime, the device plug-in module will detect the physical GPU changes and allocate a specified amount of virtual GPU resources to the container.

The device plug-in module first needs to register with Kubelet through the gRPC service, that is, it needs to implement the Register interface provided by the framework. The device plug-in module first monitors the GPU information on the node, including the GPU ID and other information, through the relevant interfaces provided by GPU manufacturers such as Nvidia. Then the device plug-in module registers with Kubelet through the gRPC service and transmits the device plug-in's Unix socket, device plug-in API version, device resource name and other information, in which the GPU memory resources and GPU computing resources are abstracted into fine-grained virtual GPU resources. Register into Kubelet and report to CRD for scheduling. For a physical GPU, the video memory is abstracted into virtual GPU video memory resources according to the granularity of 1MB, and the computing power is abstracted into virtual GPU computing resources according to the granularity of 1% utilization. Allocating a virtual GPU to a container essentially allocates a specified number of virtual GPU computing resources and virtual GPU memory resources to the container through the device plug-in.

When running, the device plug-in module also needs to start a gRPC service under its own Unix socket to provide support to Kubelet. The device plug-in module needs to implement a series of interfaces provided by the framework, including the two interfaces ListAndWatch and Allocate. The ListAndWatch interface is used to return a data stream composed of a device list, that is, the number of virtual GPU memory resources and the number of virtual GPU computing resources. When a device is updated or the device disappears, the ListAndWatch interface will also notify Kubelet to update accordingly. When a GPU acceleration container is created, it will apply for allocation of virtual GPU memory resources and virtual GPU computing resources. Kubelet will call the Allocate interface to allocate and run some device-specific operations. In the process of calling the Allocate interface, the device plug-in module mounts the directory for the container (such as mounting the directory of API interception libraries such as CUDA), sets environment variables (such as the NVIDIA_VISIBLE_DEVICES environment variable indicating which physical GPU is used), and sets the Annotation and other processes to enable the container to use physical GPU resources.

(3)API interception and forwarding and resource quota process

After the high-performance computing Pod submitted by the user is scheduled by the Pod scheduler and the device plug-in module allocates virtual GPU resources, the Pod resource manager will obtain the process ID of the Pod container, resource specified quota and other information, and then use gRPC to communicate with CUDA, etc. The interception library communicates and transmits relevant information to interception libraries such as CUDA, so that interception libraries such as CUDA can complete some initialization work. The API interception library such as CUDA is a dynamic link library customized by the present invention. It has the same API as the native CUDA, OpenCL, HIP and other computing acceleration libraries. Through the hooking mechanism of the Linux operating system, set the container environment variable LD_LIBRARY_PATH to the directory of API interception libraries such as CUDA to achieve the effect of intercepting APIs. By using system calls such as dlopen and dlsym of the Linux operating system, the address of the real CUDA and other APIs is obtained and the relevant APIs are called to achieve the effect of forwarding the API.

The container is allocated a certain amount of GPU computing resources and video memory resources. When running, it must be strictly ensured that the amount used does not exceed the allocated amount. The average GPU utilization of all processes in the container should be approximately equal to the allocated amount of GPU computing resources; the sum of the GPU memory usage of all processes in the container must not exceed the allocated amount of GPU memory resources.

As for the limit of computing resources, the present invention adopts a monitoring-based GPU computing resource limit algorithm. Since the main process of calling the GPU for acceleration of high-performance computing tasks is to launch and start computing units such as CUDA kernel functions on the GPU, the core idea of the algorithm is based on monitoring and adjustment algorithms. By using relevant monitoring interfaces provided by GPU manufacturers such as Nvidia, Monitor the GPU utilization of the container in real time, and dynamically adjust the rate at which high-performance computing tasks in the container start CUDA kernel functions and other computing units, so that the actual GPU utilization is roughly equal to the computing resource quota specified by the container.

In order to make the entire monitoring and adjustment process smoother and reduce fluctuations in GPU utilization, the present invention quantifies the execution time of computing units such as CUDA kernel functions and the available time of the GPU. The available time limit of the GPU can be regarded as a globally maintained resource, running in the Producer Consumer Pattern. The real-time GPU utilization monitoring program is a producer. When the GPU utilization is lower or higher than the allocated value, it increases or decreases the available time quota. The changing time quota can be expressed by the formula as increment=(allocation-current)×α , where α is an empirical value, each model of GPU is different, allocation represents the amount of GPU computing resources allocated to the container, that is, the expected GPU utilization, and current represents the sum of the actual GPU utilization of all processes in the container. Computing units such as CUDA kernel functions are consumers. When CUDA kernel functions and other computing units are started, a time limit is consumed. The time limit consumed can be expressed as consume=f(Block,Thread)×α, where f is experience Function, Block and Thread are the input parameters of computing units such as CUDA kernel function, α is an empirical value, and each model of GPU is different from each other. In addition, in order to ensure that the effect of GPU resources and quotas can be achieved, it is necessary to strictly ensure that the production rate of the producer is not lower than the consumption rate of the consumer, otherwise starvation will occur and the GPU utilization will not reach the target value.

As for the limit of video memory resources, the present invention adopts a quota-based GPU video memory resource limit algorithm. The container will continuously apply for and release GPU memory resources throughout its life cycle. Therefore, the present invention maintains a video memory quota for each container and records the size of the video memory currently used by the container. When the container is created, the video memory quota value is 0.

When the process in the container calls APIs related to video memory application such as cuArray3DCreate, cuMemAlloc, etc., interception libraries such as CUDA intercept the call, obtain the size of the video memory requested, and add the applied video memory value to the current video memory quota of the container to determine whether it exceeds If the size of the specified quota exceeds the specified quota, an exception of insufficient video memory will be reported, and interception libraries such as CUDA will not continue to call the real acceleration library; if the size of the specified quota is not exceeded, interception libraries such as CUDA will call the real acceleration library. Execute the API that allocates video memory and return the real result of its call. When the process in the container calls APIs related to video memory release such as cuArrayDestroy and cuMemFree, interception libraries such as CUDA intercept the call, obtain the size of the released video memory through the pointer, and subtract the released video memory value from the current video memory quota of the container, and then The interception library such as CUDA calls the real acceleration library, executes the API to release the video memory, and returns the real result of its call.

(4) Elastic shrinkage and allocation process of computing resources

High-performance computing tasks submitted by users often do not fully utilize the allocated GPU computing resources during operation, resulting in a waste of resources. In order to solve this problem, the present invention proposes an adaptive elastic allocation algorithm based on The computing resource allocation strategy monitors and detects the actual GPU computing resource usage of the container, dynamically allocates and recycles GPU computing resources, and improves the utilization of GPU resources.

The present invention divides high-performance computing tasks into two categories, one is resource-sensitive tasks (ResourceSensitive Tasks, RST), and the other is resource-insensitive tasks (Resource Insensitive Tasks, RIT). Among them, resource-sensitive tasks refer to tasks that have higher requirements for GPU computing resources. When a task requires computing resources, its specified computing resource requirements are strictly guaranteed and will not be preempted; resource-insensitive tasks refer to tasks that require GPU computing resources. The requirements for computing resources are low, and computing resources can be preempted during the entire life cycle of the execution task. Both RST and RIT will specify the specific size of GPU memory resources when submitting, while RIT does not specify the specific size of GPU computing resources when submitting, and its value is fixed at 0.

When the actual GPU utilization of a high-performance computing task within a period of time is significantly lower than the specified GPU computing resource quota and exceeds a threshold, the task can be considered to be in the "idle phase". At this time, the specified GPU computing resource quota for the task is reduced to the actual GPU utilization to avoid waste of GPU computing resources.

The present invention adopts an algorithm based on sliding window sampling to determine whether the high-performance computing task is in the "idle phase". The specific approach is to maintain a sliding window based on monitoring and obtain the GPU utilization of high-performance computing tasks within the window time period in real time. Once a high-performance computing task enters the "idle phase", the task's GPU computing resource quota will be automatically reduced, and the task will also enter a restricted state. After a period of time, the restricted state of the task is released and its specified GPU computing resource quota is restored. This process is called a performance probe, and the purpose is to detect whether the task has restored its original GPU computing resource utilization. If the task is still in the "idle phase", the resource shrinking process will be started immediately, and the task will re-enter the restricted state.

GPU computing resources can be divided into elastic resources and fixed resources. Let the total amount of computing resources of a certain GPU be Core, the set of RSTs running on the GPU is J={J ₁ , J ₂ ,..., J _n }, and the set of RITs running on the GPU is j={j ₁ , j ₂ ,..., j _m }. The fixed computing resource Core _b allocated on the GPU at a certain moment can be expressed by formula (1), and its value is equal to the sum of the computing resources of all RSTs that have not been elastically increased.

The elastic computing resource Core _e on the GPU at a certain moment can be expressed by formula (2), and its value is equal to the sum of the total resources minus the fixed resources minus the computing resources of the RIT in a restricted state.

Among them, restricted represents a task in a restricted state, and j∩restricted represents a RIT in a restricted state.

When there is no RIT in the GPU that is not in a restricted state, the elastic computing resources are evenly allocated to the RST, as shown in formula (3). Among them, Core _i is the current resource allocation of task i, Alloc _i is the initial resource allocation of task f, and n is the number of RSTs that are not in a restricted state.

When there are RITs in the GPU that are not in a restricted state, the elastic computing resources are evenly allocated to the RITs, as shown in formula (4).

Among them, Core _i is the current resource allocation of task i, and m is the number of RITs that are not in a restricted state.

In one embodiment of the present invention, a virtual GPU allocation method in a container cloud environment based on API interception and forwarding is provided. As shown in Figure 1, the method as a whole can be divided into the following steps:

Step 001: The user deploys the device plug-in module to each Worker node (worker node) in the form of DaemonSet, and the device plug-in module registers virtual GPU resources with Kubelet.

Step 101: Users submit Pods to the cluster through YAML description files.

Step 102: The Pod scheduler monitors the submitted Pod through the watch mechanism.

Step 103: The Pod scheduler schedules the Pod to the appropriate Worker node and GPU through the scheduling algorithm and binds it.

Step 104: The device plug-in module monitors the successfully scheduled Pod through the watch mechanism.

Step 105: The device plug-in module interacts with Kubelet to allocate virtual GPU resources to the Pod.

Step 106: After the device plug-in module allocates virtual GPU resources, it notifies the Pod resource manager.

Step 107: The Pod Resource Manager communicates with the API interception library module through gRPC to perform initialization and elastic allocation of computing resources.

Step 108: The API interception and forwarding module intercepts the API calls of the forwarding container to implement resource quotas.

In this embodiment, the scheduling process of the Pod scheduler is shown in Figure 2, which can be divided into the following steps:

Step 201: The monitor monitors the submitted Pod through the watch mechanism.

Step 202: After monitoring the submitted Pod, the monitor puts the Pod into the scheduler task queue.

Step 203: The notifier monitors CRD through the watch mechanism and obtains the usage of GPU resources in the cluster at all times.

Step 204: When the available GPU resources meet the conditions, the notifier notifies the scheduler that the Pod at the head of the task queue is dequeued.

Step 205: The dequeued Pod executes the scheduling and binding process. Through filtering, scoring, binding, etc., the Pod is scheduled to the appropriate node and GPU and bound.

Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, etc.), which includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be processed by the processor. Execution, the computer program includes instructions for executing each step in the method of the present invention.

Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM/RAM, magnetic disk, optical disk). The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, , implement each step of the method of the present invention.

The above description of specific implementation methods of the embodiments of the present invention is only illustrative, and the protection scope of the present invention is set forth in the claims. Any changes and modifications made by those skilled in the art based on understanding the above description will fall within the protection scope of the present invention.

Claims (10)

1. The virtual GPU distribution method under the container cloud environment based on API interception forwarding is characterized by comprising the following steps of:

abstracting a physical GPU in a cloud environment into virtual GPU resources with fine granularity of a video memory dimension and a calculation dimension;

a virtual GPU for distributing the appointed amount of computing resources and the appointed amount of video memory resources to the container in a declarative mode;

dispatching the container to a proper working node through a self-defined dispatcher, and binding the container with proper GPU equipment;

inserting an API interception library, intercepting related API calls when a container calls the API, implementing a resource quota algorithm on the intercepted computing resource and the related API of the video memory resource, and then forwarding and calling a real API;

when the flexible allocation mode is enabled, the computing resources of the virtual GPU in the idle phase container are reclaimed and allocated for use by other suitable containers.

2. The method of claim 1, wherein abstracting the physical GPU in the cloud environment into fine-grained virtual GPU resources of the memory dimension and the compute dimension comprises:

detecting information of a physical GPU on the node by using a detection interface provided by a GPU manufacturer;

the execution time of the physical GPU is uniformly divided into 100 parts, and each part of execution time corresponds to virtual GPU computing resources with one granularity and also corresponds to the utilization rate of the GPU;

dividing the video memory of the physical GPU into a plurality of parts according to the size of 1MB, wherein each part of video memory corresponds to virtual GPU video memory resources with one granularity;

and (3) expanding and realizing the function of the device plug-in provided by Kubernetes, registering and reporting the virtual GPU computing resources and the virtual GPU video memory resources with fine granularity.

3. The method according to claim 1, wherein the scheduling, by the custom scheduler, the container to the appropriate working node and binding with the appropriate GPU device includes:

the Pod scheduler monitors the Pod of the virtual GPU through a watch mechanism of Kubernetes and schedules the Pod;

after monitoring the submitted Pod, the monitor puts the Pod into a task queue of a scheduler;

the notifier monitors the CRD and acquires the use condition of GPU resources in the cluster at any time;

when the available GPU resources meet the conditions, the notifier notifies the Pod at the head of the task queue of the scheduler to dequeue;

the dequeued Pod executes a scheduling and binding procedure, and schedules the Pod to a proper node and GPU and binds the Pod through filtering, scoring and binding links;

the scheduling algorithm is set in the Pod scheduler for the user to select in different scenes or the user expands the interface through the reserved scheduling algorithm to realize the self-defined scheduling algorithm.

4. The method according to claim 1, wherein inserting the API interceptor library, intercepting the associated API call when the container calls the API, implementing a resource quota algorithm on the intercepted computing resources and the memory resource associated API, and then forwarding and calling the real API, comprises:

compiling a dynamic link library containing all APIs of a native acceleration platform; the API includes CUDA, openCL, HIP;

setting a container environment variable LD_LIBRARY_PATH as a catalog of an API interception LIBRARY through a hooking mechanism of a Linux operating system, and intercepting an API called by the API interception LIBRARY;

intercepting API calls related to computing resources and video memory resources, and executing a resource quota algorithm;

for the quota of the computing resource, a GPU computing resource quota algorithm based on monitoring is adopted;

for the quota of the video memory resource, a GPU video memory resource quota algorithm based on quota is adopted;

the address of the real API is obtained and the related API is called by using dlopen and dlsym system calls of the Linux operating system, so that the effect of forwarding the API is achieved.

5. The method of claim 4, wherein the monitoring-based GPU calculates a resource quota algorithm comprising:

monitoring the GPU utilization rate of the container in real time through a relevant interface of a GPU manufacturer;

quantifying the execution time of a computing unit including the CUDA kernel function and the available time of the GPU;

running in producer consumer mode, the rate at which high performance computing tasks in the container start the computing units is dynamically adjusted.

6. The method of claim 4, wherein the quota-based GPU video memory resource quota algorithm comprises:

maintaining a video memory quota for each container, recording the current video memory size used by the container, and setting the value of the video memory quota to be 0 when the container is created;

when a process in the container calls a related API for applying or releasing the video memory, judging whether to forward and call a real API according to the current quota size.

7. The method according to claim 1, wherein when the flexible allocation mode is enabled, recovering the computing resources of the virtual GPU in the idle phase container and allocating to other suitable containers for use, comprising:

dividing tasks into two types, wherein one type is a resource sensitive task RST and the other type is a resource insensitive task RIT;

adopting a sliding window sampling-based algorithm to judge whether the high-performance computing task is in an idle stage;

dividing the computing resources on the GPU into fixed computing resources and elastic computing resources, and reallocating the received elastic computing resources;

the fixed computing resource has a value equal to the sum of the computing resources of all the RSTs that have not been increased elastically;

the value of the elastic computing resource is equal to the sum of the total resource minus the fixed resource minus the computing resource of the RIT in the limiting state;

when the RIT which is not in a limiting state is not available in the GPU, the elastic computing resources are uniformly distributed to RST;

when there is an RIT in the GPU that is not in a restricted state, the elastic computing resources are evenly allocated to the RIT.

8. A virtual GPU distribution system in an API-based intercept forwarding container cloud environment employing the method of any of claims 1-7, the system comprising:

the Pod scheduler is used for selecting proper nodes and GPU for the high-performance computing task Pod submitted by the user, and binding and executing the nodes and GPU;

the device plug-in module is used for abstracting virtual GPU resources and releasing the virtual GPU resources to the Kubelet, and applying for and distributing virtual GPU of appointed computing resources and video memory resource limit for the high-performance computing task Pod;

the Pod resource manager is used for interacting with the API interception library, maintaining resource quota in the whole life cycle of the Pod and realizing flexible computing resource allocation;

and the API interception library module is used for intercepting and forwarding the API, executing a resource quota algorithm and achieving the effect of limiting the computing resources and the video memory resources of the virtual GPU.

9. An electronic device comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program comprising instructions for performing the method of any of claims 1-7.

10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a computer, implements the method of any of claims 1-7.