CN114356543A - A Kubernetes-based Multi-tenant Machine Learning Task Resource Scheduling Method - Google Patents

Abstract

The invention discloses a multi-tenant machine learning task resource scheduling method based on Kubernetes, which performs quota management on the computing resources available to different users, monitors the resource status information of each Node node in the Kubernetes platform, and considers the resources of the host machine where the nodes are located. Utilization problem, avoid the problem of inaccurate scheduling results, and at the same time by monitoring real-time scheduling and pre-scheduling request demand information, according to the scheduling task demand information to prioritize each Node node, get the host label of the optimal node, according to the label Reasonable allocation of resource requirements for various machine learning model training and prediction tasks. The invention effectively prevents and reduces the inclination problem of node resource usage in the Kubernetes platform, realizes multi-node load balance, and improves the utilization rate of node resources.

Description

A Kubernetes-based Multi-tenant Machine Learning Task Resource Scheduling Method

technical field

The invention relates to a Kubernetes-based multi-tenant machine learning task resource scheduling method, which belongs to the technical field of power regulation.

Background technique

At present, the application of artificial intelligence technology in the field of power grid regulation has achieved preliminary results, but in the management and control of computing power resources, the computing power is scattered, which restricts the breakthrough of applications. Repeated construction of hardware resources, scattered computing power and difficult to expand.

The IaaS layer of the cloud computing platform mainly uses virtualization technology to achieve multi-tenant resource isolation and dynamic allocation. However, the traditional virtualization technology itself has a high occupancy rate of hardware resources and is not suitable for high utilization of computing resources for machine learning model training and prediction tasks. In addition, the complexity of application configuration, operation, and management is relatively high, which is not conducive to clustered overall management.

Kubernetes has the ability to automate service arrangement, deployment and resource scheduling, which is very popular among developers. The present invention performs customized arrangement and scheduling of resources based on Kubernetes, and supports artificial intelligence application development and service support platform in the new generation scheduling technology support system. The product development work of the invention is used for the resource scheduling of machine learning training and prediction tasks such as power grid fault identification and analysis, power grid operation prediction and analysis, and power grid intelligent dispatch auxiliary decision-making. Its application results verify the technical route and reliability of the present invention.

SUMMARY OF THE INVENTION

Objective: In order to overcome the deficiencies in the prior art, the present invention provides a multi-tenant machine learning task resource scheduling method based on Kubernetes, which adopts Kubernetes and container technology to uniformly control the IaaS layer CPU, GPU and memory resources, and builds a multi-tenant system. The application of machine learning model training and prediction standardizes the operating environment, and improves the controllability, elastic expansion capability and resource isolation capability of the power grid regulation system.

Technical scheme: in order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

A Kubernetes-based multi-tenant machine learning task resource scheduling method, comprising the following steps:

Calculate the difference between the resources used by the Node node in the cluster and the resources used by the created container, and obtain the resource information occupied by all the processes of the Node node operating system itself.

Call the Kubernetes API to obtain resource information requested by all machine learning model training and prediction task containers on the Node node.

The real-time available resource information of the Node node is calculated by subtracting the inherent resource capacity of the Node node from the resource information occupied by all the processes of the Node node operating system itself and the resource information applied by all machine learning model training and prediction task containers on the Node node.

According to the real-time available resource information of the Node node and the inherent resource capacity of the Node node, the availability rate of the CPU, GPU, and memory of the Node node is calculated.

The system cluster resource management and control service presets the resource threshold percentage. Node nodes whose CPU, GPU, and memory availability rates are not lower than the preset resource threshold percentages allocate computing resources for machine learning model training and prediction tasks.

The machine learning task scheduling service sends the number of CPU, GPU, and memory resources requested by different users' machine learning model training and prediction tasks to the system cluster resource management and control service.

The system cluster resource management and control service calculates the resource difference between the multi-tenant resource quota table and the user resource usage table to obtain the remaining resources that the user can apply for, and verifies whether the number of CPUs, GPUs, and memory requested by the machine learning model training and prediction tasks exceeds the number of users. Remaining resources can be applied for.

Select the Node node that does not exceed the remaining resources that the user can apply for. The system cluster resource management service calculates the difference between the real-time available resource information of the Node node and the number of CPUs, GPUs, and memory applied for, and divides it by the inherent resource capacity of the Node node to obtain the allocated resources. The percentage of resources left for CPU, GPU, and memory.

Select Node nodes whose percentages of CPU, GPU, and memory remaining resources after resource allocation are greater than the preset resource threshold percentage, and calculate the percentage of CPU, GPU, and memory remaining resources for each Node node after resource allocation. And sort by rating from largest to smallest.

The system cluster resource management and control service selects the node node ranked first in the sequence as the optimal node, returns the node name of the optimal node to the machine learning task scheduling service, and stores it persistently in the user resource usage table.

The machine learning task scheduling service dynamically generates Kubernetes yaml files, and calls the Kubernetes API to create containers on the optimal nodes to run machine learning model training and prediction tasks.

As a preferred solution, a CPU, GPU, and memory usage collection program is deployed on each Kubernetes Node in the cluster.

As a preferred solution, the inherent resource capacity of Node nodes in the cluster uses the user ID as the namespace in Kubernetes to logically divide and isolate the virtual resource pool.

As a preferred solution, the multi-tenant resource quota table is as follows:

As a preferred solution, the user resource usage table is as follows:

As a preferred option, grant access to different user-operable namespaces through Kubernetes' role-based access control.

As a preferred solution, a Kubernetes cluster includes the following components: API Server, ControllerManager, Scheduler, Kubelet, Kube-proxy, Etcd, and Container runtime.

As a preferred solution, the method for calculating the percentage of the remaining resources of the CPU, GPU and memory of each Node node after allocating resources is as follows:

Score _i =request_cpu×percent_cpu _i +request_gpu×percent_gpu _i +request_mem×percent_mem _i Among them, Score _i is the score of the i-th Node node, percent_cpu _i , percent_gpu _i , percent_mem _i are the resources of the i-th Node node after allocating resources respectively The percentage of the remaining resources of CPU, GPU and memory, request_cpu, request_gpu, request_mem are the i-th Node section respectively

The amount of CPU, GPU, and memory requested by the point.

Beneficial effect: The invention provides a Kubernetes-based multi-tenant machine learning task resource scheduling method, which performs quota management on computing resources available to different users, and monitors resource status information of each Node node in the Kubernetes platform at the same time, considering the destination of the node. The problem of resource utilization of the host can avoid the problem of inaccurate scheduling results. At the same time, by monitoring the real-time scheduling and pre-scheduling request demand information, each Node node is prioritized according to the scheduling task demand information, and the host label of the optimal node is obtained. , according to the label, the resource requirements of various machine learning model training and prediction tasks are reasonably allocated, which can effectively prevent and reduce the skew problem of node resource usage in the Kubernetes platform, achieve multi-node load balancing, and improve the utilization of node resources.

Description of drawings

FIG. 1 is a schematic diagram of multi-tenant management of cluster resources in an example of the present invention.

FIG. 2 is a schematic diagram of a Kubernetes cluster resource management architecture in an example of the present invention.

FIG. 3 is a flowchart of machine learning training and prediction task creation in an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments.

A Kubernetes-based multi-tenant machine learning task resource scheduling method, comprising the following steps:

1) By calculating the difference between the resources used by the Node node (node_cpu_used _i , node_gpu_used _i and node_mem_used _i ) and the resources used by the created container (pod_cpu_used _i , pod_gpu_used _i and pod_mem_used _i ), the resources occupied by all processes of the node's operating system are obtained. information.

2) Obtain the resource information (pod_cpu_req _i , pod_gpu_req _i and pod_mem_req _i ) applied by all machine learning model training and prediction task containers on the node by calling the Kubernetes API.

3) Subtract the above two values from the inherent resource capacity of the Node node (node_cpu_total _i , node_gpu_total _i and node_mem_total _i ) to calculate the real-time available resource information of the Node node (node_cpu_i , node_gpu _i and node_mem _{i )} .

4) Calculate the availability of CPU, GPU and memory of each Node node by the following formula:

percent_cpu _i =node_cpu _i /node_cpu_total _i

percent_gpu _i =node_gpu _i /node_gpu_total _i

percent_mem _i =node_mem _i /node_mem_total _i

5) The system cluster resource management and control service passes the preset resource threshold percentage. Nodes below the preset resource threshold percentage will no longer allocate computing resources for machine learning model training and prediction tasks to ensure that each Node node will not be overloaded.

6) The machine learning task scheduling service sends the number of CPU, GPU and memory resources (request_cpu, request_gpu and request_mem) required by different users' machine learning model training and prediction tasks to the system cluster resource management and control service.

7) The system cluster resource management and control service calculates the resource difference between the multi-tenant resource quota table and the user resource usage table to obtain the remaining resources that the user can apply for, and verifies whether the number of request_cpu, request_gpu and request_mem applied for the machine learning model training and prediction tasks is not Exceeding users can apply for the remaining resources.

8) The system cluster resource management and control service calculates the difference between the real-time available resource information (node_cpu _i , node_gpu _i and node_mem _i ) of the Node node and the requested number of request_cpu, request_gpu and request_mem, and divides it by the inherent resource capacity of the Node node to obtain the allocated resources. The percentage of resources remaining.

percent_cpu _i =(node_cpu _i -request_cpu)/node_cpu_total _i

percent_gpu _i =(node_gpu _i -request_gpu)/node_gpu_total _i

percent_mem _i =(node_mem _i -request_mem)/node_mem_total _i

Compare and filter the nodes whose percentage of the remaining resources after allocating the resources is less than the preset resource threshold percentage, and then calculate and sort by the score the percentage of the remaining resources of the remaining nodes.

Score _i =request_cpu×percent_cpu _i +request_gpu×percent_gpu _i +request_mem×percent_mem _i

9) The system cluster resource management and control service selects the Node node with the highest score from the ranking as the optimal node, returns the node name to the machine learning task scheduling service, and stores it persistently in the user resource usage table.

10) The machine learning task scheduling service dynamically generates the Kubernetes yaml file, and calls the Kubernetes API to create a container in the optimal node to run the machine learning model training and prediction tasks.

The purpose of the present invention is to use Kubernetes and container technology to uniformly control IaaS layer CPU, GPU, memory and storage resources, build a standardized operating environment for multi-tenant machine learning model training and prediction applications, and improve the controllability and flexibility of the power grid system Expansion capability and resource isolation capability. The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention:

Label management is performed according to the available CPU, GPU, memory and storage resource information of different Node nodes in the cluster, and the cluster resources are integrated into a resource pool through Kubernetes, and the user ID is used as the namespace in Kubernetes. Logical division and isolation, as shown in Figure 1.

The system administrator allocates the required resource information to different users through the cluster multi-tenant resource management interface tool. The information is persistently stored in the multi-tenant resource quota table, as shown in Table 1. Through Kubernetes' role-based access control (RBAC), access permissions are granted to the namespaces that can be operated by different users, preventing mutual interference of resource usage among users.

The CPU, GPU, and memory usage collection programs are deployed on each Kubernetes Node node in the cluster, and the available resources and availability rates on all Node nodes are calculated based on the obtained information above.

Table 1 Multi-tenant resource quota table

field name use user_id User unique ID consistent with permissions, single sign-on cpu_capacity The total number of logical cores of the CPU memory_capacity Total memory (GB) gpu_capacity Number of GPU cards storage_capacity Storage space (GB)

Table 2 User resource usage table

FIG. 2 is a schematic diagram of the resource management architecture of the Kubernetes cluster in this embodiment. In this embodiment, the Kubernetes cluster is composed of 2 Master nodes and 6 Node nodes. The Master node is the main control unit of the cluster, which mainly performs scheduling management on the cluster to prevent the increase of project demand and the increase of access volume. Therefore, this embodiment builds a high-availability mode of dual-Master nodes; the Node node is a workload node, which is mainly used for running Containers for business applications include two clusters: CPU and GPU. The CPU cluster is mainly used to create regular pod tasks, while the GPU cluster is mainly used to create pod tasks involving image operations. The dual-node cluster mode will allow applications deployed in it to run. more reasonable and efficient.

The Kubernetes cluster mainly includes seven main components: API Server, Controller Manager, Scheduler, Kubelet, Kube-proxy, Etcd, and Container Runtime. The above components cooperate with each other to realize the operation of the entire cluster. The scheduling strategy of the present invention is mainly for The Scheduler works by calculating the evaluation scores of real-time tasks and timed tasks in each Node node. The evaluation score includes two aspects. On the one hand, it refers to the actual use of the resources of the Node node itself, and on the other hand, it takes into account the pod's preference for CPU, GPU, and memory resource requirements. Finally, the scheduling strategy of the present invention comprehensively evaluates each Node node according to real-time tasks and timing tasks, selects the Node node with the highest evaluation score as the target scheduling node, skips the pre-selection strategy and the optimization strategy of the Scheduler, and can directly use the unique label by setting the unique label. Specify the Node node to create a pod. Figure 2 shows the flow chart of creating a pod task request in this embodiment, and the specific method is as follows:

Step 1: Obtain the CPU, GPU and memory usage information of the host where each Node node in the Kubernetes platform is located, as well as the CPU, GPU, and memory usage information and request allocation information of the pod on the node, and calculate each node according to the above obtained information. The available resources and availability rate of each Node node;

First, obtain the resource usage of the host outside the pod container by calculating the difference between the resources used by the host and the pod; secondly, obtain the actual resource allocation of the pod container, and sum the usage of the host outside the container with it to calculate Get the actual availability of Node nodes; calculate the actual available resources of CPU, GPU and memory of all Node nodes by the following formula:

node_cpu _i =node_cpu_total _i -(host_cpu_used _i -pod_cpu_used _i )-pod_cpu_req _i

node_mem _i =node_mem_total _i -(host_mem_used _i -pod_mem_used _i )-pod_mem_req _i

node_gpu _i =node_gpu_total _i -(host_gpu_used _i -pod_gpu_used _i )-pod_gpu_req _i

Wherein, the node_cpu _i , node_mem _i and node_gpu _i correspond to the actual available resource information of the Node node CPU, GPU and memory respectively, and the node_cpu_total _i , node_mem_total _i and node_gpu_total _i respectively correspond to the total resource configuration of the Node node CPU, GPU and memory Information, the host_cpu_used _i , host_mem_used _i and host_gpu_used _i correspond to the host machine usage information of Node node CPU, GPU and memory respectively, and the host_cpu_used _i , pod_mem_used _i and host_gpu_used _i correspond to the pod usage information of Node node CPU, GPU and memory respectively , the pod_cpu_req _i , pod_mem_req _i and pod_gpu_req _i respectively correspond to the resource request allocation information of the current Node node CPU, GPU and memory pod;

Calculate the availability of CPU, GPU and memory of each Node node by the following formula:

percent_cpu _i =node_cpu _i /node_cpu_total _i

percent_mem _i =node_mem _i /node_mem_total _i

percent_gpu _i =node_gpu _i /node_gpu_total _i

Step 2: Compare the CPU, GPU, and memory availability of each Node node with the preset threshold. If any node is lower than the specified threshold, it means that the node is overloaded, and the node is filtered. If the number of filtered nodes is If it is 0, it will return scheduling failure; if the number of filtered nodes is greater than 0, continue to step 3;

Step 3: Obtain the request information of real-time tasks and timed task pods for CPU, GPU and memory resources through the K8s scheduler, which are request_cpu, request_gpu, request_mem and user ID respectively. According to the user ID lookup table, the remaining information of the current user resources can be obtained. By comparison, it can be judged whether it supports to continue to create pods, if not, it will return scheduling failure, if it is satisfied, continue to the next step;

Step 4: Compare the task resource request information obtained in Step 3 with the available resources of the Node node, and filter the Node nodes with insufficient CPU, GPU and memory resources. If the number of filtered nodes is 0, the scheduling failure is returned. If the number of outgoing nodes is equal to 1, the Node node is set as the host of the pod to be created. If the number of filtered out nodes is greater than 1, proceed to the next step;

Step 5: Score the filtered Node nodes, and calculate the percentage of the remaining resources of the request task after each Node node allocates resources by the following formula.

percent_cpu _i =(node_cpu _i -request_cpu)/node_cpu_total _i

percent_gpu _i =(node_gpu _i -request_gpu)/node_gpu_total _i

percent_mem _i =(node_mem _i -request_mem)/node_mem_total _i

The nodes whose percentage of the remaining resources after the above allocation of resources is less than the percentage of the remaining resource threshold are excluded, and the percentages of the remaining resources of all the nodes' CPU, GPU, and memory after the resources are allocated are accumulated and sorted.

Prioritize each Node node, and determine the optimal number of nodes according to the ranking. If the number of nodes is 1, the Node node is the optimal node and its label is obtained; if the number of nodes is greater than 1, the most optimal node is selected according to the ordering Optimize the Node node and get its label; finally, start the pod by specifying the label in the yaml file of the machine learning task.

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (8)

1. A Kubernetes-based multi-tenant machine learning task resource scheduling method is characterized by comprising the following steps: the method comprises the following steps:

calculating the difference value of the used resources of the Node nodes in the cluster and the used resources of the created containers to obtain the resource information occupied by all processes of the Node operating system;

calling a Kubernetes API to acquire resource information applied by all machine learning models and prediction task containers on Node nodes;

subtracting resource information occupied by all processes of a Node operating system and resource information applied by all machine learning model training and prediction task containers on the Node from the inherent resource capacity of the Node, and calculating real-time available resource information of the Node;

calculating the availability ratios of a Node CPU, a GPU and a memory according to the real-time available resource information of the Node and the inherent resource capacity of the Node;

the Node nodes with the availability ratios of Node CPUs, GPUs and memories not lower than the preset resource threshold percentage allocate computing resources for the machine learning model training and predicting tasks;

the machine learning task scheduling service sends the quantities of CPU, GPU and memory resources applied by machine learning model training and prediction tasks of different users to the system cluster resource management and control service;

the system cluster resource management and control service calculates resource difference values of a multi-tenant resource quota table and a user resource use condition table to obtain user-applicable residual resources, and checks whether the quantity of CPUs (central processing units), GPUs (graphic processing units) and memories applied by the machine learning model training and prediction tasks exceeds the quantity of the user-applicable residual resources or not;

selecting Node nodes which do not exceed the amount of the residual resources which can be applied by the user, calculating the difference value of the real-time available resource information of the Node nodes and the amount of the applied CPU, GPU and memory by the system cluster resource management and control service, and dividing the difference value by the inherent resource capacity of the Node nodes to obtain the percentage of the residual resources of the CPU, GPU and memory after the resources are distributed;

selecting Node nodes with the percentage of the residual resources of the CPU, the GPU and the memory after the resources are distributed being larger than the preset resource threshold percentage, carrying out score calculation on the percentage of the residual resources of the CPU, the GPU and the memory after the resources are distributed of each Node, and sequencing according to the score from large to small;

the Node ordered in the sequence of the system cluster resource management and control service is the optimal Node, the Node name of the optimal Node is returned to the machine learning task scheduling service, and persistent storage is carried out in the user resource use condition table;

and dynamically generating a Kubernets yaml file by the machine learning task scheduling service, and calling a Kubernets API to create a container in the optimal node to run a machine learning model training and predicting task.

2. The Kubernetes-based multi-tenant machine learning task resource scheduling method according to claim 1, characterized in that: and each Kubernetes Node in the cluster is provided with a CPU, a GPU and a memory use condition acquisition program.

3. The Kubernetes-based multi-tenant machine learning task resource scheduling method according to claim 1, characterized in that: the inherent resource capacity of the Node nodes in the cluster takes the user ID as a name space in Kubernetes to logically divide and isolate the virtual resource pool.

4. The Kubernetes-based multi-tenant machine learning task resource scheduling method according to claim 1, characterized in that: the multi-tenant resource quota table is as follows:

name of field Use of user_id User unique ID consistent with authority and single sign-on cpu_capacity Total core number of CPU logic memory_capacity Total memory (GB) gpu_capacity GPU card number storage_capacity Storage space (GB)

。

5. The Kubernetes-based multi-tenant machine learning task resource scheduling method according to claim 1, characterized in that: the user resource usage table is as follows:

6. the Kubernetes-based multi-tenant machine learning task resource scheduling method according to claim 1, characterized in that: the role-based access control by Kubernetes gives access rights to namespaces that are operable by different users.

7. The Kubernetes-based multi-tenant machine learning task resource scheduling method according to claim 1, characterized in that: the Kubernetes cluster includes the following components: API Server, Controller Manager, Scheduler, Kubelet, Kube-proxy, Etcd, Container runtime.

8. The Kubernetes-based multi-tenant machine learning task resource scheduling method according to claim 1, characterized in that: CPU, GPU and memory of each Node after distributing resource

The method for calculating the scores by percentage is as follows:

Score_i＝request_cpu×percent_cpu_i+request_gpu×percent_gpu_i+request_mem×percent_mem_i

wherein, Score_iRating _ cpu, rating of i Node_i、percent_gpu_i、percent_mem_iAnd respectively allocating the percentages of the residual resources of the CPU, the GPU and the memory for the ith Node after the resources are allocated, wherein the request _ CPU, the request _ GPU and the request _ mem are respectively the CPU, the GPU and the memory quantity applied by the ith Node.

Images