CN117632390A - Job scheduling method, device, scheduler and system - Google Patents

Abstract

Disclosed are job scheduling methods, devices, schedulers and systems, relating to the field of data processing. The method includes the scheduler obtaining the pending job, determining the associated tenant and scheduling result of the pending job according to the two-level scheduling policy, and executing the operation of the pending job according to the scheduling result, that is, the first-level scheduling implements tenant-level tenant scheduling, and the second-level scheduling implements tenant-level tenant scheduling. Job-level job scheduling. Compared with users of all tenants sharing and using the shared resource pool provided by the supercomputing center, the jobs of users in different tenants may be processed based on the same resources, which poses security risks. The solution provided by this application uses the scheduler to process the jobs of different tenants according to the tenant. Two-level isolation scheduling and the isolation of tenant resources provided by the system based on tenants realize the isolation between tenants. The scheduler isolates the jobs of different tenants to improve data processing security, and is based on each tenant's own tenant resources. The pool performs job scheduling to improve data processing efficiency.

Description

Job scheduling method, device, scheduler and system

Technical field

The present application relates to the field of data processing, and in particular, to a job scheduling method, device, scheduler and system.

Background technique

With the rapid development of High Performance Computing (HPC) applications, the demand for high-performance computing in scientific research, manufacturing, life sciences and other fields is growing rapidly. More and more companies are using supercomputing computing resources for scientific computing. As a deployment form of high-performance computing applications, supercomputing centers usually process high-performance computing jobs in the form of shared resource pools. However, as the number of users continues to increase, jobs submitted by users are often scheduled based on shared resource pools. The security of user data cannot be guaranteed. Therefore, a more secure job scheduling method is urgently needed.

Contents of the invention

This application provides a job scheduling method, device, scheduler and system, thereby improving data security for processing user jobs in the HPC field.

The first aspect provides a job scheduling method, which is executed by the scheduler. The method includes: the scheduler obtains the job to be processed, determines the associated tenant and scheduling result of the job to be processed according to the two-level scheduling policy, and executes the operation of the job to be processed according to the scheduling result. The two-level scheduling policy is used to indicate the job scheduling method that implements resource management in a multi-tenant isolation manner. That is, the tenant resources provided by the system are isolated according to the tenant, and the tenant's jobs are scheduled for two-level isolation. Among them, the first-level scheduling determines whether there are The tenant associated with the job to be processed implements tenant-level tenant scheduling. The secondary scheduling schedules the job to be processed based on the tenant resource pool of the tenant associated with the job to be processed to implement job-level job scheduling.

In this way, compared with users of all tenants sharing the shared resource pool provided by the supercomputing center, there are security risks in that the jobs of users in different tenants may be processed based on the same resources. The solution provided by this application uses the scheduler to handle the tasks of users in different tenants. Jobs are scheduled in two-level isolation based on tenants, and the tenant resources provided by the system are isolated based on tenants, thereby achieving isolation between tenants and allowing the scheduler to isolate jobs of different tenants, thus improving data processing security and based on Each tenant has its own tenant resource pool for job scheduling, which effectively improves the efficiency of data processing.

Among them, tenant resource pools of multiple tenants are isolated from each other, and the tenant resource pool includes at least one of computing resources, storage resources, and network resources. Therefore, the tenant resource pool provides abundant resources to process the tasks of users within the tenant.

In one possible implementation, determining the associated tenant of the pending job according to the two-level scheduling policy includes: determining the associated tenant with the user who submitted the pending job based on the association between the tenant and the user. Therefore, one-level scheduling is implemented based on the association between tenants and users, and it is determined that there are tenants associated with the jobs to be processed in the system, so that the scheduler can isolate and process the jobs of different tenants, thus improving the security of data processing.

In one example, determining the tenant associated with the user who submitted the job to be processed based on the association between the tenant and the user includes: determining the tenant associated with the user who submitted the job to be processed based on the user ID and the association. This is so that the scheduler can determine the tenant associated with the user who submitted the pending job as quickly as possible based on the user ID to implement first-level scheduling.

In some embodiments, determining the tenant associated with the user who submitted the job to be processed according to the association relationship between the tenant and the user includes: the scheduler determines the scheduling resources of the associated tenant of the job to be processed in the scheduler according to the association relationship, and the scheduling resources are used to Pending jobs are scheduled based on the tenant resource pool of the pending job's associated tenant. The scheduling resources used to schedule jobs of different tenants in the scheduler are isolated from each other.

In another possible implementation, determining the associated tenant and scheduling result of the to-be-processed job based on the two-level scheduling policy includes: scheduling the job of the user in the associated tenant based on the tenant resource pool of the associated tenant to obtain the scheduling result. Among them, the jobs associated with users in the tenant include pending jobs. The scheduling results are used to indicate the resources in the tenant resource pool of the associated tenant to process pending jobs. Therefore, since the tenant resource pools of different tenants are isolated from each other, secondary scheduling is implemented based on the tenant resource pool of the associated tenant, that is, the jobs of the users in the associated tenant are scheduled based on the tenant resource pool of the associated tenant, and the scheduler is implemented to isolate the jobs of different tenants. processing, thereby improving data processing security.

In one example, the scheduler can configure different scheduling policies for different tenants based on the tenant's business characteristics. Based on the tenant resource pool of the associated tenant, scheduling the jobs of the users in the associated tenant and obtaining the scheduling results includes: based on the tenant resource pool of the associated tenant, scheduling the jobs of the users in the associated tenant according to the scheduling policy of the associated tenant and obtaining the scheduling results. Therefore, job scheduling based on each tenant's own tenant resource pool combined with scheduling strategies further improves the efficiency of data processing.

In another example, the scheduler can configure multiple queues for a tenant, and each queue is bound to a different user of the same tenant. Based on the tenant resource pool of the associated tenant, schedule the jobs of the users in the associated tenant, and obtain the scheduling results, including: based on the tenant resource pool of the associated tenant, schedule the jobs of the users in the associated tenant in the first queue according to the scheduling policy of the associated tenant, and obtain the scheduling As a result, the first queue is the queue of the plurality of queues that contains pending jobs. Therefore, scheduling the jobs of users in the tenant based on different queues further improves the efficiency of data processing.

In another possible implementation manner, performing operations on the job to be processed according to the scheduling result includes: performing job processing operations and job management operations on the job to be processed according to the scheduling result.

In one example, performing a job processing operation on the job to be processed according to the scheduling result includes: when the tenant resource pool of the associated tenant meets the resource requirements of the job to be processed, scheduling the job of the user in the associated tenant to obtain the scheduling result.

In another example, performing a job management operation on the job to be processed according to the scheduling result includes: when the job management operation matches the job status, scheduling the job of the user in the associated tenant to obtain the scheduling result.

A second aspect provides a job scheduling device, which includes various modules for executing the job scheduling method in the first aspect or any possible design of the first aspect.

In a third aspect, a scheduler is provided. The scheduler includes at least one processor and a memory, and the memory is used to store a set of computer instructions; when the processor serves as the scheduler in the first aspect or any possible implementation of the first aspect, When the processor executes the set of computer instructions, the operation steps of the job scheduling method in the first aspect or any possible implementation of the first aspect are executed.

A fourth aspect provides a computer device. The computer device includes a scheduler, and the scheduler is configured to perform the operation steps of the job scheduling method in the first aspect or any possible implementation of the first aspect.

In the fifth aspect, a job scheduling system is provided. The job scheduling system includes a scheduler and a resource pool. The resource pool includes storage nodes, computing nodes and networks. The resource pool is used to provide tenant resource pools for multiple tenants. The scheduler is used to execute the first Operation steps of the job scheduling method in one aspect or any possible implementation of the first aspect.

In a sixth aspect, a computer-readable storage medium includes: computer software instructions; when the computer software instructions are run in a scheduler, they cause the scheduler to execute the method described in the first aspect or any possible implementation of the first aspect. operating steps.

In a seventh aspect, a computer program product is provided. When the computer program product is run on a computer, it causes the computer to perform the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

Description of drawings

Figure 1 is a schematic architectural diagram of a job scheduling system provided by an embodiment of the present application;

Figure 2 is a schematic flow chart of scheduler initialization provided by an embodiment of the present application;

Figure 3 is a schematic flowchart of a job scheduling method provided by an embodiment of the present application;

Figure 4 is a schematic flowchart of scheduling jobs in a queue provided by an embodiment of the present application;

Figure 5 is a schematic flowchart of a job management method provided by an embodiment of the present application;

Figure 6 is a schematic structural diagram of a job scheduling device provided by an embodiment of the present application;

Figure 7 is a schematic structural diagram of a scheduler provided by an embodiment of the present application.

Detailed ways

High Performance Computing (HPC) cluster refers to a computer cluster system. HPC clusters contain multiple computers connected together using various interconnect technologies. The interconnection technology may be, for example, infinite bandwidth technology (infiniband, IB), Remote Direct Memory Access over Converged Ethernet (RoCE), or transmission control protocol (Transmission Control Protocol, TCP). HPC provides ultra-high floating-point computing capabilities and can be used to solve the computing needs of computing-intensive and massive data processing services. The combined computing power of multiple computers connected together can handle large computing problems. For example, industries such as scientific research, weather forecasting, finance, simulation experiments, biopharmaceuticals, gene sequencing, and image processing involve the use of HPC clusters to solve large-scale computing problems and computing needs. Using HPC clusters to handle large-scale computing problems can effectively shorten the computing time for processing data and improve computing accuracy. Usually, the management node in the HPC cluster can decompose the job, assign the decomposed job to multiple computing nodes, and have multiple computing nodes complete the job in parallel.

Resources refer to at least one of computing resources, storage resources and network resources required to process jobs, such as: central processing unit (CPU), graphics processing unit (GPU), data processing Unit (data processing unit, DPU), neural processing unit (neural processing unit, NPU) and embedded neural network processor (neural-network processing unit, NPU), memory, input/output (Input/Output, I/O).

Tenant refers to an enterprise organization that has been granted the right to use Software-as-a-Service (SaaS).

User refers to an object with the right to use SaaS. A tenant includes multiple users.

Job refers to the executable command or script submitted by the user to the system.

A queue is a linear list with restricted operations. The data elements of the queue are also called queue elements. Inserting a queue element into the queue is called enqueuing, and deleting a queue element from the queue is called dequeuing. The end that performs the insertion operation is called the tail of the queue, and the end that performs the deletion operation is called the head of the queue. When there are no elements in the queue, it is called an empty queue. In this embodiment of the application, the queue may be a job container that contains jobs for multiple users of a tenant.

Scheduling policy refers to the job scheduling algorithm. Common scheduling strategies in the HPC field include at least one of first-come, first-served strategy, fair sharing strategy, and preemptive strategy.

In order to solve the data security problem caused by scheduling jobs submitted by users based on shared resource pools in HPC clusters, this application provides a job scheduling method, especially a multi-tenant two-level scheduling method in a supercomputing scenario, that is, the system If the provided resources are isolated by tenant, the system includes tenant resource pools of multiple tenants that are isolated from each other. After the scheduler in the system obtains the pending jobs, it determines the associated tenants and scheduling results of the pending jobs based on the two-level scheduling policy. To execute the operation of the pending job according to the scheduling result, that is, first determine that there is a tenant associated with the pending job in the system, and schedule the pending job based on the tenant resource pool of the tenant associated with the pending job. Compared with users of all tenants sharing and using the shared resource pool provided by the supercomputing center, there is a security risk that the jobs of users in different tenants may run on the same resource. The solution provided by this application implements tenant-level tenant scheduling through one-level scheduling. Second-level scheduling implements job-level job scheduling. By isolating the scheduling resources for scheduling jobs of multiple tenants in the scheduler, and isolating the tenant resources provided by the system according to tenants, the isolation between tenants is achieved, allowing the scheduler to The jobs of different tenants are processed in two-level isolation, thereby improving the security of data processing, and job scheduling based on each tenant's own resources effectively improves the efficiency of data processing.

Figure 1 is a schematic architectural diagram of a job scheduling system provided by an embodiment of the present application. As shown in Figure 1, the job scheduling system 100 is an entity that provides high-performance computing. Job scheduling system 100 includes resource pool 110 . The resource pool 110 includes computing resources 111, storage resources 112 and network resources 113. The resource pool 110 can be divided into multiple tenant resource pools based on multiple tenants (N tenant resource pools as shown in Figure 1, N is greater than or equal to 2). Multiple tenant resource pools are isolated from each other, and the tenant resource pool includes at least one of computing resources, storage resources, and network resources. Tenant resource pools of different tenants can contain the same or different resources. Each tenant resource pool is used to provide resource support for processing jobs of the tenant to which it belongs.

Computing resources 111 may be provided by computing clusters in the system. A computing cluster contains at least two computing nodes, and the computing nodes can communicate with each other. A computing node is a computing device, such as a server, desktop computer, or storage array controller.

Storage resources 112 may be provided by storage clusters in the system. A storage cluster contains at least two storage nodes. A storage node includes one or more controllers, network cards, and multiple hard disks. Hard drives are used to store data. The hard disk can be a magnetic disk or other type of storage medium, such as a solid state drive or a shingled magnetic recording hard drive. Network cards are used to communicate with the computing nodes contained in the computing cluster. The controller is used to write data to the hard disk or read data from the hard disk according to the read/write data request sent by the computing node. In the process of reading and writing data, the controller needs to convert the address carried in the read/write data request into an address that the hard disk can recognize.

Network resources 113 can be provided by network devices such as switches and routers. Network resources 113 are used to provide support for data transmission between computing nodes and storage nodes.

The job scheduling system 100 may include a tenant management platform through which tenants can purchase tenant resource pools. For example, the tenant management platform can receive a tenant's establishment request, and the establishment request includes the resource requirements of the tenant's resource pool. When granting SaaS usage rights to a tenant, a tenant resource pool is constructed for the tenant based on the tenant's resource requirements.

Job scheduling system 100 also includes a scheduler 120 . The scheduler can be a separate physical device. The scheduler 120 is used to perform two-level isolation scheduling on the acquired jobs to be processed. The jobs to be processed may be jobs of different users under N tenants obtained by the scheduler 120 as shown in Figure 1. For example, the jobs of tenant 1 include job 11, job 12 and job 13; the jobs of tenant 2 include job 21 and job 22. and job 23; tenant 23’s jobs include job 31, job 32 and job 23. In some embodiments, the user can operate a user interface (UI) displayed on the client to submit a job. Scheduler 120 obtains jobs over the network. The network may refer to an enterprise's internal network (such as a Local Area Network (LAN)) or the Internet.

The first-level scheduling is used to implement tenant-level tenant scheduling, that is, to determine the tenant space of the tenant associated with the pending job in the scheduler 120 . Tenant space can refer to the scheduling resources within the scheduler used to schedule jobs. Level one scheduling is also used to implement tenant management and job management.

Tenant management is used to manage tenants who have the right to use SaaS in the system, so that the scheduler 120 can perform first-level scheduling of acquired jobs based on the relationship between the tenant and the user.

Job management is used to manage the job status of jobs. Job statuses include paused, resumed, and terminated.

Second-level scheduling is used to implement job-level job scheduling, that is, to schedule users' jobs in the tenant space based on the tenant's tenant resource pool. In some embodiments, the scheduler 120 can configure different scheduling policies for different tenant spaces, and schedule users' jobs in the tenant spaces according to the tenant's scheduling policies. The scheduler 120 may also perform secondary scheduling on the user's jobs in at least one queue. Secondary scheduling is also used to implement resource management, scheduling policy management, queue management and job management.

Resource management is used to monitor the total resources, available resources, and used resources in the tenant resource pool.

Scheduling policy management is used to configure scheduling policies based on tenants' business characteristics. Business characteristics include computing characteristics, storage characteristics and data transmission characteristics of the business. For example, if the service characteristic is a computing-intensive service, the scheduling policy indicates that the service is preferentially scheduled to nodes with abundant computing resources. For another example, if the service characteristics are storage-intensive services, the scheduling policy instructs the service to be prioritized to nodes with abundant storage resources.

Queue management includes queue permission management, creating queues, deleting queues, modifying queues, etc.

Next, the implementation of the job scheduling method provided by the embodiment of the present application will be described in detail with reference to the accompanying drawings.

Figure 2 is a schematic flowchart of scheduler initialization provided by an embodiment of the present application.

Step 210: The scheduler 120 obtains initialization information.

The scheduler 120 may obtain initialization information from a local hard disk or a storage node in the system and store it in local memory. Initialization information includes system configuration files and tenant configuration files. The system configuration file includes the cluster name, number of computing nodes, number of storage nodes, and number of resources. The tenant configuration file includes the association between tenants and users, primary scheduling applications, secondary scheduling applications, and tenant resource pool conditions.

Step 220: The scheduler 120 starts level one scheduling.

The scheduler 120 reads the initialization information from the memory, runs the first-level scheduling application, and starts the first-level scheduling. In some embodiments, the scheduler 120 allocates a process or thread to run the first-level scheduling application, starts the first-level scheduling, and implements tenant-level tenant scheduling.

Step 230: The scheduler 120 starts secondary scheduling.

The scheduler 120 runs the secondary scheduling application to initiate secondary scheduling for each tenant. The scheduler 120 allocates a thread or sub-thread to each tenant to run the secondary scheduling application, starts the tenant's secondary scheduling, and implements tenant job-level job scheduling.

The scheduler 120 may initiate secondary scheduling for each tenant in sequence. For example, step 230 includes the following detailed steps 231 to 232.

Step 231: The scheduler 120 traverses the tenants and determines whether the secondary scheduling of the next tenant has not been started. If there is no secondary scheduling of the next tenant that has not been started, the secondary scheduling initialization is completed. If there is a next tenant whose secondary scheduling has not been started, in step 232, the scheduler 120 runs the tenant's secondary scheduling application and starts the tenant's secondary scheduling. In some embodiments, after the scheduler 120 runs the second-level scheduling application, the scheduler 120 obtains the jobs to be processed. After performing the first-level scheduling on the jobs to be processed, the scheduler 120 performs the second-level scheduling on the jobs of the users in the tenant. Jobs contains pending jobs.

Optionally, after obtaining the job to be processed, the scheduler 120 may also determine the tenant associated with the job to be processed, and initiate secondary scheduling of the tenant associated with the job to be processed. Therefore, after obtaining the to-be-processed job, the secondary scheduling is started and executed, thereby saving the computing resources of the scheduler 120 .

Figure 3 is a schematic flowchart of a job scheduling method provided by an embodiment of the present application. Here, the scheduler 120 is taken as an example for description. As shown in Figure 3, the method includes the following steps.

Step 310: The scheduler 120 obtains the jobs to be processed.

The scheduler 120 receives jobs submitted by any user in the tenant through the application interface. The pending jobs are HPC-related processing requests. The application interface may include an application platform interface (application platform interface, API) or a command-line interface (command-line interface, CLI) or a graphical user interface (Graphical UserInterface, GUI).

After obtaining the job to be processed, the scheduler 120 performs two-level isolation scheduling on the job to be processed, that is, determining the tenant and scheduling result associated with the job to be processed according to the two-level scheduling policy. The two-level scheduling policy is used to indicate the job scheduling method that implements resource management in a multi-tenant isolation manner. The tenant resource pools provided by the system for multiple tenants are isolated from each other. Regarding the two-level scheduling, steps 320 and 330 are described below.

Step 320: The scheduler 120 determines the tenant associated with the job to be processed.

The scheduler 120 may pre-configure the association between tenants and users. The scheduler 120 determines the tenant associated with the user who submitted the job to be processed based on the association relationship between the tenant and the user.

In some embodiments, the association between the tenant and the user indicates the corresponding relationship between the tenant identification and the user identification. The processing request submitted by the user contains the user ID, which is used to uniquely indicate a user. The scheduler 120 queries the association relationship between the tenant and the user according to the user ID, and determines the associated tenant of the job to be processed. Optionally, the tenant identifier included in the processing request is processed. The scheduler 120 queries the association between the tenant and the user based on the tenant identifier, and determines the associated tenant of the job to be processed. The tenant ID is used to uniquely identify a tenant.

In other embodiments, the association between tenants and users indicates the corresponding relationship between tenant identification, user identification and scheduling resources. Scheduling resources are used to schedule jobs of users within a tenant. For example, scheduling resources may refer to threads or sub-threads running secondary scheduling applications in the scheduler 120 . The scheduling resources of multiple tenants are isolated from each other. The scheduler 120 determines the associated tenant of the job to be processed, that is, determines the scheduling resources of the associated tenant in the scheduler 120. The scheduling resources of the associated tenant are used to schedule jobs of users in the associated tenant based on the tenant resource pool of the associated tenant.

In an example, the association between tenants and users can be presented in the form of a table, as shown in Table 1.

Table 1

As can be seen from Table 1, the users associated with tenant 1 include user 11, user 12, and user 13. Tenant 1 is also associated with scheduling resource 1. The users associated with tenant 2 include user 21, user 22, and user 23. Tenant 2 is also associated with scheduling resource 2. The users associated with tenant 3 include user 31, user 32, and user 33. Tenant 3 is also associated with scheduling resource 3.

Assume that the scheduler 120 receives a job submitted by user 11 in tenant 1. The scheduler 120 can obtain user 11 from the processing request. The scheduler 120 queries table 1 according to user 11 and obtains that the job submitted by user 11 is associated with tenant 1. Tenant 1 The scheduling resource of 1 is scheduling resource 1. Scheduling resource 1 is used to schedule the jobs of users in tenant 1 based on the tenant resource pool of tenant 1.

Table 1 only illustrates the storage form of the association relationship in the storage device in the form of a table, and does not limit the storage form of the association relationship in the storage device. Of course, the storage form of the association relationship in the storage device can also be in other forms. stored in the form, this embodiment does not limit this.

Assume that a user of the first tenant submitted a pending job. If the association between the tenant and the user does not include the first tenant, it means that the first tenant does not exist in the system, the first tenant is an illegal tenant, and users of the first tenant are not allowed to access the system. The scheduler 120 can also feedback an access failure response to the user. .

If the association between the tenant and the user includes the first tenant, it means that the first tenant exists in the system and the first tenant is a legal tenant. Users of the first tenant are allowed to access the system, and secondary scheduling is performed on the jobs to be processed, that is, step 330 is performed. The scheduler 120 may also feed back an access success response to the user.

Step 330: The scheduler 120 schedules jobs of users in the associated tenant based on the tenant resource pool of the associated tenant, and obtains the scheduling result.

When the scheduler 120 performs secondary scheduling, it schedules the jobs of multiple users associated with the tenant together, that is, determines the execution order of the jobs of the multiple users, and the jobs of the multiple users associated with the tenant include pending jobs. After determining the job that needs to be processed, the scheduler 120 allocates resources in the tenant resource pool used by the job. For example, the scheduler 120 schedules the job of the user of tenant 1 to at least one computing node in tenant resource pool 1 of tenant 1, and the at least one computing node provides the computing resource processing job. The scheduler 120 may first schedule the jobs to be processed and allocate resources in the tenant resource pool to the jobs to be processed; or the scheduler 120 may first schedule the jobs of other users and allocate resources in the tenant resource pool to the jobs of other users.

In some embodiments, the scheduler 120 can configure a tenant-level scheduling policy, that is, configure a scheduling policy for each tenant. The scheduler 120 runs a thread or sub-thread of the tenant's secondary scheduling. When executing the tenant's secondary scheduling on the tenant's job, the tenant's job can be scheduled according to the tenant's scheduling policy. The scheduling policies configured by different tenants can be the same or different.

For example, assuming that the scheduling policy is a first-come, first-served policy, when the scheduler 120 performs secondary scheduling, it schedules the jobs of the users obtained first.

In other embodiments, the scheduler 120 can also configure tenant-level queues, that is, configure at least one queue for each tenant. If a tenant configures a queue, and all users in the tenant have a binding relationship with the queue, the scheduler 120 performs secondary scheduling on the jobs in the queue. If a tenant is configured with at least two queues, users within the tenant can be assigned to at least two queues, that is, each queue has a binding relationship with some users within the tenant. For example, a tenant configures two queues. The tenant includes 6 users, and each queue has a binding relationship with 3 user tenants. In addition, each queue can also be configured with scheduling policies and resources in the tenant resource pool available to the queue. The users bound to each queue are different. A user's jobs belong to a queue.

When the scheduler 120 performs secondary scheduling, it may poll at least two queues of the tenant and perform secondary scheduling on the jobs of each queue. Alternatively, the scheduler 120 can also schedule jobs according to the priorities of the queues, and jobs in the queues with high priority will first obtain resources in the tenant resource pool.

The process by which the scheduler 120 schedules jobs in each queue may refer to the method steps shown in FIG. 4 .

Step 410: The scheduler 120 determines whether there is a job in the queue.

If there are no jobs waiting to be scheduled in the queue, this round of scheduling ends. If there are jobs waiting for scheduling in the queue, traverse the jobs in the queue and execute step 420.

Step 420: The scheduler 120 determines resources for processing the job.

The scheduler 120 determines whether the tenant resource pool meets the resource requirements. If the tenant resource pool meets the resource requirements, the scheduler 120 determines the resources required to process the job from the tenant resource pool, and executes step 430. If the tenant resource pool does not meet the resource requirements, step 440 is executed to feedback a submission failure response to the user, and step 410 is executed to determine whether there are jobs in the queue.

In some embodiments, processing the request includes resource requirements required to process the job. For example, the resource requirement indicates at least one of computing resources, storage resources, and network resources. Computing resources include the number of processes, threads, and computing nodes, as well as at least one of XPUs such as CPU, GPU, and NPU used for job processing. Storage resources include the storage capacity required to process jobs. Network resources include bandwidth requirements.

Step 430: The scheduler 120 dispatches the job.

After determining the associated tenant and scheduling result of the job to be processed according to the two-level scheduling policy, the scheduler 120 executes the operation of the job to be processed according to the scheduling result. For example, the scheduler 120 sends an indication to a computing node that meets the resource requirements, instructing the computing node to process the pending job. After the scheduler 120 completes dispatching the job, step 410 can be executed again.

After the scheduler 120 completes scheduling the jobs in the queue, it may also feedback a successful submission response to the user.

In this way, after the user submits the job, the scheduler will schedule the jobs submitted by different users to the scheduling resources of the tenant to which the user belongs through first-level scheduling. That is, the jobs of different users will be scheduled for the second-level tenant, and the jobs within the same tenant will be scheduled through the second-level scheduling. The jobs of different users are sorted and processed, and then the resources in the tenant resource pool are allocated to the users' jobs. This allows the scheduler to perform two-level isolation processing on the jobs of users in different tenants, thus improving the security of data processing. Moreover, job scheduling based on each tenant's own resources effectively improves the efficiency of data processing.

The above embodiment is an explanation of the user submission process. In other embodiments, user-submitted processing requests may also indicate job management operations. Figure 5 is a flow chart of a job management method provided by an embodiment of the present application.

Step 510: The scheduler 120 determines whether the job management operation matches the job status.

The scheduler 120 receives the jobs to be processed and performs first-level scheduling. For example, the pending jobs are HPC-related processing requests. Processing requests include a job number, which uniquely identifies a job. The scheduler 120 determines whether there is a job in the job scheduling system based on the job number, and determines whether the job management operation matches the job status. If the job management operation matches the job status, step 520 is executed, and step 530 is performed. The scheduler 120 feeds back a successful operation response to the user. If the job management operation does not match the job status, step 540 is executed, and the scheduler 120 feeds back an operation failure response to the user.

Step 520: The scheduler 120 operates the job indicated by the job number according to the job management operation.

After determining the associated tenant and scheduling result of the job to be processed according to the two-level scheduling policy, the scheduler 120 executes the operation of the job to be processed according to the scheduling result. For example, the scheduler 120 may send an instruction to the computing node processing the job, instructing the computing node to adjust the job status of the job indicated by the job number according to the job management operation. Job status includes running, paused, resumed, and terminated. The computing node can periodically report the running status of the job to the scheduler 120 so that the scheduler 120 can manage the job status of the job.

For example, if the job processed by the computing node is in the running state and the job management operation instructs the job to be terminated, the computing node adjusts the status of the job from the running state to the terminated state according to the instruction.

Optionally, after the computing node adjusts the job status according to the job management operation, it can also feedback an operation failure response or an operation success response to the user.

In this way, when a user manages a job, the scheduler determines that the job management operation matches the job status through primary scheduling, and schedules the job management operations of different users to the scheduling resources of the tenant to which the user belongs, that is, the job management of the jobs of different users The operation is scheduled on a tenant-by-tenant basis, and the job management operations of different users' jobs within the same tenant are sorted and processed through the second-level scheduling, and then the job management operations are dispatched to the computing nodes that process the jobs. This allows the scheduler to perform two-level isolation processing on the job management operations of users' jobs in different tenants, thus improving the security of data processing. Moreover, job scheduling based on each tenant's own resources effectively improves the efficiency of data processing.

It can be understood that, in order to implement the functions in the above embodiments, the scheduler includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that the units and method steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is performed by hardware or computer software driving the hardware depends on the specific application scenarios and design constraints of the technical solution.

The job scheduling method provided according to this embodiment is described in detail above with reference to FIGS. 1 to 5 . Next, the job scheduling device provided according to this embodiment will be described with reference to FIG. 6 .

Figure 6 is a schematic structural diagram of a possible job scheduling device provided by this embodiment. These job scheduling devices can be used to implement the functions of the scheduler in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the job scheduling device may be the scheduler 120 as shown in FIG. 1 , or may be a module (such as a chip) applied to the server.

As shown in FIG. 6 , the job scheduling device 600 includes a communication module 610 , a scheduling module 620 and a storage module 630 . The job scheduling device 600 is used to implement the functions of the scheduler 120 in the method embodiments shown in FIG. 3, FIG. 4, and FIG. 5.

The communication module 610 is used to obtain pending jobs, which are HPC-related processing requests.

The scheduling module 620 is configured to determine the associated tenant and scheduling result of the to-be-processed job according to a two-level scheduling policy, which is used to indicate a job scheduling method that implements resource management in a multi-tenant isolation manner. For example, the scheduling module 620 is used to execute step 320 and step 330 in FIG. 3 .

The storage module 630 is used to store tenant information, scheduling policies, and queues, so that the scheduling module 620 schedules the jobs of users in the associated tenant according to the scheduling policy of the associated tenant based on the tenant resource pool of the associated tenant, and obtains the Scheduling results.

The scheduling module 620 is specifically configured to schedule the jobs of the users in the associated tenant in the first queue according to the scheduling policy of the associated tenant based on the tenant resource pool of the associated tenant, and obtain the scheduling result. The first queue is A queue containing the pending job among multiple queues.

The scheduling module 620 is also configured to perform job processing operations and job management operations on the to-be-processed jobs according to the scheduling results.

The scheduling module 620 is also configured to determine the tenant associated with the user who submitted the job to be processed based on the association between the tenant and the user.

It should be understood that the job scheduling device 600 in the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex program. Logic device (complexprogrammable logical device, CPLD), field-programmable gate array (field-programmable gatearray, FPGA), general array logic (generic array logic, GAL) or any combination thereof. When the job scheduling methods shown in Figures 3, 4 and 5 can also be implemented through software, the job scheduling apparatus 600 and its respective modules can also be software modules.

The job scheduling device 600 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and/or functions of the various units in the job scheduling device 600 are respectively to implement Figures 3, 4 and The corresponding processes of each method in Figure 5 will not be described again for the sake of simplicity.

Figure 7 is a schematic structural diagram of a scheduler 700 provided in this embodiment. As shown in the figure, the scheduler 700 includes a processor 710, a bus 720, a memory 730, a communication interface 740, and a memory unit 750 (which may also be called a main memory unit). The processor 710, the memory 730, the memory unit 750 and the communication interface 740 are connected through a bus 720.

It should be understood that in this embodiment, the processor 710 may be a CPU. The processor 710 may also be other general-purpose processors, digital signal processing (DSP), ASIC, FPGA or other programmable logic devices. Discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor, etc.

The processor may also be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated devices for controlling the execution of the program of the present application. circuit.

The communication interface 740 is used to implement communication between the scheduler 700 and external devices or devices. In this embodiment, when the scheduler 700 is used to implement the functions of the scheduler 120 shown in Figure 3, Figure 4 and Figure 5, the communication interface 740 is used to obtain the jobs to be processed, and the jobs to be processed are high-performance computing HPC For related processing requests, the processor 710 determines the associated tenant and scheduling result of the job to be processed according to the two-level scheduling policy.

Bus 720 may include a path for communicating information between the components described above, such as processor 710, memory unit 750, and storage 730. In addition to the data bus, the bus 720 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, the various buses are labeled bus 720 in the figure. The bus 720 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), or a computer quick link ( compute express link (CXL), cache coherent interconnect for accelerators (CCIX), etc. The bus 720 can be divided into an address bus, a data bus, a control bus, etc.

As one example, scheduler 700 may include multiple processors. The processor may be a multi-CPU processor. A processor here may refer to one or more devices, circuits, and/or computing units for processing data (eg, computer program instructions).

It is worth noting that FIG. 7 only takes the scheduler 700 including a processor 710 and a memory 730 as an example. Here, the processor 710 and the memory 730 are respectively used to indicate a type of device or device. In specific embodiments, , the quantity of each type of device or equipment can be determined based on business needs.

Memory unit 750 may be a pool of volatile or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), Double data rate synchronous dynamic random access memory (double data date SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlinkDRAM, SLDRAM) and direct memory Bus random access memory (direct rambus RAM, DR RAM).

The memory 730 may correspond to the storage medium used to store information such as scheduling policies and queues in the above method embodiments, for example, a magnetic disk, such as a mechanical hard disk or a solid-state hard disk.

The above-mentioned scheduler 700 may be a general-purpose device or a special-purpose device. For example, the scheduler 700 may be an edge device (eg, a box carrying a chip with processing capabilities) or the like. Optionally, the scheduler 700 may also be a server or other device with computing capabilities.

It should be understood that the scheduler 700 according to this embodiment may correspond to the job scheduling device 600 in this embodiment, and may correspond to the corresponding subject executing any method according to FIG. 3, FIG. 4 or FIG. 5, and the job scheduling The above and other operations and/or functions of each module in the device 600 are respectively intended to implement the corresponding processes of each method in Figure 3, Figure 4 or Figure 5. For the sake of simplicity, they will not be described again here.

Embodiments of the present application provide a computer device. The computer device includes a scheduler. The scheduler is configured to execute the operating steps of the job scheduling method described in the above method embodiment.

Embodiments of the present application provide a job scheduling system. The job scheduling system includes a scheduler and a resource pool. The resource pool includes storage nodes, computing nodes, and networks. The resource pool is used to provide tenant resource pools for multiple tenants. The scheduler is used to execute The operation steps of the job scheduling method described in the above method embodiment.

The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. Software instructions can be composed of corresponding software modules. Software modules can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or other well-known in the art any other form of storage media. An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage media may be located in an ASIC. Additionally, the ASIC can be located in a computing device. Of course, the processor and storage medium may also exist as discrete components in a computing device.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are executed in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program or instructions may be transmitted from a website, computer, A server or data center transmits via wired or wireless means to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media, such as floppy disks, hard disks, and magnetic tapes; they may also be optical media, such as digital video discs (DVDs); they may also be semiconductor media, such as solid state drives (solid state drives). ,SSD). The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of various equivalent methods within the technical scope disclosed in the present application. Modification or replacement, these modifications or replacements shall be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (11)

1. A job scheduling method, characterized in that the method is executed by a scheduler, and the method includes: Obtain pending jobs, which are high-performance computing HPC-related processing requests; Determine the associated tenant and scheduling result of the to-be-processed job according to a two-level scheduling policy, which is used to indicate a job scheduling method that implements resource management in a multi-tenant isolation manner; Execute the operation of the pending job according to the scheduling result. 2. The method according to claim 1, characterized in that determining the associated tenant of the to-be-processed job according to a two-level scheduling policy includes: Determine the tenant associated with the user who submitted the job to be processed based on the association between the tenant and the user. 3. The method according to claim 2, characterized in that determining the tenant associated with the user who submitted the job to be processed according to the association between the tenant and the user includes: Determine a tenant associated with the user who submitted the job to be processed based on the user ID and the association relationship. 4. The method according to claim 2 or 3, characterized in that determining the associated tenant and scheduling result of the to-be-processed job according to a two-level scheduling policy includes: Based on the tenant resource pool of the associated tenant, the jobs of the users in the associated tenant are scheduled to obtain the scheduling result. The jobs of the users in the associated tenant include the to-be-processed jobs. The scheduling result is used to indicate the The resource in the tenant resource pool of the associated tenant that handles the pending job. 5. The method according to claim 4, characterized in that, based on the tenant resource pool of the associated tenant, scheduling the jobs of the users in the associated tenant to obtain the scheduling result, including: Based on the tenant resource pool of the associated tenant, jobs of users in the associated tenant are scheduled according to the scheduling policy of the associated tenant, and the scheduling result is obtained. 6. The method according to claim 4 or 5, characterized in that, based on the tenant resource pool of the associated tenant, scheduling the jobs of users in the associated tenant to obtain the scheduling result includes: Based on the tenant resource pool of the associated tenant, schedule the jobs of the users in the associated tenant in the first queue according to the scheduling policy of the associated tenant to obtain the scheduling result. The first queue contains all the tasks in the multiple queues. Describes the queue of pending jobs. 7. The method according to any one of claims 1 to 6, characterized in that tenant resource pools of multiple tenants are isolated from each other, and the tenant resource pools include at least one of computing resources, storage resources and network resources. 8. The method according to any one of claims 1 to 7, characterized in that, executing the operation of the pending job according to the scheduling result includes: Perform job processing operations and job management operations on the to-be-processed jobs according to the scheduling results. 9. A job scheduling device, characterized in that the device includes: A communication module, used to obtain pending jobs, which are high-performance computing HPC-related processing requests; A scheduling module, configured to determine the associated tenants and scheduling results of the to-be-processed job according to a two-level scheduling policy, which is used to indicate a job scheduling method that implements resource management in a multi-tenant isolation manner; The scheduling module is also configured to execute the operation of the to-be-processed job according to the scheduling result. 10. A scheduler, characterized in that the scheduler includes a memory and at least one processor, the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the above The operating steps of the method according to any one of claims 1-8. 11. A job scheduling system, characterized in that the job scheduling system includes a scheduler and a resource pool. The resource pool includes storage nodes, computing nodes and networks. The resource pool is used to provide tenant resources for multiple tenants. Pool, the scheduler is used to perform the operation steps of the method described in any one of the above claims 1-8.