CN113225269B - Container-based workflow scheduling method, device and system and storage medium - Google Patents

Abstract

The invention discloses a container-based workflow scheduling method, device, system and storage medium, which are applied to a container-based workflow scheduling system. The system includes a first-level scheduler and a second-level scheduler. The method includes: when receiving a job When a flow executes a request, determine the target workflow corresponding to the workflow execution request, and assign the target workflow to the preset workflow queue; monitor the preset workflow queue through the first-level scheduler to obtain from the preset workflow queue Target workflow, and determine the computing resource node corresponding to the target workflow through the primary scheduler; start the target secondary scheduler corresponding to the computing resource node through the primary scheduler, and schedule the target workflow through the target secondary scheduler . The invention adopts a multi-level scheduling framework to realize distributed scheduling on different computing resource nodes, so as to achieve the purpose of scalable scheduling and improve the scheduling performance of the system.

Description

Container-based workflow scheduling method, device, system and storage medium

technical field

The invention relates to the technical field of network communication, in particular to a container-based workflow scheduling method, device, system and storage medium.

Background technique

Workflow refers to the automatic process of part or whole of the business process in the computer environment. It is a calculation model of the workflow and an abstract and general description of the business rules between the workflow and its various operation steps. With the development of information technology, container technology has become one of the core technologies of the new generation of cloud computing, and the concept of workflow management also provides a new business perspective for the development of information systems. Therefore, how to implement workflow based on containers Scheduling is of great significance to the deployment and operation and maintenance of workflow systems.

The traditional workflow scheduling method focuses on scheduling on a single computing resource, and the scheduling performance is low. With the growth of data scale and the complexity of workflow, the method of scheduling on a single computing resource can no longer meet the current workflow management needs. .

Contents of the invention

The main purpose of the present invention is to propose a container-based workflow scheduling method, device, system and storage medium, aiming at improving the scalability of the workflow scheduling system.

To achieve the above object, the present invention provides a container-based workflow scheduling method, which is applied to a container-based workflow scheduling system. The container-based workflow scheduling system includes a first-level scheduler and a second-level scheduler. The method includes the following steps:

When a workflow execution request is received, determine a target workflow corresponding to the workflow execution request, and assign the target workflow to a preset workflow queue;

Monitor the preset workflow queue through the first-level scheduler to obtain the target workflow from the preset workflow queue, and determine the calculation corresponding to the target workflow through the first-level scheduler resource node;

The target secondary scheduler corresponding to the computing resource node is started by the primary scheduler, and the target workflow is scheduled by the target secondary scheduler.

Preferably, the step of determining the computing resource node corresponding to the target workflow through the primary scheduler includes:

determining an initial computing resource node corresponding to the target workflow according to the workflow execution request;

Obtain the load condition corresponding to the initial computing resource node, and determine the target computing resource node corresponding to the target workflow according to the load condition through the primary scheduler.

Preferably, the step of determining the target computing resource node corresponding to the target workflow according to the load condition by the primary scheduler includes:

When the load condition is within a preset load range, the primary scheduler determines that the initial computing resource node is a target computing resource node corresponding to the target workflow;

When the load condition is not within the preset load range, the primary scheduler detects the node load conditions of other computing resource nodes, and obtains from other computing resource nodes whose node load conditions are within the preset load range , determine the target computing resource node corresponding to the target workflow.

Preferably, the step of scheduling the target workflow through the target secondary scheduler includes:

Analyzing the target workflow through the target secondary scheduler to determine the task node corresponding to the target workflow;

The task nodes are executed concurrently by the target secondary scheduler, so as to schedule the target workflow.

Preferably, after the step of performing concurrent execution processing on the task nodes through the target secondary scheduler to schedule the target workflow, it further includes:

receiving execution processing results returned by each of the task nodes through the target secondary scheduler;

According to the execution processing result, it is determined whether the target workflow is scheduled to be completed.

Preferably, the first-level scheduler monitors the preset workflow queue to obtain the target workflow from the preset workflow queue, and determines the target workflow through the first-level scheduler The steps of computing resource nodes corresponding to the workflow include:

Monitor the preset workflow queue through the first-level scheduler, and actively pull the target workflow from the preset workflow queue through the first-level scheduler, so as to obtain the target workflow workflow information;

The computing resource node corresponding to the target workflow is determined by the primary scheduler according to the workflow information.

Preferably, before the step of determining the target workflow corresponding to the workflow execution request, it further includes:

Determining the workflow to be executed, and describing the workflow to be executed according to a preset description method, to obtain a corresponding description model;

A workflow execution request is received based on the description model.

In addition, in order to achieve the above object, the present invention also provides a container-based workflow scheduling device, which is applied to a container-based workflow scheduling system, and the container-based workflow scheduling system includes a first-level scheduler and a second-level scheduler , the container-based workflow scheduling device includes:

A receiving and determining module, configured to determine a target workflow corresponding to the workflow execution request when a workflow execution request is received, and assign the target workflow to a preset workflow queue;

A monitoring and determining module, configured to monitor the preset workflow queue through the primary scheduler, so as to obtain the target workflow from the preset workflow queue, and determine the preset workflow queue through the primary scheduler. The computing resource node corresponding to the target workflow;

The starting scheduling module is configured to start the target secondary scheduler corresponding to the computing resource node through the primary scheduler, and schedule the target workflow through the target secondary scheduler.

Preferably, the monitoring determination module further includes a node determination unit, and the node determination unit is used for:

determining an initial computing resource node corresponding to the target workflow according to the workflow execution request;

Obtain the load condition corresponding to the initial computing resource node, and determine the target computing resource node corresponding to the target workflow according to the load condition through the primary scheduler.

Preferably, the node determining unit is also used for:

When the load condition is within a preset load range, the primary scheduler determines that the initial computing resource node is a target computing resource node corresponding to the target workflow;

When the load condition is not within the preset load range, the primary scheduler detects the node load conditions of other computing resource nodes, and obtains from other computing resource nodes whose node load conditions are within the preset load range , determine the target computing resource node corresponding to the target workflow.

Preferably, the startup scheduling module is also used for:

Analyzing the target workflow through the target secondary scheduler to determine the task node corresponding to the target workflow;

The task nodes are executed concurrently by the target secondary scheduler, so as to schedule the target workflow.

Preferably, the container-based workflow scheduling device further includes an execution feedback unit, and the execution feedback unit is used for:

receiving execution processing results returned by each of the task nodes through the target secondary scheduler;

According to the execution processing result, it is determined whether the target workflow is scheduled to be completed.

Preferably, the monitoring determination module is also used for:

Monitor the preset workflow queue through the first-level scheduler, and actively pull the target workflow from the preset workflow queue through the first-level scheduler, so as to obtain the target workflow workflow information;

The computing resource node corresponding to the target workflow is determined by the primary scheduler according to the workflow information.

Preferably, the container-based workflow scheduling device further includes a description trigger unit, and the description trigger unit is used for:

Determining the workflow to be executed, and describing the workflow to be executed according to a preset description method, to obtain a corresponding description model;

A workflow execution request is received based on the description model.

In addition, in order to achieve the above object, the present invention also provides a container-based workflow scheduling system, the container-based workflow scheduling system includes: a memory, a processor, and a The container-based workflow scheduler running on the above-mentioned container-based workflow scheduler implements the steps of the container-based workflow scheduler method when executed by the processor.

In addition, in order to achieve the above object, the present invention also provides a storage medium, on which a container-based workflow scheduler is stored, and when the container-based workflow scheduler is executed by a processor, the above-mentioned Steps of a container-based workflow scheduling method.

The container-based workflow scheduling method proposed by the present invention is applied to a container-based workflow scheduling system. The system includes a first-level scheduler and a second-level scheduler. The method includes: when a workflow execution request is received, determine the workflow execution Request the corresponding target workflow, and assign the target workflow to the preset workflow queue; monitor the preset workflow queue through the first-level scheduler to obtain the target workflow from the preset workflow queue, and pass the first-level scheduling The controller determines the computing resource node corresponding to the target workflow; starts the target secondary scheduler corresponding to the computing resource node through the primary scheduler, and schedules the target workflow through the target secondary scheduler. The invention adopts a multi-level scheduling framework to realize distributed scheduling on different computing resource nodes, so as to achieve the purpose of scalable scheduling and improve the scheduling performance of the system.

Description of drawings

Fig. 1 is a schematic diagram of the system structure of the hardware operating environment involved in the solution of the embodiment of the present invention;

FIG. 2 is a schematic flowchart of the first embodiment of the container-based workflow scheduling method of the present invention;

3 is a multi-level scheduling structure diagram of a preferred embodiment of the container-based workflow scheduling method of the present invention;

Fig. 4 is a schematic diagram of functional modules of a preferred embodiment of the container-based workflow scheduling method of the present invention.

The realization of the purpose of the present invention, functional characteristics and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

As shown in FIG. 1 , FIG. 1 is a schematic diagram of the system structure of the hardware operating environment involved in the solution of the embodiment of the present invention.

The system in the embodiment of the present invention includes a first-level scheduler and a second-level scheduler.

As shown in FIG. 1 , the system may include: a processor 1001 , such as a CPU, a network interface 1004 , a user interface 1003 , a memory 1005 , and a communication bus 1002 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the system structure shown in FIG. 1 does not constitute a limitation to the system, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

As shown in FIG. 1 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a container-based workflow scheduler.

Among them, the operating system is a program that manages and controls the container-based workflow scheduling system and software resources, and supports the operation of the network communication module, user interface module, container-based workflow scheduling program, and other programs or software; the network communication module is used for Manage and control the network interface 1002; the user interface module is used to manage and control the user interface 1003.

In the container-based workflow scheduling system shown in Figure 1, the container-based workflow scheduling system calls the container-based workflow scheduler stored in the memory 1005 through the processor 1001, and executes the following container-based work Operations in various embodiments of the flow scheduling method.

Based on the above hardware structure, an embodiment of the container-based workflow scheduling method of the present invention is proposed.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of the first embodiment of the container-based workflow scheduling method of the present invention, the method comprising:

Step S10, when a workflow execution request is received, determine a target workflow corresponding to the workflow execution request, and assign the target workflow to a preset workflow queue;

The container-based workflow scheduling method of this embodiment is applied to container-based workflow scheduling systems of various enterprises and groups. For the convenience of description, the container-based workflow scheduling system is referred to as scheduling system. In this embodiment, the scheduling system includes a first-level scheduler and a second-level scheduler. The first-level scheduler is responsible for distributing workflows. Distribute to determine the corresponding operating environment of the workflow; the secondary scheduler is responsible for parsing the workflow, and then executes the logical scheduling of the workflow to schedule the node tasks of the workflow.

With the development of information technology, container technology has become one of the core technologies of the new generation of cloud computing, and the concept of workflow management also provides a new business perspective for the development of information systems. Therefore, how to schedule workflow based on containers , which is of great significance to the deployment and operation and maintenance of the workflow system. The traditional workflow scheduling method focuses on scheduling on a single computing resource, and the scheduling performance is low. With the growth of data scale and the complexity of workflow, the method of scheduling on a single computing resource can no longer meet the current workflow management. need.

In this embodiment, since the workflow execution request carries the target workflow to be executed and the corresponding workflow information, when the scheduling system receives the workflow execution request, it can determine the workflow execution request by reading the workflow execution request. , and then assign the target workflow to the preset workflow queue. Before the target workflow is assigned to the preset workflow queue, the preset middleware technology can be used to establish the preset workflow queue, wherein the preset middleware technology is preferably efficient middleware such as Redis (Remote Dictionary Service, remote dictionary service) technology, so that the workflow queue can follow the first-in-first-out rule to execute the target workflow. In addition, the preset workflow queues established using Redis include queue modes such as producer-consumer mode and publisher-subscriber mode. Specifically, the producer-consumer model is used to generate workflows through producers and assign workflows to queues, while scheduling and processing workflows through consumers, such as monitoring workflow queues through multiple consumers at the same time. Whoever receives the message will take the message from the workflow queue. If there is no message in the workflow queue, the consumer will continue to monitor. Therefore, the producer-consumer mode is generally used to handle high concurrency situations; using the message publish and subscribe mechanism, you can Asynchronous and efficient processing of workflows. A variety of queue modes can be used to meet the needs of different scenarios of workflow, and high-efficiency scheduling can be achieved by using efficient middleware technology and producer-consumer queue modes.

In addition, to assign the target workflow to the preset workflow queue, the target queue corresponding to each target workflow can be determined by obtaining the workflow running environment ID corresponding to the target workflow, and then write each target workflow into the corresponding target queue . The workflow running environment ID is a unique identifier of the workflow running environment, wherein the workflow running environment ID can be acquired through active strategy or passive adaptive strategy. The ID of the workflow running environment is obtained through the active strategy, that is, when the user executes the target workflow through the client, the user actively selects a different workflow queue to obtain the ID of the workflow running environment; the ID of the workflow running environment is obtained through the passive adaptive strategy, that is, The scheduling system determines the ID of the workflow running environment by monitoring the congestion of each preset workflow queue and performing adaptive allocation. After determining the target queue corresponding to each target workflow, obtain the running unique identifier ID corresponding to the target workflow, and then generate the key corresponding to each target workflow according to the running unique identifier ID and the corresponding running environment; then the target workflow The corresponding workflow information is used as value, and the value and key of each target workflow are correspondingly written into the preset workflow queue through key-value pairs, which can effectively solve the collision problem during workflow execution.

Step S20, monitor the preset workflow queue through the first-level scheduler to obtain the target workflow from the preset workflow queue, and determine the target workflow through the first-level scheduler The corresponding computing resource node;

In this embodiment, different workflow queues are monitored in real time by the first-level scheduler, and the workflow information of each target workflow can be read from the workflow queue by using the key generated by each target workflow. If the controller acquires the target workflow from the workflow queue, the first-level scheduler distributes the acquired target workflow to determine the computing resource node to be run by the second-level scheduler. Among them, the computing resource node corresponding to the target workflow is the operating environment of the target workflow. After determining the computing resource node of the target workflow, it is possible to determine which computing resource nodes the workflow will run on. virtual machine 1, then virtual machine 1 is the computing resource node corresponding to the target workflow.

Further, step S20 also includes:

Step a1, monitor the preset workflow queue through the first-level scheduler, and actively pull the target workflow from the preset workflow queue through the first-level scheduler to obtain the target workflow information in the workflow;

In step a2, the computing resource node corresponding to the target workflow is determined by the primary scheduler according to the workflow information.

In this embodiment, at least one primary scheduler is included. If there are multiple primary schedulers, the primary schedulers distributed on different computing resource nodes monitor different preset workflow queues in real time. The front-end configures the first-level scheduler to monitor different preset workflow queues, and expands the first-level scheduler by actively or passively adding queues; it can also automatically expand by monitoring the congestion level of the preset workflow queues. Then the first-level scheduler actively pulls the workflow information from the preset workflow queue by competing for the consumption queue, and then the first-level scheduler that pulls the workflow information determines the The corresponding computing resource node. By competing for consumption queues, the self-adaptive function of the first-level scheduler can be realized, which can not only reduce the complexity of algorithm configuration parameters, but also make the scheduling structure easier to expand.

Step S30, start the target secondary scheduler corresponding to the computing resource node through the primary scheduler, and schedule the target workflow through the target secondary scheduler.

In this embodiment, since the application programs corresponding to different tasks are different, and the computing resource nodes corresponding to different application programs are also different, the task nodes of the specified queue will be scheduled to the computing resource nodes of the corresponding queue, and the corresponding Computing resource nodes will only run task nodes of the specified queue. Therefore, after detecting that the first-level scheduler has finished distributing and processing the target workflow, the first-level scheduler starts the second-level scheduler on the computing resource node corresponding to the target workflow, that is, the target second-level scheduler, and then passes the target second-level scheduler. The level scheduler schedules target workflows according to a configurable scheduling algorithm. Utilize the multi-level scheduling architecture and adopt distributed technology to realize the scalable scheduling function.

Further, the step of scheduling the target workflow through the target secondary scheduler includes:

Step b1, analyzing and processing the target workflow through the target secondary scheduler to determine the task node corresponding to the target workflow;

In this embodiment, since a directed acyclic graph needs to be used to describe the workflow model before triggering a workflow execution request, and the workflow information stored in the model is related to the modeling language, if the workflow to be executed using the json structure Then, the elements stored in the preset workflow queue are the workflow information in json format. Therefore, when the workflow scheduling is performed through the target secondary scheduler, the target workflow needs to be analyzed and processed by the target secondary scheduler first, for example, the workflow information of the json structure is parsed into a directed acyclic graph, so as to determine the target work The task node corresponding to the flow. It can be understood that in a directed acyclic graph, the circles are task nodes, which represent the scheduled tasks; the edges connecting two circles are communication edges, which represent the communication between the scheduled tasks; the direction of the edges is expressed as The sequence of task execution, therefore, the task node corresponding to the target workflow can be determined by parsing and processing the target workflow through the target secondary scheduler.

Step b2, performing concurrent execution processing on the task nodes through the target secondary scheduler, so as to schedule the target workflow.

In this embodiment, since the task nodes in the directed acyclic graph contain task information that needs to be executed, the task nodes at each level of the scheduling system are traversed through the preset traversal algorithm, and each task is processed by the target secondary scheduler. When the nodes perform concurrent execution processing, the target workflow can be scheduled through the target secondary scheduler according to the task information in the target workflow, and the concurrent execution of child task nodes of the same layer can be realized by using concurrency technology.

Specifically, the preset traversal algorithm is preferably a width traversal algorithm, also known as breadth-first traversal (BFS). Breadth-first traversal refers to starting from an untraversed node in the graph, first traversing the adjacent nodes of this node, and then traversing the adjacent nodes of each adjacent node in turn. Since the value corresponding to each task node is their respective traversal order, breadth-first traversal is also called layer-order traversal. When traversing each task node through BFS, each task node is visited by the shortest path, which is beneficial to improve the execution efficiency of the workflow. Also, parallel processing is having a computer run several programs at the same time or multiple processes or threads of the same program at the same time. The main purpose of parallel processing is to save time in solving large and complex problems. As shown in Figure 3, Figure 3 is a multi-level scheduling structure diagram of a preferred embodiment of the container-based workflow scheduling method of the present invention. The execution environment corresponding to the workflow, the workflow is assigned to different first-level schedulers for execution, and one first-level scheduler supports concurrently launching multiple second-level schedulers, and the second-level schedulers on different computing resource nodes can also Support multiple APPs concurrently on the same node, so as to complete the scheduling work of the workflow through multiple specific APPs. Using multi-thread or multi-coroutine technology to realize the concurrent execution of each layer of scheduler can improve the scheduling ability of the scheduling system.

It should be noted that the "first-level scheduler" and "secondary scheduler" involved in the present invention only serve to distinguish different schedulers, and according to task requirements, the first-level scheduler and/or the second-level scheduler are both Can be multiple.

The container-based workflow scheduling method of this embodiment is applied to a container-based workflow scheduling system. The system includes a first-level scheduler and a second-level scheduler. The method includes: when a workflow execution request is received, determine the workflow execution Request the corresponding target workflow, and assign the target workflow to the preset workflow queue; monitor the preset workflow queue through the first-level scheduler to obtain the target workflow from the preset workflow queue, and pass the first-level scheduling The controller determines the computing resource node corresponding to the target workflow; starts the target secondary scheduler corresponding to the computing resource node through the primary scheduler, and schedules the target workflow through the target secondary scheduler. The invention adopts a multi-level scheduling framework to realize distributed scheduling on different computing resource nodes, so as to achieve the purpose of scalable scheduling and improve the scheduling performance of the system.

Further, based on the first embodiment of the container-based workflow scheduling method of the present invention, a second embodiment of the container-based workflow scheduling method of the present invention is proposed.

The difference between the second embodiment of the container-based workflow scheduling method and the first embodiment of the container-based workflow scheduling method is that before the step of determining the target workflow corresponding to the workflow execution request, it further includes:

Step c1, determining the workflow to be executed, and describing the workflow to be executed according to a preset description method, to obtain a corresponding description model;

In this embodiment, firstly, the application programs required by the workflow are encapsulated by the container technology, and then the workflow to be executed is determined according to various application programs required for workflow scheduling and the order and logic relationship of application program calls. It is understandable that packaging the applications required by the workflow through container technology can ensure that the applications have the necessary libraries, dependencies, and files. More importantly, users can also freely migrate these applications in production. without any negative impact. It can be seen that container technology provides a guarantee for the cross-platform operation of applications. In addition, the use of container cluster technology can realize the distributed operation of applications. While enhancing the stability of applications, it can also achieve the purpose of load balancing and improve the convenience of large-scale container cluster management. Among them, the container cluster technology can be k8s (Kubernetes , an open source container cluster management system), based on Docker (an open source application container engine) technology, k8s can provide containerized applications with a series of complete functions such as deployment and operation, resource scheduling, service discovery and dynamic scaling .

After the workflow to be executed is determined, the workflow to be executed is described by a preset description method, such as a directed acyclic graph (Direct Acyclic Gragh, DAG), to obtain a corresponding description model, that is, a workflow model. In graph theory, if a directed graph cannot start from a certain vertex and return to that point through several edges, then the graph is a directed acyclic graph. The workflow is composed of workflow nodes, and each workflow node can contain one or more applications. DAG is a data structure. The DAG is used to describe the workflow to be executed, that is, to store the pending execution through this data structure. Workflow nodes, node application calling relationship and other information. In addition, when modeling the DAG, it can be modeled with a modeling language such as json or yaml, so as to save the workflow information corresponding to the workflow to be executed in the workflow model. Since the directed acyclic graph supports asynchronous concurrency and can form a topological tree structure, describing the workflow to be executed through DAG can greatly improve the scalability of the workflow scheduling system.

Specifically, the description model includes a workflow vertex set and a workflow edge set, each vertex in the workflow vertex set represents an array description of an application interface; each edge in the workflow edge set is composed of an edge description array. Among them, the workflow vertex set includes the unique identifier ID of the application interface, the name of the application interface, the parameters of the application interface, etc., and the unique identifier ID of the application interface is a unique representation string generated by using the UUID (Universally Unique Identifier) specification; The interface name is represented by a string; the application interface parameter is represented by an array of key-value pairs. In addition, workflow edge collection, that is, each edge is composed of an edge description array, including edge unique identifier ID, source vertex, destination vertex, condition, etc. The edge unique identifier ID is a unique representation string generated by UUID specification; source The vertex and the destination vertex are represented by the unique identifier ID of the application interface; the condition is the set of conditions for edge execution.

It should be noted that UUID is a standard for software construction, which allows all elements in a distributed system to have unique identification information, without the need to specify the identification information through the central control terminal. In this way, each workflow can create a unique UUID, which will not conflict with other workflows, and there is no need to consider the problem of duplicate names when creating a database.

Step c2, receiving a workflow execution request based on the description model.

In this embodiment, if the workflow to be executed is described by DAG, after obtaining the corresponding description model, the description model in DAG format can be scanned and stored in a preset database. Then, the user can obtain the workflow-related information in the preset database on the client, and trigger the workflow execution request according to the actual application requirements; it is also possible to pre-set the workflow execution start time point of each workflow to be executed in the scheduling system, When this start time point is reached, the client automatically triggers a workflow execution request. When receiving a workflow execution request, the scheduling system can schedule the workflow in a stable and orderly manner according to the workflow execution request. For example, when a customer submits order information in the scheduling system, that is, when the order information is generated, the order processing process in the description model will be activated, and the scheduling system will perform workflow scheduling according to the order processing process.

The container-based workflow scheduling method of this embodiment triggers a workflow execution request by building a workflow model, which can improve workflow execution efficiency and help improve the stability of the scheduling system.

Further, based on the first and second embodiments of the container-based workflow scheduling method of the present invention, a third embodiment of the container-based workflow scheduling method of the present invention is proposed.

The difference between the third embodiment of the container-based workflow scheduling method and the first and second embodiments of the container-based workflow scheduling method is that the calculation corresponding to the target workflow is determined by the first-level scheduler The steps for a resource node include:

Step d1, determining the initial computing resource node corresponding to the target workflow according to the workflow execution request;

Step d2, obtaining the load condition corresponding to the initial computing resource node, and determining the target computing resource node corresponding to the target workflow through the primary scheduler according to the load condition.

In this embodiment, when the user executes the target workflow through the client, the computing resource node corresponding to the workflow is selected by the user, that is, the workflow execution request contains information about the initial computing resource node corresponding to the target workflow . When the scheduling system receives a workflow execution request, it will give priority to the load on the initial computing resource node, and then the first-level scheduler will distribute the target workflow according to the specific load, so as to determine the computing resources corresponding to the target workflow node.

Further, the step of determining the target computing resource node corresponding to the target workflow according to the load condition through the primary scheduler includes:

Step e1, when the load condition is within a preset load range, the primary scheduler determines that the initial computing resource node is the target computing resource node corresponding to the target workflow;

Step e2, when the load condition is not within the preset load range, the node load condition of other computing resource nodes is detected by the first-level scheduler, and other nodes whose node load condition is within the preset load range are obtained In the computing resource node, the target computing resource node corresponding to the target workflow is determined.

In this embodiment, in order to ensure efficient execution of the workflow, a load range on computing resource nodes is preset, that is, a preset load range. When it is detected that the load on the initial computing resource node is within the preset load range, it means that the resources on the initial computing resource node are sufficient, and the first-level scheduler will preferentially allocate the target workflow to the initial computing resource node; When the load on the initial computing resource node is not within the preset load range, that is, when the resources on the initial computing resource node are limited, the first-level scheduler re-arranges the target workflow and assigns it a suitable running surroundings. Specifically, the first-level scheduler detects the load on other computing resource nodes, that is, the node load, and selects a computing resource node as the target work from other computing resource nodes whose node load is within the preset load range The target computing resource node corresponding to the flow.

It is understandable that since a certain first-level scheduler can seize resources from the workflow queue by competing for consumption queues, it means that the load on this first-level scheduler is lighter than that of other first-level schedulers, and it can The workflow information is competitively obtained from the workflow queue, so the first-level scheduler will preferentially use the initial computing resource node to process the target workflow.

In the container-based workflow scheduling method of this embodiment, the target computing resource node corresponding to the target workflow is determined according to the load on the initial computing resource node, and the default initial computing resource node in the workflow execution request is prioritized. The target workflow is processed, which improves the workflow scheduling efficiency of the scheduling system.

Further, based on the first, second, and third embodiments of the container-based workflow scheduling method of the present invention, a fourth embodiment of the container-based workflow scheduling method of the present invention is proposed.

The difference between the fourth embodiment of the container-based workflow scheduling method and the first, second, and third embodiments of the container-based workflow scheduling method is that the task node After the step of performing concurrent execution processing to schedule the target workflow, it further includes:

Step f1, receiving the execution processing results returned by each of the task nodes through the target secondary scheduler;

Step f2, according to the execution processing result, determine whether the target workflow is scheduled to be completed.

In this embodiment, after the execution of each task node is completed, the execution processing result of the task node can be returned by calling the Restful API, and then the execution processing result returned by each task node can be received through the target secondary scheduler, and the execution result can be fed back to the user . After the feedback of the execution processing results of all task nodes is completed, the target secondary scheduler collects the execution processing results in a unified manner, and feeds back the workflow scheduling results of each layer according to the execution processing results, so as to determine the task to be executed according to the workflow scheduling results Whether the target workflow is scheduled to complete. Specifically, if the user is fed back the result of DAG scheduling success, the target workflow can be mapped to computing resources for scheduling after the user confirms; if the user is fed back the result of DAG scheduling failure, the user will execute the processing according to the feedback As a result, the selection of the next activity is carried out until the scheduling work of this workflow is completed.

In the container-based workflow scheduling method of this embodiment, after performing concurrent execution processing on each task node through the target secondary scheduler, by receiving the execution processing results returned by each task node, it is helpful to determine whether the target workflow scheduling is completed. It helps users know the execution progress of workflow scheduling in time, and improves the stability of the scheduling system to a certain extent.

The invention also provides a container-based workflow scheduling device. Referring to Fig. 4, the container-based workflow scheduling device of the present invention is applied to a container-based workflow scheduling system, the container-based workflow scheduling system includes a first-level scheduler and a second-level scheduler, and the device includes:

The receiving determination module 10 is configured to determine a target workflow corresponding to the workflow execution request when a workflow execution request is received, and assign the target workflow to a preset workflow queue;

The monitoring determination module 20 is configured to monitor the preset workflow queue through the first-level scheduler, so as to obtain the target workflow from the preset workflow queue, and determine the target workflow through the first-level scheduler. The computing resource node corresponding to the target workflow;

The starting scheduling module 30 is configured to start the target secondary scheduler corresponding to the computing resource node through the primary scheduler, and schedule the target workflow through the target secondary scheduler.

Preferably, the monitoring determination module further includes a node determination unit, and the node determination unit is used for:

determining an initial computing resource node corresponding to the target workflow according to the workflow execution request;

Obtain the load condition corresponding to the initial computing resource node, and determine the target computing resource node corresponding to the target workflow according to the load condition through the primary scheduler.

Preferably, the node determining unit is also used for:

When the load condition is within a preset load range, the primary scheduler determines that the initial computing resource node is a target computing resource node corresponding to the target workflow;

When the load condition is not within the preset load range, the primary scheduler detects the node load conditions of other computing resource nodes, and obtains from other computing resource nodes whose node load conditions are within the preset load range , determine the target computing resource node corresponding to the target workflow.

Preferably, the startup scheduling module is also used for:

Analyzing the target workflow through the target secondary scheduler to determine the task node corresponding to the target workflow;

The task nodes are executed concurrently by the target secondary scheduler, so as to schedule the target workflow.

Preferably, the container-based workflow scheduling device further includes an execution feedback unit, and the execution feedback unit is used for:

receiving execution processing results returned by each of the task nodes through the target secondary scheduler;

According to the execution processing result, it is determined whether the target workflow is scheduled to be completed.

Preferably, the monitoring determination module is also used for:

Monitor the preset workflow queue through the first-level scheduler, and actively pull the target workflow from the preset workflow queue through the first-level scheduler, so as to obtain the target workflow workflow information;

The computing resource node corresponding to the target workflow is determined by the primary scheduler according to the workflow information.

Preferably, the container-based workflow scheduling device further includes a description trigger unit, and the description trigger unit is used for:

Determining the workflow to be executed, and describing the workflow to be executed according to a preset description method, to obtain a corresponding description model;

A workflow execution request is received based on the description model.

The invention also provides a storage medium.

A container-based workflow scheduler is stored on the storage medium of the present invention, and when the container-based workflow scheduler is executed by a processor, the steps of the above-mentioned container-based workflow scheduler are implemented.

For the method implemented when the container-based workflow scheduling program running on the processor is executed, reference may be made to various embodiments of the container-based workflow scheduling method of the present invention, which will not be repeated here.

It should be noted that, as used herein, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or system comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or system. Without further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system comprising that element.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present invention can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , magnetic disk, optical disk), including several instructions to make a terminal system (which can be a mobile phone, a computer, a server, an air conditioner, or a network system, etc.) execute the methods described in various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (9)

1. A method for scheduling a container-based workflow is applied to a container-based workflow scheduling system, wherein the container-based workflow scheduling system comprises a primary scheduler and a secondary scheduler, and the method comprises the following steps:

when a workflow execution request is received, determining a target workflow corresponding to the workflow execution request, and distributing the target workflow to a preset workflow queue;

monitoring the preset workflow queue through the primary scheduler to obtain the target workflow from the preset workflow queue, and determining a computing resource node corresponding to the target workflow through the primary scheduler;

starting a target secondary scheduler corresponding to the computing resource node through the primary scheduler, and scheduling the target workflow through the target secondary scheduler;

before the step of determining the target workflow corresponding to the workflow execution request, the method further includes:

determining a workflow to be executed, and describing the workflow to be executed according to a preset description mode to obtain a corresponding description model, wherein the description model comprises a workflow vertex set and a workflow edge set, and each vertex in the workflow vertex set represents an application interface array description; each edge in the workflow edge set consists of an edge description array;

a workflow execution request is received based on the description model.

2. The method for container-based workflow scheduling of claim 1 wherein said step of determining by said primary scheduler the computational resource node to which said target workflow corresponds comprises:

determining an initial computing resource node corresponding to the target workflow according to the workflow execution request;

and acquiring the load condition corresponding to the initial computing resource node, and determining a target computing resource node corresponding to the target workflow according to the load condition through the primary scheduler.

3. The method for container-based workflow scheduling of claim 2 wherein said step of determining, by said primary scheduler, a target computing resource node corresponding to said target workflow based on said load condition comprises:

when the load condition is within a preset load range, determining the initial computing resource node as a target computing resource node corresponding to the target workflow through the primary scheduler;

when the load condition is not in a preset load range, detecting the node load condition of other computing resource nodes through the primary scheduler, and determining a target computing resource node corresponding to the target workflow from the other computing resource nodes of which the node load condition is in the preset load range.

4. The container-based workflow scheduling method of claim 1 wherein said step of scheduling said target workflow by said target secondary scheduler comprises:

analyzing the target workflow through the target secondary scheduler to determine a task node corresponding to the target workflow;

and performing concurrent execution processing on the task nodes through the target secondary scheduler so as to schedule the target workflow.

5. The method for container-based workflow scheduling of claim 4 wherein said step of concurrently executing processing of said task nodes by said target secondary scheduler to schedule said target workflow is followed by further comprising:

receiving execution processing results returned by the task nodes through the target secondary scheduler;

and determining whether the target workflow is scheduled to be completed or not according to the execution processing result.

6. The method for scheduling workflow based on container according to claim 1, wherein the step of listening to the preset workflow queue through the primary scheduler to obtain the target workflow from the preset workflow queue and determining the computing resource node corresponding to the target workflow through the primary scheduler comprises:

monitoring the preset workflow queue through the primary scheduler, and actively pulling the target workflow from the preset workflow queue through the primary scheduler to acquire workflow information in the target workflow;

and determining a computing resource node corresponding to the target workflow according to the workflow information through the primary scheduler.

7. A container-based workflow scheduling apparatus applied to a container-based workflow scheduling system including a primary scheduler and a secondary scheduler, the container-based workflow scheduling apparatus comprising:

the system comprises a receiving and determining module, a processing module and a processing module, wherein the receiving and determining module is used for determining a target workflow corresponding to a workflow execution request when the workflow execution request is received, and distributing the target workflow to a preset workflow queue;

the monitoring determining module is used for monitoring the preset workflow queue through the primary scheduler to acquire the target workflow from the preset workflow queue and determining a computing resource node corresponding to the target workflow through the primary scheduler;

the starting scheduling module is used for starting a target secondary scheduler corresponding to the computing resource node through the primary scheduler and scheduling the target workflow through the target secondary scheduler;

wherein the apparatus further comprises:

the monitoring determining module is further configured to determine a workflow to be executed, and describe the workflow to be executed according to a preset description mode to obtain a corresponding description model, where the description model includes a workflow vertex set and a workflow edge set, and each vertex in the workflow vertex set represents an application interface array description; each edge in the workflow edge set consists of an edge description array;

the receiving determination module is further configured to receive a workflow execution request based on the description model.

8. A container-based workflow scheduling system, the container-based workflow scheduling system comprising: memory, a processor and a container-based workflow scheduler stored on the memory and executable on the processor, the container-based workflow scheduler when executed by the processor implementing the steps of the container-based workflow scheduling method according to any of claims 1 to 6.

9. A storage medium having stored thereon a container-based workflow scheduler, the container-based workflow scheduler when executed by a processor implementing the steps of the container-based workflow scheduling method according to any of claims 1 to 6.

Images