CN105630589A - Distributed process scheduling system and process scheduling and execution method - Google Patents

Abstract

The invention discloses a distributed process scheduling system and a process scheduling and execution method, wherein the system includes: a first management node and a plurality of process processing nodes located in a cluster system, wherein the first management node includes: a load A balancing module, configured to select a process processing node that performs a process processing task from the multiple process processing nodes according to a preset load balancing strategy, and distribute the process processing task to the selected process processing node; the multiple process processing nodes Each process processing node in the nodes is used to execute process processing tasks. Through the present invention, load balancing and distributed cluster systems are adopted to realize distributed scheduling of processes.

Description

Distributed process scheduling system and process scheduling and execution method

technical field

The invention relates to the field of distributed data processing, in particular to a distributed process scheduling system and a process scheduling and execution method.

Background technique

Since the concept of process was formally proposed, process management has been valued by scholars and enterprises, and people's understanding and understanding of process has been deepened. A scientific and mature process management model has been formed in the fields of scientific research and industry. The logical relationship is also becoming more and more complex. As the number and complexity of process applications grows rapidly, the amount of data within them explodes.

At present, mainstream workflow engines including business process management (JavaBusinessProcessManagement, referred to as JPBM) and Activiti have been widely used in actual business systems. The process definition is written through industry standard Business Process Modeling and Notation (Business Process Modeling Notation, referred to as BPMN), and the process is described through Extensible Markup Language (Extensible Markup Language, referred to as XML). By calling the system interface, the loading and starting of the process and related operations of the process task can be easily realized. However, traditional workflow engines are deployed and run independently, and their processing capabilities are stretched in today's big data era.

Later, some scholars proposed Simple Workflow (SimpleWorkflow, referred to as SWF). As a workflow engine based on a distributed computing scheduling framework, SWF is composed of Worker and Decider. Worker performs actual tasks, and Decider performs process control. Its design structure It is decided that its functions are used to process simple WorkFlow, because all process control is concentrated in Decider. If it is too complicated, Decider will be huge. It brings great difficulties to maintenance and expansion.

Aiming at the problem of high-performance process scheduling and management under the condition of large data volume in related technologies, no effective solution has been proposed yet.

Contents of the invention

Aiming at the problem of high-performance process scheduling and management under the condition of large amount of data in the related art, the present invention provides a distributed process scheduling system and a process scheduling and execution method to at least solve the above problems.

According to one aspect of the present invention, a distributed process scheduling system is provided, including: a first management node and a plurality of process processing nodes located in a cluster system, wherein the first management node includes: a management module for managing all The plurality of process processing nodes; a load balancing module, configured to select a process processing node that executes process processing tasks from the plurality of process processing nodes according to a preset load balancing strategy, and distribute the process processing tasks to the selected process processing nodes A node; each process processing node among the plurality of process processing nodes is configured with a process processing engine for executing a process processing task.

Optionally, each of the plurality of process processing nodes is configured to determine the process engine used by the process processing task according to the process definition file standard, adapt the determined process engine, and start the process engine after the adaptation is completed. The process engine executes the process processing tasks.

Optionally, each of the plurality of process processing nodes is further configured to parse the process processing task, persist the data in the process processing task to the first database, store the process processing task The result of the process is persisted to the second database, and the execution state of the process processing task is cached in the memory database.

Optionally, the system further includes: a second management node, configured to monitor the working status of the plurality of process processing nodes, and request process recovery from the load balancing node when a process processing node fails; the load The balancing module is further configured to, in response to the request of the second management node, select a process processing node to execute the process processing task requested to be restored.

Optionally, the first management node is configured to manage the cluster status and process processing status of the plurality of process processing nodes, and synchronize the cluster status and the aforementioned process processing status to the second management node; The second management node is configured to monitor the working status of the plurality of process processing nodes, and request process restoration to the load balancing node according to the data synchronized by the first management node when a process processing node fails.

Optionally, the second management node includes: an active second management node and a standby second management node, and the active second management node and the standby second management node adopt an active-standby mechanism.

According to another aspect of the present invention, a process scheduling method is provided, including: responding to a process processing request, selecting a process process for executing process processing tasks from multiple process processing nodes located in the cluster system according to a preset load balancing strategy node; distribute the process processing task to the selected process processing node.

Optionally, the method further includes: monitoring the working status of the plurality of process processing nodes; when a process processing node fails, reselecting the process processing node to execute the process processing task performed by the failed process processing node.

According to another aspect of the present invention, a process execution method is provided, including: responding to a process processing task, determining the process definition file standard according to the process processing task; determining the process engine used by the process processing task according to the process definition file standard; The process engine performs adaptation, and after the adaptation is completed, the process engine is started to execute the process processing task.

Optionally, the above method further includes: parsing the process processing task, persisting the data in the process processing task into the first database; persisting the result of the process processing task into the second database; The execution status of processing tasks is cached in the memory database.

Through the present invention, load balancing and distributed cluster systems are adopted to realize distributed scheduling of processes.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

FIG. 1 is a schematic diagram of a distributed process scheduling system according to an embodiment of the present invention;

2 is a flow chart of a process scheduling method according to an embodiment of the present invention;

FIG. 3 is a flowchart of a process execution method according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of an optional distributed process scheduling system according to an embodiment of the present invention.

detailed description

Hereinafter, the present invention will be described in detail with reference to the drawings and examples. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

Fig. 1 is a schematic diagram of a distributed process scheduling system according to an embodiment of the present invention. As shown in Fig. 1, the system includes: a first management node 1 and a plurality of process processing nodes 2 located in a cluster system, wherein the first management The node 1 includes: a management module, configured to manage the plurality of process processing nodes; a load balancing module, configured to select a process processing node from among the plurality of process processing nodes 2 to perform a process processing task according to a preset load balancing strategy , and distribute the process processing task to the selected process processing node; each process processing node in the plurality of process processing nodes 2 is configured with a process processing engine for executing the process processing task.

In the embodiment of the present invention, a process processing engine (also referred to as a workflow engine) may be set on each process processing node to execute process processing tasks. Process processing tasks are workflows that request execution. Workflow is the calculation model of workflow, that is, the logic and rules of how to organize the work in the workflow are represented in the computer with an appropriate model and calculated.

In the embodiment of the present invention, the load balancing module may be used to implement load balancing at the IP layer. After receiving the process processing request, the load balancing module selects a node for processing the process processing task, replaces the IP address in the process processing request with the IP address of the selected process processing node, and sends the process processing request to the process processing node. The destination IP address in the process processing request received by the load balancing module is the address provided by the system, so there is only one IP address for external users.

In an optional implementation manner of the embodiment of the present invention, each of the above-mentioned multiple process processing nodes can be used to determine the process engine used by the process processing task according to the process definition file standard, and to determine the process engine Adaptation is performed, and after the adaptation is completed, the process engine is started to execute the process processing task. Through this embodiment, the scalability of the system is improved.

In an optional implementation manner of the embodiment of the present invention, each of the above-mentioned multiple process processing nodes is further configured to parse the process processing task, and persist the data in the process processing task to the first In the first database, the result of the process processing task is persisted in the second database, and the execution status of the process processing task is cached in the memory database. Through this implementation, the process processing task execution status is asynchronously stored in the memory database, providing users with efficient asynchronous query services; at the same time, the data related to the process processing nodes is persistently stored in the traditional relational database to provide data for the system.

In an optional implementation method of the embodiment of the present invention, the above system further includes: a second management node, configured to monitor the working status of the plurality of process processing nodes, and when a process processing node fails, load balancing The node requests process restoration; the load balancing module is further configured to respond to the request of the second management node and select a process processing node to execute the process task for which restoration is requested.

Optionally, the first management node 1 is configured to manage the cluster status and process processing status of the plurality of process processing nodes, and synchronize the cluster status and the aforementioned process processing status to the second management node; The second management node is configured to monitor the working status of the plurality of process processing nodes, and request process restoration to the load balancing node according to the data synchronized by the first management node when a process processing node fails. The above-mentioned first management node and second management node may be different management nodes, and different nodes are responsible for different tasks, which can reduce the complexity of each node.

Optionally, the above-mentioned second management node includes: an active second management node and a standby second management node, and the active second management node and the standby second management node adopt an active-standby mechanism.

Fig. 2 is a flowchart of a process scheduling method according to an embodiment of the present invention. As shown in Fig. 2, the method includes steps 201 to 202:

Step 201 , in response to a process processing request, select a process processing node to execute a process processing task from multiple process processing nodes located in the cluster system according to a preset load balancing strategy.

Step 202, distribute the process processing tasks to the selected process processing nodes.

In the above step 201, load balancing at the IP layer can be implemented. Optionally, the process scheduling system provides an IP address to the outside, and the user uses this IP address as the destination IP address when requesting. After the flow processing node is selected, the destination IP address in the flow processing request is replaced with the IP address of the selected flow processing node.

In an optional implementation of the embodiment of the present invention, the above method may further include: monitoring the working status of the plurality of process processing nodes; when a process processing node fails, reselecting the process processing node to execute the failed process processing Process processing tasks performed by nodes.

Optionally, the information of the process processing tasks performed by each node can be saved in advance. When a failure occurs, the process processing tasks to be restored can be determined according to the stored information, and the process processing nodes can be reselected to request the new process processing nodes to process the required tasks. The resumed process handles the task.

Fig. 3 is a flowchart of a process execution method according to an embodiment of the present invention. As shown in Fig. 3, the method includes steps 301 to 303:

Step 301, responding to the process processing task, determining the standard of the process definition file according to the process processing task;

Step 302, determine the process engine used by the process processing task according to the standard of the process definition file;

In step 303, the determined process engine is adapted, and after the adaptation is completed, the process engine is started to execute the process processing task.

In the above step 303, the adaptation may include service processor adaptation, business context adaptation, engine providing special service adaptation, and function adaptation based on specific requirements.

In an implementation manner of the embodiment of the present invention, the above method may further include: parsing the process processing task, persisting the data in the process processing task to the first database; persisting the result of the process processing task to In the second database; and cache the execution state of the process processing task in the memory database.

Optional implementation manners of the embodiments of the present invention are described below.

Optional implementation mode one

The purpose of the embodiments of the present invention is to provide users with highly available distributed process scheduling and management services. First build a highly available distributed cluster and deploy a load balancer in it. Deploy the workflow engine on the cluster nodes to ensure the normal operation of each process engine in the cluster while ensuring the communication between the nodes. In order to provide users with highly available distributed process scheduling and management services.

To achieve the above purpose, by deploying a cluster management system on the cluster, a mapping is created for each processing node, and node information and node process processing information are stored therein. By creating active and standby Manager nodes to back up and manage the information on the service nodes of the cluster management system, high availability of the system is achieved. In addition, the Manager node can monitor the information on each node. If the node is down, the process on the node will be activated restore on other nodes. By deploying a load balancer on the cluster and selecting an appropriate load balancing strategy to achieve load balancing at the IP layer, it can effectively prevent the load balancing node from becoming the bottleneck of the system. Setting workflow engine adapters on the above-mentioned cluster nodes can enable the system to adapt to different workflow engines as needed, uniformly schedule the cluster, and provide users with transparent process scheduling and management services.

Embodiments of the present invention are described below:

1. High availability cluster implementation:

Step 1. Install and deploy the cluster management system on the cluster;

Step 2. Install and configure a load balancer on the cluster;

Step 3. Establish event monitoring on the server side of the cluster management system on the Manager node.

The establishment of process processing sub-nodes, related information of node computers and process information of process processing will be saved in real time when the cluster management system establishes the corresponding nodes. The Manager node can monitor the data changes on the cluster management system side in real time.

Step 4. The Manager node saves the data changes on the cluster management system locally in a timely manner, and optionally stores them in an xml file.

Step 5. Establish the main and standby Managers to keep the data synchronization of the main and standby nodes. When the main Manager goes down for some reason, enable the standby node to continue data monitoring.

2. Process recovery implementation:

Step 1. The Manager node monitors the status of each process processing node, finds events such as node downtime or process processing errors, and triggers the process recovery function.

Step 2. The Manager obtains the process ID to be restored from the locally stored data.

Step 3. The Manager node obtains relevant information for execution from the database according to the process ID.

Step 4. The Manager node converts the information into a process operation request and resubmits it to the cluster system for task assignment and execution.

3. Realization of control instructions and business requests

Step 1. After the external system receives and sends the business request signal, it sends the business request to the process scheduling and service software through the external interface provided by the process scheduling (load balancing) and service software system;

Step 2. Process scheduling (load balancing) and service software analyze the business request after receiving the business request;

Step 3. If the parsing fails, the external system will be prompted for parsing request failure;

Step 4. After correctly analyzing the request, the process scheduling and service software executes tasks according to the process definition;

Step 5. After the task execution fails, the external system will be prompted to execute the task failure;

Step 6. The task is executed successfully, the execution result is saved to the database, and the operation of receiving the business request is completed.

4. Process engine adapter implementation

Step 1. The distributed process scheduling and management service software system obtains the process definition file provided by the external system through the external interface;

Step 2. Determine the process engine to be used according to the process definition file standard;

Step 3. Adapt the selected process engine, including business processor adaptation, business context adaptation, engine-provided special service adaptation, and function adaptation based on specific requirements;

Step 4. After the adaptation is completed, start the process engine, load the process definition, and start the process instance;

Step 5. After starting the process instance, execute the process according to the workflow or business flow defined in the process;

Step 6. If an error occurs during execution, a detailed error prompt is given, and the error can be corrected according to the error prompt to ensure the correct execution of the process;

Step 7. There is no abnormality during the execution process, and the process execution result is stored in the database. If the storage fails, a failure reason prompt is given;

Step 8. The process is successfully executed and completed.

Through this embodiment: (1) An efficient distributed service cluster is established through the load balancer + cluster management system; (2) High availability of the system is realized through the active/standby Manager node mechanism, and the unified process is carried out through the Manager node Restoration; (3) Improve the scalability of the system through process engine adaptation and reasonable instructions and business requests. (4) The system provides persistence services by setting two data persistence methods. The process request execution status is asynchronously stored in the memory database, providing users with efficient asynchronous query services. At the same time, all the information of the nodes is persistently stored in the traditional relational database to provide data for the system.

Optional implementation mode two

As shown in Figure 4, the system of the embodiment of the present invention consists of four parts:

The first part is realized by the cluster management system framework + load balancer, in which the cluster management system is used to manage the process processing node cluster status and process status, and at the same time back up data to the Manager node and provide node information. The load balancer provides load balancing at the IP layer. Users access the system through the virtual IP address provided by it. After the user's request reaches the load balancer, the IP address of the data packet is rewritten into an available process processing node IP according to the load balancing strategy. Complete the distribution of the request.

The second part of the management node subsystem is responsible for the high availability (HA) work of the process processing subsystem. By setting the active and standby management node mechanism and synchronizing the data of the cluster management system to the active and standby management nodes, the monitoring mechanism can monitor the work of the processing nodes in real time. When the process processing node is down, according to the backup data information, the load balancer selects the node for process recovery.

The third part of the communication proxy subsystem realizes the communication proxy between the business application and the process processing node, shields the content of the communication between the application system and other subsystems, and achieves the transparency of the communication between the application system and the process processing system.

The fourth part of the process processing subsystem is mainly responsible for process definition and process instance management, start and execute the process, and complete business request processing. In order to meet different process processing requirements, the system adopts an architecture design independent of specific process engines, and at the same time, process engines can be dynamically implanted to meet special business needs.

The following is an example to illustrate:

Step 1. Deploy the cluster management system load balancing cluster;

Step 2. Deploy the management node subsystem and deploy the process processing subsystem on the process processing sub-node;

Step 3. Start the management node subsystem; the system automatically establishes a relevant mapping on the cluster management system, and stores the node information in the cluster management system. At the same time, the database will be initialized, and the basic process information on the node will be stored in the database.

Step 4. The management node subsystem detects data changes on the cluster management system, and saves the data locally.

Step 5. The client sends a request, and the target address of the request is the virtual service IP address provided by the load balancer;

Step 6. After receiving the user request, the load balancer rewrites the IP address of the requested data packet according to the load balancing strategy, and distributes the IP data packet to the destination flow processing subsystem;

Step 7. The flow processing subsystem parses the data packet, and stores the data in the data packet. For example, when the workflow is loaded for the first time, the detailed information in the workflow is stored and persisted to the database for storage;

Step 8. Execute the user's operation control request, and return the result to the user in real time;

Step 9. Save the processing results to the memory database, and the memory database can be accessed for asynchronous data access when the user needs it;

Step 10. Save the process processing results to the cluster management system, and persist them to the database for backup storage;

Step 11. The user operation request ends;

Step 12. Node abnormalities are found in the node management subsystem, and the process ID currently being executed by the current node is read through local information;

Step 13. According to the process ID, look up the detailed information of the process in the database, resend the process execution request, and resume the process;

Step 14. The node management subsystem detects that the process execution is abnormal, and performs the process recovery operation as above;

Step 15. During system operation, the backup node detects that the master node management subsystem is down, and starts the backup node monitoring system.

The process processing subsystem in this example can provide users with efficient distributed process scheduling and management services under the reasonable configuration of the server. At the same time, the availability of the system is greatly improved through the node management subsystem to ensure the stable operation of the system.

In addition, the data persistence solution in this example uses the memory database and the traditional relational database respectively to limit the user's data access rights, speed up the system's access to data, and reduce the system's burden. Moreover, a variety of process engines can be used by setting the process engine adapter, so that the scalability of the system is greatly enhanced.

From the above description, it can be seen that the present invention achieves the following technical effects: load balance and distributed cluster system, and realizes distributed scheduling of processes.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned present invention can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network formed by multiple computing devices Alternatively, they may be implemented in program code executable by a computing device so that they may be stored in a storage device to be executed by a computing device, and in some cases in an order different from that shown here The steps shown or described are carried out, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present invention is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. a distributed flow process dispatching patcher, it is characterised in that including: the first management node and the multiple flow processing nodes being arranged in group system, wherein,

Described first management node includes: management module, is used for managing the plurality of flow processing node; Load balancing module, for selecting to perform the flow processing node of flow processing task from the plurality of flow processing node according to default load balancing, and is distributed to the flow processing node of selection by flow processing task;

Each flow processing node in the plurality of flow processing node, is configured with flow processing engine, is used for performing flow processing task.

2. system according to claim 1, it is characterized in that, each flow processing node in the plurality of flow processing node, for according to flow definition file standard, determine the flow engine that flow processing task uses, the flow engine determined is carried out adaptation, and after adaptation completes, Booting sequence engine performs described flow processing task.

3. system according to claim 1 and 2, it is characterized in that, each flow processing node in the plurality of flow processing node, it is additionally operable to resolve described flow processing task, by in the data persistence in described flow processing task to the first data base, the result of flow processing task is persisted in the second data base, and by flow processing execution status of task buffer memory to memory database.

4. system according to any one of claim 1 to 3, it is characterised in that

Described system also includes: the second management node, for monitoring the duty of the plurality of flow processing node, when flow processing one malfunctions, asks flow process to be recovered to described load balancing node;

Described load balancing module, is additionally operable to the request of the described second management node of response, selects flow processing node to perform the flow processing task that request recovers.

5. system according to claim 4, it is characterised in that

Described first management node, for managing cluster state and the flow processing state of the plurality of flow processing node, and by the above-mentioned flow processing state synchronized of described cluster state and institute to described second management node;

Described second management node, for monitoring the duty of the plurality of flow processing node, when flow processing one malfunctions, asks flow process to be recovered according to the data that described first manages node synchronization to described load balancing node.

6. the system according to claim 4 or 5, it is characterised in that described second management node includes: main second management node and standby second management node, described master second manages node and described standby second management node adopts active and standby mechanism.

7. a process dispatch method, it is characterised in that including:

Responding process processes request, selects to perform the flow processing node of flow processing task from the multiple flow processing nodes being arranged in group system according to default load balancing;

Described flow processing task is distributed to the flow processing node selected.

8. method according to claim 7, it is characterised in that also include:

Monitor the duty of the plurality of flow processing node;

When flow processing one malfunctions, reselect the flow processing task performed by flow processing node that flow processing node performs to break down.

9. a flow executing method, it is characterised in that including:

Responding process processes task, determines flow definition file standard according to flow processing task;

The flow engine that flow processing task uses is determined according to flow definition file standard;

The flow engine determined is carried out adaptation, and after adaptation completes, Booting sequence engine performs described flow processing task.

10. method according to claim 9, it is characterised in that also include:

Resolve described flow processing task, by the data persistence in described flow processing task to the first data base;

The result of flow processing task is persisted in the second data base; And

By in flow processing execution status of task buffer memory to memory database.