CN115248750A - Distributed task scheduling method, apparatus, computer equipment and medium - Google Patents

Abstract

The application relates to a distributed task scheduling method, a distributed task scheduling device, computer equipment and a medium. In the distributed task scheduling method, original main node equipment creates task nodes in the process of executing a first target task, and writes task information of the first target task into the task nodes, so that after the original main node is down, secondary node equipment is converted into target main node equipment, and then the target main node equipment scans preset directory nodes in a Zookeeper system; if at least one task node exists under the directory node, reading task information from the at least one task node; and recovering the first target task according to the task information, and executing the first target task. Because the Zookeeper system has high availability, the data storage of the distributed system can be ensured not to be easy to lose efficacy, thereby ensuring the high availability of the distributed system.

Description

Distributed task scheduling method, apparatus, computer equipment and medium

technical field

The present application relates to the technical field of distributed systems, and in particular, to a distributed task scheduling method, apparatus, computer equipment and medium.

Background technique

A distributed system includes a plurality of computer devices, including a primary node device and a secondary node device. Among them, the master node device is used to receive task requests, and establish and execute tasks according to the task requests. When the master node device is down, the distributed system can convert the secondary node device into a new master node device, and then the new master node device The node device resumes the unfinished tasks of the previous master node device.

However, since the existing data storage system of the distributed system is prone to the problem of single point of failure, the high availability of the distributed system is affected.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a distributed task scheduling method, apparatus, computer equipment and medium for the above technical problems.

A distributed task scheduling method, comprising:

After converting from the secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system; if there is at least one task node under the directory node, read task information from at least one task node; among them, at least one The task node is created by the original master node device in the process of executing the first target task, and the task information is the information related to the execution of the first target task written by the original master node device; the first target task is restored according to the task information. task, and perform the first target task.

With the support of distributed storage of the Zookeeper system, the embodiment of the present application realizes that when the original master node device supporting the distributed system is down, the target master node device restores the first target task and continues to execute the first target The ability of tasks to improve the high availability of distributed systems.

In one embodiment, the method further includes:

After receiving the task request, establish a second target task according to the task request, and execute the second target task; in the process of executing the second target task, create a connection with the second target task under the preset directory node in the Zookeeper system corresponding task node, and write the task information corresponding to the second target task into the task node corresponding to the second target task.

When the second target task is established, the task information of the second target task is written into the Zookeeper system to prepare for subsequent recovery and execution of the second target task, and to improve the high availability service of the distributed system.

In one embodiment, a task node corresponding to the second target task is established under a preset directory node in the Zookeeper system, and task information corresponding to the second target task is written into the task node corresponding to the second target task , including: dividing the second target task into a plurality of target subtasks, each target subtask is a task of a different stage of the second target task; under the directory node, according to the execution sequence of each target subtask, establish and each target subtask The task node corresponding to the task, and the task information corresponding to each target subtask is written into the task node corresponding to each target subtask.

By dividing the second target task into multiple independent target subtasks, and establishing task nodes corresponding to each target subtask, in this way, when the target master node device is down, the new master node device only needs to restore the first A part of the target subtasks of the second target task does not need to be restored and re-executed in all of the second target task, which avoids repeated operations in the distributed system and affects the execution efficiency of the task.

In one embodiment, during the process of executing the second target task, a task node corresponding to the second target task is established under a preset directory node in the Zookeeper system, and the task information corresponding to the second target task is written Entering the task node corresponding to the second target task includes: dividing the second target task into a plurality of target subtasks arranged in order of execution, and each target subtask is a task of a different stage of the second target task; The task information corresponding to the target subtask is stored in the target task node; for each target subtask, when the target subtask is executed, the target subtask corresponding to the target subtask is established in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node. subtask node of the target subtask, and delete the subtask node corresponding to the previous target subtask of the target subtask.

The subtask nodes corresponding to each target subtask are not created at one time, but when the target subtasks are executed sequentially, whichever target subtask is executed, a subtask node is created, and the subtask is deleted after the subtask is executed. Nodes, and so on, execute each target subtask sequentially, which can save storage space on the zookeeper system.

In one embodiment, the method further includes: in the process of executing the second target task, acquiring task status information of the second target task until the execution of the second target task ends; the task status information is used to indicate the status of the second target task. Execution state; write the task state information into the task node corresponding to the second target task, so as to determine whether the execution of the second target task ends after the target master node device goes down.

By writing the task status information in the embodiment of the present application, frequent deletion operations are avoided in the directory node, the data processing volume of the Zookeeper system is reduced, and the work efficiency of the distributed system is improved.

In one of the embodiments, writing the task status information into the task node corresponding to the second target task includes: establishing a transaction mark under the directory node, where the transaction mark includes at least two task nodes to which the task status information is to be written The transaction flag is used to indicate that the write operation is performed synchronously on at least two task nodes corresponding to the identifiers of at least two task nodes; the task information in the at least two task nodes is backed up to obtain backup data; the task status information Write into at least two task nodes corresponding to the identifiers of at least two task nodes.

In one embodiment, the method further includes:

If the writing is successful, the backup data and transaction mark are deleted;

If the writing fails, the task information in the at least two task nodes corresponding to the identifiers of the at least two task nodes is restored according to the backup data, and the transaction mark is deleted.

By establishing transaction marks and deleting transaction marks, operations on multiple task nodes are realized, and the atomicity of Zookeeper system operations is guaranteed. Solved the problem that the Zookeeper system does not support simultaneous transaction operations on multiple task nodes. Completely supports the highly available job framework based on the Zookeeper system.

In one embodiment, the method further includes:

After the execution of the second target task is completed, the task node corresponding to the second target task is deleted from the directory node.

By deleting the task node corresponding to the second target task whose execution ends, the data storage amount can be simplified, and the data management burden of the Zookeeper system can be reduced.

A distributed task scheduling device, the device includes:

The scanning module is used to scan the preset directory nodes in the Zookeeper system after converting from the secondary node device to the target primary node device;

The reading module is used for reading task information from at least one task node if there is at least one task node under the directory node; wherein, at least one task node is the original master node device in the process of executing the first target task Created, the task information is the information related to the execution of the first target task written by the original master node device;

The restoration module is used for restoring the first target task according to the task information, and executing the first target task.

A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: after switching from a secondary node device to a target primary node device, scan preset directory nodes in the Zookeeper system If there is at least one task node under the directory node, then read task information from at least one task node; wherein, at least one task node is created by the original master node device in the process of executing the first target task, and the task information The information related to the execution of the first target task written for the original master node device; the first target task is restored according to the task information, and the first target task is executed.

A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: after being converted from a secondary node device to a target master node device, scan a preset directory node in the Zookeeper system; If there is at least one task node under the directory node, read task information from at least one task node; wherein, at least one task node is created by the original master node device in the process of executing the first target task, and the task information is The information related to the execution of the first target task written by the original master node device; the first target task is restored according to the task information, and the first target task is executed.

The above-mentioned distributed task scheduling method, apparatus, computer equipment and medium can ensure the high availability of the distributed system. In the distributed task scheduling method, the original master node device creates a task node in the process of executing the first target task, and writes the task information of the first target task into the task node, so that when the original master node device After the downtime, the secondary node device is converted into the target master node device, and then the target master node device scans the preset directory node in the Zookeeper system; if there is at least one task node under the directory node, then the at least one task node The task information is read in the node; the first target task is restored according to the task information, and the first target task is executed. Since the Zookeeper system has high availability, it can ensure that the data storage of the distributed system is not easy to fail, thereby ensuring the high availability of the distributed system.

Description of drawings

1 is a schematic diagram of an implementation environment involved in a distributed task scheduling method in one embodiment;

2 is a schematic structural diagram of a master node device;

3 is a schematic structural diagram of another master node device;

4 is a flowchart of a distributed task scheduling method provided by an embodiment of the present application;

5 is a flowchart of another distributed task scheduling method provided by an embodiment of the present application;

6 is a flowchart of another distributed task scheduling method provided by an embodiment of the present application;

7 is a flowchart of another distributed task scheduling method provided by an embodiment of the present application;

8 is a flowchart of a method for performing a write operation on multiple task nodes in a Zookeeper system provided by an embodiment of the present application;

FIG. 9 is a structural block diagram of a distributed task scheduling apparatus provided by an embodiment of the present application;

FIG. 10 is a structural block diagram of another distributed task scheduling apparatus provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

First, the architecture of the distributed system involved in this case is explained:

A distributed system includes a cluster composed of multiple computer devices, and each computer device communicates through a network. Multiple computer devices cooperate to provide a certain service or resource to the outside world. Generally speaking, one computer device among the multiple computer devices included in the distributed system is the primary node device in the distributed system, and the other computer devices are the secondary node devices in the distributed system.

Secondly, the working process of the distributed system involved in this case is explained:

In the distributed system, the master node device executes the task, and in the process of executing the task, the task information of the task and the execution state information of the task are stored.

When the primary node device goes down, the distributed system can select a new primary node device from multiple secondary node devices based on preset rules. The new master node device can restore the tasks that were not executed and completed by the previous master node device according to the stored task information and execution state information, and then continue to execute.

In practical applications, the new master node device in the distributed system is highly dependent on the stored data when restoring tasks, so the effectiveness of the stored data seriously affects the high availability of the distributed system. However, the data storage of the existing distributed system is prone to failure, resulting in unstable high availability of the existing distributed system.

Based on the above problems, this proposal provides the following technical solutions: The master node device writes the task information into the Zookeeper system. Since the Zookeeper system is a distributed system and has high availability, it is not easy to fail. In this way, when the master node device goes down, it can be ensured that the new master node device can normally obtain the task information required for the recovery task from the Zookeeper system, thereby ensuring the high availability of the distributed system.

Further, the distributed task scheduling method provided by the embodiments of the present application is based on the distributed consistency support of the Zookeeper system, which simplifies the problem of consistency processing of high-availability task information among different servers. Moreover, the support of distributed storage based on the Zookeeper system simplifies the design of high-availability task information to synchronize data between different servers. Finally, the present application solves the technical problem that using the Zookeeper system does not support simultaneous transaction operations on multiple task nodes, and fully supports the highly available job framework based on the Zookeeper system.

Below, the implementation environment involved in the distributed task scheduling method provided by the embodiment of the present application will be briefly described.

As shown in FIG. 1 , the implementation environment includes a server cluster 101 and a computer equipment cluster 102 configured with a Zookeeper system. A distributed system is configured in the computer device cluster 102, and the distributed scheduling method provided in this application can be applied to the distributed system.

The Zookeeper system can provide support for distributed system consistency and distributed storage. The server cluster 101 configured with the Zookeeper system includes a master server and several slave servers, and the server cluster 101 configured with the Zookeeper system can be used to maintain a similar file. The data structure of the system, the data structure includes multiple directory nodes, each directory node may also include multiple sub-directory nodes, and each directory node or each sub-directory node can store data. When the master server in the server cluster 101 configured with the Zookeeper system goes down, the slave server in the server cluster 101 configured with the Zookeeper system can become the new master server to ensure the normal operation of the Zookeeper system.

The computer device cluster 102 includes multiple computer devices, one of the multiple computer devices is the primary node device 1021, and the rest of the computer devices are secondary node devices 1022, wherein the primary node device 1021 and the server configured with the Zookeeper system The master servers in the cluster 101 are connected in order to perform write operations and read operations in the Zookeeper system.

In this embodiment of the present application, the master node device may be a server or a terminal. If the master node device is a terminal, its internal structure diagram may be as shown in FIG. 2 , and the master node device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the master node device is used to provide computing and control capabilities. The memory of the master node device includes a non-volatile storage medium and an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network interface of the master node device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the distributed task scheduling method provided by the present application is implemented. The master node device may include a liquid crystal display screen or an electronic ink display screen, and the input device of the master node device may be a button, a trackball, or a touchpad, and may also be an external keyboard, touchpad, or mouse, and the like.

If the master node device is a server, its internal structure diagram may be as shown in FIG. 3 , and the master node device includes a processor, a memory and a network interface connected through a system bus. The processor of the master node device is used to provide computing and control capabilities. The memory of the master node device includes a non-volatile storage medium and an internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network interface of the master node device is used to communicate with an external terminal through a network connection. The computer program, when executed by the processor, implements a distributed task scheduling method.

Those skilled in the art can understand that the structures shown in FIG. 2 and FIG. 3 are only block diagrams of partial structures related to the solution of the present application, and do not constitute a limitation on the computer equipment to which the solution of the present application is applied. A computer device may include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.

Please refer to FIG. 4 , which shows a flowchart of a distributed task scheduling method provided by an embodiment of the present application. As shown in FIG. 4 , the distributed task scheduling method may include the following steps:

Step 401 , after switching from the secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system.

First of all, it should be noted that both the primary node device and the secondary node device configured with the distributed system are pre-stored with the path of the directory node, and the path of the directory node is used to point to the preset directory node in the Zookeeper system.

When the distributed system is running, the original master node device can receive the task request, and can establish and execute the first target task according to the task request. The first target task is an application task that needs to be executed by the distributed system, and the embodiment of the present application does not limit the actual task type of the first target task.

In the process of executing the first target task, the original master node device can access the directory node in the Zookeeper system based on the path of the pre-stored directory node, establish a task node under the directory node, and then transfer the first target task's path to the directory node. The task information is written into the task node. The task information of the first target task is information related to executing the first target task.

When the primary node device is down, the distributed system can select a new primary node device from multiple secondary node devices based on a preset rule, and the new primary node device is referred to as the target primary node device in this application. After the identity of the selected secondary node device is converted from the secondary node device to the target primary node device, the target primary node device needs to detect whether there is an unfinished task performed by the original primary node device. Therefore, the target primary node device needs to The path of the pre-stored directory node scans the preset directory node in the Zookeeper system.

Step 402: If there is at least one task node under the directory node, read task information from the at least one task node.

Wherein, at least one task node is created by the original master node device in the process of executing the first target task, and the task information is information related to executing the first target task written by the original master node device.

In an optional implementation manner, after the target master node device scans the directory node, if there is at least one task node under the directory node, it means that there is an unfinished task performed by the original master node device. In this case, The target master node device may read task information from the at least one task node.

If there is no task node under the directory node, it means that the original master node device finishes executing all the first target tasks.

In an optional implementation manner, if there is at least one task node under the directory node, it means that the original master node device once created the first target task, and the target master node device can read the at least one task node in the The task information, optionally, the task information may include the task status, and the task status is used to indicate whether the execution of the first target task has ended, so that the target master node device can determine according to the task information read from at least one task node. Whether there are unfinished tasks performed by the original master node device.

Step 403: Restore the first target task according to the task information, and execute the first target task.

In the embodiment of the present application, if the target master node device determines that there is an unfinished task performed by the original master node device, it can re-establish the first target task according to the task information, and then continue to execute the first target task.

In the embodiment of the present application, the original master node device creates a task node in the process of executing the first target task, and writes the task information of the first target task into the task node, so that when the original master node goes down After that, the secondary node device is converted into the target master node device, and then the target master node device scans the preset directory node in the Zookeeper system; if there is at least one task node under the directory node, then the at least one task node Read task information; restore the first target task according to the task information, and execute the first target task. Since the Zookeeper system has high availability, it can ensure that the data storage of the distributed system is not easy to fail, thereby ensuring the high availability of the distributed system.

In another embodiment of the present application, as shown in FIG. 5 , which shows another distributed scheduling method provided by the embodiment of the present application, the method includes:

Step 501, after receiving the task request, establish a second target task according to the task request, and execute the second target task.

In the embodiment of the present application, after receiving the task request, the target master node device may establish a second target task according to the task request, and execute the second target task.

Optionally, in this embodiment of the present application, the task request may carry task information, and the task information includes task attribute information, and the task attribute information is information related to establishing the second target task, and the target master node device may establish the task information according to the task information. the second target task, and execute the second target task.

It should be noted that, in the embodiment of the present application, the target task established by the original master node device is called the first target task, the task established by the target master node device is called the second target task, the first target task and the second target task. There is no temporal sequence relationship between tasks.

Step 502, in the process of executing the second target task, establish a task node corresponding to the second target task under a preset directory node in the Zookeeper system, and write the task information corresponding to the second target task into the second target task. in the task node corresponding to the target task.

In the process of executing the second target task, the target master node device can access the directory node in the Zookeeper system based on the address of the pre-stored directory node, and then establish a task node corresponding to the second target task under the directory node.

The task node corresponding to the second target task refers to that the task node contains a task identifier of the second target task, where the task identifier may be a task number or a name number. The target master node device can determine the target task corresponding to the task node by reading the task identifier in the task node.

After establishing the task node corresponding to the second target task, the target master node device may obtain task information corresponding to the second target task, wherein the task information corresponding to the second target task includes task attribute information, task name, task identifier, and start time. and task status, etc., the task attribute information is related to the establishment of the second target task, and the task status can be used to indicate whether the second target task has been executed or not.

In the embodiment of the present application, after establishing the second target task, the target master node device writes the task information of the second target task to the task node corresponding to the second target task under the directory node in the Zookeeper system, so that the target When the master node device is down, the new master node device can obtain the task information of the unfinished second target task from the directory node, so that the second target task can be recovered and the high availability of the distributed system is ensured.

In an optional implementation manner, in this embodiment of the present application, in the process of executing the second target task by the target master node device, after the execution of the second target task is completed, the target master node device may correspond to the second target task The task node is removed from the directory node. When there is a new second target task, the task node corresponding to the new second target task is re-established.

In this implementation manner, the task nodes existing under the directory node are all task nodes corresponding to the unexecuted second target task, so that when the target master node goes down, the new master node device can quickly identify the unexecuted second target task. The target task is convenient for task recovery and continuous execution, which improves the task execution efficiency and high availability of the distributed system.

In practical applications, since the second target task will be relatively large, if a second target task corresponds to a task node, then the second target task is not completed. When the master node device restores the second target task, it must restore the entire second target task, and for the part that the target master node device has already executed, the new target master node device will still re-execute it once, resulting in the distribution of The system repeats operations, which affects the efficiency of task execution. In order to solve this technical problem, in the embodiment of the present application, as shown in FIG. 6 , a new distributed task scheduling method is proposed, and the method includes the following steps:

Step 601, after receiving the task request, establish a second target task according to the task request, and execute the second target task.

Step 602: Divide the second target task into a plurality of target subtasks, and each target subtask is a task of different stages of the second target task.

In this embodiment of the present application, the user can divide the second target task into multiple independently designed target subtasks of independent stages through the target master node device, and each target subtask is a task of different stages of the second target task. Each stage is the smallest granularity supported by a high-availability job.

Step 603: Create a task node corresponding to each target subtask under the directory node according to the execution sequence of each target subtask, and write the task information corresponding to each target subtask into the task node corresponding to each target subtask.

The target master node device may establish multiple task nodes under the directory node, and each task node is used to store task information of a target subtask. Each target subtask sequentially writes the respective task information into the task node corresponding to each target subtask according to the execution sequence.

Optionally, the task information corresponding to the target subtask may include information such as a second target task identifier, a target subtask identifier, execution time, and execution status.

Optionally, in this embodiment of the present application, each time the target master node device finishes executing a target subtask, the task node corresponding to the target subtask may be deleted from the directory node, so that all task nodes existing under the directory node are not executed. The task node corresponding to the finished target subtask.

Optionally, in this embodiment of the present application, each time the target master node device finishes executing a target subtask, the task status information of the target subtask may be written into the task node that does not have a corresponding target subtask, and the task status information may be It is used to indicate whether the execution of the target subtask has ended. In this way, the task nodes existing under the directory node include task nodes corresponding to target subtasks whose execution has ended, and task nodes corresponding to target subtasks that have not been executed.

Step 604 , in the case that the target master node device is down, the new master node device restores the target subtask that has not been executed and ended from the directory node in the Zookeeper system.

When the target primary node device is down, the distributed system can select a new primary node device from multiple secondary node devices, the new primary node device scans the directory node, and then resumes the target subtasks that have not been executed. and proceed. In this way, the new master node device only needs to restore a part of the target subtasks of the second target task, and does not need to restore and re-execute all the second target tasks, which avoids repeated operations in the distributed system and affects the execution efficiency of the tasks.

In an optional implementation manner, since task nodes corresponding to all target subtasks are established in the Zookeeper system when the second target task is executed, more storage space in the Zookeeper system will be occupied. Another implementation method is proposed to save storage space on the Zookeeper system, including:

The second target task is divided into a plurality of target subtasks arranged in the order of execution, and each target subtask is a task of a different stage of the second target task; the task information corresponding to each target subtask is stored in the target task node; For each target subtask, when the target subtask is executed, a subtask node corresponding to the target subtask is established in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the previous target of the target subtask is deleted. The subtask node corresponding to the subtask.

In the embodiment of the present application, after the second target task is divided into multiple target subtasks, a task node corresponding to each target subtask will not be established in the Zookeeper system, but all task information of all target subtasks will be stored. to the target task node, and can be stored in the order of execution.

Then, execute each target subtask in sequence according to the execution order. When the kth target subtask is executed, the task information corresponding to the kth target subtask can be obtained from the target task node, and the kth target subtask can be established in the Zookeeper system. The subtask node corresponding to the target subtask, and then start to execute the kth target subtask.

In this way, the subtask nodes corresponding to each target subtask are not created at one time, but when the target subtasks are executed sequentially, a subtask node is created for which target subtask is executed, and the subtask node is deleted after the subtask is executed. Task nodes, and so on, execute each target subtask in sequence, thereby saving storage space on the zookeeper system.

Among them, in the embodiment of the present application, after the subtask node corresponding to the kth target subtask is established, delete the previous target subtask of the kth target subtask, that is, the subtask node corresponding to the k-1th target subtask . The k-1 th target subtask is the target subtask that has been executed, so that the target subtask that has been executed can be clearly distinguished to avoid confusion.

In the embodiment of the present application, as shown in FIG. 7 , the embodiment of the present application provides a new distributed task scheduling method, and the method includes the following contents:

Step 701: During the execution of the second target task, task status information of the second target task is acquired until the execution of the second target task ends.

In this embodiment of the present application, during the process of executing the second target task, the target master node device may periodically acquire task status information of the second target task, where the task status information is used to indicate the execution status of the second target task . Optionally, the task status information may include task name, task ID, task stage ID, execution status, task stage start time or task stage end time, wherein the execution status may be executing, or may be execution completed.

Optionally, if the task status information includes the end time of the task phase, it means that the execution of the second target task is completed, and if the task status information does not include the end time of the task phase, it means that the execution of the second target task has not ended.

Step 702: Write the task status information into the task node corresponding to the second target task, so as to determine whether the execution of the second target task ends after the target master node device goes down.

In this embodiment of the present application, the target master node device may write the task status information into the task node corresponding to the second target task. If the target master node device goes down, the new master node device can scan the directory node to determine whether the second target task has finished execution according to the task status information of the second target task in the task node corresponding to the second target task.

In this embodiment of the present application, by writing the task status information corresponding to the second target task into the task node corresponding to the second target task, it is more clearly determined whether the execution of the second target task has ended, so that the target master node device can In the event of downtime, the new master node device can quickly resume the unfinished second target task.

In practical applications, it may happen that the data amount of the task information corresponding to the second target task is relatively large, or the data amount of the task information corresponding to some target subtasks is relatively large. Since the data storage capacity of each task node in the Zookeeper system is limited, if the data volume of the task information exceeds the data storage capacity of the task node, the task information needs to be split.

In this embodiment of the present application, when establishing a task node corresponding to the second target task, the target master node device can detect whether the data volume of the task information corresponding to the second target task is greater than the data volume threshold, and if it is greater than the data volume threshold, establish a task node corresponding to the second target task. Multiple task nodes corresponding to the data amount of the task information of the task, and the multiple task nodes can be used to store the task information corresponding to the same second target task.

Correspondingly, the target master node device can also detect whether the data volume of the task information of each target subtask is greater than the data volume threshold. Multiple task nodes corresponding to the data amount of the task information of the task, and the multiple task nodes can be used to store the task information corresponding to the same target subtask.

However, in practical applications, the target master node device writes task information corresponding to a second task or task information corresponding to a target subtask to different task nodes, which will bring new transactional problems.

The following description will be given by taking the case of writing task information corresponding to a second task to different task nodes as an example. The existing Zookeeper system does not support simultaneous transaction operations on multiple task nodes. Therefore, the target master node device cannot synchronously perform write operations or read operations on multiple task nodes. When the master node device writes the task information of the same second target task, it only executes the write operation to some of the multiple task nodes corresponding to the second target task, and does not execute the write operation to the other part of the task nodes. In addition, the existing Zookeeper system cannot identify which task nodes perform the write operation and which task nodes do not perform the write operation, so it will cause difficulties for the Zookeeper system to manage the data.

In order to solve the above technical problems, the embodiments of the present application provide an implementation manner that can write to multiple task nodes in the Zookeeper system, as shown in FIG. A flowchart of a method for performing a write operation on a plurality of task nodes, the method includes the following steps:

Step 801, create a transaction mark under the directory node.

In the embodiment of the present application, when the number of task nodes corresponding to the second target task corresponding to the task status information is multiple, the target master node device needs to establish a transaction mark first when writing the target status information, wherein the transaction mark includes The identifiers of the at least two task nodes to which the task status information is to be written; the transaction flag is used to indicate that the write operation is performed synchronously on the at least two task nodes corresponding to the identifiers of the at least two task nodes.

Step 802: Back up the task information in at least two task nodes to obtain backup data.

After the transaction mark is established, the target master node device clearly needs to perform a write operation on at least two task nodes corresponding to the identifiers of the at least two task nodes, so as to avoid missing some task nodes.

It should be noted that the data in the task node under the directory node in the Zookeeper system does not support update operations, only overwrite write operations. In order to avoid abnormal data storage caused by the write operation, in the embodiment of the present application, when the write operation is performed, the original data in the at least two task nodes needs to be saved first, and the backup data is the original data.

Step 803: Write the task status information into at least two task nodes corresponding to the identifiers of the at least two task nodes.

The target master node device may determine the at least two task nodes under the directory node according to the identifiers of the at least two task nodes. Then, write operations are performed to the at least two task nodes one by one.

Step 804, if the writing is successful, delete the backup data and the transaction mark.

In the embodiment of the present application, if the writing operation is performed on at least two task nodes, it indicates that the writing is successful, and if the writing is successful, the backup data is no longer used, and is therefore deleted. And the successful writing indicates that the transaction of writing the task status information to the task node has been completed, so the transaction mark is also deleted.

Step 805, if the writing fails, restore the task information in the at least two task nodes corresponding to the identifiers of the at least two task nodes according to the backup data, and delete the transaction mark.

If the writing fails, it means that the at least two task nodes have not completed the transaction of writing the task status information. In order to ensure the accuracy of the data in the at least two task nodes, the backup data is re-stored in the at least two task nodes , to ensure the accuracy of task information. At the same time, the deletion of the transaction marker indicates the end of the write operation.

In the embodiment of the present application, by establishing a transaction mark and deleting a transaction mark, simultaneous transaction operations on multiple task nodes are implemented, and the atomicity of operations on the Zookeeper system is guaranteed. Solved the problem that the Zookeeper system does not support simultaneous transaction operations on multiple task nodes. Completely supports the highly available job framework based on the Zookeeper system.

It should be understood that although the steps in the flowcharts of FIG. 4 to FIG. 8 are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIG. 4 to FIG. 8 may include multiple steps or multiple stages. These steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. The order of execution is also not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the steps or stages within the other steps.

In one embodiment, as shown in FIG. 9, a distributed task scheduling apparatus 900 is provided, including: a scanning module 901, a reading module 902 and a recovery module 903, wherein:

The scanning module 901 is configured to scan the preset directory node in the Zookeeper system after the conversion from the secondary node device to the target primary node device;

The reading module 902 is configured to read the task information from the at least one task node if there is at least one task node under the directory node; wherein, the at least one task node is the process of the original master node device executing the first target task Created in , the task information is the information related to the execution of the first target task written by the original master node device;

The restoration module 903 is configured to restore the first target task according to the task information, and execute the first target task.

In one of the embodiments, as shown in FIG. 10 , a distributed task scheduling apparatus 1000 is provided, further comprising:

The receiving module 1001 is used to establish a second target task according to the task request after receiving the task request, and execute the second target task;

The establishment module 1002 is used to establish a task node corresponding to the second target task under a preset directory node in the Zookeeper system during the process of executing the second target task, and write the task information corresponding to the second target task in the task node corresponding to the second target task.

In one of the embodiments, the establishment module 1002 is specifically used for:

Divide the second target task into a plurality of target subtasks, and each target subtask is a task of different stages of the second target task;

A task node corresponding to each target subtask is established under the directory node according to the execution sequence of each target subtask, and the task information corresponding to each target subtask is written into the task node corresponding to each target subtask.

In one of the embodiments, the establishment module 1002 is specifically used for:

Divide the second target task into a plurality of target subtasks arranged in order of execution, and each target subtask is a task of different stages of the second target task;

Save the task information corresponding to each target subtask in the target task node;

For each target subtask, when the target subtask is executed, a subtask node corresponding to the target subtask is established in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the previous target of the target subtask is deleted. The subtask node corresponding to the subtask.

In one of the embodiments, the establishment module 1002 is specifically used for:

In the process of executing the second target task, the task status information of the second target task is obtained until the execution of the second target task ends; the task status information is used to represent the execution status of the second target task;

The task status information is written into the task node corresponding to the second target task, so as to determine whether the execution of the second target task ends after the target master node device goes down.

In one of the embodiments, the establishment module 1002 is specifically used for:

A transaction marker is established under the directory node, and the transaction marker includes the identifiers of the at least two task nodes to which the task status information is to be written; the transaction marker is used to indicate that the at least two task nodes corresponding to the identifiers of the at least two task nodes are synchronously executing writing enter operation;

Back up the task information in at least two task nodes to obtain backup data;

The task status information is written into at least two task nodes corresponding to the identifiers of the at least two task nodes.

In one of the embodiments, the establishment module 1002 is specifically used for:

If the writing is successful, the backup data and transaction mark are deleted;

If the writing fails, the task information in the at least two task nodes corresponding to the identifiers of the at least two task nodes is restored according to the backup data, and the transaction mark is deleted.

In one of the embodiments, the establishment module 1002 is specifically used for:

After the execution of the second target task is completed, the task node corresponding to the second target task is deleted from the directory node.

For specific limitations on the distributed task scheduling apparatus, reference may be made to the above limitations on the distributed task scheduling method, which will not be repeated here. Each module in the above-mentioned distributed task scheduling apparatus may be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, including a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when executing the computer program:

After converting from the secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system;

If there is at least one task node under the directory node, read task information from at least one task node; wherein, at least one task node is created by the original master node device in the process of executing the first target task, and the task information is Information related to the execution of the first target task written by the original master node device;

The first target task is restored according to the task information, and the first target task is executed.

In one of the embodiments, the processor may further implement the following steps when executing the computer program:

After receiving the task request, establish a second target task according to the task request, and execute the second target task;

In the process of executing the second target task, a task node corresponding to the second target task is established under the preset directory node in the Zookeeper system, and the task information corresponding to the second target task is written into the second target task corresponding to the second target task in the task node.

In one of the embodiments, the processor may further implement the following steps when executing the computer program:

Divide the second target task into a plurality of target subtasks, and each target subtask is a task of different stages of the second target task;

A task node corresponding to each target subtask is established under the directory node according to the execution sequence of each target subtask, and the task information corresponding to each target subtask is written into the task node corresponding to each target subtask.

In one of the embodiments, the processor may further implement the following steps when executing the computer program:

Divide the second target task into a plurality of target subtasks arranged in order of execution, and each target subtask is a task of different stages of the second target task;

Save the task information corresponding to each target subtask in the target task node;

For each target subtask, when the target subtask is executed, a subtask node corresponding to the target subtask is established in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the previous target of the target subtask is deleted. The subtask node corresponding to the subtask.

In one of the embodiments, the processor may further implement the following steps when executing the computer program:

In the process of executing the second target task, the task status information of the second target task is obtained until the execution of the second target task ends; the task status information is used to represent the execution status of the second target task;

The task status information is written into the task node corresponding to the second target task, so as to determine whether the execution of the second target task ends after the target master node device goes down.

In one of the embodiments, the processor may further implement the following steps when executing the computer program:

A transaction marker is established under the directory node, and the transaction marker includes the identifiers of the at least two task nodes to which the task status information is to be written; the transaction marker is used to indicate that the at least two task nodes corresponding to the identifiers of the at least two task nodes are synchronously executing writing enter operation;

Back up the task information in at least two task nodes to obtain backup data;

The task status information is written into at least two task nodes corresponding to the identifiers of the at least two task nodes.

In one of the embodiments, the processor may further implement the following steps when executing the computer program:

If the writing is successful, the backup data and transaction mark are deleted;

If the writing fails, the task information in the at least two task nodes corresponding to the identifiers of the at least two task nodes is restored according to the backup data, and the transaction mark is deleted.

In one of the embodiments, the processor may further implement the following steps when executing the computer program:

After the execution of the second target task is completed, the task node corresponding to the second target task is deleted from the directory node.

The implementation principles and technical effects of the computer equipment provided in the embodiments of the present application are similar to those of the foregoing method embodiments, and details are not described herein again.

In one embodiment, a computer-readable storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

After converting from the secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system;

If there is at least one task node under the directory node, read task information from at least one task node; wherein, at least one task node is created by the original master node device in the process of executing the first target task, and the task information is Information related to the execution of the first target task written by the original master node device;

The first target task is restored according to the task information, and the first target task is executed.

In one of the embodiments, when the computer program is executed by the processor, the following steps may also be implemented:

After receiving the task request, establish a second target task according to the task request, and execute the second target task;

In the process of executing the second target task, a task node corresponding to the second target task is established under the preset directory node in the Zookeeper system, and the task information corresponding to the second target task is written into the second target task corresponding to the second target task in the task node.

In one of the embodiments, when the computer program is executed by the processor, the following steps may also be implemented:

Divide the second target task into a plurality of target subtasks, and each target subtask is a task of different stages of the second target task;

A task node corresponding to each target subtask is established under the directory node according to the execution sequence of each target subtask, and the task information corresponding to each target subtask is written into the task node corresponding to each target subtask.

In one of the embodiments, when the computer program is executed by the processor, the following steps may also be implemented:

Divide the second target task into a plurality of target subtasks arranged in order of execution, and each target subtask is a task of different stages of the second target task;

Save the task information corresponding to each target subtask in the target task node;

For each target subtask, when the target subtask is executed, a subtask node corresponding to the target subtask is established in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the previous target of the target subtask is deleted. The subtask node corresponding to the subtask.

In one of the embodiments, when the computer program is executed by the processor, the following steps may also be implemented:

In the process of executing the second target task, the task status information of the second target task is obtained until the execution of the second target task ends; the task status information is used to represent the execution status of the second target task;

The task status information is written into the task node corresponding to the second target task, so as to determine whether the execution of the second target task ends after the target master node device goes down.

In one of the embodiments, when the computer program is executed by the processor, the following steps may also be implemented:

A transaction marker is established under the directory node, and the transaction marker includes the identifiers of the at least two task nodes to which the task status information is to be written; the transaction marker is used to indicate that the at least two task nodes corresponding to the identifiers of the at least two task nodes are synchronously executing writing enter operation;

Back up the task information in at least two task nodes to obtain backup data;

The task status information is written into at least two task nodes corresponding to the identifiers of the at least two task nodes.

In one of the embodiments, when the computer program is executed by the processor, the following steps may also be implemented:

If the writing is successful, the backup data and transaction mark are deleted;

If the writing fails, the task information in the at least two task nodes corresponding to the identifiers of the at least two task nodes is restored according to the backup data, and the transaction mark is deleted.

In one of the embodiments, when the computer program is executed by the processor, the following steps may also be implemented:

After the execution of the second target task is completed, the task node corresponding to the second target task is deleted from the directory node.

The implementation principle and technical effect of the computer-readable storage medium provided in this embodiment are similar to those of the foregoing method embodiments, and details are not described herein again.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the various embodiments provided in this application may include at least one of non-volatile and volatile memory. The non-volatile memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, and the like. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, the RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM).

The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, all It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be noted that, for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (10)

1. A distributed task scheduling method, wherein the method comprises: After converting from the secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system; If there is at least one task node under the directory node, read task information from the at least one task node; wherein, the at least one task node is the original master node device in the process of executing the first target task Created, the task information is the information related to the execution of the first target task written by the original master node device; The first target task is restored according to the task information, and the first target task is executed. 2. The method according to claim 1, wherein the method further comprises: After receiving the task request, establish a second target task according to the task request, and execute the second target task; In the process of executing the second target task, a task node corresponding to the second target task is established under the preset directory node in the Zookeeper system, and a task node corresponding to the second target task is created. The task information is written into the task node corresponding to the second target task. 3. The method according to claim 2, wherein the task node corresponding to the second target task is established under the preset directory node in the Zookeeper system, and the The task information corresponding to the second target task is written into the task node corresponding to the second target task, including: The second target task is divided into a plurality of target subtasks, and each of the target subtasks is a task of a different stage of the second target task; A task node corresponding to each target subtask is established under the directory node according to the execution sequence of each target subtask, and the task information corresponding to each target subtask is written into each target subtask in the task node corresponding to the task. 4. The method according to claim 3, characterized in that, in the process of executing the second target task, establishing a relationship with the second directory node in the Zookeeper system under the preset directory node The task node corresponding to the target task, and writing the task information corresponding to the second target task into the task node corresponding to the second target task, including: The second target task is divided into a plurality of target subtasks arranged in the order of execution, and each of the target subtasks is a task of different stages of the second target task; saving the task information corresponding to each of the target subtasks in the target task node; For each target subtask, when executing the target subtask, a subtask corresponding to the target subtask is established in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node. task node, and delete the subtask node corresponding to the previous target subtask of the target subtask. 5. The method according to claim 2, wherein the method further comprises: In the process of executing the second target task, the task status information of the second target task is acquired until the execution of the second target task ends; the task status information is used to indicate the execution status of the second target task ; The task status information is written into the task node corresponding to the second target task, so as to determine whether the execution of the second target task ends after the target master node device goes down. 6. The method according to claim 5, wherein the writing the task status information into the task node corresponding to the second target task comprises: A transaction marker is established under the directory node, the transaction marker includes the identifiers of at least two task nodes to which the task status information is to be written; the transaction marker is used to indicate the identifiers of the at least two task nodes The corresponding at least two task nodes perform the write operation synchronously; Backing up the task information in the at least two task nodes to obtain backup data; Writing the task status information into the at least two task nodes corresponding to the identifiers of the at least two task nodes. 7. The method according to claim 6, wherein the method further comprises: If the writing is successful, delete the backup data and the transaction mark; If the writing fails, the task information in the at least two task nodes corresponding to the identifiers of the at least two task nodes is restored according to the backup data, and the transaction flag is deleted. 8. A distributed task scheduling apparatus, wherein the apparatus comprises: The scanning module is used to scan the preset directory nodes in the Zookeeper system after converting from the secondary node device to the target primary node device; A reading module, configured to read task information from the at least one task node if there is at least one task node under the directory node; wherein, the at least one task node is the original master node device executing the first Created in the process of a target task, the task information is the information related to the execution of the first target task written by the original master node device; A restoring module, configured to restore the first target task according to the task information, and execute the first target task. 9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the method according to any one of claims 1 to 7 when the processor executes the computer program. step. 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images