CN114090198A - Distributed task scheduling method and device, electronic equipment and storage medium - Google Patents
Distributed task scheduling method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114090198A CN114090198A CN202111256508.4A CN202111256508A CN114090198A CN 114090198 A CN114090198 A CN 114090198A CN 202111256508 A CN202111256508 A CN 202111256508A CN 114090198 A CN114090198 A CN 114090198A
- Authority
- CN
- China
- Prior art keywords
- task
- executed
- scheduling
- registered
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a distributed task scheduling method, a distributed task scheduling device, electronic equipment and a storage medium, wherein the distributed task scheduling method comprises the following steps: reading a task to be executed and service registration information based on a packaged development kit, and registering the task to be executed based on the service registration information; triggering the registered task to be executed, and adding the task to be executed into a scheduling queue; and based on a scheduler, performing fragment scheduling on the tasks to be executed which are registered in the scheduling queue, and adding the tasks to be executed into an actuator. The invention provides a distributed task scheduling method, a distributed task scheduling device, electronic equipment and a storage medium, which can overcome the defect that the prior art does not support the active automatic registration and discovery of distributed tasks, realize the automatic registration and discovery of the distributed tasks and improve the efficiency of the distributed task scheduling.
Description
Technical Field
The present invention relates to the field of information technologies, and in particular, to a distributed task scheduling method and apparatus, an electronic device, and a storage medium.
Background
With the development of the internet, the number of online systems is increasing, and the complexity of the system is increased while the system enabling service is developed. Often, some business logic needs to be triggered regularly, for example, a business data report of the previous day needs to be calculated regularly in the morning; and regularly triggering some service content synchronization, such as periodical commodity information synchronization and the like. At present, a timing task solution existing in the market does not support active automatic registration and discovery of distributed tasks, tasks can only be added manually, development is time-consuming and labor-consuming, task configuration needs to be changed manually when server nodes are changed, and efficiency is low.
Disclosure of Invention
The invention provides a distributed task scheduling method, a distributed task scheduling device, electronic equipment and a storage medium, which are used for overcoming the defect that the prior art does not support the active automatic registration and discovery of distributed tasks, realizing the automatic registration and discovery of the distributed tasks and improving the distributed task scheduling efficiency.
The invention provides a distributed task scheduling method, which comprises the following steps:
reading a task to be executed and service registration information based on a packaged development kit, and registering the task to be executed based on the service registration information;
triggering the registered task to be executed, and adding the task to be executed into a scheduling queue;
and based on a scheduler, performing fragment scheduling on the tasks to be executed which are registered in the scheduling queue, and adding the tasks to be executed into an actuator.
According to the distributed task scheduling method provided by the invention, the triggering of the registered task to be executed and the joining of the task to be executed into the scheduling queue comprises the following steps:
the registered task to be executed is triggered regularly based on a first thread and/or a second thread, and the task to be executed is added into the scheduling queue;
the first thread acquires the registered to-be-executed task based on a polling mode and adds the registered to-be-executed task into a hash queue; and the second thread acquires the registered task to be executed based on a preset frequency.
The distributed task scheduling method provided by the invention further comprises the following steps:
and after the registered task to be executed is executed based on the executor, sending a corresponding task execution result to an operation interface for displaying.
The distributed task scheduling method provided by the invention further comprises the following steps:
and acquiring server node information, and updating the service registration information based on new server node information under the condition that the server node information changes.
The distributed task scheduling method provided by the invention further comprises the following steps:
acquiring the repeated execution times of the corresponding failed task under the condition that the execution of the registered task to be executed fails based on the task execution result;
and under the condition that the repeated execution times do not reach the preset limit times, re-executing the failed task.
According to the distributed task scheduling method provided by the invention, the re-executing the failed task comprises the following steps:
after the registered task to be executed fails to be executed and a target time length is passed, adding the failed task to the executor based on the scheduler;
wherein the size of the target duration is positively correlated with the number of repeated executions.
According to the distributed task scheduling method provided by the invention, the method for performing fragment scheduling on the tasks to be executed registered in the scheduling queue based on the scheduler and adding the tasks to be executed into the executor comprises the following steps:
performing heartbeat detection on the server node;
and based on the scheduler, the registered tasks to be executed are dispatched to the server nodes with successful heartbeat detection in a fragmentation mode, and the server nodes are added into the executor to carry out fragmentation dispatching.
The invention also provides a distributed task scheduling device, comprising:
the system comprises a registration module, a service registration module and a task execution module, wherein the registration module is used for reading a task to be executed and service registration information based on a packaged development toolkit and registering the task to be executed based on the service registration information;
the trigger module is used for triggering the registered tasks to be executed and adding the tasks to the scheduling queue;
and the scheduling module is used for performing fragment scheduling on the tasks to be executed which are registered in the scheduling queue based on the scheduler and adding the tasks to be executed into the executor.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of any one of the distributed task scheduling methods.
The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the distributed task scheduling method as described in any of the above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of the distributed task scheduling method as described in any one of the above.
According to the distributed task scheduling method, the distributed task scheduling device, the electronic equipment and the storage medium, the to-be-executed tasks and the service registration information are read through the packaged development toolkit, the to-be-executed tasks can be registered, the registered to-be-executed tasks are triggered and added into the scheduling queue, and when the tasks need to be executed, the tasks in the scheduling queue are scheduled in a slicing mode and added into the actuator. The distributed task scheduling method can completely realize tasks such as task reading, registration, triggering and scheduling, and does not need manual scheduling. Therefore, the distributed task scheduling method provided by the invention can overcome the defect that the active automatic registration and discovery of the distributed tasks are not supported in the prior art, realize the automatic registration and discovery of the distributed tasks and improve the distributed task scheduling efficiency.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a flow chart of a distributed task scheduling method according to the present invention;
FIG. 2 is a second flowchart of a distributed task scheduling method according to the present invention;
FIG. 3 is a schematic structural diagram of a distributed task scheduling apparatus provided in the present invention;
fig. 4 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following describes a distributed task scheduling method, apparatus, electronic device and storage medium according to the present invention with reference to fig. 1 to 4.
The invention provides a distributed task scheduling method which is applied to a server cluster. The distributed task scheduling method comprises the following steps:
and step 110, reading the task to be executed and the service registration information based on the packaged development toolkit, and registering the task to be executed based on the service registration information.
It is understood that the Development Kit may be a JDK (Java Development Kit), and the task to be executed is a logical task that needs to be executed. The service registration information includes task information and server node information for executing the task, such as an IP (Internet Protocol Address) Address and port information of the server node.
The logic code of the active registration task can be encapsulated into a functional module, namely JDK, and the task to be executed is configured with annotation, and the task to be executed configured with annotation can be read by the functional module, so as to implement active registration of the task to be executed.
Registering the task to be executed, which may be to register the task to be executed in the packaged scheduling management module.
The scheduling management module realizes high-concurrency scheduling by relying on multithreading and concurrency capability of a data queue, supports cluster deployment, and realizes task scheduling of CASCENTRAL Authentication Service and optimistic lock of a central Authentication server in a mode of task version number.
The task automatically registers task information to the scheduling management module when the service is started, and the scheduling management module monitors the change of each node of the service through the original monitoring capability of k8s (namely kubernets), automatically manages the fragment node information of the task and realizes the automatic registration and monitoring of the task.
And 120, triggering the registered tasks to be executed, and adding the tasks to a scheduling queue.
It is understood that the join scheduling queue is a scheduling queue in the scheduling management module. Triggering the registered tasks to be executed based on a trigger, wherein the trigger is a trigger component of the task platform, and triggering related tasks to be added into the scheduler for scheduling execution at a point according to a corresponding trigger strategy.
The scheduling queue is also a task scheduling queue, and tasks in the scheduling queue are sequentially executed according to the order of the queue. The scheduling queue is packaged in the scheduling management module, and no additional operation is required to be performed by personnel.
And step 130, based on the scheduler, performing fragment scheduling on the tasks to be executed registered in the scheduling queue, and adding the tasks to be executed into the executor.
It can be understood that the scheduler is a scheduling component of the task scheduling platform, and performs scheduling execution of tasks to corresponding executors according to corresponding scheduling policies and fragmentation information.
The executor is a server node corresponding to each task service, for example, if the service in which the task 1 is located has 3 server nodes, the task 1 has 3 executors. The distributed executor enables highly available scheduling of timed tasks.
The task executors are distributed in each task server node, so that the performance of each node of the server can be fully exerted, the abnormal isolation of the service level is realized, the service abnormality cannot influence each other, and the high availability is realized.
The scheduler and the scheduling management module are two mutually independent functional modules, the scheduler and the scheduling management module are separated and processed, relevant logics are not coupled, and the scheduler and the scheduling management module interact through an RESTful (namely: a design style and a development mode of a network application program) interface.
The relevant calls between the scheduler and the scheduling management module are hidden in respective frames, after the service is started, the register task is automatically triggered to the scheduling management module, the corresponding process is performed on a task developer black box, the developer does not need to care about and realize coding, and the development efficiency is greatly improved.
The scheduler selects corresponding execution nodes according to the scheduling strategy and the fragmentation strategy, namely the server nodes perform fragmentation scheduling, the tasks to be executed are stored in the execution queue of the executor, the concurrent execution of the execution queue can improve the scheduling speed, and the problem that the task scheduling speed is influenced by the time consumed by remote calling is avoided.
The task is executed through the execution thread after being called, the execution supports concurrent execution, the effect of fragment concurrent calling can be achieved even if only one server node is arranged at the server node, the execution efficiency of the task is improved, and time consumption is reduced.
In some embodiments, triggering the registered to-be-executed task to join the scheduling queue includes:
and performing timing trigger on the registered tasks to be executed based on the first thread and/or the second thread, and adding the tasks to be executed into a scheduling queue.
The first thread acquires a registered task to be executed based on a polling mode and adds the registered task to be executed into a Hash queue; and the second thread acquires the registered task to be executed based on the preset frequency.
It can be understood that the timing trigger is to perform task processing through two resident threads, namely, the first thread and the second thread, where one thread acquires an upcoming task in a polling manner, puts the task in a second-level hash queue, and updates the next scheduling time according to a target scheduling policy; and the other thread executes 1 time per second, acquires the task to be triggered at the current moment and puts the task into the scheduling queue.
The target scheduling policy comprises a trigger policy, an expiration policy and a blocking policy.
The trigger strategy includes four types: the first trigger strategy does not actively trigger execution scheduling; triggering a Cron expression of the task by a second trigger strategy; the third trigger strategy is to trigger the task based on a fixed time interval, and the time unit is second; a fourth trigger strategy is to trigger the task based on a fixed delay time, the time unit being seconds.
The expiration policy includes two types: the first is to ignore the schedule if the task is outdated and not scheduled; the second is that the task is expired and not scheduled, which triggers execution once.
The blocking strategy includes two types: the first is that if the task is not executed and the scheduling time is reached, the strategy will continue to trigger the execution of the task; the second is that if the task is not executed and the scheduling time is reached, the strategy ignores the scheduling and does not affect the scheduling of the subsequent tasks.
In some embodiments, the distributed task scheduling method further comprises:
and after the registered task to be executed is executed based on the executor, sending a corresponding task execution result to an operation interface for displaying.
It can be understood that the corresponding task execution result is sent to the operation interface for display, specifically, the corresponding task execution result is called back to the scheduling management module, so that the task execution result is sent to the operation interface for display based on the scheduling management module.
And after the task execution is finished, notifying the scheduling management module of the execution result of the task in a callback mode. Specifically, the task callback is to automatically judge the task execution result in an AOP (Aspect Programming) mode, and does not need the user to care about the callback logic, and the callback is also in an asynchronous mode, so that the service logic is not affected.
Based on the scheduling management module, the task execution result is sent to the operation interface for display, and the task execution result can be sent to a Web (World Wide Web) operation interface for display, so that report statistics of the execution condition and progress display of task execution are provided, and convenience and high efficiency are achieved.
In some embodiments, the distributed task scheduling method further comprises:
and acquiring server node information, and updating the service registration information based on the new server node information under the condition that the server node information changes.
It can be understood that the server node information changes, which may be that the IP address or the port of the server changes, and since the task to be executed is registered based on the service registration information when registering, the IP address and the port in the service registration information need to be updated in time when the server node information changes.
In some embodiments, the distributed task scheduling method further comprises:
acquiring the repeated execution times of the corresponding failed task under the condition of determining that the execution of the registered task to be executed fails based on the task execution result;
and under the condition that the repeated execution times do not reach the preset limit times, re-executing the failed task.
It can be understood that after receiving the task execution result of the callback, the task execution result is used for judging whether the task is successfully executed or not, and if the task is successfully executed, the relevant execution log and the scheduling log are modified to complete the call.
In some embodiments, re-executing the failed task includes:
after the registered task to be executed fails to be executed and the target time length is passed, adding the failed task into the executor based on the scheduler;
wherein the size of the target duration is positively correlated with the number of repeated executions.
It can be understood that, if the task execution is determined to be failed according to the task execution result, the executor may determine a retry strategy configuration of the task, and if the retry number has not reached the limit, may initiate a retry schedule, and add the failed task to the executor based on the scheduler until the retry number of the task is used up or the execution is successful.
The retry strategy includes two kinds: the first is to perform task retry by configuring the retry number, and the task will be retried immediately after failure until success or retry number is completed, for example, 5 times at most. The second is to configure the interval time within the target number, for example, within 5 interval times, and after the task fails, the execution will be retried at intervals until the task is successful or the task retry strategy is completed, and the interval time configuration example is: 5 sec/10 sec/30 sec/1 min/5 min; namely: after the primary task fails to be executed, carrying out primary retry after 5 seconds; if the first retry fails, after the first retry is finished, carrying out second retry after 10 seconds; if the second retry fails, the third retry is carried out 30 seconds after the second retry is finished, and so on until the task retry times are used up or the execution is successful.
In some embodiments, based on the scheduler, the to-be-executed tasks registered in the scheduling queue are scheduled in a fragmentation manner, and are added to the executor, including:
performing heartbeat detection on the server node;
based on the scheduler, the registered tasks to be executed are dispatched to the server nodes with successful heartbeat detection in a fragmentation mode, and then the server nodes are added into the executor to conduct fragmentation dispatching.
It is understood that the scheduling management module may perform the reachability check of the task node at preset time intervals by means of heartbeat detection, for example, perform the reachability check of the task node at 10 minutes as a time interval. If the heartbeat detection of the corresponding server node fails, the task is suspended from being distributed to the server node, and the reliability of task execution is ensured. The heartbeat probe will continue to check the server node thereafter, and if the heartbeat probe returns to normal, the availability of the server node will be restored.
In some embodiments, the flow of the distributed task scheduling method provided by the present invention is shown in fig. 2, the work division between the scheduling management module, the scheduler, and the executor is clear, and the loose coupling of the codes is realized.
In summary, the distributed task scheduling method provided by the present invention includes: reading a task to be executed and service registration information based on the packaged development toolkit, and registering the task to be executed based on the service registration information; triggering the registered task to be executed, and adding the task to be executed into a scheduling queue; and based on the scheduler, performing fragment scheduling on the tasks to be executed registered in the scheduling queue, and adding the tasks to be executed into the executor.
In the distributed task scheduling method provided by the invention, the tasks to be executed can be registered after reading the tasks to be executed and the service registration information, the registered tasks to be executed are triggered and added into the scheduling queue, and when the tasks need to be executed, the tasks in the scheduling queue are scheduled in a slicing mode and added into the actuator.
The distributed task scheduling method can completely realize tasks such as task reading, registration, triggering and scheduling, and does not need manual scheduling. Therefore, the distributed task scheduling method provided by the invention can overcome the defect that the active automatic registration and discovery of the distributed tasks are not supported in the prior art, realize the automatic registration and discovery of the distributed tasks and improve the distributed task scheduling efficiency.
In addition, the invention automatically registers tasks by integrating JDK, realizes code loose coupling, updates actuator information in real time by automatically monitoring the change of nodes, and can ensure flexible processing without modifying configuration.
The task scheduling is realized in a multi-thread mode, so that the problems of task scheduling delay and the like caused by single thread blockage are avoided; the dispatching management module realizes the cluster deployment, achieves the purpose of high availability, avoids the performance problem of lock preemption in a CAS optimistic lock mode, and improves the stability and the execution efficiency.
The following describes the distributed task scheduling apparatus provided by the present invention, and the distributed task scheduling apparatus described below and the distributed task scheduling method described above may be referred to in correspondence with each other.
As shown in fig. 3, the distributed task scheduling apparatus 300 according to the present invention includes: a registration module 310, a triggering module 320, and a scheduling module 330.
The registration module 310 is configured to read a to-be-executed task and service registration information based on the encapsulated development kit, and register the to-be-executed task based on the service registration information.
It will be appreciated that the task to be performed is a logical task that needs to be performed. The service registration information includes task information and server node information for performing the task, such as an IP address and port information of the server node.
The logic code of the active registration task can be encapsulated into a functional module, namely JDK, and the task to be executed is configured with annotation, and the task to be executed configured with annotation can be read by the functional module, so as to implement active registration of the task to be executed.
Registering the task to be executed, which may be to register the task to be executed in the packaged scheduling management module.
The triggering module 320 is configured to trigger the registered task to be executed to join the scheduling queue.
It is understood that the join scheduling queue is a scheduling queue in the scheduling management module. Triggering the registered tasks to be executed based on a trigger, wherein the trigger is a trigger component of the task platform, and triggering related tasks to be added into the scheduler for scheduling execution at a point according to a corresponding trigger strategy.
The scheduling queue is also a task scheduling queue, and tasks in the scheduling queue are sequentially executed according to the order of the queue. The scheduling queue is packaged in the scheduling management module, and no additional operation is required to be performed by personnel.
The scheduling module 330 is configured to perform fragment scheduling on the to-be-executed tasks registered in the scheduling queue based on the scheduler, and add the to-be-executed tasks to the executor.
It can be understood that the scheduler is a scheduling component of the task scheduling platform, and performs scheduling execution of tasks to corresponding executors according to corresponding scheduling policies and fragmentation information.
The executor is a server node corresponding to each task service, for example, if the service in which the task 1 is located has 3 server nodes, the task 1 has 3 executors.
The scheduler and the scheduling management module are two mutually independent functional modules, the scheduler and the scheduling management module are separated and processed, relevant logics are not coupled, and the scheduler and the scheduling management module interact through a RESTful interface.
The relevant calls between the scheduler and the scheduling management module are hidden in respective frames, after the service is started, the register task is automatically triggered to the scheduling management module, the corresponding process is performed on a task developer black box, the developer does not need to care about and realize coding, and the development efficiency is greatly improved.
The scheduler selects corresponding execution nodes according to the scheduling strategy and the fragmentation strategy, namely the server nodes perform fragmentation scheduling, the tasks to be executed are stored in the execution queue of the executor, the concurrent execution of the execution queue can improve the scheduling speed, and the problem that the task scheduling speed is influenced by the time consumed by remote calling is avoided.
The task is executed through the execution thread after being called, the execution supports concurrent execution, the effect of fragment concurrent calling can be achieved even if only one server node is arranged at the server node, the execution efficiency of the task is improved, and time consumption is reduced.
In some embodiments, the triggering module 320 is further configured to add to the scheduling queue based on the first thread and/or the second thread triggering the registered task to be executed in a timed manner.
The first thread acquires a registered task to be executed based on a polling mode and adds the registered task to be executed into a Hash queue; and the second thread acquires the registered task to be executed based on the preset frequency.
It can be understood that the timing trigger is to perform task processing through two resident threads, namely, the first thread and the second thread, where one thread acquires an upcoming task in a polling manner, puts the task in a second-level hash queue, and updates the next scheduling time according to a target scheduling policy; and the other thread executes 1 time per second, acquires the task to be triggered at the current moment and puts the task into the scheduling queue.
The target scheduling policy comprises a trigger policy, an expiration policy and a blocking policy.
The trigger strategy includes four types: the first trigger strategy does not actively trigger execution scheduling; triggering a Cron expression of the task by a second trigger strategy; the third trigger strategy is to trigger the task based on a fixed time interval, and the time unit is second; a fourth trigger strategy is to trigger the task based on a fixed delay time, the time unit being seconds.
The expiration policy includes two types: the first is to ignore the schedule if the task is outdated and not scheduled; the second is that the task is expired and not scheduled, which triggers execution once.
The blocking strategy includes two types: the first is that if the task is not executed and the scheduling time is reached, the strategy will continue to trigger the execution of the task; the second is that if the task is not executed and the scheduling time is reached, the strategy ignores the scheduling and does not affect the scheduling of the subsequent tasks.
In some embodiments, the distributed task scheduler 300 further comprises: and a callback module.
And the callback module is used for sending a corresponding task execution result to the operation interface for displaying after the registered task to be executed is executed based on the executor.
It can be understood that the corresponding task execution result is sent to the operation interface for display, specifically, the corresponding task execution result is called back to the scheduling management module, so that the task execution result is sent to the operation interface for display based on the scheduling management module.
And after the task execution is finished, notifying the scheduling management module of the execution result of the task in a callback mode. Specifically, the task callback is to automatically judge the task execution result in an AOP (Aspect Programming) mode, and does not need the user to care about the callback logic, and the callback is also in an asynchronous mode, so that the service logic is not affected.
Based on the scheduling management module, the task execution result is sent to the operation interface for display, and the task execution result can be sent to a Web (World Wide Web) operation interface for display, so that report statistics of the execution condition and progress display of task execution are provided, and convenience and high efficiency are achieved.
In some embodiments, the distributed task scheduler 300 further comprises: and updating the module.
The updating module is used for acquiring the server node information and updating the service registration information based on the new server node information under the condition that the server node information changes.
It can be understood that the server node information changes, which may be that the IP address or the port of the server changes, and since the task to be executed is registered based on the service registration information when registering, the IP address and the port in the service registration information need to be updated in time when the server node information changes.
In some embodiments, the distributed task scheduler 300 further comprises: the device comprises an acquisition module and an execution module.
The acquisition module is used for acquiring the repeated execution times of the corresponding failed task under the condition that the execution of the registered task to be executed fails based on the task execution result.
And the execution module is used for re-executing the failed task under the condition that the repeated execution times do not reach the preset limit times.
It can be understood that after receiving the task execution result of the callback, the task execution result is used for judging whether the task is successfully executed or not, and if the task is successfully executed, the relevant execution log and the scheduling log are modified to complete the call.
In some embodiments, the execution module is further configured to add the failed task to the executor based on the scheduler after determining that the registered task to be executed fails to execute and the target duration has elapsed.
Wherein the size of the target duration is positively correlated with the number of repeated executions.
It can be understood that, if the task execution is determined to be failed according to the task execution result, the executor may determine a retry strategy configuration of the task, and if the retry number has not reached the limit, may initiate a retry schedule, and add the failed task to the executor based on the scheduler until the retry number of the task is used up or the execution is successful.
The retry strategy includes two kinds: the first is to perform task retry by configuring the retry number, and the task will be retried immediately after failure until success or retry number is completed, for example, 5 times at most. The second is to configure the interval time within the target number, for example, within 5 interval times, and after the task fails, the execution will be retried at intervals until the task is successful or the task retry strategy is completed, and the interval time configuration example is: 5 sec/10 sec/30 sec/1 min/5 min; namely: after the primary task fails to be executed, carrying out primary retry after 5 seconds; if the first retry fails, after the first retry is finished, carrying out second retry after 10 seconds; if the second retry fails, the third retry is carried out 30 seconds after the second retry is finished, and so on until the task retry times are used up or the execution is successful.
In some embodiments, the scheduling module 330 includes: a sounding unit and a scheduling unit.
The detection unit is used for executing heartbeat detection on the server node.
And the scheduling unit is used for scheduling the registered tasks to be executed to the server nodes with successful heartbeat detection in a fragmentation mode based on the scheduler, adding the server nodes into the executor, and performing fragmentation scheduling.
It is understood that the scheduling management module may perform the reachability check of the task node at preset time intervals by means of heartbeat detection, for example, perform the reachability check of the task node at 10 minutes as a time interval. If the heartbeat detection of the corresponding server node fails, the task is suspended from being distributed to the server node, and the reliability of task execution is ensured. The heartbeat probe will continue to check the server node thereafter, and if the heartbeat probe returns to normal, the availability of the server node will be restored.
In summary, the distributed task scheduling apparatus provided in the present invention includes: a registration module 310, a triggering module 320, and a scheduling module 330. The registration module 310 is configured to read a to-be-executed task and service registration information based on the encapsulated development kit, and register the to-be-executed task based on the service registration information. The triggering module 320 is configured to trigger the registered task to be executed to join the scheduling queue. The scheduling module 330 is configured to perform fragment scheduling on the to-be-executed tasks registered in the scheduling queue based on the scheduler, and add the to-be-executed tasks to the executor.
The distributed task scheduling device provided by the invention can register the tasks to be executed after reading the tasks to be executed and the service registration information, trigger the registered tasks to be executed, add the tasks to the scheduling queue, and schedule the tasks in the scheduling queue in a slicing mode and add the tasks to the executor when the tasks need to be executed.
The distributed task scheduling device can completely realize tasks such as reading, registering, triggering and scheduling, and does not need manual scheduling. Therefore, the distributed task scheduling method provided by the invention can overcome the defect that the active automatic registration and discovery of the distributed tasks are not supported in the prior art, realize the automatic registration and discovery of the distributed tasks and improve the distributed task scheduling efficiency.
In addition, the invention automatically registers tasks by integrating JDK, realizes code loose coupling, updates actuator information in real time by automatically monitoring the change of nodes, and can ensure flexible processing without modifying configuration.
The task scheduling is realized in a multi-thread mode, so that the problems of task scheduling delay and the like caused by single thread blockage are avoided; the dispatching management module realizes the cluster deployment, achieves the purpose of high availability, avoids the performance problem of lock preemption in a CAS optimistic lock mode, and improves the stability and the execution efficiency.
The electronic device, the computer program product, and the storage medium provided by the present invention are described below, and the electronic device, the computer program product, and the storage medium described below and the distributed task scheduling method described above may be referred to in correspondence with each other.
Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor)410, a communication Interface 420, a memory (memory)430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. Processor 410 may invoke logical instructions in memory 430 to perform a distributed task scheduling method comprising:
and step 130, based on the scheduler, performing fragment scheduling on the tasks to be executed registered in the scheduling queue, and adding the tasks to be executed into the executor.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer-readable storage medium, the computer program being capable of executing, when executed by a processor, the distributed task scheduling method provided by the above methods, the method including:
and step 130, based on the scheduler, performing fragment scheduling on the tasks to be executed registered in the scheduling queue, and adding the tasks to be executed into the executor.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium, on which a computer program is stored, the computer program, when being executed by a processor, implementing a distributed task scheduling method provided by the above methods, the method including:
and step 130, based on the scheduler, performing fragment scheduling on the tasks to be executed registered in the scheduling queue, and adding the tasks to be executed into the executor.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A distributed task scheduling method, comprising:
reading a task to be executed and service registration information based on a packaged development kit, and registering the task to be executed based on the service registration information;
triggering the registered task to be executed, and adding the task to be executed into a scheduling queue;
and based on a scheduler, performing fragment scheduling on the tasks to be executed which are registered in the scheduling queue, and adding the tasks to be executed into an actuator.
2. The distributed task scheduling method according to claim 1, wherein the triggering the registered task to be executed to join the scheduling queue comprises:
the registered task to be executed is triggered regularly based on a first thread and/or a second thread, and the task to be executed is added into the scheduling queue;
the first thread acquires the registered to-be-executed task based on a polling mode and adds the registered to-be-executed task into a hash queue; and the second thread acquires the registered task to be executed based on a preset frequency.
3. The distributed task scheduling method of claim 1, further comprising:
and after the registered task to be executed is executed based on the executor, sending a corresponding task execution result to an operation interface for displaying.
4. The distributed task scheduling method of claim 1, further comprising:
and acquiring server node information, and updating the service registration information based on new server node information under the condition that the server node information changes.
5. The distributed task scheduling method of claim 3, further comprising:
acquiring the repeated execution times of the corresponding failed task under the condition that the execution of the registered task to be executed fails based on the task execution result;
and under the condition that the repeated execution times do not reach the preset limit times, re-executing the failed task.
6. The distributed task scheduling method of claim 5, wherein said re-executing the failed task comprises:
after the registered task to be executed fails to be executed and a target time length is passed, adding the failed task to the executor based on the scheduler;
wherein the size of the target duration is positively correlated with the number of repeated executions.
7. The distributed task scheduling method according to any one of claims 1 to 6, wherein the performing, based on the scheduler, the task to be executed that is registered in the scheduling queue is scheduled in a fragmented manner, and the adding to the executor includes:
performing heartbeat detection on the server node;
and based on the scheduler, the registered tasks to be executed are dispatched to the server nodes with successful heartbeat detection in a fragmentation mode, and the server nodes are added into the executor to carry out fragmentation dispatching.
8. A distributed task scheduler, comprising:
the system comprises a registration module, a service registration module and a task execution module, wherein the registration module is used for reading a task to be executed and service registration information based on a packaged development toolkit and registering the task to be executed based on the service registration information;
the trigger module is used for triggering the registered tasks to be executed and adding the tasks to the scheduling queue;
and the scheduling module is used for performing fragment scheduling on the tasks to be executed which are registered in the scheduling queue based on the scheduler and adding the tasks to be executed into the executor.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the distributed task scheduling method according to any one of claims 1 to 7 are implemented when the processor executes the program.
10. A non-transitory computer readable storage medium, having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the distributed task scheduling method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111256508.4A CN114090198A (en) | 2021-10-27 | 2021-10-27 | Distributed task scheduling method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111256508.4A CN114090198A (en) | 2021-10-27 | 2021-10-27 | Distributed task scheduling method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114090198A true CN114090198A (en) | 2022-02-25 |
Family
ID=80298156
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111256508.4A Pending CN114090198A (en) | 2021-10-27 | 2021-10-27 | Distributed task scheduling method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114090198A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114579278A (en) * | 2022-03-08 | 2022-06-03 | 阿维塔科技(重庆)有限公司 | A distributed scheduling method, apparatus, system and computer-readable storage medium |
| CN114610466A (en) * | 2022-03-23 | 2022-06-10 | 杭州隆埠科技有限公司 | Distributed task scheduling system and method |
| CN114968557A (en) * | 2022-04-28 | 2022-08-30 | 中国电力科学研究院有限公司 | Method, system, equipment and medium for scheduling timed tasks in power industry |
| CN115146982A (en) * | 2022-07-15 | 2022-10-04 | 浙江欧菲克斯交通科技有限公司 | Intelligent scheduling control system, method and interface for mobile variable traffic information board |
| CN115220866A (en) * | 2022-06-23 | 2022-10-21 | 中国工商银行股份有限公司 | Distributed task scheduling method and device based on sidecar mode and electronic equipment |
| CN115858118A (en) * | 2022-11-29 | 2023-03-28 | 北京数势云创科技有限公司 | A distributed task scheduling method, system, device and medium |
| CN116126499A (en) * | 2023-01-18 | 2023-05-16 | 中银金融科技有限公司 | Distributed process scheduling parallel processing device and method |
| CN117785431A (en) * | 2024-02-27 | 2024-03-29 | 云粒智慧科技有限公司 | Task scheduling distribution method and device, electronic equipment and storage medium |
| CN118093214A (en) * | 2024-04-26 | 2024-05-28 | 华芯智上半导体设备(上海)有限公司 | Handling task scheduling method and device, electronic equipment and storage medium |
| CN119003136A (en) * | 2024-10-22 | 2024-11-22 | 中博信息技术研究院有限公司 | Timing task scheduling method and system for cluster environment decentralization |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070266390A1 (en) * | 2005-10-31 | 2007-11-15 | Mark Emmerich | Automated management of application-specific tasks from the Internet via distributed task manager agents in a local area network |
| CN106547635A (en) * | 2015-09-18 | 2017-03-29 | 阿里巴巴集团控股有限公司 | A kind of operation of operation retries method and apparatus |
| CN109558239A (en) * | 2018-10-26 | 2019-04-02 | 北京中关村科金技术有限公司 | A kind of method for scheduling task, device, system, computer equipment and storage medium |
| CN112015534A (en) * | 2020-08-27 | 2020-12-01 | 中国平安财产保险股份有限公司 | Configurated platform scheduling method, system and storage medium |
| CN112416581A (en) * | 2020-11-13 | 2021-02-26 | 五八同城信息技术有限公司 | Distributed calling system for timed tasks |
-
2021
- 2021-10-27 CN CN202111256508.4A patent/CN114090198A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070266390A1 (en) * | 2005-10-31 | 2007-11-15 | Mark Emmerich | Automated management of application-specific tasks from the Internet via distributed task manager agents in a local area network |
| CN106547635A (en) * | 2015-09-18 | 2017-03-29 | 阿里巴巴集团控股有限公司 | A kind of operation of operation retries method and apparatus |
| CN109558239A (en) * | 2018-10-26 | 2019-04-02 | 北京中关村科金技术有限公司 | A kind of method for scheduling task, device, system, computer equipment and storage medium |
| CN112015534A (en) * | 2020-08-27 | 2020-12-01 | 中国平安财产保险股份有限公司 | Configurated platform scheduling method, system and storage medium |
| CN112416581A (en) * | 2020-11-13 | 2021-02-26 | 五八同城信息技术有限公司 | Distributed calling system for timed tasks |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114579278A (en) * | 2022-03-08 | 2022-06-03 | 阿维塔科技(重庆)有限公司 | A distributed scheduling method, apparatus, system and computer-readable storage medium |
| CN114610466A (en) * | 2022-03-23 | 2022-06-10 | 杭州隆埠科技有限公司 | Distributed task scheduling system and method |
| CN114968557A (en) * | 2022-04-28 | 2022-08-30 | 中国电力科学研究院有限公司 | Method, system, equipment and medium for scheduling timed tasks in power industry |
| CN115220866A (en) * | 2022-06-23 | 2022-10-21 | 中国工商银行股份有限公司 | Distributed task scheduling method and device based on sidecar mode and electronic equipment |
| CN115146982A (en) * | 2022-07-15 | 2022-10-04 | 浙江欧菲克斯交通科技有限公司 | Intelligent scheduling control system, method and interface for mobile variable traffic information board |
| CN115858118A (en) * | 2022-11-29 | 2023-03-28 | 北京数势云创科技有限公司 | A distributed task scheduling method, system, device and medium |
| CN116126499A (en) * | 2023-01-18 | 2023-05-16 | 中银金融科技有限公司 | Distributed process scheduling parallel processing device and method |
| CN117785431A (en) * | 2024-02-27 | 2024-03-29 | 云粒智慧科技有限公司 | Task scheduling distribution method and device, electronic equipment and storage medium |
| CN117785431B (en) * | 2024-02-27 | 2024-06-04 | 云粒智慧科技有限公司 | Task scheduling distribution method and device, electronic equipment and storage medium |
| CN118093214A (en) * | 2024-04-26 | 2024-05-28 | 华芯智上半导体设备(上海)有限公司 | Handling task scheduling method and device, electronic equipment and storage medium |
| CN119003136A (en) * | 2024-10-22 | 2024-11-22 | 中博信息技术研究院有限公司 | Timing task scheduling method and system for cluster environment decentralization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114090198A (en) | Distributed task scheduling method and device, electronic equipment and storage medium | |
| CN112416581B (en) | Distributed calling system for timed tasks | |
| US8762929B2 (en) | System and method for exclusion of inconsistent objects from lifecycle management processes | |
| US8375244B2 (en) | Managing processing of a computing environment during failures of the environment | |
| US9558459B2 (en) | Dynamic selection of actions in an information technology environment | |
| US8682705B2 (en) | Information technology management based on computer dynamically adjusted discrete phases of event correlation | |
| US8560889B2 (en) | Adding scalability and fault tolerance to generic finite state machine frameworks for use in automated incident management of cloud computing infrastructures | |
| EP2008400B1 (en) | Method, system and computer program for the centralized system management on endpoints of a distributed data processing system | |
| CN108804215B (en) | Task processing method and device and electronic equipment | |
| CN110895488B (en) | Task scheduling method and device | |
| US20090172682A1 (en) | Serialization in computer management | |
| US9712418B2 (en) | Automated network control | |
| CN110895487B (en) | Distributed Task Scheduling System | |
| JPH10214199A (en) | Process restart method and system for realizing process restart | |
| CN110895484A (en) | Task scheduling method and device | |
| CN101777020A (en) | Fault tolerance method and system used for distributed program | |
| CN112131188B (en) | Batch file distribution processing method and device | |
| CN110895486B (en) | Distributed Task Scheduling System | |
| CN107066339A (en) | Distributed job manager and distributed job management method | |
| CN110895485A (en) | task scheduling system | |
| CN106775620B (en) | A timing method and device | |
| US11599387B2 (en) | Re-initiation of microservices utilizing context information provided via service calls | |
| CN111176687A (en) | Method, device, equipment and storage medium for updating cloud host client program | |
| CN117850998A (en) | Distributed task scheduling framework based on Quartz and application thereof | |
| CN111211973B (en) | Information processing method and device in invoice field and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |