CN112667384A - Task flow scheduling method, device, storage medium and electronic device - Google Patents
Task flow scheduling method, device, storage medium and electronic device Download PDFInfo
- Publication number
- CN112667384A CN112667384A CN202011641926.0A CN202011641926A CN112667384A CN 112667384 A CN112667384 A CN 112667384A CN 202011641926 A CN202011641926 A CN 202011641926A CN 112667384 A CN112667384 A CN 112667384A
- Authority
- CN
- China
- Prior art keywords
- task flow
- task
- list
- flow
- scheduler
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a method, a device, a storage medium and an electronic device for scheduling a task flow, wherein the method comprises the following steps: the method comprises the steps of obtaining an execution state of a first task flow in a scheduler, wherein the first task flow is a task flow in a first project, obtaining a first task flow list under the condition that the execution state of the first task flow represents execution completion, wherein the first task flow list is configured to record a second task flow, the second task flow is other task flows which depend on the first task flow in other projects except the first project, and calling the scheduler to execute the second task flow under the condition that the first task flow list comprises the second task flow.
Description
Technical Field
The embodiment of the invention relates to the field of communication, in particular to a task flow scheduling method, a task flow scheduling device, a task flow scheduling storage medium and an electronic device.
Background
In the current related art, the scheduling of task flows can be realized in various ways, such as the azkaban3.25 workflow scheduling task, which supports the interdependence between task flows inside the engineering, but does not support the task flow scheduling between cross-engineering. For example, after each task flow is scheduled, the execution condition of the dependent task flow list needs to be searched, and the task flow can be executed only after all the dependent task flows are completely executed.
Aiming at the technical problem of low execution efficiency of task flow in the related technology, no effective solution is provided at present.
Disclosure of Invention
Embodiments of the present invention provide a method, an apparatus, a storage medium, and an electronic apparatus for scheduling a task flow, so as to at least solve the technical problem of low execution efficiency of the task flow in the related art.
According to an embodiment of the present invention, there is provided a method for scheduling a task flow, including: acquiring an execution state of a first task flow in a scheduler, wherein the first task flow is a task flow in a first project; acquiring a first task flow list when the execution state of the first task flow represents the completion of execution, wherein the first task flow list is configured to record a second task flow, and the second task flow is other task flows which depend on the first task flow in other projects except the first project; and calling the scheduler to execute the second task flow under the condition that the second task flow is included in the first task flow list.
According to another embodiment of the present invention, there is provided a task flow scheduling apparatus including: a first obtaining module, configured to obtain an execution state of a first task flow in a scheduler, where the first task flow is a task flow in a first project; a first obtaining module, configured to obtain a first task flow list when an execution state of the first task flow indicates that execution is completed, where the first task flow list is configured to record a second task flow, and the second task flow is another task flow that depends on the first task flow in another project other than the first project; and the scheduling module is used for calling the scheduler to execute the second task flow under the condition that the first task flow list comprises the second task flow.
According to yet another embodiment of the invention, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps in any of the above method embodiments.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps in any of the above method embodiments when executing the computer program.
By the invention, the executing state of the first task flow in the scheduler is obtained, the first task flow list is obtained under the condition that the executing state of the first task flow represents the completion of the executing, the first task flow list is configured to record the second task flow, the second task flow is other task flows which depend on the first task flow in other projects except the first project, and under the condition that the first task flow list comprises the second task flow, the mode that the scheduler executes the second task flow is called, the task flow which needs to be executed at present is executed first, then other task flows which depend on the task flow which is executed at present are obtained, and other task flows which depend on the task flow which is executed at present are indicated to be executed at the same time, so that the ordered execution of the task flows which have the dependency relationship among the projects is realized, and the technical problems that the executing efficiency of the task flows in the related technology is low and the task flows among the projects are difficult to be scheduled are solved, the technical effects of improving the execution efficiency of the task flow and effectively realizing the scheduling of the cross-engineering task flow are achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a block diagram of a hardware structure of a mobile terminal according to an alternative task flow scheduling method in an embodiment of the present invention;
FIG. 2 is a flowchart illustrating an alternative task flow scheduling method according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an alternative task flow scheduling method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an alternative task flow scheduling method according to an embodiment of the present invention;
fig. 5 is a block diagram of an alternative task flow scheduling apparatus according to an embodiment of the present invention.
Detailed Description
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings in conjunction with the embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking an example of the scheduling method running on a mobile terminal, fig. 1 is a hardware structure block diagram of the mobile terminal of the scheduling method of a task flow according to the embodiment of the present invention. As shown in fig. 1, the mobile terminal may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, wherein the mobile terminal may further include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store a computer program, for example, a software program of an application software and a module, such as a computer program corresponding to the scheduling method of the task flow in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In this embodiment, a method for scheduling task flows running on a mobile terminal, a computer terminal, or a similar computing device is provided, and fig. 2 is a schematic flow chart of an alternative method for scheduling task flows according to an embodiment of the present invention, as shown in fig. 2, the flow includes the following steps:
s202, acquiring an execution state of a first task flow in a scheduler, wherein the first task flow is a task flow in a first project;
s204, under the condition that the execution state of the first task flow represents the completion of the execution, acquiring a first task flow list, wherein the first task flow list is configured to record a second task flow, and the second task flow is other task flows which depend on the first task flow in other projects except the first project;
s206, when the first task flow list comprises the second task flow, the dispatcher is called to execute the second task flow.
Optionally, in this embodiment, the scheduler may include, but is not limited to, an element or a module for scheduling or executing the task flow, and for example, may include, but is not limited to, an executor in an Azkaban task scheduling scenario, or the like.
Optionally, in this embodiment, the task flow may include, but is not limited to, a flow in an Azkaban task scheduling scenario, or a task flow in another task scheduling scenario.
Optionally, in this embodiment, the execution state of the first task flow may include, but is not limited to, not executed, in execution, abnormal execution, complete execution, and the like.
Optionally, in this embodiment, the first task flow list is configured to record the second task flow.
For example, in an Azkaban task scheduling scenario, Azkaban is a batch workflow task scheduler open by Linkedin for running a set of work and processes in a particular order within a workflow. Dynamic state detection during engineering execution is achieved by recording the configured engineering dependencies in mysql.
The job flow is operated in project, that is, the task flow is operated in project, the flow forms project, and the second task flow is other task flows belonging to different projects from the first task flow and depending on the first task flow.
Optionally, in this embodiment, in the case that the second task flow is included in the first task flow list, the invoking the scheduler to execute the second task flow may include, but is not limited to, actively sending a notification message to instruct the scheduler to schedule the second task flow.
Specifically, the second task flow may be added to a queue associated with the scheduler by actively sending a notification message, so as to enable the scheduler to schedule the second task flow based on the queue.
According to the embodiment, the execution state of the first task flow in the scheduler is acquired, the first task flow list is acquired when the execution state of the first task flow indicates that the execution is completed, the scheduler is called to execute the second task flow when the first task flow list comprises the second task flow, and a mode that other task flows depended on by the task flow are searched after the task flow is scheduled and the current task flow required to be executed can be executed after the other task flows are executed is completed in the related art is replaced, so that the technical problem that the scheduling and execution efficiency of the task flow is too low in the related art is solved, cross-engineering dependent scheduling can be added on the basis of the existing azkaban, and the technical effect of improving the execution efficiency of the task flow is achieved.
As an optional scheme, after obtaining the first task flow list, the method further includes: and under the condition that the second task flow is not included in the first task flow list, calling the scheduler to execute a third task flow, wherein the third task flow is other task flows of which the execution sequence in the first project is after the first task flow.
Optionally, in this embodiment, when the first task flow list does not include the second task flow, that is, when the second task flow list is empty, the scheduler is called to execute a third task flow, where the third task flow and the first task flow are task flows included in the same project, and the third task flow is a task flow after the first task flow in the first project in the execution order.
According to the embodiment, the method for executing the third task flow by calling the scheduler under the condition that the first task flow list does not include the second task flow replaces the mode that the task flow which depends on the task flow is retrieved after the task flow is scheduled and the current task flow which needs to be executed can be executed after the other task flows are executed in the related art, solves the technical problem that the scheduling and executing efficiency of the task flow in the related art is too low, can increase cross-engineering dependent scheduling on the basis of the existing azkaban, and achieves the technical effect of improving the executing efficiency of the task flow.
As an optional scheme, after obtaining the execution state of the first task flow in the scheduler, the method further includes: if the execution state represents that the first task flow is abnormally executed, determining the first task flow with the abnormal execution as a fourth task flow; adding the fourth task flow to a second task flow list and saving the fourth task flow to a clustered database.
Optionally, in this embodiment, the case that the execution state indicates that the execution of the first task flow is abnormal may include, but is not limited to, that other task flows depended on by the first task flow are not executed completely, or that the execution condition of the first task flow does not meet a preset requirement, and the like.
Optionally, in this embodiment, the fourth task flow is used to indicate the first task flow whose execution state is an abnormal state, and the second task flow list is used to record the task flow whose execution state is an abnormal state.
Optionally, in this embodiment, the saving of the fourth task stream in the cluster database may include, but is not limited to, saving the fourth task stream in a mysql cluster database, and the mysql cluster data may include, but is not limited to, clustering ignorance in a manner of 1 master and 2 slaves, so as to satisfy a situation that there are fewer write operations but more query operations in the current business requirement.
By the embodiment, when the execution state indicates that the first task flow is abnormal in execution, the first task flow with abnormal execution is determined as the fourth task flow; the method of adding the fourth task flow to the second task flow list and storing the fourth task flow in the cluster database replaces the method of searching other task flows depended on by the task flow after scheduling the task flow and executing the task flow required to be executed currently after the other task flows are executed, solves the technical problem that the scheduling and executing efficiency of the task flow in the related art is too low, can add cross-engineering dependent scheduling on the basis of the existing azkaban, and achieves the technical effect of improving the executing efficiency of the task flow.
As an optional solution, the method further comprises: obtaining retry configuration parameters, wherein the retry configuration parameters comprise polling times and/or polling intervals, and the retry configuration parameters have a corresponding relation with the first project; polling the second task flow list based on the retry configuration parameters to obtain a polling result; and processing the task flow in the second task flow list according to the polling result.
Optionally, in this embodiment, the retry configuration parameter may include, but is not limited to, a polling number and/or a polling interval, where the polling number may include, but is not limited to, a preconfigured number of times that the task flow whose execution state is the abnormal state needs to be re-executed, and the polling period may include, but is not limited to, a time interval that the task flow whose execution state is the abnormal state needs to be re-executed.
Optionally, in this embodiment, the polling result may include, but is not limited to, a retry success or a retry failure, and when the polling result is the retry success, the fourth task flow is used as the first task flow to perform subsequent processing, and when the polling result is the retry failure, the retried number is stored in the cluster database, so as to ensure that the critical data is not lost, thereby achieving a technical effect of improving the task flow execution efficiency.
According to the embodiment, the retry configuration parameters are obtained, the retry configuration parameters comprise polling times and/or polling intervals, and the retry configuration parameters have a corresponding relation with the first project; polling the second task flow list based on the retry configuration parameters to obtain a polling result; the method for processing the task flow in the second task flow list according to the polling result replaces the method in the related art that other task flows depending on the task flow are retrieved after the task flow is scheduled, and the task flow required to be executed currently can be executed after the other task flows are executed, so that the technical problem that the scheduling and executing efficiency of the task flow in the related art is too low is solved, cross-engineering dependent scheduling can be added on the basis of the existing azkaban, and the technical effect of improving the executing efficiency of the task flow is achieved.
As an optional scheme, processing the task flow in the second task flow list according to the polling result includes: when a fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow; acquiring an execution state of the fourth task flow being executed in the scheduler; acquiring a third task flow list under the condition that the execution state of the fourth task flow represents the completion of execution, wherein the third task flow list is configured to record a fifth task flow, and the fifth task flow is other task flows which depend on the fourth task flow in other projects except the first project; and under the condition that the third task flow list comprises the fifth task flow, calling the scheduler to execute the fifth task flow.
Optionally, in this embodiment, the third task flow list is similar to the first task flow list and is configured to record other task flows that depend on a fourth task flow in projects other than the first project, except that the first task flow list records a second task flow corresponding to the first task flow that is directly executed and completed, and the third task flow list records a fifth task flow corresponding to the fourth task flow that includes a retry success of executing the exception history.
According to the embodiment, when the fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow; acquiring the execution state of the fourth task flow being executed in the scheduler; under the condition that the execution state of the fourth task flow indicates that the execution is completed, a way of obtaining a third task flow list replaces a way that the task flow which needs to be executed currently can be executed only after the other task flows are executed after the task flow is scheduled in the related art, so that the technical problem that the scheduling and execution efficiency of the task flow in the related art is too low is solved, cross-engineering dependent scheduling can be added on the basis of the existing azkaban, and the technical effect of improving the execution efficiency of the task flow is achieved.
As an optional scheme, processing the task flow in the second task flow list according to the polling result includes: when a fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow; acquiring an execution state of the fourth task flow being executed in the scheduler; acquiring the retry number of the fourth task flow when the execution state of the fourth task flow indicates abnormal execution, wherein the retry number is determined by the number of times the fourth task flow is saved to a cluster database; continuing to re-execute the fourth task flow if the retry number is less than or equal to the predetermined threshold until the retry number is greater than a predetermined threshold; deleting the fourth task stream if the number of retries is greater than a predetermined threshold.
Optionally, in this embodiment, the manner of retrying the fourth task flow may include, but is not limited to, determining according to a preset threshold of retry times, and the retry times of the fourth task flow may include, but is not limited to, saving in a mysql cluster database, and flexibly adjusting the threshold of retry times according to actual needs of different projects, so that existing services may be more flexibly satisfied without intruding code.
According to the embodiment, when the fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow; acquiring the execution state of the fourth task flow being executed in the scheduler; acquiring the retry times of the fourth task flow under the condition that the execution state of the fourth task flow indicates abnormal execution, wherein the retry times are determined by the times that the fourth task flow is saved to the cluster database; under the condition that the retry number is less than or equal to the preset threshold value, continuing to re-execute the fourth task flow until the retry number is greater than the preset threshold value; under the condition that the retry number is greater than the preset threshold value, a mode of deleting the fourth task flow is replaced by a mode of searching other task flows depended on by the task flow after the task flow is scheduled and executing the task flow required to be executed currently after the other task flows are executed completely in the related technology, the technical problem that the scheduling and executing efficiency of the task flow is too low in the related technology is solved, cross-engineering dependent scheduling can be added on the basis of the existing azkaban, and the technical effect of improving the executing efficiency of the task flow is achieved.
As an optional scheme, after the first task flow with the execution exception is determined as the fourth task flow, the method further comprises: sending an abnormal indication message to a preset account; in response to the exception indication message, adjusting the tasks in the fourth task flow such that the tasks in the fourth task flow are allowed to execute normally.
Optionally, in this embodiment, the preset account may include, but is not limited to, a work mailbox, a communication number, and the like of the maintenance worker, so as to prompt the maintenance worker that the fourth task flow is an abnormally executed task flow, and specifically, may also include, but is not limited to, performing preliminary analysis on the abnormally executed task flow, so as to correspondingly send an analysis result to the preset account.
According to the embodiment, an abnormal indication message is sent to the preset account; responding to the abnormal indication message, adjusting the fourth task flow to enable the fourth task flow to be executed normally, replacing the mode that the task flow required to be executed currently can be executed only after other task flows are executed after the task flow is scheduled in the related art, solving the technical problem that the scheduling and executing efficiency of the task flow in the related art is too low, and increasing cross-engineering dependent scheduling on the basis of the existing azkaban to achieve the technical effect of improving the executing efficiency of the task flow.
The present application is further explained below with reference to a specific example:
fig. 3 is a schematic diagram of a scheduling method of a task flow according to an embodiment of the present invention, and as shown in fig. 3, the flow may include, but is not limited to, the following steps:
s1, the Azkaban project starts to execute at the task submission of the web server (corresponding to the scheduler);
s2, project executes the submitted task in the executor (corresponding to the aforementioned scheduler);
s3, waiting for the current flow (corresponding to the task flow) to finish executing, and finishing the current flow;
s4, searching flow lists of other projects depending on the project;
s5, the Flow list is not empty, and indicates that other engineering flows depending on the Flow exist;
s6, actively notifying the flow of the flow list to execute, and executing the step S2;
s7, if the Flow list is empty, it shows that there is no other project depending on the Flow, the cross-project dependence does not exist, and the execution is performed according to the internal dependence sequence of the project;
s8, if the current flow is abnormal, the mail informs the responsible person of the dependent project to repair the abnormality in time;
s9, adding the failed flow into a retry list, and persisting the flow into a mysql cluster database;
s10, polling the task at regular time, acquiring a flow list needing polling from a retry configuration table, and acquiring polling times and polling interval time custom configuration;
s11, if the retry number is less than the preset number, continuing the retry;
s12, if the retry number is larger than the specified number, the retry flow in the delete configuration will not be retried again next time.
Fig. 4 is a schematic diagram of another task flow scheduling method according to an embodiment of the present invention, and as shown in fig. 4, the flow of the active notification flow list performs the following specific steps:
s402, actively generating a notification message;
s404, sending a flow task to be executed;
s406, adding the information into a to-be-executed information queue (KAFKA);
s408, the task is executed when the data is monitored.
By the method, the task flow scheduling method is high in executable efficiency and low in time delay, all flow lists of other projects are not required to be relied on to be executed completely, then the flow to be executed is executed, and the downstream dependence is executed immediately after the current flow is executed, so that the problems of high dependence on configured flow, invalid waiting and low execution efficiency are solved.
On the other hand, on the premise that the execution efficiency of the task flow scheduling method is greatly improved, the resource utilization rate in unit time is also obviously improved.
In addition, the user-defined configuration time interval and the retry times are used, so that the existing service can be more flexibly met under the condition of not invading codes.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
In this embodiment, a task flow scheduling apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and details of which have been already described are omitted. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a block diagram of a structure of an alternative task flow scheduling apparatus according to an embodiment of the present invention, as shown in fig. 5, the apparatus includes:
a first obtaining module 502, configured to obtain an execution state of a first task flow in a scheduler, where the first task flow is a task flow in a first project;
a second obtaining module 504, configured to obtain a first task flow list when the execution state of the first task flow indicates that execution is completed, where the first task flow list is configured to record a second task flow, and the second task flow is another task flow that depends on the first task flow in another project other than the first project;
a scheduling module 506, configured to invoke the scheduler to execute the second task flow when the second task flow is included in the first task flow list.
As an optional solution, the apparatus is further configured to: after a first task flow list is obtained, under the condition that the first task flow list does not include the second task flow, the scheduler is called to execute a third task flow, wherein the third task flow is other task flows of which the execution sequence in the first project is after the first task flow.
As an optional solution, the apparatus is further configured to: after acquiring the execution state of a first task flow in a scheduler, determining the first task flow with abnormal execution as a fourth task flow when the execution state represents that the first task flow has abnormal execution; adding the fourth task stream to a second task stream list and persisting the fourth task stream into a clustered database.
As an optional solution, the apparatus is further configured to: obtaining retry configuration parameters, wherein the retry configuration parameters comprise polling times and/or polling intervals, and the retry configuration parameters have a corresponding relation with the first project; polling the second task flow list based on the retry configuration parameters to obtain a polling result; and processing the task flow in the second task flow list according to the polling result.
As an alternative, the apparatus is configured to process the task flows in the second task flow list according to the polling result by: when a fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow; acquiring an execution state of the fourth task flow being executed in the scheduler; acquiring a third task flow list under the condition that the execution state of the fourth task flow represents the completion of execution, wherein the third task flow list is configured to record a fifth task flow, and the fifth task flow is other task flows which depend on the fourth task flow in other projects except the first project; and under the condition that the third task flow list comprises the fifth task flow, calling the scheduler to execute the fifth task flow.
As an alternative, the apparatus is configured to process the task flows in the second task flow list according to the polling result by: when a fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow; acquiring an execution state of the fourth task flow being executed in the scheduler; acquiring the retry number of the fourth task flow under the condition that the execution state of the fourth task flow represents abnormal execution, wherein the retry number is determined by the persistence time of the fourth task flow to a cluster database; continuing to re-execute the fourth task flow if the retry number is less than or equal to the predetermined threshold until the retry number is greater than a predetermined threshold; deleting the fourth task stream if the number of retries is greater than a predetermined threshold.
As an optional solution, the apparatus is further configured to: after the first task flow with abnormal execution is determined as a fourth task flow, an abnormal indication message is sent to a preset account; in response to the exception indication message, adjusting the tasks in the fourth task flow such that the tasks in the fourth task flow are allowed to execute normally.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above-mentioned method embodiments when executed.
In the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, acquiring the execution state of a first task flow in the scheduler, wherein the first task flow is a task flow in a first project;
s2, acquiring a first task flow list when the execution state of the first task flow indicates that the execution is completed, wherein the first task flow list is configured to record a second task flow, and the second task flow is other task flows which depend on the first task flow in other projects except the first project;
s3, when the second task flow is included in the first task flow list, the scheduler is called to execute the second task flow.
The computer readable storage medium is further arranged to store a computer program for performing the steps of:
s1, acquiring the execution state of a first task flow in the scheduler, wherein the first task flow is a task flow in a first project;
s2, acquiring a first task flow list when the execution state of the first task flow indicates that the execution is completed, wherein the first task flow list is configured to record a second task flow, and the second task flow is other task flows which depend on the first task flow in other projects except the first project;
s3, when the second task flow is included in the first task flow list, the scheduler is called to execute the second task flow.
In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
In an exemplary embodiment, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
In an exemplary embodiment, the processor may be configured to execute the following steps by a computer program:
s1, acquiring the execution state of a first task flow in the scheduler, wherein the first task flow is a task flow in a first project;
s2, acquiring a first task flow list when the execution state of the first task flow indicates that the execution is completed, wherein the first task flow list is configured to record a second task flow, and the second task flow is other task flows which depend on the first task flow in other projects except the first project;
s3, when the second task flow is included in the first task flow list, the scheduler is called to execute the second task flow.
For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary embodiments, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the various modules or steps of the invention described above may be implemented using a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and they may be implemented using program code executable by the computing devices, such that they may be stored in a memory device and executed by the computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into various integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for scheduling a task flow, comprising:
acquiring an execution state of a first task flow in a scheduler, wherein the first task flow is a task flow in a first project;
acquiring a first task flow list when the execution state of the first task flow represents the completion of execution, wherein the first task flow list is configured to record a second task flow, and the second task flow is other task flows which depend on the first task flow in other projects except the first project;
and calling the scheduler to execute the second task flow under the condition that the second task flow is included in the first task flow list.
2. The method of claim 1, wherein after obtaining the first task flow list, the method further comprises:
and under the condition that the second task flow is not included in the first task flow list, calling the scheduler to execute a third task flow, wherein the third task flow is other task flows of which the execution sequence in the first project is after the first task flow.
3. The method of claim 1, wherein after obtaining the execution state of the first task flow in the scheduler, the method further comprises:
if the execution state represents that the first task flow is abnormally executed, determining the first task flow with the abnormal execution as a fourth task flow;
adding the fourth task flow to a second task flow list and saving the fourth task flow to a clustered database.
4. The method of claim 3, further comprising:
obtaining retry configuration parameters, wherein the retry configuration parameters comprise polling times and/or polling intervals, and the retry configuration parameters have a corresponding relation with the first project;
polling the second task flow list based on the retry configuration parameters to obtain a polling result;
and processing the task flow in the second task flow list according to the polling result.
5. The method of claim 3, wherein processing the task flow in the second task flow list according to the polling result comprises:
when a fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow;
acquiring an execution state of the fourth task flow being executed in the scheduler;
acquiring a third task flow list under the condition that the execution state of the fourth task flow represents the completion of execution, wherein the third task flow list is configured to record a fifth task flow, and the fifth task flow is other task flows which depend on the fourth task flow in other projects except the first project;
and under the condition that the third task flow list comprises the fifth task flow, calling the scheduler to execute the fifth task flow.
6. The method of claim 3, wherein processing the task flow in the second task flow list according to the polling result comprises:
when a fourth task flow is included in the second task flow list, the scheduler is called again to execute the fourth task flow;
acquiring an execution state of the fourth task flow being executed in the scheduler;
acquiring the retry number of the fourth task flow when the execution state of the fourth task flow indicates abnormal execution, wherein the retry number is determined by the number of times the fourth task flow is saved to a cluster database;
continuing to re-execute the fourth task flow if the retry number is less than or equal to the predetermined threshold until the retry number is greater than a predetermined threshold;
deleting the fourth task stream if the number of retries is greater than a predetermined threshold.
7. The method of claim 3, wherein after determining the first task flow that performs the exception as a fourth task flow, the method further comprises:
sending an abnormal indication message to a preset account;
in response to the exception indication message, adjusting the tasks in the fourth task flow such that the tasks in the fourth task flow are allowed to execute normally.
8. A task flow scheduling apparatus, comprising:
a first obtaining module, configured to obtain an execution state of a first task flow in a scheduler, where the first task flow is a task flow in a first project;
a second obtaining module, configured to obtain a first task flow list when an execution state of the first task flow indicates that execution is completed, where the first task flow list is configured to record a second task flow, and the second task flow is another task flow that depends on the first task flow in another project other than the first project;
and the calling module is used for calling the scheduler to execute the second task flow under the condition that the first task flow list comprises the second task flow.
9. A computer-readable storage medium, in which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method as claimed in any of claims 1 to 7 are implemented when the computer program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011641926.0A CN112667384A (en) | 2020-12-31 | 2020-12-31 | Task flow scheduling method, device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011641926.0A CN112667384A (en) | 2020-12-31 | 2020-12-31 | Task flow scheduling method, device, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112667384A true CN112667384A (en) | 2021-04-16 |
Family
ID=75412298
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011641926.0A Pending CN112667384A (en) | 2020-12-31 | 2020-12-31 | Task flow scheduling method, device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112667384A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106293971A (en) * | 2016-08-15 | 2017-01-04 | 张家林 | A kind of method and apparatus of distributed task dispatching |
| CN110362362A (en) * | 2019-07-24 | 2019-10-22 | 北京明略软件系统有限公司 | Method for scheduling task and device, storage medium, electronic device |
| WO2019218708A1 (en) * | 2018-05-15 | 2019-11-21 | 华为技术有限公司 | Task processing method and device, and computer system |
| CN111399840A (en) * | 2020-03-04 | 2020-07-10 | 腾讯音乐娱乐科技（深圳）有限公司 | Module development method and device |
| CN111459629A (en) * | 2020-03-24 | 2020-07-28 | 深圳市云智融科技有限公司 | Azkaban-based project operation method and device and terminal equipment |
| CN111625344A (en) * | 2019-02-27 | 2020-09-04 | 阿里巴巴集团控股有限公司 | Resource scheduling system, method and device in application system |
-
2020
- 2020-12-31 CN CN202011641926.0A patent/CN112667384A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106293971A (en) * | 2016-08-15 | 2017-01-04 | 张家林 | A kind of method and apparatus of distributed task dispatching |
| WO2019218708A1 (en) * | 2018-05-15 | 2019-11-21 | 华为技术有限公司 | Task processing method and device, and computer system |
| CN111625344A (en) * | 2019-02-27 | 2020-09-04 | 阿里巴巴集团控股有限公司 | Resource scheduling system, method and device in application system |
| CN110362362A (en) * | 2019-07-24 | 2019-10-22 | 北京明略软件系统有限公司 | Method for scheduling task and device, storage medium, electronic device |
| CN111399840A (en) * | 2020-03-04 | 2020-07-10 | 腾讯音乐娱乐科技（深圳）有限公司 | Module development method and device |
| CN111459629A (en) * | 2020-03-24 | 2020-07-28 | 深圳市云智融科技有限公司 | Azkaban-based project operation method and device and terminal equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106802826B (en) | Service processing method and device based on thread pool | |
| CN106406993A (en) | Timed task management method and system | |
| CN111274052A (en) | Data distribution method, server, and computer-readable storage medium | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| CN104657212A (en) | Task scheduling method and system | |
| CN111507608B (en) | Work order early warning method and device and storage medium | |
| CN110471746B (en) | Distributed transaction callback method, device and system | |
| CN111026602A (en) | Health inspection scheduling management method and device of cloud platform and electronic equipment | |
| CN110955506A (en) | Distributed job scheduling processing method | |
| CN111090532A (en) | Application service calling method and device, electronic equipment and computer storage medium | |
| CN107577527B (en) | Task generation and scheduling method and device | |
| CN112667383B (en) | Task execution and scheduling method, system, device, computing equipment and medium | |
| CN114020368A (en) | Information processing method and device based on state machine and storage medium | |
| CN108616668B (en) | Method for monitoring dialing task, electronic device and computer readable storage medium | |
| CN110069465A (en) | HDFS data managing method, device, equipment and medium based on workflow | |
| CN114780177B (en) | Transaction execution method and device, storage medium and electronic device | |
| CN115509714A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN112748990A (en) | Quartz-based data quality task execution method and device and computer equipment | |
| CN112667384A (en) | Task flow scheduling method, device, storage medium and electronic device | |
| CN114564249B (en) | Recommendation scheduling engine, recommendation scheduling method and computer readable storage medium | |
| CN114090268B (en) | Container management method and container management system | |
| CN110780869A (en) | Distributed batch scheduling | |
| CN112672369B (en) | Electronic price tag management method, server and computer readable storage medium | |
| CN116303276A (en) | Method for realizing file export by spring batch nested script | |
| CN115712572A (en) | Task testing method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210416 |