CN112181621A

CN112181621A - Task scheduling system, method, equipment and storage medium

Info

Publication number: CN112181621A
Application number: CN202011034859.6A
Authority: CN
Inventors: 史振华
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2020-09-27
Filing date: 2020-09-27
Publication date: 2021-01-05
Anticipated expiration: 2040-09-27
Also published as: CN112181621B

Abstract

The embodiment of the invention discloses a task scheduling system, a method, equipment and a storage medium. The system comprises: the system comprises a timing scheduling module, a task scanning and distributing module, a task preprocessing module, a task executing module and a task queue; the timing scheduling module is used for adding the unprocessed tasks meeting the timing scheduling conditions into the task queue; the task scanning and distributing module is used for scanning unprocessed tasks in the task queue and distributing the tasks meeting the task distributing condition to the task preprocessing module; the task preprocessing module is used for arranging and executing the tasks meeting the task arranging conditions in the distributed unprocessed tasks and submitting the arranged tasks to the task executing module; and the task execution module is used for executing the successfully acquired scheduled tasks according to the matched execution steps. The system realizes effective management of the tasks, avoids repeated execution of the tasks and avoids task blockage.

Description

Task scheduling system, method, equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of software, in particular to a task scheduling system, method, equipment and storage medium.

Background

At present, many business processes are automatically processed at a specified time, such as financial statement generation at the end of a month, or the transmission of preferential information before traditional festivals.

In the prior art, two ways are generally adopted for performing the timing-triggered scheduling execution of the service processing. The first mode is that a crontab command in the Linux system is adopted to configure timing trigger for each task execution script program, and a cron process in the Linux system can schedule and execute each task execution script program according to a preset timing trigger rule. The second approach is to use a Quartz framework to integrate the timing scheduling of tasks into the application system. Quartz is an open source scheduling framework, provides various scheduling methods, supports scheduling of various granularities, supports various storage modes, and has certain distributed clustering capability.

However, the first method cannot effectively manage and monitor the timing task, and the task is repeatedly executed in a distributed environment. The second method lacks effective management of task concurrency and flow control, and when a large number of tasks are to be executed in a certain time period, the tasks are easily seriously blocked.

Disclosure of Invention

Embodiments of the present invention provide a task scheduling system, method, device, and storage medium, which can effectively manage tasks, avoid repeated execution of tasks, and avoid task congestion.

In a first aspect, an embodiment of the present invention provides a task scheduling system, where the system includes: the system comprises a timing scheduling module, a task scanning and distributing module, a task preprocessing module, a task executing module and a task queue;

the timing scheduling module is used for adding the unprocessed tasks meeting the timing scheduling conditions into the task queue;

the task scanning and distributing module is used for scanning unprocessed tasks in the task queue and distributing the tasks meeting the task distributing condition to the task preprocessing module;

the task preprocessing module is used for arranging and executing the tasks meeting the task arranging conditions in the distributed unprocessed tasks and submitting the arranged tasks to the task executing module;

and the task execution module is used for executing the successfully acquired scheduled tasks according to the matched execution steps.

In a second aspect, an embodiment of the present invention further provides a task scheduling method, where the method includes:

adding unprocessed tasks meeting the timing scheduling conditions into a task queue through a scheduling module;

scanning unprocessed tasks in the task queue through a task scanning and distributing module, and distributing the tasks meeting task distribution conditions to a task preprocessing module;

arranging and executing steps for tasks meeting task arrangement conditions in the distributed unprocessed tasks through a task preprocessing module, and submitting the arranged tasks to a task executing module;

and executing the successfully acquired scheduled tasks according to the matched execution steps through the task execution module.

In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a method of task scheduling as in any embodiment of the invention.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a task scheduling method according to any embodiment of the present invention.

According to the technical scheme of the embodiment of the invention, the task scanning and distributing module and the task preprocessing module are arranged between the timing scheduling module and the task execution module to perform timing scheduling, scanning, distributing, preprocessing and executing on unprocessed tasks in the task queue, so that the problem of task scheduling is solved, the effective management on the tasks is realized, the repeated execution of the tasks is avoided, and the effect of task blocking is avoided.

Drawings

FIG. 1 is a schematic structural diagram of a task scheduling system according to an embodiment of the present invention;

FIG. 2 is a flowchart of the operation of a timing scheduling module provided by an embodiment of the present invention;

FIG. 3 is a flowchart of the task scanning distribution module according to an embodiment of the present invention;

FIG. 4 is a flowchart of the task pre-processing module provided by an embodiment of the present invention;

FIG. 5 is a flowchart of the task execution module provided by an embodiment of the present invention;

FIG. 6 is a flowchart of a task scheduling method according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Fig. 1 is a schematic structural diagram of a task scheduling system according to an embodiment of the present invention, which is applicable to a situation where a large number of tasks are scheduled and batched at regular time, as shown in fig. 1, the system includes: a timing scheduling module 110, a task scanning distribution module 120, a task preprocessing module 130, a task execution module 140, and a task queue 150.

Specifically, the timing scheduling module 110 is configured to add the unprocessed task meeting the timing scheduling condition to the task queue 150.

The timing scheduling module 110 can be implemented based on a Quartz framework, which is an open-source task scheduling system completely developed by Java and can be responsible for executing (notifying) other software component systems at a predetermined time. The Quartz framework may provide rich scheduling methods, such as scheduling tasks according to calendar or time rules. The Quartz framework may include a task scheduler, triggers, and job triplets.

The number of flip-flops may be one or more. A trigger is an element that defines the scheduling time, i.e. according to what time rule a task is to be executed. The trigger may be one of SimpleTrigger, calendarrintervaltrigger, or CronTrigger. The SimpleTrigger can specify that a task is triggered at a certain time interval (unit is millisecond) from a certain time; the CalendarIntervalTrigger can specify that tasks are executed at certain time intervals (the unit can be seconds, minutes, hours, days, months, years or weeks) from a certain time; CronTrigger is suitable for more complex tasks and supports syntax similar to cron expression for Linux systems.

Jobs are used to represent scheduled tasks and are classified into two categories, stateful and stateless. For the same trigger, the jobs with states cannot be executed in parallel, and only after the task triggered last time is executed, the next execution can be triggered. The stateless operation means that when the trigger condition is satisfied, the task is triggered to execute. In this embodiment, the timing scheduling module may adopt stateful operations, so as to avoid that when the task is not completely executed, the task is triggered again when the next trigger time arrives, and the task is repeatedly executed. Jobs may be persisted to a database store, which may be a relational database table. Persistence may ensure that data is not lost, e.g., when a task scheduling system exits abnormally, and when the task scheduling system is restarted, jobs are not lost.

The task scheduler is a controller for actually executing scheduling, the task scheduler can poll all triggers through scheduling threads, and when the triggering time of the task is up, idle threads are acquired from the task execution thread pool to execute the tasks related to the triggers. Alternatively, the task scheduler may scan all triggers to see if there is a task that misses the trigger time, and if so, process the task according to the missed trigger policy. In the prior art, a task is directly executed by a task scheduler of a Quartz frame, but in the embodiment, the task scheduler of the Quartz frame does not directly execute the task but adds the task into the task queue 150, so that the task can be effectively controlled, and the problem that the task is seriously blocked due to limited Quartz frame resources when a large number of tasks to be executed exist in a certain time period is avoided.

In this embodiment, the timing scheduling condition may include a trigger time condition and/or a traffic trigger condition. The trigger time condition may refer to that the task is triggered according to a preset time rule, for example, a data report is sent to the client every morning. The service triggering condition may refer to that the task is triggered according to a preset service detection rule, for example, when it is determined that the account of the client changes, a data report is sent to the client. And various triggering modes of the tasks can be met through the timing scheduling conditions. The preset time rule and the service detection rule of the task can be stored in the relational data table, a persistent storage mode can be adopted, the corresponding relation between the task and the timing scheduling condition can be established, and data loss when the task scheduling system is abnormal can be avoided.

Optionally, the timing scheduling module is specifically configured to: when the unprocessed task reaches the trigger time, determining whether a listener exists in the unprocessed task; when the fact that the listener does not exist in the unprocessed task is determined, the fact that the unprocessed task meets a timing scheduling condition is determined; when the fact that the unprocessed task exists in the monitor is determined, whether the unprocessed task meets a service triggering condition or not is judged through the monitor according to a preset detection rule corresponding to the unprocessed task; when the unprocessed task is determined to meet the service triggering condition, determining that the unprocessed task meets the timing scheduling condition; otherwise, determining that the unprocessed task does not meet the timing scheduling condition.

Each unprocessed task may have a corresponding preset time rule, and when the processing task satisfies the corresponding preset time rule, it may be determined that the unprocessed task reaches the trigger time. For example, the preset time rule for an unprocessed task is 6 pm per saturday, and when the current time is 6 pm on saturday, the trigger time to reach the unprocessed task is determined.

The listener can be added on a trigger of the Quartz framework and can be used for detecting the service of the unprocessed task. If the unprocessed task does not need to carry out service detection, a listener can not be set for the unprocessed task, and the unprocessed task can be determined to meet the timing scheduling condition when the trigger time of the unprocessed task is reached. And if the unprocessed task needs to carry out service detection, a listener can be set for the unprocessed task. The listener can judge whether the unprocessed task meets the triggering condition according to a preset detection rule corresponding to the unprocessed task. And when the triggering time of the unprocessed task is reached and the service triggering condition corresponding to the unprocessed task is met, determining that the unprocessed task meets the timing scheduling condition. A layer of service requirement triggering can be added for triggering the task on the basis of timing requirement triggering, the triggering form of the task is enriched, and the requirements of various application scenes are met.

For example, when the current time is 8 o' clock on saturday night and the account amount of the customer changes, it may be determined that the unprocessed task sending the account amount change data report to the customer satisfies the timing scheduling condition. The unprocessed task may be added to the task queue 150. The tasks in the task queue 150 may have a task state, which may be determined based on the execution of the tasks. For example, when the timing scheduling module 110 adds an unprocessed task to the task queue 150, the task state of the unprocessed task may default to "not executed". The task state may be stored in a database.

Optionally, the timing scheduling module is further specifically configured to: configuring task parameters of unprocessed tasks through an API (application programming interface); wherein the task parameters comprise the monitoring packet condition of the unprocessed task; and when the unprocessed task is in the snooping packet corresponding to the snooper, determining that the unprocessed task exists in the snooper.

An Application Programming Interface (API) Interface is disposed on the timing scheduling module, and task parameters of the unprocessed task, such as a job status, a preset time rule, a preset detection rule, and other parameters of the unprocessed task, may be dynamically configured through the API Interface. The task parameters configured through the API interface can be stored in a relational database table in a persistent mode. The task can be dynamically configured through the API, and the situation that the Quartz framework can only configure the task by modifying a static configuration file can be avoided. The static configuration needs to be configured and then the task scheduling system is restarted, the relevant task parameters can take effect, and the configuration of the task parameters is carried out through the API interface, so that the configuration can be updated without restarting the task scheduling system after the configuration. In addition, the database table can be customized by configuring the task parameters through the API, the problem that only the database table defaulted by the Quartz frame can be adopted when the Quartz frame is adopted can be avoided, the database table defaulted by the Quartz frame can be simplified, a plurality of database tables are not needed, complex access operation of the database table is avoided, and development, operation and maintenance are facilitated.

To facilitate setting up a listener for an unprocessed task, the task parameters may include a snoop packet condition for the unprocessed task. When the unprocessed task needs to perform traffic detection, the unprocessed task may be divided into snoop packets. A listener can be set for unprocessed tasks in the snoop packet to perform service detection.

Fig. 2 is a flowchart of a timing scheduling module according to an embodiment of the present invention, and as shown in fig. 2, when the timing scheduling module is started, scheduling may be initialized, for example, task scheduling rules (including preset time rules and preset detection rules, etc.) corresponding to unprocessed tasks in a database table may be packaged as triggers and jobs of a Quartz framework, and a listener is added to a trigger corresponding to an unprocessed task that needs to perform service detection. The scheduling thread of the timing scheduling module can poll all triggers, when the triggering time is determined to be reached and the service triggering condition is met (the condition is not needed to be judged when the service detection is not needed), a task request is initiated, an unprocessed task is initialized to be unexecuted and persisted to a database table; otherwise, all triggers continue to be polled. The timing scheduling module of the embodiment can realize the timing scheduling and the service scheduling of the tasks, can meet the dynamic real-time configuration of the task parameters, does not need a large number of database tables, and is convenient for the development, operation and maintenance of the system.

As shown in fig. 1, in the present embodiment, the task scanning and distributing module 120 is configured to scan unprocessed tasks in the task queue 150 and distribute the tasks meeting the task distribution condition to the task preprocessing module 130.

The tasks may all be buffered in the task queue, that is, the tasks in different task states may exist in the task queue. The task scan distribution module 120 may scan the unprocessed tasks in the task queue 150 whose task status is "not executing". The manner of scanning may be to poll the task queue 150. The task scan distribution module 120 may distribute the scanned "unexecuted" tasks to the task pre-processing module 130. Specifically, the scanned "unexecuted" tasks may be distributed to the respective preprocessing threads of the task preprocessing module 130 in an evenly distributed manner.

Optionally, the task scanning and distributing module is specifically configured to: and acquiring the task concurrency corresponding to the unprocessed task, and distributing the task meeting the task concurrency to the task preprocessing module.

The task concurrency degree may be the concurrency degree of different classified tasks, for example, for task a1, task a2, and task A3 may all belong to a task classified as a; corresponding to task B1, task B2 may belong to a task classified as B. The task classified in the category a may have task concurrency, the task classified in the category B may also have task concurrency, and the task concurrency of the two may be the same or different. For example, the task concurrency of the a classification task is 2, and the task concurrency of the B classification task is 5. When a task satisfies a corresponding task concurrency, the task may be distributed to the task pre-processing module 130. For example, the number of currently executed tasks of the class a task is 1, which is smaller than the task concurrency degree of the class a task, and the class a task can be scanned and distributed.

Optionally, the task parameters include: task numbering, task classification and task priority; the task scanning and distributing module is specifically configured to: determining task grouping results and sequencing results of unprocessed tasks according to task numbers, task classifications and task priorities; and distributing the tasks meeting the task concurrency degree to a task preprocessing module according to the task grouping result and the sequencing result.

Here, the task number is a number that each task has, and examples thereof include a1, a2, A3, B1, and B2. Task classification refers to the classification of individual tasks, e.g., a1, a2, A3 belong to task classification a; b1, B2 belong to task category B. The task priority may be a priority of the task classification or may be a priority of each task.

Unprocessed tasks may be grouped and ordered according to task number, task classification, and task priority. For example, tasks corresponding to task numbers belonging to the same task category may be grouped; the grouped tasks are sorted according to a priority order, for example, according to a descending order of priority. Wherein, the tasks in the same group can have the same priority and carry out the in-group sequencing according to the task number; alternatively, tasks in the same group may have different priorities, with the ordering within the group being by priority. For another example, tasks corresponding to task numbers belonging to the same priority may be grouped into one group; and sorting the grouped tasks according to a priority order, wherein the sorting order in the group can be determined according to task classification and/or task numbers.

In this embodiment, the task priority may be a priority that the task classification has, for example, a is higher than B. Optionally, the task scanning and distributing module is further specifically configured to: and determining a task grouping result of the unprocessed tasks according to the task numbers and the task classification, and determining a sequencing result of the unprocessed tasks after the tasks are grouped according to the task priority.

Tasks corresponding to task numbers belonging to the same task classification can be divided into a group; the grouped tasks are sorted according to a priority order, for example, according to a descending order of priority. For example, group a1, group a2, group a 2; dividing B1 and B2 into B groups; and (4) sorting the group A and the group B according to the descending order of the priority to generate a sorting result of the group A and the group B.

Specifically, the tasks meeting the task concurrency are distributed to the task preprocessing module according to the task grouping result and the sequencing result, whether each task group meets the corresponding task concurrency or not can be judged, and if yes, the tasks in the task group can be distributed to the task preprocessing module. For example, the tasks of the group B are currently executed by 3 tasks, the task concurrency degree corresponding to the group B is 5, and the currently executed number is smaller than the task concurrency degree, so that the tasks of the group B can be scanned and distributed to the task preprocessing module.

Optionally, the task scanning and distributing module is further specifically configured to: and when the total number of tasks in the scheduled tasks and the execution is smaller than the task concurrency degree corresponding to the task group, determining that the unprocessed tasks in the task group meet the task concurrency degree, and recording the task group as a target task group.

The scheduled tasks can mean that the execution steps of the tasks are scheduled by the task preprocessing module, the task preprocessing module can modify the task state of the scheduled tasks into 'to be executed', and the task state can be stored in a database table in a persistent mode. The task in execution may mean that the task execution module has already executed the task, but has not yet finished executing the task, and the task execution module may modify the task state to "in execution".

The task state is that the tasks to be executed and in execution already occupy the task resources, and in order to avoid the problems of task resource shortage and task blockage caused by the fact that the task scanning and distributing module continues to scan and distribute the tasks under the condition that the task resources are already saturated, whether the total number of the tasks to be executed and in execution is smaller than the task concurrency degree corresponding to the task group can be judged. If so, the task packet may be scanned.

For example, for the task in group a, it may be determined that the total number of tasks in group a that are "to be executed" and "in execution" is 1 and is less than the task concurrency degree 2 of group a, it is determined that the unprocessed task in group a satisfies the task concurrency degree, group a may be marked as a target task group, and the task in group a that is "not executed" may be scanned. When the task scanning is performed on the group a, the task scanning may be performed according to the task concurrency of the group a, for example, the task concurrency of the group a is 2, the total number of tasks "to be executed" and "in execution" is 1, and 2-1 — 1 "tasks" of the group a that are not executed "may be scanned. The concurrency and the flow of the tasks can be effectively controlled, and the task blockage is avoided.

Optionally, the task scanning and distributing module is further specifically configured to: acquiring a preset task grouping scanning threshold before distributing tasks meeting task concurrency to a task preprocessing module according to task grouping results and sequencing results; and scanning unprocessed tasks in the target task groups which are not greater than a preset task group scanning threshold value in the task queue, and distributing the scanned unprocessed tasks to a task preprocessing module.

The preset task group scanning threshold may be an upper limit of the task group that can be scanned by the task scanning distribution module at the same time. The preset task group scanning threshold value can be stored in a database table, and the task scanning distribution module can read the configuration information to obtain the preset task group scanning threshold value in the database table. When the task scanning and distributing module scans each task group, only the target task group meeting the task concurrency degree can be scanned, and for the task group not meeting the task concurrency degree, the task scanning and distributing module can choose to skip during the scanning. The number of the target task groups scanned by the task scanning and distributing module can be smaller than any value of a preset task group scanning threshold value. The task scanning distribution module can distribute unprocessed tasks in the scanned target task group to the task preprocessing module.

When the task scanning and distributing module distributes the tasks, the tasks can be distributed to each preprocessing thread of the task preprocessing module in an average distribution mode. The task priority may be considered when distributing. Illustratively, the task scanning and distributing module scans task groups, and the task groups are sorted into A, B, C and D according to task priorities. There are 2 preprocessing threads for the task preprocessing module. The task scanning and distributing module can distribute the A and the C to the preprocessing thread 1 and distribute the B and the D to the preprocessing thread 2, so that task groups with high task priority can be processed preferentially. For example, A is processed before C and B is processed before D.

Fig. 3 is a flowchart of a task scanning and distributing module according to an embodiment of the present invention, and as shown in fig. 3, the task scanning and distributing module may poll a task queue to scan a task packet of an acquired task. Judging whether unprocessed tasks in an unexecuted state exist in each task group or not, and if not, continuing to poll the task queue for scanning; if the task group exists, judging whether the task group of the unprocessed task with the non-execution state meets the task concurrency degree, for example, judging whether the total number of the tasks to be executed and not executed in the task group is less than the task concurrency degree. If the task grouping does not meet the task concurrency degree, continuing to poll the task queue for scanning; if the task group meets the task concurrency degree, acquiring an unprocessed task list of 'unexecuted' required to be processed, for example, acquiring an unprocessed task list of 'unexecuted' in the task group meeting the task concurrency degree. And distributing the tasks in the unprocessed task list to the preprocessing thread (1, 2, … n), and continuously polling the task queue for scanning after waiting for the preprocessing thread to return. The task scanning and distributing module in this embodiment can perform scheduling management of task priority, concurrency and flow control on the tasks in the task queue, and can avoid task congestion.

As shown in fig. 1, the task preprocessing module 130 is configured to schedule execution steps for tasks satisfying the task scheduling condition among the distributed unprocessed tasks, and submit the scheduled tasks to the task execution module 140.

The task scheduling condition may be a condition that the "unexecuted" unprocessed task received by the preprocessing thread of the task preprocessing module 130 can be processed by the preprocessing thread. In this embodiment, the task preprocessing module 130 is specifically configured to: receiving an unprocessed task through at least one preprocessing thread, and applying for task resources for the received unprocessed task in a mutual exclusion locking mode; and when the unprocessed task applies for the task resource successfully, determining that the unprocessed task meets the task arranging condition.

The task pre-processing module 130 may include one or more pre-processing threads, each of which may pre-process a task in parallel. When the preprocessing thread receives the unprocessed tasks in the "unexecuted" state, the unprocessed tasks may be preprocessed according to the receiving order of the unprocessed tasks (i.e., the task priority order). When the preprocessing thread performs preprocessing, task resources can be applied for each unprocessed task first. The task resources may be devices needed to perform the task, e.g., the task resources may be device nodes in a distributed cluster. In actual task execution, each task group may occupy multiple task resources, while each unprocessed task in the "not executed" state in the task group generally occupies only one task resource.

For example, task group a may occupy one task resource in task resource I and task resource II, and unprocessed tasks a1, a2 in the "not executed" state in task group a may occupy one task resource, for example, a1 occupies task resource I and a2 occupies task resource II, respectively.

The required task resources are typically different for different task groups, and therefore no limitation on task resources between different task groups may be imposed. However, the task resources required by the tasks in the same task group are generally the same, and therefore, the task resources of the unprocessed tasks in the unexecuted state of the same task group can be limited. For task grouping, the amount of available task resources may be equal to the task concurrency. The preprocessing thread can apply for task resources for unprocessed tasks in an 'unexecuted' state of each task group in a mutually exclusive locking mode.

When the task resource application is successful, the number of the task resources can be reduced by 1, and the task state of the unprocessed task corresponding to the successful application of the task resources can be modified into 'to be executed'. When the task resource application fails, the number of the task resources may be kept unchanged, the task state of the unprocessed task corresponding to the task resource application failure may be kept as "unexecuted", and the next scanning and distributing operation of the task scanning and distributing module may be waited for, where the specific process is the same as the foregoing process, and is not described again. The number of the task resources can be recorded in a database table and can correspond to the task grouping.

In this embodiment, when the task resource application is successful, it may be determined that the unprocessed task satisfies the task scheduling condition, and the task state may be modified to "to be executed". For the task of "waiting to execute", the task preprocessing module can perform the arrangement of the task execution steps. The arrangement of the task execution steps can be various, for example, each task can have information related to the task execution steps, and the arrangement of the execution steps can be performed according to the information related to the execution steps; alternatively, an execution template of the task may be determined according to task parameters, such as task number or task classification, and the execution steps may be arranged for the task according to the execution template.

Specifically, in this embodiment, the task preprocessing module is further specifically configured to: when the unprocessed task is determined to have the corresponding execution template, taking the execution step of the execution template as the execution step arranged for the unprocessed task through the preprocessing thread; when the unprocessed task is determined to have no corresponding execution template, arranging execution steps for the unprocessed task through the preprocessing thread according to step parameter information in task parameters of the unprocessed task, and taking the arranged execution steps as the execution template corresponding to the unprocessed task; the task parameters are configured by the timing scheduling module through an API (application program interface).

The execution template may be a template of execution steps set in advance for different task groups, or may be a template of execution steps set for a task group to which a task belongs according to step parameter information of a certain task. The execution template or step parameter information can be configured through an API (application programming interface) of the timing scheduling module and stored in a database table.

When the task preprocessing module performs task step arrangement for the task to be executed, the configuration information can be read first, and whether an execution template corresponding to the task group to which the task to be executed belongs exists in the database table or not is obtained. If the corresponding execution template exists, the preprocessing thread may use the execution step in the execution template corresponding to the "to-be-executed" task as the execution step of the "to-be-executed" task. If the corresponding execution template does not exist, the preprocessing thread can read the configuration information first, obtain the step parameter information in the task parameters of the task to be executed in the database table, and arrange and execute the step for the task to be executed according to the step parameter information. After the scheduling is finished, the preprocessing thread may use the execution steps of the tasks to be executed as the execution templates of the task groups to which the tasks to be executed belong.

It should be noted that, if the execution step of the task packet needs to be adjusted, the updated step parameter information of the task in the task packet may be configured through the API interface of the timing scheduling module. When the preprocessing thread acquires the execution template of the task, whether the step parameter information of the task is updated or not can be judged. If there is an update, the execution step may be orchestrated and an execution template may be created according to the updated step parameter information. If no updates exist, execution steps may continue as determined by the execution template.

In this embodiment, when the preprocessing thread schedules the execution step for the task, the execution step may be performed according to a preset step scheduling rule. Specifically, the task preprocessing module is further configured to: and arranging and executing steps for the unprocessed tasks according to step parameter information in the task parameters and a preset step arranging rule.

The preset step arrangement rule may be a rule including a plurality of step sequences. The preset step arrangement rule can be configured through an API (application programming interface) of the timing scheduling module and stored in a database table. Illustratively, the preset step arrangement rule includes three symbols, for example, "#", "; "and", "and". The three symbols may define the concurrent and sequential execution of the steps. Wherein, "#" may indicate that steps on either side of a symbol are performed concurrently; "; "may indicate that the steps on either side of the symbol are performed in a left-to-right order; "," may indicate that the steps in the concurrent set of steps are performed sequentially from left to right. The priority of the three symbols can be sequentially from high to low; "," # "and", ".

Illustratively, the execution template is "1" for a certain task; 2# 4; 3; 5, 6#7 "wherein the numbers indicate the step numbers, the execution template indicates the meaning: firstly, executing the step 1; then, concurrently executing the step 2 and the step 4; step 3 is executed after the step 2 and the step 4 are executed; and finally, concurrently executing a concurrent step set and a step 7, wherein the concurrent step set is composed of the step 5 and the step 6, and the execution sequence of the concurrent step set is that the step 5 is executed first and then the step 6 is executed.

The task preprocessing module can submit the scheduled tasks to the task execution module, and can submit the execution templates and the task parameters of the tasks when the tasks are submitted. The task parameters may include external parameters such as the source time and organization of the task data. Illustratively, the task classification is a calculated interest, which may include tasks of calculated interest on different dates or by different institutions. For example, task group A is the individual tasks that calculate interest. A1 in task group A is the interest of X organization calculating 8 months and 20 days; a2 is the interest of the X organization for 8 months and 21 days; a3 is the interest of Y organization for 8 months and 30 days; a4 is the interest in calculating the Y institution on day 31/8. For a1, a2, A3 and a4, all the interest calculation steps are the same, i.e. one execution template can be used, but there is a difference in the data sources when interest calculation is performed, and the information of the data sources can be used as external parameters, such as time of source and organization. The external parameters can also be configured through an API interface and stored in a database table.

Fig. 4 is a work flow diagram of the task preprocessing module according to the embodiment of the present invention, and as shown in fig. 4, a preprocessing thread in the task preprocessing module receives an "unexecuted" task distributed by the task scanning and distributing module, obtains a task record from a task queue according to a task number of the "unexecuted" task, and applies for a task resource for the "unexecuted" task. When the task resource application fails, the current task of the preprocessing thread is finished, the task state of the current task which is not executed is kept unchanged, the next scanning and distribution of the task scanning and distribution module is waited, and the preprocessing thread starts to process the next task. When the task resource application is successful, the preprocessing process updates the task state in the task queue to be 'to be executed', determines whether the task has a corresponding execution template, and the execution template can be stored in the memory or the database table. When it is determined that the task has the corresponding execution template, the pre-processing thread may further determine whether the execution step of the task is updated. If the execution steps of the task are updated or it is determined that the corresponding execution template does not exist in the task, the preprocessing thread can arrange the execution steps of the task according to the step parameter information of the task and create the execution template. If the execution template of the task is not updated or the execution template is created, the preprocessing thread can submit the execution template and the task parameters of the task to the task execution module. If the task is successfully submitted, the current task of the preprocessing thread is finished, and the next task can be processed. If the submission fails, the preprocessing thread can modify the task state of the task into 'unexecuted', release the task resources, end the processing of the current task and process the next task. The task and processing module of the embodiment can apply for task resources for the task, perform arrangement of execution steps on the task, and control parallelism of the task, thereby facilitating the normative execution of the task.

As shown in fig. 1, the task execution module 140 is configured to execute the successfully acquired scheduled tasks according to the matched execution steps.

The task execution module 140 may perform asynchronous execution and management of tasks based on a Spring Batch framework, and may perform Batch processing on the tasks. There may be one or more threads of execution for the task execution module 140. The pre-processing thread may submit the scheduled task of "waiting to be executed" to the thread pool of the task execution module 140, and the execution thread acquires the task from the thread pool and modifies the task state of the task to "executing". The task execution template 140 determines a specific execution step of the task according to the execution template of the task, executes the task according to task parameters (such as time of data source, mechanism, etc.) of the task, and records the execution condition of the task step. The execution of the task steps may be persisted to a database table.

The task execution module 140 may determine the task state according to the task execution condition, and optionally, in this embodiment, the task execution module is specifically configured to: and determining the task state of the scheduled task according to the execution condition of the scheduled task, and releasing the task resource.

When the task is successfully executed, the execution thread of the task execution module 140 may modify the task state of the task to "successfully executed"; when the task fails to execute, the execution thread of the task execution module 140 may modify the task state of the task to "execution failed". When the execution of the execution thread is finished, the task resources can be released, the execution of the current task is finished, and the execution of the next task is carried out.

It should be noted that there may be an emergency situation in the execution of the task, for example, the task resource may not be released when the information of the task queue is still in the situation before the interruption, which is caused by a system or hardware abnormality such as a server power failure. In this embodiment, the task scheduling system may be restarted, and after the restart, the task state may be forcibly modified to "execution failure" through the API interface, and the task resource may be released. For the task of 'execution failure', whether rescheduling is needed or not can be determined in a manual intervention mode, and configuration can be carried out through an API (application programming interface).

In addition, it should be further noted that operations such as suspension and validation of the task can be configured through the API interface, and breakpoint execution of the task can also be configured through the API interface, and when the execution mode corresponding to the task is breakpoint execution, the execution module 140 needs to adopt a step-by-step execution mode when executing the task, which is convenient for maintenance and debugging, and can be applied to various application scenarios.

Fig. 5 is a work flow diagram of a task execution module according to an embodiment of the present invention, and as shown in fig. 5, an execution thread of the task execution module may modify a task state of a scheduled task that is "to be executed" to "in execution", determine an execution step of the task according to a task execution template, execute the task according to a task parameter and the execution step of the task, and record an execution situation of the task step. And updating the task state into 'execution success' or 'execution failure' according to the task execution condition. And when the task execution is finished, releasing the task resources, finishing the current task execution of the execution thread, and executing the next task. The task execution module of this embodiment can Batch process the task through the Spring Batch frame, wherein, the Spring Batch frame is a lightweight comprehensive Batch frame, and can be used to develop Batch processing services of data in the enterprise information system.

According to the technical scheme of the embodiment of the invention, the task scanning and distributing module and the task preprocessing module are arranged between the timing scheduling module and the task execution module to perform timing scheduling, scanning, distributing, preprocessing and executing on unprocessed tasks in the task queue, so that the problem of task scheduling is solved, the effective management on the tasks is realized, the repeated execution of the tasks is avoided, and the effect of task blocking is avoided. The timing scheduling module adopts a Quartz frame, the task execution module adopts a Spring Batch frame, and the boundary of timing scheduling and asynchronous scheduling is opened through the task queue, the task scanning and distributing module and the task preprocessing module, so that the timing task can be executed in one step in a Batch processing mode, the concurrency and flow control management of the task is supported, and the arrangement of the task steps and the monitoring of the task execution condition are supported.

Fig. 6 is a flowchart of a task scheduling method according to an embodiment of the present invention. The embodiment is applicable to the case of performing timing scheduling and batch processing on a large number of tasks, the method may be performed by a task scheduling system, the system may be implemented by software, and/or hardware, and the system may be integrated in an electronic device, as shown in fig. 6, and the method specifically includes:

and step 610, adding the unprocessed tasks meeting the timing scheduling conditions into a task queue through a scheduling module.

And step 620, scanning the unprocessed tasks in the task queue through the task scanning and distributing module, and distributing the tasks meeting the task distributing condition to the task preprocessing module.

Step 630, through the task preprocessing module, arranging and executing steps for tasks meeting the task arranging conditions among the distributed unprocessed tasks, and submitting the arranged tasks to the task executing module.

And step 640, executing the successfully acquired scheduled tasks according to the matched executing step through the task executing module.

Optionally, adding the unprocessed task meeting the timing scheduling condition to the task queue, including:

when the unprocessed task reaches the trigger time, determining whether a listener exists in the unprocessed task;

when the fact that the listener does not exist in the unprocessed task is determined, the fact that the unprocessed task meets a timing scheduling condition is determined;

when the fact that the unprocessed task exists in the monitor is determined, whether the unprocessed task meets a service triggering condition or not is judged through the monitor according to a preset detection rule corresponding to the unprocessed task;

when the unprocessed task is determined to meet the service triggering condition, determining that the unprocessed task meets the timing scheduling condition;

otherwise, determining that the unprocessed task does not meet the timing scheduling condition.

Optionally, determining whether a listener exists for the unprocessed task includes:

configuring task parameters of unprocessed tasks through an API (application programming interface); wherein the task parameters comprise the monitoring packet condition of the unprocessed task;

and when the unprocessed task is in the snooping packet corresponding to the snooper, determining that the unprocessed task exists in the snooper.

Optionally, scanning the unprocessed task in the task queue, and distributing the task meeting the task distribution condition to the task preprocessing module, including:

and acquiring the task concurrency corresponding to the unprocessed task, and distributing the task meeting the task concurrency to the task preprocessing module.

Optionally, the task parameters include: task numbering, task classification and task priority;

the method for acquiring the task concurrency corresponding to the unprocessed task and distributing the task meeting the task concurrency to the task preprocessing module comprises the following steps:

determining task grouping results and sequencing results of unprocessed tasks according to task numbers, task classifications and task priorities;

and distributing the tasks meeting the task concurrency degree to a task preprocessing module according to the task grouping result and the sequencing result.

Optionally, determining a task grouping result and an ordering result of the unprocessed task according to the task number, the task classification and the task priority includes:

and determining a task grouping result of the unprocessed tasks according to the task numbers and the task classification, and determining a sequencing result of the unprocessed tasks after the tasks are grouped according to the task priority.

Optionally, distributing the tasks meeting the task concurrency to the task preprocessing module according to the task grouping result and the sorting result, including:

and when the total number of tasks in the scheduled tasks and the execution is smaller than the task concurrency degree corresponding to the task group, determining that the unprocessed tasks in the task group meet the task concurrency degree, and recording the task group as a target task group.

Optionally, before distributing the tasks meeting the task concurrency to the task preprocessing module according to the task grouping result and the sorting result, the task scheduling method further includes:

acquiring a preset task group scanning threshold;

and scanning unprocessed tasks in the target task groups which are not greater than a preset task group scanning threshold value in the task queue, and distributing the scanned unprocessed tasks to a task preprocessing module.

Optionally, in the distributed unprocessed task, the step of scheduling and executing the task meeting the task scheduling condition includes:

receiving an unprocessed task through at least one preprocessing thread, and applying for task resources for the received unprocessed task in a mutual exclusion locking mode;

and when the unprocessed task applies for the task resource successfully, determining that the unprocessed task meets the task arranging condition.

when the unprocessed task is determined to have the corresponding execution template, taking the execution step of the execution template as the execution step arranged for the unprocessed task through the preprocessing thread;

when the unprocessed task is determined to have no corresponding execution template, arranging execution steps for the unprocessed task through the preprocessing thread according to step parameter information in task parameters of the unprocessed task, and taking the arranged execution steps as the execution template corresponding to the unprocessed task;

wherein, the task parameters are configured by the timing scheduling module through an application programming API interface.

Optionally, the scheduling and executing step for the unprocessed task by the preprocessing thread according to the step parameter information in the task parameter of the unprocessed task includes:

and arranging and executing steps for the unprocessed tasks according to step parameter information in the task parameters and a preset step arranging rule.

Optionally, executing the successfully acquired scheduled task according to the matching executing step includes:

and determining the task state of the scheduled task according to the execution condition of the scheduled task, and releasing the task resource.

The task scheduling method provided by the embodiment of the invention corresponds to the work flow of the task scheduling system provided by the embodiment of the invention, and has corresponding functions and beneficial effects.

Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 7, the electronic device includes:

one or more processors 410, one processor 410 being exemplified in FIG. 7;

a memory 420;

the apparatus may further include: an input device 430 and an output device 440.

The processor 410, the memory 420, the input device 430 and the output device 440 of the apparatus may be connected by a bus or other means, for example, in fig. 7.

The memory 420 serves as a non-transitory computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to a task scheduling method in the embodiment of the present invention (for example, the timing scheduling module 110, the task scanning and distributing module 120, the task preprocessing module 130, the task execution module 140, and the task queue 150 shown in fig. 1). The processor 410 executes various functional applications and data processing of the computer device by executing the software programs, instructions and modules stored in the memory 420, namely, a task scheduling method for implementing the above method embodiments is realized, that is:

The memory 420 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the computer device, and the like. Further, the memory 420 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 420 may optionally include memory located remotely from processor 410, which may be connected to the terminal device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 430 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the computer apparatus. The output device 440 may include a display device such as a display screen.

An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a task scheduling method according to an embodiment of the present invention:

Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A task scheduling system, comprising: the system comprises a timing scheduling module, a task scanning and distributing module, a task preprocessing module, a task executing module and a task queue;

2. The system of claim 1, wherein the timing scheduling module is specifically configured to:

when it is determined that the listener does not exist for the unprocessed task, determining that the unprocessed task satisfies the timing scheduling condition;

when the unprocessed task is determined to exist in the listener, judging whether the unprocessed task meets a service triggering condition or not through the listener according to a preset detection rule corresponding to the unprocessed task;

3. The system of claim 2, wherein the timing scheduling module is further configured to:

configuring task parameters of the unprocessed task through an application programming API (application programming interface); wherein the task parameter comprises a monitoring packet condition of the unprocessed task;

and when the unprocessed task is in the monitoring packet corresponding to the monitor, determining that the unprocessed task exists in the monitor.

4. The system of claim 3, wherein the task scanning distribution module is specifically configured to:

5. The system of claim 4, wherein the task parameters comprise: task numbering, task classification and task priority;

the task scanning and distributing module is specifically configured to:

determining task grouping results and sequencing results of the unprocessed tasks according to the task numbers, the task classifications and the task priorities;

and distributing the tasks meeting the task concurrency degree to the task preprocessing module according to the task grouping result and the sequencing result.

6. The system of claim 5, wherein the task scanning distribution module is further configured to:

7. The system according to claim 5 or 6, wherein the task scanning distribution module is further configured to:

and when the total number of tasks in the process of arranging and executing tasks is less than the task concurrency degree corresponding to the task group, determining that the unprocessed tasks in the task group meet the task concurrency degree, and recording the task group as a target task group.

8. The system of claim 7, wherein the task scanning distribution module is further configured to:

acquiring a preset task grouping scanning threshold before distributing the tasks meeting the task concurrency degree to the task preprocessing module according to the task grouping result and the sequencing result;

scanning the unprocessed tasks in the target task group which are not greater than the preset task group scanning threshold value in the task queue, and distributing the scanned unprocessed tasks to the task preprocessing module.

9. The system of claim 1, wherein the task preprocessing module is specifically configured to:

receiving the unprocessed task through at least one preprocessing thread, and applying for task resources for the received unprocessed task in a mutual exclusion locking mode;

10. The system of claim 9, wherein the task preprocessing module is further configured to:

when determining that the unprocessed task has a corresponding execution template, taking the execution step of the execution template as the execution step arranged for the unprocessed task through the preprocessing thread;

when it is determined that the unprocessed task does not have the corresponding execution template, arranging execution steps for the unprocessed task through the preprocessing thread according to step parameter information in task parameters of the unprocessed task, and taking the arranged execution steps as the execution template corresponding to the unprocessed task;

wherein the task parameters are configured by the timing scheduling module through an application programming API interface.

11. The system of claim 10, wherein the task preprocessing module is further configured to:

12. The system of claim 9, wherein the task execution module is specifically configured to:

13. A method for task scheduling, comprising:

14. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a method of task scheduling as claimed in claim 13.

15. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out a method for task scheduling according to claim 13.