CN118394468A - Task scheduling method, system and computing device - Google Patents
Task scheduling method, system and computing device Download PDFInfo
- Publication number
- CN118394468A CN118394468A CN202410458202.4A CN202410458202A CN118394468A CN 118394468 A CN118394468 A CN 118394468A CN 202410458202 A CN202410458202 A CN 202410458202A CN 118394468 A CN118394468 A CN 118394468A
- Authority
- CN
- China
- Prior art keywords
- type
- task
- time
- tasks
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the field of service clusters, in particular to a task scheduling method, a task scheduling system and computing equipment, wherein the task scheduling method comprises the following steps: setting the type of each task node in one or more task nodes, and dividing the first type task nodes into a first number and a second number of first type task nodes, wherein the first type task nodes are suitable for processing real-time tasks, and the second type task nodes are suitable for processing offline tasks; setting a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule; and processing the offline tasks according to the first number of the first-type task nodes and the second-type task nodes, and processing the real-time tasks according to the second number of the first-type task nodes. According to the invention, the first type task node for processing the real-time task and the second type task node for processing the offline task are uniformly scheduled by using one server, so that the management cost of the task nodes is reduced.
Description
Technical Field
The present invention relates to the field of service clusters, and in particular, to a task scheduling method, system and computing device.
Background
The current large data platform for providing service is generally divided into an offline platform and a real-time platform, the offline platform has early origin and is a technology represented by a MapReduce and Spark batch processing operation framework, the data aging is usually T+1, the real-time platform is a technology represented by a Flink technology and a SPARK STREAMING technology which are developed in recent years, and the technology has higher data aging compared with the offline technology and can reach the minute level generally.
The offline platform can be used for processing offline tasks, and the real-time platform can be used for processing real-time tasks. The timeliness required for offline tasks is typically t+1, and the timeliness of real-time tasks can reach the minute level. The offline task and the real-time task are not used in the task types, and the timeliness is lower than that of the real-time task. The task characteristics of the offline tasks and the real-time tasks are different, and one condition is that the number of the offline tasks required to be processed in the daytime is low, and the number of the real-time tasks is high; in contrast, the number of offline tasks that need to be handled during the night is higher, while the number of real-time tasks is lower. It is therefore necessary to employ different platforms, i.e. to handle offline tasks and real-time tasks using an offline platform and a real-time platform, respectively.
The real-time platform or the offline platform each includes a plurality of task nodes, each of which is adapted to receive and process a task. However, when task nodes in the real-time platform and the offline platform are managed, cluster management services are often required to be respectively constructed for the real-time platform and the offline platform, so that maintenance cost and machine cost are increased. And because the characteristics of the offline tasks and the real-time tasks are different, the task processing pressures of the offline platform and the real-time platform are unbalanced in different time periods, so that the computing resources of the whole data platform cannot be fully utilized.
For this reason, a new task scheduling method is required.
Disclosure of Invention
To this end, the present invention provides a task scheduling method in an effort to solve or at least alleviate the above-presented problems.
According to a first aspect of the present invention there is provided a task scheduling method adapted to run in a server in communication with one or more task nodes, the method comprising: setting the type of each task node in one or more task nodes, wherein the types of the task nodes comprise a first type and a second type, the first type task nodes are divided into a first number and a second number of first type task nodes, the first type task nodes are suitable for processing real-time tasks, and the second type task nodes are suitable for processing offline tasks; setting a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule; and processing the offline tasks according to the first number of the first-type task nodes and the second-type task nodes, and processing the real-time tasks according to the second number of the first-type task nodes.
Optionally, in the method according to the present invention, further comprising: setting a first number of first type task nodes to process real-time tasks in response to reaching a second scheduling time specified by a preset time period scheduling rule; and processing real-time tasks according to the first number and the second number of the first type task nodes, and processing offline tasks according to the second type task nodes.
Optionally, in the method according to the present invention, setting the time period scheduling rule includes: setting one or more scheduling time periods included in the daily time, wherein the starting time of each scheduling time period is used as a first scheduling time, and the ending time is used as a second scheduling time; the first type task nodes are arranged at the first scheduling time to schedule the first number of the first type task nodes to process the offline tasks, and the second scheduling time to schedule the first number of the first type task nodes to process the real-time tasks.
Optionally, in the method according to the invention, the method further comprises: determining a minimum number of first type task nodes required for processing the real-time task; setting a task amount control rule according to the minimum number, wherein the task amount control rule comprises setting the second number to be larger than or equal to the minimum number; the first type of task nodes are divided into a first number and a second number of first type of task nodes according to the task volume control rule.
Optionally, in the method according to the present invention, processing the offline task according to the first number of first-type task nodes and the second-type task nodes comprises: submitting one or more offline tasks to a real-time queue and an offline queue according to the queue binding rule; distributing the offline tasks submitted to the real-time queue to a first number of first type task nodes for execution; and distributing the offline tasks submitted to the offline queue to the second type task nodes for execution.
Optionally, in the method according to the present invention, processing the real-time tasks according to the second number of task nodes of the first type comprises: submitting the real-time task to a real-time queue according to the queue binding rule; real-time tasks submitted to the real-time queue are distributed to a second number of first type task nodes for execution.
Optionally, in the method according to the present invention, further comprising: setting a queue binding rule, wherein the queue binding rule comprises: submitting the offline tasks to a real-time queue and an offline queue in a scheduling time period, and submitting the real-time tasks to the real-time queue; and when the real-time task does not belong to the scheduling time period, submitting the offline task to an offline queue, and submitting the real-time task to a real-time queue.
Optionally, in the method according to the present invention, processing the offline task according to the first number of first-type task nodes and the second-type task nodes comprises: and distributing the offline tasks in the real-time queue to a first number of first-type task nodes for execution, and distributing the offline tasks in the offline queue to a second-type task node for execution.
According to a second aspect of the present invention, there is provided a task scheduling system comprising a server and one or more task nodes communicatively connected to the server, the server being adapted to set a type of each of the one or more task nodes, the type of task node comprising a first type and a second type, and to divide the first type of task node into a first number and a second number of first type of task nodes, the first type of task node being adapted to process real-time tasks, the second type of task node being adapted to process offline tasks; setting a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule; the first number of first-type task nodes and the second-type task node are adapted to process offline tasks, and the second number of first-type task nodes are adapted to process real-time tasks.
According to a third aspect of the present invention there is provided a computing device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the task scheduling method according to the present invention.
According to a fourth aspect of the present invention there is provided a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform a task scheduling method according to the present invention.
According to a fifth aspect of the present invention there is provided a computer program product comprising a computer program/instruction which when executed by a processor implements the static library determination method of the present invention.
The task scheduling method of the invention is suitable for being executed in a server, and the server is in communication connection with one or more task nodes, and the method comprises the following steps: setting the type of each task node in one or more task nodes, wherein the types of the task nodes comprise a first type and a second type, the first type task nodes are divided into a first number and a second number of first type task nodes, the first type task nodes are suitable for processing real-time tasks, and the second type task nodes are suitable for processing offline tasks; setting a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule; and processing the offline tasks according to the first number of the first-type task nodes and the second-type task nodes, and processing the real-time tasks according to the second number of the first-type task nodes. According to the invention, the first type task node for processing the real-time task and the second type task node for processing the offline task are uniformly scheduled by using one server, so that the management cost of the task nodes is reduced; and scheduling task nodes at a first scheduling time specified by a time period scheduling rule, setting a first number of first type task nodes to process offline tasks, fully utilizing the computing resources of the whole data platform, balancing the task processing pressure of the offline platform and the real-time platform, and improving the task processing efficiency.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which set forth the various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to fall within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. Like reference numerals generally refer to like parts or elements throughout the present disclosure.
FIG. 1 shows a schematic diagram illustrating a task scheduling system according to an exemplary embodiment of the present invention.
FIG. 2 illustrates a block diagram of a computing device 200 according to an exemplary embodiment of the invention.
Fig. 3 shows a schematic diagram of a task scheduling method 300 according to an exemplary embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like reference numerals generally refer to like parts or elements.
FIG. 1 shows a schematic diagram illustrating a task scheduling system according to an exemplary embodiment of the present invention. The task scheduling system comprises a server and one or more task nodes which are in communication connection with the server, wherein the server is suitable for setting the type of each task node in the one or more task nodes, the type of the task node comprises a first type and a second type, the first type task node is divided into a first type task node comprising a first number and a second number, the first type task node is suitable for processing real-time tasks, and the second type task node is suitable for processing offline tasks; in response to reaching a first scheduling time specified by a preset time period scheduling rule, a first number of first type task nodes are set to process offline tasks.
The first number of first-type task nodes and the second-type task node are adapted to process offline tasks, and the second number of first-type task nodes are adapted to process real-time tasks. The task nodes are provided with a plurality of types, including a first type and a second type. A plurality of task nodes of a first type may form a cluster of task nodes of the first type; the first type of task node cluster 110, as shown in fig. 1, includes task nodes 111, 112, 113, and 114. A plurality of task nodes of a second type may form a cluster of task nodes of the second type; the second type of task node cluster 120, shown in fig. 1, includes task nodes 121, 122, 123, and 124. The invention does not limit the mode of setting the types of the task nodes and the number of the task nodes included in each type of task node cluster.
According to one embodiment of the invention, the task node is adapted to process tasks assigned by the server 130. Each task node comprises a computing framework (NodeManager); the computing framework comprises an application program controller (APPMASTER) which is responsible for managing and executing specific tasks, applying resources for the tasks executed in the task nodes and distributing the resources, tracking the task state and monitoring the progress of task execution.
According to one embodiment of the invention, the server 130 may distribute a variety of tasks, including offline tasks and real-time tasks. The offline task may be implemented in particular as a task with low timeliness requirements, i.e. the required timeliness is typically t+1, i.e. the task is completed when or next day. Offline tasks such as: a task to aggregate data of users within a day, a task to process business processes on a day, a task to change settings to take effect the next day. The invention is not limited to the specific type of offline task. The real-time task can be specifically realized as a task with high timeliness requirement, namely, the required timeliness can reach the minute level, namely, the task needs to be processed immediately and the processing result is returned. Real-time tasks such as: searching search terms input by a user, processing a login request of the user, modifying data of a server side and the like. The invention is not limited to the specific type of real-time task.
The task characteristics of the offline task and the real-time task are different. During daytime, the activity of a user or a server is large; the number of real-time tasks is more submitted, and at night, the activity amount of a user or a service end is lower, the number of real-time tasks at night is reduced, and the processing pressure of the real-time tasks is reduced. In contrast to the off-line tasks, the off-line tasks have low timeliness requirements, so that most off-line tasks can be completed at night when the processing pressure of the service cluster tasks is low. Thus, during the daytime, the number of submitted offline tasks is low; but at night, a large number of offline tasks are submitted to meet the relevant processing requirements.
In order to cope with the situation that the processing pressure of the offline task and the real-time task is unbalanced and the processing time is asynchronous, the real-time queue 131, the offline queue 132, the queue binding rule 133, the time period scheduling rule 134 and the task quantity control rule 135 are arranged in the server 130, and the task distribution and the task monitoring in the task processing process are realized by using the rules through the queue resource monitoring module 136 and the queue resource scheduling module 137.
The real-time queue 131 is adapted to store and distribute real-time tasks and the offline queue 132 is adapted to store and distribute offline tasks. The queue resource monitoring module 136 is adapted to monitor the resources corresponding to the real-time queue 131 and the offline queue 132, that is, monitor the first type task node cluster 110 and the second type task node cluster 120, and determine the load and the task running condition thereof.
The queue resource scheduling module 137 is adapted to allocate tasks added in the real-time queue 131 and the offline queue 132 into the first type task node cluster 110 or the second type task node cluster 120.
Both the server and the task node of the present invention may be implemented as a single computing device.
FIG. 2 illustrates a block diagram of a computing device according to an exemplary embodiment of the invention. In a basic configuration, computing device 200 includes at least one processing unit 220 and system memory 210. According to one aspect, depending on the configuration and type of computing device, system memory 210 includes, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 210 includes an operating system 211.
According to one aspect, operating system 211 is suitable, for example, for controlling the operation of computing device 200. Further, examples are practiced in connection with a graphics library, other operating systems, or any other application program and are not limited to any particular application or system. This basic configuration is illustrated in fig. 2 by those components within dashed line 215. According to one aspect, computing device 200 has additional features or functionality. For example, according to one aspect, computing device 200 includes additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape.
As set forth hereinabove, according to one aspect, program modules 212 are stored in system memory 210. According to one aspect, program modules 212 may be implemented as one or more computer program products, the application is not limited to the type of computer program product, and may include, for example: email, word processing applications, spreadsheet applications, database applications, slide show applications, drawing or computer-aided application, web browser, etc. In some embodiments according to the present application, the computer programs/instructions associated with the task scheduling method 100 are packaged as a computer program product, which, when executed by a processor (i.e. the processing unit 220), implements the task scheduling method 300 according to the present application.
According to one aspect, the examples may be practiced in a circuit comprising discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic components or a microprocessor. For example, examples may be practiced via a system on a chip (SOC) in which each or many of the components shown in fig. 2 may be integrated on a single integrated circuit. According to one aspect, such SOC devices may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operating via an SOC, the functionality described herein may be operated via dedicated logic integrated with other components of computing device 200 on a single integrated circuit (chip). Embodiments of the invention may also be practiced using other techniques capable of performing logical operations (e.g., AND, OR, AND NOT), including but NOT limited to mechanical, optical, fluidic, AND quantum techniques. In addition, embodiments of the invention may be practiced within a general purpose computer or in any other circuit or system.
According to one aspect, computing device 200 may also have one or more input devices 231, such as a keyboard, mouse, pen, voice input device, touch input device, or the like. Output devices 232 such as a display, speakers, printer, etc. may also be included. The foregoing devices are examples and other devices may also be used. Computing device 200 may include one or more communication connections 233 that allow communication with other computing devices 240. Examples of suitable communication connections 233 include, but are not limited to: RF transmitter, receiver and/or transceiver circuitry; universal Serial Bus (USB), parallel and/or serial ports. Computing device 200 may be communicatively connected to other computing devices 240 via communication connection 233.
Embodiments of the present invention also provide a non-transitory readable storage medium storing instructions for causing the computing device to perform a method according to embodiments of the present invention. The readable media of the present embodiments, including both permanent and non-permanent, removable and non-removable media, may be any method or technology for information storage. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of readable storage media include, but are not limited to: phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transitory readable storage medium.
According to one aspect, communication media is embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., carrier wave or other transport mechanism) and includes any information delivery media. According to one aspect, the term "modulated data signal" describes a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio Frequency (RF), infrared, and other wireless media.
It should be noted that although the above-described computing device only shows processing unit 220, system memory 210, input device 231, output device 232, and communication connection 233, the device may include other components necessary to achieve proper operation in a particular implementation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.
Fig. 3 shows a schematic diagram of a task scheduling method 300 according to an exemplary embodiment of the invention. The task scheduling method of the present invention is suitable for execution in a server. The server is communicatively coupled to one or more of the task nodes.
The task scheduling method 300 of the present invention first performs step 310: setting a type of each task node in one or more task nodes, wherein the type of the task node comprises a first type and a second type, the first type task node is divided into a first type task node comprising a first number and a second number, the first type task node is suitable for processing real-time tasks, and the second type task node is suitable for processing offline tasks. The present invention is not limited to the specific values of the first number and the second number, and the sum of the first number and the second number is the total number of the first type task nodes. Taking task nodes 111-114 included in the first type task node cluster 110 as an example, if the first number is 2, including task node 111 and task node 112, the second number is 2, including task node 113 and task node 114; if the second number is 3, including task node 111, task node 112, and task node 113, then the first number is 1, including task node 114.
According to one embodiment of the invention, when the first type task nodes are divided into first type task nodes comprising a first number and a second number, a minimum number of the first type task nodes required for processing the real-time task is predetermined by a queue resource monitoring module; setting a task amount control rule according to the minimum number, wherein the task amount control rule comprises setting the second number to be larger than or equal to the minimum number; the first type of task nodes are divided into a first number and a second number of first type of task nodes according to the task volume control rule.
According to one embodiment of the invention, the minimum number of task nodes of the first type required to process a real-time task dictates how many task nodes are at a minimum required to process a real-time task. When the minimum number of the first type task nodes required for processing the real-time tasks is determined, the minimum number of the set minimum number of task nodes can be ensured to deal with the processing of the real-time tasks according to the daily submitting condition and the daily processing condition of the real-time tasks, and the situation that the number of the real-time tasks in a real-time queue exceeds the limit of the queue does not occur. By setting the task quantity control rule, when the first type task nodes are divided and called subsequently, the processing requirement of the real-time task can be met, and the problem that the specific service cannot be processed in real time is avoided.
Subsequently, step 320 is performed to set a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule. The queue resource scheduling module is adapted to set a first number of task nodes of a first type to process offline tasks.
The preset time period scheduling rule of the invention comprises the following steps: setting one or more scheduling time periods included in the daily time, wherein the starting time of each scheduling time period is used as a first scheduling time, and the ending time is used as a second scheduling time; the first type task nodes are arranged at the first scheduling time to schedule the first number of the first type task nodes to process the offline tasks, and the second scheduling time to schedule the first number of the first type task nodes to process the real-time tasks.
According to one embodiment of the invention, the scheduling time period can be set at night, such as 21 to 5 a.m. on the next day, the first scheduling time is 21, and the second scheduling time is 5 a.m. on the next day. The method and the system do not limit the specific setting mode of the scheduling time period, and a developer can flexibly set according to the execution requirement of the offline task and the submission rule of the real-time task.
According to one embodiment of the present invention, one or more scheduling periods, such as a first scheduling period and a second scheduling period, may be set. The first scheduling time of the first scheduling time period is 21 points, and the second scheduling time is 5 points in the morning of the next day. The first scheduling time of the second scheduling time period is 12 points, and the second scheduling time of the second scheduling time period is 14 points.
Subsequently, step 330 is performed to process offline tasks according to the first number of first-type task nodes and the second-type task nodes, and to process real-time tasks according to the second number of first-type task nodes.
Specifically: submitting one or more offline tasks to a real-time queue and an offline queue according to a queue binding rule by a queue resource scheduling module; distributing the offline tasks submitted to the real-time queue to a first number of first type task nodes for execution; and distributing the offline tasks submitted to the offline queue to the second type task nodes for execution.
According to one embodiment of the present invention, a queue binding rule is preset, where the queue binding rule includes: submitting the offline tasks to a real-time queue and an offline queue in a scheduling time period, and submitting the real-time tasks to the real-time queue; and when the task does not belong to the scheduling time period, submitting the offline task to an offline queue, and submitting the real-time task to the real-time queue so as to meet the corresponding task execution requirement.
According to one embodiment of the invention, when the queue resource scheduling module submits the offline tasks to the real-time queue and the offline queue, the first type task nodes and the second type task nodes for processing the offline tasks can be allocated according to the number of the first type task nodes and the number of the second type task nodes, namely, the first type task nodes and the second type task nodes for processing the offline tasks are allocated according to the ratio of the number of the first type task nodes to the number of the second type task nodes, so that the processing pressure of the first type task nodes and the second type task nodes for processing the offline tasks is the same.
Processing real-time tasks according to a second number of first type task nodes includes: submitting the real-time task to a real-time queue according to the queue binding rule; real-time tasks submitted to the real-time queue are distributed to a second number of first type task nodes for execution.
According to one embodiment of the invention, the invention further sets the first number of task nodes of the first type to process real-time tasks in response to reaching a second scheduling time specified by a scheduling rule for a preset time period; and submitting the offline tasks to the offline queue and submitting the real-time tasks to the real-time queue after the second scheduling time arrives, so as to process the real-time tasks according to the first number and the second number of the first type task nodes and process the offline tasks according to the second type task nodes. The first type task nodes of the first number and the second number process real-time tasks together in the time except the preset time period, so that the service cluster can process a large number of real-time tasks submitted in the daytime normally, and the service cluster can maintain normal service provision.
The task scheduling method of the invention is suitable for being executed in a server, and the server is in communication connection with one or more task nodes, and the method comprises the following steps: setting the type of each task node in one or more task nodes, wherein the types of the task nodes comprise a first type and a second type, the first type task nodes are divided into a first number and a second number of first type task nodes, the first type task nodes are suitable for processing real-time tasks, and the second type task nodes are suitable for processing offline tasks; setting a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule; and processing the offline tasks according to the first number of the first-type task nodes and the second-type task nodes, and processing the real-time tasks according to the second number of the first-type task nodes. According to the invention, the first type task node for processing the real-time task and the second type task node for processing the offline task are uniformly scheduled by using one server, so that the management cost of the task nodes is reduced; and scheduling task nodes at a first scheduling time specified by a time period scheduling rule, setting a first number of first type task nodes to process offline tasks, fully utilizing the computing resources of the whole data platform, balancing the task processing pressure of the offline platform and the real-time platform, and improving the task processing efficiency.
The method of the invention further comprises the following steps: a11, a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the method of any of A1-A9. A12, a computer program product comprising computer programs/instructions which, when executed by a processor, implement the method according to any of A1-A9.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects.
Those skilled in the art will appreciate that the modules or units or groups of devices in the examples disclosed herein may be arranged in a device as described in this embodiment, or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into a plurality of sub-modules.
Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or groups of embodiments may be combined into one module or unit or group, and furthermore they may be divided into a plurality of sub-modules or sub-units or groups. Any combination of all features disclosed in this specification, as well as all processes or units of any method or apparatus so disclosed, may be employed, except that at least some of such features and or processes or units are mutually exclusive. Each feature disclosed in this specification may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments.
Furthermore, some of the embodiments are described herein as methods or combinations of method elements that may be implemented by a processor of a computer system or by other means of performing the functions. Thus, a processor with the necessary instructions for implementing the described method or method element forms a means for implementing the method or method element. Furthermore, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is for carrying out the functions performed by the elements for carrying out the objects of the invention.
The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the task scheduling method of the present invention according to instructions in said program code stored in the memory.
By way of example, and not limitation, computer readable media comprise computer storage media and communication media. Computer-readable media include computer storage media and communication media. Computer storage media stores information such as computer readable instructions, data structures, program modules, or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of computer readable media.
As used herein, unless otherwise specified the use of the ordinal terms "first," "second," "third," etc., to describe a general object merely denote different instances of like objects, and are not intended to imply that the objects so described must have a given order, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments are contemplated within the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. The disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention.
Claims (10)
1. A task scheduling method adapted to run in a server communicatively connected to one or more task nodes, the method comprising:
Setting the type of each task node in the one or more task nodes, wherein the type of the task node comprises a first type and a second type, the first type task node is divided into a first number and a second number of first type task nodes, the first type task node is suitable for processing real-time tasks, and the second type task node is suitable for processing offline tasks;
setting a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule;
and processing offline tasks according to the first number of first-type task nodes and the second-type task nodes, and processing real-time tasks according to the second number of first-type task nodes.
2. The method of claim 1, wherein the method further comprises:
setting a first number of first type task nodes to process real-time tasks in response to reaching a second scheduling time specified by a preset time period scheduling rule;
And processing real-time tasks according to the first number and the second number of the first type task nodes, and processing offline tasks according to the second type task nodes.
3. The method of claim 1 or 2, wherein setting the time period scheduling rule comprises:
setting one or more scheduling time periods included in the daily time, wherein the starting time of each scheduling time period is used as a first scheduling time, and the ending time is used as a second scheduling time;
the first type task nodes are arranged at the first scheduling time to schedule the first number of the first type task nodes to process the offline tasks, and the second scheduling time to schedule the first number of the first type task nodes to process the real-time tasks.
4. A method according to any one of claims 1-3, wherein the method further comprises:
determining a minimum number of first type task nodes required to process the real-time task;
setting a task amount control rule according to the minimum number, wherein the task amount control rule comprises setting the second number to be larger than or equal to the minimum number;
The first type task nodes are divided into first type task nodes comprising a first number and a second number according to the task amount control rule.
5. The method of claim 1, wherein the processing offline tasks according to the first number of first-type task nodes and the second-type task node comprises:
submitting one or more offline tasks to a real-time queue and an offline queue according to the queue binding rule;
distributing the offline tasks submitted to the real-time queue to the first number of first type task nodes for execution;
and distributing the offline tasks submitted to the offline queue to the second type task nodes for execution.
6. The method of claim 5, wherein said processing real-time tasks according to said second number of first type task nodes comprises:
submitting the real-time task to the real-time queue according to a queue binding rule;
And distributing the real-time tasks submitted to the real-time queue to the second number of first type task nodes for execution.
7. The method of any one of claims 1-6, wherein the method further comprises: setting the queue binding rule, wherein the queue binding rule comprises:
Submitting the offline tasks to a real-time queue and an offline queue in a scheduling time period, and submitting the real-time tasks to the real-time queue;
and submitting the offline tasks to the offline queue and submitting the real-time tasks to the real-time queue when the tasks do not belong to the scheduling time period.
8. A method as defined in claim 7, wherein the processing offline tasks according to the first number of first-type task nodes and the second-type task node comprises:
and distributing the offline tasks in the real-time queue to a first number of first-type task nodes for execution, and distributing the offline tasks in the offline queue to the second-type task nodes for execution.
9. A task scheduling system comprising a server and one or more task nodes communicatively connected to the server, the server being adapted to set a type of each of the one or more task nodes, the type of task nodes comprising a first type and a second type, and to divide the first type of task nodes into a first number and a second number of first type of task nodes, the first type of task nodes being adapted to process real-time tasks, the second type of task nodes being adapted to process offline tasks;
setting a first number of first type task nodes to process offline tasks in response to reaching a first scheduling time specified by a preset time period scheduling rule;
A first number of task nodes of a first type and said task nodes of a second type are adapted to handle offline tasks and said second number of task nodes of the first type are adapted to handle real-time tasks.
10. A computing device, comprising:
one or more processors;
A memory; and
One or more programs, wherein the one or more programs are stored in memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410458202.4A CN118394468A (en) | 2024-04-16 | 2024-04-16 | Task scheduling method, system and computing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410458202.4A CN118394468A (en) | 2024-04-16 | 2024-04-16 | Task scheduling method, system and computing device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118394468A true CN118394468A (en) | 2024-07-26 |
Family
ID=91986190
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410458202.4A Pending CN118394468A (en) | 2024-04-16 | 2024-04-16 | Task scheduling method, system and computing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118394468A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118819864A (en) * | 2024-09-18 | 2024-10-22 | 北京景行锐创软件有限公司 | Resource unified scheduling method and system for multiple types of loads |
-
2024
- 2024-04-16 CN CN202410458202.4A patent/CN118394468A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118819864A (en) * | 2024-09-18 | 2024-10-22 | 北京景行锐创软件有限公司 | Resource unified scheduling method and system for multiple types of loads |
| CN118819864B (en) * | 2024-09-18 | 2024-12-20 | 北京景行锐创软件有限公司 | Unified scheduling method and system for resources of multiple types of loads |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10754704B2 (en) | Cluster load balancing based on assessment of future loading | |
| US12099483B2 (en) | Rules based scheduling and migration of databases using complexity and weight | |
| CN108549583B (en) | Big data processing method and device, server and readable storage medium | |
| US10108458B2 (en) | System and method for scheduling jobs in distributed datacenters | |
| US20200104376A1 (en) | Data Migration Using Customizable Database Consolidation Rules | |
| US8595732B2 (en) | Reducing the response time of flexible highly data parallel task by assigning task sets using dynamic combined longest processing time scheme | |
| US20190065225A9 (en) | Virtual machine packing method using scarcity | |
| US7925755B2 (en) | Peer to peer resource negotiation and coordination to satisfy a service level objective | |
| US10680975B2 (en) | Method of dynamic resource allocation for public clouds | |
| CN110008018A (en) | A kind of batch tasks processing method, device and equipment | |
| CN114356587B (en) | Calculation power task cross-region scheduling method, system and equipment | |
| US7681196B2 (en) | Providing optimal number of threads to applications performing multi-tasking using threads | |
| CN118394468A (en) | Task scheduling method, system and computing device | |
| CN112130997A (en) | Resource allocation method and device | |
| CN119065848B (en) | Resource scheduling method and device based on cloud service platform | |
| CN118550671A (en) | Job set scheduling method, device, computer equipment and storage medium | |
| CN111353766A (en) | Service process processing system and method of distributed service system | |
| CN117707777A (en) | Task execution method, system, device and storage medium for dynamic allocation of resources | |
| CN115443642B (en) | Distribution of rules across instances of the rule engine | |
| CN112612579B (en) | Virtual machine deployment method, storage medium and computer equipment | |
| CN114338694A (en) | One-stop cloud data center server scheduling method and system | |
| CN108196936A (en) | A kind of resource regulating method, equipment and system | |
| CN114625540B (en) | Resource providing method and device | |
| US20240256361A1 (en) | Predictive load modeling using a digital twin of a computing infrastructure | |
| EP4528505A1 (en) | Data processing method, apparatus, device, and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |