CN102763086A - Distributed computing task processing system and task processing method - Google Patents

Abstract

The embodiment of the invention provides a distributed computing task processing system and a task processing method. The system comprises: the first layer scheduler is used for receiving a request for executing the task, starting or selecting a second layer scheduler corresponding to the task and forwarding the request to the second layer scheduler; and the second-layer scheduler is used for decomposing the task into a plurality of subtasks according to the logical relation of the task when receiving the request forwarded by the first-layer scheduler. The embodiment of the invention adopts a two-layer scheduling architecture, the second-layer scheduler corresponds to the tasks, and the first-layer scheduler starts or selects the second-layer scheduler corresponding to the tasks, thereby being suitable for different tasks and improving the processing efficiency and the scheduling flexibility.

Description

Distributed computing task processing system and task processing method

technical field

Embodiments of the present invention relate to the field of network communication, and more specifically, to a distributed computing task processing system and a task processing method.

Background technique

Currently, with the development of the Internet, the need for fast processing of a large amount of information becomes urgent. Therefore, parallel processing of data becomes very important. The distributed computing environment provides an effective means of resource sharing and interoperability between different software and hardware platforms in the network environment, and has become a common architecture for parallel processing. At present, the well-known parallel processing system in the industry adopts the MapReduce architecture. MapReduce is a distributed computing software architecture that can support distributed processing of large amounts of data. This architecture originally originated from the two functions of map (mapping) and reduce (reduction) of functional programming. Map refers to processing the original document according to custom mapping rules and outputting intermediate results. reduce merges the intermediate results according to the custom reduction rules.

In a distributed computing environment, the general architecture of MapReduce includes scheduling nodes and multiple working nodes. The scheduling node is responsible for task scheduling and resource management; it is responsible for decomposing the tasks submitted by the user into two subtasks of map and reduce according to the user configuration, and assigning the subtasks of map and reduce to the working nodes. Worker nodes are responsible for running map and reduce subtasks and maintaining communication with scheduling nodes.

In this parallel processing architecture, since a scheduling node is responsible for task and resource management, it needs to process tasks strictly in the order of map and reduce. If there are many steps of processing, it needs to be completed by submitting many task requests, the processing efficiency is low, and the scheduling is not flexible enough.

Contents of the invention

Embodiments of the present invention provide a task processing system and a task processing method, which can solve the problem of processing efficiency in the existing parallel processing architecture.

In one aspect, a distributed computing task processing system is provided, including: a first-level scheduler, configured to receive a request for executing a task, start or select a second-level scheduler corresponding to the task, and forward the requested information to the second-level scheduler. The above-mentioned request; the second-level scheduler is used to decompose the task into multiple sub-tasks according to the logical relationship of the task when receiving the request forwarded by the first-level scheduler.

On the other hand, a distributed computing task processing method is provided, the method includes: when the first-level scheduler receives a request to execute the task, start or select the second-level scheduler corresponding to the task; the first-level scheduler The request is forwarded to the second-level scheduler; when the second-level scheduler receives the request forwarded by the first-level scheduler, it decomposes the task into multiple subtasks according to the logical relationship of the task.

The embodiment of the present invention adopts a two-tier scheduling architecture, the second-tier scheduler corresponds to the task, and the first-tier scheduler starts or selects the second-tier scheduler corresponding to the task, so that it can be applied to different tasks, and the processing efficiency and scheduling are improved. flexibility.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a block diagram of a task processing system according to an embodiment of the present invention.

Figure 2 is a schematic diagram of the processing architecture of one embodiment of the present invention.

Fig. 3 is a flowchart of a task processing method according to an embodiment of the present invention.

Fig. 4 is a schematic flowchart of a task processing process according to an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

FIG. 1 is a block diagram of a distributed computing task processing system according to an embodiment of the present invention. The task processing system 10 in FIG. 1 includes two layers of schedulers, namely a first layer scheduler 11 and a second layer scheduler 12 .

The first-level scheduler 11 receives a request for executing a task, starts or selects a second-level scheduler 12 corresponding to the task, and forwards the request to the second-level scheduler 12 .

For example, when there is no suitable second-level scheduler in the system, the first-level scheduler 11 can start the second-level scheduler 12 corresponding to the task. When there are suitable second-level schedulers in the system, the first-level scheduler 11 can select the second-level scheduler 12 corresponding to the task from these suitable second-level schedulers.

Optionally, the first-level scheduler is further configured to perform priority management on the tasks, and start or select the second-level scheduler to process the tasks according to the priorities.

When the second-level scheduler 12 receives the request forwarded by the first-level scheduler 11, it decomposes the task into multiple subtasks according to the logical relationship of the task.

The embodiment of the present invention adopts a two-tier scheduling architecture, the second-tier scheduler corresponds to the task, and the first-tier scheduler starts or selects the second-tier scheduler corresponding to the task, so that it can be applied to different tasks, and the processing efficiency and scheduling are improved. flexibility.

In the existing parallel processing architecture, there is only one layer of scheduling, so task processing needs to be strictly followed by two steps of map and reduce, but the embodiment of the present invention has no such limitation. The first-layer scheduler 11 of the embodiment of the present invention can accept various forms of tasks, and the form of tasks is not necessarily limited to the strict map and reduce steps in the prior art. The second-level schedulers 12 correspond to tasks, so that the first-level scheduler 11 can send different tasks to corresponding second-level schedulers 12 for scheduling processing. The second-level scheduler 12 decomposes the task into subtasks to process the task, such as scheduling the execution of each subtask. Such scheduling has higher flexibility.

In addition, in the prior art, task processing needs to be performed strictly in the order of map and reduce. If there are many steps of processing, it needs to be completed by submitting many task requests, and the processing efficiency is low. The embodiment of the present invention has no such limitation. The embodiment of the present invention does not limit the task itself and the manner of executing the task or subtask. For example, the number of subtasks included in a task may be more than that of map and reduce in the prior art, such as three or more subtasks; and it is not limited to the form of map and reduce subtasks. In addition, the subtasks do not have to follow a strict sequence, and can be executed in parallel, serially, or partly in parallel and partly in serially. In this way, even for processing with many steps, only a small number of task requests are required, which improves the processing efficiency.

The number of subtasks is related to specific tasks, such as transcoding, face recognition business and other tasks. Depending on the logical relationship of tasks, these tasks may have the same or different number of subtasks. Optionally, as an embodiment, the logical relationship of the task may be carried in the description file of the task. For example, the task processing system 10 (specifically, such as the second-level scheduler 12) can receive a description file uploaded by the user, such as a description file in XML (Extensible Markup Language, Extensible Markup Language) format, which contains the description file of the task. Logic.

Optionally, when receiving the request forwarded by the first-level scheduler 11, the second-level scheduler 12 obtains the XML-formatted description file corresponding to the task. According to the logical relationship of the task carried in the description file in XML format, the task is decomposed into multiple subtasks.

In addition, each subtask can be further decomposed into smaller granularity subtasks. That is to say, the subtasks in this embodiment of the present invention may be multi-layer subtasks, and the way of further decomposing each layer of subtasks can be determined by the logical relationship carried in the description file. For example, subtask 1 can be decomposed into multiple subtasks 2, and subtask 2 can be further decomposed into multiple subtasks 3, and so on.

Optionally, as an embodiment, the logical relationship of tasks may indicate the execution dependencies of multiple subtasks. The so-called execution dependency refers to whether the execution operations of each subtask depend on each other.

For example, assuming that subtask 2 must depend on the execution result of subtask 1, then subtask 2 should be executed after subtask 1 is executed (that is, subtask 1 and subtask 2 need to be executed serially). On the other hand, if subtask 2 does not depend on all execution results of subtask 1, then subtask 1 and subtask 2 can be executed in parallel or serially.

A non-limiting example of execution dependency may include: two or more subtasks in multiple subtasks are executed in serial, or parallel, or partially parallel and partially serial order, not limited to the map in the prior art , reduce these two steps. In this way, if there are many steps of processing for a certain task, there is no need to submit many task requests like the MapReduce architecture, and the embodiment of the present invention may only need to submit one or a few task requests, thereby improving the task processing efficiency.

The logical relationship of the tasks may explicitly indicate the execution dependencies among the subtasks, for example, it may explicitly indicate that the task is composed of subtasks 1-3 executed serially. Alternatively, the logical relationship of tasks may implicitly indicate the execution dependencies between subtasks. For example, for a certain task, the system knows in advance that the task is composed of subtasks 1-3 executed serially.

Optionally, as another embodiment, the second-level scheduler 12 is further configured to create corresponding queues for the multiple subtasks to store tasks included in the subtasks. When the tasks contained in the subtasks are stored in the queue, the second-level scheduler 12 can also be used to apply for resources for the subtasks, and instruct the work unit manager of the requested resources to start the work unit to The working unit is made to obtain the tasks included in the subtasks from the queue to execute the tasks. Optionally, as another embodiment, the second-level scheduler 12 may also be configured to instruct the work unit to put the task execution result into another queue or output the task execution result.

Further, as another embodiment, the second-level scheduler 12 may also be configured to obtain progress information of the queue and the work unit, so as to determine the execution progress of the task.

In a word, the embodiment of the present invention does not limit the specific form of the task. Optionally, the setting or selection of the logical relationship of the task may support user customization, for example, receiving the user's setting or selection through a plug-in mechanism.

The task processing system 10 of the embodiment of the present invention can be applied to cloud computing architecture. Cloud computing proposes a high-reliability, low-cost, on-demand, elastic business model. Many systems can achieve high reliability, elasticity, and low cost by using cloud services.

Figure 2 is a schematic diagram of the processing architecture of one embodiment of the present invention. The processing architecture 20 in FIG. 2 is a cloud computing architecture, including the task processing system 10 in FIG. 1 . The difference from FIG. 1 is that the task processing system 10 in FIG. 2 may include multiple second-level schedulers 12 . For simplicity, only two second-level schedulers 12 are depicted in FIG. 2 , but the number of second-level schedulers 12 is not limited by this example (it can be more or less). Each second-level scheduler 12 corresponds to a type of task to adapt to or support different computing models. Optionally, multiple second-level schedulers 12 may also correspond to one type of task, so as to achieve high concurrency of system scheduling. If there is a suitable second-level scheduler 12 corresponding to the task among the existing plurality of second-level schedulers 12, the first-level scheduler 11 can select the suitable second-level scheduler 12 to process the task; If there is no suitable second-level scheduler 12 corresponding to the task among the existing multiple second-level schedulers 12, the first-level scheduler 11 can start a new suitable second-level scheduler 12 to process the task .

In the processing architecture 20, the first-level scheduler 11 may be distributed to support high concurrency. The first layer scheduler 11 can receive the task request sent by the Webservice (network service) 21 . Webservice 21 is responsible for receiving and forwarding the user's web (network) request, and the specific implementation may refer to the prior art, so details are not repeated here.

Optionally, as an embodiment, when the first-level scheduler 11 receives a plurality of task requests, the first-level scheduler 11 can also perform priority management (for example, perform priority sorting) on the tasks, and Start or select the second-level scheduler 12 to process the task. For example, the first-level scheduler 11 may preferentially start or select the second-level scheduler 12 corresponding to a task with a higher priority.

Optionally, as another embodiment, the first-level scheduler 11 may implement additional functions such as task priority adjustment. The way of prioritizing or adjusting can support user customization, for example, receiving user's settings through a plug-in mechanism.

After the first-level scheduler 11 starts or selects the second-level scheduler 12 corresponding to the task, it forwards the task request to the second-level scheduler 12 . The second-level scheduler 12 decomposes the task into multiple subtasks according to the logical relationship of the task, and manages the execution of the multiple subtasks. Optionally, as an embodiment, the execution of subtasks may be managed through a queue (distributed queue 22 shown in FIG. 2 ). The distributed queue 22 may include multiple queues, respectively storing tasks contained in corresponding subtasks.

Specifically, the second-level scheduler 12 may create corresponding queues for multiple subtasks to store tasks included in the subtasks. The second-level scheduler 12 can arrange the order of the queues according to the logical relationship of the tasks. For example, assuming that the task is composed of subtasks 1-3 (subtask 1->subtask 2->subtask 3) executed serially, the second-level scheduler 12 can establish queues 1-3 to store subtask 1 respectively. -3 tasks included, and determine the order of queues 1-3, that is, execute the tasks contained in the corresponding subtasks in the order of queue 1->queue 2->queue 3. The task execution result of subtask 1 is placed in queue 2, the task execution result of subtask 2 is placed in queue 3, and the task execution result of subtask 3 is output to an appropriate location, for example, to the distributed storage shown in Figure 2 device 24 or returned to the user.

Optionally, as another embodiment, when the tasks included in the subtask are stored in the distributed queue 22 , the second-level scheduler 12 may also apply for resources for the subtask, for example, apply for resources from the resource manager 25 . The resource manager 25 is responsible for satisfying the resource application and release of the scheduler 11 or 12 . The main functions of the resource manager 25 include resource management, resource matching, and automatic scaling of resources. Among them, the resource matching method can adopt a plug-in mechanism to support user-defined. In addition, the so-called automatic resource scaling means that when the user configures the cluster size within a certain range, the cluster can be automatically expanded or reduced according to the cluster load. For other implementation manners of the resource manager 25, reference may be made to the prior art, so details are not repeated here. For example, the resource manager 25 may also adopt a distributed solution to achieve high concurrency.

After setting up the queue and applying for resources for the subtasks, the second-level scheduler 12 can instruct the work unit (worker) manager 26 of the applied resources to start the work unit 27, so that the work unit 27 obtains the tasks contained in the subtasks from the queue and Execute the task. Worker manager 26 is responsible for creation, deletion and monitoring of worker 27. There is a worker manager 26 on each node (physical machine or virtual machine) in the cloud computing architecture. For other implementations of the worker manager 26, reference may be made to the prior art, so details are not repeated here.

Worker 27 is responsible for obtaining the tasks contained in the user's subtasks from the corresponding queue in the distributed queue 22, performing preprocessing, and then calling the processing program developed by the user. After the processing is completed, it can be determined according to the second layer scheduler 12 The order of the queue, put the result of executing the task into another queue or output the result of executing the task. For other implementations of Worker 27, reference may be made to existing technologies, so details are not repeated here.

In addition, the second-level scheduler 12 can also implement other scheduling processes, such as task exception handling or task progress statistics. For example, the second-level scheduler 12 can obtain the progress information of the queue and the work unit (worker) (such as whether each subtask is completed or how much is completed, whether the subtasks in each queue are completed or how much is completed, etc.), to Determine the progress of the task. In this way, real-time query of task progress can be realized. For example, the user can go to the second-level scheduler 12 to query the execution progress of the corresponding task. Alternatively, the second-level scheduler 12 may report the progress information of the task to the first-level scheduler 11, so that the user can query the execution progress of the corresponding task from the first-level scheduler 11, which is convenient for the user to monitor.

For simplicity, three worker managers 26 and corresponding three workers 27 are illustrated in FIG. 2 , but the embodiment of the present invention is not limited to this specific example, and the number of worker managers 26 and worker 27 can be more or less.

The cluster management software 28 is responsible for the automatic deployment and basic monitoring of the cluster processing parallel tasks, and its implementation method can refer to the existing technology, so it will not be repeated here.

Distributed queue 22, database 23 (such as nosql database), and distributed storage device 24 realize the task storage, database and file storage required by processing architecture 20. The specific implementation method can also refer to the existing technology, so it will not be described again. For example, the database 23 can be used for persistent storage of information to meet system operation requirements or implement fault tolerance.

The bottom layer of the processing architecture 20 supports various heterogeneous hardware such as physical machines or virtual machines 29 , and there is no need to care about it for user applications. For the implementation manner of the physical machine or the virtual machine 29, reference may be made to the prior art, so details are not repeated here.

The processing architecture 20 adopts a "queue-worker" computing model, but the embodiment of the present invention is not limited thereto. The processing architecture 20 may also adopt other computing models. For example, part of the second-layer schedulers 12 in the processing architecture 20 may also adopt the aforementioned MapReduce method without queues.

Therefore, the processing architecture 20 of the embodiment of the present invention adopts a two-tier scheduling architecture, the second-tier scheduler corresponds to the task, and the first-tier scheduler starts or selects the second-tier scheduler corresponding to the task, so that it can be applied to different tasks , improving processing efficiency and scheduling flexibility. Moreover, through the above-mentioned "queue-worker" computing model, multiple second-level schedulers for different tasks can be started at the same time, further improving concurrency performance.

In addition, the embodiment of the present invention provides a high-performance, flexible parallel processing architecture 20 that can support physical machines and currently popular cloud computing platforms, support large-scale clusters, support user scheduling policy configuration and customization, and support different computing models.

Fig. 3 is a flowchart of a distributed computing task processing method according to an embodiment of the present invention. The method in FIG. 3 can be executed by the task processing system 10 in FIG. 1 and FIG. 2 , so the method in FIG. 3 will be described below in conjunction with FIG. 1 and FIG. 2 , and repeated descriptions will be appropriately omitted.

301. When receiving a request to execute a task, the first-level scheduler 11 starts or selects a second-level scheduler 12 corresponding to the task.

For example, when there is no suitable second-level scheduler in the system, the first-level scheduler 11 can start the second-level scheduler 12 corresponding to the task. When there are suitable second-level schedulers in the system, the first-level scheduler 11 can select the second-level scheduler 12 corresponding to the task from these suitable second-level schedulers.

302. The first-level scheduler 11 forwards the request to the second-level scheduler 12.

303. When receiving the request forwarded by the first-level scheduler, the second-level scheduler 12 decomposes the task into multiple subtasks according to the logical relationship of the task.

The embodiment of the present invention adopts a two-tier scheduling architecture, the second-tier scheduler corresponds to the task, and the first-tier scheduler starts or selects the second-tier scheduler corresponding to the task, so that it can be applied to different tasks, improving the processing efficiency and Scheduling flexibility.

In the existing parallel processing architecture, there is only one layer of scheduling, so task processing needs to be strictly followed by two steps of map and reduce, but the embodiment of the present invention has no such limitation. The first-layer scheduler 11 of the embodiment of the present invention can accept various forms of tasks, and the form of tasks is not necessarily limited to the strict map and reduce steps in the prior art. The second-level schedulers 12 correspond to tasks, so that the first-level scheduler 11 can send different tasks to corresponding second-level schedulers 12 for scheduling processing. The second-level scheduler 12 decomposes the task into subtasks to process the task, such as scheduling the execution of each subtask. Such scheduling has higher flexibility.

In addition, in the prior art, task processing needs to be performed strictly in the order of map and reduce. If there are many steps of processing, it needs to be completed by submitting many task requests, and the processing efficiency is low. The embodiment of the present invention has no such limitation. The embodiment of the present invention does not limit the task itself and the manner of executing the task or subtask. For example, the number of subtasks included in a task may be more than that of map and reduce in the prior art, such as three or more subtasks; and it is not limited to the form of map and reduce subtasks. In addition, the subtasks do not have to follow a strict sequence, and can be executed in parallel, serially, or partly in parallel and partly in serially. In this way, even for processing with many steps, only a small number of task requests are required, which improves the processing efficiency.

Optionally, as an embodiment, the logical relationship of tasks may indicate the execution dependencies of multiple subtasks. The so-called execution dependency refers to whether the execution operations of each subtask depend on each other.

For example, assuming that subtask 2 must depend on the execution result of subtask 1, then subtask 2 should be executed after subtask 1 is executed (that is, subtask 1 and subtask 2 need to be executed serially). On the other hand, if subtask 2 does not depend on all execution results of subtask 1, subtask 1 and subtask 2 can be executed in parallel or serially.

A non-limiting example of execution dependency may include: two or more subtasks in multiple subtasks are executed in serial, or parallel, or partially parallel and partially serial order, not limited to the map in the prior art , reduce these two steps. In this way, if there are many steps of processing for a certain task, there is no need to submit many task requests like the MapReduce architecture, and the embodiment of the present invention may only need to submit one or a few task requests, thereby improving the task processing efficiency.

Optionally, as another embodiment, the second-level scheduler 12 may also create corresponding queues for multiple subtasks to store the tasks included in the subtasks, and arrange the order of the queues according to the logical relationship of the tasks.

Optionally, as another embodiment, the second-level scheduler 12 can also apply for resources for the subtasks when the tasks contained in the subtasks are stored in the queue, and instruct the work unit (worker) manager of the requested resources to start The unit of work, so that the unit of work fetches the tasks contained in the subtasks from the queue to execute the tasks.

Optionally, as another embodiment, the second-level scheduler 12 may also instruct the work unit to put the result of executing the task into another queue or output the result of executing the task.

Optionally, as another embodiment, the second-level scheduler 12 may also obtain progress information of queues and work units, so as to determine the execution progress of tasks. In this way, real-time query of task progress can be realized, which is convenient for users to monitor.

Optionally, as another embodiment, in step 301, the first-level scheduler 11 may perform priority management on the tasks, and start or select the second-level scheduler 12 to process the tasks according to the priority.

In the following, the embodiments of the present invention will be described in more detail in combination with specific examples. Fig. 4 is a schematic flowchart of a task processing process according to an embodiment of the present invention. For example, the process in FIG. 4 can be executed by the processing architecture 20 in FIG. 2 , so repeated descriptions are appropriately omitted.

In the example in FIG. 4 , it is assumed that the task is composed of subtasks 1-3 (subtask 1 -> subtask 2 -> subtask 3) which are executed serially. However, this embodiment of the present invention is not limited to this specific example, and any other type of task can similarly apply the processing procedure of this embodiment of the present invention. Such applications all fall within the scope of the embodiments of the present invention.

401. A web service (webservice) receives a request for executing a task submitted by a user. The logical relationship of tasks can be defined by the user.

402. The network service forwards the request to the first-layer scheduler.

403. The first-layer scheduler returns a response that the request is submitted successfully to the network service. Step 403 is an optional step.

404. The first-level scheduler calculates and sorts the priorities of the received tasks according to the priority calculation method, and selects tasks with higher priorities.

405, the first-level scheduler starts (if there is no second-level scheduler corresponding to this type in the system at that time) or selects (there is a second-level scheduler corresponding to this type available in the system) according to the task selected in step 404. layer scheduler) a suitable second layer scheduler.

406 , the first-level scheduler forwards the task request to the second-level scheduler started or selected in step 405 .

407. After receiving the task request, the second-layer scheduler preprocesses the task according to the logical relationship of the task. Specifically, as a non-limiting example, the second-level scheduler may decompose a task into multiple subtasks (subtask 1, subtask 2, subtask 3).

408. The second-level scheduler creates queues 1-3 for subtasks 1-3 according to the "queue-worker" computing model. Optionally, the second-level scheduler may generate an initial subtask (corresponding to subtask1) and put it in queue1. At this time, the second-level scheduler can arrange the execution order of queues 1-3 according to the execution dependency relationship between subtasks 1-3 "subtask 1->subtask 2->subtask 3" as "queue 1-> Queue 2 -> Queue 3".

409. The second-level scheduler finds that there is subtask 1 in queue 1. For example, a second-tier scheduler could periodically check the queue to see if there are tasks in the queue. However, this embodiment of the present invention does not limit this, and the second-level scheduler may discover subtasks in the queue in other ways.

410. The second-level scheduler applies for resources for subtask 1 from the resource manager.

411 , the second-level scheduler instructs the work unit (worker) manager of the applied resource to start the worker to process the subtask 1 in the queue 1 .

412. The worker manager starts the worker, and notifies the worker to put the result (corresponding to the subtask 2) obtained after processing the subtask 1 into the queue 2.

413: After the worker is started, it automatically goes to queue 1 to obtain and execute the tasks contained in subtask 1, and after the execution is completed, puts the execution result (corresponding to subtask 2) into queue 2.

414. The second-level scheduler finds that there is subtask 2 in queue 2.

415. The second-level scheduler applies for resources for subtask 2 from the resource manager.

416. The second-level scheduler instructs the work unit (worker) manager to start the worker to process the subtask 2 in the queue 2.

417. The worker manager starts the worker, and informs the worker to put the result (corresponding to the subtask 3) into the queue 3 after processing the subtask 2.

418: After the worker is started, it automatically goes to the queue 2 to obtain and execute the tasks contained in the subtask 2, and after the execution is completed, puts the execution result (corresponding to the subtask 3) into the queue 3.

419. The second-level scheduler finds that there is subtask 3 in queue 3.

420. The second-level scheduler applies for resources for subtask 3 from the resource manager.

421. The second-level scheduler instructs the work unit (worker) manager to start the worker to process the subtask 3 in the queue 3.

422. The worker manager starts the worker, and informs the worker to put the result obtained after processing subtask 3 into an appropriate location (such as putting it into a distributed storage device or returning it to the user).

423. After the worker is started, it automatically goes to the queue 3 to obtain and execute the tasks contained in the subtask 3, and after the execution is completed, puts the execution result into an appropriate position.

Through the above steps 409-423: the worker automatically fetches and puts subtasks. In this way, the entire task can be processed according to the logical relationship defined by the task. In addition, although the embodiment of FIG. 4 depicts 409-423 as being performed serially, the embodiment of the present invention is not limited thereto. In other embodiments, when the queues 1-3 do not need to be executed sequentially, the execution order of steps 409-413, steps 414-418, and steps 419-423 may be exchanged or overlapped. For example, if subtask 2 of queue 2 does not depend on the execution results of all subtasks 1 in queue 1, when workers in queue 1 are working, workers in queue 2 can also work without the worker in queue 1 executing all the subtasks. Task 1 can start the worker of queue 2.

424. The second-layer scheduler can obtain the progress information of the queue or worker, and judge the execution progress of the entire task.

425. If the execution of the task is completed, the second-level scheduler reports to the first-level scheduler. The second-level scheduler can also directly allow users to query the progress of tasks in real time.

In this way, the embodiment of the present invention adopts a two-tier scheduling architecture, the second-tier scheduler corresponds to the task, and the first-tier scheduler starts or selects the second-tier scheduler corresponding to the task, so that it can be applied to different tasks and improves the processing efficiency. Efficiency and scheduling flexibility, and can meet the needs of a variety of parallel processing business.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (15)

1. A distributed computing task processing system, characterized in that, comprising: The first-level scheduler is configured to receive a request for executing a task, start or select a second-level scheduler corresponding to the task, and forward the request to the second-level scheduler; The second-level scheduler is configured to decompose the task into multiple subtasks according to the logical relationship of the task when receiving the request forwarded by the first-level scheduler. 2. The system according to claim 1, wherein the logical relationship of the tasks indicates the execution dependencies of the plurality of subtasks. 3. The system according to claim 1 or 2, wherein the second-level scheduler is further configured to create corresponding queues for the multiple subtasks to store tasks included in the subtasks. 4. The system according to claim 3, wherein when the tasks contained in the subtasks are stored in the queue, the second-level scheduler is also used to apply for resources for the subtasks, and Instructing the work unit manager of the requested resource to start the work unit, so that the work unit obtains the tasks contained in the subtasks from the queue to execute the tasks. 5. The system according to claim 4, wherein the second-level scheduler is further configured to instruct the work unit to put the result of executing the task into another queue or output the result of executing the task. 6. The system according to claim 4 or 5, wherein the second-level scheduler is further configured to obtain progress information of the queue and the work unit, so as to determine the execution progress of the task. 7. The system according to any one of claims 2-6, wherein the execution dependencies of the plurality of subtasks include: two or more subtasks in the plurality of subtasks are executed in series or in parallel Execute sequentially. 8. The system according to any one of claims 1-7, wherein the first-level scheduler is also used to perform priority management on the tasks, and start or select the task according to the priority The second layer scheduler processes the tasks. 9. A distributed computing task processing method, characterized in that the method comprises: When the first-level scheduler receives a request to execute a task, it starts or selects a second-level scheduler corresponding to the task; forwarding the request to the second-tier scheduler by the first-tier scheduler; When the second-level scheduler receives the request forwarded by the first-level scheduler, it decomposes the task into multiple subtasks according to the logical relationship of the task. 10. The method according to claim 9, wherein the logical relationship of the tasks indicates the execution dependencies of the plurality of subtasks. 11. The method according to claim 9 or 10, further comprising: the second-level scheduler creating corresponding queues for the plurality of subtasks to store tasks included in the subtasks. 12. The method according to claim 11, further comprising: when the second-level scheduler stores tasks contained in the subtask in the queue, applying for the subtask resource, and instruct the work unit manager of the applied resource to start the work unit, so that the work unit obtains the tasks contained in the subtasks from the queue to execute the task. 13. The method according to claim 12, further comprising: the second-level scheduler instructing the work unit to put the execution task result into another queue or output the execution task the result of. 14. The method according to claim 12 or 13, further comprising: the second-level scheduler acquiring progress information of the queue and the work unit, so as to determine the execution progress of the task. 15. The method according to any one of claims 9-14, wherein the first-level scheduler also performs priority management on the tasks, and starts or selects the second task according to the priority. The layer scheduler processes the tasks.

Images