CN111290854A - Task management method, device and system, computer storage medium and electronic equipment - Google Patents

Abstract

The present disclosure provides a task management method, an apparatus, and a distributed task management system. The method is applied to a distributed task management system, and the distributed task management system includes a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server; the method includes: each task scheduling service monitors the parent node corresponding to the node. node; when monitoring the existence of the deleted first node in the node, create a second node corresponding to the first node through the target task scheduling service, and the target task scheduling service corresponding to the second node is different from the task corresponding to the first node. Scheduling service; the target task scheduling service obtains the task information corresponding to the first node from the database, and adds the task information to the task queue corresponding to the target task scheduling service. The present disclosure takes over the tasks of the faulty task scheduling service through the normal working task scheduling service, which ensures that the task is not lost and improves the availability of the task management system.

Description

Task management method, apparatus, system, computer storage medium and electronic device

technical field

The present disclosure relates to the technical field of cloud computing, and in particular, to a task management method, a task management device, a distributed task management system, a computer storage medium, and an electronic device.

Background technique

Task management systems are widely used in computing tasks that require a long execution time, such as offline speech recognition, big data statistics tasks, machine learning model training, etc. The task management system manages these computing tasks that require a long execution time and executes tasks. Issue, monitor task progress, store task status, etc.

The traditional task management system is mainly composed of a console, a task management service and a task execution service. Users send tasks to the task management service through the console, and the task management service sends the tasks to the task execution service. When the task execution service receives After the task starts to execute the task, and the task execution progress is updated to the database. And the task management service can display the task execution status to the console in real time for users to query. The traditional task management system has a simple architecture and is only suitable for processing atomic tasks, that is, the task itself only involves one task execution service. When a task consists of multiple subtasks with sequential order, these subtasks need to be composed of multiple different If the traditional task management system is used, the task execution efficiency will be reduced, and the availability of the task management service will be low.

It should be noted that the information disclosed in the above Background section is only for enhancement of understanding of the background of the present disclosure, and therefore may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

SUMMARY OF THE INVENTION

The embodiments of the present disclosure provide a task management method, a task management apparatus, a task management system, a computer storage medium and an electronic device, thereby ensuring that tasks are not lost at least to a certain extent and realizing high availability of the task management system.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or be learned in part by practice of the present disclosure.

According to an aspect of the embodiments of the present disclosure, a task management method is provided, the task management method is applied to a distributed task management system, and the distributed task management system includes a task scheduling server and a task scheduling server and the task scheduling server. Nodes corresponding to a plurality of task scheduling services, wherein the task scheduling services are used to manage and schedule tasks of users; the method includes: each of the task scheduling services monitors the parent node corresponding to the node; When there is a deleted first node in the node, a second node corresponding to the first node is created through the target task scheduling service, and the target task scheduling service corresponding to the second node is different from that corresponding to the first node. The target task scheduling service obtains the task information corresponding to the first node from the database, and adds the task information to the task queue corresponding to the target task scheduling service.

According to an aspect of the embodiments of the present disclosure, a task management apparatus is provided, the task management apparatus is applied to a distributed task management system, and the distributed task management system includes a task scheduling server and a task scheduling server and the task scheduling server. Nodes corresponding to multiple task scheduling services, wherein the task scheduling services are used to manage and schedule user tasks; the device includes: a monitoring module, used for each task scheduling service to monitor the parent node corresponding to the node The node creation module is used to create a second node corresponding to the first node through the target task scheduling service when monitoring the existence of the deleted first node in the node, and the target task corresponding to the second node The scheduling service is different from the task scheduling service corresponding to the first node; the task takeover module is used for the target task scheduling service to obtain the task information corresponding to the first node from the database, and add the task information into the task queue corresponding to the target task scheduling service.

In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus further includes: a registration module, configured to register in the distributed coordination service system through each of the task scheduling services, so as to obtain information related to each of the tasks Nodes with different node numbers corresponding to the scheduling service.

In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus further includes: an identification modification module, configured to modify the task scheduling service identification corresponding to the task information to a task corresponding to the target task scheduling service Dispatch service ID.

In some embodiments of the present disclosure, the distributed task management system further includes a task management server and a task execution server; based on the foregoing solution, the task management apparatus further includes: a request generation module, configured to use the task scheduling server Receive the task to be executed sent by the task management server, and form a task execution request according to the to-be-executed task; a task execution module is configured to send the task execution request to the task execution server, so that the task is executed The task execution service in the server executes the task.

In some embodiments of the present disclosure, based on the foregoing solution, the request generation module includes: a task adding unit, configured to add the to-be-executed task to a task queue corresponding to the task scheduling service, and determine the Whether there is an idle thread in the thread pool of the task scheduling service; the task pulling unit is used to pull the to-be-executed task from the task queue corresponding to the task scheduling service when the idle thread exists, and obtain all the A subtask in the task to be executed; a request generating unit, configured to form the task execution request according to the attribute information of the subtask and the subtask.

In some embodiments of the present disclosure, based on the foregoing solution, the request generating unit is configured to: determine whether the subtask is a subtask to be executed; when it is determined that the subtask is a subtask to be executed, according to the subtask to be executed Executing a subtask forms the task execution request.

In some embodiments of the present disclosure, the number of the task execution services is multiple, and each task execution service has a different task type label; based on the foregoing solution, the task execution module is configured to: Matching the task type of the task to be executed in the execution request with the task type label to determine the target task execution service; sending the task execution request to the target task execution service, so that the target task execution service executes the task .

In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus is further configured to: query whether the execution of the subtask to be executed is completed; if the execution of the subtask to be executed is completed, update the database in the Task progress information corresponding to the task to be executed; if the execution of the subtask to be executed is not completed, the task to be executed is re-added to the task queue corresponding to the task scheduling service.

In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus is further configured to: determine whether there is an unexecuted subtask in the to-be-executed task; if there is, the unexecuted subtask is to be executed The unexecuted subtask is executed when the subtask is executed until there is no unexecuted subtask in the to-be-executed task; if there is no unexecuted subtask, the task status information corresponding to the to-be-executed task in the database is updated.

In some embodiments of the present disclosure, based on the foregoing solution, the task management apparatus is further configured to: each of the task scheduling services update the number of tasks in the node according to the number of tasks in the task queue corresponding to each of the task scheduling services The task amount; the task management server obtains the task amount from each of the nodes, determines the task allocation weight according to the task amount, and performs task allocation based on the task allocation weight.

In some embodiments of the present disclosure, the distributed task management system further includes a console; based on the foregoing solution, the task management apparatus is further configured to: the console responds to a user's first trigger operation to enter the to-be-to-be Execute the task; or, the console generates a task progress query request in response to the user's second trigger operation, and sends the task progress query request to the task management server to obtain task progress information.

According to an aspect of the embodiments of the present disclosure, a distributed task management system is provided, which is characterized by comprising: a console for generating a task issuing instruction or a task progress query request in response to a user's trigger operation; a task management server , connected with the console, and used to issue a task in response to the task issuing instruction or in response to the task progress query request to feed back task progress information to the console; a task scheduling server, connected with the task management server , including a plurality of task scheduling services, each of which is registered in the distributed coordination service system to form a node for receiving tasks issued by the task management server, and dispatches and manages the tasks through each of the task scheduling services. a task execution server, connected to the task scheduling server, and comprising a plurality of task execution services for executing tasks in response to a task execution request sent by the task scheduling service; a data storage device, connected with the task management server and the task management server and The task scheduling server is connected to store information related to the task.

According to an aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the task management method described in the foregoing embodiments.

According to an aspect of the embodiments of the present disclosure, there is provided an electronic device including one or more processors; and a storage device for storing one or more programs, when the one or more programs are stored by the one or more programs When executed by multiple processors, the one or more processors are caused to execute the task management method described in the above-mentioned embodiments.

In the technical solutions provided by the embodiments of the present disclosure, multiple task scheduling services in the task management system are registered in the distributed coordination service framework to form nodes corresponding to each task scheduling service, that is, the task management The system is a multi-node distributed task management system. Each task scheduling service monitors the parent node corresponding to the node. When there is a deleted first node, a second node corresponding to the first node is created through the target task scheduling service. At the same time, the target task scheduling service obtains the first node from the database. The task information corresponding to the node is added to the task queue corresponding to the target task scheduling service. The technical solution of the present disclosure can take over the tasks of the faulty task scheduling service through other normally working task scheduling services after a certain task scheduling service fails, ensuring that tasks are not lost and improving the availability of the task management system.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort. In the attached image:

FIG. 1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present disclosure can be applied;

FIG. 2 schematically shows a schematic diagram of the architecture of a conventional task management system in the related art;

3 schematically shows a schematic flowchart of a task management method according to an embodiment of the present disclosure;

FIG. 4 schematically shows a schematic diagram of the architecture of a distributed task management system according to an embodiment of the present disclosure;

5 schematically shows a distributed architecture diagram between task scheduling services in a distributed task processing system according to an embodiment of the present disclosure;

6 schematically shows a schematic structural diagram of task information stored in a data storage device according to an embodiment of the present disclosure;

FIG. 7 schematically shows a relationship diagram between task scheduling services when the task scheduling service fails according to an embodiment of the present disclosure;

FIG. 8 schematically shows a relationship diagram between task scheduling services after task recovery according to an embodiment of the present disclosure;

9 schematically shows a schematic structural diagram of restored task information stored in a data storage device according to an embodiment of the present disclosure;

FIG. 10 schematically shows a schematic diagram of an execution flow of a task scheduling service according to an embodiment of the present disclosure;

FIG. 11 schematically shows a block diagram of a task management apparatus according to an embodiment of the present disclosure;

FIG. 12 shows a schematic structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

The block diagrams shown in the figures are merely functional entities and do not necessarily necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices entity.

The flowcharts shown in the figures are only exemplary illustrations and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can be decomposed, and some operations/steps can be combined or partially combined, so the actual execution order may be changed according to the actual situation.

FIG. 1 shows a schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of the present disclosure may be applied.

As shown in FIG. 1 , the system architecture 100 may include a task management front end 101 , a network 102 and a task management back end 103 . The network 102 is the medium used to provide the communication link between the task management front end 101 and the task management back end 103 . The network 102 may include various connection types, such as wired communication links, wireless communication links, and the like.

It should be understood that the numbers of task management front ends, networks and task management back ends in FIG. 1 are merely illustrative. There can be any number of task management front-ends, networks, and task management back-ends according to actual needs. For example, the task management backend 103 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, and network services. , cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The task management front end 101 may be a terminal device such as a notebook computer, a tablet computer, a desktop computer, etc., but is not limited thereto.

In an embodiment of the present disclosure, a user can enter task information in the task management front end 101 or send a task progress query request through the task management front end 101 , and the task information and task progress query request are sent to the task management back end 103 through the network 102 . The task management backend 103 includes a task management server, a task scheduling server, and a task execution server, wherein the task scheduling server includes a plurality of task scheduling services for managing and scheduling tasks of users; the task execution server includes a plurality of task execution services, which are used for to perform the tasks received by the user. After receiving the task issued by the task management front end 101, the task management server sends the task to the task scheduling server, schedules the task through multiple task scheduling services, and invokes the corresponding task execution service to execute each task. In the process of scheduling tasks by the task scheduling service, if a task scheduling service fails, the distributed coordination service system can take over the tasks scheduled by the faulty task scheduling service through the non-faulty task scheduling service to ensure that the task is not lost. In addition, when the task management server sends tasks to the task scheduling server, it can dynamically adjust the weight of the tasks to be delivered according to the task volume in the task queue corresponding to each task scheduling service, so as to ensure that the tasks are evenly distributed to each task scheduling service. The technical solutions of the embodiments of the present disclosure can set up multiple task scheduling services and nodes corresponding to the task scheduling services in the task management system to form a distributed task management system, in which multiple tasks can be executed simultaneously, The task execution efficiency is improved. In addition, the distributed task management system has the characteristics of service self-registration and fault self-healing, which ensures that tasks are not lost, and the weighted task issuance can ensure the smooth operation of the distributed task management system, further improving High availability of the task management system.

It should be noted that, the task management method provided by the embodiments of the present disclosure is generally executed by a server, and accordingly, a task management apparatus is generally set in the server. However, in other embodiments of the present disclosure, the task management method provided by the embodiments of the present disclosure may also be executed by the terminal device.

In the related art in the art, FIG. 2 shows an architecture diagram of a traditional task management system. As described in FIG. 2 , the traditional task management system includes a console 201 , a task management server 202 , a task execution server 203 and a database 204 The specific task execution process is as follows: the user sends the task to the task management server 202 through the console 201; the task management server 202 stores the task information in the database 204, and simultaneously issues the task to the task execution server 203; the task execution server 203 receives After arriving at the task, start to execute the task, and periodically update the task execution progress to the database 204; the user sends a task progress query request to the task management server 202 through the console 201, and the task management server 202 responds to the task progress query request, will The task execution status in the database 204 is displayed in the console 201 in real time for the user to browse.

However, the traditional task management system has a simple structure and is suitable for processing atomic tasks. If a task consists of multiple sub-tasks with sequential order, and these sub-tasks are executed by multiple different task execution services, in order to make the task The management system has high availability, so it is necessary to set the task management service to multiple nodes, and manage tasks through different nodes. However, there are corresponding problems, such as how to coordinate and distribute tasks among multiple nodes, how to deal with the tasks of a node when a node fails, and so on.

In view of the problems existing in the related technologies, the embodiments of the present disclosure provide a task management method, which is based on cloud technology, and cloud technology (Cloud technology) is based on network technology, information technology, integration technology, The general term for management platform technology, application technology, etc., which can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. Background services of technical network systems require a lot of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the back-end system for logical processing. Data of different levels will be processed separately, and all kinds of industry data need to be strong. The system backing support can only be achieved through cloud computing.

Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space and information services as needed. The network that provides the resources is called the "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for according to usage.

As a basic capability provider of cloud computing, it will establish a cloud computing resource pool (referred to as cloud platform, generally called IaaS (Infrastructure as a Service) platform, and deploy various types of virtual resources in the resource pool for External customers choose to use. The cloud computing resource pool mainly includes: computing devices (which are virtualized machines, including operating systems), storage devices, and network devices.

According to the division of logical functions, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. service) layer, or directly deploy SaaS on IaaS. PaaS is a platform on which software runs, such as databases and web containers. SaaS is a variety of business software, such as web portals, SMS group senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

Cloud processing is usually used in the field of big data. Big data refers to the collection of data that cannot be captured, managed and processed by conventional software tools within a certain time frame. It requires new processing modes to have stronger decision-making power, Massive, high-growth and diverse information assets for insight discovery and process optimization capabilities. With the advent of the cloud era, big data is also attracting more and more attention, and big data requires special technologies to efficiently process a large amount of data that tolerates elapsed time. Technologies applicable to big data, including massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

An embodiment of the present disclosure first provides a task management method. FIG. 3 schematically shows a flowchart of a task management method according to an embodiment of the present disclosure. The task management method may be implemented by a task management front-end and a task management back-end. Execution, specifically, can be performed by a terminal device and a server. In addition, the task management method is applied to a distributed task management system, and the distributed task management system includes a task scheduling server and a plurality of task scheduling services in the task scheduling server. Node, where the task scheduling service is used to manage and schedule user tasks. Referring to FIG. 3 , the task management method includes at least steps S310 to S330, which are described in detail as follows:

In step S310, each task scheduling service monitors the parent node corresponding to the node.

In an embodiment of the present disclosure, the task management system is a distributed task management system, the distributed task management system includes a task scheduling server, the task scheduling server includes multiple task scheduling services, and the multiple task scheduling services form a task The cluster is scheduled, and each task scheduling service is registered in the distributed coordination service system to form a node. FIG. 4 shows a schematic diagram of the architecture of the distributed task management system. As shown in FIG. 4 , the distributed task management system 400 includes a console 401 , a task management server 402 , a task scheduling server 403 , a task execution server 404 and a data storage device 405 , where the console 401 may be a Web console specifically, and is used to generate a task issuing instruction or a task progress query request in response to a user's trigger operation, that is, the user can perform task issuing, task status monitoring and Task execution result query and other operations; the task management server 402 is also called the task management background, which is the background service of the console 401 and is connected to the console 401 to issue tasks or respond to control in response to the task issuing instructions sent by the console 401 The task progress query request sent by the console 401 is used to feed back the task progress information to the console 401; the task scheduling server 403 is connected to the task management server 402, and includes a task scheduling service cluster, and the task scheduling service cluster includes a plurality of task scheduling services, wherein each The task scheduling service is registered in the distributed coordination service framework to form a node, and each task scheduling service receives the task issued by the task management device 402 and is responsible for scheduling the entire life cycle of the task. The execution progress is updated in the data storage device 405; the task execution server 404 is connected to the task scheduling server 403, and includes a task execution service cluster, and the task execution service cluster includes a plurality of task execution services, and each task execution service is an atomic calculation The task execution service receives subtask execution requests submitted by the task scheduling service, and executes corresponding computing tasks; the data storage device 405 may be a hard disk or other media that can be used to store data, and is used to store the tasks submitted by the task management server 402 and the task scheduling server 403. The data. In the embodiment of the present disclosure, each task execution service has different task type labels, and the user will mark the type of each task when entering a task through the console 401. When the task execution server 404 receives a task, it can label the task type. The type of the task execution service is matched with the task type label of the task execution service to determine the matching task execution service, and execute the task through the matching task execution service.

In an embodiment of the present disclosure, the task management server 402, the task scheduling server 403, and the task execution server 404 may be the same server or different servers, which are not specifically limited in this embodiment of the present disclosure.

In an embodiment of the present disclosure, the distributed task management system is constructed based on a distributed coordination service system, and each task scheduling service is registered in the distributed coordination service system to form a node corresponding to each task scheduling service. Each node corresponds to a different node number. In the embodiment of the present disclosure, the distributed coordination service system may specifically be zookeeper. Zookeeper is a sub-project of Apache Hadoop. It is mainly used to solve some data management problems often encountered in distributed applications, such as: unified Naming service, state synchronization service, cluster management, management of distributed application configuration items, etc. Zookeeper has an event monitoring function, which allows users to register some listener watchers on specified nodes, and when some specific events are triggered, the zookeeper server will notify interested clients of the event. The structure of the zookeeper data model can be regarded as a tree as a whole, and each node is called a Znode. Figure 5 shows the distributed architecture diagram between the task scheduling services in the distributed task management system, as shown in Figure 5, There are N task scheduling services Broker1, Broker2, ..., BrokerN in the distributed task management system. Each task scheduling service is registered in zookeeper to form nodes corresponding to each task scheduling service. These nodes are /broker/ 1. /broker/2, ..., /broker/N, these nodes are all child nodes of the node /brokers, that is to say, /brokers is the parent node of the node corresponding to each task scheduling service. After the tree structure is formed, each task scheduling service can register listeners on its nodes to monitor the parent node/brokers. When the information in the parent node or child node changes, zookeeper can notify each task of the change information. Dispatch service.

In an embodiment of the present disclosure, the information of the task scheduling service and its corresponding nodes can be stored in the database, that is, the data storage device 405, so that the task management server 402 can obtain relevant information therefrom, and when the task scheduling service updates the task progress , you can also update your own node serial number to the data storage device 405. Figure 6 shows a schematic structural diagram of the task information stored in the data storage device. As shown in Figure 6, there are multiple tasks, and each task has Corresponding task ID, task status, task scheduling service ID of the task scheduling service, and other information, for example, the task status corresponding to the task whose task ID is 1 is Step1, the task scheduling service ID of the task scheduling service is 1; the task ID is 2 The task status corresponding to the task of the task scheduling service is Step2, and the task scheduling service of the task scheduling service is 2; the task status corresponding to the task whose task ID is 3 is Complete, and the task scheduling service of the task scheduling service is N. It should be noted that the task ID, task status and task scheduling service identifier may also be in other representation forms, including but not limited to the forms in the above examples, which are not specifically limited in this embodiment of the present disclosure.

In an embodiment of the present disclosure, when a task scheduling service fails and hangs up, the node of the faulty task scheduling service will be automatically deleted, and the corresponding task may not be executed smoothly, thereby causing the task execution to fail and reducing the user experience.

In step S320, when it is detected that the deleted first node exists in the node, a second node corresponding to the first node is created through the target task scheduling service, and the target task scheduling service corresponding to the second node Different from the task scheduling service corresponding to the first node.

In an embodiment of the present disclosure, in order to ensure that all tasks can be executed smoothly, when there is a task scheduling service that fails and hangs up, the normal working task scheduling service can take over the corresponding tasks, and perform the tasks in order. Execute all tasks in the task queue. The specific process of taking over the task of the task scheduling service that fails and hangs up is as follows: when monitoring the existence of the deleted first node in the node, the target task scheduling is determined by the competition of other task scheduling services except the suspended task scheduling service. service; then create a second node corresponding to the first node in zookeeper through the target task scheduling service. It is a feature of zookeeper to uniquely determine the target task scheduling service from multiple competing task scheduling services. As for how to compete and determine the target task scheduling service, the embodiment of the present disclosure will not repeat them here. Figure 7 shows the relationship between the task scheduling services when the task scheduling service fails. As shown in Figure 7, the task scheduling service corresponding to the number /brokers/2 fails and hangs up, so the number is /brokers/2 node is deleted. Then a new node can be registered through the target task scheduling service to generate a new node corresponding to the node numbered /brokers/2.

In an embodiment of the present disclosure, the number of the first node and the number of the second node may not be exactly the same. For example, the number of the second node may be set to /brokers/N-r, for example, the node numbered /brokers/2 corresponds to If the task scheduling service fails and hangs up, the node is deleted, then the target task scheduling service can take over the task of the task scheduling service and register the node corresponding to /brokers/2 on zookeeper. The number of the newly registered node can be Set to /brokers/2-r; you can also set the number of the second node to other forms based on the number of the first node, of course, you can also set the number of the first node and the number of the second node to a completely different form, The embodiments of the present disclosure do not specifically limit this.

In step S330, the target task scheduling service acquires task information corresponding to the first node from a database, and adds the task information to a task queue corresponding to the target task scheduling service.

In an embodiment of the present disclosure, after the second node corresponding to the first node is successfully created, the task information corresponding to the first node can be obtained from the data storage device 405 through the target task scheduling service, and the task information Add to the task queue corresponding to the target task scheduling service. In the embodiment of the present disclosure, each task scheduling service corresponds to a task queue respectively, and after each task scheduling service receives the task issued by the task management server 402, it can add the received task to its corresponding task queue, In this way, the task processing efficiency can be improved, and the high availability of the task processing system can be ensured.

Further, after adding the task information to the task queue corresponding to the target task scheduling service, the task scheduling service identifier corresponding to the task information in the data storage device 405 can also be modified to the task scheduling service corresponding to the target task scheduling service. logo.

Continuing to take the example in step S320 as an example, when other task scheduling services monitor that the node numbered /brokers/2 is deleted, they compete to create a new node corresponding to the node numbered /brokers/2. After being the node of /brokers/2-r, the target task scheduling service can obtain the task information of all Broker2 from the database, put the task information into its own task queue, and store the data storage device corresponding to the task information. The task scheduling service ID of is modified to the task scheduling service ID corresponding to the target task scheduling service. Figure 8 shows the relationship between the task scheduling services after task recovery. As shown in Figure 8, the task scheduling service Broker1 is the target task scheduling service determined after the competition. It is registered in zookeeper to form a new node. is numbered /brokers/2-r, which corresponds to the deleted node numbered /brokers/2. Meanwhile, FIG. 9 shows a schematic structural diagram of the restored task information stored in the data storage device. As shown in FIG. 9 , the task statuses corresponding to the tasks with the original task IDs 1 and 3 are changed to Running, and the task is scheduled The task scheduling service identifier of the service remains unchanged, while the task status corresponding to the task whose original task ID is 2 becomes Running, and the task scheduling service identifier of the task scheduling service changes from 2 to 1.

In an embodiment of the present disclosure, information such as task status information and task scheduling service identifiers corresponding to tasks are regularly updated to the data storage device. In addition, a new task scheduling service can be created and registered in zookeeper to form a corresponding node. The task scheduling service identifier and the corresponding node information corresponding to the newly created task scheduling service will also be updated to the data storage device for the task management server to acquire.

In an embodiment of the present disclosure, based on the system architecture diagram of the distributed task management system shown in FIG. 4 , the task management server 402 can deliver the tasks to be executed sent by the user to each task scheduling service in the task scheduling server 403 , after the task scheduling service receives the task to be executed sent by the task management server 402, it can form a task execution request according to the task to be executed, and then send the task execution request to the corresponding task execution service in the task execution server 404, so that the task execution service perform tasks.

In an embodiment of the present disclosure, the execution process of each task scheduling service is the same. In order to make the technical solution of the present disclosure clearer, the following uses a task scheduling service as an example to describe the execution process of the task scheduling service.

Figure 10 shows a schematic diagram of the execution flow of the task scheduling service. As shown in Figure 10, in step S1001, the task to be executed is added to the task queue corresponding to the task scheduling service, and it is determined whether the thread pool of the task scheduling service is in the thread pool. There is an idle thread; in step S1002, when there is an idle thread, pull the task to be executed from the task queue corresponding to the task scheduling service, and obtain the subtask in the task to be executed; in step S1003, determine the subtask Whether it is a subtask to be executed; in step S1004, when it is determined that the subtask is a subtask to be executed, a task execution request is formed according to the subtask to be executed; in step S1005, the task type of the task to be executed in the task execution request Match with the task type label to determine the target task execution service; in step S1006, send the task execution request to the target task execution service, so that the target task execution service execution task; in step S1007, query whether the subtask to be executed is The execution is completed; in step S1008, if the execution of the subtask to be executed is completed, the task progress information corresponding to the task to be executed in the database is updated; in step S1009, if the execution of the subtask to be executed is not completed, then the task to be executed is renewed. Add to the task queue corresponding to the task scheduling service; in step S1010, determine whether there is an unexecuted subtask in the to-be-executed task; in step S1011, if there is, execute when the unexecuted subtask is a to-be-executed subtask The unexecuted subtask is until there is no unexecuted subtask in the to-be-executed task; in step S1012, if there is no unexecuted subtask, update the task status information corresponding to the to-be-executed task in the data storage device.

Before step S1001, after the task scheduling service receives the to-be-executed task, it can put it into the task queue corresponding to the task scheduling service, and the to-be-executed tasks in the task queue can be sorted according to the order of reception time. Earlier tasks to be executed can be located at the front of the task queue, and tasks to be executed with a later reception time can be located at the back end of the task queue. Of course, the tasks to be executed in the task queue can also be sorted according to other rules, such as according to the tasks to be executed. Priority and the like are sorted, which is not specifically limited in this embodiment of the present disclosure. In step S1003, after the idle thread in the task scheduling service obtains a scheduled task to be executed, since the task to be executed contains multiple subtasks, the task scheduling service needs to obtain the attribute information of the subtask, and then according to the subtask's attribute information Attribute information and subtasks form a task execution request. Obtaining attribute information of subtasks includes determining how many subtasks are included in the pulled task to be executed, and whether the subtasks are subtasks to be executed. For example, a task to be executed contains 5 subtasks. The subtasks arranged in sequence are subtasks No. 1, No. 2, No. 3, No. 4, and No. 5. When the task scheduling service pulls the subtask No. 1, it needs to obtain the execution status of the subtask, and according to the subtask The execution status judges whether it is a subtask to be executed. For example, subtask 1 and subtask 2 in the task to be executed have been executed. When querying whether the execution of subtask 2 is completed, the query result is that the execution is not completed. The task to be executed is put back into the task queue, so the execution status of subtask No. 1 obtained by the task scheduling service is that subtask No. 1 has been executed. Correspondingly, subtask No. 1 is not a subtask to be executed. The execution status of the remaining subtasks is judged to determine whether these subtasks are subtasks to be executed. In step S1007, the query of the execution state of the subtask to be executed may be performed at a preset time point, for example, the query is performed immediately after the task execution service receives and executes the subtask to be executed, and the query is performed at the 3s and 5s after execution, etc., If the execution status of the to-be-executed subtask queried at the preset time point is not completed, the to-be-executed task is put back into the tail of the task queue for sorting. In step S1008, the task progress information is the execution progress of the subtask to be executed. For example, Step1, Step2, Running, etc. in FIG. 5 and FIG. In S1012, the task status information is the execution result of the task to be executed. For example, Complete in FIG. 5 indicates that the execution of the to-be-executed task is completed. Of course, other identifications may also be used, such as successful execution, waiting for re-execution and so on.

In an embodiment of the present disclosure, after the execution of the entire task to be executed is completed, the task execution result, that is, the task status information, is updated to the data management device 405, and then steps S1001-S1012 are repeatedly executed to execute other tasks in the task queue. task to be performed.

In an embodiment of the present disclosure, the task management server issues tasks to the task scheduling service in the task scheduling server. In order to ensure balanced assignment of tasks, it is avoided that some task scheduling services have an excessively high load and some task scheduling services have an excessively low load. , the task delivery amount of each task scheduling service can be dynamically adjusted according to the task amount corresponding to the task scheduling service. Each task scheduling service in the task scheduling server has its corresponding task queue, and the number of tasks in the task queue reflects the load of the task scheduling service itself. The task scheduling service periodically updates the task volume in the nodes corresponding to each task scheduling service according to the number of tasks in the respective task queues, and then the task management server can obtain the task volume of all nodes from zookeeper, and determine the task allocation according to the task volume. weight, and further perform task assignment based on the task assignment weight. For example, the task scheduling service cluster includes three task scheduling services A, B, and C. The number of tasks stored in the node corresponding to the task scheduling service A is 10, and the number of tasks stored in the node corresponding to the task scheduling service B is 5 , the number of tasks stored in the node corresponding to the task scheduling service C is 15, then it can be determined that the task allocation weight corresponding to the task scheduling service B is the largest, the task allocation weight corresponding to the task scheduling service A is second, and the task scheduling service The task assignment weight corresponding to C is the smallest, and the specific task assignment weights are 5/12, 1/3, and 1/4. If the task management background needs to deliver 60 tasks to be executed, it can assign 25 tasks to the task scheduling service B. , 20 tasks are allocated to task scheduling service A and 15 tasks are allocated to task scheduling service C, so that the number of tasks in task scheduling service A, task scheduling service B and task scheduling service C can all be guaranteed to be 30, which ensures the task allocation balance.

In an embodiment of the present disclosure, only the console 401 can be contacted by the user, and the internal details of the distributed task management system 400 are invisible to the user, so the user can only perform tasks by performing different trigger operations on the console Delivery, task status monitoring, and task execution result query. Specifically, the console 401 may enter the task to be executed in response to the user's first trigger operation; or, the console 401 may generate a task progress query request in response to the user's second trigger operation, and send the task progress query request to the task management server 402 , after the task management server 402 receives the task progress query request, it can obtain the task status of the task corresponding to the task ID in the task progress query request from the data storage device 405, and feed back the task status and display it on the console In the display interface, it is convenient for users to browse.

The task management method in the embodiment of the present disclosure can be applied to multiple scenarios, especially computing tasks that require a long execution time, such as offline speech recognition, big data statistics tasks, machine learning model training, and so on.

Taking the big data statistical task as an example, suppose a task is to push advertisements to tourists from other places in the scenic spot, then firstly, the crowd located in the scenic spot can be obtained. Specifically, the geographical coordinates submitted by the positioning device carried by the tourists can be stored in the database. The data determines the composition of the crowd in the scenic spot, that is, the tourists included in the crowd; then it can determine which tourists are out-of-town tourists based on the pages recently clicked by each tourist, chat content, location sharing information and other information; tourists, build user portraits based on their user information, browsing information and other information; finally, based on the user portraits of each tourist, relevant advertisements can be pushed to each tourist. For example, if some users frequently browse train and air ticket booking platforms, they can send The user pushes advertisement information of trains and air tickets, and some users frequently browse the food recommendation platform, so food advertisements can be pushed to the user, and so on. In the process of pushing advertisements, there are three subtasks. The first subtask is to filter crowd data, the second subtask is to build user portraits of foreign tourists, and the third subtask is to place advertisements based on user portraits. These three subtasks According to the task management method of the embodiment of the present disclosure, the task and related subtasks can be entered in the console by the user, and then the task and related subtasks can be sent to the task scheduling server through the task management server. The task scheduling server schedules the task and sends a task execution request to the task execution service, so that the task execution service executes each subtask to complete the task. When performing task management through the task management system in the embodiment of the present disclosure, even if some subtasks of the three subtasks are executed and some subtasks are not completed, only the unfinished subtasks can be executed when the task is repeatedly executed, and the There is no need to repeat the completed subtasks, which improves the efficiency of task execution.

In one embodiment of the present disclosure, the task management system is a distributed task management system, which can schedule multiple tasks through a task scheduling server, and each task scheduling service can completely schedule the entire life cycle of multiple tasks, ensuring that High availability of the task management system.

The task management method in the embodiment of the present disclosure combines a task management system with a distributed coordination service system, and sets a plurality of task scheduling services in the task management system, and each task scheduling service is registered in the distributed coordination service system to form a Nodes, and then form a multi-node distributed task management system. Each task scheduling service in the distributed task management system can completely schedule the entire life cycle of multiple tasks. When an individual task scheduling service fails and hangs up, Other normally working task scheduling services can compete to take over the tasks corresponding to the failed task scheduling services, which realizes service self-registration and fault self-healing, ensures that tasks are not lost, and ensures the normal service of the entire distributed task management system. At the same time, there are multiple task scheduling services and multiple task execution services in the distributed task management system. Different task execution services can perform different tasks, thus improving the task execution efficiency. In addition, the distributed task management system has weighted tasks under the It can balance the load of each task scheduling service, prevent the task scheduling service failure caused by the high load of individual task scheduling service, and further ensure the normal service of the distributed task management system.

The apparatus embodiments of the present disclosure are introduced below, which can be used to execute the task management methods in the above-mentioned embodiments of the present disclosure. For details not disclosed in the embodiments of the apparatus of the present disclosure, please refer to the above-mentioned task management method of the present disclosure.

FIG. 11 schematically shows a block diagram of a task management apparatus according to an embodiment of the present disclosure.

Referring to FIG. 11 , according to a task management apparatus 1100 according to an embodiment of the present disclosure, the task management apparatus is applied to a distributed task management system, and the distributed task management system includes a task scheduling server and a task scheduling server and a task scheduling server. Nodes corresponding to multiple task scheduling services, wherein the task scheduling services are used to manage and schedule user tasks; the task management device 1100 includes: a monitoring module 1101 , a node creation module 1102 and a task takeover module 1103 .

Among them, the monitoring module 1101 is used for each of the task scheduling services to monitor the parent node corresponding to the node; the node creation module 1102 is used for monitoring the existence of the deleted first node in the node, through the target task The scheduling service creates a second node corresponding to the first node, and the target task scheduling service corresponding to the second node is different from the task scheduling service corresponding to the first node; the task takeover module 1103 is used for the target The task scheduling service acquires task information corresponding to the first node from the database, and adds the task information to the task queue corresponding to the target task scheduling service.

In an embodiment of the present disclosure, the task management apparatus 1100 further includes: a registration module, configured to register in the distributed coordination service system through each of the task scheduling services, so as to obtain information corresponding to each of the task scheduling services of nodes with different node numbers.

In an embodiment of the present disclosure, the task management apparatus 1100 further includes: an identification modification module, configured to modify the task scheduling service identification corresponding to the task information to the task dispatching service identification corresponding to the target task dispatching service .

In an embodiment of the present disclosure, the distributed task management system further includes a task management server and a task execution server; the task management apparatus 1100 further includes: a request generation module, configured to receive the task scheduling server through the task scheduling server. The task to be executed sent by the task management server, and a task execution request is formed according to the to-be-executed task; the task execution module is configured to send the task execution request to the task execution server, so that the tasks in the task execution server are executed. The task execution service executes the task.

In an embodiment of the present disclosure, the request generating module includes: a task adding unit, configured to add the to-be-executed task to a task queue corresponding to the task scheduling service, and determine the Whether there is an idle thread in the thread pool; the task pulling unit is used to pull the to-be-executed task from the task queue corresponding to the task scheduling service when the idle thread exists, and obtain the to-be-executed task A subtask in ; a request generating unit, configured to form the task execution request according to the attribute information of the subtask and the subtask.

In an embodiment of the present disclosure, the request generating unit is configured to: determine whether the subtask is a subtask to be executed; when it is determined that the subtask is a subtask to be executed, form a subtask according to the subtask to be executed the task execution request.

In an embodiment of the present disclosure, the number of the task execution services is multiple, and each of the task execution services has a different task type label; the task execution module is configured to: The task type of the execution task is matched with the task type label to determine the target task execution service; the task execution request is sent to the target task execution service, so that the target task execution service executes the task.

In an embodiment of the present disclosure, the task management apparatus 1100 is further configured to: query whether the execution of the subtask to be executed is completed; if the execution of the subtask to be executed is completed, update the Task progress information corresponding to the task to be executed; if the execution of the subtask to be executed is not completed, the task to be executed is re-added to the task queue corresponding to the task scheduling service.

In an embodiment of the present disclosure, the task management apparatus 1100 is further configured to: determine whether there is an unexecuted subtask in the to-be-executed task; if so, when the unexecuted subtask is a to-be-executed subtask Execute the unexecuted subtask until there is no unexecuted subtask in the to-be-executed task; if there is no unexecuted subtask, update the task status information corresponding to the to-be-executed task in the database.

In an embodiment of the present disclosure, the task management apparatus 1100 is further configured to: each of the task scheduling services update the task amount in the node according to the number of tasks in the task queue corresponding to each of the task scheduling services; The task management server acquires the task amount from each of the nodes, determines a task assignment weight according to the task amount, and performs task assignment based on the task assignment weight.

In an embodiment of the present disclosure, the distributed task management system further includes a console; the task management apparatus 1100 is further configured to: the console responds to a user's first trigger operation to input the to-be-executed task; Alternatively, the console generates a task progress query request in response to the user's second trigger operation, and sends the task progress query request to the task management server to obtain task progress information.

FIG. 12 shows a schematic structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure.

It should be noted that the computer system 1200 of the electronic device shown in FIG. 12 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 12, the computer system 1200 includes a central processing unit (Central Processing Unit, CPU) 1201, which can be loaded into a random device according to a program stored in a read-only memory (Read-Only Memory, ROM) 1202 or from a storage part 1208 The program in the memory (Random Access Memory, RAM) 1203 is accessed to execute various appropriate actions and processes, so as to realize the image labeling method described in the above embodiments. In the RAM 1203, various programs and data necessary for system operation are also stored. The CPU 1201 , the ROM 1202 , and the RAM 1203 are connected to each other through a bus 1204 . An Input/Output (I/O) interface 1205 is also connected to the bus 1204 .

The following components are connected to the I/O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc. ; a storage section 1208 including a hard disk and the like; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1209 performs communication processing via a network such as the Internet. Drivers 1210 are also connected to I/O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

In particular, according to embodiments of the present disclosure, the processes described below with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 1209, and/or installed from the removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present disclosure are executed.

It should be noted that the computer-readable medium shown in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Erasable Programmable Read Only Memory (EPROM), flash memory, optical fiber, portable Compact Disc Read-Only Memory (CD-ROM), optical storage device, magnetic storage device, or any suitable of the above The combination. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The units involved in the embodiments of the present disclosure may be implemented in software or hardware, and the described units may also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.

As another aspect, the present disclosure also provides a computer-readable medium. The computer-readable medium may be included in the task management apparatus described in the above-mentioned embodiments; it may also exist alone without being assembled into the electronic device. in the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, enables the electronic device to implement the methods described in the above-mentioned embodiments.

It should be noted that although several modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.

From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein may be implemented by software, or may be implemented by software combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.) or on the network , which includes several instructions to cause a computing device (which may be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to an embodiment of the present disclosure.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common general knowledge or techniques in the technical field not disclosed by this disclosure .

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (15)

1. A task management method is characterized in that the task management method is applied to a distributed task management system, the distributed task management system comprises a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server, wherein the task scheduling services are used for managing and scheduling tasks of users; the method comprises the following steps:

each task scheduling service monitors a father node corresponding to the node;

when the deleted first node exists in the nodes, creating a second node corresponding to the first node through a target task scheduling service, wherein the target task scheduling service corresponding to the second node is different from the task scheduling service corresponding to the first node;

and the target task scheduling service acquires task information corresponding to the first node from a database and adds the task information to a task queue corresponding to the target task scheduling service.

2. The task management method according to claim 1, characterized in that:

and registering each task scheduling service in the distributed coordination service system to acquire nodes with different node numbers corresponding to each task scheduling service.

3. The method of claim 1, wherein after adding the task information to a task queue corresponding to the target task scheduling service, the method further comprises:

and modifying the task scheduling service identifier corresponding to the task information into a task scheduling service identifier corresponding to the target task scheduling service.

4. The task management method according to claim 1, wherein the distributed task management system further comprises a task management server and a task execution server;

the task scheduling server receives a task to be executed sent by the task management server and forms a task execution request according to the task to be executed;

and sending the task execution request to the task execution server so that the task execution service in the task execution server executes the task.

5. The task management method according to claim 4, wherein the task scheduling server receives the to-be-executed task sent by the task management server and forms a task execution request according to the to-be-executed task, and the task execution request includes:

adding the task to be executed into a task queue corresponding to the task scheduling service, and judging whether an idle thread exists in a thread pool of the task scheduling service;

when the idle thread exists, pulling the task to be executed from a task queue corresponding to the task scheduling service, and acquiring a subtask in the task to be executed;

and forming the task execution request according to the attribute information of the subtasks and the subtasks.

6. The task management method according to claim 5, wherein the forming the task execution request according to the attribute information of the subtask and the subtask includes:

judging whether the subtask is a subtask to be executed;

and when the subtask is judged to be the subtask to be executed, forming the task execution request according to the subtask to be executed.

7. The task management method according to claim 4, wherein the number of the task execution services is plural, and each of the task execution services has a different task type tag;

the sending the task execution request to the task execution server to enable the task execution service in the task execution server to execute the task includes:

matching the task type of the task to be executed in the task execution request with the task type label to determine a target task execution service;

and sending the task execution request to the target task execution service so that the target task execution service executes the task.

8. The task management method according to claim 4, wherein after sending the task execution request to the task execution server, the method further comprises:

inquiring whether the sub task to be executed is executed or not;

if the execution of the subtask to be executed is completed, updating task progress information corresponding to the task to be executed in the database;

and if the execution of the subtasks to be executed is not finished, adding the tasks to be executed into the task queues corresponding to the task scheduling services again.

9. The task management method according to claim 8, wherein after updating the task progress information corresponding to the task to be executed in the database, the method further comprises:

judging whether the tasks to be executed have unexecuted subtasks or not;

if the task exists, the unexecuted subtask is executed when the unexecuted subtask is a subtask to be executed until the unexecuted subtask does not exist in the task to be executed;

and if not, updating the task state information corresponding to the task to be executed in the database.

10. The task management method of claim 1, further comprising:

each task scheduling service updates the task quantity in the node according to the task quantity in the task queue corresponding to each task scheduling service;

and the task management server acquires the task quantity from each node, determines task distribution weight according to the task quantity, and executes task distribution based on the task distribution weight.

11. The task management method according to claim 4, wherein the distributed task management system further comprises a console; the method further comprises the following steps:

the console responds to a first trigger operation of a user to enter the task to be executed; or,

and the console responds to a second trigger operation of the user to generate a task progress query request and sends the task progress query request to the task management server so as to acquire task progress information.

12. A task management device is applied to a distributed task management system, the distributed task management system comprises a task scheduling server and nodes corresponding to a plurality of task scheduling services in the task scheduling server, wherein the task scheduling services are used for managing and scheduling tasks of users; the device comprises:

the monitoring module is used for monitoring a father node corresponding to the node by each task scheduling service;

a node creating module, configured to create, when it is monitored that a deleted first node exists in the nodes, a second node corresponding to the first node through a target task scheduling service, where the target task scheduling service corresponding to the second node is different from a task scheduling service corresponding to the first node;

and the task taking module is used for acquiring the task information corresponding to the first node from a database by the target task scheduling service and adding the task information into a task queue corresponding to the target task scheduling service.

13. A distributed task management system, comprising:

the console is used for responding to the triggering operation of a user to generate a task issuing instruction or a task progress query request;

the task management server is connected with the console and used for responding to the task issuing instruction to issue the task or responding to the task progress inquiry request to feed back task progress information to the console;

the task scheduling server is connected with the task management server and comprises a plurality of task scheduling services, and each task scheduling service is registered in the distributed coordination service system to form a node and is used for receiving the tasks issued by the task management server and scheduling and managing the tasks through each task scheduling service;

the task execution server is connected with the task scheduling server, comprises a plurality of task execution services and is used for responding to a task execution request sent by the task scheduling service so as to execute a task;

and the data storage equipment is connected with the task management server and the task scheduling server and is used for storing information related to the tasks.

14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the task management method of any one of claims 1 to 11.

15. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to perform a task management method as claimed in any one of claims 1-11.

Images