CN110362390B - Distributed data integration job scheduling method and device - Google Patents
Distributed data integration job scheduling method and device Download PDFInfo
- Publication number
- CN110362390B CN110362390B CN201910489422.2A CN201910489422A CN110362390B CN 110362390 B CN110362390 B CN 110362390B CN 201910489422 A CN201910489422 A CN 201910489422A CN 110362390 B CN110362390 B CN 110362390B
- Authority
- CN
- China
- Prior art keywords
- job
- scheduling
- information
- module
- resource
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000010354 integration Effects 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000007726 management method Methods 0.000 claims description 52
- 238000012545 processing Methods 0.000 claims description 16
- 238000013468 resource allocation Methods 0.000 claims description 14
- 230000002085 persistent effect Effects 0.000 claims description 8
- 230000015654 memory Effects 0.000 claims description 4
- 208000033748 Device issues Diseases 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000012986 modification Methods 0.000 claims description 3
- 230000004048 modification Effects 0.000 claims description 3
- 230000001960 triggered effect Effects 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000011084 recovery Methods 0.000 claims description 2
- 238000012217 deletion Methods 0.000 claims 1
- 230000037430 deletion Effects 0.000 claims 1
- 238000002955 isolation Methods 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000002688 persistence Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- RYYVLZVUVIJVGH-UHFFFAOYSA-N caffeine Chemical compound CN1C(=O)N(C)C(=O)C2=C1N=CN2C RYYVLZVUVIJVGH-UHFFFAOYSA-N 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- LPHGQDQBBGAPDZ-UHFFFAOYSA-N Isocaffeine Natural products CN1C(=O)N(C)C(=O)C2=C1N(C)C=N2 LPHGQDQBBGAPDZ-UHFFFAOYSA-N 0.000 description 1
- 241000508269 Psidium Species 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 229960001948 caffeine Drugs 0.000 description 1
- VJEONQKOZGKCAK-UHFFFAOYSA-N caffeine Natural products CN1C(=O)N(C)C(=O)C2=C1C=CN2C VJEONQKOZGKCAK-UHFFFAOYSA-N 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 235000003642 hunger Nutrition 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000037351 starvation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明涉及一种分布式数据集成作业调度方法及装置，本发明针对数据集成可能面临的特殊场景，由作业调度装置负责将数据集成作业下发到作业运行装置，作业运行装置接收调度任务并启动作业执行，并将作业运行的状态信息并发送给反馈给作业管理模块，将工作节点计算资源反馈给资源调度模块，将失联或故障信息反馈给作业预加载模块。本发明具备如下综合特点：(1)高可用与故障容错，弱一致性；(2)面向准实时作业调度的低延迟特性；(3)面向云服务应用的多租户并发控制；(4)计算资源隔离与多作业并行调度；(5)优先调度机制。
The invention relates to a distributed data integration job scheduling method and device. The present invention is aimed at special scenarios that data integration may face. The job scheduling device is responsible for delivering data integration jobs to a job running device, and the job running device receives the scheduling task and starts it. The job is executed, and the status information of the job running is sent to the job management module, and the computing resources of the worker nodes are fed back to the resource scheduling module, and the disconnection or failure information is fed back to the job preloading module. The present invention has the following comprehensive features: (1) high availability and fault tolerance, weak consistency; (2) low-latency characteristics for quasi-real-time job scheduling; (3) multi-tenant concurrency control for cloud service applications; (4) computing Resource isolation and parallel scheduling of multiple jobs; (5) Priority scheduling mechanism.
Description
Technical Field
The invention relates to the technical field of big data foundation, in particular to a distributed data integration job scheduling method and device.
Background
With the evolution of digital economy, service digitization in various industries has been fully developed, and digital service has gradually become a new center of gravity. However, since the service digitization derives a large number of data islands and becomes a common pain point for realizing the digital service, data integration is urgently needed in various industries, the data islands are opened and avoided, and data resources are integrated and managed, so that the associated value among data is effectively developed.
Data integration often faces thousands of job scheduling including job types such as data exchange and data preprocessing, and the design of a scheduling system needs to consider various complex scenarios. For example, some scenarios not only have a large number of jobs but also require parallel processing; some scenes require quasi-real-time operation, and priority scheduling needs to be considered; some jobs need to occupy more or longer computing resources, and resource isolation needs to be considered so as not to influence other jobs; some jobs may suffer from failures such as downtime of data sources and targets, interruption of networks, downtime of running nodes and the like, and a failure fault tolerance mechanism is needed; in a multi-tenant scenario, concurrent control needs to be handled; and so on.
The prior art fails to meet the complex data integration job scheduling requirements. For example: in the traditional LTS scheduling system, the operation is isolated based on the thread, if the execution thread of one operation exhausts all the memories of the current process, all the operations in the process are abnormal, the capability of scheduling data integration operation is lacked, and the traditional LTS scheduling system is more suitable for scheduling light-weight tasks. Chinese patent CN201610800080 discloses a distributed task scheduling system and method for solving the problems of large code writing amount and heavy development task of developers in a parallel computing program development mode, which is essentially scheduling for a single large-scale distributed computing job without considering multi-task parallel scheduling. Chinese patent CN201610197298 discloses a task scheduling method, apparatus and system, which provides a multi-channel multi-task distributed scheduling method, solves the problem of starvation of other jobs caused by too long scheduling time of a single task, but does not consider the problems of low latency of job metadata access, how to ensure job metadata consistency, and the like. Chinese patent No. CN201410748604 of the invention discloses a distributed task scheduling system and method, which provides a distributed task scheduling system and method for ensuring the reliability of the system itself, supporting independent or associated tasks, and supporting task rollback distribution, but is not suitable for complex data integration scenarios where rollback of tasks is not a key point, association does not need to occur between data integration tasks, and does not consider the problem that downtime under large-scale complex data integration tasks is high, and high availability needs to be achieved, and the problems of how to reduce delay as much as possible, how to ensure consistency of operation metadata, and the like under the premise that a quasi-real-time task exists.
Disclosure of Invention
Aiming at special scenes possibly faced by data integration, the invention is characterized in that an operation scheduling device is responsible for sending data integration operation to an operation running device, the operation running device receives scheduling tasks and starts operation execution, and sends operation running state information to an operation management module, feeds back working node calculation resources to a resource scheduling module, and feeds back loss connection or fault information to an operation preloading module; the invention has the following comprehensive characteristics: (1) high availability, fault tolerance, weak consistency; (2) the low-delay characteristic facing the quasi-real-time job scheduling; (3) multi-tenant concurrency control facing to cloud service application; (4) computing resource isolation and multi-job parallel scheduling; (5) a priority scheduling mechanism.
The invention achieves the aim through the following technical scheme: a distributed data integration job scheduling method comprises the following steps:
(1) the job scheduling device issues the data integration job to the job running device, wherein the job scheduling device comprises a job management module, a job preloading module and a resource scheduling module: (1.1) the operation management module receives, caches and stores operation related meta information to perform concurrency control;
(1.2) the job preloading module acquires the jobs to be processed from the job management module and determines the scheduling priority sequence;
(1.3) the resource scheduling module completes resource allocation and scheduling distribution by acquiring the job preloading information and the calculation resource information of the job running device;
(2) the operation running device receives the scheduling task and starts operation execution, and feeds back the operation running state information to the operation management module, feeds back the working node calculation resources to the resource scheduling module, and feeds back the loss connection or fault information to the operation preloading module.
Preferably, the job management module includes an information receiving unit, an information caching unit, a persistent storage unit, and a concurrency control unit, wherein the specific operations are as follows:
(i) the information receiving unit receives job submission, job meta-information modification and scheduling strategy updating; receiving unallocated resource job information fed back by the resource scheduling module, and updating job states; receiving operation state information fed back by the operation running device and updating the operation state;
(ii) the information caching unit is used for locally caching the operation meta information and the state information and supporting frequent real-time query;
(iii) the persistent storage unit maintains the data consistency of the cache layer and the storage layer according to the metadata information of the persistent operation of the cache content;
(iv) the concurrency control unit assigns a read-write lock to access of each of the job resources.
Preferably, in step (iii), the data consistency between the cache layer and the storage layer is maintained by the following method:
(a) writing updated data into a local file by using fault-tolerant storage, and writing the updated data into a storage layer after the network is recovered to be normal;
(b) the fuse is used, when the fault-tolerant mechanism is triggered and reaches a preset threshold value, the fuse is disconnected, the service performs degradation processing, and a new task is not scheduled;
(c) and encapsulating the operation state interface of the operation running device, acquiring the operation running state and auditing the operation running state when the operation scheduling node is started every time, and ensuring that the operation state in the operation running device is consistent with the state in the metadata storage layer.
Preferably, the job preloading module includes a real-time query unit, a job preloading unit, and a fault processing unit, wherein the specific operations are as follows:
(I) the real-time query unit queries the job metadata cache in real time to acquire the unlocked job to be scheduled;
(II) adding the unlocked job to be scheduled into a bounded ordered queue by a job preloading unit, and sequencing according to job scheduling time and job priority;
(III) the fault processing unit receives fault information from the operation device and carries out fault tolerance processing; the fault tolerance processing means that a working node is down or a long-time network is disconnected in the operation process, the operation running device informs the operation preloading module, the operation preloading module is connected with the working node to judge whether the connection is unavailable, if the connection is not available, the operation is directly put into a queue, and the lost operation is finally dispatched to other available nodes; if the unconnected node recovers again at the moment, the operation running device can directly kill the operation process, so that the same operation cannot be simultaneously run on two nodes under one operation running device.
Preferably, the bounded ordered blocking queue is used for loading all jobs to be scheduled, wherein bounded means that the upper limit of the number of jobs is guaranteed, and an upper limit parameter can be given through scene evaluation; the order refers to that the job with earlier trigger time and higher priority is placed at the position at the front of the queue for preferential scheduling; in the process of stopping deleting the job, supporting to remove the specified job to be scheduled from the queue; with the producer-consumer model, the CPU burden is reduced using a thread blocking-wakeup approach.
Preferably, the resource scheduling module includes a resource obtaining unit, a resource allocating unit, and a scheduling distributing unit, wherein the specific operations are as follows:
1) the resource obtaining unit obtains the computing resources of the operation cluster and caches the computing resources in the memory;
2) the resource allocation unit acquires all the jobs from the bounded ordered queue and allocates computing resources to each job according to the job priority order;
3) and the scheduling and distributing unit appoints an actuator for the scheduled job and sends the job meta-information, the distributed computing resources and the actuator configuration to the job running cluster.
Preferably, in the step (2), the job running device includes a main control node and a work node, the main control node is responsible for management and coordination, and the work node is responsible for executing the data integration job; the master control node receives the job meta-information, the job resource allocation information and the job executor information which are distributed by the resource scheduling module and starts the job execution; the agent program on the working node collects the state information of the operation and sends the state information to the main control node, and the main control node sends the state information to the operation management module; the agent program on the working node collects the working node computing resources and sends the working node computing resources to the main control node, and the main control node sends the working node computing resources to the resource scheduling module; the agent program on the working node sends heartbeat information to the main control node, and the main control node sends loss of connection or fault information to the operation preloading module; the executor on the working node is provided with a retry mechanism, and once the data flow source or the data flow target goes down or loses connection, the executor carries out timing retry to ensure that the data flow source and the data flow target can continue to normally operate after recovery.
Preferably, the operation of the job running device is based on a meso cluster system to perform distributed system resource management, a master control node provides low-delay local metadata management in a RAM + WAL log mode, a PAXOS algorithm is adopted to maintain job state synchronization of a large number of working nodes, and specific physical resources are pushed to a job scheduling system based on a cluster physical resource unified management interface and a specific resource sharing strategy of the master control node; the job operation cluster provides two modes of a multi-language driver package and JSON RPC for the registration of the job scheduling system and the acquisition of a specific callback event; the agent program is responsible for collecting the resources of the working nodes, running a specific scheduling task through the actuator and returning the execution result and the task state of the actuator to the master control node; and then the main control node forwards the data to the job scheduling device.
A distributed data integrated job scheduling apparatus, comprising: a job scheduling device and a job running device; the job scheduling device and the job running device perform information interaction with each other; the job scheduling device comprises a job management module, a job pre-recording module and a resource scheduling module; the operation management module is used for receiving, caching and storing operation related meta information and performing concurrency control; the job preloading module is used for acquiring the jobs to be processed from the job management module and determining the scheduling priority sequence; the resource scheduling module is used for completing resource allocation and scheduling distribution by acquiring the job preloading information and the calculation resource information of the job running device; the operation running device comprises a main control node and a working node, wherein the main control node is responsible for management and coordination, and the working node is responsible for executing data integration operation.
Preferably, the job scheduling device and the job running device are both registered in the ZooKeeper, the job scheduling device adopts a master-slave mode, and once the master device goes down, the ZooKeeper selects the backup device and replaces the job scheduling work; the main control nodes in the operation running device adopt a main standby mode, and once the main control nodes are down, the ZooKeeper elects the standby main control nodes to take over the management coordination work.
Preferably, the job scheduling device performs auditing and maintenance based on job status information fed back by the job running device, and the job metadata database caches and maintains consistency based on job metadata of the job scheduling device; when the job scheduling device fails, the standby job scheduling device needs to interact with the job metadata database once taking over the work, a job metadata cache mechanism is rebuilt, and the metadata cache information is audited and maintained by receiving job state feedback information from the job running device, so that the metadata cache information is kept consistent in a distribution system.
The invention has the beneficial effects that: (1) according to CAP theorem, the method of the invention meets two indexes of high availability and fault tolerance, and adopts a mechanism for ensuring the consistency of operation metadata as much as possible; (2) the multi-tenant concurrency control is realized based on the distributed read-write lock, and the multi-tenant data integration service is provided in a cloud service mode; (3) aiming at the particularity that the data integration operation needs frequent scheduling, the operation metadata adopts a cache mechanism, so that the delay and interruption risks caused by frequent metadata access can be effectively reduced.
Drawings
FIG. 1 is a schematic flow diagram of the apparatus of the present invention;
FIG. 2 is a schematic diagram of the high availability mechanism of the apparatus of the present invention;
FIG. 3 is a schematic flow diagram of the method of the present invention;
FIG. 4 is a schematic flow diagram of a job management module of the present invention;
FIG. 5 is a schematic representation of the operation of the job management module of the present invention;
FIG. 6 is a process flow diagram of a job preloading module of the present invention;
FIG. 7 is a schematic flow diagram of a resource scheduling module of the present invention;
fig. 8 is a schematic view showing an operation flow of the work running apparatus of the present invention.
Detailed Description
The invention will be further described with reference to specific examples, but the scope of the invention is not limited thereto:
example (b): as shown in fig. 1, a distributed data-integrated job scheduling apparatus is composed of a job scheduling apparatus and a job execution apparatus. The job scheduling device and the job running device perform information interaction with each other; the job scheduling device comprises a job management module, a job pre-recording module and a resource scheduling module; the operation management module is used for receiving, caching and storing operation related meta information and performing concurrency control; the job preloading module is used for acquiring the jobs to be processed from the job management module and determining the scheduling priority sequence; the resource scheduling module is used for completing resource allocation and scheduling distribution by acquiring the job preloading information and the calculation resource information of the job running device; the operation running device comprises a main control node and a working node, wherein the main control node is responsible for management and coordination, and the working node is responsible for executing data integration operation.
As shown in fig. 2, both the job scheduling device and the job running device are registered in the ZooKeeper, the job scheduling device adopts a master-slave mode, and once the master device goes down, the ZooKeeper selects the backup device and replaces the job scheduling work; the main control nodes in the operation running device adopt a main standby mode, and once the main control nodes are down, the ZooKeeper elects the standby main control nodes to take over the management coordination work.
The job scheduling device audits and maintains based on job state information fed back by the job running device, and the job metadata base caches and maintains consistency based on job metadata of the job scheduling device; when the job scheduling device fails, the standby job scheduling device needs to interact with a job metadata database once taking over work, a job metadata cache mechanism is rebuilt, and the metadata cache information is audited and maintained by receiving job state feedback information from the job running device so as to keep the metadata cache information consistent in a distribution system; thereby ensuring weak consistency.
As shown in fig. 3, a distributed data integration job scheduling method includes the following steps:
s100: the job scheduling device issues the data integration job to the job running device, and the job scheduling device consists of a job management module, a job preloading module and a resource scheduling module, and specifically comprises the following parts:
s101: and the operation management module receives, caches and stores the operation related meta information and performs concurrency control. The job management module consists of an information receiving unit, a storage processing unit and a concurrency control unit. As shown in fig. 4, the specific operations are as follows:
(1) the information receiving unit S101-1 is responsible for receiving job submission, job meta-information modification and scheduling policy update; receiving unallocated resource job information fed back by the resource scheduling module, and updating job states; receiving operation state information fed back by the operation running device and updating the operation state;
(2) the information caching unit S101-2 is responsible for locally caching the operation meta information and the state information and supporting frequent real-time query;
(3) the persistent storage unit S101-3 maintains the data consistency of the cache layer and the storage layer according to the metadata information of the persistent job of the cache content;
(4) the concurrency control unit S101-4 is responsible for assigning a read-write lock to access of each job resource.
Specifically, as shown in fig. 5, the job management module of the present invention is responsible for receiving jobs and maintaining job state machines, and the main job states may include: the method comprises the following steps of not starting operation, waiting to be scheduled operation, suspending operation, running operation, stopping operation, abnormal operation and finishing operation. The job management module provides job state operation interfaces, such as operation interfaces for stopping running jobs, suspending jobs, scheduling jobs, suspending jobs, normally stopping jobs, abnormally stopping jobs, and the like. Maintaining various job scheduling policies, such as: repeated operation, timed operation, Cron operation, disposable operation, etc. And a history storage module is internally maintained and used for recording all scheduling histories. And adding a read-write lock to the operation of the cache layer and the metadata persistence layer to realize concurrency control: if there are concurrent threads that are write operations, the lock is upgraded to an exclusive lock and other threads cannot seize the lock. Conversely, if the concurrent thread is a read operation, the lock is upgraded to a shared lock and other threads can concurrently seize the lock. The operation management module adds a cache layer on an operation metadata storage layer, ensures frequent metadata query and call, supports frequent access and frequent scheduling of quasi-real-time data integration tasks, abstracts the cache layer into an SPI interface on an implementation layer, and supports the implementation of cache layer interfaces such as Caffeine, JDK, Guava, Redis and the like. The persistence layer is abstracted into an SPI interface on the implementation level and supports databases such as relational databases, MongoDB databases and the like. Because the job data may cause the problem of data inconsistency due to network and other instability factors when written into the persistence layer, in the implementation level, the job management module adopts "triple insurance" to ensure the metadata consistency as much as possible: (1) writing updated data into a local database/file by using fault-tolerant storage, and writing the updated data into a storage layer after the network is recovered to be normal; (2) the fuse is used, when the fault-tolerant mechanism is triggered and reaches a certain threshold value, the fuse is disconnected, the service performs degradation processing, and a new task is not scheduled; (3) and encapsulating an operation state interface of the operation running system, and acquiring the operation running state for auditing when the operation scheduling node is started every time, so as to ensure that the operation state in the operation running system is consistent with the state in the metadata storage layer.
S102, as shown in figure 6, the job preloading module acquires the job to be processed from the job management module and determines the scheduling priority sequence, wherein the job preloading module consists of a real-time query unit, a job preloading unit and a fault processing unit. The method comprises the following specific operations:
(1) the real-time query unit S102-1 is responsible for querying job metadata cache in real time and acquiring unlocked jobs to be scheduled;
(2) the job preloading unit S102-2 is responsible for adding the unlocked job to be scheduled into the bounded ordered queue and sorting the job according to the job scheduling time and the job priority;
(3) failure processing unit S102-3: and the system is responsible for receiving fault information from the operation cluster and performing fault tolerance processing.
Specifically, a bounded ordered blocking queue is built to load all jobs to be scheduled. The bounded state refers to the condition that the upper limit of the number of the jobs is guaranteed, and upper limit parameters can be given through scene evaluation; in order means that jobs with earlier trigger times and higher priority levels will be placed in the queue front position for priority scheduling. And in the process of stopping deleting the job, supporting to remove the specified job to be scheduled from the queue. With the producer-consumer model, the CPU burden is reduced using a thread blocking-wakeup approach.
The fault tolerance processing means that in the operation process of the operation, a working node is down or a long-time network is disconnected, the operation running device informs the operation preloading module, the operation preloading module is connected with the working node to judge whether the connection is unavailable, if the connection is not available, the operation is directly put into a queue, and the lost operation is finally scheduled to other available nodes. If the disconnected node is recovered again, the operation running device can directly kill the operation process to ensure that the same operation cannot run on two nodes simultaneously under one operation running device.
And S103, the resource scheduling module finishes resource allocation and scheduling distribution by acquiring the job preloading information and the calculation resource information of the job running device. The resource scheduling module consists of a resource acquisition unit, a resource allocation unit and a scheduling and distributing unit; as shown in fig. 7:
(1) the resource obtaining unit S103-1 is responsible for obtaining the computing resources of the job running cluster and caching the computing resources in the memory;
(2) the resource allocation unit S103-2 is responsible for acquiring all the jobs from the bounded ordered queue and allocating computing resources to each job according to the job priority order;
(3) the scheduling and distributing unit S103-3 is responsible for assigning an executor to the scheduled job and sending the job meta-information, the allocated computing resource and the executor configuration to the job running cluster.
The implementation of the Executor may be Linux container Executor, Docker Executor, or other executors, and these container executors can implement the isolation of the computing resources.
And S200, the job running device receives the scheduling task and starts job execution, sends the state information of job running to the job management module, feeds back the working node calculation resources to the resource scheduling module, and feeds back the loss of connection or fault information to the job preloading module.
The operation running device is provided with a main control node and a working node, the main control node is responsible for management and coordination, and the working node is responsible for executing data integration operation. The master control node receives job meta-information, job resource allocation information and job executor information distributed by the resource scheduling module and starts job execution; the agent program on the working node collects the state information of the operation and sends the state information to the main control node, and the main control node sends the state information to the operation management module; the agent program on the working node collects the working node computing resources and sends the working node computing resources to the main control node, and the main control node sends the working node computing resources to the resource scheduling module; and the agent program on the working node sends heartbeat information to the main control node, and the main control node sends loss of connection or fault information to the operation preloading module. The executor on the working node is provided with a retry mechanism, and once the data flow source or the data flow target goes down or loses connection, the executor carries out timing retry to ensure that the data flow source and the data flow target can continue to normally operate after being recovered.
As shown in fig. 8, the job running apparatus may perform distributed system resource management based on a meso cluster system, where the master node provides low-latency local metadata management in a RAM + WAL log manner, maintains job state synchronization of a large number of working nodes by using a PAXOS algorithm, and pushes specific physical resources to a job scheduling system based on a cluster physical resource unified management interface and a specific resource sharing policy of the master node. The job operation cluster provides two modes of a multi-language driver package and JSON RPC for the registration of the job scheduling system and the acquisition of a specific callback event; the agent program is responsible for collecting the resources of the working nodes, running a specific scheduling task through the actuator and returning the execution result and the task state of the actuator to the master control node. And then the main control node forwards the data to the job scheduling device.
While the invention has been described in connection with specific embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (8)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910489422.2A CN110362390B (en) | 2019-06-06 | 2019-06-06 | Distributed data integration job scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910489422.2A CN110362390B (en) | 2019-06-06 | 2019-06-06 | Distributed data integration job scheduling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362390A CN110362390A (en) | 2019-10-22 |
| CN110362390B true CN110362390B (en) | 2021-09-07 |
Family
ID=68215696
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910489422.2A Active CN110362390B (en) | 2019-06-06 | 2019-06-06 | Distributed data integration job scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362390B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111045802B (en) * | 2019-11-22 | 2024-01-26 | 中国联合网络通信集团有限公司 | Redis cluster component scheduling system and method, platform equipment |
| CN111124806B (en) * | 2019-11-25 | 2023-09-05 | 山东鲁软数字科技有限公司 | Method and system for monitoring equipment state in real time based on distributed scheduling task |
| CN111338770A (en) * | 2020-02-12 | 2020-06-26 | 咪咕文化科技有限公司 | A task scheduling method, server and computer-readable storage medium |
| CN111580990A (en) * | 2020-05-08 | 2020-08-25 | 中国建设银行股份有限公司 | Task scheduling method, scheduling node, centralized configuration server and system |
| CN113961318A (en) * | 2020-07-20 | 2022-01-21 | 百度在线网络技术（北京）有限公司 | Distributed scheduling method, device, equipment and storage medium |
| CN114265672A (en) * | 2020-09-16 | 2022-04-01 | 广州天维信息技术股份有限公司 | Parallel scheduling scheme and system based on multi-tenant distributed performance system |
| CN112200534A (en) * | 2020-09-24 | 2021-01-08 | 中国建设银行股份有限公司 | Method and device for managing time events |
| CN112328383A (en) * | 2020-11-19 | 2021-02-05 | 湖南智慧畅行交通科技有限公司 | Priority-based job concurrency control and scheduling algorithm |
| CN112131318B (en) * | 2020-11-30 | 2021-03-16 | 北京优炫软件股份有限公司 | Pre-written log record ordering system in database cluster |
| CN112527488A (en) * | 2020-12-21 | 2021-03-19 | 浙江百应科技有限公司 | Distributed high-availability task scheduling method and system |
| CN112835717B (en) * | 2021-02-05 | 2024-06-28 | 远光软件股份有限公司 | Integrated application processing method and device for clusters |
| CN113778676B (en) * | 2021-09-02 | 2023-05-23 | 山东派盟网络科技有限公司 | Task scheduling system, method, computer device and storage medium |
| CN113986507A (en) * | 2021-11-01 | 2022-01-28 | 佛山技研智联科技有限公司 | Job scheduling method and device, computer equipment and storage medium |
| CN114118043A (en) * | 2021-11-17 | 2022-03-01 | 粤港澳国际供应链(广州)有限公司 | A method of dynamically generating PDF |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6490572B2 (en) * | 1998-05-15 | 2002-12-03 | International Business Machines Corporation | Optimization prediction for industrial processes |
| CN101309208A (en) * | 2008-06-21 | 2008-11-19 | 华中科技大学 | A Job Scheduling System Based on Reliability Cost for Grid Environment |
| CN101599026A (en) * | 2009-07-09 | 2009-12-09 | 浪潮电子信息产业股份有限公司 | A Cluster Job Scheduling System with Elastic Architecture |
| CN104317650A (en) * | 2014-10-10 | 2015-01-28 | 北京工业大学 | Map/Reduce type mass data processing platform-orientated job scheduling method |
| CN104462370A (en) * | 2014-12-09 | 2015-03-25 | 北京百度网讯科技有限公司 | Distributed task scheduling system and method |
| US9141433B2 (en) * | 2009-12-18 | 2015-09-22 | International Business Machines Corporation | Automated cloud workload management in a map-reduce environment |
| CN109327509A (en) * | 2018-09-11 | 2019-02-12 | 武汉魅瞳科技有限公司 | A kind of distributive type Computational frame of the lower coupling of master/slave framework |
-
2019
- 2019-06-06 CN CN201910489422.2A patent/CN110362390B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6490572B2 (en) * | 1998-05-15 | 2002-12-03 | International Business Machines Corporation | Optimization prediction for industrial processes |
| CN101309208A (en) * | 2008-06-21 | 2008-11-19 | 华中科技大学 | A Job Scheduling System Based on Reliability Cost for Grid Environment |
| CN101599026A (en) * | 2009-07-09 | 2009-12-09 | 浪潮电子信息产业股份有限公司 | A Cluster Job Scheduling System with Elastic Architecture |
| US9141433B2 (en) * | 2009-12-18 | 2015-09-22 | International Business Machines Corporation | Automated cloud workload management in a map-reduce environment |
| CN104317650A (en) * | 2014-10-10 | 2015-01-28 | 北京工业大学 | Map/Reduce type mass data processing platform-orientated job scheduling method |
| CN104462370A (en) * | 2014-12-09 | 2015-03-25 | 北京百度网讯科技有限公司 | Distributed task scheduling system and method |
| CN109327509A (en) * | 2018-09-11 | 2019-02-12 | 武汉魅瞳科技有限公司 | A kind of distributive type Computational frame of the lower coupling of master/slave framework |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362390A (en) | 2019-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110362390B (en) | Distributed data integration job scheduling method and device | |
| US10817478B2 (en) | System and method for supporting persistent store versioning and integrity in a distributed data grid | |
| Lin et al. | Towards a non-2pc transaction management in distributed database systems | |
| CN104793988B (en) | The implementation method and device of integration across database distributed transaction | |
| US8799248B2 (en) | Real-time transaction scheduling in a distributed database | |
| KR102013005B1 (en) | Managing partitions in a scalable environment | |
| US9548912B2 (en) | System and method for supporting smart buffer management in a distributed data grid | |
| US11334422B2 (en) | System and method for data redistribution in a database | |
| CN110019469B (en) | Distributed database data processing method and device, storage medium and electronic device | |
| US20090172142A1 (en) | System and method for adding a standby computer into clustered computer system | |
| EP2673711A1 (en) | Method and system for reducing write latency for database logging utilizing multiple storage devices | |
| CN114064414A (en) | High-availability cluster state monitoring method and system | |
| EP4229516B1 (en) | System and method for rapid fault detection and repair in a shared nothing distributed database | |
| US11550820B2 (en) | System and method for partition-scoped snapshot creation in a distributed data computing environment | |
| US9703634B2 (en) | Data recovery for a compute node in a heterogeneous database system | |
| US11003550B2 (en) | Methods and systems of operating a database management system DBMS in a strong consistency mode | |
| US11537574B2 (en) | Autonomous database defragmentation | |
| EP4404059A1 (en) | Unified resource management architecture for workload schedulers | |
| US9424147B2 (en) | System and method for supporting memory allocation control with push-back in a distributed data grid | |
| CN117331751A (en) | Multi-node backup system and method for database | |
| CN113342511A (en) | Distributed task management system and method | |
| CN117931302B (en) | Parameter file saving and loading method, device, equipment and storage medium | |
| WO2025010735A1 (en) | Hybrid database implementations | |
| WO2025010725A1 (en) | Hybrid database implementations | |
| WO2025010728A1 (en) | Hybrid database implementations |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 310012 1st floor, building 1, 223 Yile Road, Hangzhou City, Zhejiang Province Patentee after: Yinjiang Technology Co.,Ltd. Address before: 310012 1st floor, building 1, 223 Yile Road, Hangzhou City, Zhejiang Province Patentee before: ENJOYOR Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20191022 Assignee: HANGZHOU ENJOYOR SMART CITY TECHNOLOGY GROUP CO.,LTD. Assignor: Yinjiang Technology Co.,Ltd. Contract record no.: X2024980042648 Denomination of invention: A distributed data integration job scheduling method and device Granted publication date: 20210907 License type: Common License Record date: 20250102 |
|
| EE01 | Entry into force of recordation of patent licensing contract |