CN108958840B - A cluster configuration dynamic detection merge loading method - Google Patents
A cluster configuration dynamic detection merge loading method Download PDFInfo
- Publication number
- CN108958840B CN108958840B CN201810744414.3A CN201810744414A CN108958840B CN 108958840 B CN108958840 B CN 108958840B CN 201810744414 A CN201810744414 A CN 201810744414A CN 108958840 B CN108958840 B CN 108958840B
- Authority
- CN
- China
- Prior art keywords
- configuration
- node
- cluster
- state
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Stored Programmes (AREA)
Abstract
本发明公开了一种集群配置动态探测合并加载方法，包括：步骤1，通过配置管理客户端管理集群配置；步骤2，配置管理服务器端接收配置管理客户端的配置修改请求并把配置按指定格式写入配置管理中心；步骤3，配置探测模块实时监听配置管理中心的集群配置节点；当配置探测模块检测到节点的内容出现变化时，集群同步新的配置到本地，并更新相应节点的状态。本发明为所有配置提供版本管理和日志，维护过程中能够应用任一版本的配置；集群配置文件动态探测加载的同时，保证不中断当前正执行的任务；提供统一的配置修改入口，对修改的配置增加合法性检查；配置管理流量控制和配置合并应用，保证集群不会因为配置修改而频繁重启。
The invention discloses a method for dynamically detecting and merging and loading a cluster configuration, comprising: step 1, managing the cluster configuration through a configuration management client; step 2, receiving a configuration modification request of the configuration management client by the configuration management server and writing the configuration in a specified format In step 3, the configuration detection module monitors the cluster configuration nodes of the configuration management center in real time; when the configuration detection module detects that the content of the node changes, the cluster synchronizes the new configuration to the local, and updates the state of the corresponding node. The invention provides version management and logs for all configurations, and any version of the configuration can be applied in the maintenance process; the cluster configuration file is dynamically detected and loaded, while ensuring that the currently executing task is not interrupted; Validity check is added to the configuration; the configuration management flow control and configuration merge application ensure that the cluster will not be restarted frequently due to configuration changes.
Description
Technical Field
The invention relates to the field of data processing cluster management, in particular to a dynamic detection, merging and loading method for cluster configuration.
Background
With the rapid development and popularization of computers and information technologies, the scale of industrial application systems is rapidly expanded, and data generated by industrial application is explosively increased. The concept of big data has been proposed. With the release of MapReduce and GoogleFile System (GFS), Google, big data is no longer used to describe a large amount of data, but also covers the speed of processing data. Big data processing is a computing model that provides dynamically scalable virtualized resources in a service-like manner over the Internet. Cluster management is an essential part of this computing model.
The existing distributed cluster has the following defects in management:
1. in the cluster management process, for modification of a certain configuration, the same modification needs to be made to each node of the cluster. It is easy to modify only part of the nodes due to errors or unfairness to the system, resulting in inconsistent system configuration state.
2. In the configuration modification process, the cluster needs to be restarted to enable the configuration to be effective, so that part of tasks are repeatedly calculated due to the restarting of the nodes, the calculation resources are wasted, and the efficiency is reduced.
3. Most of the cases are directly modifying the configuration file in a text editing mode, and the possibility of human errors is further increased.
4. Some errors due to incorrect configuration are difficult to trace back.
Disclosure of Invention
The present invention is to solve the above technical problem, and aims to provide a method for dynamically detecting, merging and loading cluster configuration.
The invention is realized by the following technical scheme:
a cluster configuration dynamic detection merging loading method comprises the following steps:
step 1, managing cluster configuration through a configuration management client;
step 2, the configuration management server receives the configuration modification request of the configuration management client and writes the configuration into the configuration management center according to the specified format;
step 3, the configuration detection module monitors cluster configuration nodes of the configuration management center in real time; when the configuration detection module detects that the content of the node changes, the cluster synchronizes the new configuration to the local and updates the state of the corresponding node;
step 4, the master node cluster configuration state maintenance module acquires the configuration state of each working node from each working node of the cluster; the configuration state of the working node comprises a configuration version number, a configuration applied state and a configuration synchronous unapplied state of the node; when each working node is started or updated, each working node registers the state of the working node to the main node;
step 5, the heavy-load module is positioned on each node; when the heavy-load module is positioned on the main node of the cluster, executing the step 6; when the heavy-load module is positioned on the working node of the cluster, executing the step 7;
step 6, when the reloading module is positioned at the main node, the reloading module checks the state of the corresponding working node before distributing the task to the working node, and if the configuration of the corresponding working node is updated and the configuration is applied, the reloading module distributes the task to the corresponding working node;
step 7, when the heavy-load module is positioned at the working node,
the reloading module checks whether the configuration version number of the corresponding working node is latest, and if the configuration version number of the corresponding working node is not latest, the step 3 is executed;
if the configuration version number of the corresponding working node is latest, the reloading module checks whether the corresponding working node is in a configuration synchronization non-application state, and if the corresponding working node is in the configuration synchronization non-application state, the reloading module informs the main node that the corresponding working node refuses to accept a new task;
if the corresponding worker node is in the configuration applied state, the reload module assigns a task to the corresponding worker node.
The invention provides version management and logs for all configurations, and any version configuration can be applied in the maintenance process; when the cluster configuration file is dynamically detected and loaded, the currently executed task is ensured not to be interrupted; providing a uniform configuration modification entrance, and adding validity check to the modified configuration; and the configuration management flow control and the configuration are combined for application, so that the cluster is not frequently restarted due to configuration modification.
Further, the step 1 of managing the cluster configuration by the configuration management client includes modifying the cluster configuration, adding validity check and authority control to each configuration item input, viewing all historical versions and related information of the configuration application, and merging the configuration application of any version into any cluster state.
Further, the configuration management server in step 2 has a flow control function.
Further, the flow control function includes discarding unreasonable configuration management client requests.
Further, the nodes in step 3 include a master node and a worker node, wherein the configuration detection module is located on each node of the cluster.
Further, the configuration detection module in step 3 monitors the cluster configuration nodes of the configuration management center in real time by using the API interface.
Further, the master node cluster configuration state maintenance module in step 4 is located in a master node of the cluster.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the invention uses the configuration management client to submit the relevant configuration parameters to the configuration management center. The client has the functions of validity detection and authority control, and the configuration is ensured to be correctly modified;
2. the invention utilizes the monitoring mechanism of the configuration management middle body, and each node of the cluster detects the change of the configuration in real time and synchronizes to the local. Each node determines the time for updating the configuration according to the condition of each load, combines all the configuration items with update or conflict during updating, and submits the completion state of the configuration update to the main node. The main node performs task scheduling and resource allocation according to the configuration update state table;
3. the configuration management client side of the invention provides a configuration history backtracking function, and can check all history configuration applications and related information thereof of any node of the cluster, including configuration application date, configuration effective time, configuration application cluster state and the like. The configuration of any cluster state of any version can be merged and applied to the current cluster state through the management client.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
FIG. 1 is a flow chart of dynamic probing and loading for configuration of a working node according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to examples and accompanying drawings, and the exemplary embodiments and descriptions thereof are only used for explaining the present invention and are not meant to limit the present invention.
Examples
As shown in fig. 1, a method for dynamic probing, merging and loading of cluster configuration includes:
step 1, managing cluster configuration through a configuration management client; the configuration management client side in the step 1 manages the cluster configuration, and comprises the steps of modifying the cluster configuration, adding validity check and authority control to each configuration item input, checking all historical versions and related information of the configuration application, and merging the configuration application of any version into any cluster state.
The cluster configuration is managed through the configuration management client, the cluster configuration can only be modified through the configuration management client, meanwhile, the validity check and the authority control are added for the input of each configuration item, and the validity of the configuration is greatly guaranteed; meanwhile, all historical versions and related information of the configuration application can be checked, and the configuration application of any version can be merged into any cluster state.
Step 2, the configuration management server receives the configuration modification request of the configuration management client and writes the configuration into the configuration management center according to the specified format; the configuration management server side in the step 2 has a flow control function.
The flow control function includes discarding unreasonable configuration management client requests.
The configuration management server receives the client configuration modification request and writes the configuration into the configuration management center according to the specified format; meanwhile, the flow control function is added, unreasonable client requests are abandoned, and the situation that the cluster is frequently restarted or even paralyzed due to flow attack and other abnormalities is avoided.
Step 3, the configuration detection module monitors cluster configuration nodes of the configuration management center in real time; when the configuration detection module detects that the content of the node changes, the cluster synchronizes the new configuration to the local and updates the state of the corresponding node; the nodes in step 3 include a master node and a working node, wherein the configuration detection module is located on each node of the cluster. And 3, the configuration detection module monitors the cluster configuration nodes of the configuration management center in real time by using the API interface.
Step 4, the master node cluster configuration state maintenance module acquires the configuration state of each working node from each working node of the cluster; the configuration state of the working node comprises a configuration version number of the node, a configuration applied state and a configuration synchronous unapplied state, and as shown in table 1, the NodeId is used as a unique identifier of each node; version is used as the configuration Version number of each node, the Version number must be up-to-date, and the configuration of the node is valid; the State is used as the configuration application State of each node, and is further divided into an Applied State and a synchronized State, wherein the Applied State: the configuration is applied, and the node can accept a new task; synced: the node is synchronously configured to the local, but is not applied yet; when each working node is started or updated, each working node registers the state of the working node to the main node; and 4, the master node cluster configuration state maintenance module in the step 4 is positioned in the master nodes of the cluster.
TABLE 1
Step 5, the heavy-load module is positioned on each node; when the heavy-load module is positioned on the main node of the cluster, executing the step 6; when the heavy-load module is positioned on the working node of the cluster, executing the step 7;
step 6, when the reloading module is positioned at the main node, the reloading module checks the state of the corresponding working node before distributing the task to the working node, and if the configuration of the corresponding working node is updated and the configuration is applied, the reloading module distributes the task to the corresponding working node;
step 7, when the heavy-load module is positioned at the working node,
the reloading module checks whether the configuration Version number of the corresponding working node is latest, namely whether Version is latest, and if the configuration Version number of the corresponding working node is not latest, the step 3 is executed;
if the configuration version number of the corresponding working node is latest, the reloading module checks whether the corresponding working node is in a configuration synchronization non-application state, namely Synced, and if the corresponding working node is in the configuration synchronization non-application state, the reloading module informs the main node that the corresponding working node refuses to accept a new task; and after all tasks of the corresponding node are completed, applying the new configuration, modifying the state to be Applied after the application is completed, and informing the main node.
If the corresponding working node is in the configuration applied state, namely Synced, the reload module allocates a task to the corresponding working node.
The invention avoids the situation that the cluster configuration is inconsistent because maintenance personnel are not familiar with the cluster or because of human error, and the client checks the validity of the parameters of the configuration items to ensure that all the configurations are correct and effective; the cluster configuration is managed by using a configuration management center, so that the consistency and the real-time performance of the configuration are ensured; the reloading module based on the node load state ensures that the new configuration can be correctly applied to all new tasks and the normal execution of the existing tasks is not influenced. The accuracy is ensured, and meanwhile, the resource utilization rate of the cluster is improved.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (6)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810744414.3A CN108958840B (en) | 2018-07-09 | 2018-07-09 | A cluster configuration dynamic detection merge loading method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810744414.3A CN108958840B (en) | 2018-07-09 | 2018-07-09 | A cluster configuration dynamic detection merge loading method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108958840A CN108958840A (en) | 2018-12-07 |
| CN108958840B true CN108958840B (en) | 2021-01-22 |
Family
ID=64482572
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810744414.3A Expired - Fee Related CN108958840B (en) | 2018-07-09 | 2018-07-09 | A cluster configuration dynamic detection merge loading method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108958840B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112612545A (en) * | 2020-12-23 | 2021-04-06 | 北京浪潮数据技术有限公司 | Configuration hot loading system, method, equipment and medium of server cluster |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105005487A (en) * | 2015-06-29 | 2015-10-28 | 清华大学 | Continuous-service high-performance cluster operating system online upgrading method |
| CN106603281A (en) * | 2016-12-07 | 2017-04-26 | 乐视控股（北京）有限公司 | Configuration file management method and system |
| CN106843930A (en) * | 2016-12-23 | 2017-06-13 | 江苏途致信息科技有限公司 | Streaming dynamic configuration more new architecture and method based on zookeeper |
| CN107025139A (en) * | 2017-03-21 | 2017-08-08 | 北京天云融创软件技术有限公司 | A kind of high-performance calculation Scheduling Framework based on cloud computing |
| CN107465714A (en) * | 2017-01-23 | 2017-12-12 | 北京思特奇信息技术股份有限公司 | A kind of configuration data dynamic update system and method based on application cluster |
| CN107506213A (en) * | 2017-08-30 | 2017-12-22 | 深圳市小牛在线互联网信息咨询有限公司 | Dynamic Configuration, device, storage medium and computer equipment |
| CN108255592A (en) * | 2017-12-19 | 2018-07-06 | 武汉市烽视威科技有限公司 | A kind of Quartz clusters timing task processing system and method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5223707B2 (en) * | 2009-02-05 | 2013-06-26 | 富士通株式会社 | Software update instruction program, software update instruction method, and information processing apparatus |
| US9207956B2 (en) * | 2013-10-15 | 2015-12-08 | Fujitsu Technology Solutions Intellectual Property Gmbh | Class loading device for a java runtime environment, cluster system and method of executing a function |
-
2018
- 2018-07-09 CN CN201810744414.3A patent/CN108958840B/en not_active Expired - Fee Related
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105005487A (en) * | 2015-06-29 | 2015-10-28 | 清华大学 | Continuous-service high-performance cluster operating system online upgrading method |
| CN106603281A (en) * | 2016-12-07 | 2017-04-26 | 乐视控股（北京）有限公司 | Configuration file management method and system |
| CN106843930A (en) * | 2016-12-23 | 2017-06-13 | 江苏途致信息科技有限公司 | Streaming dynamic configuration more new architecture and method based on zookeeper |
| CN107465714A (en) * | 2017-01-23 | 2017-12-12 | 北京思特奇信息技术股份有限公司 | A kind of configuration data dynamic update system and method based on application cluster |
| CN107025139A (en) * | 2017-03-21 | 2017-08-08 | 北京天云融创软件技术有限公司 | A kind of high-performance calculation Scheduling Framework based on cloud computing |
| CN107506213A (en) * | 2017-08-30 | 2017-12-22 | 深圳市小牛在线互联网信息咨询有限公司 | Dynamic Configuration, device, storage medium and computer equipment |
| CN108255592A (en) * | 2017-12-19 | 2018-07-06 | 武汉市烽视威科技有限公司 | A kind of Quartz clusters timing task processing system and method |
Non-Patent Citations (1)
| Title |
|---|
| Zookeeper分布式过程协同技术详解笔记（二）;codingBoyJack;《https://www.jianshu.com/p/034f1c6ecd6b》;20180612;第1-8页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108958840A (en) | 2018-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11630692B2 (en) | Efficient worker utilization | |
| WO2021088437A1 (en) | Version releasing method and device based on multi-environment offline task | |
| US7523196B2 (en) | Session monitoring using shared memory | |
| US8381212B2 (en) | Dynamic allocation and partitioning of compute nodes in hierarchical job scheduling | |
| CN104503845B (en) | A kind of task distribution method and system | |
| US9063783B2 (en) | Coordinating parallel execution of processes using agents | |
| CN113569987A (en) | Model training method and device | |
| CN105119913A (en) | Web server architecture based on Docker and interactive method between modules | |
| JP2014123365A (en) | DEVICE AND METHOD FOR OPTIMIZATION OF DATA PROCESSING IN MapReduce FRAMEWORK | |
| EP3226133A1 (en) | Task scheduling and resource provisioning system and method | |
| CN104850416B (en) | A kind of upgrade-system, method, apparatus and cloud computing node | |
| CN109117252B (en) | Method and system for task processing based on container and container cluster management system | |
| US9077613B2 (en) | System and method for graph based K-redundant resiliency for IT cloud | |
| CN112445598B (en) | Task scheduling method and device based on quartz, electronic equipment and medium | |
| CN112199192B (en) | Method and system for deploying Kubernetes cluster refined management quota based on server | |
| US20140258250A1 (en) | Flexible Control Framework Featuring Standalone Rule Engine | |
| CN106354563A (en) | Distributed computing system for 3D (three-dimensional reconstruction) and 3D reconstruction method | |
| JP7645877B2 (en) | Providing optimization in microservices architecture | |
| CN113439258B (en) | Hosting virtual machines on secondary storage systems | |
| CN108958840B (en) | A cluster configuration dynamic detection merge loading method | |
| CN109614242B (en) | A computing power sharing method, device, equipment and medium | |
| CN113641498A (en) | Data checking method, system, equipment and storage medium | |
| CN108701035B (en) | Management of application properties | |
| US11288004B1 (en) | Consensus-based authority selection in replicated network-accessible block storage devices | |
| CN110069343B (en) | Power equipment distributed storage and calculation architecture for complex high concurrency calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210122 |