CN110647379A - Hadoop cluster automatic scaling deployment and Plugin deployment method based on OpenStack cloud - Google Patents

Abstract

The invention belongs to the technical field of cloud computing, and in particular relates to a method for Hadoop cluster automatic scaling deployment and Plugin deployment based on OpenStack cloud. The method of the invention includes: in the Hadoop cluster deployment stage, a cluster automatic scaling strategy based on resource utilization rate and a replacement mechanism based on task success rate. The invention enables OpenStack to provide better support for the cluster, the cluster scale can be adjusted according to the business processing volume in different periods, and the speed of business processing is guaranteed.

Description

Hadoop cluster automatic scaling deployment and Plugin based on OpenStack cloud Deployment method

technical field

The invention belongs to the technical field of cloud computing, and relates to a method for Hadoop cluster automatic scaling deployment and Plugin deployment based on OpenStack cloud.

Background technique

The prior art discloses that Sahara can be integrated with third-party management tools (such as Apache Ambari and Cloudera management console) through a plug-in mechanism. The core part of Sahara is responsible for interacting with users, and provides OpenStack resources (such as virtual machines, servers, security groups, etc.) through Heat components; Plugin is responsible for installing and configuring Hadoop clusters in pre-allocated virtual machines. In addition, Plugin can also Tools for cluster deployment management and monitoring. Sahara provides a unified mechanism for Plugins to work in pre-provisioned virtual machines: on the one hand, Plugins must inherit the sahara.plugins.provisioning:ProvisioningPluginBase class and need to implement all necessary methods/interfaces; on the other hand, Sahara provides The virtual machine object has a remote attribute that can be used to interact with the virtual machine, and remotely invoke commands to operate the virtual machine through the instance.remote statement (available commands can be found in sahara.utils.remote.InstanceInteropHelper).

Based on the current state of the prior art, the inventor of the present application proposes a method for automatic scaling deployment and Plugin deployment of Hadoop clusters based on OpenStack cloud to supplement and optimize the automatic scaling mechanism of Hadoop cluster deployment. Scale to remove redundant nodes, replace problem nodes, deploy new nodes.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for Hadoop cluster automatic scaling deployment and Plugin deployment based on OpenStack cloud based on the current state of the prior art, supplement and optimize the deployment automatic scaling mechanism related to Hadoop cluster, and its automatic scaling will be very important to the cluster. Scale to remove redundant nodes, replace problem nodes, deploy new nodes.

The object of the present invention is implemented through the following technical solutions:

A method for automatic scaling and deployment of Hadoop clusters based on OpenStack cloud proposed by the present invention includes, based on the Sahara module in the OpenStack cloud, integrating with third-party management tools through the Plugin mechanism, referring to requirements, and combining with the automatic scaling and deployment method, for all Allocate an appropriate amount of virtual machines to the required Hadoop cluster, and install and configure the Hadoop cluster in the pre-allocated virtual machines.

Specifically, a method for automatic scaling deployment of Hadoop clusters based on OpenStack cloud of the present invention is characterized in that it completes the automatic scaling deployment of Hadoop clusters in a cloud environment according to prediction and real-time conditions; specifically, it includes:

(1) Automatic scaling strategy based on utilization

分别代表用户对于集群CPU、内存RAM和硬盘Disk这三个利用率的期待值，而实际情况下这三者利用率值分别为

由于用户对于不同的资源之间的重视程度不同，所以在其中引入了λ_C、λ_R、λ_D这三项分别作为三者的权值，由此可以得到以下数据，Introduced by the present invention

Represents the user's expected value of the three utilization rates of the cluster CPU, memory RAM, and hard disk Disk, respectively, and the actual utilization values of the three are

Since users attach different importance to different resources, three items λ _C , λ _R , and λ _D are introduced as the weights of the three, respectively, and the following data can be obtained:

φ=λ _C · η _C +λ _R · η _R +λ _D · η _D (Definition 2)

Among them, definition 1 represents the difference between the actual utilization rate and the expected utilization rate under each indicator, and this indicator can specifically reflect the gap between various indicators; definition 2 represents the comprehensive difference between these three items and the expectation , the range of this value is also within [0,1), the closer it is to 0, the closer it is to the expected value, and the more in line with the expected cluster utilization;

Based on the above, the present invention proposes an automatic scaling strategy based on utilization;

(2) Automatic scaling and rapid deployment strategy based on task success rate

是一个介于[0,1]的百分数，代表了单个节点上任务能够成功运行的比例；在最理想的情况下，各个节点

表示所有的任务都能被成功执行，结果也能顺利输出；由于节点任务执行的失败不可避免，不可能只要出错就立刻对该节点进行置换，这是不合理的，也会导致系统开销的增大；所以会对

有一个预估值

如果该值趋近于0，则代表该节点的任务成功率过于小，这种情况下，该节点的继续使用会降低集群的运行效率，所以将启用节点置换策略；The present invention introduces a variable in the strategy

is a percentage between [0,1], representing the proportion of tasks that can run successfully on a single node; in the most ideal case, each node

Indicates that all tasks can be successfully executed and the results can be output smoothly; since the failure of node task execution is inevitable, it is impossible to replace the node immediately as long as there is an error, which is unreasonable and will also lead to an increase in system overhead. large; so will

have an estimate

If the value is close to 0, it means that the task success rate of the node is too small. In this case, the continued use of the node will reduce the operating efficiency of the cluster, so the node replacement strategy will be enabled;

Based on the above, the present invention proposes an automatic scaling and rapid deployment strategy based on task success rate.

In the present invention, the automatic scaling deployment of Hadoop clusters based on the OpenStack cloud includes the following two processes:

1. Automatic scaling deployment mechanism

(1) Automatic scaling strategy based on utilization

The ultimate goal of the automatic scaling strategy algorithm based on cloud platform resource utilization is to combine cloud platform resources with the current application scenario requirements, make more rational use of cloud platform computing resources, optimize the Hadoop cluster-related cluster deployment automatic scaling mechanism, and make the cluster operating efficiency to achieve better results;

Usually, the usage of the three indicators of CPU, memory and hard disk plays an important role in the realization of the automatic scaling function based on the utilization rate, and the present invention mainly starts from the three indicators to reflect the usage rate;

分别代表用户对于集群CPU、内存RAM和硬盘Disk三个利用率的期待值，而实际情况下这三者利用率值分别为

由于用户对于不同的资源之间的重视程度不同，所以在其中引入了λ_C、λ_R、λ_D这三项分别作为三者的权值，由此得到以下数据，Introduced by the present invention

Represents the user's expected value of the three utilization rates of the cluster CPU, memory RAM, and hard disk Disk, respectively, and the actual utilization values of the three are

Since users attach different importance to different resources, three items λ _C , λ _R , and λ _D are introduced as the weights of the three, respectively, and the following data are obtained:

φ=λ _C · η _C +λ _R · η _R +λ _D · η _D (Definition 2)

是一个可以的区间范围，故η的计算结果为一个值区间，由于需要计算与用户期望之间的符合程度，一区这个包含在在[0,1)区间的子区间范围内的最小值；定义2代表了综合这三项与期望之间的差值，这个值的范围也在[0,1)之内，越接近于0，则与期待值越接近，集群的利用率越符合预期；Definition 1 represents the difference between the actual utilization rate and the expected utilization rate under each indicator. This indicator specifically reflects the gap between various indicators, and the platform adjusts the cluster resource configuration accordingly;

is a possible interval range, so the calculation result of n is a value interval, because the degree of conformity between the calculation and the user's expectation needs to be calculated, the minimum value of one area is included in the sub-interval range of the [0,1) interval; Definition 2 represents the difference between these three items and the expectation. The range of this value is also within [0, 1). The closer it is to 0, the closer it is to the expected value, and the cluster utilization is more in line with expectations;

In the algorithm of the present invention, the user's expected value of the three utilization rates of the cluster CPU, memory RAM and hard disk Disk is first obtained, and the actual utilization value of the three in the platform is compared. If the actual utilization value is less than the corresponding expected value If the actual utilization value is greater than the minimum value of the corresponding expected value but less than the maximum value of the corresponding expected value, then start the virtual machine, deploy the Hadoop cluster, and start it;

(2) Automatic scaling and rapid deployment strategy based on task success rate

Based on the design process of Hadoop, it is believed that any node may have various problems and cause the execution of the assigned task to fail. The present invention adopts the method of replacing the node with a higher failure rate, which can avoid some physical aspects of the node itself. failure caused by factors;

At the same time, before the node that replaces the node that provides computing services officially serves the cluster, the present invention needs to perform point-to-point copying of data blocks to ensure that the data can be stored correctly. The implementation of this strategy can better improve the state of the entire cluster. Adjustment to avoid the increase in the load of other nodes due to the failure of a single node, or even the occurrence of scale effects;

是一个介于[0,1]的百分数，代表了单个节点上任务能够成功运行的比例；在最理想的情况下，各个节点

表示所有的任务都能被成功执行，结果也能顺利输出，由于节点任务执行的失败不可避免，不可能只要出错就立刻对该节点进行置换，这不仅是不合理的，也会导致系统开销的增大，所以本发明中，对

有一个预估值

如果该值趋近于0，则代表该节点的任务成功率过于小，该节点的继续使用会降低集群的运行效率，所以将启用节点置换策略；In order to realize this strategy, in the present invention, a variable is first introduced

is a percentage between [0,1], representing the proportion of tasks that can run successfully on a single node; in the most ideal case, each node

Indicates that all tasks can be successfully executed, and the results can be output smoothly. Since the failure of node task execution is inevitable, it is impossible to replace the node immediately as soon as there is an error. This is not only unreasonable, but also causes system overhead. increases, so in the present invention, the

have an estimate

If the value is close to 0, it means that the task success rate of the node is too small, and the continued use of the node will reduce the operating efficiency of the cluster, so the node replacement strategy will be enabled;

The procedural steps of the algorithm of the present invention are described as follows:

值作为衡量任务成功率的最低标准step1: choose a suitable

value as the minimum standard to measure the success rate of the task

step2: Calculate the success rate of node tasks within a certain period of time value

值进行比较，若大于，则继续下一个节点开始step2；若小于，则进入下一步step3: will value with

The value is compared, if it is greater than, continue to the next node to start step2; if it is less than, go to the next step

step4: apply for a new node

step5: Deploy the Hadoop application on the newly applied node and copy the data on the original node

step6: Start the service on the new node and suspend the service of the original node

step7: Start the service on the new node, terminate the original node, and enter step2 of the next node

Through node replacement, an optimized task execution effect of the entire cluster is finally achieved. During this replacement process, failures caused by the physical aspects of the virtual machine can be avoided to the greatest extent.

2. Cluster automatic deployment Plugin implementation

(1) Cluster Plugin implementation interface

As an independent plugin, the cluster exists in the form of an independent directory under the sahara/sahara/plugins directory, and its main directory structure is shown in Figure 1;

in:

1. As shown in Figure 2, the v2_7_1 directory is the special content of the sandbox plugin version 2.7.1, the hadoop2 is the general content, and the outermost versionfactory.py identifies the directory name through regular matching to get the corresponding version number ;

2.plugin.py is the core and is responsible for implementing all the necessary interfaces. The specific interfaces to be implemented are shown in Table 1:

3. As shown in Figure 3, in terms of function implementation, it is mainly divided into two parts: cluster configuration and startup. config_helper.py is the core module of configuration, in which the path of relevant configuration files is configured, and the configuration of environment variables is performed;

4. As shown in Figure 5, in versionhandle.py, the configuration and startup of the sandbox are completed according to the current plugin version;

5. And run_scripts.py/starting_scripts.py specifically realizes the startup of the cluster, in which the instance.remote() method is used to remotely connect to the virtual machine that has been started through ssh, and execute the corresponding linux command, so as to specifically control the process, node, Startup of clusters, etc.

(2) Implementation of cluster image packaging and production tools

1. OpenStack virtual machine image

In the embodiment of the present invention, taking the CentOS operating system as an example, the production process and principle of the OpenStack virtual machine image are briefly introduced;

2. Cluster mirroring

In the present invention, Diskimage-builder is used to make a cluster image.

In the present invention, the experiment of automatic scaling deployment of Hadoop cluster is carried out,

Select any set of deployments as the representative of the experiment, conduct multiple tests, and compare the indicator data before and after optimization on this basis, and analyze the deployment services. Table 2 shows the cluster configuration.

Table 2 Cluster Configuration

cluster vCPU RAM disk Node number Cluster 1 4core 10GB 5GB 8 Cluster 2 2core 100GB 100GB 16 Cluster 3 1core 5GB 80GB 8 Cluster 4 1core 5GB 80GB 16 Cluster 5 1core 5GB 80GB twenty four Cluster 6 1core 5GB 80GB 48

The test results show that the deployment speed after optimization has been improved considerably compared with that before optimization. When the number of cluster nodes to be deployed is small, the optimization effect is not significant, but when the number of cluster nodes increases, The deployment time before optimization has increased significantly, while the deployment time after optimization has increased as the cluster size increases, but the increase is moderate. The six cluster deployment times are relatively close, and the deployment time is stable at 10 minutes to 20 minutes. The results show that the deployment time of the optimized cluster deployment has been significantly improved, and the required time is more stable; the optimized deployment service is better than before the optimization, even in the small-scale cluster deployment process , it still reflects its optimization effect, and in terms of success rate, the effect after optimization has been significantly improved, which shows that the deployment service optimization of the present invention is relatively successful. The automatic scaling deployment strategy of the Hadoop cluster proposed by the invention can optimize the automatic deployment of the Hadoop cluster and make the deployment service more stable and efficient.

Description of drawings

Figure 1 shows the directory structure diagram that the cluster exists as an independent plugin in the sahara/sahara/plugins directory in the form of an independent directory.

Figure 2 shows that the v2_7_1 directory is the content specially made for version 2.7.1 of the sandbox plugin, and the general content in hadoop2. The outermost versionfactory.py identifies the directory name through regular matching to get the corresponding version number.

Figure 3 shows that in terms of function implementation, it is mainly divided into two parts: cluster configuration and startup. config_helper.py is the core module of configuration, which configures the path of relevant configuration files and configures environment variables. config_helper.py also It is responsible for generating the corresponding configuration file according to the user configuration for the sandbox to be started.

Figure 4 is a screenshot of what config_helper.py does when configuring spark environment variables as shown in Figure 3.

Figure 5 shows that in versionhandle.py, the configuration and startup of the sandbox are completed according to the current plugin version.

The start_cluster method describes the overall process of sandbox startup.

Figure 6 shows that a virtual machine image in qcow2 format is created with a size of 10G.

FIG. 7 is a schematic diagram of a cluster.

Figure 8 shows the web page of the cluster creation service, on which the relevant requirements for cluster creation are submitted, including the selection of the Hadoop version, the configuration of the nodes listed in Table 1, the selection of images, etc. After completing the relevant designations , the cluster can enter the deployment phase.

Figure 9 shows that the deployment speed after optimization has a considerable improvement compared with that before optimization, and the deployment time of the optimized cluster deployment is significantly improved, and the time required is more stable.

Figure 10 shows that compared to before optimization, the optimized deployment service still reflects its optimization effect even in the small-scale cluster deployment process. In terms of success rate, the optimized effect has been significantly improved, reflecting the The present invention deploys the success of service optimization.

Detailed ways

The technical solutions of the present invention are described below in detail with reference to the accompanying drawings and embodiments.

The purpose of the present invention is to propose a method for automatic scaling and deployment of Hadoop clusters based on OpenStack cloud. As shown in Figure 1, the present invention is based on the Sahara module in the OpenStack cloud, and is integrated with a third-party management tool through the Plugin mechanism. With reference to the requirements, combined with the automated scaling deployment method, an appropriate amount of virtual machines are allocated to the required Hadoop clusters. Install and configure a Hadoop cluster in an assigned virtual machine.

In the present invention, the automatic scaling deployment of Hadoop clusters based on the OpenStack cloud is specifically divided into the following two processes:

1. Automatic scaling deployment mechanism

(1) Automatic scaling strategy based on utilization

The ultimate goal of the automatic scaling strategy algorithm based on cloud platform resource utilization of the present invention is to combine cloud platform resources and current application scenario requirements, make more rational use of cloud platform computing resources, and optimize Hadoop cluster-related cluster deployment automatic scaling mechanism , so that the operating efficiency of the cluster can achieve better results.

When allocating the number of resources occupied by virtual machines, we usually pay attention to three aspects: CPU, memory, and hard disk. Therefore, in the implementation of the automatic scaling function based on utilization, the usage of these three indicators must be among them. There are several important aspects; in this embodiment, in terms of the utilization rate, the utilization rate is mainly reflected from these three indicators.

分别代表用户对于集群CPU、内存RAM和硬盘Disk这三个利用率的期待值，而实际情况下这三者利用率值分别为

由于用户对于不同的资源之间的重视程度不同，所以在其中引入了λ_C、λ_R、λ_D这三项分别作为三者的权值。由此可以得到以下数据，Introduced by the present invention

Represents the user's expected value of the three utilization rates of the cluster CPU, memory RAM, and hard disk Disk, respectively, and the actual utilization values of the three are

Since users place different emphasis on different resources, three items λ _C , λ _R , and λ _D are introduced as the weights of the three, respectively. From this, the following data can be obtained,

φ=λ _C · η _C +λ _R · η _R +λ _D · η _D (Definition 2)

一般会是一个可以的区间范围，故η的计算结果也为一个值区间，由于需要计算与用户期望之间的符合程度，一区这个包含在在[0,1)区间的子区间范围内的最小值；定义2代表了综合这三项与期望之间的差值，这个值的范围也在[0,1)之内，越接近于0，则与期待值越接近，集群的利用率越符合预期；Definition 1 represents the difference between the actual utilization and the expected utilization under each indicator. This indicator can specifically reflect the gap between various indicators, and the platform can adjust the cluster resource configuration accordingly;

Generally, it is a valid range, so the calculation result of n is also a value range. Since it is necessary to calculate the degree of compliance with the user's expectation, the first range is included in the sub-range of the [0,1) range. The minimum value; Definition 2 represents the difference between these three items and the expectation. The range of this value is also within [0, 1). The closer it is to 0, the closer it is to the expected value, and the higher the utilization rate of the cluster. In line with expectations;

The algorithm first obtains the user's expected values for the three utilization rates of the cluster CPU, memory RAM and hard disk Disk, and compares the actual utilization values of the three in the platform. If the actual utilization value is less than the minimum value of the corresponding expected value. If the actual utilization value is greater than the minimum value of the corresponding expected value but less than the maximum value of the corresponding expected value, then start the virtual machine, deploy the Hadoop cluster, start,

(2) Automatic scaling and rapid deployment strategy based on task success rate

In the design process of Hadoop, it is believed that any node may have various problems and cause the execution of the assigned task to fail, and the failed task will be re-run. If the probability of node failure increases, although Hadoop can assign tasks to its own. Considering the standard, the running time of the entire cluster will increase. The higher the node task failure rate, the greater the growth rate. Even at this time, the node has been performing task calculation, and the utilization rate of each resource is relatively reasonable, but It actually has a certain impact on the operation of the entire task, and there are many reasons for the failure of the task. In order to cause as little impact as possible on other nodes in the entire cluster, the present invention adopts the method of replacing nodes with a higher failure rate to avoid Failure caused by some physical factors of the node itself;

At the same time, due to the principle that mobile computing is more economical than mobile data, Hadoop will try to allocate computing tasks to nodes that already have data blocks required for the calculation. Before, this node has completed the required data in safe mode. Storage, if these data are deleted directly, it will inevitably have a certain impact on the entire cluster, which may cause the cluster to enter the safe mode again and need to wait. The point-to-point copy of data blocks ensures that the data can be stored correctly. The implementation of this strategy can better adjust the state of the entire cluster, avoid the increase in the load of other nodes caused by the failure of a single node, and even cause scale effects. Happening;

是一个介于[0,1]的百分数，代表单个节点上任务能够成功运行的比例；在最理想的情况下，各个节点

表示所有的任务都能被成功执行，结果也能顺利输出，由于节点任务执行的失败不可避免，不可能只要出错就立刻对该节点进行置换，这是不合理的，也会导致系统开销的增大；本发明中对

有一个预估值

如果该值趋近于0，则代表该节点的任务成功率过于小，此时，该节点的继续使用会降低集群的运行效率，所以本发明将启用节点置换策略；In order to realize the above strategy, the present invention first introduces a variable

is a percentage between [0,1], representing the proportion of tasks that can run successfully on a single node; in the most ideal case, each node

Indicates that all tasks can be successfully executed and the results can be output smoothly. Since the failure of node task execution is inevitable, it is impossible to replace the node immediately as long as there is an error. This is unreasonable and will also lead to an increase in system overhead. large; in the present invention

have an estimate

If the value is close to 0, it means that the task success rate of the node is too small. At this time, the continued use of the node will reduce the operating efficiency of the cluster, so the present invention will enable the node replacement strategy;

The process steps of the algorithm are described as follows:

值作为衡量任务成功率的最低标准step1: choose a suitable

value as the minimum standard to measure the success rate of the task

值step2: Calculate the success rate of node tasks within a certain period of time

value

值与

值进行比较，若大于，则继续下一个节点开始step2；若小于，则进入下一步step3: will

value with

The value is compared, if it is greater than, continue to the next node to start step2; if it is less than, go to the next step

step4: apply for a new node

step5: Deploy the Hadoop application on the newly applied node and copy the data on the original node

step6: Start the service on the new node and suspend the service of the original node

step7: Start the service on the new node, terminate the original node, and enter step2 of the next node

Through node replacement, a more optimized task execution effect for the entire cluster is finally achieved. During this replacement process, failures caused by the physical aspects of the virtual machine can be avoided to the greatest extent.

3. Cluster automatic deployment Plugin implementation

(3) Cluster Plugin implementation interface

As an independent plugin, the cluster exists in the form of an independent directory under the sahara/sahara/plugins directory. Its main directory structure is shown in Figure 1.

in:

As shown in Figure 2, the v2_7_1 directory is the content specially made for version 2.7.1 of the sandbox plugin, the general content in hadoop2, and the outermost versionfactory.py identifies the directory name through regular matching to get the corresponding version number .

plugin.py is the core and is responsible for implementing all the necessary interfaces. The specific interfaces to be implemented are shown in Table 1:

Table 1

(cluster,instances)

Figure 3 shows that in terms of function implementation, it is mainly divided into two parts: cluster configuration and startup. config_helper.py is the core module of configuration, in which the path of relevant configuration files is configured, and environment variables are configured; in addition, config_helper .py is also responsible for generating the corresponding configuration file according to the user's configuration for the sandbox to be started. Figure 4 intercepts a small part of the work done by config_helper.py when configuring the spark environment variables;

As shown in Figure 5, in versionhandle.py, the configuration and startup of the sandbox are completed according to the current plugin version. The start_cluster method specifies the overall process of sandbox startup;

And run_scripts.py/starting_scripts.py specifically implements the startup of the cluster, in which the instance.remote() method is used to remotely connect to the virtual machine that has been started through ssh, and execute the corresponding linux commands, so as to specifically control the process, node, cluster, etc. start.

(4) Implementation of cluster image packaging and production tools

OpenStack virtual machine image:

Taking the CentOS operating system as an example, the production process and principle of the OpenStack virtual machine image in this embodiment are as follows:

1) Download the CentOS installation ISO image;

2) Install through the virt-manager tool or the virt-install command. Figure 6 is an example of installation using the command line;

Figure 6 shows the creation of a virtual machine image in qcow2 format with a size of 10G. If virt-manager is used, it can be installed step-by-step through graphical prompts. During the installation process, some additional configurations are required, such as changing the Ethernet Network status, hostname settings, specifying installation sources and storage devices, partitioning disks, setting root passwords, etc.;

3) After step 2), log in to the newly installed virtual machine through the root user, and perform related configurations, such as installation of ACPI services, installation of cloud-init packages, configuration of partition resizing support, disabling of zeroconf routing, configuration control log output, etc.;

4) After completing the configuration, shut down the virtual machine;

5) Clear the MAC address information;

6) Compress the image.

After completing the above steps, a common OpenStack virtual machine image is created and can be uploaded to the OpenStack platform for use.

Cluster image:

For clusters of different types and versions, images of corresponding types and versions are required to support them. When creating a cluster image, in addition to the basic OpenStack image creation steps, all related software packages (such as Hadoop and Spark) need to be performed in the image. Download, install and configure, in the present invention, use Diskimage-builder to make cluster image.

The present invention conducts the experiment of automatic scaling deployment of Hadoop cluster

Select any set of deployments as the representative of the experiment, conduct multiple tests, and compare the indicator data before and after optimization on this basis, and analyze the deployment services.

Table 2 Cluster Configuration

cluster vCPU RAM disk Node number Cluster 1 4core 10GB 5GB 8 Cluster 2 2core 100GB 100GB 16 Cluster 3 1core 5GB 80GB 8 Cluster 4 1core 5GB 80GB 16 Cluster 5 1core 5GB 80GB twenty four Cluster 6 1core 5GB 80GB 48

In the experiment, six different Hadoop clusters will be deployed. As shown in Figure 6, six clusters of different scales and configurations are studied. There are 6 physical computing nodes in this experiment. Since Openstack will determine the resource usage for the location of the virtual machine. Therefore, in the first and second clusters, the virtual machines will be evenly distributed on each node. In order to reduce the uncertainty caused by other factors, in this embodiment, it is directly performed on 6 identical machines. Cluster deployment, Table 2 shows the specific node resource configuration of these two clusters;

Figure 8 shows the web page of the cluster creation service, on which the relevant requirements for cluster creation are submitted, including the selection of Hadoop version, the configuration of nodes as shown in Table 1, the selection of images, etc. After the designation of , the cluster can enter the deployment stage. In this experiment, the experimental results are compared before and after optimization in terms of deployment speed and success rate;

In these two sets of comparative experiments, six Hadoop clusters were deployed respectively. The deployment of each cluster was carried out several times, and the abnormal or failed situations were excluded, and the average value was taken as the deployment time of the cluster. Figure 9 shows the results. Compared with before optimization, the deployment speed after optimization is significantly improved. When the number of cluster nodes to be deployed is small, the optimization effect is not significant, but when the number of cluster nodes increases, the deployment time before optimization The increase is obvious, and the deployment time after the optimization increases with the increase of the cluster size, but the increase is relatively moderate. The six cluster deployment times are relatively close, and the deployment time is stable within the range of 10 minutes to 20 minutes. , the results show that the deployment time of the optimized cluster deployment of the present invention is significantly improved, and the required time is more stable; as shown in Figure 10, the optimized deployment service is compared to before the optimization, even in a small-scale cluster deployment In the process, its optimization effect is still reflected; the results show that with the increase of cluster size, various uncertainties lead to a certain reduction in the success rate of cluster deployment. Before optimization, this decline is obvious. The stability of the deployment service is poor; after optimization, although the success rate is also reduced, the decline is small, and the decline tends to be gentle, still maintained at a high level; the experimental results confirm that in terms of the success rate, After optimization, the effect is obviously improved, and this deployment service optimization is relatively successful. The automatic scaling deployment strategy of the Hadoop cluster of the present invention can optimize the automatic deployment of the Hadoop cluster, so that the deployment service is more stable and efficient.

Claims (2)

1. a Hadoop cluster automatic scaling deployment method is carried out based on OpenStack cloud, it is characterized in that, it comprises, based on the Sahara module in OpenStack cloud, integrates by Plugin mechanism with third-party management tool, with reference to requirements, in conjunction with automatic scaling deployment method, Allocate the right amount of virtual machines for the desired Hadoop cluster, install and configure the Hadoop cluster in the pre-allocated virtual machines. 2. by the described method of claim 1, it is characterized in that, this method completes the automatic scaling deployment of Hadoop cluster under cloud environment according to prediction and real-time situation, comprising: (1) Automatic scaling strategy based on utilization introduce

Represents the user's expected value of the three utilization rates of the cluster CPU, memory RAM and hard disk Disk, respectively. Replace the actual situation of the three utilization values as l _C , l _R , and l _D respectively. The importance of λ C , λ R , and λ D are different, and three items λ _C , λ _R , and λ _D are introduced as the weights of the three respectively; thus the following data are obtained, φ=λ _C · η _C +λ _R · η _R +λ _D · η _D (Definition 2) Among them, Definition 1 represents the difference between the actual utilization rate and the expected utilization rate under each indicator, and this indicator specifically reflects the gap between various indicators; Definition 2 represents the comprehensive difference between the three items and the expectation, the The range of the value is within [0, 1), the closer it is to 0, the closer it is to the expected value, and the more in line with the expected cluster utilization; (2) Automatic scaling and rapid deployment strategy based on task success rate Introduce a variable into the strategy

is a percentage between [0,1], representing the proportion of tasks that can run successfully on a single node; each node

Indicates that all tasks can be successfully executed and the results can be output smoothly; The failure of node-based task execution is inevitable.

set an estimate

If the value is close to 0, it means that the task success rate of this node is too small, and the continued use of this node will reduce the operating efficiency of the cluster, and the node replacement strategy will be enabled.

Images