CN103106044B - Classification storage power-economizing method - Google Patents

Abstract

The invention provides a hierarchical storage energy-saving method. The method includes the following steps: automatic storage classification: cluster startup, automatically identifying the storage level of each host, and adjusting the nodes with lower storage levels to the energy-saving mode in proportion; directional storage Fetch: Select a storage layer with a short distance and a high storage level to store and read files; find hot data: record the access information of each data block in the file, and obtain the value of each accessed data block according to the recorded information. A queue is formed from high to low. The hierarchical storage method of the invention ensures the energy saving of the cluster.

Description

Hierarchical Storage Energy Saving Method

technical field

The invention relates to a storage technology in the computer field, in particular to a hierarchical storage energy-saving method.

Background technique

With the explosive growth of data volume, server clusters for storing and processing massive amounts of data are becoming more and more common. The energy consumption of these server clusters has attracted more and more attention.

According to statistics, in the cost of building a server cluster, only the power consumption of servers and cooling systems accounts for 20%, while most servers are in a low-load state most of the time, generally not higher than 30%, resulting in a large waste of electricity. In order to try our best to reduce the unnecessary loss caused by this kind of power waste, cluster energy-saving technology came into being.

The key point of the current cluster energy-saving technology is to concentrate the tasks in the cluster to run on individual servers, while other servers are adjusted to the energy-saving state or turned off, so as to achieve the purpose of cluster energy conservation.

The foothold of these current cluster energy-saving technologies is that the access to data in the cluster is scattered and not fixed, which is related to the distribution of data in the entire cluster. Many of the current server clusters have implemented load balancing technology, so that the data in the cluster can be evenly distributed on the servers, preventing the situation that some servers are overloaded and other servers are idle, so as to achieve the purpose of concurrent processing.

However, industrial research shows that only 20% of data is active, while the remaining 80% of data is inactive, and the activity of these data will also change over time. Therefore, even if the cluster achieves load balancing, due to inconsistent data access characteristics, some servers must be heavily loaded while other servers are lightly loaded.

The current cluster energy-saving technology actually concentrates the load so that the entire cluster is in an unbalanced load state, and then adjusts idle nodes to an energy-saving state. This approach is actually the inverse process of load balancing. Although some problems have been temporarily solved, a price has also been paid. For example, monitoring the load of each node in the cluster requires instruments such as sensors, which also increases part of the cost.

Therefore, the low utilization rate of servers in the cluster and the large waste of power are actually the inevitable result of implementing load balancing technology in the entire cluster. However, if load balancing is not implemented, individual servers in the cluster may become access bottlenecks. Therefore, to solve the problem of cluster power consumption and ensure that individual servers in the cluster will not become an access bottleneck, a new data configuration method is required.

Contents of the invention

In order to solve the above-mentioned technical problems, the present invention provides a low-cost, highly automated hierarchical storage energy-saving method, the method comprising the following steps:

Automatic storage classification: when the cluster is started, the host name is used to identify the storage level of each host, and the nodes with lower storage levels are adjusted to the energy-saving mode in proportion;

Directed access: select a storage layer with a short distance, a high storage level, and a normal working mode to store and read files;

Find hot data: record the access information of each data block in the file, judge the migration opportunity, when the migration opportunity arrives, obtain the value of each access data block according to the recorded information, and form a queue according to the value from high to low;

Data block migration: Migrate data blocks with high value to the storage layer with high storage level, and migrate data blocks with low value to the storage layer with low storage level.

Preferably, the method further includes: adaptive adjustment: after the data migration is completed, update the access information of the data block, and restart the monitoring.

Preferably, the record information is processed through an information valuation model, and the data block access information includes access user, access time, and data block information.

Preferably, a specific data migration task is formed on the basis of the data block value queue obtained after processing by the information valuation model through the queue filtering model and the path matching model, and the data migration is completed using the migration control model.

Preferably, the queue filtering model is: filter out data segments that do not need to be migrated according to a threshold, and all data segments in the queue formed after filtering have already determined the migration direction, and the threshold reflects the previous migration on the storage level result.

Preferably, the path matching model is: after all the blocks in the queue have determined the migration direction, determine the migration source and migration target with a closer distance, and the migration source preferably chooses the one with less remaining space, light load, and normal working mode Nodes, the migration target prefers nodes with light loads.

Preferably, the migration control model is: performing migration rate control, using multithreading to execute the data migration tasks in batches, and reducing the impact of the migration process on node access performance in the cluster.

Preferably, the step of updating data block information and restarting monitoring is specifically:

Store the valuation results of the data block for use in the next valuation;

For the data blocks that have been deleted, delete them in the access records kept by the system;

Update the threshold value of each storage level according to the actual situation of migration;

Wake up the monitoring process and wait for the arrival of the next data migration.

Preferably, when the storage is automatically graded, the storage level includes at least two levels, and the storage level is divided according to the following criteria: the higher the storage level, the better the access performance, and the shorter the response time for processing user requests.

Preferably, 40% of the secondary storage tier and 60% of the tertiary storage tier are adjusted to an energy-saving mode.

The hierarchical storage energy-saving method of the present invention realizes the hierarchical storage technology in the cluster, uses the hierarchical storage method, uses hierarchical storage media in the cluster, fixes access hotspots on higher-level storage, and adjusts nodes with lower storage levels in proportion to The energy-saving mode ensures the energy saving of the cluster and saves costs.

Description of drawings

FIG. 1 is a schematic flowchart of a hierarchical storage energy-saving method according to an embodiment of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, it is a schematic flow chart of a hierarchical storage energy-saving method according to an embodiment of the present invention. The hierarchical storage method of the present invention includes the following steps:

Step S1: Storage automatic grading.

When the cluster is started, the host name is used to identify the storage level of each host including the storage layer, and the nodes with a lower storage level are adjusted to the energy-saving mode in proportion. In this embodiment, when the Hadoop cluster starts, the system can Automatically identify the access performance of each node. In this embodiment, 40% of the secondary storage layer and 60% of the tertiary storage layer are adjusted to the energy-saving mode; of course, in other embodiments, the number of storage layers and the proportion adjusted to the energy-saving mode can be adjusted arbitrarily. Belong to the scope of this patent protection.

Step S2: Directed access.

Select a storage layer with a short distance, a high storage layer, and a normal working mode to store and read files.

Step S3: Find hot data.

Record the access information of each data block in the file, and judge the migration opportunity. When the migration opportunity arrives, according to the recorded information, the value of each access data block is obtained, and a queue is formed according to the value from high to low. In this embodiment, The nodes in the cluster are divided into three different storage levels. The higher the storage level, the better the access performance of the configured hard disk, the smaller the capacity, and the more expensive the price. Therefore, only a small amount of data can be stored on the node with the highest storage level. Typically, only a small amount of all data in a cluster is accessed frequently. We record the access information of the file, process the information through the information valuation model, and obtain a value. The larger the value, the more frequently the data is accessed, and the higher the storage level should be; the client reads the file The fetching is in units of blocks, and the system records every read operation of a block. The recorded content includes: access user, access time, and data block information, etc., and the system generates a record every time it is read. At a specific moment, use the information valuation model to process these records. The processing object of the model is a block. The parameters used are: access time, number of visits, number of users, size of block, degree of association between block and other blocks, history of block Value, etc., use the formula to calculate a specific value to measure the "hot" degree of the block, and form a queue according to the value from high to low, the block value queue after the initial processing of the information valuation model, the data migration algorithm uses the queue filtering model, The path matching model forms a specific migration task, and finally uses the migration control model to complete the final data migration; the queue filtering model filters out data blocks that do not need to be migrated through thresholds at each storage level. These thresholds record the maximum value of all data blocks moved down and the minimum value of all data blocks moved up. All the blocks in the queue formed after filtering have already determined the migration direction. In other embodiments, when the storage is automatically classified, the storage level includes at least 2 levels. The division standard of the storage level is: the higher the storage level, the higher the access performance. The better, the shorter the response time to process user requests.

Step S4: data block migration.

Migrate data blocks with high value to the storage layer with high storage level, and migrate data blocks with low value to the storage layer with low storage level. After all the blocks in the queue have determined the migration direction, you need to determine the source and target of the migration . The migration source prefers nodes with less remaining space, light load, and normal working mode. If the space of the normal working mode node is insufficient, the node using the energy-saving mode will be automatically upgraded to the normal working mode. The migration target needs to have enough space to accommodate the migration Blocks, preferentially select nodes with light load. At the same time, the distance between the migration source and the migration target should be close enough. When all blocks in the queue have specific migration sources and migration targets, a specific migration task is formed. The control model uses multithreading to perform these migration tasks in batches. For example, only 50 threads are used for migration in each batch, and each node has at most 5 threads for performing migration tasks, so that the migration process affects the access performance of nodes in the cluster. have as little impact as possible.

Step S5: adaptive adjustment.

After the data migration is completed, the access information of the data block is updated, and the monitoring is restarted. In this embodiment, the migration period is adjusted in time according to the triggering condition of the migration. The steps of updating data block information and restarting monitoring are specifically:

Store the valuation results of the data block for use in the next valuation;

For the data blocks that have been deleted, delete them in the access records kept by the system;

Update the threshold value of each storage level according to the actual situation of migration;

Wake up the monitoring process and wait for the arrival of the next data migration.

During the migration process, some nodes in energy-saving mode (located on the secondary storage and tertiary storage) may change to normal working mode, indicating that the remaining space of the nodes in normal working mode in this level of storage is insufficient. According to the principle of locality of data access, nodes with heavy loads and no access records for two consecutive cycles are set to energy-saving mode, and some nodes in energy-saving mode are set to normal working mode to ensure the available space of this level of storage More than 10% of the total capacity is stored at this level.

After step S5, return to step S2, and the process of data scheduling is cyclically performed.

The hierarchical storage energy-saving method of the present invention uses a hierarchical storage method, uses hierarchical storage media in the hadoop cluster, and fixes access hotspots on higher-level storage, so that tasks do not need to be migrated, and only low-level storage The node is in the energy-saving state. In this way, the energy saving of the cluster is guaranteed, and individual servers in the cluster will not become the bottleneck of access, killing two birds with one stone.

It can be understood that those skilled in the art can make various other corresponding changes and deformations according to the technical concept of the present invention, and all these changes and deformations should belong to the protection scope of the claims of the present invention.

Claims (3)

1. A hierarchical storage energy-saving method, characterized in that said method comprises the following steps: Automatic storage grading: Hadoop cluster starts, uses the host name to identify the storage level of each host, and adjusts the nodes with lower storage levels to energy-saving mode in proportion; Directed access: select a storage layer with a short distance, a high storage level, and a normal working mode to store and read files; Find hot data: record the access information of each data block in the file, judge the migration opportunity, when the migration opportunity arrives, get the value of each access data block according to the recorded information, and form a queue according to the value from high to low; The value model processes the record information, and the access information of the data block includes access user, access time and data block information; Data block migration: Migrate high-value data blocks to a storage layer with a high storage level, and migrate low-value data blocks to a storage layer with a low storage level; through the queue filtering model and path matching model, after the information valuation model processes On the basis of the obtained data block value queue, a specific data migration task is formed, and the migration control model is used to complete the data migration; All the data segments in the queue have determined the migration direction, and the threshold reflects the previous migration result on the storage level; the path matching model is: after all the blocks in the queue have determined the migration direction, determine the migration with a closer distance source and migration target, the migration source preferentially selects nodes with less remaining space, light load, and normal working mode, and the migration target preferentially selects nodes with light load; the migration control model is: perform migration rate control, use multi-threaded batches Execute the data migration task to reduce the impact of the migration process on the access performance of nodes in the cluster; Adaptive adjustment: After the data migration is completed, update the data block access information and restart the monitoring. The specific steps are: Store the valuation results of the data block for use in the next valuation; For the data blocks that have been deleted, delete them in the access records kept by the system; Update the threshold value of each storage level according to the actual situation of migration; Wake up the monitoring process and wait for the arrival of the next data migration. 2. The hierarchical storage energy-saving method according to claim 1, characterized in that: when storage is automatically classified, the storage hierarchy includes at least 2 levels, and the division standard of storage hierarchy is: the higher the storage hierarchy, the better the access performance, The response time to process user requests is shorter. 3. The energy-saving method for tiered storage according to claim 1, characterized in that: 40% of the secondary storage tier and 60% of the tertiary storage tier are adjusted to an energy-saving mode.