CN103106044A - Classification storage energy-saving method - Google Patents
Classification storage energy-saving method Download PDFInfo
- Publication number
- CN103106044A CN103106044A CN2012105394424A CN201210539442A CN103106044A CN 103106044 A CN103106044 A CN 103106044A CN 2012105394424 A CN2012105394424 A CN 2012105394424A CN 201210539442 A CN201210539442 A CN 201210539442A CN 103106044 A CN103106044 A CN 103106044A
- Authority
- CN
- China
- Prior art keywords
- migration
- data block
- data
- storage
- memory hierarchy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Supply And Distribution Of Alternating Current (AREA)
Abstract
本发明提供一种分级存储节能方法，所述方法包括以下步骤：存储自动分级：集群启动，自动识别各主机所处于的存储层次，并按照比例将存储层次低的节点调整为节能模式；定向存取：选择距离近、存储层次高的存储层存储和读取文件；寻找热数据：记录文件中各数据块的访问信息，根据所述记录信息，得出每个访问数据块的价值，按照价值从高到低形成队列。本发明的分级存储方法保证了集群的节能。
The invention provides a hierarchical storage energy-saving method. The method includes the following steps: automatic storage classification: cluster startup, automatically identifying the storage level of each host, and adjusting the nodes with lower storage levels to the energy-saving mode in proportion; directional storage Fetch: Select a storage layer with a short distance and a high storage level to store and read files; find hot data: record the access information of each data block in the file, and obtain the value of each accessed data block according to the recorded information. A queue is formed from high to low. The hierarchical storage method of the invention ensures the energy saving of the cluster.
Description
Technical field
The present invention relates to a kind of memory technology of computer realm, relate in particular to a kind of classification storage power-economizing method.
Background technology
Along with the explosive growth of data volume, the server cluster of Storage and Processing mass data is more and more general.The energy consumption problem of these server clusters more and more causes people's concern.
According to statistics, in the cost that builds a server cluster, only the power consumption of server and cooling system has just occupied 20%, and most of server all is in low load condition in the time of majority, generally not higher than 30%, caused very large power wastage.The unnecessary loss that brings in order to reduce as possible this power wastage, the cluster power-saving technology is arisen at the historic moment.
The power-saving technology of current cluster, its key point are to operate on individual servers in the task-set in cluster, and other servers are adjusted into power save mode or turn off, thereby reach the energy-conservation purpose of cluster.
The foothold of current these cluster power-saving technologies is, in cluster, the access of data is to disperse and unfixed, and this is relevant with the distribution of data in whole cluster.Present server cluster has much all been realized load-balancing technique, make data in cluster can be on server mean allocation, prevent the individual servers overload and the idle situation of other servers, to reach the purpose of concurrent processing.
But industrial research shows, only has 20% data to enliven, and be in disabled state and remain 80% data, and the activity of these data also can time to time change.Even if therefore cluster has reached load balancing, but because the access characteristics of data is inconsistent, be bound to occur the individual servers load heavy, and the light situation of all the other server loads.
Current this cluster power-saving technology is with load centralization in fact, makes whole cluster be in again the state of load imbalance, then idle node is adjusted to power save mode.This way is the inverse process of load balancing in fact.Though temporarily solved subproblem, also paid cost, for example the load of each node in cluster is monitored, need the sensor instrument, increased again departmental cost.
Thus, the server utilization rate in cluster is low, wastes in a large number electric energy, is in fact to carry out the inevitable outcome that load-balancing technique brings in whole cluster.If but do not realize load balancing, may make the individual servers in cluster become access bottleneck.Therefore, solve the problem of cluster power consumption, guarantee that again the individual servers in cluster can not become access bottleneck, just need a brand-new data configuration mode.
Summary of the invention
The present invention provides the classification storage that a kind of cost is low, automaticity is high power-economizing method for solving the problems of the technologies described above, and said method comprising the steps of:
The storage automatic classification: cluster starts, and utilizes the present memory hierarchy of host name identification each main frame of identification, and proportionally knot adjustment that memory hierarchy is low is energy saver mode;
Directed access: chosen distance is near, memory hierarchy is high, the accumulation layer of normal mode of operation is stored and file reading;
Seek dsc data: the visit information of each data block in log file, judgement migration opportunity when migration arrives opportunity, according to described recorded information, draw the value of each visit data piece, form from high to low formation according to being worth;
Data block migration: costly data block is moved to the high accumulation layer of memory hierarchy, move to the low accumulation layer of memory hierarchy with being worth low data block.
Preferably, described method also comprises: the self-adaptation adjustment: after Data Migration was completed, more the new data block visit information, restarted monitoring.
Preferably, process described recorded information by the information Valuation Modelling, described data block visit information comprises calling party, access time and data block information.
Preferably, by formation filtering model and route matching model, on the basis of the data block value formation that obtains, form concrete Data Migration task after the information Valuation Modelling is processed, utilize migration to control model and complete Data Migration.
Preferably, described formation filtering model is: fall the not data sectional of needs migration according to threshold filtering, all data sectionals in the formation that forms after filtering have all been determined migratory direction, and threshold value has reflected previous migration results on this memory hierarchy.
Preferably, described route matching model is: after all pieces have all been determined migratory direction in formation, determine migration source and the migration target of close together, the migration source selects preferentially that remaining space is less, load light, the node of normal mode of operation, and the migration target priority is selected the light node of load.
Preferably, described migration is controlled model and is: carry out migration rate and control, use multithreading to carry out in batches described Data Migration task, reduce transition process to the impact of node visit performance in cluster.
Preferably, described renewal data block information, the step that restarts monitoring is specially:
The valuation result of storage data block is used during in order to valuation next time;
For deleted data block, delete in the Visitor Logs that system keeps;
Carry out the threshold value of each memory hierarchy upgrades according to the actual conditions of migration;
The awaking monitoring process is waited for the arrival of Data Migration next time.
Preferably, when the storage automatic classification, described memory hierarchy comprises 2 grades at least, and the criteria for classifying of memory hierarchy is: memory hierarchy is higher, and access performance is better, and the response time of processing user's request is shorter.
Preferably, 40% secondary storage layer and 60% tertiary storage layer are adjusted into energy saver mode.
Classification storage power-economizing method of the present invention is realized the classification memory technology at cluster, use the method for classification storage, use the level storage medium in cluster, the access focus is fixed in the storage of higher level, proportionally the knot adjustment that memory hierarchy is low is energy saver mode, has guaranteed the energy-conservation of cluster and has saved cost.
Description of drawings
Fig. 1 is one embodiment of the invention classification storage power-economizing method schematic flow sheet.
Embodiment
Below in conjunction with accompanying drawing and specific embodiment, the present invention is described in further detail.
As shown in Figure 1, be one embodiment of the invention classification storage power-economizing method schematic flow sheet, the method for classification storage of the present invention comprises the following steps:
Step S1: storage automatic classification.
Cluster starts, and utilizes host name to identify the memory hierarchy that each main frame comprises accumulation layer, and proportionally knot adjustment that memory hierarchy is low is energy saver mode, in the present embodiment, when the hadoop cluster started, by " host name identification method ", system can identify the access performance of each node automatically.In the present embodiment, the secondary storage layer with 40% and 60% tertiary storage layer are adjusted into energy saver mode; Certainly, in other embodiments, what and the ratio that is adjusted to energy saver mode of accumulation layer can be regulated arbitrarily, all belong to the scope of this patent protection.
Step S2: directed access.
Chosen distance is near, memory hierarchy is high, the accumulation layer of normal mode of operation is stored and file reading.
Step S3: seek dsc data.
The visit information of each data block in log file, judgement migration opportunity is when migration arrives opportunity, according to described recorded information, draw the value of each visit data piece, form from high to low formation according to being worth, in the present embodiment, the node in cluster is divided into 3 different memory hierarchys, memory hierarchy is higher, the hard disk access performance of configuration is better, and capacity is just less, and price is also more expensive.Therefore can only be by a small amount of deposit data on the highest node of memory hierarchy.Generally, only have low volume data to be accessed frequently in all data in cluster.We process these information by the visit information of log file by the information Valuation Modelling, draw a value, and this value larger represents the frequent of this data access, and memory hierarchy should be higher; Client reads take piece as unit file, and system all records each read operation of piece, and the content of record comprises: calling party, access time and data block information etc., often read primary system and will generate a record.in particular moment, use information Valuation Modelling is processed these records, the processing of model is to liking piece, the parameter of using has: the access time, access times, number of users, block size, the degree of association of piece and other pieces, the history value of piece etc., utilize formula to calculate specific value, weigh " heat " degree of piece, and form from high to low formation according to being worth, piece value formation after the rough handling of information Valuation Modelling, the Data Migration algorithm utilizes formation filtering model, the route matching model, form concrete migration task, utilize at last migration to control model and complete final Data Migration, formation filtering model filters out by the threshold value on each memory hierarchy the data block that need not to move.What these threshold values recorded is the minimum value of moving the maximal value of data block under all and moving data block on all.All pieces in the formation that forms after filtering have all been determined migratory direction, in other embodiments, and when the storage automatic classification, described memory hierarchy comprises 2 grades at least, the criteria for classifying of memory hierarchy is: memory hierarchy is higher, and access performance is better, and the response time of processing user's request is shorter.
Step S4: data block migration.
Costly data block is moved to the high accumulation layer of memory hierarchy, move to the low accumulation layer of memory hierarchy with being worth low data block, after all pieces have all been determined migratory direction in formation, need to determine the source and target of migration.The migration source preferentially selects remaining space less, load is light, the node of normal mode of operation, if the node space of normal mode of operation is not enough, use the node of energy saver mode to be automatically upgraded to normal mode of operation, the migration target need to have enough spaces to hold the migration piece, preferentially selects the light node of load.Move simultaneously source and the distance of migration target and want enough near, when in formation, all pieces have had concrete migration source and migration target, just formed concrete migration task.Controlling model uses multithreading to carry out in batches these migration tasks, only have 50 threads to be used for migration as every batch, and each node has 5 threads to be used for carrying out the migration task at the most, makes transition process as far as possible little on the impact of node visit performance in cluster.
Step S5: self-adaptation adjustment.
After Data Migration was completed, more the new data block visit information, restarted monitoring, in the present embodiment, in time adjusts the migration cycle according to the trigger condition of migration.Described renewal data block information, the step that restarts monitoring is specially:
The valuation result of storage data block is used during in order to valuation next time;
For deleted data block, delete in the Visitor Logs that system keeps;
Carry out the threshold value of each memory hierarchy upgrades according to the actual conditions of migration;
The awaking monitoring process is waited for the arrival of Data Migration next time.
May there be some node that is in energy saver mode (being positioned on secondary storage and tertiary storage) to become normal mode of operation in transition process, shows that the node remaining space that is in normal mode of operation in this grade storage is not enough.Principle of locality according to data access, there is no the node of Visitor Logs in load is heavy and continuous 2 cycles, be set to energy saver mode, and the node that partly is in energy saver mode is set to normal mode of operation, guarantee that the free space of this grade storage is at more than 10% of this grade storage total volume.
After step S5, return to execution in step S2, the process of data dispatch loops.
Classification storage power-economizing method of the present invention uses the method for classification storage, use the level storage medium in the hadoop cluster, the access focus is fixed in the storage of higher level, with regard to not needing, task is moved like this, only the memory node of low level need be in power save mode and get final product.Guaranteed like this energy-conservation of cluster, can make again the individual servers in cluster can not become the bottleneck of access, killed two birds with one stone.
Be understandable that, for the person of ordinary skill of the art, can make other various corresponding changes and distortion by technical conceive according to the present invention, and all these change and distortion all should belong to the protection domain of claim of the present invention.
Claims (10)
1. a classification storage power-economizing method, is characterized in that, said method comprising the steps of:
The storage automatic classification: cluster starts, and utilizes host name to identify the present memory hierarchy of each main frame, and proportionally knot adjustment that memory hierarchy is low is energy saver mode;
Directed access: chosen distance is near, memory hierarchy is high, the accumulation layer of normal mode of operation is stored and file reading;
Seek dsc data: the visit information of each data block in log file, judgement migration opportunity when migration arrives opportunity, according to described recorded information, draw the value of each visit data piece, form from high to low formation according to being worth;
Data block migration: costly data block is moved to the high accumulation layer of memory hierarchy, move to the low accumulation layer of memory hierarchy with being worth low data block.
2. classification storage power-economizing method according to claim 1, is characterized in that, described method also comprises: the self-adaptation adjustment: after Data Migration was completed, more the new data block visit information, restarted monitoring.
3. power-economizing method is stored in classification according to claim 1, it is characterized in that: process described recorded information by the information Valuation Modelling, described data block visit information comprises calling party, access time and data block information.
4. power-economizing method is stored in classification according to claim 3, it is characterized in that: by formation filtering model and route matching model, on the basis of the data block value formation that obtains after the information Valuation Modelling is processed, form concrete Data Migration task, utilize migration to control model and complete Data Migration.
5. power-economizing method is stored in classification according to claim 4, it is characterized in that: described formation filtering model is: fall the not data sectional of needs migration according to threshold filtering, all data sectionals in the formation that forms after filtering have all been determined migratory direction, and threshold value has reflected previous migration results on this memory hierarchy.
6. power-economizing method is stored in classification according to claim 4, it is characterized in that: described route matching model is: after all pieces have all been determined migratory direction in formation, determine migration source and the migration target of close together, the migration source selects preferentially that remaining space is less, load is light, the node of normal mode of operation, and the migration target priority is selected the light node of load.
7. power-economizing method is stored in classification according to claim 4, it is characterized in that: described migration is controlled model and is: carry out migration rate and control, use multithreading to carry out in batches described Data Migration task, reduce transition process to the impact of node visit performance in cluster.
8. power-economizing method is stored in classification according to claim 2, it is characterized in that: described renewal data block information, and the step that restarts monitoring is specially:
The valuation result of storage data block is used during in order to valuation next time;
For deleted data block, delete in the Visitor Logs that system keeps;
Carry out the threshold value of each memory hierarchy upgrades according to the actual conditions of migration;
The awaking monitoring process is waited for the arrival of Data Migration next time.
9. power-economizing method is stored in classification according to claim 1, it is characterized in that: when the storage automatic classification, described memory hierarchy comprises 2 grades at least, and the criteria for classifying of memory hierarchy is: memory hierarchy is higher, access performance is better, and the response time of processing user's request is shorter.
10. power-economizing method is stored in classification according to claim 1, and it is characterized in that: the secondary storage layer with 40% and 60% tertiary storage layer are adjusted into energy saver mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210539442.4A CN103106044B (en) | 2012-12-13 | 2012-12-13 | Classification storage power-economizing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210539442.4A CN103106044B (en) | 2012-12-13 | 2012-12-13 | Classification storage power-economizing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103106044A true CN103106044A (en) | 2013-05-15 |
| CN103106044B CN103106044B (en) | 2016-09-07 |
Family
ID=48313940
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210539442.4A Active CN103106044B (en) | 2012-12-13 | 2012-12-13 | Classification storage power-economizing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103106044B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294167A (en) * | 2013-05-21 | 2013-09-11 | 暨南大学 | Data behavior based low-energy consumption cluster storage replication device and method |
| CN104462389A (en) * | 2014-12-10 | 2015-03-25 | 上海爱数软件有限公司 | Method for implementing distributed file systems on basis of hierarchical storage |
| CN106933859A (en) * | 2015-12-30 | 2017-07-07 | 中国移动通信集团公司 | The moving method and device of a kind of medical data |
| CN106990924A (en) * | 2017-04-07 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of data migration method and system |
| CN107122126A (en) * | 2016-12-22 | 2017-09-01 | 华为技术有限公司 | The moving method of data, device and system |
| CN107741940A (en) * | 2016-11-24 | 2018-02-27 | 腾讯科技（深圳）有限公司 | A kind of data storage method and storage system |
| CN107807796A (en) * | 2017-11-17 | 2018-03-16 | 北京联想超融合科技有限公司 | A kind of data hierarchy method, terminal and system based on super fusion storage system |
| CN108595108A (en) * | 2017-12-29 | 2018-09-28 | 北京奇虎科技有限公司 | A kind of moving method and device of data |
| CN109408401A (en) * | 2017-08-18 | 2019-03-01 | 旺宏电子股份有限公司 | Management system and management method of memory device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101201801A (en) * | 2006-12-11 | 2008-06-18 | 南京理工大学 | Hierarchical storage management method for video on demand system |
| WO2011049315A2 (en) * | 2009-10-19 | 2011-04-28 | Choi Hae-Yong | Energy-saving apparatus using computer user information, and energy-saving method using same |
| CN102158513A (en) * | 2010-02-11 | 2011-08-17 | 联想(北京)有限公司 | Service cluster and energy-saving method and device thereof |
| CN102480469A (en) * | 2010-11-29 | 2012-05-30 | 北京中和威软件有限公司 | Load scheduling method and device based on energy balance in SIP service cluster |
-
2012
- 2012-12-13 CN CN201210539442.4A patent/CN103106044B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101201801A (en) * | 2006-12-11 | 2008-06-18 | 南京理工大学 | Hierarchical storage management method for video on demand system |
| WO2011049315A2 (en) * | 2009-10-19 | 2011-04-28 | Choi Hae-Yong | Energy-saving apparatus using computer user information, and energy-saving method using same |
| CN102158513A (en) * | 2010-02-11 | 2011-08-17 | 联想(北京)有限公司 | Service cluster and energy-saving method and device thereof |
| CN102480469A (en) * | 2010-11-29 | 2012-05-30 | 北京中和威软件有限公司 | Load scheduling method and device based on energy balance in SIP service cluster |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294167B (en) * | 2013-05-21 | 2016-02-10 | 暨南大学 | A kind of low energy consumption cluster-based storage reproducing unit based on data behavior and method |
| CN103294167A (en) * | 2013-05-21 | 2013-09-11 | 暨南大学 | Data behavior based low-energy consumption cluster storage replication device and method |
| CN104462389A (en) * | 2014-12-10 | 2015-03-25 | 上海爱数软件有限公司 | Method for implementing distributed file systems on basis of hierarchical storage |
| CN104462389B (en) * | 2014-12-10 | 2018-01-30 | 上海爱数信息技术股份有限公司 | Distributed file system implementation method based on classification storage |
| CN106933859A (en) * | 2015-12-30 | 2017-07-07 | 中国移动通信集团公司 | The moving method and device of a kind of medical data |
| CN106933859B (en) * | 2015-12-30 | 2020-10-20 | 中国移动通信集团公司 | Method and device for migrating medical data |
| CN107741940A (en) * | 2016-11-24 | 2018-02-27 | 腾讯科技（深圳）有限公司 | A kind of data storage method and storage system |
| CN107741940B (en) * | 2016-11-24 | 2021-03-02 | 腾讯科技（深圳）有限公司 | Data storage method and storage system |
| CN107122126B (en) * | 2016-12-22 | 2020-09-08 | 华为技术有限公司 | Data migration method, device and system |
| CN107122126A (en) * | 2016-12-22 | 2017-09-01 | 华为技术有限公司 | The moving method of data, device and system |
| CN106990924A (en) * | 2017-04-07 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of data migration method and system |
| CN109408401A (en) * | 2017-08-18 | 2019-03-01 | 旺宏电子股份有限公司 | Management system and management method of memory device |
| CN109408401B (en) * | 2017-08-18 | 2023-03-24 | 旺宏电子股份有限公司 | Management system and management method of memory device |
| CN107807796A (en) * | 2017-11-17 | 2018-03-16 | 北京联想超融合科技有限公司 | A kind of data hierarchy method, terminal and system based on super fusion storage system |
| CN107807796B (en) * | 2017-11-17 | 2021-03-05 | 北京联想超融合科技有限公司 | Data layering method, terminal and system based on super-fusion storage system |
| CN108595108A (en) * | 2017-12-29 | 2018-09-28 | 北京奇虎科技有限公司 | A kind of moving method and device of data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103106044B (en) | 2016-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103106044A (en) | Classification storage energy-saving method | |
| CN103150263B (en) | Classification storage means | |
| CN103106152B (en) | Based on the data dispatching method of level storage medium | |
| CN111190688B (en) | A kind of Docker migration method and system for cloud data center | |
| CN100476742C (en) | Load Balancing Method Based on Object Storage Device | |
| CN105159751B (en) | The virtual machine migration method of energy efficient in a kind of cloud data center | |
| EP2122502B1 (en) | Automated and dynamic management of query views for database workloads | |
| CN102521154B (en) | For creating the dynamic memory allocation of low power section and reorientating | |
| JP5600277B2 (en) | Cloud service relocation system, method and program | |
| CN102868763A (en) | Energy-saving dynamic adjustment method of virtual web application cluster in cloud computing environment | |
| KR20150043102A (en) | Apparatus and method for managing data in hybrid memory | |
| Herodotou et al. | Automating distributed tiered storage management in cluster computing | |
| CN106095531B (en) | A virtual machine scheduling method based on grade and physical machine load in cloud platform | |
| CN103095805A (en) | Cloud storage system of data intelligent and decentralized management | |
| Rajabzadeh et al. | Energy-aware framework with Markov chain-based parallel simulated annealing algorithm for dynamic management of virtual machines in cloud data centers | |
| CN102156665A (en) | Differential serving method for virtual system competition resources | |
| CN110321331A (en) | The object storage system of storage address is determined using multistage hash function | |
| CN103390061A (en) | Multi-level cache-based sea spatio-temporal data accessing method taking user as center | |
| CN112269632A (en) | Scheduling method and system for optimizing cloud data center | |
| CN107315540A (en) | A kind of AUTOMATIC ZONING storage method and system | |
| WO2024187900A1 (en) | Data storage method, system and device for distributed storage system, and storage medium | |
| CN103150245A (en) | Method for determining visiting characteristic of data entityand store controller | |
| CN102662859A (en) | Data cache system based on service grade and method thereof | |
| CN114629906B (en) | Reliable cloud container cluster resource scheduling method and device based on deep reinforcement learning | |
| Séguéla et al. | Dynamic energy and expenditure aware data replication strategy |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |