CN103064902A - Method and device for storing and reading data in hadoop distributed file system (HDFS) - Google Patents
Method and device for storing and reading data in hadoop distributed file system (HDFS) Download PDFInfo
- Publication number
- CN103064902A CN103064902A CN2012105502202A CN201210550220A CN103064902A CN 103064902 A CN103064902 A CN 103064902A CN 2012105502202 A CN2012105502202 A CN 2012105502202A CN 201210550220 A CN201210550220 A CN 201210550220A CN 103064902 A CN103064902 A CN 103064902A
- Authority
- CN
- China
- Prior art keywords
- cold
- data
- hot
- check code
- data blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and device for storing and reading data in a hadoop distributed file system (HDFS). The storing method includes: obtaining a cold and hot value of a file data block in the HDFS according to a data cold hot distinguishing mechanism, and comparing the cold and hot value and a set distinguishing threshold; if the cold and hot value is not larger than the distinguishing threshold, the file data block is cold data; dividing the file data block of the cold data into n blocks, and calculating m verifying coding blocks corresponding to the n data blocks, wherein m and n are positive integral numbers, and m<n; and storing the n data blocks and the m verifying coding blocks. The storing method and the storing system can achieve distinguishing of the cold data and hot data, stores the cold data in distinguishing mode, and reduces storing cost of the cold data in the HDFS. The reading method and the read system can achieve distinguishing of the cold data and the hot data, read and recover the cold data in distinguishing mode, and reduce time and cost for reading and recovering the cold data.
Description
Technical field
The present invention relates to the data storage and read the field, especially, relate to data storage and read method and storage and reading system among a kind of HDFS.
Background technology
Hadoop is by the leading cluster distributed project of Apache fund, mainly comprises two nucleus modules: Map/Reduce programming mode and HDFS(Hadoop distributed File System) distributed file system.Wherein, mainly the many back mechanisms (being generally three parts) by file data blocks, heartbeat mechanism etc. are realized high availability, the cluster extendability of data, the characteristics such as high-speed read-write of data to HDFS.Because the above-mentioned characteristic of HDFS, at present, the esbablished corporation of storage reaches nearly thousand take HDFS as the fundamental construction cloud.
In the storage and read-write of dsc data, there is not any problem in the memory mechanism of HDFS, but in cold data storage and read-write, because this memory mechanism reckons without the differentiation storage to hot and cold data, therefore, can there be the higher problem of financial cost owing to cold data storage takies too many hard-disk capacity.For example, the cost of supposing hard disk is 1 yuan/G, its cost of memory mechanism according to HDFS just needs 3 yuan/G, the data of storage 1TB just need 3*1024=3072 unit, the data of storage 1PB just need 3*1024*1024=3145728 unit, that is to say, the storage of 1 part of storage backup piece of every reduction PB level just will save more than 100 ten thousand.
Hence one can see that, need the urgent technical matters that solves of those skilled in the art to be exactly: how to solve detection, differentiation and the difference storage of cold and hot data among the HDFS of prior art and read cold data, to reduce among the HDFS cold data storage and to read the high problem of cost.
Summary of the invention
Technical matters to be solved by this invention provide in HDFS, can distinguish cold and hot data and can to cold data distinguish storage method and system and cold data are distinguished the method and system that reads.
The invention provides data storage method among a kind of HDFS, it comprises the steps: to obtain by the cold and hot determine mechanism of data the cold and hot value of the file data blocks among the HDFS, and the discrimination threshold of described cold and hot value and setting is compared; If described cold and hot value is not more than described discrimination threshold, then described file data blocks is cold data; The file data blocks of described cold data is divided into the n piece, and calculates m check code piece corresponding to a described n data block, wherein, m, n are positive integer, and m＜n; A described n data block and m check code piece are stored.
Preferably, described cold and hot determine mechanism comprises by the cold and hot discriminant parameter value of each file data blocks of testing mechanism acquisition sets up the cold and hot chained list of data, goes out the cold and hot value of each file data blocks in the described cold and hot chained list according to described parameter value calculation; Described cold and hot determine mechanism also can comprise by reading the configuration file that comprises the cold and hot discriminant parameter of data in each file data blocks, to obtain the cold and hot discriminant parameter value of each file data blocks, go out the cold and hot value of described each file data blocks according to described parameter value calculation.
Preferably, described check code piece is RS check code piece, and described RS check code piece calculates by model moral gate matrix.
Preferably, the file backup piece of described cold data is one.
The present invention also provides a kind of read method corresponding with data storage method among the HDFS, it comprises the steps: to obtain by the cold and hot determine mechanism of data the cold and hot value of the file data blocks among the HDFS, and the discrimination threshold of described cold and hot value and setting is compared; If described cold and hot value is not more than described discrimination threshold, then described file data blocks is cold data; N file data blocks in the described cold data read; Calculate m check code piece corresponding to a described n data block, wherein, m, n are positive integer, and m＜n; If read failure, then choose the individual file data blocks of m ' according to the check code principle and carry out code restoration; Wherein, m ' [m, n].
Preferably, described check code piece is RS check code piece, and described RS check code piece calculates by model moral gate matrix.
The present invention also provides the corresponding storage system of data storage method among the HDFS a kind of and of the present invention, it comprises: comparing unit, be used for obtaining by the cold and hot determine mechanism of data the cold and hot value of the file data blocks of HDFS, and the discrimination threshold of described cold and hot value and setting is compared; Judgement unit is used for when described cold and hot value is not more than described discrimination threshold, described file data blocks is differentiated be cold data; Cold data division unit is used for the file data blocks of described cold data is divided into the n piece, and calculates m check code piece corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n; Cold data storage cell is used for a described n data block and m check code piece are stored.
Preferably, described comparing unit comprises detection sub-unit, sets up the cold and hot chained list of data for the cold and hot discriminant parameter value of each file data blocks that obtains by testing mechanism, goes out the cold and hot value of each file data blocks in the described cold and hot chained list according to described parameter value calculation; Described comparing unit also can comprise the parameter acquiring subelement, be used for reading the configuration file that comprises cold and hot discriminant parameter of each file data blocks, to obtain the cold and hot discriminant parameter value of each file data blocks, go out the cold and hot value of described each file data blocks according to described parameter value calculation.
Preferably, described check code piece is RS check code piece, and described RS check code piece calculates by model moral gate matrix.
Preferably, the file backup piece of described cold data is one.
The present invention also provides the corresponding reading system of data storage system among a kind of and above-mentioned HDFS simultaneously, and it comprises; Comparing unit is used for obtaining by the cold and hot determine mechanism of data the cold and hot value of the file data blocks of HDFS, and the discrimination threshold of described cold and hot value and setting is compared; Judgement unit is used for when described cold and hot value is not more than described discrimination threshold, described file data blocks is differentiated be cold data; Cold data-reading unit is used for n file data blocks of described cold data read; The check code computing unit is used for calculating m check code piece corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n; Data recovery unit is used for reading failure when cold data, then chooses the individual file data blocks of m ' according to the check code principle and carries out code restoration; Wherein, m ' [m, n].
Preferably, described check code piece is RS check code piece, and described RS check code piece calculates by model moral gate matrix.
Data storage method and storage system among the HDFS provided by the invention, realization is distinguished cold and hot data, and cold data are distinguished storage, can obviously reduce the high problem of cold data carrying cost among the existing HDFS, especially for the above data storage of PB level, can greatly reduce cold data storage cost among the HDFS; Read method and the reading system of data among the while HDFS provided by the invention, can realize cold and hot data are distinguished, and cold data are distinguished read and recover, thereby can reduce the time that cold data read and recover, simultaneously, carry out data by cryptoprinciple especially RS cryptoprinciple and recover, can obviously reduce the quantity of losing of file data blocks.
Description of drawings
Fig. 1 is the process flow diagram of data storage embodiment of the method among the HDFS of the present invention;
Fig. 2 is the process flow diagram of the read method embodiment of data among the HDFS of the present invention;
Fig. 3 is the schematic diagram of data storage system embodiment among the HDFS of the present invention;
Fig. 4 is the schematic diagram of the reading system embodiment of data among the HDFS of the present invention.
Embodiment
For above-mentioned purpose of the present invention, feature and advantage can be become apparent more, the present invention is further detailed explanation below in conjunction with the drawings and specific embodiments.
With reference to Fig. 1, show the process flow diagram of data storage embodiment of the method among the HDFS of the present invention, comprise: step S11, obtain the cold and hot value of the file data blocks among the HDFS, and the discrimination threshold of described cold and hot value and setting compared by the cold and hot determine mechanism of data; Step S12, if described cold and hot value is not more than described discrimination threshold, then described file data blocks is cold data; Step S13 is divided into the n piece with the file data blocks of described cold data, and calculates m check code piece corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n; Step S14 stores a described n data block and m check code piece.
By data storage method among the HDFS of the present invention, can realize cold and hot data are distinguished, and cold data are distinguished storage, can obviously reduce the high problem of cold data carrying cost among the existing HDFS, especially for the above data storage of PB level, can greatly reduce cold data storage cost among the HDFS.
Wherein, in step S11, according to system and user's request, can the different cold and hot determine mechanism of data of free choice for use.Preferably, the cold and hot determine mechanism of described data can be to set up the cold and hot chained list of data by the cold and hot discriminant parameter value of each file data blocks of testing mechanism acquisition, described cold and hot discriminant parameter value comprises data storage and reading frequency value, data storage and reads the time equivalence, goes out the cold and hot value of each file data blocks in the described cold and hot chained list according to described parameter value calculation; The cold and hot determine mechanism of described data also can be by reading the configuration file that comprises the cold and hot discriminant parameter of data in each file data blocks, HCFile.properties for example, to obtain the cold and hot discriminant parameter value of each file data blocks, go out the cold and hot value of described each file data blocks according to described parameter value calculation.
Wherein, described check code capable of choosing multiple mode, such as RS coding (Reed-Solomon, in coded), the LT code, Raptor code etc., but the RS of choice for use coding in the preferred embodiment of the present invention, although it need to set up long RS coding, its overall efficiency is better than other coded systems.Preferably, the RS encoding block calculates by model moral gate matrix.
Especially, the file backup piece of described cold data is set to one, increase simultaneously certain RS check code piece, wherein the quantity m of check code piece is less than the quantity n of file data blocks, therefore, the backup factor of described cold data is 1+m/n, less than the backup factor 3 of giving tacit consent among the HDFS, thereby can reduce the umber of cold data backup, and then reduce cold data carrying cost.
Referring to Fig. 2, show the process flow diagram of the method for reading data embodiment corresponding with data storage method among the above-mentioned HDFS, it comprises: step S21, obtain the cold and hot value of the file data blocks among the HDFS by the cold and hot determine mechanism of data, and the discrimination threshold of described cold and hot value and setting is compared; Step S22, if described cold and hot value is not more than described discrimination threshold, then described file data blocks is cold data; Step S23 merges n file data blocks in the described cold data and to read; Step S24 calculates m check code piece corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n; Step S25 if read failure, then chooses the individual file data blocks of m ' according to the check code principle and carries out code restoration; Wherein, m ' [m, n].
Wherein, described check code capable of choosing multiple mode, such as RS coding (Reed-Solomon, in coded), the LT code, Raptor code etc., but the RS of choice for use coding in the preferred embodiment of the present invention, although it need to set up long RS coding, its overall efficiency is better than other coded systems.Preferably, the RS encoding block calculates by model moral gate matrix.
Wherein, be m ' piece and m ' [m, n] owing to carrying out the file data blocks of RS code restoration, hence one can see that, and file data blocks allows at most to lose the m piece.
Read method by data among the HDFS of the present invention, can realize cold and hot data are distinguished, and cold data are distinguished read and recover, carry out data by cryptoprinciple especially RS cryptoprinciple and recover, can obviously reduce the quantity of losing of file data blocks.
Referring to Fig. 3, show the schematic diagram of the data-storage system embodiment corresponding with data storage method among the above-mentioned HDFS, it comprises: comparing unit 31, be used for obtaining by the cold and hot determine mechanism of data the cold and hot value of the file data blocks of HDFS, and the discrimination threshold of described cold and hot value and setting is compared; Judgement unit 32 is used for when described cold and hot value is not more than described discrimination threshold, described file data blocks is differentiated be cold data; Cold data division unit 33 is divided into the n piece with the file data blocks of described cold data, and calculates m verification code block corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n; Cold data storage cell 34 is used for a described n data block and m check code piece are stored.
Wherein, described check code capable of choosing multiple mode, such as RS coding (Reed-Solomon, in coded), the LT code, Raptor code etc., but the RS of choice for use coding in the preferred embodiment of the present invention, although it need to set up long RS coding, its overall efficiency is better than other coded systems.Preferably, the RS encoding block calculates by model moral gate matrix.
By data storage system among the HDFS of the present invention, can realize cold and hot data are distinguished, and cold data are distinguished storage, can obviously reduce the high problem of cold data carrying cost among the existing HDFS, especially for the above data storage of PB level, can greatly reduce cold data storage cost among the HDFS.
Wherein, according to system and user's request, but the comparison module that option and installment is different, preferably, described comparing unit 31 can comprise the detection sub-unit (not shown), cold and hot discriminant parameter value for each file data blocks that obtains by testing mechanism is set up the cold and hot chained list of data, and described cold and hot discriminant parameter value comprises data storage and reading frequency value, data storage and reads the time equivalence, goes out the cold and hot value of each file data blocks in the described cold and hot chained list according to described parameter value calculation; Described comparing unit also can comprise the parameter acquiring subelement, be used for reading the configuration file that comprises cold and hot discriminant parameter of each file data blocks, HCFile.properties for example, to obtain the cold and hot discriminant parameter value of each file data blocks, go out the cold and hot value of described each file data blocks according to described parameter value calculation.
Preferably, the file backup piece of described cold data is set to one, increase simultaneously certain RS check code piece, wherein the quantity m of check code piece is less than the quantity n of file data blocks, therefore, the backup factor of described cold data is 1+m/n, less than the backup factor 3 of giving tacit consent among the HDFS, thereby can reduce the umber of cold data backup, and then reduce cold data carrying cost.
Referring to Fig. 4, show with above-mentioned HDFS in data the schematic diagram of the corresponding data reading system embodiment of read method, comprise: comparing unit 41, be used for obtaining by the cold and hot determine mechanism of data the cold and hot value of the file data blocks of HDFS, and the discrimination threshold of described cold and hot value and setting is compared; Judgement unit 42 is used for when described cold and hot value is not more than described discrimination threshold, described file data blocks is differentiated be cold data; Cold data-reading unit 43 is used for n file data blocks of described cold data read; Check code computing unit 44 is used for calculating m check code piece corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n; Data recovery unit 45 is used and is read failure when cold data, then chooses the individual file data blocks of m ' according to the check code principle and carries out code restoration; Wherein, m ' [m, n].
Wherein, described check code capable of choosing multiple mode, such as RS coding (Reed-Solomon, in coded), the LT code, Raptor code etc., but the RS of choice for use coding in the preferred embodiment of the present invention, although it need to set up long RS coding, its overall efficiency is better than other coded systems.Preferably, the RS encoding block calculates by model moral gate matrix.
Wherein, be m ' piece and m ' [m, n] owing to carrying out the file data blocks of RS code restoration, hence one can see that, and file data blocks allows at most to lose the m piece.
Reading system by data among the HDFS of the present invention, can realize cold and hot data are distinguished, and cold data are distinguished read and recover, thereby can reduce the time that cold data read and recover, simultaneously, carry out data by cryptoprinciple especially RS cryptoprinciple and recover, can obviously reduce the quantity of losing of file data blocks.
Each embodiment in this instructions all adopts the mode of going forward one by one to describe, and what each embodiment stressed is and the difference of other embodiment that identical similar part is mutually referring to getting final product between each embodiment.For system embodiment because itself and embodiment of the method basic simlarity, so describe fairly simple, relevant part gets final product referring to the part explanation of embodiment of the method.
Above to data storage method and system among the HDFS provided by the present invention, reaching read method and system is described in detail, used specific case herein principle of the present invention and embodiment are set forth, the explanation of above embodiment just is used for helping to understand method of the present invention and core concept thereof; Simultaneously, for one of ordinary skill in the art, according to thought of the present invention, all will change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention.
Claims (18)
1. data storage method among the HDFS is characterized in that, comprising:
Obtain the cold and hot value of the file data blocks among the HDFS by the cold and hot determine mechanism of data, and the discrimination threshold of described cold and hot value and setting is compared;
If described cold and hot value is not more than described discrimination threshold, then described file data blocks is cold data;
The file data blocks of described cold data is divided into the n piece, and calculates m check code piece corresponding to a described n data block, wherein, m, n are positive integer, and m＜n;
A described n data block and m check code piece are stored.
2. the method for claim 1, it is characterized in that, described cold and hot determine mechanism comprises by the cold and hot discriminant parameter value of each file data blocks of testing mechanism acquisition sets up the cold and hot chained list of data, goes out the cold and hot value of each file data blocks in the described cold and hot chained list according to described parameter value calculation.
3. the method for claim 1, it is characterized in that, described cold and hot determine mechanism comprises by reading the configuration file that comprises the cold and hot discriminant parameter of data in each file data blocks, to obtain the cold and hot discriminant parameter value of each file data blocks, go out the cold and hot value of described each file data blocks according to described parameter value calculation.
4. the method for claim 1 is characterized in that, described check code piece is RS check code piece.
5. method as claimed in claim 4 is characterized in that, described RS check code piece calculates by model moral gate matrix.
6. such as each described method in the claim 1 to 5, it is characterized in that, the file backup piece of described cold data is one.
7. the read method of data among the HDFS is characterized in that, comprising:
Obtain the cold and hot value of the file data blocks among the HDFS by the cold and hot determine mechanism of data, and the discrimination threshold of described cold and hot value and setting is compared;
If described cold and hot value is not more than described discrimination threshold, then described file data blocks is cold data;
N file data blocks in the described cold data read;
Calculate m check code piece corresponding to a described n data block, wherein, m, n are positive integer, and m＜n;
If read failure, then choose the individual file data blocks of m ' according to the check code principle and carry out code restoration; Wherein, m ' [m, n].
8. method as claimed in claim 7 is characterized in that, described check code piece is RS check code piece.
9. method as claimed in claim 8 is characterized in that, described RS check code piece calculates by model moral gate matrix.
10. data storage system among the HDFS is characterized in that, comprising:
Comparing unit is used for the cold and hot value by the file data blocks of the cold and hot determine mechanism acquisition of data HDFS, and the discrimination threshold of described cold and hot value and setting is compared;
Judgement unit is used for when described cold and hot value is not more than described discrimination threshold, described file data blocks is differentiated be cold data;
Cold data division unit is used for the file data blocks of described cold data is divided into the n piece, and calculates m check code piece corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n;
Cold data storage cell is used for a described n data block and m check code piece are stored.
11. system as claimed in claim 10, it is characterized in that, described comparing unit comprises detection sub-unit, cold and hot discriminant parameter value for each file data blocks that obtains by testing mechanism is set up the cold and hot chained list of data, goes out the cold and hot value of each file data blocks in the described cold and hot chained list according to described parameter value calculation.
12. system as claimed in claim 10, it is characterized in that, described comparing unit comprises the parameter acquiring subelement, be used for reading the configuration file that comprises cold and hot discriminant parameter of each file data blocks, to obtain the cold and hot discriminant parameter value of each file data blocks, go out the cold and hot value of described each file data blocks according to described parameter value calculation.
13. system as claimed in claim 10 is characterized in that, described check code piece is RS check code piece.
14. system as claimed in claim 13 is characterized in that, described RS check code piece calculates by model moral gate matrix.
15. such as each described system in the claim 10 to 14, it is characterized in that, the file backup piece of described cold data is one.
16. the reading system of data is characterized in that among the HDFS, comprises;
Comparing unit is used for obtaining by the cold and hot determine mechanism of data the cold and hot value of the file data blocks of HDFS, and the discrimination threshold of described cold and hot value and setting is compared;
Judgement unit is used for when described cold and hot value is not more than described discrimination threshold, described file data blocks is differentiated be cold data;
Cold data-reading unit is used for n file data blocks of described cold data read;
The check code computing unit is used for calculating m check code piece corresponding to a described n data block, and wherein, m, n are positive integer, and m＜n;
Data recovery unit is used for reading failure when cold data, then chooses the individual file data blocks of m ' according to the check code principle and carries out code restoration; Wherein, m ' [m, n].
17. system as claimed in claim 16 is characterized in that, described check code piece is RS check code piece.
18. system as claimed in claim 17 is characterized in that, described RS check code piece calculates by model moral gate matrix.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012105502202A CN103064902A (en) | 2012-12-18 | 2012-12-18 | Method and device for storing and reading data in hadoop distributed file system (HDFS) |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012105502202A CN103064902A (en) | 2012-12-18 | 2012-12-18 | Method and device for storing and reading data in hadoop distributed file system (HDFS) |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103064902A true CN103064902A (en) | 2013-04-24 |
Family
ID=48107532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012105502202A Pending CN103064902A (en) | 2012-12-18 | 2012-12-18 | Method and device for storing and reading data in hadoop distributed file system (HDFS) |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103064902A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103259856A (en) * | 2013-04-27 | 2013-08-21 | 华为技术有限公司 | Method, device and system for data processing |
| CN104573119A (en) * | 2015-02-05 | 2015-04-29 | 重庆大学 | Energy-saving-oriented Hadoop distributed file system storage policy in cloud computing |
| CN104917788A (en) * | 2014-03-11 | 2015-09-16 | 中国移动通信集团公司 | Data storage method and apparatus |
| CN104935481A (en) * | 2015-06-24 | 2015-09-23 | 华中科技大学 | A Data Recovery Method Based on Redundancy Mechanism in Distributed Storage |
| CN105138422A (en) * | 2015-08-10 | 2015-12-09 | 联想(北京)有限公司 | Control method and electronic device |
| CN105956128A (en) * | 2016-05-09 | 2016-09-21 | 南京大学 | Self-adaptive encoding storage fault-tolerant method based on simple regenerating code |
| CN106528608A (en) * | 2016-09-27 | 2017-03-22 | 中国电力科学研究院 | Cold and hot storage method and system for power grid GIS (Geographic Information System) data in cloud architecture |
| CN106649406A (en) * | 2015-11-04 | 2017-05-10 | 华为技术有限公司 | Method and device for storing file in self-adaption mode |
| CN107153661A (en) * | 2016-03-04 | 2017-09-12 | 中国移动通信集团山东有限公司 | A kind of storage, read method and its device of the data based on HDFS systems |
| CN107665224A (en) * | 2016-07-29 | 2018-02-06 | 北京京东尚科信息技术有限公司 | Scan the mthods, systems and devices of HDFS cold datas |
| CN107870726A (en) * | 2016-09-26 | 2018-04-03 | 北京优朋普乐科技有限公司 | Date storage method and device, server based on redundancy scheme |
| CN111930555A (en) * | 2020-09-02 | 2020-11-13 | 平安国际智慧城市科技股份有限公司 | Erasure code based file processing method and device and computer equipment |
| CN111966293A (en) * | 2020-08-18 | 2020-11-20 | 北京明略昭辉科技有限公司 | Cold and hot data analysis method and system |
| CN112948171A (en) * | 2019-12-11 | 2021-06-11 | 华为技术有限公司 | Data processing method and device, terminal equipment and computer readable storage medium |
| CN113360553A (en) * | 2020-03-03 | 2021-09-07 | 中国移动通信集团贵州有限公司 | Data cold and hot degree evaluation method and server |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411542A (en) * | 2011-10-26 | 2012-04-11 | 上海文广互动电视有限公司 | Dynamic hierarchical storage system and method |
| US20120182891A1 (en) * | 2011-01-19 | 2012-07-19 | Youngseok Lee | Packet analysis system and method using hadoop based parallel computation |
-
2012
- 2012-12-18 CN CN2012105502202A patent/CN103064902A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120182891A1 (en) * | 2011-01-19 | 2012-07-19 | Youngseok Lee | Packet analysis system and method using hadoop based parallel computation |
| CN102411542A (en) * | 2011-10-26 | 2012-04-11 | 上海文广互动电视有限公司 | Dynamic hierarchical storage system and method |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103259856A (en) * | 2013-04-27 | 2013-08-21 | 华为技术有限公司 | Method, device and system for data processing |
| CN104917788A (en) * | 2014-03-11 | 2015-09-16 | 中国移动通信集团公司 | Data storage method and apparatus |
| CN104573119A (en) * | 2015-02-05 | 2015-04-29 | 重庆大学 | Energy-saving-oriented Hadoop distributed file system storage policy in cloud computing |
| CN104573119B (en) * | 2015-02-05 | 2017-10-27 | 重庆大学 | Towards the Hadoop distributed file system storage methods of energy-conservation in cloud computing |
| CN104935481A (en) * | 2015-06-24 | 2015-09-23 | 华中科技大学 | A Data Recovery Method Based on Redundancy Mechanism in Distributed Storage |
| CN104935481B (en) * | 2015-06-24 | 2018-03-09 | 华中科技大学 | Data reconstruction method based on redundancy scheme under a kind of distributed storage |
| CN105138422B (en) * | 2015-08-10 | 2018-09-21 | 北京联想核芯科技有限公司 | Control method and electronic equipment |
| CN105138422A (en) * | 2015-08-10 | 2015-12-09 | 联想(北京)有限公司 | Control method and electronic device |
| US11182080B2 (en) | 2015-11-04 | 2021-11-23 | Huawei Technologies Co., Ltd. | Adaptive file storage method and apparatus |
| CN106649406A (en) * | 2015-11-04 | 2017-05-10 | 华为技术有限公司 | Method and device for storing file in self-adaption mode |
| CN106649406B (en) * | 2015-11-04 | 2020-04-28 | 华为技术有限公司 | A method and device for adaptively storing files |
| CN107153661A (en) * | 2016-03-04 | 2017-09-12 | 中国移动通信集团山东有限公司 | A kind of storage, read method and its device of the data based on HDFS systems |
| CN105956128A (en) * | 2016-05-09 | 2016-09-21 | 南京大学 | Self-adaptive encoding storage fault-tolerant method based on simple regenerating code |
| CN105956128B (en) * | 2016-05-09 | 2019-09-17 | 南京大学 | A kind of adaptive coding storage fault-tolerance approach based on simple regeneration code |
| CN107665224A (en) * | 2016-07-29 | 2018-02-06 | 北京京东尚科信息技术有限公司 | Scan the mthods, systems and devices of HDFS cold datas |
| CN107665224B (en) * | 2016-07-29 | 2021-04-30 | 北京京东尚科信息技术有限公司 | Method, system and device for scanning HDFS cold data |
| CN107870726A (en) * | 2016-09-26 | 2018-04-03 | 北京优朋普乐科技有限公司 | Date storage method and device, server based on redundancy scheme |
| CN106528608A (en) * | 2016-09-27 | 2017-03-22 | 中国电力科学研究院 | Cold and hot storage method and system for power grid GIS (Geographic Information System) data in cloud architecture |
| CN106528608B (en) * | 2016-09-27 | 2022-07-01 | 中国电力科学研究院 | Power grid GIS data cold and hot storage method and system under cloud architecture |
| CN112948171A (en) * | 2019-12-11 | 2021-06-11 | 华为技术有限公司 | Data processing method and device, terminal equipment and computer readable storage medium |
| CN113360553A (en) * | 2020-03-03 | 2021-09-07 | 中国移动通信集团贵州有限公司 | Data cold and hot degree evaluation method and server |
| CN111966293A (en) * | 2020-08-18 | 2020-11-20 | 北京明略昭辉科技有限公司 | Cold and hot data analysis method and system |
| CN111930555A (en) * | 2020-09-02 | 2020-11-13 | 平安国际智慧城市科技股份有限公司 | Erasure code based file processing method and device and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103064902A (en) | Method and device for storing and reading data in hadoop distributed file system (HDFS) | |
| CN102279776B (en) | Error checking and correcting ability testing method and device | |
| KR101580454B1 (en) | Selecting between non-volatile memory units having different minimum addressable data unit sizes | |
| CN109542342A (en) | Metadata management and data reconstruction method, equipment and storage medium | |
| CN102841931A (en) | Storage method and storage device of distributive-type file system | |
| US8762771B2 (en) | Method for completing write operations to a RAID drive pool with an abnormally slow drive in a timely fashion | |
| CN105138290A (en) | High-performance storage pool organization method and device | |
| CN106033320A (en) | A high-performance redundant array of inexpensive disc (RAID) mechanism processing method and device | |
| CN101937721A (en) | Method for testing memory device | |
| CN103019893A (en) | Multi-disk fault-tolerant two-dimensional hybrid disk RAID4 system architecture and read-write method thereof | |
| CN103700396A (en) | SRAM (static random access memory)-oriented anti-SEU (single-event upset) error accumulation controller and method | |
| US8037332B2 (en) | Quad-state power-saving virtual storage controller | |
| CN104699577A (en) | Counter to locate faulty die in a distributed codeword storage system | |
| CN102981783A (en) | Cache accelerating method based on Nand Flash | |
| CN103729149A (en) | Data storage method | |
| CN103577115A (en) | Data configuration processing method and data configuration processing device | |
| CN102591746B (en) | Data reconstruction method and storage equipment | |
| CN105094700B (en) | The data temperature of bedding storage calculates method and apparatus in cloud storage system | |
| CN102270102B (en) | Method for optimizing writing performance of RAID6 (Redundant Array of Independent Disks) disk array | |
| CN109375868B (en) | Data storage method, scheduling device, system, equipment and storage medium | |
| Rosenthal | The medium-term prospects for long-term storage systems | |
| CN102033717B (en) | Data storage method and system based on RAID (Redundant Array of Inexpensive Disk) | |
| US20160292035A1 (en) | Indicating rebuild state of storage devices | |
| CN104375906B (en) | A kind of extensive Backup Data fast calibration method based on file system | |
| CN103399783A (en) | Storage method and device of mirror image documents of virtual machines |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20130424 |
|
| RJ01 | Rejection of invention patent application after publication |