[go: up one dir, main page]

CN107193494B - A RDD Persistence Method Based on SSD and HDD Hybrid Storage System - Google Patents

A RDD Persistence Method Based on SSD and HDD Hybrid Storage System Download PDF

Info

Publication number
CN107193494B
CN107193494B CN201710358093.9A CN201710358093A CN107193494B CN 107193494 B CN107193494 B CN 107193494B CN 201710358093 A CN201710358093 A CN 201710358093A CN 107193494 B CN107193494 B CN 107193494B
Authority
CN
China
Prior art keywords
rdd
data
module
preset
block manager
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710358093.9A
Other languages
Chinese (zh)
Other versions
CN107193494A (en
Inventor
陆克中
黄泽成
毛睿
廖好
朱金彬
隋秀峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Baode Network Security System Shenzhen Co ltd
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to CN201710358093.9A priority Critical patent/CN107193494B/en
Publication of CN107193494A publication Critical patent/CN107193494A/en
Application granted granted Critical
Publication of CN107193494B publication Critical patent/CN107193494B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3034—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a storage system, e.g. DASD based or network based
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
    • G06F11/324—Display of status information
    • G06F11/325—Display of status information by lamps or LED's
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/068—Hybrid storage device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides an RDD (remote data description) persistence method based on a SSD (solid State disk) and HDD (hard disk drive) hybrid storage system, which comprises the following steps: the RDD module transmits the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager; the disk block manager transmits the preset persistence level to a device adapter; the equipment adapter receives a preset persistence level of data and reads two directory management variables in a configuration file, matches the preset persistence level with a temporary file directory in a corresponding directory management variable according to the preset persistence level of the data, and returns the temporary file directory obtained by matching to the disk block manager; the disk block manager obtains a file name according to the block identifier, obtains a data storage address according to the temporary file directory and the file name obtained by matching, and returns the data storage address to the block manager; and the block manager stores the data in the RDD module in the SSD or the HDD according to the data storage address.

Description

RDD (remote data description) persistence method based on SSD (solid State disk) and HDD (hard disk drive) hybrid storage system
Technical Field
The invention relates to the technical field of data processing, in particular to an RDD (remote data description) persistence method based on a SSD (solid state drive) and HDD (hard disk drive) hybrid storage system.
Background
In the existing big data era, in the face of massive data, how to manage, analyze and extract valuable information in an effective time becomes a problem which people need to solve urgently. However, big data, whether it be of scale, variety, or structure, presents a significant challenge to people's ability to host data.
Spark is a big data computing framework which is currently efficient and widely used in the industry, and is a universal and fast large-scale data processing engine. Firstly, Spark provides a uniform solution, and can be used for complex tasks such as interactive query, real-time stream processing, machine learning and the like; secondly, the Spark divides phases and tasks through an elastic distributed data set (RDD), optimizes the execution sequence of subtasks through a high-efficiency Directed Acyclic Graph (DAG) execution engine, and greatly improves the data processing efficiency through memory-based calculation; thirdly, Spark data management depends on multiple data sources such as HDFS and Hive, Spark in a cluster mode realizes horizontal expansion, and large-scale data processing is supported. RDD is the most important concept of Spark to distinguish from other big data computing frameworks, which is a read-only distributed data set with a highly fault-tolerant mechanism. In the Spark application, each RDD is divided into a plurality of partitions, and Spark performs various operations on the RDD in units of partitions. And the data of the persistent (Persist) RDD partition is cached in a memory or a hard disk, so that the intermediate result of the calculation task can be directly read by the subsequent iteration task, the repeated calculation is avoided, and the data processing efficiency is greatly improved. In addition, the data is durably transmitted to the hard disk, the limitation of insufficient memory capacity on the size of the data set is broken, and spare processing of large data by Spark is enabled.
However, at present, the initial RDD dataset is divided according to a random proportion, and the persistence framework provided by Spark persists data to different storage media according to the proportion, so that persistence on demand cannot be realized.
Disclosure of Invention
The invention aims to solve the technical problem that on-demand persistence cannot be realized in the prior art, and provides an RDD persistence method based on an SSD and HDD hybrid storage system, which can not realize on-demand persistence.
The embodiment of the invention provides an RDD (remote data description) persistence method based on a SSD (solid State disk) and HDD (hard disk drive) hybrid storage system, which comprises the following steps:
the RDD module transmits the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager;
the block manager transmits the block identifier and a preset persistence level to a disk block manager;
the disk block manager transmits the preset persistence level to a device adapter;
the equipment adapter receives a preset persistence level of data and reads two directory management variables in a configuration file, matches the preset persistence level with a temporary file directory in a corresponding directory management variable according to the preset persistence level of the data, and returns the temporary file directory obtained by matching to the disk block manager;
the disk block manager obtains a file name according to the block identifier, obtains a data storage address according to the temporary file directory and the file name obtained by matching, and returns the data storage address to the block manager;
and the block manager stores the data in the RDD module in the SSD or the HDD according to the data storage address.
The invention also provides a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that: and storing the data in the RDD module in the SSD or the HDD according to the preset persistence level by using the data storage address so as to realize the on-demand persistence of the Spark application program.
Drawings
FIG. 1 is a block diagram of one embodiment of a distributed computing system according to the present invention.
FIG. 2 is a flow chart of one embodiment of a data processing method of the distributed computing system of the present invention.
FIG. 3 is a flow chart of one embodiment of the RDD persistence method based on the SSD and HDD hybrid storage system of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
Specifically, the emergence of a Solid-State Drive (SSD) brings a new opportunity for improving the performance of a storage system, and the SSD has the advantages of low power consumption, low latency, small size, and the like. Unlike the conventional Hard disk drive (Hard disk drive for short) which addresses by moving a robot arm, the SSD is completely built on a semiconductor chip, and thus has a random access performance. However, due to the disadvantages of high cost and limited life span of the SSD, the complete replacement of the HDD with the SSD will significantly increase the cost of the industry. In order to make reasonable use of the advantages of high performance of SSDs and low price of HDDs, heterogeneous data centers based on hybrid storage of SSDs and HDDs are widely researched and applied.
As shown in fig. 1, the distributed computing system according to an embodiment of the present invention includes a spare platform module 1 and a hybrid storage module 2, where the hybrid storage module 2 includes an SSD unit 21 and an HDD unit 22, and the spare platform module 1 is connected to the SSD unit 21 and the HDD unit 22 respectively;
the Spark platform module 1 uses a big data processing frame Spark as a calculation engine, and sends the processed data to the SSD unit 21 or the HDD unit 22 for storage, and the Spark platform module 1 is further configured to receive a query instruction, and fetch and output data corresponding to the query instruction from the SSD unit 21 or the HDD unit 22.
The Spark platform module is respectively connected with the SSD unit and the HDD unit, so that the processed data is sent to the SSD unit or the HDD unit for storage, and accurate mapping and storage of the data can be realized.
In a specific implementation, the Spark platform module 1 includes a first API (application programming interface) corresponding to the SSD unit 21 and a second API corresponding to the HDD unit, the Spark platform module 1 is connected to the SSD unit 21 through the first API, and the Spark platform module 1 is connected to the HDD unit 22 through the second API, so as to perform data transmission. The Spark platform module 1 may expose the structural features of the hybrid storage system to the user through the first API and the second API. The selection of the storage medium is realized by calling the first API or the second API interface, that is, the selection of the storage in the SSD unit 21 or the HDD unit 22 is realized by calling the first API or the second API interface.
In a specific implementation, the SSD unit 21 and the HDD unit 22 are persistent storage units in the same layer. The processed data specifically includes RDD partition data. The Spark platform module is further used for persisting the RDD partition data into the SSD unit or the HDD unit according to a preset partition proportion value.
In a specific implementation, the spare platform module 1 is further configured to persist the RDD partition data into the SSD unit or the HDD unit according to a hot degree of the RDD partition data. The I/O bandwidth and reduced access latency due to SSD can be effectively increased. HDDs still provide substantial storage efficiency for data that requires less storage performance. After an additional large amount of data is collected and captured by the data center, it is not often accessed, called cold data, which accounts for about 90% of global data. While the remaining 10% of the data is collected and captured and is accessed frequently, referred to as hot data. Clearly, it is not reasonable to store all of the data on a high performance, low latency storage device, and the cost is prohibitively expensive. Therefore, according to the heat of the RDD partition data, the SSD unit 21 and the HDD unit 22 are combined in a reasonable manner, performance can be greatly improved by constructing a hybrid storage system, and cost controllability is ensured.
In specific implementation, the distributed computing system further includes a capacity monitoring module connected to the hybrid storage module, where the capacity monitoring module is configured to monitor the remaining capacity of the hybrid storage module and output an alarm signal when the remaining capacity is smaller than a preset threshold. That is to say, the distributed computing system may further include a capacity monitoring module connected to the hybrid storage module 2, where the capacity monitoring module is configured to monitor the remaining capacity of the hybrid storage module 2, and output alarm information when the remaining capacity is smaller than a preset threshold. The specific value of the preset threshold can be determined according to the capacity of the hybrid storage module 2, and the output alarm information can be the control of the loudspeaker to sound or the control of the alarm lamp to flash and the like. When the residual capacity of the hybrid storage module 2 is too low, an alarm is given to remind a worker to transfer the stored data or replace a storage hard disk and the like in time so as to improve the reliability of data storage.
The present invention also provides a data processing method of a distributed computing system according to an embodiment, as shown in fig. 2, the data processing method includes the following steps:
step S21, the Spark platform module sends the processed data to the SSD unit or the HDD unit for storage by using a big data processing frame Spark as a calculation engine;
step S22, the Spark platform module receives the query instruction, and acquires data corresponding to the query instruction from the SSD unit or the HDD unit and outputs the data.
The Spark platform module is respectively connected with the SSD unit and the HDD unit, so that the processed data is sent to the SSD unit or the HDD unit for storage, and accurate mapping and storage of the data can be realized.
In specific implementation, the data processing method further includes the following steps of monitoring the remaining capacity of the hybrid storage module through a capacity monitoring module, and outputting alarm information when the remaining capacity is smaller than a preset threshold value. The specific value of the preset threshold can be determined according to the capacity of the hybrid storage module 2, and the output alarm information can be the control of the loudspeaker to sound or the control of the alarm lamp to flash and the like. When the residual capacity of the hybrid storage module 2 is too low, an alarm is given to remind a worker to transfer the stored data or replace a storage hard disk and the like in time so as to improve the reliability of data storage.
In a specific implementation, the Spark platform module 1 includes a first API (application programming interface) corresponding to the SSD unit 21 and a second API corresponding to the HDD unit, the Spark platform module 1 is connected to the SSD unit 21 through the first API, and the Spark platform module 1 is connected to the HDD unit 22 through the second API, so as to perform data transmission. The Spark platform module 1 may expose the structural features of the hybrid storage system to the user through the first API and the second API. The selection of the storage medium is realized by calling the first API or the second API interface, that is, the selection of the storage in the SSD unit 21 or the HDD unit 22 is realized by calling the first API or the second API interface.
In a specific implementation, the SSD unit 21 and the HDD unit 22 are persistent storage units in the same layer. The processed data specifically includes RDD partition data. The Spark platform module is further used for persisting the RDD partition data into the SSD unit or the HDD unit according to a preset partition proportion value.
In a specific implementation, the spare platform module 1 is further configured to persist the RDD partition data into the SSD unit or the HDD unit according to a hot degree of the RDD partition data. The I/O bandwidth and reduced access latency due to SSD can be effectively increased. HDDs still provide substantial storage efficiency for data that requires less storage performance. After an additional large amount of data is collected and captured by the data center, it is not often accessed, called cold data, which accounts for about 90% of global data. While the remaining 10% of the data is collected and captured and is accessed frequently, referred to as hot data. Clearly, it is not reasonable to store all of the data on a high performance, low latency storage device, and the cost is prohibitively expensive. Therefore, according to the heat of the RDD partition data, the SSD unit 21 and the HDD unit 22 are combined in a reasonable manner, performance can be greatly improved by constructing a hybrid storage system, and cost controllability is ensured.
In specific implementation, the RDD partition data is persisted by calling RDD. The operation of persisting the RDD is started by an RDD initiator method, and the content shown in fig. 3 is a persistence flow of RDD data. In addition, to persist RDD partition data, two conditions need to be met: the method comprises the steps of partitioning data and addresses, wherein the partitioning data are stored in an RDD module, the addresses need to be obtained through calculation, the addresses are paths/file names, the paths are stored in a configuration file, the paths need to be obtained according to a preset persistence level mapping configuration file of the partitioning data, and the file names need to be generated according to block identifiers.
The invention provides an embodiment of an RDD persistence method based on a SSD and HDD hybrid storage system, which is based on an optimized Spark framework to realize the persistence of RDD partition data, and comprises the following steps:
the RDD module transmits the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager;
the block manager transmits the block identifier and a preset persistence level to a disk block manager;
the disk block manager transmits the preset persistence level to a device adapter;
the equipment adapter receives a preset persistence level of data and reads two directory management variables in a configuration file, matches the preset persistence level with a temporary file directory in a corresponding directory management variable according to the preset persistence level of the data, and returns the temporary file directory obtained by matching to the disk block manager;
the disk block manager obtains a file name according to the block identifier, obtains a data storage address according to the temporary file directory and the file name obtained by matching, and returns the data storage address to the block manager;
and the block manager stores the data in the RDD module in the SSD or the HDD according to the data storage address.
The data storage address stores the data in the RDD module in the SSD or the HDD according to the preset persistence level, so that the on-demand persistence of the Spark application program is realized. That is, when the preset persistence level is SSD _ ONLY, the data in the RDD module is stored in the SSD, and when the preset persistence level is HDD _ ONLY, the data in the RDD module is stored in the HDD.
Specifically, as shown in fig. 3, the steps of the persistence method are as follows:
step 1, the RDD module calls a doputiterer method of a block manager Blockmanager through an iterer method to transmit a block identifier blockId in the RDD module and a preset persistence level of data in the RDD module to the block manager Blockmanager;
step 2, the doPutIterator method of the block manager BlockManager calls the getFile method of the disk block manager, and transmits the block identification blockId in the RDD module and the preset persistence level of the data in the RDD module to the DiskBlockManager;
step 3, the getFile method of the disk block manager DiskBlockManager calls a getACCURateDir method of the device adapter to transfer the preset persistence level to the device adapter;
step 4, the device adapter DeviceAdapter reads two directory management variables in the configuration file, specifically, the two directory management variables include an SSD directory management variable and an HDD directory management variable;
step 5, the device adapter DeviceAdapter matches the preset persistence level with the temporary file directory in the corresponding directory management variable according to the preset persistence level of the data, that is, the device adapter DeviceAdapter can obtain the preset persistence level from the upper layer, can obtain the configuration file such as the SSD directory management variable and the HDD directory management variable from the lower layer, and can complete the preset persistence level and the temporary file directory, that is, the getAccurateDir method reads the configuration file, wherein the configuration file includes two variables of the SSD directory management variable and the HDD directory management variable, and then matches the two variables according to the received preset persistence level. If the preset persistence level is SSD _ ONLY, matching an SSD directory management variable; if the preset persistence level is HDD _ ONLY, matching HDD directory management variables, obtaining a specific storage address of RDD data persistence at the moment, and then returning the address to the disk block manager DiskBlockManager;
step 6, returning the temporary file directory obtained by matching to the disk block manager DiskBlockManager, that is, the temporary file directory obtained by matching contains a specific storage address, and then returning the address to the disk block manager DiskBlockManager;
step 7, the disk block manager DiskBlockManager obtains a fileName according to the block identification blockId, and obtains a data storage address according to the temporary file directory obtained by matching and the fileName, that is, the specific address + fileName is a complete address, that is, a data storage address, where RDD _ and Index are digital indexes, and are sequentially incremented, and the data storage address is a directory/fileName, and the temporary file directory is a storage path;
step 8, the disk block manager DiskBlockManager returns the data storage address to the block manager BlockManager;
and 9, after the block manager BlockManager obtains the data storage address of the RDD, calling a writeFunc method of the DiskStore block storage module to finish the data storage task.
In a specific implementation, the RDD persistence method further comprises the steps of;
judging whether the heat degree of the data in the RDD module is greater than a first preset value or not;
if so, the preset persistence level of the data in the RDD module is SSD _ ONLY;
and if not, the preset persistence level of the data in the RDD module is HDD _ ONLY.
That is, according to the heat of the data in the RDD partition, the preset persistence level of the data is set to realize the combination of the SSD unit 21 and the HDD unit 22 in a reasonable manner, and the performance can be greatly improved by constructing the hybrid storage system, while ensuring the controllability of the cost.
That is, by means of the optimized Spark persistence framework, on-demand persistence of Spark data is achieved. Furthermore, the user can call an SSD persistence-oriented API provided by the optimized Spark framework to persist the partition data of the high-heat RDD into the SSD, so that the Spark performance is effectively improved.
The present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method of fig. 3 described above.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims (10)

1.一种基于SSD和HDD混合存储系统的RDD持久化方法，其特征在于：所述方法包括以下步骤：1. A method for RDD persistence based on SSD and HDD hybrid storage system, characterized in that: the method comprises the following steps: RDD模块将RDD模块中的块标识和RDD模块中数据的预设持久化级别传递给块管理器，所述预设持久化级别为SSD_ONLY或HDD_ONLY；The RDD module transmits the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager, where the preset persistence level is SSD_ONLY or HDD_ONLY; 所述块管理器将所述块标识和预设持久化级别传递给磁盘块管理器；The block manager passes the block identifier and the preset persistence level to the disk block manager; 所述磁盘块管理器将所述预设持久化级别传递给设备适配器；the disk block manager passing the preset persistence level to the device adapter; 所述设备适配器接收数据的预设持久化级别和读取配置文件中两个目录管理变量，所述两个目录管理变量包括SSD目录管理变量和HDD目录管理变量，根据数据的预设持久化级别进行预设持久化级别和对应目录管理变量中临时文件目录匹配，并将匹配得到的临时文件目录返回给所述磁盘块管理器；The device adapter receives the preset persistence level of the data and reads two directory management variables in the configuration file. The two directory management variables include the SSD directory management variable and the HDD directory management variable. According to the preset persistence level of the data Matching the preset persistence level with the temporary file directory in the corresponding directory management variable, and returning the matching temporary file directory to the disk block manager; 所述磁盘块管理器根据所述块标识得到文件名，并根据匹配得到的临时文件目录和所述文件名得到数据存储地址，并将所述数据存储地址返回至所述块管理器；The disk block manager obtains a file name according to the block identifier, and obtains a data storage address according to the temporary file directory and the file name obtained by matching, and returns the data storage address to the block manager; 所述块管理器根据所述数据存储地址对RDD模块中的数据在SSD或HDD中进行存储。The block manager stores the data in the RDD module in the SSD or HDD according to the data storage address. 2.如权利要求1所述的RDD持久化方法，其特征在于：所述RDD模块将RDD模块中的块标识和RDD模块中数据的预设持久化级别传递给块管理器的步骤，具体为：2. The RDD persistence method of claim 1, wherein the RDD module transmits the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager, specifically: : 所述RDD模块通过Iterator方法调用块管理器的doPutIterator方法将RDD模块中的块标识和RDD模块中数据的预设持久化级别传递给块管理器。The RDD module calls the doPutIterator method of the block manager through the Iterator method to transfer the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager. 3.如权利要求1所述的RDD持久化方法，其特征在于：所述块管理器将所述块标识和预设持久化级别传递给磁盘块管理器的步骤，具体为：3. The RDD persistence method according to claim 1, wherein the step of the block manager passing the block identifier and the preset persistence level to the disk block manager is specifically: 所述块管理器调用磁盘块管理器的getFile方法，将RDD模块中的块标识和RDD模块中数据的预设持久化级别传递给所述磁盘块管理器。The block manager calls the getFile method of the disk block manager, and transmits the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the disk block manager. 4.如权利要求1所述的RDD持久化方法，其特征在于：所述磁盘块管理器根据所述块标识得到文件名，并将所述预设持久化级别传递给设备适配器的步骤，具体为：4. The RDD persistence method according to claim 1, wherein the disk block manager obtains the file name according to the block identifier, and transmits the preset persistence level to the device adapter, specifically for: 所述磁盘块管理器通过getFile方法根据所述块标识得到文件名；The disk block manager obtains the file name according to the block identifier by the getFile method; 所述磁盘块管理器调用设备适配器的getAccurateDir方法将所述预设持久化级别传递给设备适配器。The disk block manager calls the getAccurateDir method of the device adapter to transfer the preset persistence level to the device adapter. 5.如权利要求1所述的RDD持久化方法，其特征在于：所述设备适配器接收数据的预设持久化级别和读取配置文件中两个目录管理变量，根据数据的预设持久化级别进行预设持久化级别和对应目录管理变量中临时文件目录匹配，并将匹配得到的临时文件目录返回给所述磁盘块管理器的步骤，具体为：5. The RDD persistence method according to claim 1, wherein the device adapter receives a preset persistence level of the data and reads two directory management variables in the configuration file, according to the preset persistence level of the data The steps of matching the preset persistence level with the temporary file directory in the corresponding directory management variable, and returning the matching temporary file directory to the disk block manager are specifically: 所述设备适配器通过getAccurateDir方法根据数据的预设持久化级别进行预设持久化级别和对应目录管理变量中临时文件目录匹配；The device adapter uses the getAccurateDir method to match the preset persistence level with the temporary file directory in the corresponding directory management variable according to the preset persistence level of the data; 所述设备适配器通过getAccurateDir方法将匹配得到的临时文件目录返回给所述磁盘块管理器。The device adapter returns the matching temporary file directory to the disk block manager through the getAccurateDir method. 6.如权利要求5所述的RDD持久化方法，其特征在于：两个目录管理变量包括SSD目录管理变量和HDD目录管理变量。6 . The RDD persistence method according to claim 5 , wherein the two directory management variables include SSD directory management variables and HDD directory management variables. 7 . 7.如权利要求6所述的RDD持久化方法，其特征在于：所述设备适配器通过getAccurateDir方法根据数据的预设持久化级别进行预设持久化级别和对应目录管理变量中临时文件目录匹配的步骤，具体为：7. The RDD persistence method according to claim 6, wherein the device adapter performs the matching between the preset persistence level and the temporary file directory in the corresponding directory management variable according to the preset persistence level of the data by the getAccurateDir method. The steps are as follows: 当数据的预设持久化级别为SSD_ONLY时，执行数据的预设持久化级别与SSD目录管理变量中临时文件目录的映射匹配；When the preset persistence level of the data is SSD_ONLY, the preset persistence level of the execution data matches the mapping of the temporary file directory in the SSD directory management variable; 当数据的预设持久化级别为HDD_ONLY时，执行数据的预设持久化级别与HDD目录管理变量中临时文件目录的映射匹配。When the preset persistence level of the data is HDD_ONLY, the preset persistence level of the execution data matches the mapping of the temporary file directory in the HDD directory management variable. 8.如权利要求1所述的RDD持久化方法，其特征在于，所述块管理器根据所述数据存储地址对RDD模块中的数据在SSD或HDD中进行存储的步骤，包括：8. The RDD persistence method according to claim 1, wherein the step of the block manager storing the data in the RDD module in the SSD or the HDD according to the data storage address, comprises: 所述块管理器获得RDD的数据存储地址后，调用块存储模块的writeFunc方法对RDD模块中的数据在SSD或HDD中进行存储。After the block manager obtains the data storage address of the RDD, it calls the writeFunc method of the block storage module to store the data in the RDD module in the SSD or HDD. 9.如权利要求1所述的RDD持久化方法，其特征在于：所述RDD持久化方法还包括以下步骤；9. The RDD persistence method according to claim 1, characterized in that: the RDD persistence method further comprises the following steps; 判断RDD模块中数据的热度是否大于第一预设值；Determine whether the heat of the data in the RDD module is greater than the first preset value; 如果是，所述RDD模块中数据的预设持久化级别为SSD_ONLY；If so, the preset persistence level of the data in the RDD module is SSD_ONLY; 如果否，所述RDD模块中数据的预设持久化级别为HDD_ONLY。If not, the preset persistence level of the data in the RDD module is HDD_ONLY. 10.一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现如权利要求1-9任意一项所述方法的步骤。10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the method according to any one of claims 1-9 are implemented.
CN201710358093.9A 2017-05-19 2017-05-19 A RDD Persistence Method Based on SSD and HDD Hybrid Storage System Active CN107193494B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710358093.9A CN107193494B (en) 2017-05-19 2017-05-19 A RDD Persistence Method Based on SSD and HDD Hybrid Storage System

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710358093.9A CN107193494B (en) 2017-05-19 2017-05-19 A RDD Persistence Method Based on SSD and HDD Hybrid Storage System

Publications (2)

Publication Number Publication Date
CN107193494A CN107193494A (en) 2017-09-22
CN107193494B true CN107193494B (en) 2020-05-12

Family

ID=59875380

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710358093.9A Active CN107193494B (en) 2017-05-19 2017-05-19 A RDD Persistence Method Based on SSD and HDD Hybrid Storage System

Country Status (1)

Country Link
CN (1) CN107193494B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018209693A1 (en) * 2017-05-19 2018-11-22 深圳大学 Rdd persistence method based on ssd and hdd hybrid storage system
CN107590003B (en) * 2017-09-28 2020-10-23 深圳大学 Spark task allocation method and system
CN109375868B (en) * 2018-09-14 2022-07-08 深圳爱捷云科技有限公司 Data storage method, scheduling device, system, equipment and storage medium
CN112799597A (en) * 2021-02-08 2021-05-14 东北大学 A fault-tolerant method of hierarchical storage for stream data processing
CN113590536B (en) * 2021-05-20 2023-12-29 济南浪潮数据技术有限公司 Data storage method, system, electronic equipment and storage medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104216988A (en) * 2014-09-04 2014-12-17 天津大学 SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data
CN105893541A (en) * 2016-03-31 2016-08-24 中国科学院软件研究所 Streaming data self-adaption persistence method and system based on mixed storage

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104216988A (en) * 2014-09-04 2014-12-17 天津大学 SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data
CN105893541A (en) * 2016-03-31 2016-08-24 中国科学院软件研究所 Streaming data self-adaption persistence method and system based on mixed storage

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
hStorag e-DB: Heterog eneity-aware Data Manag ement to;Luo Tian;《August 27th - 31st 2012, Istanbul, Turkey.》;20120831;第10卷(第5期);全文 *
Hybrid HBase: Leveraging Flash SSDs to Improve Cost per;Awasthi A;《The 18th International Conference on Management of Data (COMAD)》;20121216;全文 *

Also Published As

Publication number Publication date
CN107193494A (en) 2017-09-22

Similar Documents

Publication Publication Date Title
CN107193494B (en) A RDD Persistence Method Based on SSD and HDD Hybrid Storage System
CN106354425B (en) Data attribute-based data layout method and system
US9021189B2 (en) System and method for performing efficient processing of data stored in a storage node
US9092321B2 (en) System and method for performing efficient searches and queries in a storage node
US8332367B2 (en) Parallel data redundancy removal
CN104462225B (en) The method, apparatus and system of a kind of digital independent
CN102307206B (en) Caching method of caching system for quickly accessing virtual machine mirror image based on cloud storage
TWI771933B (en) Method for performing deduplication management with aid of command-related filter, host device, and storage server
TW201220197A (en) for improving the safety and reliability of data storage in a virtual machine based on cloud calculation and distributed storage environment
US11061676B2 (en) Scatter gather using key-value store
CN112632069A (en) Hash table data storage management method, device, medium and electronic equipment
CN104054071A (en) Method for accessing storage device and storage device
US11042328B2 (en) Storage apparatus and method for autonomous space compaction
CN104270412A (en) A three-level cache method based on Hadoop distributed file system
CN106126124A (en) A kind of data processing method and electronic equipment
US20240273074A1 (en) Efficient analytical calculations on a data set and method for use therewith
CN105718561A (en) Particular distributed data storage file structure redundancy removing construction method and system
CN112711564A (en) Merging processing method and related equipment
WO2024045545A1 (en) Domain address change method and apparatus, and device and readable storage medium
CN110162395B (en) Memory allocation method and device
CN109002255B (en) Memory system and method of operating the same
CN110781159B (en) Ceph directory file information reading method and device, server and storage medium
CN107179883B (en) Spark architecture optimization method of hybrid storage system based on SSD and HDD
CN112804003A (en) Optical module communication-based storage method, system and terminal
US20240111743A1 (en) Efficient evaluation of queries across multiple columnar storage tiers

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20220523

Address after: 518000 east of the fourth floor of plant 1 (Building 1) of Baode technology R & D and production base, gaoxinyuan, Guanlan street, Longhua new area, Shenzhen, Guangdong

Patentee after: Baode network security system (Shenzhen) Co.,Ltd.

Address before: 518000 No. 3688 Nanhai Road, Shenzhen, Guangdong, Nanshan District

Patentee before: SHENZHEN University

TR01 Transfer of patent right