CN107193494A - RDD (remote data description) persistence method based on SSD (solid State disk) and HDD (hard disk drive) hybrid storage system - Google Patents

Abstract

The present invention provides an RDD persistence method based on SSD and HDD hybrid storage system, comprising: the RDD module transmits the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager; the disk block The manager passes the preset persistence level to the device adapter; the device adapter receives the preset persistence level of the data and reads two directory management variables in the configuration file, and presets according to the preset persistence level of the data The persistence level matches the temporary file directory in the corresponding directory management variable, and returns the matched temporary file directory to the disk block manager; the disk block manager obtains the file name according to the block identifier, and obtains the file name according to the matching Temporary file directory and the file name to obtain the data storage address, and return the data storage address to the block manager; the block manager stores the data in the RDD module on SSD or HDD according to the data storage address stored in.

Description

A RDD Persistence Method Based on SSD and HDD Hybrid Storage System

technical field

The invention relates to the technical field of data processing, in particular to an RDD persistence method based on an SSD and HDD hybrid storage system.

Background technique

In the current era of big data, in the face of massive data, how to manage, analyze and extract valuable information in an effective time has become an urgent problem that people need to solve. However, regardless of size, variety or structure, big data poses a huge challenge to people's ability to manage data.

Spark is currently an efficient and widely used big data computing framework in the industry. It is a general and fast large-scale data processing engine. First of all, Spark provides a unified solution that can be used for complex tasks such as interactive query, real-time stream processing, and machine learning; secondly, Spark divides stages and tasks through Resilient Distributed Dataset (RDD). The Directed Acyclic Graph (DAG for short) execution engine optimizes the execution order of subtasks, and greatly improves data processing efficiency through memory-based calculations; third, Spark data management relies on HDFS, Hive and other data sources , and Spark in cluster mode realizes horizontal expansion and supports large-scale data processing. RDD is the most important concept that distinguishes Spark from other big data computing frameworks. It is a read-only distributed dataset with a highly fault-tolerant mechanism. In a Spark application, each RDD is divided into multiple partitions, and Spark performs various operations on the RDD in units of partitions. Persisting (Persist) RDD partition data to memory or hard disk realizes the caching of the intermediate results of computing tasks, so that subsequent iterative tasks can directly read the intermediate results, avoiding repeated calculations, and greatly improving the efficiency of data processing. In addition, persisting data to the hard disk breaks the limitation of insufficient memory capacity on the size of the data set, allowing Spark to handle large data with ease.

However, the current initial RDD data set is divided according to a random ratio, and the persistence framework provided by Spark persists the data to different storage media according to this ratio, which cannot achieve on-demand persistence.

Contents of the invention

The present invention aims to solve the technical problem that the on-demand persistence cannot be realized in the prior art, and provides an RDD persistence method based on the SSD and HDD hybrid storage system that cannot realize the on-demand persistence.

The embodiment of the present invention provides a kind of RDD persistent method based on SSD and HDD hybrid storage system, and described method comprises the following steps:

The RDD module passes the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager;

The block manager passes the block identifier and the preset persistence level to the disk block manager;

The disk block manager passes the preset persistence level to the device adapter;

The device adapter receives the preset persistence level of the data and reads the two directory management variables in the configuration file, performs the preset persistence level matching with the temporary file directory in the corresponding directory management variable according to the preset persistence level of the data, and return the matched temporary file directory to the disk block manager;

The disk block manager obtains the file name according to the block identifier, and obtains the data storage address according to the matching temporary file directory and the file name, and returns the data storage address to the block manager;

The block manager stores the data in the RDD module in SSD or HDD according to the data storage address.

The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are realized.

Compared with the prior art, the technical solution of the present invention has the beneficial effect of: storing the data in the data storage address pair RDD module in SSD or HDD according to the preset persistence level, so as to realize the on-demand application of Spark Persistence.

Description of drawings

FIG. 1 is a schematic structural diagram of an embodiment of the distributed computing system of the present invention.

Fig. 2 is a flowchart of an embodiment of the data processing method of the distributed computing system of the present invention.

FIG. 3 is a flowchart of an embodiment of the RDD persistence method based on the SSD and HDD hybrid storage system of the present invention.

detailed description

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

Specifically, the emergence of solid-state drives (Solid-State Drive, SSD for short) has brought new opportunities for improving the performance of storage systems. SSDs have advantages such as low power consumption, low latency, and small size. Different from the traditional enterprise hard disk (Hard DiskDrive, referred to as HDD) which is addressed by moving the mechanical arm, the SSD is completely built on the semiconductor chip, so it has random access performance. However, due to the high cost of SSD capacity and limited lifespan, replacing HDD with SSD will greatly increase the cost of the industry. In order to reasonably utilize the advantages of high performance of SSD and low price of HDD, heterogeneous data centers based on hybrid storage of SSD and HDD have been widely researched and applied.

The distributed computing system of one embodiment of the present invention, as shown in Figure 1, comprises Spark platform module 1 and hybrid storage module 2, and described hybrid storage module 2 comprises SSD unit 21 and with HDD unit 22, and described Spark platform module 1 Connect with the SSD unit 21 and the HDD unit 22 respectively;

The Spark platform module 1 utilizes the big data processing framework Spark as a computing engine, and the data obtained by processing is sent to the SSD unit 21 or the HDD unit 22 for storage, and the Spark platform module 1 is also used for receiving query instructions, And fetch the data corresponding to the query command from the SSD unit 21 or the HDD unit 22 and output it.

The Spark platform module is respectively connected to the SSD unit and the HDD unit, so that the processed data is sent to the SSD unit or the HDD unit for storage, and accurate mapping and storage of data can be realized.

In a specific implementation, the Spark platform module 1 includes a first API (ApplicationProgrammingInterface, application programming interface) corresponding to the SSD unit 21 and a second API corresponding to the HDD unit, and the Spark platform module 1 passes The first API is connected to the SSD unit 21, and the Spark platform module 1 is connected to the HDD unit 22 through the second API for data transmission. The Spark platform module 1 can display the structural features of the hybrid storage system to the user through the first API and the second API. The selection of the storage medium is realized by calling the first API or the second API interface, that is, selecting to store in the SSD unit 21 or the HDD unit 22 is realized by calling the first API or the second API interface.

In a specific implementation, the SSD unit 21 and the HDD unit 22 are persistent storage units of the same layer. The processed data specifically includes RDD partition data. The Spark platform module is also used to persist the RDD partition data into the SSD unit or the HDD unit according to a preset partition ratio value.

In a specific implementation, the Spark platform module 1 is further configured to persist the RDD partition data into the SSD unit or the HDD unit according to the heat of the RDD partition data. Due to SSD's I/O bandwidth and lower access latency can be effectively improved. And HDD can still provide a lot of storage efficiency for those data that have lower storage performance requirements. In addition, after a large amount of data is collected and captured by the data center, it is not frequently accessed, which is called cold data, accounting for about 90% of the global data. After the remaining 10% of data is collected and captured, it will be frequently accessed, which is called hot data. Obviously, it is unreasonable to store all data in high-performance, low-latency storage devices, and the cost is extremely expensive. Therefore, according to the popularity of the RDD partition data, the SSD unit 21 and the HDD unit 22 can be combined in a reasonable manner, and the performance can be greatly improved by building a hybrid storage system, while ensuring cost control.

In a specific implementation, the distributed computing system further includes a capacity monitoring module connected to the hybrid storage module, the capacity monitoring module is used to monitor the remaining capacity of the hybrid storage module, and when the remaining capacity is less than a preset An alarm signal is output when the threshold is reached. That is to say, the distributed computing system may also include a capacity monitoring module connected to the hybrid storage module 2, the capacity monitoring module is used to monitor the remaining capacity of the hybrid storage module 2, and output an alarm message when the remaining capacity is less than a preset threshold. The specific value of the preset threshold can be determined according to the capacity of the hybrid storage module 2, and the output of the alarm information can be to control the sound of the speaker or control the flashing of the alarm light. When the remaining capacity of the hybrid storage module 2 is too low, an alarm is issued to remind the staff to transfer the stored data or replace the storage hard disk in time, so as to improve the reliability of data storage.

The present invention also provides a data processing method of the distributed computing system of an embodiment. As shown in FIG. 2, the data processing method includes the following steps:

Step S21, the Spark platform module uses the big data processing framework Spark as a computing engine, and sends the processed data to the SSD unit or the HDD unit for storage;

Step S22, the Spark platform module receives the query command, and obtains data corresponding to the query command from the SSD unit or the HDD unit, and then outputs it.

The Spark platform module is respectively connected to the SSD unit and the HDD unit, so that the processed data is sent to the SSD unit or the HDD unit for storage, and accurate mapping and storage of data can be realized.

In a specific implementation, the data processing method further includes the following steps of monitoring the remaining capacity of the hybrid storage module through the capacity monitoring module, and outputting an alarm message when the remaining capacity is less than a preset threshold. The specific value of the preset threshold can be determined according to the capacity of the hybrid storage module 2, and the output of the alarm information can be to control the sound of the speaker or control the flashing of the alarm light. When the remaining capacity of the hybrid storage module 2 is too low, an alarm is issued to remind the staff to transfer the stored data or replace the storage hard disk in time, so as to improve the reliability of data storage.

In a specific implementation, the Spark platform module 1 includes a first API (ApplicationProgrammingInterface, application programming interface) corresponding to the SSD unit 21 and a second API corresponding to the HDD unit, and the Spark platform module 1 passes The first API is connected to the SSD unit 21, and the Spark platform module 1 is connected to the HDD unit 22 through the second API for data transmission. The Spark platform module 1 can display the structural features of the hybrid storage system to the user through the first API and the second API. The selection of the storage medium is realized by calling the first API or the second API interface, that is, selecting to store in the SSD unit 21 or the HDD unit 22 is realized by calling the first API or the second API interface.

In a specific implementation, the SSD unit 21 and the HDD unit 22 are persistent storage units of the same layer. The processed data specifically includes RDD partition data. The Spark platform module is also used to persist the RDD partition data into the SSD unit or the HDD unit according to a preset partition ratio value.

In a specific implementation, the Spark platform module 1 is further configured to persist the RDD partition data into the SSD unit or the HDD unit according to the heat of the RDD partition data. Due to SSD's I/O bandwidth and lower access latency can be effectively improved. And HDD can still provide a lot of storage efficiency for those data that have lower storage performance requirements. In addition, after a large amount of data is collected and captured by the data center, it is not frequently accessed, which is called cold data, accounting for about 90% of the global data. After the remaining 10% of data is collected and captured, it will be frequently accessed, which is called hot data. Obviously, it is unreasonable to store all data in high-performance, low-latency storage devices, and the cost is extremely expensive. Therefore, according to the popularity of the RDD partition data, the SSD unit 21 and the HDD unit 22 can be combined in a reasonable manner, and the performance can be greatly improved by building a hybrid storage system, while ensuring cost control.

In a specific implementation, the RDD partition data is persisted by calling RDD.persist(StorageLevel.SSD_ONLY), and at the same time, the preset persistence level of the partition data is set to SSD_ONLY. The operation of persisting the RDD is started by the RDD.iterator method, and the content shown in Figure 3 is the persistence process of the RDD data. In addition, to persist RDD partition data, two conditions need to be met: partition data + address, the partition data has been saved in the RDD module, and the address needs to be obtained through calculation, address = path/file name, and the path has been saved in the configuration file , it needs to be obtained according to the preset persistence level mapping configuration file of the partition data, and the file name needs to be generated according to the block identifier.

The present invention provides an embodiment of an RDD persistence method based on an SSD and HDD hybrid storage system. The persistence method is based on an optimized Spark framework to achieve persistence of RDD partition data. The persistence method includes the following step:

The RDD module passes the block identifier in the RDD module and the preset persistence level of the data in the RDD module to the block manager;

The block manager passes the block identifier and the preset persistence level to the disk block manager;

The disk block manager passes the preset persistence level to the device adapter;

The device adapter receives the preset persistence level of the data and reads the two directory management variables in the configuration file, performs the preset persistence level matching with the temporary file directory in the corresponding directory management variable according to the preset persistence level of the data, and return the matched temporary file directory to the disk block manager;

The disk block manager obtains the file name according to the block identifier, and obtains the data storage address according to the matching temporary file directory and the file name, and returns the data storage address to the block manager;

The block manager stores the data in the RDD module in SSD or HDD according to the data storage address.

The present invention stores the data in the data storage address pair RDD module in the SSD or HDD according to the preset persistence level, so as to realize the on-demand persistence of the Spark application program. That is to say, when the preset persistence level is SSD_ONLY, the data in the RDD module is stored in SSD, and when the preset persistence level is HDD_ONLY, the data in the RDD module is stored in HDD.

Specifically, as shown in Figure 3, the steps of the persistence method are as follows:

Step 1, the RDD module calls the doPutIterator method of the block manager BlockManager through the Iterator method to pass the block identifier blockId in the RDD module and the preset persistence level of the data in the RDD module to the block manager BlockManager;

Step 2, the doPutIterator method of the block manager BlockManager calls the getFile method of the disk block manager, and the block identifier blockId in the RDD module and the preset persistence level of the data in the RDD module are passed to the disk block manager DiskBlockManager;

Step 3, the getFile method of the disk block manager DiskBlockManager calls the getAccurateDir method of the device adapter to pass the preset persistence level to the device adapter DeviceAdapter;

Step 4, the device adapter DeviceAdapter reads two directory management variables in the configuration file, specifically, the two directory management variables include SSD directory management variables and HDD directory management variables;

Step 5, the device adapter DeviceAdapter matches the preset persistence level with the temporary file directory in the corresponding directory management variable according to the preset persistence level of the data, that is to say, the device adapter DeviceAdapter can obtain the preset persistence level from the upper layer , can obtain configuration files from the lower layer such as SSD directory management variables and HDD directory management variables, and can complete the preset persistence level and temporary file directory, that is, the getAccurateDir method reads the configuration file, and the configuration file includes two variables for SSD Directory management variable and HDD directory management variable, and then match the above two variables according to the received preset persistence level. If the preset persistence level is SSD_ONLY, match the SSD directory management variable; if the preset persistence level is HDD_ONLY, match the HDD directory management variable. At this time, the specific storage address of the RDD data persistence is obtained, and then the address is returned to the disk block manager DiskBlockManager;

Step 6, return the matched temporary file directory to the disk block manager DiskBlockManager, that is, the matched temporary file directory contains a specific storage address, and then return the address to the disk block manager DiskBlockManager;

Step 7, the disk block manager DiskBlockManager obtains the file name filename according to the block identifier blockId, and obtains the data storage address according to the temporary file directory obtained by matching and the file name, that is to say, the specific address+fileName is the RDD data The complete address stored to the disk is the data storage address, where fileName=“rdd_”+Index, Index is a numerical index, which increases in order, and the data storage address=directory/file name, and the temporary file directory is also the storage path;

Step 8, the disk block manager DiskBlockManager returns the data storage address to the block manager BlockManager;

Step 9: After obtaining the data storage address of the RDD, the block manager BlockManager calls the writeFunc method of the block storage module DiskStore to complete the data storage task.

In a specific implementation, the RDD persistence method also includes the following steps;

Judging whether the heat of the data in the RDD module is greater than a first preset value;

If yes, the preset persistence level of data in the RDD module is SSD_ONLY;

If not, the preset persistence level of data in the RDD module is HDD_ONLY.

That is, according to the heat of the data in the RDD partition, the preset persistence level of the data is set to realize the combination of the SSD unit 21 and the HDD unit 22 in a reasonable manner. By building a hybrid storage system, the performance can be greatly improved, and at the same time The cost of guarantee is controllable.

That is to say, through the optimized Spark persistence framework, the on-demand persistence of Spark data is realized. Furthermore, the user can call the SSD persistence-oriented API provided by the optimized Spark framework to persist the partition data of the hot RDD to the SSD, thereby effectively improving Spark performance.

The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above-mentioned method in FIG. 3 are realized.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (10)

1. a kind of RDD persistence methods that storage system is mixed based on SSD and HDD, it is characterised in that：Methods described includes following Step：

The default persistence rank of data in block identification in RDD modules and RDD modules is passed to block manager by RDD modules；

The block identification and default persistence rank are passed to disk block manager by described piece of manager；

The default persistence rank is passed to device adapter by the disk block manager；

The device adapter receives the default persistence rank of data and reads two directory management variables, root in configuration file Temporary file directory in default persistence rank and correspondence directory management variable is carried out according to the default persistence rank of data to match, And the temporary file directory for obtaining matching returns to the disk block manager；

The disk block manager obtains filename, and the temporary file directory obtained according to matching and institute according to the block identification State filename and obtain address data memory, and the address data memory is back to described piece of manager；

Described piece of manager is stored according to the address data memory to the data in RDD modules in SSD or HDD.

2. RDD persistence methods as claimed in claim 1, it is characterised in that：The RDD modules are by the block mark in RDD modules The step of default persistence rank known with data in RDD modules passes to block manager, be specially：

The RDD modules are by the doPutIterator methods of Iterator method call block managers by the block in RDD modules The default persistence rank of data passes to block manager in mark and RDD modules.

3. RDD persistence methods as claimed in claim 1, it is characterised in that：Described piece of manager is by the block identification and in advance If the step of persistence rank passes to disk block manager, it is specially：

Described piece of manager calls the getFile methods of disk block manager, by the block identification in RDD modules and RDD modules The default persistence rank of data passes to the disk block manager.

4. RDD persistence methods as claimed in claim 1, it is characterised in that：The disk block manager is according to described piece of mark The step of knowing and obtain filename, and the default persistence rank is passed into device adapter, be specially：

The disk block manager obtains filename by getFile methods according to the block identification；

The disk block manager calls the getAccurateDir methods of device adapter by the default persistence level supplementary biography Pass device adapter.

5. RDD persistence methods as claimed in claim 1, it is characterised in that：The device adapter receives the default of data Two directory management variables in persistence rank and reading configuration file, default hold is carried out according to the default persistence rank of data Temporary file directory is matched in longization rank and correspondence directory management variable, and the temporary file directory that matching is obtained is returned to The step of disk block manager, it is specially：

It is default lasting that the device adapter is carried out by getAccurateDir methods according to the default persistence rank of data Change temporary file directory in rank and correspondence directory management variable to match；

The device adapter returns to the magnetic by getAccurateDir methods by obtained temporary file directory is matched Disk block manager.

6. RDD persistence methods as claimed in claim 5, it is characterised in that：Two directory management variables include SSD catalogue pipes Manage variable and HDD directory management variables.

7. RDD persistence methods as claimed in claim 6, it is characterised in that：The device adapter passes through GetAccurateDir methods carry out default persistence rank according to the default persistence rank of data and correspondence directory management becomes The step of temporary file directory is matched in amount, be specially：

When the default persistence rank of data is SSD_ONLY, default persistence rank and the SSD directory managements of data are performed The mapping matching of temporary file directory in variable；

When the default persistence rank of data is HDD_ONLY, default persistence rank and the HDD directory managements of data are performed The mapping matching of temporary file directory in variable.

8. RDD persistence methods as claimed in claim 1, it is characterised in that described piece of manager is according to the data storage The step of address is stored to the data in RDD modules in SSD or HDD, including：

Described piece of manager is obtained after RDD address data memory, calls the writeFunc methods of block memory module to RDD moulds Data in block are stored in SSD or HDD.

9. RDD persistence methods as claimed in claim 1, it is characterised in that：The RDD persistence methods also include following step Suddenly；

Judge whether the temperature of data in RDD modules is more than the first preset value；

If it is, the default persistence rank of data is SSD_ONLY in the RDD modules；

If not, the default persistence rank of data is HDD_ONLY in the RDD modules.

10. a kind of computer-readable recording medium, is stored thereon with computer program, it is characterised in that the program is by processor Realized during execution such as the step of claim 1-9 any one methods describeds.