CN106354805A - Optimization method and system for searching and caching distribution storage system NoSQL - Google Patents

Abstract

The present invention provides a method and system for optimizing the NoSQL search cache of a distributed storage system. The method includes: step S101, in the HBase client, pre-set the local SSD as a read-only cache; step S102, when the HBase client reads When getting the target data, judge whether the target data is located on the local SSD, if yes, then enter step S104; if not, then enter step S103; step S103, read the target data from the HDFS cluster and cache to the local SSD above; step S104, the local SSD returns the target data. The above technical solution introduces the local SSD to provide the read-only cache function, giving full play to the random read performance of the SSD, and caches related files in the SSD. Because the accessed data is local, the files cached in the SSD can effectively reduce the time to read HDFS. I/O number, thereby improving the performance of the distributed storage system, and achieving the purpose of improving the efficiency of data search.

Description

Method and system for optimizing NoSQL search cache in a distributed storage system

technical field

The invention belongs to the technical field of computers, and in particular relates to an optimization method and system for a distributed storage system NoSQL search cache.

Background technique

A single distributed storage system can no longer meet the storage, management and search needs of large-scale massive data. At present, distributed storage system and multi-data center storage are the key technical approaches to meet the demand for EB-level data storage. How to quickly search for data that meets user requirements from a large-scale data set is an urgent problem to be solved in the current cross-data center storage system, especially the emergence of new storage media such as SSD and its excellent performance. Therefore, in order to improve the search efficiency, it is necessary to break through the bottleneck of the search technology based on SSD and load locality characteristics.

Compared with traditional mechanical hard drives, solid state drives (SSDs) have great advantages in terms of performance. SSD has extremely high performance and is very suitable for requests requiring fast response time and high read/write times per second. At present, the application of SSD in large-scale storage systems is divided into SSD hierarchical storage technology and SSD cache technology. SSD tiered storage technology can store high throughput and low access response time of the system through the tiered storage method, but the difficulty of using the tiered technology method is how to determine the value of different stored data and files to obtain good throughput At the same time, the layered tier method requires more SSDs, which means higher storage costs; SSD caching technology uses SSDs as the caching technology of the storage system according to the temporal locality and spatial locality of data access.

At present, the common NoSQL storage systems represented by BigTable include HBase, Cassandra, etc. The bottom layer uses LSM tree to organize data, and the data writing will not change. However, for general solutions such as SSD cache storage, no specific SSD cache method has been established for the specific load characteristics of the upper layer, nor has a data search cache method been established for the above-mentioned NoSQL system that writes once and reads many times, especially in cross-data Data access to distributed storage system NoSQL under remote environment conditions in the central environment. Correspondingly, the data search efficiency is low, and there is a large room for improvement.

Contents of the invention

In view of this, the embodiments of the present invention provide a distributed storage system NoSQL search cache optimization method and system to solve the problem that the prior art does not address the specific load characteristics of the upper layer and the characteristics of the distributed storage system NoSQL write once and read multiple times The caching method leads to the technical problem of low data search efficiency, and achieves the purpose of improving data search efficiency.

The technical scheme provided by the invention is as follows:

A method for optimizing NoSQL search cache in a distributed storage system, comprising:

Step S101, in the HBase client, pre-set the local SSD as a read-only cache;

Step S102, when the HBase client reads the target data, judge whether the target data is located on the local SSD, if yes, then enter step S104; if no, then enter step S103;

Step S103, read the target data from the HDFS cluster and cache it on the local SSD;

Step S104, the local SSD returns the target data.

Preferably, before the step S102, it also includes:

The HBase client sends the read request to the corresponding HRegionServe;

The HRegionServer judges whether the target data is located in the local memory according to the received read request, if yes, returns the target data from the local memory, and if not, proceeds to step S102.

Preferably, the step S104 includes:

The local SSD returns the target data to the local memory, and the local memory returns the target data.

Preferably, before the step S103, it also includes:

Judging whether the remaining storage space of the local SSD meets the preset requirements, if yes, proceed to step S103; if not, execute a replacement algorithm to replace existing data in the local SSD with the read target data.

Preferably, the method also includes:

The HBase client generates a Compact operation request, judges whether the target HFile file corresponding to the Compact operation request is located on the local SSD, if yes, then executes the Compact operation; if not, then reads the target HFile file cache from the HDFS cluster to the described The Compact operation is performed on the local SSD. After the Compact operation is completed, the original HFile file is deleted from the local SSD, and the newly generated HFile file is written to the HDFS cluster and cached in the local SSD.

Preferably, the method also includes:

The HBase client generates a Split operation request, judges whether the target HFile file corresponding to the Split operation request is located on the local SSD, if yes, then executes the Split operation; if not, then reads the target HFile file cache from the HDFS cluster to the described The split operation is performed on the local SSD. After the split operation is completed, the original HFile file is deleted from the local SSD.

Corresponding to the above method, the present invention also provides an optimization system for a distributed storage system NoSQL search cache, including:

Local SSD for setting in HBase client as read-only cache;

The first judging module is used to judge whether the target data is located on the local SSD when the HBase client reads the target data;

A data reading module, for when the target data is not located on the local SSD, read the target data from the HDFS cluster and cache it on the local SSD;

A data returning module, configured to return the target data from the local SSD when the target data is located on the local SSD.

Preferably, the system further includes:

A second judging module, configured to judge whether the remaining storage space of the local SSD meets a preset requirement;

The replacement module is configured to execute a replacement algorithm when the remaining storage space of the local SSD does not meet the preset requirements, and replace the existing data in the local SSD with the read target data.

Preferably, the system further includes:

The Compact module is configured to perform a Compact operation when the target HFile file is located on the local SSD, delete the original HFile file from the local SSD, and write the newly generated HFile file into the HDFS cluster and cache it in the local SSD.

Preferably, the system further includes:

The Split module is configured to perform a Split operation when the target HFile file is located on the local SSD, and delete the original HFile file from the local SSD.

Adopt above-mentioned technical scheme, the present invention can obtain following technical effect at least:

The technical scheme provided by the present invention introduces a local SSD to provide a read-only cache function, fully exerts the random read performance of the SSD, and caches related files in the SSD. Because the accessed data has locality, the files cached in the SSD can effectively reduce Read the I/O number of HDFS, thereby improving the performance of the distributed storage system and achieving the purpose of improving the efficiency of data search.

In the application of the above technical solution, in view of HBase using the LSM tree to organize data and the load of upper-level read data, the local SSD is used as the read-only cache function of the HBase cluster, making full use of the characteristics of HBase itself, especially suitable for upper-level remote sensing satellites, etc. The application has load characteristics of more reads and less writes for data access.

Description of drawings

Fig. 1 is the flow chart of the method for optimizing the NoSQL search cache of the distributed storage system provided by Embodiment 1;

FIG. 2 is a schematic diagram of the HRregionServer architecture of HBase after adding the distributed storage system after the local SSD provided by Embodiment 1;

Fig. 3 is the flow chart of the Compact operation method provided by embodiment two;

Fig. 4 is the flow chart of the Split operation method that embodiment three provides;

FIG. 5 is a system composition diagram of an optimized NoSQL search cache of a distributed storage system provided in Embodiment 4.

detailed description

In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the technical solutions of the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only the technical solutions of the present invention. Some, but not all, embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and through specific implementation methods. It should be noted that the present invention uses the following examples as examples to describe the technical solution of the present invention, but it is not limited thereto. Those skilled in the art can understand that the method and system for optimizing the NoSQL search cache of the distributed storage system proposed by the present invention can be widely used in other same or similar fields besides the distributed storage system, and achieve similar technical effect.

Under the cross-data center environment, the technical solution adopted by the underlying database system is the HBase distributed storage system. HBase is a storage system that uses LSM to organize the underlying data. Data access locality and features such as write once and read multiple times.

The technical terms related to the HBase storage system used in the embodiments are described below:

(1) HBase components: HBase storage system includes HMaster, HRegionServer and Client. Among them, HMaster is responsible for creating and deleting data tables in the HBase storage system. The HRegionServer stores the data of a specific table, including multiple HRegions of the table.

(2) HRegion: Each table is divided into multiple HRegions, each HRegion contains multiple Stores, each Store corresponds to a column family, and each Store contains a MemStore and multiple StoreFiles.

(3) Compact and Split: During the operation of the HBase system, multiple HFile files will be generated. When the number of HFile files exceeds a certain number, the HBase cluster will execute Compact. When the HFile file size exceeds a certain threshold, the Split operation is performed.

An important feature of the HBase storage system is that once the HFile file is written into the HDFS cluster, the HFile file will not change.

Embodiment one:

FIG. 1 is a flowchart of a method for optimizing a NoSQL search cache in a distributed storage system provided by this embodiment. Shown in Fig. 1 with reference to, this method comprises the steps:

Step S101, in the HBase client, pre-set the local SSD as a read-only cache;

Figure 2 is a schematic diagram of the architecture after setting the local SSD as a read-only cache in the HBase client (distributed storage system cluster). Among them, local SSD can be added to HRegionServer.

Step S102, when the HBase client reads the target data, judge whether the target data is located on the local SSD, if yes, then enter step S104; if no, then enter step S103;

Before this step, the HBase client generates a Get/Scan operation request according to the requirement. According to the target data of the operation, the HBase client sends the Get/Scan operation request to the corresponding HRegionServer, and the HRegionServer judges the target data according to the received read request. Whether it is located in the local memory, assuming that the HFile file needs to be read, locate the specific HFile file, and judge whether the database in the HFile file is in the local memory, if yes, return the target data from the local memory, if not , that is, the corresponding HFileBlock data block is not in the local memory, then enter step S102.

Step S103, read the target data from the HDFS cluster and cache it on the local SSD;

Wherein, before step S103, it may also include: judging whether the remaining storage space of the local SSD meets the preset requirements, if yes, then enter step S103; if not, then perform a replacement algorithm such as LRU (Least Recently Used, least recently used ) algorithm, replace the existing data in the local SSD with the read target data, such as replacing the old HFile file in the local SSD with the read HFile file.

Step S104, the local SSD returns the target data.

Correspondingly, the step S104 may specifically include: the local SSD returns the target data to the local memory, and the local memory returns the target data.

In addition, in the above embodiment, when data is written, it can be directly written to the underlying file system, and no cache is performed in the local SSD.

In a large-scale massive data environment, storing data through a NoSQL storage system requires low latency for fast lookup. At present, NoSQL systems represented by BigTable generally use LSM trees to organize semi-structured data, which is used to write once and read multiple times.

The technical solution provided by this embodiment introduces a local SSD to provide a read-only cache function, fully utilizes the random read performance of the SSD, and caches related files in the SSD. Because the accessed data has locality, the files cached in the SSD can be effectively Reduce the number of I/Os for reading HDFS, thereby improving the performance of the distributed storage system and achieving the purpose of improving data search efficiency.

In the application of the above technical solution, in view of HBase using the LSM tree to organize data and the load of upper-level read data, the local SSD is used as the read-only cache function of the HBase cluster, making full use of the characteristics of HBase itself, especially suitable for upper-level remote sensing satellites, etc. The application has load characteristics of more reads and less writes for data access.

Embodiment two:

During the operation of the HBase storage system, multiple HFile files will be generated. When the number of HFile files exceeds a certain number, the HBase cluster needs to perform the Compact operation to reduce the number of HFile files. This embodiment provides a method for performing a Compact operation in an HBase cluster on the basis of the method for optimizing the NoSQL search cache of the distributed storage system in Embodiment 1. Figure 3 is a schematic flow diagram of the method, which specifically includes the following steps :

Step S301, the HBase client generates a Compact operation request;

Step S302, judging whether the target HFile file corresponding to the Compact operation request is located on the local SSD, if yes, then enter step S304; if no, then enter step S303;

Step S303, read the target HFile file from the HDFS cluster and cache it on the local SSD;

Step S304, performing a Compact operation;

Step S305, after the Compact operation is completed, the original HFile file is deleted from the local SSD;

Step S306, write the newly generated HFile into the HDFS cluster and cache it in the local SSD.

Through the method provided in this embodiment, when the number of HFiles exceeds a certain number, the Compact operation can be performed to reduce the number of HFiles.

Embodiment three:

During the operation of the HBase storage system, multiple HFile files are generated. When the size of the HFile file exceeds a certain threshold, the HBase cluster needs to perform the Split operation to reduce the size of the HFile file. On the basis of the optimization method for the NoSQL search cache of the distributed storage system in Embodiment 1, this embodiment provides a method for executing the Split operation in the HBase cluster, as shown in Figure 4, which is a schematic flow diagram of the method, specifically including the following steps :

Step S401, the HBase client generates a Split operation request;

Step S402, judging whether the target HFile file corresponding to the Split operation request is located on the local SSD, if yes, then enter step S304; if no, then enter step S303;

Step S403, read the target HFile file from the HDFS cluster and cache it on the local SSD;

Step S404, performing a Split operation;

Step S405, after the Split operation is completed, the original HFile file is deleted from the local SSD.

Through the method provided in this embodiment, when the size of the HFile exceeds a certain threshold, the Split operation can be performed to reduce the size of the HFile.

Embodiment four:

Corresponding to the above method, this embodiment also provides a NoSQL search cache optimization system for a distributed storage system, as shown in Figure 5, a schematic diagram of the system architecture, including:

Local SSD501, used as a read-only cache set in the HBase client;

The first judging module 502 is used to judge whether the target data is located on the local SSD when the HBase client reads the target data;

Data reading module 503, for when the target data is not located on the local SSD, read the target data from the HDFS cluster and cache it on the local SSD;

A data return module 504, configured to return the target data from the local SSD when the target data is located on the local SSD.

In addition, the system may also include:

A second judging module, configured to judge whether the remaining storage space of the local SSD meets a preset requirement;

A replacement module, configured to execute a replacement algorithm such as LRU (Least Recently Used) when the remaining storage space of the local SSD does not meet the preset requirements, and replace the data in the local SSD with the target data read. Existing data, such as replacing the old HFile file in the local SSD with the read HFile file.

During the operation of the HBase storage system, multiple HFile files will be generated. When the number of HFile files exceeds a certain number, the HBase cluster needs to perform the Compact operation to reduce the number of HFile files. Therefore, the system may also include:

The Compact module is configured to perform a Compact operation when the target HFile file is located on the local SSD, delete the original HFile file from the local SSD, and write the newly generated HFile file into the HDFS cluster and cache it in the local SSD.

When the size of the HFile file exceeds a certain threshold, the HBase cluster needs to perform the Split operation to reduce the size of the HFile file. Therefore described system can also comprise:

The Split module is configured to perform a Split operation when the target HFile file is located on the local SSD, and delete the original HFile file from the local SSD.

The technical solution provided by this embodiment introduces a local SSD to provide a read-only cache function, fully utilizes the random read performance of the SSD, and caches related files in the SSD. Because the accessed data has locality, the files cached in the SSD can be effectively Reduce the number of I/Os for reading HDFS, thereby improving the performance of the distributed storage system and achieving the purpose of improving data search efficiency.

In the application of the above technical solution, in view of HBase using the LSM tree to organize data and the load of upper-level read data, the local SSD is used as the read-only cache function of the HBase cluster, making full use of the characteristics of HBase itself, especially suitable for upper-level remote sensing satellites, etc. The application has load characteristics of more reads and less writes for data access.

The present invention is on software, and the operating system is preferably a Linux system, running on the software that provides file IO services in the Linux cluster, such as distributed file systems such as HDFS and GFS and NoSql distributed database systems such as HBase, and HDFS distributed files The system configures multiple DataNodes.

All or part of the content in the technical solutions provided by the above embodiments can be realized by software programming or special hardware equipment, wherein the software program is stored in a readable storage medium, the storage medium is for example: a hard disk, an optical disk or a floppy disk in a computer; The hardware device can be ASIC, FPGA, SoC, or IP Core with corresponding circuits.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the present invention The scope is determined by the scope of the appended claims.

Claims (10)

1. an optimization method for distributed storage system NoSQL search cache, characterized in that, comprising: Step S101, in the HBase client, pre-set the local SSD as a read-only cache; Step S102, when the HBase client reads the target data, judge whether the target data is located on the local SSD, if yes, then enter step S104; if no, then enter step S103; Step S103, read the target data from the HDFS cluster and cache it on the local SSD; Step S104, the local SSD returns the target data. 2. The method according to claim 1, characterized in that, before the step S102, further comprising: The HBase client sends the read request to the corresponding HRegionServe; The HRegionServer judges whether the target data is located in the local memory according to the received read request, if yes, returns the target data from the local memory, and if not, proceeds to step S102. 3. The method according to claim 2, characterized in that the step S104 comprises: The local SSD returns the target data to the local memory, and the local memory returns the target data. 4. The method according to claim 1, characterized in that, before the step S103, further comprising: Judging whether the remaining storage space of the local SSD meets the preset requirements, if yes, proceed to step S103; if not, execute a replacement algorithm to replace existing data in the local SSD with the read target data. 5. The method of claim 1, further comprising: The HBase client generates a Compact operation request, judges whether the target HFile file corresponding to the Compact operation request is located on the local SSD, if yes, then executes the Compact operation; if not, then reads the target HFile file cache from the HDFS cluster to the described The Compact operation is performed on the local SSD. After the Compact operation is completed, the original HFile file is deleted from the local SSD, and the newly generated HFile file is written to the HDFS cluster and cached in the local SSD. 6. The method of claim 1, further comprising: The HBase client generates a Split operation request, judges whether the target HFile file corresponding to the Split operation request is located on the local SSD, if yes, then executes the Split operation; if not, then reads the target HFile file cache from the HDFS cluster to the described The split operation is performed on the local SSD. After the split operation is completed, the original HFile file is deleted from the local SSD. 7. An optimization system for distributed storage system NoSQL search cache, characterized in that, comprising: Local SSD for setting in HBase client as read-only cache; The first judging module is used to judge whether the target data is located on the local SSD when the HBase client reads the target data; A data reading module, for when the target data is not located on the local SSD, read the target data from the HDFS cluster and cache it on the local SSD; A data returning module, configured to return the target data from the local SSD when the target data is located on the local SSD. 8. The system of claim 7, further comprising: A second judging module, configured to judge whether the remaining storage space of the local SSD meets a preset requirement; The replacement module is configured to execute a replacement algorithm when the remaining storage space of the local SSD does not meet the preset requirements, and replace the existing data in the local SSD with the read target data. 9. The system of claim 7, further comprising: The Compact module is configured to perform a Compact operation when the target HFile file is located on the local SSD, delete the original HFile file from the local SSD, and write the newly generated HFile file into the HDFS cluster and cache it in the local SSD. 10. The system of claim 7, further comprising: The Split module is configured to perform a Split operation when the target HFile file is located on the local SSD, and delete the original HFile file from the local SSD.