CN105183839A - Hadoop-based storage optimizing method for small file hierachical indexing - Google Patents

Abstract

The invention discloses a hadoop-based storage optimization method for hierarchical indexing of small files, capable of uploading a large amount of small files to an HDFS file system and improving the indexing efficiency of small files stored in the HDFS system. It mainly includes the following steps: (1) WebServer listens to file read and write requests, and judges whether the size of the file is smaller than the block size set by the HDFS file system; (2) files larger than the block size are read or written according to normal HDFS files, and if the file size is smaller than The file of block size is divided into small files, and is sent to the small file processing server for processing; (3) inside the small file processing server, a judging module is provided to set the small files of 1～1023KB as K-level small files, Set the small files from 1M to 64M as M-level small files; (4) There is also a cache in the small file processing server, which can prefetch some small files when the files are separated according to the file correlation, and design a cache update strategy to Improve file access efficiency.

Description

A storage optimization method for hierarchical indexing of small files based on Hadoop

technical field

The invention belongs to the technical field of cloud storage, and more specifically relates to a Hadoop-based storage optimization method for hierarchical indexing of small files.

Background technique

Hadoop is a distributed system infrastructure developed by the Apache Foundation. Hadoop is the current mainstream cloud storage platform. It consists of a NameNode and multiple DataNodes. The NameNode is responsible for managing the file system namespace and controlling the access of external clients, and the DataNode is responsible for the storage of specific data. Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system, HDFS for short. HDFS is highly fault-tolerant and designed to be deployed on inexpensive hardware. And it provides a high transfer rate to access application data, suitable for those applications with very large data sets.

Hadoop is designed to handle large files. It stores incoming large files on different data blocks of different datanodes, and records the metadata of large files on the namenode. When operating on a file, first visit the namenode, locate each block according to the feedback information from the namenode, find the file, and then use the corresponding command to operate on it. It should be noted that these operations are performed in blocks. For small files, Hadoop is no longer so efficient, because the amount of data contained in each independent small file is very small, and can only be stored or operated as a block (far smaller than the default size of the data block), and a large number of files means The metadata is also massive. How to effectively store and manage a large number of small files has become an urgent problem to be solved.

In the HDFS distributed file system, any file information and file block information are stored in the form of an object in the memory of the NameNode (master node). Each object occupies about 150 bytes. A large number of small files make the memory usage of the NameNode Severely restricts the expansion of the cluster.

HDFS access to a large number of small files is much slower than accessing large files of the same size. It is primarily designed for streaming access to large files. Reading small files usually results in a large number of fetched files from Datanode to Datanode, which is a very inefficient access method.

MapReduce (parallel data processing) takes much longer to process batches of small files than to process large files of the same size. Processing each small file takes up a task (task). If there are too many small files, most of the processing time will be spent on starting and releasing tasks. The Map function usually processes a block-sized input. If there are a lot of small files, a large number of Map functions will be generated for execution, and these Map functions will undoubtedly occupy a large amount of memory.

For the problems caused by a large number of small files, Hadoop technology itself provides three solutions, namely HadoopArchive (Hadoop archive), Sequencefile (sequence file) and CombineFileInputFormat (combined file input format).

HadoopArchive packs small files into HAR files. First, a main index structure is established. The main index points to the secondary index, and the secondary index points to specific small files. Therefore, accessing each HAR file will require two layers of index file reading and file data reading, so actually reading a file through HAR may be less efficient than reading a file directly from HDFS. The three tools HadoopArchive, Sequencefile and CombineFileInputFormat cannot be automated and need to be manually added by the administrator. The efficiency is very low and it is not suitable for a large number of file operations.

Contents of the invention

The main purpose of the present invention is to solve the problem of low efficiency of the existing Hadoop distributed file system for massive small file storage, reading and writing operations, and proposes a storage optimization method based on hadoop-based small file hierarchical indexing.

In order to achieve the above object, the present invention provides a storage optimization method based on Hadoop's small file hierarchical index, in addition to the Hadoop distributed file system HDFS, add a web server WebServer for file read and write requests, add A small file processing server for processing small files, the method includes the following steps:

(1) The web server WebServer receives a file read or write request, and judges whether the file is larger than the preset block size. Files larger than the preset block size are processed according to the normal process of the HDFS file system, and files smaller than the preset block size are sent to Go to the small file processing server for subsequent processing;

(2) The small file processing server further divides the small files, divides the small files between 1～1023KB into K-level small files, and divides the small files between 1M and the preset block size into M-level small files;

(3) Establish a secondary index for K-level small files, wherein the primary index is stored in the small file processing server, the secondary index is stored in the datanode, and a linear index structure is established for M-level small files and stored in the small file processing server middle;

(4) The cache area of the small file processing server saves some accessed small files and their metadata information, and updates the small files and their metadata in the cache area according to the access frequency of the small files.

Further, the web server WebServer is a web server that monitors file read and write requests, and the small file processing server is a server for small file processing.

Further, the small file size is smaller than the block size on the distributed file system.

Further, the small file processing server further divides the small files, and the steps are as follows:

1) After the small file processing server receives the operation of reading or writing small files, the small file classification module divides small files from 1 to 1023KB into K grades, and divides small files between 1M and block size into M grades;

2) The K-level small files adopt the secondary index. The primary index is divided into ranges, with 1000 records as a range, and stored in the small file processing server. The secondary index is to actually record the small files in the merged large file. Offset index files are stored in the datanode; M-level small files adopt a linear index structure and are stored in the small file processing server.

Further, the cache in the small file processing server stores related files and corresponding metadata information of some accessed files, and updates the small files and corresponding metadata information stored in the cache area according to the corresponding updating strategy.

Further, for file write requests, the processing process is as follows:

(1.1) Web server WebServer judges that the request type of file is a file write request;

(1.2) Web server WebServer judges whether the file is greater than the preset block size, and the file larger than the preset block size is processed according to the normal process of the HDFS file system, and the file smaller than the preset block size is sent to the small file processing server for subsequent processing;

(1.3) The small file processing server further divides the small files, divides the small files between 1～1023KB into K-level small files, and divides the small files between 1M and the preset block size into M-level small files;

(1.4) Establish a secondary index for K-level small files, wherein the primary index is stored in the small file processing server, the secondary index is stored in the datanode, and a linear index structure is established for M-level small files and stored in the small file processing server middle.

Further, for file read requests, the processing process is as follows:

(2.1) The small file processing server receives a read request for a small file, first checks whether there is a small file to be read in the cache, if it exists, the small file processing server takes out the read request file in the cache and returns it to the client, and completes the Take the operation, otherwise, perform step (2.2);

(2.2) The small file processing server checks whether the metadata information of the read requested small file exists in the cache. If it exists, the small file processing server will directly interact with the HDFS client, take the small file from HDFS and return it to the user, and complete the reading operation, otherwise perform step (2.3);

(2.3) According to the file name of the small file and the combined file, the small file processing server maps the received small file to the combined file of the small file, and sends the combined file to the client of the Hadoop distributed file system HDFS end;

(2.4) The client of the Hadoop distributed file system HDFS reads the merged file received from the Hadoop distributed file system HDFS to obtain the metadata information and index information of the merged file, thereby completing the reading operate.

Further, the small file processing server adopts a small file separation method to read the merged file from the Hadoop distributed file system HDFS, and separate the requested small file from the merged file and return it to the user.

Further, the method further includes: the small file processing server extracts the file name, data block position, offset and file length of each small file from the metadata information and index information of the merged file obtained in step (2.4) , get the metadata information of the small files, and store each small file and its corresponding metadata information in the cache.

Further, the method further includes adding a 32-bit file access identifier for recording file access frequency to the metadata prefetch record of the small file and the header of the small file prefetch record respectively, with five minutes as the timing standard , if there is a request for accessing a small file within five minutes, add 1 to the file access ID, otherwise, decrement the file access ID by 1, and if the file access ID is 0, remove the small file from the cache.

Compared with prior art, the beneficial effect of the present invention is:

(1) A large number of small files can be uploaded to the HDFS file system and fast indexing of small files can be realized. The small file processing server independent of the HDFS file system can effectively complete the small file processing by establishing a corresponding small file index and small file processing mechanism Merge, map, and prefetch, which can greatly reduce the overhead on the namenode, effectively improve the efficiency of the system to process small files, and will not change the existing HDFS system, ensuring the versatility of the system; in small file processing The server can save the storage space of the index files by further dividing the small files and establishing appropriate indexes in different ways; the small file processing server caches the related files and corresponding metadata information of some accessed files, and formulates Update strategies to improve access efficiency

(2) Merge a large number of small files into a limited number of large files, and further divide the indexing method of small files, reducing the memory occupancy rate of the NameNode node in the Hadoop cluster;

(3) Encapsulate the small file management function into a service, which reduces the difficulty of use for users;

(4) The independent small file processing server will not affect the normal operation of the HDFS file system, which improves the file access efficiency.

Description of drawings

Fig. 1 is a flow chart of the storage optimization method of a Hadoop-based small file hierarchical index of the present invention;

Fig. 2 is a schematic diagram of the small file storage process of the present invention;

Fig. 3 is a schematic diagram of a small file reading process of the present invention;

FIG. 4 is a schematic diagram of an index structure of a K-level index file.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

As shown in Figure 1, for the storage optimization method of a kind of Hadoop-based small file classification index of the present invention, outside Hadoop distributed file system HDFS, add a web server WebServer that is used for file read and write request, add A small file processing server for processing small files, the steps are as follows:

(1) The web server WebServer receives a file read or write request, and judges whether the file is larger than the preset block size. Files larger than the preset block size are processed according to the normal process of the HDFS file system, and files smaller than the preset block size are sent to Go to the small file processing server for subsequent processing;

(2) The small file processing server further divides the small files, divides the small files between 1～1023KB into K-level small files, and divides the small files between 1M and the preset block size into M-level small files;

(3) Establish a secondary index for K-level small files, wherein the primary index is stored in the small file processing server, the secondary index is stored in the datanode, and a linear index structure is established for M-level small files and stored in the small file processing server In order to reduce the overhead of small file processing server;

(4) The cache area of the small file processing server saves some accessed small files and their metadata information, and updates the small files and their metadata in the cache area according to the access frequency of the small files.

As for the secondary index, the primary index is a range index, and the secondary index is an index file that actually records the offset of the small file in the merged large file. The index structure is shown in FIG. 4 . Whenever the secondary index file records more than 1000 records (the default setting is 1000), record a range to the primary index, including the index value of the first record of these 1000 records, the index value of the last record, and the first record The position of the index value of the index in the secondary index file, and the end position of the index value of the last item in the secondary index file. The information recorded in the secondary index includes the index value of the file name, the relative path name of the file, whether it is a directory or a file, the position in the merged file, the length of the file, and the merged file name.

The small file processing server is mainly responsible for merging, mapping and prefetching of small files. Its function has three aspects:

(1) When writing a file, it builds an index according to the size of the small file. Small files from 1 to 1023 KB use a linear index structure, and small files between 1M and block size use a secondary index structure to merge files;

(2) When reading a file, it completes the mapping process from the small file to its merged file, and separates the required small file from the large file according to the index file;

(3) The small file processing server also has a cache area, which can prefetch some small files when the files are separated according to file dependencies, and design a corresponding cache update strategy to improve file access efficiency.

For the HDFS part of the present invention, at the NameNode layer, we maintain the relationship between the original NameNode and SecondaryNameNode, and its function is the same as that of the NameNode in the original HDFS. At the DataNode layer, we divide the DataNode into two cloud storage areas. Add the secondary index file of K-level small files to the head of the DataNode, and the other parts are still used as data storage areas.

For an independent small file processing server, it is only responsible for processing small files, further subdividing small files, caching related files and corresponding metadata information of some accessed files, and formulating update strategies.

Under the framework of the method of the present invention, large files and processed small files are all waiting to be written or read in the file waiting queue, which leads to the concept of concurrent computer execution, so that when processing small files, they can be merged, mapped or read At the same time, it does not affect the writing or reading of large files or small files that have been merged to improve the efficiency of concurrent storage access.

Fig. 2 is the writing flowchart of the storage optimization method of the small file hierarchical index based on hadoop of the present invention, specifically, as Fig. 2, described method comprises the judgment of small file, classification and writing: network server WebServer receives file and writes Request, and determine whether the file is larger than the preset block size, the file larger than the preset block size is processed according to the normal process of the HDFS file system, and the file smaller than the preset block size is sent to the small file processing server for subsequent processing;

Specifically, the writing process includes:

(1.1) Web server WebServer judges the request type of file, and in this step, the request type of detection should be file writing request;

(1.2) The network server WebServer judges whether the file is larger than the preset block size, and the files larger than the preset block size are processed according to the normal process of the HDFS file system, and the files smaller than the preset block size are sent to the small file processing server for subsequent processing. In the step, the requested file size is smaller than the preset block size;

(1.3) The small file processing server further divides the small files, divides the small files between 1～1023KB into K-level small files, and divides the small files between 1M and the preset block size into M-level small files;

(1.4) Establish a secondary index for K-level small files, wherein the primary index is stored in the small file processing server, the secondary index is stored in the datanode, and a linear index structure is established for M-level small files and stored in the small file processing server In order to reduce the overhead of small file processing server;

Fig. 3 is the reading flowchart of the storage optimization method of the small file hierarchical index based on hadoop in the present invention, concretely, as Fig. 3, described method comprises the judgment of small file, classification and reading: network server WebServer receives file and reads Request, and determine whether the file is larger than the preset block size, the file larger than the preset block size is processed according to the normal flow of the HDFS file system, and the file read request smaller than the preset block size is sent to the small file processing server for subsequent processing;

Specifically, the reading process includes:

(2.1) When the small file processing server receives a read request for a small file, it will first check whether there is a small file that needs to be read in the cache. If it exists, the small file processing server will take out the read request file in the cache and return it to the client. Read operation, otherwise, perform step (2.2);

(2.2) The small file processing server will check whether the metadata information of the read requested small file exists in the cache. If it exists, the small file processing server will directly interact with the HDFS client, take the small file from HDFS and return it to the user, and complete the read Take the operation, otherwise, perform step (2.3);

(2.3) According to the file name of the small file and the combined file, the small file processing server maps the received small file to the combined file of the small file, and sends the combined file to the client of the Hadoop distributed file system HDFS end;

(2.4) The client of the Hadoop distributed file system HDFS reads the merged file received from the Hadoop distributed file system HDFS to obtain the metadata information and index information of the merged file, thereby completing the reading operate;

Specifically, the small file processing server adopts a small file separation method to read the combined file from the Hadoop distributed file system HDFS, and separate the small file requested to be read from the combined file and return it to the user;

The small file processing server extracts the file name, data block position, offset and file length of each small file from the metadata information and index information of the merged file obtained in step (2.4), and obtains the metadata information of the small file, And store each small file and its corresponding metadata information into the cache.

Add a 32-bit file access identifier for recording the file access frequency to the header of the metadata prefetch record and the small file prefetch record of the small file respectively, with five minutes as the timing standard, if there is an access within five minutes For a small file request, add 1 to the file access ID, otherwise, subtract 1 from the file access ID, and if the file access ID is 0, remove the small file from the cache.

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (10)

1. a storage optimization method based on Hadoop's small file hierarchical index, it is characterized in that, outside Hadoop distributed file system HDFS, add a network server WebServer that is used for file read and write request, add a new one with A small file processing server for processing small files, the method includes the following steps: (1) The web server WebServer receives a file read or write request, and judges whether the file is larger than the preset block size. Files larger than the preset block size are processed according to the normal process of the HDFS file system, and files smaller than the preset block size are sent to Go to the small file processing server for subsequent processing; (2) The small file processing server further divides the small files, divides the small files between 1～1023KB into K-level small files, and divides the small files between 1M and the preset block size into M-level small files; (3) Establish a secondary index for K-level small files, wherein the primary index is stored in the small file processing server, the secondary index is stored in the datanode, and a linear index structure is established for M-level small files and stored in the small file processing server middle; (4) The cache area of the small file processing server saves some accessed small files and their metadata information, and updates the small files and their metadata in the cache area according to the access frequency of the small files. 2. the storage optimization method of the small file hierarchical index based on hadoop according to claim 1, is characterized in that, described network server WebServer is a network server that monitors file read and write requests, and small file processing server is a A server for small file processing. 3. The storage optimization method of hadoop-based small file hierarchical indexing according to claim 1 or 2, wherein the size of the small file is smaller than the size of the block on the distributed file system. 4. according to claim 1 and 2 described storage optimization methods based on the small file hierarchical index of hadoop, it is characterized in that, described small file processing server divides small file further, and the steps are as follows: 1) After the small file processing server receives the operation of reading or writing small files, the small file classification module divides small files from 1 to 1023KB into K grades, and divides small files between 1M and block size into M grades; 2) The K-level small files adopt the secondary index. The primary index is divided into ranges, with 1000 records as a range, and stored in the small file processing server. The secondary index is to actually record the small files in the merged large file. Offset index files are stored in the datanode; M-level small files adopt a linear index structure and are stored in the small file processing server. 5. according to claim 1 or 2 described storage optimization methods based on the small file hierarchical index of hadoop, it is characterized in that, the relevant file and the corresponding metadata information of the cache storage part of the accessed file in the small file processing server, and The small files stored in the cache area and the corresponding metadata information are updated according to the corresponding updating strategy. 6. according to claim 1 and 2 described storage optimization methods based on the small file hierarchical index of hadoop, it is characterized in that, for file writing request, its process is as follows: (1.1) Web server WebServer judges that the request type of file is a file write request; (1.2) Web server WebServer judges whether the file is greater than the preset block size, and the file larger than the preset block size is processed according to the normal process of the HDFS file system, and the file smaller than the preset block size is sent to the small file processing server for subsequent processing; (1.3) The small file processing server further divides the small files, divides the small files between 1～1023KB into K-level small files, and divides the small files between 1M and the preset block size into M-level small files; (1.4) Establish a secondary index for K-level small files, wherein the primary index is stored in the small file processing server, the secondary index is stored in the datanode, and a linear index structure is established for M-level small files and stored in the small file processing server middle. 7. according to claim 1 and 2 described storage optimization methods based on the small file hierarchical index of hadoop, it is characterized in that, for file reading request, its processing procedure is as follows: (2.1) The small file processing server receives a read request for a small file, first checks whether there is a small file to be read in the cache, if it exists, the small file processing server takes out the read request file in the cache and returns it to the client, and completes the Take the operation, otherwise, perform step (2.2); (2.2) The small file processing server checks whether the metadata information of the read requested small file exists in the cache. If it exists, the small file processing server will directly interact with the HDFS client, take the small file from HDFS and return it to the user, and complete the reading operation, otherwise perform step (2.3); (2.3) According to the file name of the small file and the combined file, the small file processing server maps the received small file to the combined file of the small file, and sends the combined file to the client of the Hadoop distributed file system HDFS end; (2.4) The client of the Hadoop distributed file system HDFS reads the merged file received from the Hadoop distributed file system HDFS to obtain the metadata information and index information of the merged file, thereby completing the reading operate. 8. the storage optimization method of the small file hierarchical index based on hadoop according to claim 7, is characterized in that, the small file processing server adopts the small file separation method, reads the merged file from the Hadoop distributed file system HDFS, and requests The read small files are separated from the merged file and returned to the user. 9. the storage optimization method of the small file hierarchical index based on hadoop according to claim 7, is characterized in that, also comprises: in the metadata information and the index information of the merged file that the small file processing server obtains from step (2.4), Extract the file name, data block location, offset and file length of each small file, obtain the metadata information of the small file, and store each small file and its corresponding metadata information in the cache. 10. The storage optimization method of hadoop-based small file hierarchical indexing according to claim 1 or 2, characterized in that the method further comprises, at the head of the metadata prefetch record and the small file prefetch record of the small file , respectively add a 32-bit file access identifier for recording file access frequency, with five minutes as the timing standard, if there is a request to access a small file within five minutes, add 1 to the file access identifier, otherwise, delete the file The access identifier is decremented by 1, and if the file access identifier is 0, the small file is removed from the cache.