CN108268208B - A Distributed Memory File System Based on RDMA - Google Patents

Abstract

The invention discloses an RDMA-based distributed memory file system. In the initialization stage of the distributed memory file system, the memory used by the cluster for file storage is uniformly divided and registered to a network card to support the remote node for direct memory access. Then, a distributed shared memory pool is constructed; on the distributed memory shared pool, file indexing and file data block indexing are carried out respectively through two-level hash indexes to provide query services for the file system; processing is performed through a self-identifying remote procedure call method The client's request, and returns the processing result. The present invention has the following advantages: reducing data duplication during file reading and writing, reducing response delay, and improving the overall efficiency of file access in program software.

Description

A Distributed Memory File System Based on RDMA

technical field

The invention relates to the field of distributed storage systems, in particular to an RDMA-based distributed memory file system.

Background technique

Remote Direct Memory Access (RDMA) refers to direct access to remote memory without the direct participation of both host operating systems, thereby providing high bandwidth and low latency.

Data transmission in a distributed environment determines the overall I/O performance of the system, and this type of technology is widely used in distributed file systems and database systems. Most traditional distributed systems use disks as storage media, and transmit data through TCP/IP-based remote procedure call modules. Due to low disk bandwidth and high latency, the network transmission module itself will not become a bottleneck. In recent years, memory has become increasingly Cheap, in-memory computing that transfers storage and computing to memory has become a trend. While the performance of storage media is improving, network transmission also faces great challenges.

At present, there is no general data transmission module to efficiently handle different network I/O characteristics. At the same time, in terms of concurrency control, a centralized synchronization model is still used, which seriously affects the scalability of the system.

SUMMARY OF THE INVENTION

The present invention aims to solve at least one of the above-mentioned technical problems.

Therefore, the purpose of the present invention is to propose a distributed memory file system based on RDMA, so as to reduce the data duplication during file reading and writing, reduce the response delay, and improve the overall efficiency of file access in the program software.

In order to achieve the above purpose, an embodiment of the present invention discloses an RDMA-based distributed memory file system, the distributed memory file system interconnects the memory of each node through RDMA, and the file system includes a client and a server , the client provides a file access interface for upper-layer applications to call, the server provides metadata services and data services, and the distributed memory file system performs the following actions: S1: In the initialization phase of the distributed memory file system , uniformly divide the memory used by the cluster for file storage, and register it to the network card to support direct memory access by remote nodes, and then build a distributed shared memory pool; S2: On the distributed memory shared pool, through the two-level hash The wish index performs file indexing and file data block indexing, respectively, to provide query services for the file system; S3: Process the client's request through a self-identifying remote procedure call method, and return a processing result.

According to the RDMA-based distributed memory file system according to the embodiment of the present invention, data duplication during file reading and writing is reduced, response delay is reduced, and the overall efficiency of file access in program software is improved.

In addition, the RDMA-based distributed memory file system according to the foregoing embodiments of the present invention may also have the following additional technical features:

Further, the distributed shared memory pool sequentially stores a super block, a message pool, a chain hash index table, a metadata storage block and a data storage block.

Further, in the data layout area, the direct access of the remote node is received; in the chain hash index table and the metadata storage area, the service node responds to the concurrent request of the client to perform query and update on the metadata; The data is distributed to the entire cluster according to the file path name hash, and each node maintains the metadata and data of the file independently.

Further, the super block is used to store the number of metadata blocks, the size of the metadata block, the number of data blocks and the size of the data block, and the super block is remotely read by each node when the file system system is started.

Further, the message pool includes a plurality of message areas, and the plurality of message areas are allocated to different clients connected to the system, so that when a client has a new request, the client remotely writes the new request to the service In the message area to which the node belongs, after the server receiving thread detects the new request, it uses the self-identification method to quickly locate the message and process the return.

Further, the chained hash index table is used to calculate the hash value of the full path name of the file when querying the file metadata; the hash value is used as the index number of the index table, and the entry under the index number is queried. , perform file name matching, if successful, obtain the metadata address and access the metadata according to the address; if the file name does not match, continue to find the next entry until the match is successful.

Further, the metadata storage block and the data storage block are respectively used to store metadata and data, the metadata area and the data area are divided into fixed-size metadata blocks and data blocks, The data area and the header of the data area store free block metadata of the corresponding area to describe the memory usage.

Further, the step of the two-level hash index includes: when the client initiates a file access request, calculating the metadata server ID storing the file metadata according to the full path name of the file, wherein the metadata server ID is determined by the system configuration file. The client sends the request information to the metadata server corresponding to the ID. After the metadata server detects the new message, it parses out the request content, and performs a second hash value calculation according to the file path name in the request content. The second hash value accesses the chain hash index table, obtains metadata information, performs corresponding logical processing, and returns the request result.

Further, the self-identifying remote procedure calling method includes: when the client sends a message to the server, using the RDMA_WRITE_WITH_IMM primitive to carry the message content, and storing client metadata in the message header; When the server returns the request result, the returned result is directly written back to the memory area designated by the client through the RDMA primitive, and the client will poll the memory area used to store the returned result until the data is returned successfully.

Further, the client stores its own ID and timestamp in the message header, and the client ID is allocated by the server master node when the connection is established, and is globally unique.

Further, after the RDMA_WRITE_WITH_IMM message is successfully delivered, the server obtains the client metadata in the message header according to the completion information, parses out the client ID, and directly queries the fixed offset position of the local message pool according to the client ID, and obtains the New request information.

Further, when the number of clients exceeds the number of message pools allocated in advance by the server, the server queries the disconnected client and transfers the message area occupied by it to the current client; if all clients If both remain connected, you need to re-apply for the message area, register with the network card, and notify the current client.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a flowchart of an execution action of an RDMA-based distributed memory file system according to an embodiment of the present invention;

2 is a schematic diagram of RDMA data transmission according to an embodiment of the present invention;

3 is a layout diagram of a shared memory of a service node according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the statistics of the number of copies during data transmission and reception according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a self-identifying remote procedure call according to an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the present invention, and should not be construed as a limitation of the present invention.

These and other aspects of embodiments of the present invention will become apparent with reference to the following description and accompanying drawings. In these descriptions and drawings, some specific implementations of the embodiments of the invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the invention, but it should be understood that the scope of the embodiments of the invention is not limited by this limit. On the contrary, embodiments of the present invention include all changes, modifications and equivalents falling within the spirit and scope of the appended claims.

The present invention is described below with reference to the accompanying drawings. Before introducing the embodiments of the present invention, the terminology used in the present invention is explained.

Direct Memory Access (DMA) allows some hardware devices to directly read and write memory independently, without the need for a lot of CPU participation. This technology is used to relieve the processing pressure of the CPU on peripherals, and the entire data transfer process only requires the CPU Initialize the transfer operation at the very beginning, and then hand over the entire transfer operation to the DMA controller for execution.

Remote Direct Memory Access (RDMA) is a new type of network communication technology, which can directly access the remote memory without the direct participation of the two operating systems, and achieve the characteristics of high throughput and low latency. RDMA achieves zero-copy data transfer by allowing network adapters to transfer data directly to each other's memory, thereby eliminating the direct involvement of the CPU and Cache and reducing redundant on-site switching. The network protocol stacks currently supporting RDMA technology include Infiniband, RoCE (RDMA over Converged Ethernet) and iWARP. The first two are supported by Mellanox hardware technology. In particular, the latter two use the data link layer of ordinary Ethernet, so All are fully compatible with Ethernet. Figure 2 shows the specific process of RDMA communication: first, the local CPU initiates a communication command to the network card in the form of MMIO. After the local network card detects the new command, it reads the data to be transmitted from the memory in the form of DMA, and packs the data , and perform data transmission on the RDMA network. After the other party's network card receives the data, it directly writes the data to the corresponding address area of the memory by DMA, and writes the corresponding completion information to the completion queue. The whole process does not involve the other party's CPU. Participate in and bypass the kernels of both parties to achieve zero-copy data transfer. Before establishing communication, both parties need to go through the following steps: open the network card device; create a protection domain, which will be bound to the object created in the later stage to ensure the security of data transmission, any cross-domain operation will cause communication errors; Register the memory, register the communication memory at this stage. The specific method is to establish the mapping between the user mode address and the memory address of the memory segment, save the mapping table to the network card cache, and generate the key pair (lkey and rkey) of the memory segment. ), the network card needs to carry the corresponding key for identity confirmation when accessing the memory locally or remotely; to create a CQ (Completion Queue), the sender will put the corresponding completion information after the message is sent successfully or the receiver receives the message successfully. Completion queue, the user can repeatedly check the completion queue to verify whether the message sending is completed; create QP (Queue Pair), QP can peer to TCP/IP socket, QP consists of SendQueue and Receive Queue, the sender puts the message to be sent into Sending queue, at the same time, the receiver puts the receiving request into the receiving queue, and the two parties conduct network communication in this way; the QP state is initialized. After the two parties create a one-to-one corresponding QP, they need to perform a series of handshake state transitions until they are successfully established. communication link. QP can establish different connection types, including RC (Reliable Connection), UC (Unreliable Connection) and UD (Unreliable Datagram). In RC mode, QP can only perform one-to-one reliable transmission, and there will be a corresponding confirmation after the packet is sent successfully Information feedback. In UC mode, QP performs one-to-one transmission without confirmation information feedback. In UD mode, there is no one-to-one definition and no confirmation information feedback. The above three transmission modes have different characteristics. The level of support also varies.

In-memory computing means that in the face of massive data and high real-time processing requirements, traditional storage systems using disks as storage media are difficult to deal with new challenges due to slow access speeds, so the storage system is transferred to memory for real-time processing. A new processing mode for processing. There are two main types of in-memory storage systems: in-memory database systems and in-memory file systems. The invention reconstructs the memory file system in combination with RDMA network communication. At present, the mainstream memory file systems include Alluxio, IGFS, etc. Alluxio is mainly used to solve the existing problems of the Spark computing framework, accelerate data processing performance, and use lineage to achieve single-copy storage and reliable recovery of data. IGFS is a cache file system between the computing framework and HDFS. It provides an interface compatible with HDFS to the upper layer. However, unlike HDFS, IGFS does not have a separate metadata server, but uses a hash method. data distribution.

Remote Procedure Call (RPC) is a remote communication protocol that enables a program running on one computer to remotely call a function on another computer without the user needing to care about the underlying communication interaction strategy. Remote procedure call is widely used in the field of distributed systems. It adopts the client-server model. The calling process is always initiated by the client, including the serial number of the calling function, the parameters of the calling function and other information are packaged and sent to the server. , and then the server receives the request and executes it. After the server executes, it returns the execution result to the client.

FIG. 1 is a flow chart of execution actions of an RDMA-based distributed memory file system according to an embodiment of the present invention. As shown in FIG. 1 , according to an RDMA-based distributed memory file system according to an embodiment of the present invention, the memory of each node is firstly networked through RDMA, the distributed memory file system includes a client and a server, and the client provides a file access interface , which is called by the upper-layer application, the server provides metadata services and data services, and the distributed memory file system performs the following actions:

S1: In the initialization stage of the distributed memory file system, the memory used by the cluster for file storage is uniformly divided and registered to the network card to support the remote nodes to directly access the memory, and then build a distributed shared memory pool.

S2: On top of the distributed memory shared pool, file indexing and file data block indexing are performed respectively through two-level hash indexes to provide query services for the file system;

S3: Process the client's request through the self-identifying remote procedure call method, and return the processing result.

It should be noted that the distributed shared memory pool is composed of the shared memory of each node, and the shared memory of each node has a unified data layout. Specifically, the shared memory sequentially stores the super block, the message pool, the chain hash index table, and the metadata storage. Blocks and data storage blocks (as shown in Figure 3), the shared memory pool is used for file storage and message transmission, so the storage medium and communication method of this file system have changed. Through this unified management method, the overall software stack becomes thinner , the processing speed is higher.

In one embodiment of the present invention, in the data layout area, the super block, message pool and data area are registered to the network card, and this area can be directly accessed by remote nodes, thereby reducing memory copy to improve efficiency; chain hash index table and metadata storage area is independently maintained by the local service thread. The specific method is that the service node responds to all metadata requests, and completes the corresponding index and metadata query and update; metadata is distributed to the entire cluster according to the file path name hash , each node independently maintains the metadata and data of the file to improve the overall performance of the file system.

In an embodiment of the present invention, the super block is used to store the core data structure of the file system, and specifically includes the number of metadata blocks, the size of the metadata block, the number of data blocks, the size of the data block, and the like. This area will be remotely read by each node when the file system system is started for initial identification and positioning.

In an implementation of the present invention, the message pool is used for the communication between the client and the server. The specific method is to divide the message pool into message areas of the same size, and each message area is occupied by a client alone, that is, the client is bound to the The fixed offset of the message pool of the service node. When the client has a new request, the new request of the client is remotely written into the message area of the service node. After the receiving thread of the server detects the new request, it uses the unique The ID number queries the corresponding message area, identifies the message type, and returns it after processing.

In an embodiment of the present invention, a chained hash index table is used for local metadata indexing, and the index table sets a global unified entry, arranged in the form of a linear table, and is used to index specific chained table entries. Each table entry contains three fields, namely file name, metadata address, and next entry address. The specific method is: when querying file metadata, first calculate the hash value of the full path name of the file, and use this value as the index table Index number, query the linear table, read the file name of the corresponding entry and match, if successful, obtain the metadata address and access the metadata according to the address; if the file name does not match, continue to search according to the next entry address until the match is successful .

In one embodiment of the present invention, the metadata storage block and the data storage block are used to store metadata and data, respectively. The bitmap of the corresponding area is stored in the header of the above two areas to indicate the occupancy of the area.

In an embodiment of the present invention, the chained hash index table is used for local metadata indexing, and each entry contains three fields, namely file name, metadata address, and next entry address. The specific method is: in the query When the file metadata is used, first calculate the hash value of the full path name of the file, use the value as the index number of the index table, query the file name of the corresponding entry and match, if successful, obtain the metadata address and access the metadata according to the address; if the file If the name does not match, continue to search according to the next entry address until the match is successful.

According to the reconstruction method of the memory file system in the RDMA network of the present invention, the overall performance of the system is greatly improved. The traditional distributed file system uses the slow disk as the storage medium and uses the Gigabit network for network communication. Due to the high latency of the disk and the Gigabit network itself (millisecond level), the performance loss caused by the file system itself is small. When this type of file system uses memory as the storage medium, the file system itself occupies a large amount of delay in the entire data path, during which a large number of data copies (as shown in Figure 4) and redundant on-site switching are introduced, making the overall performance impossible. Linear improvement, for this reason, combined with RDMA technology, the present invention specially proposes an optimization scheme for memory media, proposes a thinner data management layer, builds a distributed memory shared pool, locates files in a hash way and obtains them quickly, so that The latency of the entire data path becomes extremely low, while also ensuring high system throughput.

Fig. 5 illustrates the self-identifying remote calling technology of the embodiment of the present invention. The method is based on a large memory cluster supporting RDMA hardware technology network interconnection. The RDMA technology means that the node can realize the direct reading and writing of the remote memory without the direct participation of the remote CPU. A large-memory cluster means that each node in the cluster is equipped with large-capacity memory and has spare memory to build a distributed memory file system. The methods include:

When the client sends a message to the server, the RDMA_WRITE_WITH_IMM primitive is used for data transmission and self-identification. When the server returns the request result, the RDMA primitive is used to write the returned data back. The specific method is:

When the client sends a message to the server, the RDMA_WRITE_WITH_IMM primitive allows the client to carry the client metadata when sending the request. In particular, the client stores its own ID and timestamp in this area to facilitate the server to quickly identify and locate;

When the server returns the request result, the returned result is directly written back to the memory area specified by the client through the RDMA primitive. At this time, the client will poll the memory area used to store the returned result until the data is returned successfully.

In an embodiment of the present invention, the client ID is assigned by the server master node when the connection is established, and is globally unique, which enables the client to automatically occupy a message area in the server message pool through its ID.

In an embodiment of the present invention, self-identification does not require the server to scan the entire message pool. After the RDMA_WRITE_WITH_IMM message is successfully delivered, the server receives the request in the queue and completes it, and puts the completion information into the completion queue. The server uses an independent thread Iteratively accesses the completion queue to detect new requests. After the server finds a new message, it first obtains the auxiliary information carried by the message according to the completion information, parses out the client ID, and directly queries the fixed bias of the local message pool according to the client ID. Move the location, obtain the new request information, then parse the request content, execute the corresponding function on the server, and then return the execution result.

In an embodiment of the present invention, when the number of clients reaches a certain number and exceeds the number of message pools allocated by the server in advance, the server queries the disconnected clients and transfers the message area occupied by them to the current Client; if all clients remain active, you need to apply for a new message area, register with the network card, and notify the client.

According to the self-identifying remote calling technology of the present invention, the remote request can be responded in time. This technology has the following advantages: the RDMA_WRITE_WITH_IMM primitive is selected to send messages, and on the premise of ensuring lower delay, the server can quickly detect and identify by carrying auxiliary information, and make timely processing; The request result is written back, which has the characteristics of extremely low latency, which makes the entire round-trip delay lower. At the same time, the client will allocate the memory area for storing the returned result in advance before sending the remote request, and attach the corresponding address to the request information. Therefore, the server can directly perform remote writing according to the given address, and the concurrency control of the memory area can be well controlled inside the client, so this technology can be well adapted to high concurrency scenarios.

In addition, other structures and functions of the RDMA-based distributed memory file system according to the embodiment of the present invention are known to those skilled in the art, and in order to reduce redundancy, detailed descriptions are omitted.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, The scope of the invention is defined by the claims and their equivalents.

Claims (10)

1. An RDMA-based distributed memory file system, wherein the distributed memory file system interconnects memory of each node through RDMA, the file system includes a client and a server, the client provides a file access interface for upper layer applications to call, the server provides metadata services and data services, and the distributed memory file system performs the following actions:

s1: in the initialization stage of the distributed memory file system, uniformly dividing the memory of the cluster for file storage, and registering the memory to a network card to support a remote node to directly access the memory, thereby constructing a distributed shared memory pool;

s2: on the distributed shared memory pool, file indexing and file data block indexing are respectively carried out through two-stage Hash indexing, and query service is provided for the file system;

s3: processing the request of the client by a self-identification remote procedure call method, and returning a processing result;

wherein the two-level hash index comprises: when a client side initiates a file access request, calculating a metadata server ID for storing metadata of a file according to the full path name of the file, wherein the metadata server ID is determined by a system configuration file; the client sends the request information to a metadata server corresponding to the ID, the metadata server analyzes the request content after detecting a new message, performs second hash value calculation according to the file path name in the request content, accesses the chained hash index table according to the second hash value, acquires the metadata information, performs corresponding logic processing and returns a request result;

the self-recognition remote procedure call method comprises the following steps: when the client sends a message to the server, using RDMA _ WRITE _ WITH _ IMM primitive to carry message content, and storing client metadata at the head of the message; when the server side returns the request result, the returned result is directly written back to the memory area designated by the client side through the RDMA primitive, and the client side monitors the memory area for storing the returned result in a polling mode until the data is successfully returned.

2. The RDMA-based distributed memory file system of claim 1, wherein the distributed shared memory pool stores, in order, a superblock, a message pool, a chain hash index table, a metadata store, and a data store.

3. The RDMA-based distributed memory file system of claim 2, wherein direct access by a remote node is received in a data layout area; in the chain hash index table and the metadata storage area, a service node responds to the client concurrent request and executes query and update on metadata; and the metadata is hashed and dispersed to the whole cluster according to the file path name, and each node independently maintains the metadata and the data of the file.

4. The RDMA-based distributed memory file system of claim 2, wherein the superblock is used to house a number of metadata blocks, a metadata block size, a number of data blocks, and a data block size, the superblock being read remotely by nodes at startup of the file system.

5. The RDMA-based distributed memory file system of claim 2, wherein the message pool comprises a plurality of message areas, and the plurality of message areas are allocated to different clients connected to the system, so that when a new request is made by a client, the client remotely writes the new request into the message area of the service node, and after the new request is monitored by the server receiving thread, the message is quickly located by using a self-recognition method and processed back.

6. The RDMA-based distributed memory file system of claim 2, wherein the chained hash index table is used for metadata indexing, and when querying file metadata, the hash value of the full pathname of the file is first calculated;

taking the hash value as an index number of an index table, inquiring the table items under the index number, matching file names, and if the file names are successfully matched, acquiring a metadata address and accessing the metadata according to the address;

if the file names do not match, the next table entry is continuously searched until the matching is successful.

7. The RDMA-based distributed memory file system according to claim 2, wherein the metadata storage area and the data storage area are used for storing metadata and data, respectively, the metadata storage area and the data storage area are divided into fixed-size metadata blocks and data blocks, and free block bitmaps of corresponding areas are stored at the headers of the metadata area and the data area to describe the memory usage.

8. The RDMA-based distributed memory file system of claim 1, wherein the client deposits its ID and a timestamp to the packet header, the client ID being assigned by a server master node at connection setup and being globally unique.

9. The RDMA-based distributed memory file system of claim 8, wherein after the RDMA _ WRITE _ WITH _ IMM message is successfully delivered, the server obtains the client metadata of the header of the message according to the completion information, parses out the client ID, and directly queries the fixed offset location of the local message pool according to the client ID to obtain new request information.

10. The RDMA-based distributed memory file system of claim 8, wherein when the number of clients exceeds the number of message pools allocated in advance by the server, the server queries the disconnected clients and transfers the message area occupied by the disconnected clients to the current client; and if all the clients keep the connection state, the clients need to reapply the message area, register to the network card and inform the current clients.

Images