CN108319634A - The directory access method and apparatus of distributed file system - Google Patents

Abstract

The present invention provides a directory access method and device for a distributed file system. The method includes: step 10: the client generates a specified directory access request for the distributed file system according to user operations; step 11: judges whether the client has specified The first memory cache of the directory, if there is the first memory cache of the specified directory, then perform step 12; step 12: load the directory item in the first memory cache of the specified directory on the client; step 13: continue to determine whether the client There is a local directory file, and the local directory file is used to save the directory items in the specified directory. If there is a local directory file, perform step 14; step 14: load the directory items in the specified directory file on the client. The directory access method and device of the distributed file system of the present invention can increase the access speed of a small number of directories or files and more than one million directories or files, and improve the user experience of the client.

Description

Directory access method and device of distributed file system

technical field

The invention relates to the computer field, in particular to a directory access method and device for a distributed file system.

Background technique

At present, large-scale distributed file systems can provide PB-level or even EB-level data storage, and storage space is no longer the bottleneck of storage technology. In theory, a distributed file system can provide an uncountable number of files.

On the other hand, if a directory in the distributed file system contains millions of sub-files or sub-directories, when the client accesses the directory through the network, since the client memory cannot cache more than one million sub-files or sub-directories, Therefore, the memory of the client is often insufficient or the memory is exhausted, which affects the user's access speed in the slightest, and causes the client to crash in severe cases, affecting the user experience.

For the above problems, no effective solution has been proposed yet.

Contents of the invention

In view of this, the present invention provides a directory access method and device of a distributed file system to solve the problem of slow speed or crash when a client accesses a large number of files or directories.

The present invention provides a directory access method of a distributed file system, the method comprising:

Step 10: The client generates a specified directory access request to the distributed file system according to the user operation;

Step 11: Determine whether the client has the first memory cache of the specified directory, and if there is the first memory cache of the specified directory, then perform step 12;

Step 12: Load the directory items in the first memory cache of the specified directory on the client;

Step 13: continue to determine whether the client has a local directory file, which is used to save the directory items in the specified directory, and if there is a local directory file, then perform step 14;

Step 14: Load the directory items in the specified directory file on the client side.

The present invention also provides a directory access device of a distributed file system, the device comprising:

User request generation module: the client generates a specified directory access request to the distributed file system according to user operations;

Comparison module: judging whether the first memory cache of the specified directory exists in the client, and if there is the first memory cache of the specified directory, execute the cache loading module;

Cache loading module: load the directory items in the first memory cache of the specified directory on the client;

Comparison module 1: continue to judge whether there is a local directory file in the client, the local directory file is used to save the directory items in the specified directory, if there is a local directory file, execute the file loading module;

File loading module: load the directory items in the specified directory file on the client side.

In the present invention, the directory entry storage of each specified directory is divided into two parts, and the directory entry is smaller than the first preset directory entry (such as 100,000) and cached in the first memory cache of the specified directory of the client; Part of the first default directory entry (eg, 100,000) is stored in a file on the local disk of the client. When the user accesses the directory of the distributed file system server through the client, by loading the corresponding directory item in the cache and/or the directory item stored locally, it can not only improve the speed of accessing sub-file items or sub-directory items in the directory, but also The client of this application limits part of the memory in the client memory, that is, the first memory cache, to save the directory items of the specified directory downloaded by the client from the distributed file system server, which will not affect the overall operation of the client system and will not bring Insufficient or exhausted memory problems caused by accessing a large number of files or directories.

The directory access method and device of the distributed file system of the present invention not only improve the directory access speed of a small number of files, but also improve the access and query speed of file directories with more than one million levels, and improve the user experience of the client.

Description of drawings

Fig. 1 is the first embodiment of the directory access method of the distributed file system of the present invention;

Fig. 2 is the second embodiment of the directory access method of the distributed file system of the present invention;

Fig. 3 is the third embodiment of the directory access method of the distributed file system of the present invention;

Fig. 4 is the fourth embodiment of the directory access method of the distributed file system of the present invention;

Fig. 5 is a structural diagram of the directory access device of the distributed file system of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

In the present invention, the directory entry obtained when the client accesses the distributed file system server is divided into two parts in the storage of the client, and the part of the directory entry smaller than the first preset directory entry (such as 100,000) is cached in the specified directory of the client In the first memory cache; the part of the directory entry greater than the first preset directory entry (eg, 100,000) is stored in a file on the client's local disk.

Based on this, the directory access method of the distributed file system of the present invention is proposed, as shown in Figure 1, the method comprises the following steps:

Step 10 (S101): The client generates an access request to a specified directory of the distributed file system according to user operations.

In step 10, the user operation may be a mouse click operation on a specified directory.

Step 11 (S102): Determine whether the client has the first memory cache of the specified directory, and if there is the first memory cache of the specified directory, then perform step 12;

Step 12 (S103): Load the directory item in the first memory cache of the specified directory on the client;

Step 13 (S104): continue to judge whether there is a local directory file in the client, the local directory file is used to save the directory items under the specified directory, if there is a local directory file, then perform step 14;

Step 14 (S105): Load the directory items in the specified directory file on the client side.

In the method shown in FIG. 1, the local directory file is used to save the directory items in the specified directory. In order to facilitate searching, the local directory file can be saved in a preset storage location, such as a certain folder. Each local directory file only stores directory items in a specified directory, and different specified directories have different local directory files. Further, in order to locate the specified local directory file, the file name of the local directory file can be the link information of the specified directory, such as the full directory information of the specified directory as the file name, or the specified directory inode (index node) as the file name.

Optionally, step 12 further includes: subdividing the directory items under the specified directory in the first memory cache into X directory groups, and the client sequentially loads the directory items under the X directory groups.

Optionally, step 14 further includes: subdividing the directory items included in the local directory file into Y directory groups, and the client loads the directory items under the Y directory groups sequentially.

When the user accesses the directory of the distributed file system through the client, by loading the corresponding directory item in the cache and/or the directory item stored locally, not only the speed of accessing the sub-file items or sub-directory items in the directory can be improved. At the same time, because the client of this application limits part of the memory in the client memory, that is, the first memory cache, to save the directory items downloaded by the client from the distributed file system, it will not affect the overall operation of the client system, nor will it bring access Insufficient or exhausted memory problems caused by a large number of files or directories.

Further, if the first memory cache of the specified directory is not updated within a preset time, the first memory cache of the specified directory is cleared.

When users access different designated directories, the first memory caches of different designated directories will be correspondingly generated in the memory. Using the "memory automatic clearing" strategy can avoid generating more and more first memory caches and continuously consuming client memory , to ensure the running speed of the client system. In addition, "memory automatic clearing" can also ensure that the directory entries saved in the first memory cache or local directory file in Figure 1 are the directory entries at the latest moment, and only need to load the directory entries saved by the client without accessing the distributed file system The server obtains the latest directory item information, saves network traffic, and improves user access speed.

The preset time can be set according to experience or user needs or the system memory of the client, such as 30 seconds or 1 minute.

Further, as shown in FIG. 2, step 11 in FIG. 1 also includes: if there is no first memory cache of the specified directory, then execute step 21;

Step 21 (S201): the client sends the specified directory access request to the server of the distributed file system;

Step 22 (S202): The server of the distributed file system reads the directory entries under the specified directory according to the specified directory access request, subdivides the directory entries into N directory groups and sends them to the client;

Step 23 (S203): judging whether the total number of directory entries in the second memory cache of the currently designated directory is greater than or equal to the first preset directory entry, if yes, then perform step 24, if not, then perform step 25;

Step 24 (S204): The client updates the received directory entry of the nth directory group into the client's local directory file, n=1, 2...N, and the client loads the directory entry of the nth directory group ( Display the directory entry of the nth directory group), return to step 23, until the N directory groups have been received;

Step 25 (S205): the client saves the received directory entry of the nth directory group to the end of the second memory cache of the specified directory, and the client loads the directory entry of the nth directory group (displaying the nth directory group's directory entry), return to step 23, until the N directory groups have been received;

Step 26 (S206): After saving the directory entries in the second memory cache of the specified directory to the first memory cache of the specified directory, clear the second memory cache.

Further, considering that after all the directory entries are received, the directory entries in the second memory cache may be larger than the first preset directory entries, as shown in Figure 3, step 26 may also include the following steps:

Step 27 (S207): judging whether the total number of directory entries in the second memory cache of the currently designated directory is greater than the first preset directory entry, if yes, then perform step 28, if not, then perform step 29;

Step 28 (S208): Save the previous first preset directory item in the second memory cache of the specified directory to the first memory cache of the specified directory, and save the previous first preset directory item in the second memory cache of the specified directory The other directory items other than the item are updated to the local directory file of the specified directory, and the second memory cache is cleared;

Step 29 (S209): After saving the directory entries in the second memory cache of the specified directory to the first memory cache of the specified directory, clear the second memory cache.

Further, in step 24 of FIG. 2 , updating the directory entry of the received nth directory group to the local directory file of the client by the client includes: judging whether there is a local directory file, and if not, creating a new local directory file , and save the directory entry of the nth directory group to the newly created local directory file of the client; if there is a local directory file, continue to judge whether the update time of the local directory file is the last update or this update, if it is the last update , then replace the original content of the client’s local directory file with the directory entry of the nth directory group, if it is this update, append the directory entry of the nth directory group to the client’s local directory file.

To determine whether the update of the local directory file is the last update or this update, you can judge by comparing the update timestamp of the local directory file with the current time. If the two times are very close, such as less than 1 minute, it is this update. Otherwise it is the last update.

In the method of Fig. 2, the server can determine the value of N according to the total number of directory entries under the specified directory, for example: if the total number of directory entries is less than 20, N=1 can be set; if the total number of directory entries is greater than 1000, N= 100.

Figure 2 shows that when the user accesses the distributed file system through the client, there is no first memory cache of the specified directory, which means that the user may be accessing the specified directory for the first time, or the first memory cache of the specified directory previously accessed has been " Automatically clear", at this time, the directory items in the specified directory can only be obtained by connecting to the distributed file system server.

In the method shown in Figure 2, the distributed file system divides the obtained directory items into N directory groups, and sends the directory items in the specified directory to the client with the directory group as a unit. The directory entries of the directory group are stored in the second memory cache, and after receiving, the directory entries in the second memory cache are transferred to the first memory cache. The second memory cache may be a background cache, and correspondingly, the first memory cache may be a front cache. The directory entry of the first (second) memory cache of the present application sets the upper limit of the directory entry as the first default directory entry, and the first default directory entry can be the largest directory entry that the first memory cache can store, such as the first default Catalog items are 100,000 items.

The method in Fig. 2 divides the directory items into N directory groups, which can refresh the directory items loaded by the client while receiving, so that it is not necessary to wait for the entire directory item to be received before loading it to the user, which can improve user experience.

In the above method, regardless of whether the client reads the directory item locally (the first memory cache or the local directory file), or the distributed file system server reads the directory item, it can be read in multiples by constructing an iterator.

For example, the number of directory entries read by the iterator each time is set to a fixed value, which can be changed according to requirements, and it is assumed to be 128. After reading 128 files each time, the position of the iterator will record the position (offset, corresponding to the serial number of the directory item) and name (name, corresponding to the name of the directory item) of the directory item currently read in the cache container. When reading 128 directory items next time, directly use offset or name to locate the iterator position, avoid retraversing the entire container, and speed up the reading of directory items.

The offset and name will be saved after reading 128 directory items each time. Even if the reading is interrupted when the network is abnormal, the position of the iterator can be relocated through the saved offset and name to ensure that the data will not be read less or misread. . At the same time, even if the last directory item has been deleted, the iterator position can be located by offset and name to ensure that the iterated read data is accurate.

As shown in Figure 4, the distributed file system directory access method of the present application also includes:

Step 30 (S301): When the client creates a new file in the distributed file system, add the directory entry of the new file to the first memory cache of the parent directory of the new file;

Step 31 (S302): Sorting the directory items in the first memory cache of the parent directory of the new file, the sorting method is consistent with the sorting method of the directory items of the distributed system server;

Step 32 (S303): judging whether the number of directory entries in the first memory cache of the parent directory of the new file is greater than the sum of the first preset directory entry and the second preset directory entry, if yes, then perform step 33;

Step 33 (S304): Delete the excess directory entry in the first memory cache of the parent directory of the new file; the excess directory entry is a directory entry exceeding the first preset directory entry in the first memory cache of the new file parent directory.

In step 31 of FIG. 3 , the client and the distributed file system use the same sorting algorithm to ensure that the reading order of the directory items included in the client is consistent with that when the distributed file system reads the directory items. When displayed, The displayed content is consistent.

In step 32 of Fig. 3, the second default directory entry is to avoid step 33 once for every additional file, assuming that the first default directory entry is 100,000, and the second default directory entry is 280, when Step 33 is executed only once after the number of directory entries in the first memory cache of the designated directory exceeds 100280, and centralized processing can avoid resource occupation and improve efficiency.

In addition, the distributed file system directory access method of the present application also includes:

Step 40: When the client deletes a file in the distributed file system, determine whether the directory entry corresponding to the deleted file is stored in the first memory cache of the parent directory of the deleted file on the client, and if so, delete the first memory cache of the parent directory of the file. The directory entry corresponding to the deleted file in the memory cache is deleted.

The directory access method and device of the distributed file system of the present invention not only improve the directory access speed of a small number of files, but also improve the access and query speed of file directories of more than one million levels, and improve the user experience of the client.

The present invention also provides a directory access device of a distributed file system, as shown in Figure 5, the device includes the following modules:

User request generation module: the client generates a specified directory access request to the distributed file system according to user operations;

Comparison module: judging whether the first memory cache of the specified directory exists in the client, and if there is the first memory cache of the specified directory, execute the cache loading module;

Cache loading module: load the directory items in the first memory cache of the specified directory on the client;

Comparison module 1: continue to judge whether there is a local directory file in the client, the local directory file is used to save the directory items in the specified directory, if there is a local directory file, execute the file loading module;

File loading module: load the directory items in the specified directory file on the client side.

Optionally, if the first memory cache of the specified directory is not updated within a preset time, the first memory cache of the specified directory is cleared.

Optionally, the comparison module in FIG. 5 also includes: if there is no first memory cache of the specified directory, execute the user request sending module;

User request sending module: the client sends the specified directory access request to the server of the distributed file system;

Server-side directory module: the server of the distributed file system reads the directory items under the specified directory according to the specified directory access request, subdivides the directory items into N directory groups and sends them to the client;

Comparison module 2: judging whether the number of directory entries in the second memory cache of the currently specified directory is greater than or equal to the first preset directory entry, if yes, execute the local directory file update module, if not, execute the second cache update module ;

Local directory file update module: the client updates the received directory entry of the nth directory group to the local directory file of the client, n=1, 2...N, the client loads the directory entry of the nth directory group, Return to the comparison module 2 until the N directory groups are received;

The second cache update module: the client saves the received directory entry of the nth directory group to the end of the second memory cache of the specified directory, the client loads the directory entry of the nth directory group, and returns to the comparison module 2, Until N directory groups are received;

The first cache update module: after saving the directory entries in the second memory cache of the specified directory to the first memory cache of the specified directory, clear the second memory cache.

Optionally, the first cache update module includes: a comparison module 3 , a first cache update module 1 and a first cache update module 2 .

Comparison module 3: judging whether the total number of directory entries in the second memory cache of the currently specified directory is greater than the first preset directory entry, if yes, execute the first cache update module 1, if not, execute the first cache update module 2.

First cache update module 1: save the previous first preset directory item in the second memory cache of the specified directory to the first memory cache of the specified directory, and save the previous first preset directory item in the second memory cache of the specified directory The directory entries other than the directory entry are updated to the local directory file of the specified directory, and the second memory cache is cleared.

The first cache updating module 2: After saving the directory items in the second memory cache of the specified directory to the first memory cache of the specified directory, clear the second memory cache.

Optionally, the device in Figure 5 also includes:

New file module: when the client creates a new file in the distributed file system, the directory entry of the new file is added to the first memory cache of the parent directory of the new file;

Directory item sorting module: sort the directory items in the first memory cache of the parent directory of the new file, and the sorting method is consistent with the sorting method of the directory items of the distributed system server;

Comparison module 4: judging whether the number of directory entries in the first memory cache of the parent directory of the new file is greater than the sum of the first preset directory entry and the second preset directory entry, if yes, perform step 43;

Deletion module 1: delete the excess directory entry in the first memory cache of the parent directory of the new file; the excess directory entry is the directory entry in the first memory cache of the new file parent directory that exceeds the first preset directory entry.

Optionally, the device in Figure 5 also includes:

Deletion module 2: When the client deletes a file in the distributed file system, it judges whether the directory entry corresponding to the deleted file is stored in the first memory cache of the parent directory of the deleted file on the client, and if so, deletes the first memory cache of the parent directory of the file. A directory entry corresponding to the deleted file in the memory cache is deleted.

Optionally, in the directory access device of the distributed file system of the present invention, the local directory file uses the specified directory inode as the file name.

It should be noted that the embodiment of the directory access device of the distributed file system of the present invention has the same principle as the embodiment of the directory access method of the distributed file system, and relevant parts can be referred to each other.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the technical solutions of the present invention are Should be included within the protection scope of the present invention.

Claims (10)

1. a kind of directory access method of distributed file system, which is characterized in that the described method comprises the following steps：

Step 10：Client generates the specified directory access request to distributed file system according to user's operation；

Step 11：Judge that client whether there is the first memory cache of the specified directory, if there are the specified directories First memory cache, thens follow the steps 12；

Step 12：Directory entry in the first memory cache that client is loaded into the specified directory；

Step 13：Continue to judge that client whether there is local directory file, the local directory file is preserving the finger Determine the directory entry under catalogue, if there are the local directory file, thens follow the steps 14；

Step 14：It is loaded into the directory entry in the specified directory file in client.

2. according to the method described in claim 1, it is characterized in that, if the first memory cache of the specified directory is default Time in do not update, then empty the first memory cache of the specified directory.

3. according to the method described in claim 1, it is characterized in that, the step 11 further includes：If the specified mesh is not present First memory cache of record, thens follow the steps 21；

Step 21：The specified directory access request is sent to the server-side of distributed file system by client；

Step 22：The server-side of the distributed file system reads the specified directory according to the specified directory access request Under directory entry, the directory entry is subdivided into N number of directory group and is sent to client；

Step 23：It is pre- to judge whether the directory entry sum in the second memory cache of presently described specified directory is more than or equal to first If directory entry, if so, 24 are thened follow the steps, if not, thening follow the steps 25；

Step 24：The directory entry of n-th of the directory group received is updated the local directory file to client by client In, n=1,2 ... N, client are loaded into the directory entry of n-th of directory group, return to the step 23, until N number of catalogue Group receives；

Step 25：The second memory that the directory entry of n-th of the directory group received is saved in the specified directory by client delays The end deposited, client are loaded into the directory entry of n-th of directory group, return to the step 23, until N number of directory group connects It harvests complete；

Step 26：Directory entry in second memory cache of the specified directory is saved in the first memory of the specified directory After caching, second memory cache is emptied.

4. according to the method described in claim 1, it is characterized in that, the method further includes：

Step 30：When client creates new file in the distributed file system, the directory entry of the new file is increased Into the first memory cache of the new file parent directory；

Step 31：Directory entry in first memory cache of the new file parent directory is ranked up, the method for the sequence It is consistent with the sort method of directory entry at distribution system services end；

Step 32：Judge whether the directory entry number in the first memory cache of the new file parent directory is more than the first default mesh The sum of item and the second predetermined directory item are recorded, if so, thening follow the steps 33；

Step 33：Catalogue entry deletion will be exceeded in first memory cache of the new file parent directory；It is described to exceed directory entry To exceed the directory entry of the first predetermined directory item in the first memory cache of the new file parent directory.

5. according to the method described in claim 1, it is characterized in that, the method further includes：

Step 40：When client deletes the file in the distributed file system, the corresponding mesh of the deletion file is judged Whether record item is stored in the first memory cache of the deletion file parent directory of client, if so, by deletion text The corresponding catalogue entry deletion of the deletion file in first memory cache of part parent directory.

6. a kind of directory access device of distributed file system, which is characterized in that described device includes：

User requests to generate module：Client is generated and is asked to the specified directory access of distributed file system according to user's operation It asks；

Comparison module：Judge that client whether there is the first memory cache of the specified directory, if there are the specified directories The first memory cache, then execute caching be loaded into mould；

Caching insmods：Directory entry in the first memory cache that client is loaded into the specified directory；

Comparison module 1：Continue to judge that client whether there is local directory file, the local directory file is described to preserve If directory entry under specified directory executes file and insmods there are the local directory file；

File insmods：It is loaded into the directory entry in the specified directory file in client.

7. device according to claim 6, which is characterized in that if the first memory cache of the specified directory is default Time in do not update, then empty the first memory cache of the specified directory.

8. device according to claim 6, which is characterized in that the comparison module further includes：If there is no described specified First memory cache of catalogue, then execute user's request sending module；

User's request sending module：The specified directory access request is sent to the service of distributed file system by client End；

Server-side catalogue module：The server-side of the distributed file system is according to described in specified directory access request reading The directory entry is subdivided into N number of directory group and is sent to client by the directory entry under specified directory；

Comparison module 2：Judge whether directory entry number in the second memory cache of presently described specified directory is more than or equal to the One predetermined directory item, if it is, local directory file update module is executed, if it is not, then executing the second buffer update module；

Local directory file update module：The directory entry of n-th of the directory group received is updated the institute to client by client It states in local directory file, n=1,2 ... N, client is loaded into the directory entry of n-th of directory group, returns to the comparison module 2, until N number of directory group receives；

Second buffer update module：The directory entry of n-th of the directory group received is saved in the specified directory by client The end of second memory cache, client are loaded into the directory entry of n-th of directory group, return to the comparison module 2, until institute N number of directory group is stated to receive；

First buffer update module：Directory entry in second memory cache of the specified directory is saved in the specified directory The first memory cache after, empty second memory cache.

9. device according to claim 6, which is characterized in that described device further includes：

New files module：When client creates new file in the distributed file system, by the catalogue of the new file Item increases in the first memory cache of the new file parent directory；

Directory entry sorting module：Directory entry in first memory cache of the new file parent directory is ranked up, the row The method of sequence is consistent with the sort method of directory entry at distribution system services end；

Comparison module 4：It is pre- to judge whether the directory entry number in the first memory cache of the new file parent directory is more than first If the sum of directory entry and the second predetermined directory item, if it is, executing removing module 1；

Removing module 1：Catalogue entry deletion will be exceeded in first memory cache of the new file parent directory；It is described to exceed catalogue Item is the directory entry beyond the first predetermined directory item in the first memory cache of the new file parent directory.

10. device according to claim 6, which is characterized in that described device further includes：

Removing module 2：When client deletes the file in the distributed file system, judge that the deletion file is corresponding Whether directory entry is stored in the first memory cache of the deletion file parent directory of client, if so, by the deletion The corresponding catalogue entry deletion of the deletion file in first memory cache of file parent directory.