Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, the first objective of the present invention is to provide a multi-core extensible file system construction method, so as to improve the throughput rate of a file system, improve the parallelism of data stored in the file system, and improve the capability of the file system of expanding the performance with the increase of CPU cores.
The second purpose of the invention is to provide a multi-core extensible file system construction device.
To achieve the above object, an embodiment of a first aspect of the present invention provides a method, including:
s1, obtaining the authority of accessing the file system by modifying the counter of the local CPU core;
s2, after the authority is obtained, modifying the index and the file cache of the file combination where each file is located;
and S3, writing the memory cache data into the storage devices of the independent partitions in parallel, and completing the file system expansion.
The method for constructing the multi-core extensible file system comprises the steps of obtaining the authority for accessing the file system by modifying a counter of a local CPU core, modifying the index and the file cache of the file combination where each file is located after obtaining the authority, and writing memory cache data into storage equipment of an independent partition in parallel to complete file system extension. The counter of the local CPU core is modified, competition of file modification operation on access permission of a file system is avoided, conflict of competition of file operation for access of the memory cache is reduced by segmenting the memory cache of the file system according to file granularity, file metadata and data are parallelized in the process of storing the file metadata and the data into the storage device by segmenting the storage device into independent partitions, throughput rate of the file system is improved, parallelism of data stored in the file system is improved, and capacity of expanding performance of the file system along with increase of the CPU core is improved.
To achieve the above object, an embodiment of a second aspect of the present invention provides a multi-core extensible file system building apparatus, including:
the authority acquisition module is used for acquiring the authority for accessing the file system by modifying the counter of the local CPU core;
the execution module is used for modifying the index and the file cache of the file combination where each file is located after the authority is obtained;
and the writing module is used for writing the memory cache data into the storage equipment of the independent partition in parallel to complete the file system expansion.
The multi-core extensible file system construction device of the embodiment of the invention obtains the authority of accessing the file system by modifying the counter of the core of the local CPU, modifies the index and the file cache of the file combination of each file after obtaining the authority, and writes the memory cache data into the storage equipment of the independent partition in parallel to complete the file system extension. The counter of the local CPU core is modified, competition of file modification operation on access permission of a file system is avoided, conflict of competition of file operation for access of the memory cache is reduced by segmenting the memory cache of the file system according to file granularity, file metadata and data are parallelized in the process of storing the file metadata and the data into the storage device by segmenting the storage device into independent partitions, throughput rate of the file system is improved, parallelism of data stored in the file system is improved, and capacity of expanding performance of the file system along with increase of the CPU core is improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
The following describes a method and an apparatus for constructing a multi-core extensible file system according to an embodiment of the present invention with reference to the drawings.
Modern computers are generally multi-core CPU architectures, i.e., one computer is equipped with multiple complete computing units. The multi-core computer can run a plurality of different tasks simultaneously and can access shared data of the file system simultaneously. To ensure that the system operates correctly, the file system uses concurrency control techniques to handle conflicting accesses by different tasks on the same shared data. However, the concurrent control techniques of the existing file systems are inefficient; this results in that as the number of computer CPU cores increases, the throughput provided by the file system to the upper layer application does not increase, and may even decrease, i.e., the file system is not scalable with multiple cores or has poor scalability with multiple cores.
According to the multi-core extensible file system construction method, competition of file modification operation on permission of accessing the file system is avoided by modifying the counter of the local CPU core, conflict of competition of file operation for accessing the memory cache is reduced by dividing the memory cache of the file system according to the file granularity, file metadata and data are parallelized in the process of storing the file metadata and the data into the storage device by dividing the storage device into independent partitions, throughput rate of the file system is improved, parallelism of data stored in the file system is improved, and capacity of expanding performance of the file system along with increase of the CPU core is improved.
Fig. 1 is a flowchart of a method for constructing a multi-core extensible file system according to an embodiment of the present invention.
As shown in fig. 1, the method for constructing a multi-core extensible file system includes:
in step S1, the authority to access the file system is obtained by modifying the counter of the local CPU core.
Specifically, the file modification operation obtains the authority to access the file system by modifying a counter of a core of a local CPU (central processing unit of a computer); file system internal checkpointing relies on the CPU scheduler checking the counters of all CPU cores to zero to gain access to the file system.
Further, in one embodiment of the present invention, the file modification operation includes modifying a file, creating a file, deleting a file, creating a directory, deleting a directory, renaming a file, renaming a directory.
The above-mentioned file modification operation is explained in detail below with reference to fig. 2. Fig. 2 is a schematic diagram of a concurrency control technique of a multi-core extensible file system construction method provided in an embodiment of the present application.
It will be appreciated that operations that modify a file system fall into two categories, one in which a user issues file modification operations, such as write file operations, and the other in which checkpointing operations are internal to the file system. As shown in FIG. 2, the file system maintains a counter on each core. When a write request comes, the write operation needs to increment the counter by 1 to gain access to the file system. Multiple writes may modify the counters on their cores simultaneously in parallel. When a checkpointing operation triggers, it needs to check the counters on all cores. The file system can give checkpoint operation authority if and only if the counters on all cores are zeroed. In performing the checking, the file system is checked incidentally when scheduling user tasks to a specific core using the CPU scheduler.
Further, a file modification operation refers to an operation that requires updating of file system metadata or user data.
Further, the counter of the local CPU core includes the number of unfinished file modification operations on the current CPU core.
Further, the authority to access the file system refers to the ability of the file system to exclusively read and modify a single or multiple files.
Further, a file system internal checkpointing includes the file system writing all updates to the file system onto the storage device.
Step S2, after the authority is obtained, the index and the file cache of the file combination where each file is located are modified.
It will be appreciated that the counter of the local CPU core is incremented before the file modification operation is performed and the file modification operation is completed by decrementing the counter of the local CPU core.
Specifically, the file system divides the memory cache by taking a file as a unit, and combines the divided memory caches into a plurality of file combinations; each file combination is organized by a separate index; when the file modification operation is executed, only the index and the file cache of the file combination where the file is located are modified.
Further, the memory cache data includes file system metadata, file metadata, and a copy of user data temporarily stored in the memory.
The above-mentioned memory cache is described in detail in units of files in conjunction with fig. 3. Fig. 3 is a schematic diagram of file system data splitting in the multi-core extensible file system construction method provided in the embodiment of the present application.
It is to be understood that one embodiment of the present invention segments and reassembles the memory cache in units of files, as shown in fig. 3. Multiple files are combined into one file combination, and all files in the file system are divided into multiple file combinations. A file assembly uses a tree index to organize all files within the assembly. The leaf nodes of the tree structure maintain a file cache including file system metadata, file metadata, and user data associated with the file. For a file modification operation, the modification operation is first located to a specific file combination by a hashing algorithm (e.g., a modulo operation of the unique identifier of the file and the total number of file combinations). Since the file modification operation does not alter the tree index, it then reads the lock on the tree index. The file modification operation is finally positioned to a specific file cache through the tree index, and the file system metadata, the file metadata and the user data are updated. In one embodiment of the invention, the file creation and deletion operations are similar in steps to the file modification operation, except that locks need to be written on the tree index, which may alter the tree index structure by adding or leaving a leaf node.
And step S3, writing the memory cache data into the storage devices of the independent partitions in parallel to complete the file system expansion.
Specifically, the file system divides the storage device into a plurality of independent partitions; the file modification operation selects any one of the free independent partitions and writes the data in parallel to the storage device.
It is to be understood that independent partitions comprise a disjoint portion of memory space of the address space; the free independent partition includes an independent partition to which no data is written at the current time.
The above-mentioned parallel writing of data to the storage device is explained in detail below with reference to fig. 4. Fig. 4 is a schematic diagram of a storage device partition in the multi-core extensible file system building method provided in the embodiment of the present application.
One embodiment of the present invention divides the storage device space into multiple part subspaces as shown in fig. 4. The storage device subspace is idle in an initial state, and when a write file 1 operation is received, the file system allocates the storage subspace 1 to the write operation. Both the metadata and data of file 1 are written into storage subspace 1. At the same time, the file system receives a create file 2 operation, and allocates the remaining free storage subspace, e.g., subspace 2, to the create operation. The metadata and data for file 2, and the metadata and data for its parent directory, are written into the subspace 2.
Further, when the file system is crashed and recovered, the latest file metadata is obtained by comparing the version numbers of the file metadata in different independent partitions, and the file system is rebuilt.
It will be appreciated that in one embodiment of the invention, crash recovery comprises a computer being suddenly powered down or a file system running in error and not working; the version number refers to the sequence of writing the files into the storage equipment; the file with the larger version number is the latest file.
The above-mentioned system recovery is explained in detail below with reference to fig. 5. Fig. 5 is a schematic diagram of file system crash recovery in the method for constructing a multi-core extensible file system according to the embodiment of the present application.
It will be appreciated that one embodiment of the present invention resolves data conflicts in storage subspaces in a system crash by comparing the version numbers of the files, as shown in FIG. 5. To maximize parallelism, one embodiment of the invention may distribute files to any free storage subspace. This makes it possible for different storage subspaces to exist simultaneously for the same file. To determine the latest file, the file system stores the version number of the file in the file metadata when writing the file. Even if the computer is suddenly powered down and crashed, the file system can determine the latest file by comparing the version number when the computer is started next time. As in FIG. 5, version number 2 in subspace 2 is greater than the version number in subspace 1, so the file is up-to-date in subspace 2.
According to the method for constructing the multi-core extensible file system, the authority for accessing the file system is obtained by modifying the counter of the core of the local CPU, the index and the file cache of the file combination where each file is located are modified after the authority is obtained, and the memory cache data are written into the storage equipment of the independent partition in parallel to complete the file system extension. The counter of the local CPU core is modified, competition of file modification operation on access permission of a file system is avoided, conflict of competition of file operation for access of the memory cache is reduced by segmenting the memory cache of the file system according to file granularity, file metadata and data are parallelized in the process of storing the file metadata and the data into the storage device by segmenting the storage device into independent partitions, throughput rate of the file system is improved, parallelism of data stored in the file system is improved, and capacity of expanding performance of the file system along with increase of the CPU core is improved.
Next, a multi-core extensible file system building apparatus according to an embodiment of the present invention is described with reference to the drawings.
Fig. 6 is a schematic structural diagram of a multi-core extensible file system building apparatus according to an embodiment of the present invention.
As shown in fig. 6, the multi-core extensible file system building apparatus includes:
the system comprises a permission acquisition module 100, an execution module 200 and a writing module 300.
The authority acquisition module is used for acquiring the authority for accessing the file system by modifying the counter of the local CPU core;
the execution module is used for modifying the index and the file cache of the file combination where each file is located after the authority is obtained;
and the writing module is used for writing the memory cache data into the storage equipment of the independent partition in parallel to complete the file system expansion.
Further, the embodiment of the invention also comprises: and the system recovery module is used for acquiring the latest file metadata by comparing the version numbers of the file metadata in different independent partitions when the file system is crashed and recovered, and reconstructing the file system.
It should be noted that the foregoing explanation on the embodiment of the multi-core extensible file system construction method is also applicable to the multi-core extensible file system construction apparatus of the embodiment, and is not repeated here.
According to the multi-core extensible file system construction device provided by the embodiment of the invention, the authority for accessing the file system is obtained by modifying the counter of the local CPU core, the index and the file cache of the file combination where each file is located are modified after the authority is obtained, and the memory cache data are written into the storage equipment of the independent partition in parallel to complete the file system extension. In the invention, competition of file modification operation on access permission of a file system is avoided by modifying a counter of a local CPU core, conflict of file operation competition access of the memory cache is reduced by dividing the memory cache of the file system by file granularity, file metadata and data are parallelized in the process of storing the file metadata and the data in the storage device by dividing the storage device into independent partitions, the throughput rate of the file system is improved, the parallelism of the data stored in the file system is improved, and the capability of expanding the performance of the file system along with the increase of the CPU core is improved.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.