CN1255748C - Metadata hierarchy management method and system of storage virtualization system - Google Patents

Abstract

The invention discloses a metadata layer management method of a storage virtualization system and a system thereof. The method manages the logical tree of metadata hierarchically, introduces a matching table to reduce the pressure on the root directory server, not only can quickly and accurately direct to the directory requested by the user, but also enables the system to achieve better scalability. The matching table is resident in memory, because it only records the root entry address of each directory server, so it occupies a very small space and is easy to manage. Since the match table does not have many entries, searching the match table to match the user request directory is also very efficient. The system includes a metadata server and a directory server provided with a matching table and a matching table management module; the matching table at least includes the name of the directory server and the root directory of the metadata sub-logic tree stored on the directory server. The invention has the characteristics of high efficiency, good expansibility, easy management and the like.

Description

Metadata level management method and system for storage virtualization system

technical field

The invention belongs to the field of computer storage, and in particular relates to a metadata layer management method of a storage virtualization system and a system thereof.

Background technique

In modern high-performance computing, scientific computing and military applications have increasingly high requirements for storage, including large capacity, distributed, high performance, and high reliability. The virtual storage management method of WAN integrates various high-performance storage systems that are geographically distributed to form a huge distributed storage space, fully realize resource sharing, improve resource utilization, and effectively solve the explosive growth of stored data and storage management capabilities The contradiction between relative inadequacy.

In a wide-area storage system, these massive storage resources must be managed effectively, thus introducing the concept of metadata. Metadata is the data that describes the data. It provides the system with a mapping between the physical location of the object and its logical name. A logical file can correspond to multiple physical file copies. In addition, metadata also includes file directory information, file information, storage device information and related system information, etc.

In many systems such as SRB or GridFTP, a hierarchical directory structure is used to manage metadata. When the metadata becomes huge, the system will enable multiple metadata servers at the same time, which will lead to the following problems: on the one hand, the logical tree must be distributed on these metadata servers, and on the other hand, the directory servers must be Collaborative operation, because the metadata to be returned may be distributed on different metadata servers, which puts pressure on the root directory server of metadata. Third, if the root directory server fails, the entire directory service system will no longer be able to function normally, and it is difficult to achieve high availability of the system. Finally, this structure makes it difficult to scale the metadata server.

Contents of the invention

The object of the present invention is to provide a metadata level management method of a storage virtualization system that can overcome the above defects. This approach is more efficient, scales better, and is easier to manage. The invention also provides a system for realizing the method.

A metadata level management method for a storage virtualization system provided by the present invention, the steps of which are as follows:

(1) Divide the entire storage virtualization system into multiple logical domains, and the metadata in each logical domain is distributed on multiple directory servers;

(2) Set up a metadata server in each logical domain to manage the directory server in the domain;

(3) A matching table is set in each metadata server, and the matching table includes at least two fields: the root directory of the metadata sub-logic tree stored on the directory server name and the directory server, for directly directing user requests to the directory server on which it resides;

(4) When the user makes a request, extract the path of the directory to be accessed from the user request;

(5) carry out long match to path in the matching table of this domain, judge whether there is the directory server that conforms to in the matching table;

(6) If there is a matching directory server, directly access the matching directory server and return the result; otherwise, an error message will be displayed.

The system for implementing the above method is divided into multiple logical domains, and the metadata in each logical domain is distributed on multiple directory servers. It is characterized in that: each logical domain specifies one server as the metadata server, and multiple servers As a directory server for storing metadata information; a matching table and a matching table management module are arranged in the metadata server, the matching table includes at least the name of the directory server and the root directory of the metadata sub-logic tree stored on the directory server, and the matching table management module It is used for maintaining the records in the matching table, parsing the matching table file, and matching to the appropriate directory server or displaying error messages according to the path requested by the user.

The metadata hierarchical management system of the present invention has the following advantages and effects.

(1) Higher efficiency: the matching table is resident in memory, and the number of entries it saves is limited by the number of directory servers in the domain, so the speed of traversing the matching table to match user requested directories is also very fast. After traversing, you can directly access the directory server where the requested directory is located, without going through the root directory server, which will greatly reduce the load on the directory server where the root directory is located, improve the speed of metadata search and then improve the efficiency of the entire system.

(2) Better scalability: The hierarchical tree management mode makes maintenance and management easy to expand; when the system load increases and the directory server needs to be added, only a few entries need to be added to the matching table to mark the new additions in the system directory server and its corresponding root directory. Because of the scheduling of the matching table, the directory servers can be independent from each other, and do not need to be aware of the existence of other directory servers in the domain. In this way, the system efficiency will not decrease with the increase of the number of directory servers, and the scalability of the system efficiency is achieved.

(3) Ease of management: the structure of the matching table is very simple, only recording the name of each directory server and the root directory of the metadata stored thereon. So it is very convenient to add, delete and modify.

When the invention uses JAVA as a development tool, platform independence can be achieved; when the storage virtualization system adds metadata copies, since the copy distribution can be completely displayed in the matching table, a suitable directory server can be selected through the matching table, In this way, copies can be effectively utilized and system efficiency can be improved.

Description of drawings

Figure 1 is an example diagram of a logical tree of metadata in a domain;

Fig. 2 is the metadata access process of the present invention;

Fig. 3 is the access method of the existing metadata logic tree;

Fig. 4 is the access mode of the logical tree of metadata of the present invention;

Fig. 5 is a structural diagram of metadata management.

Detailed ways

In our storage virtualization system GSP (Global Storage Provider), the entire system is managed hierarchically by dividing logical domains. In each domain, metadata is distributed on multiple directory servers, and forms a "local domain metadata logical tree" in an organization similar to the Linux file system. Figure 1 is an example of a domain metadata logical tree. In the figure, 1.1, 1.2, 1.3, 1.4 represent the directory servers in the domain; A0 represents the root directory of the metadata logical tree in the domain; A1, A2..., represent the subdirectories of the metadata logical tree in the domain.

Each user belongs to a domain, and the user's root directory is bound to a subdirectory in the domain. When the user logs in, the system automatically directs to the user's root directory.

Establish a metadata directory server (Directory Server) based on the directory tree structure to store metadata and its copies. However, within the scope of the wide area network, with the increase of resources in the system, the information of files and directories will become larger and larger. Obviously, if the directory server is not well organized, it will become the bottleneck of the system. So how to quickly and accurately direct to the user's root directory is a very prominent problem, which will affect the search speed of metadata and then affect the efficiency of the entire system. In our system, matching tables are introduced to solve the above problems.

A metadata server is set up in each domain to manage multiple directory servers in the domain, and the matching table is saved on the metadata server. The matching table is a data structure resident in memory, which records the name of each directory server and the root directory of the metadata stored on it; if a copy of a directory is saved on a directory server, this directory is also placed in the Match table to go. In the match table, an index field can also be set as a unique identifier of an entry in the match table.

When the system receives a request from a user, it first traverses the matching table on the metadata server, performs a longest match on all paths contained in the matching table according to the path requested by the user, and finds the directory server where the directory the user wants to access is located. Directly access this directory server instead of searching through the directory servers where the metadata root directory in the domain is located; if there is a copy of the metadata, you can also select a suitable directory server from the matching table for the user to access. The metadata access process is shown in Figure 2.

Figure 1 shows an example of a logic tree. This logical tree represents the file directory structure within a domain and is distributed across four directory servers. Among them, the root /A0 of the logical tree is stored on the directory server 1.1, the directory /A0/B1 is stored on the directory server 1.2, the directory /A0/A1/A2/C11 and the directory /A0/A1/A2/C12 are stored on the directory server On 1.3, the directory /A0/B1/B2/B3/B4/D1 is placed on directory server 1.4. Table 2 shows the matching table corresponding to the logic tree in Fig. 1.

Assuming that the user wants to access /A0/B1/B2/B3/B4/D1/D2, according to the existing metadata access method, it will first search from the directory server 1.1 where the root directory of the logical tree in the domain is located, and go through the directory server 1.3. Reach the directory server 1.4, and then return the result to the user, as shown in Figure 3. In the figure, R represents the path of the file/directory that the user requests to access; 1.1, 1.2, 1.3, 1.4 represent the directory server in the domain; A0 represents the root directory of the logical tree of metadata in the domain; B1, D1..., represent the metadata in the domain Directories in the logical tree; dotted arrows and r1, r2... denote user access steps. However, if the matching table mode used in the present invention is adopted, after the system receives the user's request, it will first search the matching table, and do a longest match according to the path requested by the user, so that metadata can be obtained on the directory server 1.4. The system will directly access the directory server 1.4 without going through the root directory entry 1.1, as shown in Figure 4. In the figure, 1 indicates all directory servers in the domain, 2 indicates the metadata server GNS in the domain, 3 indicates the matching table, dashed arrows and r1, r2 indicate user access steps.

In order to achieve the purpose of effective search, all directory servers must maintain the logical structure of the metadata tree in the domain. For example, directory server 1.4 still saves the root of the metadata tree in the domain, but it is just a path and does not save the content of the root, so that when a search request reaches 1.4, it does not need to make any changes to the request. Because the content of this root entry is empty, it only needs to occupy very little storage space, and it is also very convenient to maintain.

The content of the matching table is stored in a file, and when the whole system is started, the matching table saved in the file is read into the memory. The structure and function of the matching table determine that the reading operation of the matching table is generally performed in the memory, which does not involve the reading and writing of the matching table file. Only when a new directory server is added or the root directory of an existing directory server changes, the matching table is modified, and then the file is modified. It does not need to read and write the files of the matching table frequently, which greatly saves the time of accessing the matching table. time.

Add a matching table management module to the metadata server to maintain the records in the matching table. The specific operations include adding, deleting, and updating records, parsing the matching table file, and matching to the appropriate directory server according to the path requested by the user. Wait, the specific explanation is as follows.

Adding matching table records: When a directory server is added in the domain, a record is added to the matching table; if a subdirectory in a certain directory server is too large and needs to be migrated to another directory server, the matching table should also be notified Table, add a related record to record this subdirectory and the directory server it migrates to.

Deletion of matching table records: When a directory server no longer serves a certain domain, delete the corresponding record from the matching table;

Modification of matching table records: When the root directory of the directory tree stored on a directory server changes, modify the corresponding record;

Analysis of the matching table file: when the system starts, read the contents of the matching table file into the string array, which is resident in memory;

Match the path requested by the user: Find the directory server where the path requested by the user is located in the matching table.

Use five nodes in the cluster system to construct a metadata hierarchical management system, and its basic configuration is shown in Table 1.

CPU Memory hard drive network card operating system network Double PIII 866 256M 30G 3C905B Linux 6.2 100M switch

Table 1 Hardware and network configuration of each node

Among them, one acts as a metadata server, and the other four act as directory servers. The metadata server is responsible for the maintenance of the matching table, including a series of operations such as generating, adding, deleting, and updating. The directory server is responsible for storing metadata information.

The specific implementation is as follows: one of the nodes acts as a metadata server, and loads the file for saving the matching table and the matching table management module; the other four nodes act as 1.1, 1.2, 1.3, and 1.4, loading the LDAP database, and storing metadata in the domain.

According to Figure 5, 4.1 represents a logical domain Domain, and the shaded parts 4.2 and 4.3 represent multiple logical domains with the same structure as 4.1; 5 represents the matching table management module, and the dotted arrow represents the interaction between the metadata server and the directory server visit.

The configuration of the whole system is described as follows:

(1) The matching table includes two fields, an example of which is shown in Table 2. root directory path directory server /A0 1.1 /A0/B1 1.2 /A0/A1/A2/C11 1.3 /A0/A1/A2/C12 1.3 /A0/B1/B2/B3/B4/D1 1.4

Table 2 Example of matching table

The fields are explained as follows:

Path: the root directory path of the metadata sub-logical tree saved by the directory server;

Directory server name: the name of the directory server, each directory server name in the domain is unique.

(2) Metadata in the domain are stored on four directory servers, and the specific distribution example is shown in Table 3. directory server name Metadata information stored by the directory server 1.1 Metadata under /A0, but not under directories /A0/B1, /A0/A1/A2/C11, /A0/A1/A2/C12 and /A0/B1/B2/B3/B4/D1 1.2 Metadata under /A0/B1, but not including content under the directory /A0/B1/B2/B3/B4/D1 1.3 Metadata under directories /A0/A1/A2/C11 and /A0/A1/A2/C12 1.4 Metadata under directory /A0/B1/B2/B3/B4/D1

Table 3 Example of matching table

(3) Example of matching table operation

When the system starts, it parses the file that saves the matching table, and reads the following three string arrays, which are resident in memory:

Sindex[i]: Save the index value of the i-th record in the matching table, and the index value in the matching table is incremented. It is not required, but can simplify and facilitate the writing of programs;

Spath[i]: save the content of the field "path" in the i-th entry;

sDS[i]: Save the content of the field "Directory Server Name" in the i-th entry.

Sindex[i], spath[i], sDS[i] are a set corresponding to each other. Start recording the file from array[1]. The value of sindex[0] is the number of entries existing in the table; spath[0] is empty; sDS[0] is empty.

After receiving the file or directory path path requested by the user, first traverse the array spath, if the value of spath[i] is path, return sDS[i]; if path does not exist in spath, return the directory where the parent directory of path is located servers, or parent parent directories up to the root of the entire domain.

Claims (4)

1, a kind of metadata layer management method of memory virtualization system the steps include:

(1) whole memory virtualization system is divided into a plurality of domain logics, the metadata in each domain logic is distributed on a plurality of LIST SERVER;

(2) meta data server, the LIST SERVER that is used for managing this territory are set in each domain logic;

(3) matching list is set in each meta data server, this matching list comprises two fields at least: the root directory of the sub-logic tree of being preserved on LIST SERVER title and the LIST SERVER of metadata is used for the direct LIST SERVER that user's request is directed to its place;

(4) when the user files a request, from user's request, extract and to visit the path of catalogue;

(5) in the matching list in this territory, coupling is grown in the path, judged in matching list, whether there is the LIST SERVER that meets;

(6) if there is the LIST SERVER that meets, the LIST SERVER of directly visit coupling, and return results; Otherwise demonstration error message.

2, method according to claim 1 is characterized in that: also be provided with the catalogue copy in the described matching list.

3, method according to claim 1 is characterized in that: also be provided with index field in the described matching list.

4, a kind of metadata layer management system of memory virtualization system, this system is divided into a plurality of domain logics, metadata in each domain logic is distributed on a plurality of LIST SERVER, it is characterized in that: each domain logic specifies a station server as meta data server (2), specifies multiple servers as the LIST SERVER of preserving metadata information; Be provided with matching list (3) and matching table management module (5) in the meta data server (2), matching list (3) comprises the root directory of the sub-logic tree of being preserved on LIST SERVER title and the LIST SERVER of metadata at least, matching table management module (5) is used for that the record of matching list (3) is safeguarded, the parsing of matching list file and according to the route matching of user's request to suitable LIST SERVER or show error message.

Images