CN116089364A - Storage file management method and device, AI platform and storage medium - Google Patents

Abstract

This application relates to the field of computers, and discloses a storage file management method, device, AI platform, and storage medium, including: when the latest modification time of the first directory taken from the stack changes, obtain the subdirectory of the first directory ; The first directory is stored on the AI platform; judge whether the subdirectory is empty; if the subdirectory is not empty, put the subdirectory into the stack; if the subdirectory is empty, put the The subdirectory and the parent directory of the subdirectory that meet the preset condition are put into the queue; the preset condition is that all the subdirectories of the parent directory are all put into the queue; the second directory until the directory in the stack and the queue is empty, and perform corresponding operations on the database used to store the directory according to the last modification time of the second directory. This application can shorten the statistical management time, improve storage operation efficiency, and reduce the consumption of AI platform resources.

Description

A storage file management method, device, AI platform and storage medium

technical field

The present application relates to the field of computers, in particular to a storage file management method, device, AI platform and computer-readable storage medium.

Background technique

AI (Artificial Intelligence, artificial intelligence) platform is a platform that can manage and schedule the use of computing resources (such as GPU (Graphics Processing Unit, graphics processor), CPU (Central Processing Unit, central processing unit), etc.) and storage resources, and It can support business scenarios such as AI training and AI reasoning on a large scale.

A prominent feature of storage in the AI platform is that the number of stored files is massive (above TB level), so an important basic function of the AI platform is the statistical management of massive stored files. At present, there are several methods for statistically managing massive files in the AI platform as follows. The first is to directly traverse all the stored files to obtain the size of each file directory; the second is to use concurrent statistics to split the storage directory scheme; the third is to provide Quota quota function for the storage itself, such as Nfs and Beegfs file storage system. The first management method will continuously consume storage IO (Input/Output, input/output) resources during the traversal process, and also consume resources such as CPU and memory of business services, which will easily cause other file operations on storage nodes to freeze At the same time, the statistical results obtained by this method are not ideal. The statistical size of a large number of files is delayed, and the statistical results obtained have errors. The second method also consumes a lot of resources, which may easily cause other file operations on the storage node to freeze. The third method also consumes a lot of network, disk IO, CPU, and memory of storage resources, which increases the pressure on the storage of AI platforms.

Therefore, how to solve the above technical problems should be the focus of those skilled in the art.

Contents of the invention

The purpose of this application is to provide a storage file management method, device, AI platform and computer-readable storage medium, so as to reduce resource consumption and shorten statistical management time.

In order to solve the above technical problems, this application provides a storage file management method, including:

When the latest modification time of the first directory taken out from the stack changes, obtain the subdirectory of the first directory; the first directory is saved on the AI platform;

Determine whether the subdirectory is empty;

If the subdirectory is not empty, then put the subdirectory into the stack;

If the subdirectory is empty, put the subdirectory and the parent directory of the subdirectory that meet the preset condition into the queue; the preset condition is that all the subdirectories of the parent directory are put into the queue. in the queue;

Take out the second directory from the queue until the directories in the stack and the queue are empty, and perform corresponding operations on the database for storing the directory according to the latest modification time of the second directory.

Optionally, performing corresponding operations on the database for storing the directory according to the latest modification time of the second directory includes:

When the latest modification time of the second directory does not change, determine whether the database needs to be updated according to the size change of the second directory;

When the latest modification time of the second directory changes, corresponding operations are performed on the database according to the storage situation of the second directory in the database.

Optionally, according to the size change of the second directory, determining whether to update the database includes:

judging whether the size of the second directory changes;

If the size of the second directory changes, updating the storage path information of the second directory in the database;

If the size of the second directory does not change, no update operation is performed on the database.

Optionally, performing corresponding operations on the database according to the storage situation of the second directory in the database includes:

judging whether the second directory is saved in the database;

If the second directory is stored in the database, updating the storage path information of the second directory in the database;

If the second directory is not saved in the database, insert the second directory into the database.

Optionally, also include:

Index the tables in the database.

Optionally, also include:

According to the storage path information, the second directory is stored in sub-tables.

Optionally, also include:

It is judged whether the latest modification time of the second directory changes.

Optionally, the database is a database that can be embedded in microservices in the AI platform, can be migrated, and can be installed without configuration.

Optionally, removing the second directory from the queue includes:

The second directories that are not mutually parent-child directories are taken out from the queue at the same time.

Optionally, also include:

A directory that exists in the database and does not exist in the bottom layer is deleted from the database, and the directory includes a parent directory and all subdirectories under the parent directory.

Optionally, if the subdirectory is not empty, it also includes:

Record the information of the parent directory of the subdirectory.

Optionally, if the subdirectory is empty, putting the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue includes:

If the subdirectory is empty, then put the subdirectory into the queue;

Judging whether all subdirectories of the parent directory of the subdirectory are all put into the queue;

If all the subdirectories of the parent directory of the subdirectory are put into the queue, put the parent directory into the queue until the parent directory that does not meet the preset condition.

Optionally, when the latest modification time of the first directory taken from the stack changes, before acquiring the subdirectories of the first directory, the method includes:

It is judged whether the latest modification time of the first directory taken out from the stack changes.

Optionally, before judging whether the latest modification time of the first directory taken out of the stack changes, the method also includes:

put said first directory on a stack;

The first directory is removed from the stack.

Optionally, obtaining the subdirectories of the first directory includes

The first directory is traversed in a cleanable manner to obtain subdirectories of the first directory.

Optionally, traversing the first directory in a cleanable manner includes:

The first directory is opened with an open function, and the first directory is read with a read function.

Optionally, also include:

When the latest modification time of the first directory taken from the stack has not changed, acquire the subdirectories of the first directory from the database.

The present application also provides a storage file management device, including:

The first acquisition module is used to obtain the subdirectories of the first directory when the latest modification time of the first directory taken out from the stack changes; the first directory is stored on the AI platform;

The first judging module is used to judge whether the subdirectory is empty;

The first storage module is used to put the subdirectory into the stack if the subdirectory is not empty;

The second storage module is used to put the subdirectory and the parent directory of the subdirectory that meet the preset condition into the queue if the subdirectory is empty; the preset condition is the parent directory of the parent directory All subdirectories are put into the queue;

The removal and processing module is used to take out the second directory from the queue until the directory in the stack and the queue is empty, and according to the change of the latest modification time of the second directory, perform corresponding operations on the database used to store the directory operation.

This application also provides an AI platform, including:

memory for storing computer programs;

A processor configured to implement the steps of any one of the storage file management methods described above when executing the computer program.

The present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any one of the storage file management methods described above are implemented.

A storage file management method provided by the present application includes: when the latest modification time of the first directory taken out from the stack changes, obtaining a subdirectory of the first directory; the first directory is saved on the AI platform; judging whether the subdirectory is empty; if the subdirectory is not empty, then put the subdirectory into the stack; if the subdirectory is empty, then put the subdirectory and all the The parent directory of the subdirectory is put into the queue; the preset condition is that all the subdirectories of the parent directory are all put into the queue; the second directory is taken out from the queue until the stack and the directory in the queue is empty, and according to the change of the latest modification time of the second directory, corresponding operations are performed on the database used to store the directory.

It can be seen that the first directory of this application is stored in the stack. By taking out the first directory from the stack and when the latest modification time of the first directory changes, the subdirectory of the first directory is obtained, and the subdirectory is determined according to whether the subdirectory is empty. Whether the directory is queued or stacked. When there is a directory in the queue, remove the directory from the queue until the stack and the queue are all empty, and at the same time perform corresponding operations on the database according to whether the latest modification time of the directory removed from the queue has changed. The stack is a data structure method that realizes data first-in-first-out, and the queue is a data structure method that realizes data first-in-first-out, so this application repeats "out of the stack - into the stack - out of the stack and into the queue - out of the queue" until the stack and the queue All of them are empty to realize the "bottom-up" storage statistics management of the directory. In the process of statistics management, due to the continuous deletion (removal) in the stack and queue, the memory of the system will not overflow, and the statistics of the changed directory will avoid the waste of time and platform resources caused by full statistics.

Therefore, on the one hand, this application occupies very few AI platform resources, reduces the consumption of AI platform storage CPU, memory, IO and other resources, reduces the storage and load pressure of the AI platform, prevents other files on the storage node from being stuck, and improves the artificial intelligence model training. Improve efficiency, reduce resource consumption and long-term occupation of business modules, and enhance the performance of AI platform storage. On the other hand, this application shortens the statistical management time, improves the efficiency of each storage operation in the AI platform, shortens the model training time, reduces the operation and maintenance costs of the operation and maintenance personnel, and improves the market competitiveness of the AI platform.

In addition, the present application also provides a device, an AI platform, and a computer-readable storage medium having the above-mentioned advantages.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only For some embodiments of the present application, those of ordinary skill in the art can also obtain other drawings based on these drawings without creative effort.

FIG. 1 is a flow chart 1 of a storage file management method provided by an embodiment of the present application;

Fig. 2 is a schematic diagram of the stack data structure provided by the embodiment of the present application;

FIG. 3 is a schematic diagram of the queue data structure provided by the embodiment of the present application;

FIG. 4 is a second flowchart of a storage file management method provided by an embodiment of the present application;

FIG. 5 is a flowchart three of a storage file management method provided by an embodiment of the present application;

FIG. 6 is a flowchart 4 of a storage file management method provided by an embodiment of the present application;

FIG. 7 is a flowchart five of a storage file management method provided by the embodiment of the present application;

FIG. 8 is a flowchart six of a storage file management method provided by an embodiment of the present application;

FIG. 9 is the seventh flowchart of a storage file management method provided by the embodiment of the present application;

FIG. 10 is a structural block diagram of a storage file management device provided by an embodiment of the present application;

Fig. 11 is a structural block diagram of the AI platform provided by the embodiment of the present application;

Fig. 12 is a frame diagram of the storage statistics management system on the AI platform provided by the embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the drawings and specific implementation methods. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

As mentioned in the background technology section, due to the massive files stored on the AI platform, the related technologies have the following disadvantages when performing statistical management on massive files. First, it consumes various resources of the AI platform and increases the pressure on the storage of the AI platform. , and at the same time, it is easy to cause other file operations on the storage node to be stuck; second, the statistics management takes a long time.

In view of this, this application provides a storage file management method, please refer to Figure 1, the method includes:

Step S101: When the latest modification time of the first directory taken from the stack changes, obtain the subdirectories of the first directory; the first directory is saved on the AI platform.

In this application, the AI platform may be an AI cluster platform.

A stack is a linear storage structure that can only access data from one end and follows the "first in, last out" principle.

Based on the characteristics of the stack structure, in practical applications, only the following two operations are usually performed on the stack: the first one is to add data to the stack, and this process is called "pushing" (pushing or pushing); The second is to extract data from the stack. This process is called "popping the stack" (popping the stack). The position where the data is first pushed into the stack is called the "bottom of the stack", and the position where the data is last pushed into the stack is called the "top of the stack".

For example, as shown in Figure 2, when there are n directories a1, a2, ..., an, put a1, a2, ..., an into the stack in sequence. When taking out directories from the stack, n directories are taken out in the reverse order of the putting order, that is, the an directory is taken out first, then the a(n-1) directory, ..., the a2 directory is taken out, and the a1 directory is finally taken out.

The first directory is stored in the stack, and the first directory is taken out from the stack, that is, the first directory is popped out of the stack.

This application is based on the characteristics of Linux operating system files. The file system only has the size of the file, and the directory does not have the actual size information. The AI platform storage performs directory statistics, which can reduce the consumption of storage CPU, memory, IO and other resources, thereby reducing the storage and load pressure of the AI platform, and enhancing the performance of the AI platform storage. .

Operations that may cause the latest modification time of the first directory to change include but are not limited to adding soft links, adding hard links, compressing and decompressing, renaming, adding, deleting, and modifying hidden files.

Adding, deleting, and modifying files will cause the latest modification time of its parent directory to change, but only for the parent directory, and the last modification time of the parent directory of the parent directory will not change, that is, the stored directory changes When , it will only cause changes to the directory of the upper storage, and will not cause changes to other directories. Therefore, when counting the size of the stored directory, it is necessary to perform statistics on the size of the directory according to the change of the latest modification time of the directory, and start from the bottommost directory to the outermost directory.

It should be noted that, in this application, the type of the first directory is not limited, and it depends on the situation. For example, the first directory includes, but is not limited to, a user's home directory, a public directory, a dataset directory, a model directory, and the like.

Since the first directory can be the user's home directory, public directory, dataset directory, model directory, etc., the storage file management method in this application can be applied to different AI business scenarios to count the size of the public directory and the dataset directory.

The subdirectory of the first directory is also the directory of the next level of the first directory. The number of subdirectories under the first directory is not specifically limited in this application, and depends on circumstances. For example, the number of subdirectories of the first directory may be one, and the number of subdirectories of the first directory may be more than two.

Step S102: Determine whether the subdirectory is empty.

It should be noted that judging whether the subdirectory is empty means judging whether the subdirectory has a subdirectory, that is, judging whether there is a directory in the lower layer of the subdirectory.

Step S103: If the subdirectory is not empty, put the subdirectory on the stack.

If the subdirectory of the first directory is not empty, that is, there is a next-level directory under the subdirectory, then put the subdirectory into the stack.

In one embodiment of the application, if the subdirectory is not empty, it also includes:

Record the information of the parent directory of the subdirectory to ensure that the database is not repeatedly queried.

Step S104: if the subdirectory is empty, put the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue;

The default condition is that all subdirectories of the parent directory are put into the queue.

If the subdirectory of the first directory is empty, that is, there is no next directory under the subdirectory, that is, the first directory taken out from the stack at this time is already the bottom directory, and then the subdirectory is put into the queue, and then it will meet the preset All parent directories of the set condition are put into the queue.

For example, when there is only one B directory under the A directory, and the B directory has no next layer directory, after the B directory is put into the queue, the A directory is also put into the queue. When there are B1 directory and B2 directory in the lower layer of A directory, and there is no lower layer directory in B1 directory and B2 directory, after B1 directory and B2 directory are all put into the queue, then A directory is also put into the queue.

A queue (Queue), like a stack, is also a linear storage structure that has strict requirements on the "storage" and "fetch" of data. Different from the stack structure, both ends of the queue are "open", requiring data to only enter from one end and exit from the other end. Usually, the end of the data entry is called the "queue tail", and the end of the data output is called the "queue head". The process of data elements entering the queue is called "queue entry", and the process of leaving the queue is called "queue exit".

The entry and exit of data in the queue must follow the principle of "first in, first out", that is, the data in the most advanced queue must also be out of the queue first.

For example, as shown in Figure 3, when there are n directories such as a1 directory, a2 directory, ..., an directory, the a1 directory, a2 directory, ..., an directory are put into the queue in order, that is, the a1 directory is put into the queue first queue, then put the a2 directory into the queue, ..., and finally put the an directory into the queue. When taking out directories from the queue, n directories are taken out in the same order as they were put in, that is, the a1 directory will be taken out first, then the a2 directory, ..., and finally the an directory will be taken out.

Step S105: Take out the second directory from the queue until the directory in the stack and queue is empty, and perform corresponding operations on the database for storing the directory according to the latest modification time of the second directory.

Repeat the process of "popping - pushing - popping and merging into the queue - dequeue" until the stack and the queue are all empty. Then, follow-up processing is performed according to whether the latest modification time of the second directory taken out of the queue has changed. The specific processing will be described in the following embodiments, please refer to the following.

Only directories are saved in the database, and files are not stored. The directory information is stored in the database, which is convenient for quickly searching the file directory in the massive storage directory.

As a possible implementation, the database is a database that can be embedded in microservices in the AI platform, can be migrated, and has no configuration and installation. There can be one or more microservices in the AI platform, and the database can be embedded in one or more microservices as needed.

Preferably, the database is an SQLite3 database.

SQLite3 database is a lightweight database that implements "storage management". It is a relational database management system that complies with ACID, and realizes a self-sufficient, serverless, zero-configuration, and transactional SQL database engine. In addition, the SQLite3 database is guaranteed not to affect the use of the database by the AI platform business module, and does not need to be installed in the AI platform business module. It has the advantage of convenient migration along with storage, and can be accompanied by AI platform upgrades and patches, and the stored data will not be lost. .

When using multiple data sources to operate the SQLite3 database, it does not affect the use of other databases, such as Mysql. The SQLite3 database does not need to be installed and configured. Tens of millions of data are indexed within milliseconds (ms). SQLite3 will generate a binary file (database name.db) on the operating system. If the AI platform machine is down, you only need to You can continue to use the file by copying it to other machines, and the database can support a maximum of 128TiB. It is very suitable for solving the storage of storage directory data files in AI scenarios, and can quickly get the size of the storage directory.

Optionally, in one embodiment of the present application, before taking out the second directory from the queue, further include:

Determine whether there is a second directory in the queue;

If there is a second directory in the queue, then perform the step of taking out the second directory from the queue;

If the second directory does not exist in the queue, wait for the end of the stack.

After waiting for the end of the stack, exit the queue, that is, take out the second directory from the queue, so as to realize the "bottom-up" statistics of the directory.

In this embodiment, the first directory is stored in the stack. By taking out the first directory from the stack and when the latest modification time of the first directory changes, the subdirectory of the first directory is obtained, and the subdirectory is determined according to whether the subdirectory is empty. Whether the directory is queued or stacked. When there is a directory in the queue, remove the directory from the queue until the stack and the queue are all empty, and at the same time perform corresponding operations on the database according to whether the latest modification time of the directory removed from the queue has changed. The stack is a data structure method that realizes data first-in-first-out, and the queue is a data structure method that realizes data first-in-first-out, so this application repeats "out of the stack - into the stack - out of the stack and into the queue - out of the queue" until the stack and the queue All of them are empty to realize the "bottom-up" storage statistics management of the directory. In the process of statistics management, due to the continuous deletion (removal) in the stack and queue, the memory of the system will not overflow, and the statistics of the changed directory will avoid the waste of time and platform resources caused by full statistics. Therefore, on the one hand, this application occupies very few AI platform resources, reduces the consumption of AI platform storage CPU, memory, IO and other resources, reduces the storage and load pressure of the AI platform, prevents other files on the storage node from being stuck, and improves the artificial intelligence model training. Improve efficiency, reduce resource consumption and long-term occupation of business modules, and enhance the performance of AI platform storage. On the other hand, this application shortens the statistical management time, improves the efficiency of each storage operation in the AI platform, shortens the model training time, reduces the operation and maintenance costs of the operation and maintenance personnel, and improves the market competitiveness of the AI platform.

On the basis of the foregoing embodiments, in one embodiment of the present application, please refer to FIG. 4 , the storage file management method includes:

Step S201: When the latest modification time of the first directory taken from the stack changes, obtain the subdirectories of the first directory; the first directory is saved on the AI platform.

Step S202: Determine whether the subdirectory is empty.

Step S203: If the subdirectory is not empty, put the subdirectory on the stack.

Step S204: If the subdirectory is empty, put the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue; the preset condition is that all the subdirectories of the parent directory are put into the queue.

Step S205: Take out the second directory from the queue until the directories in the stack and the queue are empty.

Optionally, in one embodiment of the present application, after taking out the second directory from the queue until the directory in the stack and the queue is empty, further includes:

It is judged whether the latest modification time of the second directory has changed.

Step S206: When the latest modification time of the second directory has not changed, determine whether the database needs to be updated according to the size change of the second directory.

Operations that do not cause the latest modification time of the second directory to change include: modification of file permissions, and the like.

As an implementable manner, according to the size change of the second directory, determining whether to update the database includes:

Step S2061: Determine whether the size of the second directory changes.

Step S2062: If the size of the second directory changes, update the storage path information of the second directory in the database.

The storage path information is the directory path of the second directory.

The directory information stored in the database includes directory storage path information, the latest modification time of the directory, directory size, directory owner information, and the like.

Step S2063: If the size of the second directory does not change, no update operation is performed on the database.

When the size of the second directory does not change, no processing needs to be performed on the database.

Step S207: When the latest modification time of the second directory changes, perform corresponding operations on the database according to the storage situation of the second directory in the database.

Operations that can cause the last modification time of the second directory to change include but are not limited to adding soft links, adding hard links, compressing and decompressing, renaming, adding, deleting, and modifying hidden files.

As an implementable manner, according to the storage situation of the second directory in the database, performing corresponding operations on the database includes:

Step S2071: Determine whether the second directory is stored in the database.

Step S2072: If the second directory is stored in the database, update the storage path information of the second directory in the database.

The latest modification time of the second directory changes, and the second directory is also stored in the database, that is, the storage path information of the second directory in the database needs to be updated.

Step S2073: If the second directory is not stored in the database, insert the second directory into the database.

When the latest modification time of the second directory changes and the second directory is not saved in the database, this is the case of the first statistical traversal, so the second directory needs to be inserted into the database.

The purpose of updating the database is to ensure that there is only one catalog query.

It should be pointed out that, for steps S201 to S205, please refer to the content of the above-mentioned embodiments, which will not be described in detail here.

In order to improve retrieval efficiency, quickly obtain directory size information, and reduce the consumption of resources such as CPU and IO generated by AI platform storage during statistics, on the basis of any of the above embodiments, in one embodiment of the present application, storage file management Methods also include:

Create an index on a table in the database.

Further, in one embodiment of the present application, the stored file management method further includes:

According to the storage path information, the second directory is stored in sub-tables.

Create more than two tables in the database, store the second directory in separate tables according to the storage path information, avoid using the database of the AI platform to cause a certain degree of pressure on the database, making the AI platform more suitable for large-scale cluster file search, and at the same time can Realize fast search for directory information and improve retrieval efficiency.

When a table is established in the database, updating the database means updating the table in the database.

Please refer to FIG. 5, on the basis of any of the above embodiments, in one embodiment of the present application, the storage file management method includes:

Step S301: When the latest modification time of the first directory taken from the stack changes, obtain the subdirectories of the first directory; the first directory is saved on the AI platform.

Step S302: Determine whether the subdirectory is empty.

Step S303: If the subdirectory is not empty, put the subdirectory on the stack.

Step S304: If the subdirectory is empty, put the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue; the preset condition is that all the subdirectories of the parent directory are put into the queue.

Step S305: Take out the second directories that are not mutually parent-child directories from the queue at the same time, until the directories in the stack and the queue are empty.

Step S306: Determine whether the latest modification time of the second directory has changed.

Step S307: When the latest modification time of the second directory has not changed, determine whether the database needs to be updated according to the size change of the second directory.

Step S308: When the latest modification time of the second directory changes, perform corresponding operations on the database according to the storage situation of the second directory in the database.

It should be pointed out that, for steps S301 to S304 and steps S306 to S308, please refer to the contents of the above embodiments, and details are not repeated here.

In this embodiment, when the second directory is taken out from the queue, the second directories that are not mutually parent-child directories are taken out from the queue at the same time, that is, the directory data that are not mutually parent-child directories are counted asynchronously to speed up the "upward" process statistics , shorten statistical management time, and improve statistical efficiency.

Please refer to FIG. 6, on the basis of any of the above embodiments, in one embodiment of the present application, the storage file management method includes:

Step S401: When the latest modification time of the first directory taken from the stack changes, obtain the subdirectories of the first directory; the first directory is saved on the AI platform.

Step S402: Determine whether the subdirectory is empty.

Step S403: If the subdirectory is not empty, put the subdirectory on the stack.

Step S404: If the subdirectory is empty, put the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue; the preset condition is that all the subdirectories of the parent directory are put into the queue.

Step S405: Take out the second directory from the queue until the directory in the stack and the queue is empty, and perform corresponding operations on the database for storing the directory according to the latest modification time of the second directory.

Step S406: delete from the database the directories that exist in the database and do not exist in the bottom layer, the directories include the parent directory and all subdirectories under the parent directory.

It should be pointed out that, for steps S401 to S405, please refer to the content of the above-mentioned embodiments, which will not be described in detail here.

In this implementation, delete and clean up the directory that does not exist in the bottom layer but exists in the database to ensure that there is no dirty data in the database and prevent the database from growing larger.

Please refer to FIG. 7, on the basis of any of the above embodiments, in one embodiment of the present application, the storage file management method includes:

Step S501: When the latest modification time of the first directory taken from the stack changes, obtain the subdirectories of the first directory; the first directory is saved on the AI platform.

Step S502: Determine whether the subdirectory is empty.

Step S503: If the subdirectory is not empty, put the subdirectory on the stack.

Step S504: If the subdirectory is empty, put the subdirectory into the queue.

Step S505: Determine whether all the subdirectories of the parent directory of the subdirectory are put into the queue.

Step S506: If all the subdirectories of the parent directory of the subdirectory are put into the queue, put the parent directory into the queue until the parent directory that does not meet the preset condition.

Step S507: Take out the second directory from the queue until the directory in the stack and the queue are empty, and perform corresponding operations on the database for storing the directory according to the latest modification time of the second directory.

Step S508: If not all subdirectories of the parent directory of the subdirectory are put into the queue, the parent directory is not put into the queue.

It should be pointed out that, for steps S501 to S505 and S507, please refer to the contents of the above embodiments, and details are not repeated here.

On the basis of any of the above embodiments, in one embodiment of the present application, when the latest modification time of the first directory taken from the stack changes, before obtaining the subdirectories of the first directory, include:

It is judged whether the latest modification time of the first directory taken out from the stack changes.

When the latest modification time of the first directory taken from the stack changes, the step of obtaining the subdirectory of the first directory is executed.

When the latest modification time of the first directory fetched from the stack has not changed, specific operations performed are described in the following embodiments.

Further, in one embodiment of the present application, before judging whether the latest modification time of the first directory taken out of the stack changes, it also includes:

Put the first directory on the stack;

Remove the first directory from the stack.

Please refer to FIG. 8. On the basis of any of the above embodiments, in one embodiment of the present application, the storage file management method includes:

Step S601: When the latest modification time of the first directory taken from the stack changes, traverse the first directory in a cleanable manner to obtain subdirectories of the first directory; the first directory is saved on the AI platform.

In the cleanable mode, after opening the first directory, the first directory can be closed, which has the characteristics of high performance. Optionally, the clearable mode can be a stream (stream) mode.

As an implementable manner, traversing the first directory in a cleanable manner includes:

The first directory is opened with the open function, and the first directory is read with the read function.

The opening function may be an opendir function, and the reading function may be a readdir function.

Step S602: Determine whether the subdirectory is empty.

Step S603: If the subdirectory is not empty, put the subdirectory on the stack.

Step S604: If the subdirectory is empty, put the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue; the preset condition is that all the subdirectories of the parent directory are put into the queue.

Step S605: Take out the second directory from the queue until the directory in the stack and the queue is empty, and perform corresponding operations on the database for storing the directory according to the latest modification time of the second directory.

It should be pointed out that, for steps S602 to S605, please refer to the content of the above-mentioned embodiments, which will not be described in detail here.

Please refer to FIG. 9, on the basis of any of the above embodiments, in one embodiment of the present application, the storage file management method includes:

Step S701: Determine whether the latest modification time of the first directory taken from the stack has changed.

Step S702: when the latest modification time of the first directory taken from the stack has not changed, obtain the subdirectories of the first directory from the database.

If there are more than 10,000 files in a single directory, since the directory has not changed, you only need to obtain the directories under the directory in the database, and do not need to use the opendir function and readdir function to traverse the underlying storage. Since the number of directories is much smaller than the number of files, the number of directories is about 20,000 in the case of 1TB storage usage on the AI platform, so only the database is required to store directory information.

In the AI platform, within a period of time (such as 5 minutes), 99% of the directories will not change. Use the previous size for the directory that has not changed, and only need to count the size of the directory that has changed.

Step S703: When the latest modification time of the first directory taken from the stack changes, obtain the subdirectories of the first directory; the first directory is saved on the AI platform.

Step S704: Determine whether the subdirectory is empty.

Step S705: If the subdirectory is not empty, put the subdirectory on the stack.

Step S706: If the subdirectory is empty, put the subdirectory into the queue.

Step S707: Determine whether all the subdirectories of the parent directory of the subdirectory are put into the queue.

Step S708: If all the subdirectories of the parent directory of the subdirectory are put into the queue, put the parent directory into the queue until the parent directory that does not meet the preset condition.

Step S709: Take out the second directory from the queue until the directories in the stack and the queue are empty.

Step S710: Determine whether the latest modification time of the second directory has changed.

Step S711: When the latest modification time of the second directory has not changed, determine whether the database needs to be updated according to the size change of the second directory.

Step S712: When the latest modification time of the second directory changes, perform corresponding operations on the database according to the storage situation of the second directory in the database.

When the latest modification time of the directory has not changed, the subdirectory of the first directory is obtained from the database, that is, the size of the previous directory is used, and only the size of the directory that has changed is counted. Each time the statistics are stored to prevent repeated directory traversal, avoid The traditional full storage statistics method improves storage statistics efficiency.

The storage file management method in this application will be described below with a situation.

The first step: first define the use of data structure stack and data structure queue.

Step 2: After continuously putting the directory into the stack, then take out the directory in the stack, if there is no subdirectory under the directory (that is, the data taken out of the stack must be the bottom layer), then define the data structure queue to be used, and the stack The directory fetched from is put into the queue.

Step 3: After the second step is completed, it is necessary to judge whether all subdirectories of the parent directory of the subdirectory are queued. If all subdirectories are queued, the parent directory is queued until the parent directory that does not meet the conditions. This step ensures that all directories can be queued. Time efficiency O(1) (indicating no time consumption), efficiency is related to directory hierarchy.

Step 4: Take out the directory of the queue, which is divided into three situations:

1) The latest modification time has changed, and the table in the SQLite3 database does not exist (the first statistical traversal);

2) The latest modification time has changed, and the table in the SQLite3 database already exists, so update the table fields;

3) If the latest modification time has not changed, if the directory size changes, the table fields of the SQLite3 database will be updated, otherwise it will not be updated, (this is the case for most scenarios encountered in statistics);

Wherein, the storage path information is updated during updating.

You can use multiple threads to exit the queue at the same time to speed up the statistics of the "upward" process. It is necessary to ensure that the directories that are exported are not parent-child directories.

Step 5: Continuously "pop-up-stack-pop-up and put into queue-out queue" until the stack and queue are empty. Due to the continuous deletion (removal) in the stack and queue, it will not cause memory overflow of the system.

Step 6: Delete and clean up the directory that does not exist in the bottom layer and the SQLite3 database, and delete the database including all subdirectories under this directory to ensure that there is no dirty data in the database and the data table will not become larger and larger.

The storage file management method in this application has the following advantages:

First, quickly count and store the size of each directory in full, do not count the directories that have not changed, update and store the size of each directory after counting, use SQLite3 lightweight database for system design, and finally use it to store the size of the directory for each business function of the AI platform Use to quickly obtain the size of the storage space used by the AI platform, which is convenient for the AI platform to manage, display and limit the storage space. This application reduces the storage and load pressure of storage servers, reduces resource consumption and long-term occupation of business modules, and enhances the performance of AI platform storage. This application can improve the business performance of the AI platform, use SQLite3 to build a storage directory index, and divide the storage directory into tables, quickly get the size request of the storage statistics directory, build the storage statistics management system to improve the efficiency of each storage operation in the AI business, and shorten the model Training time, improve model training efficiency, reduce operation and maintenance personnel's operation and maintenance costs, and improve the market competitiveness of AI platforms;

Second, this application ensures the efficient and stable operation of the AI platform, effectively shortens the time for algorithm personnel to conduct model training, improves the storage performance of the AI platform, solves the technical problem of large storage file directory statistics, and solves the network and IO of the AI platform The performance pain point of frequent interactive statistics improves the file operation and management performance, reduces the overall resource utilization of the AI platform, makes the use of the AI platform smoother, and enhances the competitiveness of the AI platform.

The storage file management device provided by the embodiment of the present application is introduced below, and the storage file management device described below and the storage file management method described above can be referred to in correspondence.

FIG. 10 is a structural block diagram of a storage file management device provided in an embodiment of the present application. Referring to FIG. 10, the storage file management device may include:

The first obtaining module 100 is used to obtain the subdirectories of the first directory when the latest modification time of the first directory taken out from the stack changes; the first directory is stored on the AI platform;

The first judging module 200 is used to judge whether the subdirectory is empty;

The first storage module 300 is used to put the subdirectory into the stack if the subdirectory is not empty;

The second storage module 400 is used to put the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue if the subdirectory is empty; the preset condition is that all subdirectories of the parent directory are all put into the queue;

The removal and processing module 500 is used to take out the second directory from the queue until the directory in the stack and the queue is empty, and perform corresponding operations on the database used to store the directory according to the latest modification time of the second directory.

The storage file management device in this embodiment is used to implement the aforementioned storage file management method, so the specific implementation of the storage file management device can be seen in the embodiment part of the storage file management method above, for example, the first acquisition module 100, the first A judging module 200, a first storage module 300, a second storage module 400, and a removal and processing module 500 are respectively used to implement steps S101, S102, S103, S104 and S105 in the above storage file management method, so, its specific implementation Reference may be made to the descriptions of the corresponding partial embodiments, and details are not repeated here.

Optionally, the removal and processing module 500 includes:

The first operation submodule is used to determine whether the database needs to be updated according to the size change of the second directory when the last modification time of the second directory has not changed;

The second operation sub-module is used for performing corresponding operations on the database according to the storage situation of the second directory in the database when the latest modification time of the second directory changes.

Optionally, the first operation submodule includes:

a first judging unit, configured to judge whether the size of the second directory changes;

The first update unit is configured to update the storage path information of the second directory in the database if the size of the second directory changes;

The stop operation unit is configured to not update the database if the size of the second directory does not change.

Optionally, the second operation submodule includes:

The second judging unit is used to judge whether there is a second directory stored in the database;

The second update unit is used to update the storage path information of the second directory in the database if the second directory is stored in the database;

The insertion unit is configured to insert the second directory into the database if the second directory is not saved in the database.

Optionally, the storage file management device also includes:

The index building module is used for building indexes on the tables in the database.

Optionally, the storage file management device also includes:

The storage module is configured to store the second directory in sub-tables according to the storage path information.

Optionally, the storage file management device also includes:

The second judging module is used for judging whether the latest modification time of the second directory has changed.

Optionally, the removing and processing module 500 is specifically configured to simultaneously remove the second directories that are not mutually parent-child directories from the queue.

Optionally, the storage file management device also includes:

The delete module is used to delete from the database the directories that exist in the database and do not exist in the bottom layer. The directories include the parent directory and all subdirectories under the parent directory.

Optionally, if the subdirectory is not empty, the storage file management device also includes:

The recording module is used to record the information of the parent directory of the subdirectory.

Optionally, the second storage module 400 includes:

The first storage submodule is used to put the subdirectory into the queue if the subdirectory is empty;

The judging sub-module is used to judge whether all the sub-directories of the parent directory of the sub-directory are put into the queue;

The second storage submodule is used to put the parent directory into the queue if all the subdirectories of the parent directory of the subdirectory are put into the queue until the parent directory that does not meet the preset condition.

Optionally, the storage file management device also includes:

The third judging module is used for judging whether the latest modification time of the first directory taken out from the stack has changed.

Optionally, the storage file management device also includes:

The third storage module is used to put the first directory into the stack;

Remove module, used to remove the first directory from the stack.

Optionally, the first obtaining module 100 is specifically configured to traverse the first directory in a cleanable manner to obtain subdirectories of the first directory.

Optionally, the first acquisition module 100 includes:

Open the sub-module for opening the first directory by using the open function;

The read submodule is used to read the first directory by using a read function.

Optionally, the storage file management device also includes:

The second obtaining module is configured to obtain the subdirectories of the first directory from the database when the latest modification time of the first directory taken out from the stack has not changed.

The following is an introduction to the AI platform provided by the embodiment of the present application. The AI platform described below and the storage file management method described above can be referred to in correspondence.

FIG. 11 is a structural block diagram of the AI platform provided by the embodiment of the present application. The AI platform includes:

memory 11 for storing computer programs;

The processor 12 is configured to implement the steps of the storage file management method in any of the above embodiments when executing the computer program.

The framework diagram of the storage statistics management system on the AI platform is shown in Figure 12, including framework dependencies, framework input parameters, framework methods, and SQLite3 databases. Among them, framework dependencies include database drivers (Sqlite-jdbc), dynamic loading tables, and database query statements (mybats) and dynamic loading of multiple data sources (Dynamic-datasource); framework input parameters include statistics storage path (Storage path), statistics thread number (ThreadNum), filter directory list (Filter path); framework methods include storage statistics task delivery (Storage Statistic Service.storage Statistic (StorageParmeter)), Get the size of the storage path (Storage Statistic Service.size By Storage Path(Storage Path)), Get the size of all files belonging to the user for group sharing and global sharing (StorageStatistic Service.size By Share Path Owner (owner, path)); when the database file is mapped to the storage directory, the storage path can be /mnt/inspurfs/db/storageSqlite.db.

On the premise of introducing framework dependencies, the business modules of the AI platform only need to call the management method and pass in the statistical parameters: the storage path of the statistics, the statistical thread data, and the filtering directory list (supporting statistical filtering of some directories, In actual scenarios, there are some directories that do not need to be counted, such as those that do not belong to the user’s home directory to ensure statistical efficiency, and the directories that can be filtered when the statistical task is delivered), and the size of each directory can be quickly obtained. The purpose of the directory size is not only for disk quota restrictions In addition, it can also be used for file management display, data set size acquisition, etc. This patent system is based on the AI platform storage and integrates the AI platform business, performs efficient storage statistics, and manages the storage directory. Using this solution for storage size statistics can improve the performance and management efficiency of storage in the AI platform, and expand the nodes of the AI platform Scale management such as the number of users and the number of users.

The computer-readable storage medium provided by the embodiments of the present application is introduced below, and the computer-readable storage medium described below and the storage file management method described above may be referred to in correspondence.

A computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the storage file management method in any of the above embodiments are implemented.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for relevant details, please refer to the description of the method part.

Professionals can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the possible For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The storage file management method, device, AI platform, and computer-readable storage medium provided by the present application have been introduced in detail above. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application, and the descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application. It should be pointed out that those skilled in the art can make some improvements and modifications to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims (20)

1. A storage file management method, characterized in that, comprising: When the latest modification time of the first directory taken out from the stack changes, obtain the subdirectory of the first directory; the first directory is saved on the AI platform; Determine whether the subdirectory is empty; If the subdirectory is not empty, then put the subdirectory into the stack; If the subdirectory is empty, put the subdirectory and the parent directory of the subdirectory that meet the preset condition into the queue; the preset condition is that all the subdirectories of the parent directory are put into the queue. in the queue; Take out the second directory from the queue until the directories in the stack and the queue are empty, and perform corresponding operations on the database for storing the directory according to the latest modification time of the second directory. 2. The storage file management method according to claim 1, wherein, according to the change of the latest modification time of the second directory, performing corresponding operations on the database for storing the directory comprises: When the latest modification time of the second directory does not change, determine whether the database needs to be updated according to the size change of the second directory; When the latest modification time of the second directory changes, corresponding operations are performed on the database according to the storage situation of the second directory in the database. 3. The storage file management method according to claim 2, wherein, according to the size variation of the second directory, determining whether the database needs to be updated comprises: judging whether the size of the second directory changes; If the size of the second directory changes, updating the storage path information of the second directory in the database; If the size of the second directory does not change, no update operation is performed on the database. 4. The storage file management method according to claim 3, wherein, according to the storage situation of the second directory in the database, performing corresponding operations on the database comprises: judging whether the second directory is saved in the database; If the second directory is stored in the database, updating the storage path information of the second directory in the database; If the second directory is not saved in the database, insert the second directory into the database. 5. The storage file management method according to claim 4, further comprising: Index the tables in the database. 6. The storage file management method according to claim 5, further comprising: According to the storage path information, the second directory is stored in sub-tables. 7. The storage file management method according to claim 2, further comprising: It is judged whether the latest modification time of the second directory changes. 8. The storage file management method according to claim 1, wherein the database is a database that can be embedded in microservices in the AI platform, can be migrated, and is installed without configuration. 9. The storage file management method according to claim 1, wherein taking out the second directory from the queue comprises: The second directories that are not mutually parent-child directories are taken out from the queue at the same time. 10. The storage file management method according to claim 1, further comprising: A directory that exists in the database and does not exist in the bottom layer is deleted from the database, and the directory includes a parent directory and all subdirectories under the parent directory. 11. The storage file management method according to claim 1, wherein if the subdirectory is not empty, further comprising: Record the information of the parent directory of the subdirectory. 12. The storage file management method according to claim 1, wherein if the subdirectory is empty, putting the subdirectory and the parent directory of the subdirectory satisfying the preset condition into the queue comprises: If the subdirectory is empty, then put the subdirectory into the queue; Judging whether all subdirectories of the parent directory of the subdirectory are all put into the queue; If all the subdirectories of the parent directory of the subdirectory are put into the queue, put the parent directory into the queue until the parent directory that does not meet the preset condition. 13. The storage file management method according to claim 1, wherein when the latest modification time of the first directory taken out from the stack changes, before obtaining the subdirectories of the first directory, the steps include: It is judged whether the latest modification time of the first directory taken out from the stack changes. 14. The storage file management method according to claim 13, wherein before judging whether the latest modification time of the first directory taken out from the stack changes, the method further includes: put said first directory on a stack; The first directory is removed from the stack. 15. The storage file management method according to claim 1, wherein obtaining the subdirectory of the first directory comprises The first directory is traversed in a cleanable manner to obtain subdirectories of the first directory. 16. The storage file management method according to claim 15, wherein traversing the first directory in a cleanable manner comprises: The first directory is opened with an open function, and the first directory is read with a read function. 17. The storage file management method according to any one of claims 1 to 16, further comprising: When the latest modification time of the first directory taken from the stack has not changed, acquire the subdirectories of the first directory from the database. 18. A stored file management device, characterized in that it comprises: The first acquisition module is used to obtain the subdirectories of the first directory when the latest modification time of the first directory taken out from the stack changes; the first directory is stored on the AI platform; The first judging module is used to judge whether the subdirectory is empty; The first storage module is used to put the subdirectory into the stack if the subdirectory is not empty; The second storage module is used to put the subdirectory and the parent directory of the subdirectory that meet the preset condition into the queue if the subdirectory is empty; the preset condition is the parent directory of the parent directory All subdirectories are put into the queue; The removal and processing module is used to take out the second directory from the queue until the directory in the stack and the queue is empty, and according to the change of the latest modification time of the second directory, perform corresponding operations on the database used to store the directory operation. 19. An AI platform, characterized in that, comprising: memory for storing computer programs; A processor, configured to implement the steps of the storage file management method according to any one of claims 1 to 17 when executing the computer program. 20. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the storage file according to any one of claims 1 to 17 is realized. The steps of the management method.

Images