CN107291878A - A kind of data-erasure method and device of distributed storage file system - Google Patents

Abstract

This application discloses a kind of data-erasure method of distributed storage file system, methods described includes：The term of validity label of data is set；By term of validity label storage into detection table；The detection table is traveled through in setting time, the label of exhaustion of effect in the detection table is filtered out；Delete label data corresponding with its of the exhaustion of effect in the detection table.By the above method, solving invalid data in distributed storage file system can only artificial screening and the problem of high inefficiency maloperation risk caused by deleting.Disclosed herein as well is a kind of data deletion apparatus of distributed storage file system, with above-mentioned beneficial effect.

Description

A method and device for deleting data in a distributed storage file system

technical field

The present application relates to the field of data storage, in particular to a data deletion method and device in a distributed storage file system.

Background technique

In the era of big data, the role of data information is increasing. Inevitably, data storage is a crucial issue. In the process of data storage, there will always be some invalid data, which will occupy a considerable part of storage resources, and will also interfere with valid data, seriously affecting people's use of data resources in the database and the results of data analysis.

Currently the most popular storage system is undoubtedly a distributed file storage system based on a distributed file storage architecture, which is mainly used to store data such as documents, images, and videos. The system manages system resources in a global manner, and it can schedule storage resources in the cluster arbitrarily, and the scheduling process is "transparent".

The distributed storage system adopts a scalable system structure, which not only improves the reliability, availability and access efficiency of the system, but also is easy to expand. A distributed storage system with high performance and high capacity as its main features generally meets the following four conditions: it is applied in a network environment; the data of a single file is distributed and stored on different nodes; it supports concurrent access by multiple terminals and multiple processes; Provides a unified directory space and access name.

However, at present, in the distributed storage file system, invalid data is mainly screened and deleted manually, which is inefficient and has the risk of misoperation. The content of the database is huge, and manual deletion of invalid data will inevitably lead to a high risk of misoperation due to factors such as work fatigue.

Contents of the invention

In view of this, this application provides a method and device for data deletion. By setting the validity period for the data, the data detection finds and deletes the expired data, which solves the inefficiency and high risk of manual screening and deletion of invalid data in the prior art. question. The method is as follows:

Set the validity period label of the data;

Store the expiration date label in the detection table;

traversing the detection table at a set time, and filtering out tags whose validity period has expired in the detection table;

Delete the expired label and its corresponding data in the detection table.

Wherein, the traversing the detection table at the set time includes:

The set time is obtained through linear regression calculation using the validity period labels of the remaining data in the detection table.

Wherein, after the deletion of the expired label and its corresponding data in the detection table, it also includes:

After the set time is calculated again using the linear regression method, the set time is used to correct the previously calculated set time.

Wherein, the deletion of the data label and its corresponding data in the detection table includes:

Call the deletion program to delete the expired label and its corresponding data in the detection table.

The present application also includes a data deletion device for a distributed storage file system, the device comprising:

A label setting module is used to set the validity period label of the data;

A label mapping module, configured to store the validity period label in a detection table;

A label traversal module, configured to traverse the detection table at a set time, and filter out labels whose validity period has expired in the detection table;

A data deletion module, configured to delete the expired label and its corresponding data in the detection table.

Wherein, the label traversal module includes:

The time calculation sub-module is used to calculate the time for traversing the inspection table through linear regression method according to the validity period of the remaining data in the inspection table.

Wherein, the label traversal module also includes:

The time correction sub-module is used to correct the previously calculated set time by using the set time after calculating the set time again by using the linear regression method.

Wherein, the data deletion module is specifically configured to call a deletion program to delete the data tag and its corresponding data in the detection table.

The present application provides a method for deleting data in a distributed file storage system. The method sets a validity period for the data, detects the validity period of the data at a preset time, and deletes the data when the validity period expires. It enables the data storage system to automatically filter out invalid data and delete it, avoiding the low efficiency and high risk caused by manual operation. The present application also provides a data deletion device of a distributed file storage system, which has the above-mentioned beneficial effects, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a flowchart of a data deletion method of a distributed file storage system provided by an embodiment of the present application;

FIG. 2 is a flow chart of another data deletion method of a distributed file storage system provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a data deletion device of a distributed file storage system provided by an embodiment of the present application.

detailed description

The core of the present invention is to provide a method and device for deleting invalid data in a distributed file storage system, so that the system can automatically screen and delete invalid data, and solve the problem of low efficiency and high risk caused by manual screening of invalid data.

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment one

Please refer to FIG. 1 . FIG. 1 is a method for deleting invalid data in a distributed file storage system provided by the present application. The specific method is as follows:

S101: Setting a validity period label of the data.

Set the validity period label for the data in the distributed file storage system, and the type and form are not limited. For example, a separate label data can be set for a certain data, including one or several types of information such as data validity period, data type, data size, data location, etc., or it can be used as an attribute of the original data and added to the original data physical location or logical location.

S102: Store the expiration date label in a detection table;

Create a test table in the system, or use the existing table type data in the system to store the validity period label set in S101 in the test table. The storage here can be to move the label that has been set, or to copy a label to the detection table. From the perspective of the amount of data stored in the system, because the tag already contains its corresponding data information, the data can be addressed through the tag, so moving the tag directly to the detection table will save space.

S103: traverse the detection table at a set time, and filter out the labels whose validity period has expired in the detection table;

The set time can be calculated according to a mathematical algorithm, for example, the next traversal time is obtained as the set time by processing existing data tags according to a linear regression method. In addition, after the corrected set time is calculated by using the linear regression method again, the set time is corrected by using the corrected set time. The advantage of doing this is that it can ensure that the obtained calibration setting time is the optimal traversal time, and can minimize the consumption rate of system resources. You can also manually set a period, such as traversing once every three days, etc., or set a shorter period when the amount of data is large, and set a longer period when the amount of data is small, which can save certain resources.

When traversing the detection table, it can be checked whether the validity period of each label has expired after it is stored in the detection table, or it can be tested after all the labels to be detected are stored in the detection table, or it can be a Tags with different attributes are stored for detection and block detection of data is carried out. The same attribute here includes any one of attributes such as date, data type, data length, and data storage location.

Screening refers to selecting all the expired data. "Select" can store all the expired tags in a certain storage area for centralized processing, or find the expired tags when the system traverses the validity period of the detected data tags. When the data is processed, it is processed immediately; it can also be marked when the validity period of the label is found, and the marked labels are processed in batches after the traversal is completed.

S104: Delete the expired label and its corresponding data in the detection table.

After traversing all the data in the detection table, no matter which screening method in S103 is used, all tags whose expiration date can always be obtained. At this point, these expired and invalid data need to be deleted from the storage system. Generally, it is deleted manually or by machine. Machine deletion is to call the deletion program to completely delete all the labels and the data corresponding to the labels that have expired in the system. The advantage of this is that it has high efficiency, extremely low error rate, and eliminates manual operations, making it more automated and intelligent.

Embodiment two

Referring to FIG. 2, FIG. 2 is a flowchart of the second embodiment provided by the present application. On the basis of the above embodiments, the algorithm for setting the time is specified as a linear regression method, and its ergodic period is described in more detail. The specific method is as follows:

S201: Set a validity period label of the data.

A validity period label is set for the data in the distributed file storage system, and the label includes the name of the data, the logical location information of the data and the validity period of the data. In this way, the actual address of the data can be addressed through the logical location information on the label, so as to facilitate subsequent operations.

S202: Store the expiration date label in a detection table.

Create a new blank table data in the system as the detection table, and move the validity period label set in S201 to the blank table. The labels in the table are stored according to the address relationship of their corresponding data. For example, both data 1 and data 2 are files under folder A, then the positional relationship between data 1 and data 2 in the detection table should also be stored side by side in the detection table. The advantage of this is that once the system makes an error when setting the validity period label, so that the address corresponding to a certain label cannot be found in subsequent operations, the corresponding data can be judged by the labels around the label, which provides the fault tolerance rate of the entire system.

S203: Calculate the traversal time.

The traversal time of the system can be obtained through the linear regression method. This process does not depend on the setting of the validity period label. We only need to know the validity period of each data. The abscissa of the statistical chart is the validity period, and the ordinate is the label quantity, and a scatter diagram can be obtained. According to the scatter diagram system, an optimal time can be obtained to traverse the data in the detection table. The so-called optimal means that the amount of data that can be deleted within a unit is the largest. For example, if there are 20 pieces of data whose validity period expires on the 2nd, but only one piece of data expires on the 1st, the system will preferably select the expired data whose validity period expires in the traversal detection table on the 2nd. However, if the program traversal is run simultaneously on the 1st and 2nd, it will cause unnecessary waste of resources. Of course, if there is also a lot of data on No. 1, the optimal time can be calculated through the linear regression algorithm, and the nearest optimal time will be used as the start time of the next traversal.

S204: Correct the traversal time.

In addition, because the linear regression method needs to be used for calculation many times, it is obvious that the traversal time calculated based on the latest remaining data must be optimal. Therefore, after each calculation using the linear regression method, the latest calculation results are used to replace the previous optimal time. This adaptive time correction reduces the performance overhead of the data filtering process.

S205: Screen the test form.

traversing the detection table, and filtering out labels whose validity period has expired in the detection table;

When traversing the detection table, because the amount of data in the distributed file storage system is extremely large, block traversal is used. The data of the system is classified by address, so there may be multiple corresponding detection tables, which are not limited in this application. After storing all the labels of a certain block in the detection table, start to traverse the validity period of the labels in the detection table.

During the screening process, a mark is made on the tag data whose validity period has expired, which is only used to distinguish between tags whose validity period has not expired and tags whose validity period has expired. After the traversal ends, the screening process ends at the same time, and all tags that have expired are marked, and the next step proceeds.

S206: Delete the label whose validity period has expired and its corresponding data in the detection table.

The delete program in the system is invoked to delete the expired tags marked in S203 and their corresponding data.

S207: Determine whether the detection table is empty, if not, return to step S203; if yes, end the process.

For a system, it is necessary to filter and automatically delete invalid data periodically. The "periodical" mentioned here is not strictly limited to the process of a fixed time period, but refers to the need for this process to occur repeatedly.

Embodiment three

The present application also provides a data deletion device of a distributed file storage system, the device comprising:

The label setting module 100 is used to set the validity period label of the data;

A label mapping module 200, configured to store the validity period label in a detection table;

The label traversal module 300 is used to traverse the detection table at a set time, and filter out the labels whose validity period has expired in the detection table;

In the label traversal module, a time calculation sub-module and a time correction sub-module may also be included. The time calculation sub-module is used to calculate the time for traversing the inspection table through linear regression according to the validity period of the remaining data in the detection table; the time correction sub-module is used to calculate the set time by using the linear regression method again, and use the set time to correct the original time. The calculated set time is corrected.

The data deletion module 400 is configured to delete the expired tags and their corresponding data in the detection table.

Therefore, the workflow of the device is as follows:

1. The label setting module sets validity period labels for the data in the file storage system.

The label setting module sets validity period labels for the data in the distributed file storage system, and the types and forms are not limited. For example, a separate label data can be set for a certain data, including one or several types of information such as data validity period, data type, data size, data location, etc., or it can be used as an attribute of the original data and added to the original data physical location or logical location.

2. The label mapping module stores the set validity period label in the detection table.

Create a detection table in the system, or use the existing table type data in the system, and the label mapping module stores the validity period label set in 1 in the detection table. The storage here can be to move the label that has been set, or to copy a label to the detection table. From the perspective of the amount of data stored in the system, because the tag already contains its corresponding data information, the data can be addressed through the tag, so moving the tag directly to the detection table will save space.

3. The label traversal module traverses the detection table at a set time, and screens out all labels whose validity period has expired. Among them, the time calculation sub-module calculates the time for traversing the inspection table through linear regression method according to the validity period of the remaining data in the detection table, and the time correction sub-module uses the time calculated by the time calculation sub-module to correct the previously calculated set time .

The set time can be calculated according to a mathematical algorithm, for example, the next traversal time is obtained as the set time by processing existing data tags according to a linear regression method. In addition, after the corrected set time is calculated by using the linear regression method again, the set time is corrected by using the corrected set time. The advantage of doing this is that it can ensure that the obtained calibration setting time is the optimal traversal time, and can minimize the consumption rate of system resources. If there is no time calculation sub-module and time correction sub-module in the device, a cycle can also be manually set, such as traversing once every three days, etc., or a shorter cycle can be set when the amount of data is large, and a longer cycle can be set when the amount of data is small , can save certain resources.

When traversing the detection table, it can be checked whether the validity period of each label has expired after it is stored in the detection table, or it can be detected after all the labels to be detected are stored in the detection table, or it can be shared The label of an attribute is stored and detected, and then the data is segmented and detected. The same attribute here includes any one of attributes such as date, data type, data length, and data storage location.

Screening refers to selecting all the expired data. "Select" can store all the expired tags in a certain storage area for centralized processing, or find the expired tags when the system traverses the validity period of the detected data tags. When the data is processed, it is processed immediately; it can also be marked when the validity period of the label is found, and the marked labels are processed in batches after the traversal is completed.

4. The data deletion module deletes all tags and their corresponding data that are screened out in 3 and have expired.

The data deletion module will call the deletion program to completely delete all expired tags and their corresponding data filtered out in 3. You can also prompt the user to delete manually.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. It should also be noted that in this specification, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations There is no such actual relationship or order between the operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (8)

1. A data deletion method of a distributed storage file system, characterized in that the method comprises: Set the validity period label of the data; storing the expiration date label in a detection table; traversing the detection table at a set time, and filtering out tags whose validity period has expired in the detection table; Delete the expired label and its corresponding data in the detection table. 2. The method according to claim 1, wherein traversing the detection table at the set time comprises: The set time is obtained through linear regression calculation using the validity period labels of the remaining data in the detection table. 3. The method according to claim 2, wherein, after deleting the expired label and its corresponding data in the detection table, further comprising: After calculating the corrected set time by using the linear regression method again, the set time is corrected by using the corrected set time. 4. The method according to claim 1, wherein deleting the data label and its corresponding data in the detection table comprises: Call the deletion program to delete the expired label and its corresponding data in the detection table. 5. A data deletion device of a distributed storage file system, characterized in that the device comprises: A label setting module is used to set the validity period label of the data; A tag mapping module, configured to store the validity period tag into a detection table; A label traversal module, configured to traverse the detection table at a set time, and filter out labels whose validity period has expired in the detection table; A data deletion module, configured to delete the expired tags and their corresponding data in the detection table. 6. The device according to claim 5, wherein the label traversal module comprises: The time calculation sub-module is used to use the validity period labels of the remaining data in the detection table to calculate the set time through linear regression. 7. The device according to claim 6, wherein the label traversal module further comprises: The time correction sub-module is used to correct the set time by using the corrected set time after calculating the corrected set time again by using the linear regression method. 8 . The device according to claim 5 , wherein the data deletion module is specifically a module that calls a deletion program to delete the expired label and its corresponding data in the detection table.