CN117193656A - A data hierarchical storage migration process method - Google Patents

Abstract

The present invention provides a data hierarchical storage migration process method, which relates to the field of data storage technology; the present invention realizes dynamic management of data in multi-level storage devices between various levels of storage devices by performing a data storage migration process based on multi-level storage devices. ; Establish a standard and standardized process management method for hierarchical storage of massive data, solve the unreasonable factors such as subjectivity and randomness in the management of hierarchical storage devices of massive data, and realize the online, near-line and offline operation of massive data within the life cycle. Efficient organization and dynamic storage management of storage resources improve data security and service efficiency.

Description

A data hierarchical storage migration process method

Technical field

The present invention relates to the field of information technology, and in particular to a data hierarchical storage migration process method.

Background technique

At present, with the continuous in-depth application of information technology in all walks of life, the amount of accumulated data is also increasing. A large number of storage devices with different performance and application levels are deployed in data centers of industries or departments with relatively concentrated data. Massive amounts of Data storage management has become a bottleneck problem facing the informatization work of some industries and departments.

One of the outstanding manifestations is the vague storage hierarchical management of various types of data: dynamic storage management such as data migration in and out of different levels of storage devices lacks a standard and scientific process management method, which is based on full life cycle data storage management. The process is unscientific and imperfect. The above problems have caused some subjective and arbitrary factors in data storage management. These factors have also caused security issues in data management and efficiency issues in data services, thus overall affecting the improvement of data service capabilities and application value. Therefore, while providing data management and services, information construction in various industries urgently needs to establish scientific and effective storage management process methods, make full use of online, near-line and offline hierarchical storage resources, and do a good job in dynamic storage management of various data life cycles. , ensure data security and improve data management and service efficiency.

Contents of the invention

The purpose of the present invention is to provide a data hierarchical storage migration process method, thereby solving the aforementioned problems existing in the prior art.

In order to achieve the above objects, the technical solutions adopted by the present invention are as follows:

A data hierarchical storage migration process method, which performs a data storage migration process based on multi-level storage devices to realize dynamic management of data in multi-level storage devices between storage devices at all levels; wherein, the storage hierarchies of the multi-level storage devices It refers to the storage categories divided according to different device access methods, access performance and device security. From high to low, it is divided into first-level storage devices, second-level storage devices,..., M-th level storage devices; the storage migration process It includes a data upgrading process and a data downgrading process. The data upgrading process refers to the migration of data in low-level storage devices to higher-level storage devices according to the upgrading rules. The data downgrading process refers to the data in high-level storage devices. The data is migrated to lower-level storage devices according to the downgrade rules;

The data upgrading process includes an upgrading process based on user needs, specifically including the following steps:

S1, for user access to data X, confirm the specific location of data If it is in the storage device, go to step S3; if it is in the promotion queue, go to step S4;

S2, confirm whether the downgrading process of data

S3, re-use the value evaluation method to evaluate data X, adjust the data value sorting in lower-level storage devices, place data In the storage device, enter step S5;

S4, confirm whether data X has completed the data upgrading process. If it has not been migrated, place data If the migration has been completed, proceed directly to step S5;

S5, data X remains in a premium storage device;

The data value assessment method specifically includes value assessment through formula (1):

Among them, IP(X) represents the inherent attributes of the data file and migration costs.

Preferably, the data migration process includes active migration and forced migration, wherein the active migration includes the following steps:

When the value of part of the data stored in a higher-level storage device is lower than the value of part of the data stored in a lower-level storage device, the migration condition is triggered and the downgrade operation is performed according to the downgrade rules until all the data in the higher-level storage device is The data value is higher than all the data in the lower-level storage device, and the downgrade process ends;

The forced downgrade includes the following: when the proportion of the storage device capacity in the entire storage space is higher than the set high water level threshold, the migration condition is triggered, and the storage device performs data migration according to the downgrade rules until the storage capacity is lower than the high water level threshold. water level threshold.

Preferably, the downgrade rules specifically include:

Use the data value assessment method to evaluate the value of all the stored data in the storage device to be migrated, and migrate the data in order from low to high data value, migrating data with lower value to lower-level storage devices.

Preferably, the promotion process includes active promotion, specifically including the following steps:

When the value of part of the data stored in the lower-level storage device is higher than the value of part of the data stored in the higher-level storage device, the migration condition is triggered and the upgrading operation is performed according to the upsizing rules until the value of all data in the lower-level storage device is equal to that of the data stored in the lower-level storage device. With all data in the storage device below the higher level, the upsizing process ends.

Preferably, the upgrading rules are specifically: use a data value evaluation method to evaluate the value of all stored data in lower-level storage devices, and perform upgrading operations in the order of data value from high to low, and move data with higher value to Migrate to a higher level storage device.

Preferably, the calculation formula of the file access time T is as shown in Formula 2:

Among them, Ti refers to the total time for a certain user to access (including modify) the file, and N represents the number of users.

The calculation formula of data reading and writing frequency F is as shown in formula (3):

Among them, Ri and Wi respectively represent the frequency of reading and writing of the file by a certain user, expressed by the reciprocal of the access (including modification) interval time; N represents the number of users.

The number of users accessing the file C is calculated using formula (4):

Among them, Ui represents the record of a certain user accessing the file; N represents the number of users.

The data correlation R refers to the correlation between data files, including but not limited to: topic correlation, time version correlation, achievement form correlation, and causal relationship correlation;

The storage cost and migration cost assessment Q includes but is not limited to: accounting for investment in current or future storage equipment, labor costs of storage migration, impact on business systems, and data migration risk costs.

The beneficial effects of the present invention are:

The present invention provides a data hierarchical storage migration process method, which solves the problem of online, near-line and offline hierarchical storage of massive data by establishing a standard and standardized process management method for hierarchical storage of massive data, especially a data upgrading process based on user needs. Unreasonable factors such as subjectivity and randomness in management can be eliminated to achieve efficient organization and dynamic storage management of online, near-line and offline storage resources in the entire life cycle of massive data, and improve data security and service efficiency.

Description of the drawings

Figure 1 is a schematic diagram of the principle of the data hierarchical storage migration process method provided in the embodiment;

Figure 2 is a management diagram of the data storage upgrade and migration process based on user needs.

Figure 3 is a management diagram of the data storage upgrade and migration process;

Figure 4 is a management diagram of the data storage downgrade and migration process;;

Figure 5 is a schematic diagram of the high and low water level method based on storage space;

Figure 6 is a storage sequence diagram of various types of result data;

Figure 7 is a schematic diagram of location monitoring for hierarchical data storage;

Figure 8 is a comprehensive application diagram of storage data at all levels;

Figure 9 is a schematic diagram of monitoring the migration process of data hierarchical storage.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

Example

This embodiment provides a data hierarchical storage migration process method, which targets the online storage devices, near-line storage devices and offline storage devices of the data center to realize the dynamic migration of data in multi-level storage devices between storage devices at all levels. Management; specifically includes the data upgrading process and the data downgrading process, as shown in Figure 1. The data upgrading process refers to the migration of data in low-level storage devices to higher-level storage devices according to the upgrading rules, including offline storage ( Level 3) is upgraded to near-line storage (Level 2); near-line storage (Level 2) is upgraded to online storage (Level 1); offline storage (Level 3) is upgraded to online storage (Level 1). The data migration process refers to the migration of data in high-level storage devices to lower-level storage devices according to the downgrade rules. Online storage (level one) is downgraded to near-line storage (level two); near-line storage (level two) ) is downgraded to offline storage (Level 3); online storage (Level 1) is downgraded to offline storage (Level 3).

Based on the upgrading process based on user needs, at specific times and under special circumstances, users will strengthen the use and access of data on a certain level of storage devices. The value of this part of the data is highlighted. In order to improve service efficiency, this part of the data needs to be upgraded. For this part of the data requested by users, data storage upgrade must be carried out according to different management and storage situations. The migration process management is shown in Figure 2, which specifically includes the following steps:

S1, for user access to data X, confirm the specific location of data If it is in the storage device, go to step S3; if it is in the promotion queue, go to step S4;

S2, confirm whether the downgrading process of data

S3, re-use the value evaluation method to evaluate data X, adjust the data value sorting in lower-level storage devices, place data In the storage device, enter step S5;

S4, confirm whether data X has completed the data upgrading process. If it has not been migrated, place data If the migration has been completed, proceed directly to step S5;

S5, data X remains in premium storage device.

For the data downgrade process, as shown in Figure 3, it can include active downgrade and forced downgrade according to the downgrade conditions. The condition for active downgrade is that the data application level is lowered and the data does not meet the data value standard of the storage level. , specifically including the following steps:

When the value of part of the data stored in a higher-level storage device is lower than the value of part of the data stored in a lower-level storage device, the migration condition is triggered and the downgrade operation is performed according to the downgrade rules until all the data in the higher-level storage device is The data value is higher than all the data in the lower-level storage device, and the downgrade process is over; use the data value evaluation method to evaluate the value of all the storage data in the storage device to be migrated, and conduct data evaluation in order from low to high data value. Migration, migrating less valuable data to lower-level storage devices.

The conditions for the forced downgrade are that the storage space on the storage device is full or nearly full, and the data is forced to be migrated. Specifically, the conditions include the following: According to the high and low water level method (as shown in Figure 5), when the storage device capacity occupies 10% of the entire storage space, When the capacity ratio is higher than the set high water level threshold, the migration condition is triggered, and the storage device performs data migration according to the downgrade rules until the storage capacity is lower than the high water level threshold.

For the upgrading process, there are two upgrading migration conditions: due to the improvement of the data application level, that is, the active upgrading of data value exceeding the data standard of the current storage level and the upgrading process of activating data implementation based on access requests to the data:

Among them, the active upgrading process is shown in Figure 4, which specifically includes: when the value of part of the stored data in the lower-level storage device is higher than the value of part of the stored data in the higher-level storage device, the migration conditions are triggered and the upgrading rules are followed. The upsizing operation is completed until the value of all data in the lower-level storage device is lower than that of all data in the higher-level storage device.

The specific upgrading rules adopted are as follows: Use the data value evaluation method to evaluate the value of all the stored data in lower-level storage devices, and perform upgrading operations in order from high to low data value, migrating data with higher value. to a higher level storage device.

The data value assessment method used in this embodiment specifically includes value assessment through formula (1):

The data value assessment method specifically includes value assessment through formula (1):

Among them, IP(X) represents the inherent attributes of the data file and migration costs.

The calculation formula of file access time T is as shown in formula (2):

Among them, Ti refers to the total time for a certain user to access (including modify) the file, and N represents the number of users.

The calculation formula of data reading and writing frequency F is as shown in formula (3):

Among them, Ri and Wi respectively represent the frequency of reading and writing of the file by a certain user, expressed by the reciprocal of the access (including modification) interval time; N represents the number of users.

The number of users accessing the file C is calculated using formula (4):

Among them, Ui represents the record of a certain user accessing the file; N represents the number of users.

Data relevance R and storage cost and migration cost assessment Q, where data relevance R is the relationship between data, including: topic relevance, time version relevance, result form relevance, causality relevance, etc. Supervisor or objective contact. Storage cost and migration cost assessment Q is the calculation of current or future storage equipment investment, labor cost of storage migration, impact on business systems, data migration risk cost, etc.

This embodiment takes the dynamic storage management of various types of data in the data center of the Ministry of Natural Resources as an example, and uses the storage management of the second national land survey results data (including many types of data) as a typical application to illustrate the application of this process method in natural resources. The role and effect of various types of data management in the Ministry of Data Center.

1. Storage capacity estimation of existing hierarchical storage equipment and water level threshold setting

At present, the online, near-line and offline storage capacities of the Ministry of Natural Resources data center have reached 120TB, 300TB and 600TB respectively. Considering the current situation of data management and services in the data center, the high, medium and low-level storage of the above three-level storage The water level zone thresholds are defined as 85%, 50%, and 15%. That is, when the storage capacity exceeds 85%, some data will be forcibly migrated and no new data will be stored.

It is estimated that the current available online, near-line and offline storage capacities are approximately 42TB, 108TB and 320TB respectively, and there is a certain storage margin.

2. Carry out data value assessment and hierarchical storage

Taking the data reception and storage management of the second national land survey results as an example, we carry out hierarchical data storage management. The results of the Second National Land Survey are divided by data type, including: raster image data, spatial vector data, document results, etc.; by the processing stage of the results, they include: original results, intermediate results and final results. From the perspective of access frequency and application value, the final result data is the largest, followed by intermediate results, and the original result is the smallest. In terms of data size, raster image data is the largest, followed by space vector data, and document results are the smallest.

Based on the analysis of the existing storage capacity in the first stage, the value of various result data of the second national land survey data is estimated. The evaluation content includes: the number of natural resource supervision business users that can provide services and the frequency of access, Combined with the characteristics of various types of data (data volume, update cycle), the data value assessment method based on the information life cycle is used for qualitative or quantitative analysis, and the hierarchical storage order of the second national land survey results data is determined from high to low as follows : final results, intermediate results, original results, and the hierarchical storage order of each type of results from high to low is document results, space vector results, and image results. The application value and hierarchical storage sequence of the above types of achievement data are shown in Figure 6.

3. Carry out the implementation of hierarchical data storage

Based on the storage hierarchical priorities established above and combined with the existing storage capacity estimates of all levels of storage in the data center, the 5G document data, 200G spatial vector data, and 2TB raster image data in the final result data will be stored in online storage. , so that the total amount of data stored online is 80.205T, accounting for approximately 66.84% of the total storage capacity, not exceeding 85%.

The 3G document data, 100GB spatial vector data, and 20TB raster image data in the intermediate result data are all stored in the near-line storage device, making the total amount of near-line storage data 212.103T, accounting for approximately 70.70% of the total storage capacity. , no more than 85%.

The original result data mainly includes some image data, with a data volume of about 18TB. Considering that data services cannot be directly provided, it is stored in an offline storage device, making the total amount of offline stored data 298T, accounting for approximately 49.67% of the total storage capacity. No more than 85%.

Figures 7, 8, and 9 are renderings after the implementation of hierarchical storage.

By adopting the above technical solutions disclosed in the present invention, the following beneficial effects are obtained:

Based on the characteristics of the storage device itself and natural resource big data, this invention proposes a standard process and method for storage and migration of natural resource big data, standardizes the dynamic storage management method of natural resource big data, and solves the following problems:

First, it solves the standardization and standardization issues of dynamic management of natural resource data storage. This method is based on storage cost accounting and data value assessment methods, and establishes a standardized process for natural resource data full life cycle storage management. It provides a normative and standard process method for dynamic storage management of data at different life cycle stages. It avoids the subjectivity and randomness in the data storage management process and reduces the risks in data storage management.

Second, it saves storage costs and improves the utilization efficiency of storage equipment. This process method standardizes the storage management method of data in different life cycle stages, achieves reasonable and dynamic allocation of storage resources, maximizes the characteristics of different types of storage devices, saves storage costs, improves the utilization efficiency of storage devices, and achieves Efficient utilization of storage resources.

Third, it improves the management and service efficiency of various types of natural resource data. This process method can realize the dynamic management of different types of data on different types of storage by providing standard and standardized process methods based on the application characteristics of various types of natural resource data at different life cycle stages, ensuring the service efficiency of different types of data. Improved the management level and service efficiency of various types of natural resource data.

The above are only preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications can also be made. The scope of protection of the present invention should be considered.

Claims (6)

1. The data hierarchical storage migration flow method is characterized in that a data storage migration process is carried out based on multi-stage storage equipment, so that dynamic management of data in the multi-stage storage equipment among all stages of storage equipment is realized; the storage classification of the multi-level storage device refers to storage categories which are divided into a first-level storage device, a second-level storage device, … and an Mth-level storage device from high to low according to different access modes, access performances and device safety of the device; the storage migration process comprises a data migration process and a data migration process, wherein the data migration process refers to that data in the low-level storage equipment is migrated to the storage equipment at a higher level according to a migration rule, and the data migration process refers to that the data in the high-level storage equipment is migrated to the storage equipment at a lower level according to the migration rule;

the data migration process comprises a migration process based on user requirements, and specifically comprises the following steps:

s1, for user access data X, confirming the specific position of the data X, if the data X is located in the highest-level storage device, not migrating, and directly entering step S5; if the storage device is positioned in the descending queue, the step S2 is performed, and if the storage device is positioned in the lower-level storage device, the step S3 is performed; if the queue is in the transition queue, the step S4 is entered;

s2, confirming whether the data X descent process is completed or not, and if so, entering a step S3; if the migration is not completed, stopping the migration and moving out of the migration queue, and entering step S5, wherein the data X is reserved in the advanced storage equipment;

s3, evaluating the data X by adopting a value evaluation method again, adjusting the data value sequence in the lower-level storage equipment, placing the data X in the front part of the transition queue to perform a data transition process, entering the higher-level storage equipment after the data transition process is finished, and entering a step S5;

s4, confirming whether the data X completes the data migration process, if not, placing the data X at the front part of a migration queue, and continuing the migration process; if the transition is in the process of transition, accelerating the process of transition until the transition is completed, and entering step S5; if the migration is completed, directly entering step S5;

s5, the data X is reserved in the advanced storage device;

the data value evaluation method specifically comprises the following steps of evaluating the value through a formula (1):

wherein, IP (X) represents the inherent attribute of the data file X, S represents the file size, T represents the file access time, F is the data read-write frequency, C is the number of access users of the file, R is the data association degree, and Q is the storage cost and migration cost evaluation.

2. The data hierarchical storage migration flow method of claim 1, wherein the data migration process comprises an active descent and a forced descent, wherein the active descent comprises the steps of:

when the value of part of the stored data in the higher-level storage device is lower than that of part of the stored data in the lower-level storage device, triggering migration conditions, and performing migration operation according to a migration rule until all the data values in the higher-level storage device are higher than all the data in the lower-level storage device, and ending the migration process;

the forced drop comprises the following: when the capacity proportion of the capacity of the storage device to the whole storage space is higher than a set high water level threshold value, triggering a migration condition, and performing data migration by the storage device according to a migration rule until the storage capacity is lower than the high water level threshold value.

3. The method of data hierarchical storage migration flow according to claim 2, wherein the migration rule specifically includes:

and performing value evaluation on all the stored data in the storage equipment to be migrated by adopting a data value evaluation method, performing data migration according to the sequence of low data value to high data value, and migrating the data with lower value to the storage equipment with lower level.

4. The method of data hierarchical storage migration process according to claim 1, wherein the data migration process further comprises active migration, specifically comprising the steps of:

when the value of part of the stored data in the lower-level storage device is higher than that of part of the stored data in the higher-level storage device, triggering migration conditions, and performing migration operation according to an active migration rule until all the data values in the lower-level storage device are lower than all the data in the higher-level storage device, and ending the migration process.

5. The method of data hierarchical storage migration flow according to claim 4, wherein the active migration rule specifically comprises: and carrying out value evaluation on all stored data in the lower-level storage equipment by adopting a data value evaluation method, carrying out migration operation according to the sequence of the data value from high to low, and migrating the data with higher value into the higher-level storage equipment.

6. The data hierarchical storage migration flow method according to claim 1, wherein in the data value evaluation formula, a calculation formula of the file access time T is as shown in formula (2):

where Ti refers to the sum of the times a certain user accesses (including modifies) the file, and N represents the number of users;

the calculation formula of the data read-write frequency F is shown in formula (3):

wherein, ri, wi respectively represent the frequency of reading and writing the file by a certain user, and are represented by the reciprocal of the access (including modification) interval time; n represents the number of users;

the number of access users C of the file is calculated by adopting a formula (4):

where Ui represents a record of a certain user accessing the file; n represents the number of users;

the data association degree R refers to association relations between data files, including but not limited to: thematic relevance, time version relevance, achievement form relevance and causal relation relevance;

the storage cost and migration cost assessment Q includes, but is not limited to: investment accounting for current or future storage devices, labor costs for storage migration, impact on business systems, and data migration risk costs.