CN103078933A - Method and device for determining data migration time - Google Patents

Abstract

The present invention is applicable to the field of Internet communication, and provides a method for determining the timing of data migration, comprising the following steps: starting two threads, respectively monitoring the space usage rate and data migration period of the primary storage in the hierarchical storage system; the two threads, Periodically judge whether the data migration condition has been satisfied, if not, wait for the same time and then judge again until it is judged that the data migration condition is satisfied, if yes, execute data migration. The present invention can adjust the migration cycle according to different situations of migration, and can adapt to the dynamic change of the load in the cluster, thereby not only making full use of the first-level storage space in the hierarchical storage system, but also ensuring that the system adjusts the data location in time, Make the access performance of the system reach the best.

Description

A method and device for determining the timing of data migration

technical field

The invention belongs to the field of Internet communication, and in particular relates to a method and a device for determining the timing of data migration.

Background technique

With the explosive growth of data, clusters that store massive amounts of data emerge as the times require. Because the amount of data stored in the cluster is large, and some have reached the PB level, there may be a large amount of data access at the same time. Therefore, whether the data storage location is reasonable directly affects the data access delay.

Data migration refers to the movement of data between different storage media. To complete data migration, the system needs to consume certain resources (including hardware resources, bandwidth, etc.).

In a cluster that stores massive amounts of data, situations that require data migration mainly include system upgrades, data backup/recovery, and dynamic adjustment of data in hierarchical storage systems. Among them, system upgrades are large-scale and one-time; data backup/restoration is based on security considerations; usually, these two types of migrations occur rarely, and most of them must be completed at a certain moment.

However, the purpose of dynamic adjustment of data in the tiered storage system is to rationalize the configuration of the data location. Hierarchical storage is based on the "twenty-eight principle" of data activity: that is, 20% of the data is active and 80% of the data is inactive. The servers are divided into different storage levels according to hardware performance. The higher the storage level, the better the performance. the better. The goal of the hierarchical storage system is to store active data on high-level storage, but the activity of the data will change with time, so even if the data configuration at a certain time is reasonable, after a period of time, this The configuration will also appear outdated due to changes in data activity. Therefore, in order to achieve the goal of rational data configuration, data migration is required.

However, migration needs to consume resources. If the number of migrations is too small, the advantages of the hierarchical storage system cannot be fully utilized. If the number of migrations is too large, the system will have too many resources for internal consumption, which may reduce the system’s services to other businesses. quality. Therefore, the timing of migration is very important. It is necessary to ensure that the advantages of the tiered storage system can be utilized, and too many resources should not be used for internal consumption.

Currently, there are two methods for determining when to migrate. One is the fixed period method, that is, the tiered storage system will migrate data after a certain period of time, regardless of other situations; the other is the remaining space monitoring method, that is, monitor the remaining space of the first-level storage, if the remaining space is insufficient, then start the migration.

However, these two methods have disadvantages: if the period set by the fixed period method is too long, it will be difficult to take advantage of hierarchical storage; if it is too short, system resources will be frequently used for internal consumption. However, the load of the cluster is changing dynamically, and different applications access data in different situations. A reasonable cycle suitable for all application scenarios does not exist. That is to say, the fixed cycle method is difficult to adapt to the dynamic changes of the system load.

The remaining space monitoring method can be effective when there is a large amount of data written in the cluster, but in many cases, there will be no more data written in the cluster and frequent data reads, but the activity of the data is constantly increasing. change. In this case, the hierarchical storage system cannot perform data migration, nor can it achieve reasonable data configuration, and cannot play its due role.

Therefore, the method of determining the timing of migration needs to be rationalized. On the one hand, it must be able to adapt to the dynamic changes of the system load. On the other hand, the data can be stored on the appropriate node, so that the system can improve the efficiency of processing active data, and finally The data access performance is improved, and the overall access delay is reduced.

Contents of the invention

Embodiments of the present invention provide a method and device for determining the timing of data migration, aiming at solving the problem of currently being unable to dynamically select a suitable timing for data migration, ensuring that the advantages of the hierarchical storage system can be brought into play, and that too many resources will not be used for internal consumption The problem.

For this reason, the embodiment of the present invention provides following technical scheme:

A device for determining when to migrate data, comprising:

The monitoring module is connected with the judging module, and is used to start dual threads to monitor respectively the space utilization rate and the data migration period of the first-level storage in the hierarchical storage system;

A judging module is connected to the monitoring module and the data migration module respectively, and is used to regularly judge whether the data migration condition has been met through the dual threads;

A data migration module, configured to perform the data migration;

An adjustment module, connected to the data migration module, is used to adjust the data migration period.

The embodiment of the present invention also provides a method for determining the timing of data migration, including:

A: Start two threads to monitor the space usage rate and data migration cycle of the first-level storage in the tiered storage system respectively;

B: The dual thread regularly judges whether the data migration condition has been satisfied, if not, then waits for the same time before judging until it is judged that the data migration condition is satisfied, and if so, executes step C;

C: Execute the data migration;

D: Adjust the data migration cycle according to the migration situation;

E: After the data migration cycle is adjusted, the two threads re-read the parameters and continue the next round of monitoring.

Compared with the prior art, the embodiments of the present invention have the following advantages:

The embodiment of the present invention monitors the space usage rate and data migration period of the first-level storage in the hierarchical storage system respectively by starting dual threads, and regularly judges whether the data migration condition has been satisfied, and adjusts the migration condition according to the reason for satisfying the migration condition. Data migration cycle. The selected data migration timing is ensured, so that not only can the space of the first-level storage in the hierarchical storage system be fully utilized, but also ensure that the system adjusts the data location in time, so that the access performance of the system can reach the best.

Description of drawings

FIG. 1 is a flow chart of a method for determining the timing of data migration provided by the first embodiment of the present invention;

Fig. 2 is a structural diagram of an apparatus for determining a data migration opportunity provided by a second embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that what is described here is only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

FIG. 1 is a flow chart of a method for determining data migration timing provided by the first embodiment of the present invention. For ease of description, only parts related to the embodiment of the present invention are shown.

As shown in Figure 1, the method includes the following steps:

In step 101, two threads are started to respectively monitor the space usage rate and data migration period of the primary storage in the hierarchical storage system.

Step 102, the two threads periodically judge whether the data migration condition is met, if not, wait for the same time and then judge again until it is judged that the data migration condition is satisfied, if yes, execute step 103.

Specifically, the data migration condition includes that the space usage rate of the primary storage in the tiered storage system exceeds a set threshold or the data migration period arrives.

Specifically, when it is judged that the space usage rate of the primary storage in the tiered storage system does not exceed the set threshold and the data migration period has not yet arrived, the data migration is not triggered, and the judgment is made after the same time interval. When it is judged that the space usage rate of the primary storage in the tiered storage system exceeds the set threshold, or when the data migration period arrives, the migration is triggered.

Step 103, execute the data migration.

Preferably, if the migration is triggered, the system will record which background thread triggers the migration, and select a migration object to complete the migration.

Step 104, adjusting the data migration cycle according to the migration situation.

Specifically, when the reason for triggering the migration is that the space usage rate of the primary storage in the tiered storage system exceeds the set threshold, that is, the migration is caused by insufficient remaining space of the primary storage, which means that there is a large amount of data written at this time. Make timely adjustments to the data location and shorten the data migration cycle as the new data migration cycle.

Specifically, the migration caused by the exhaustion of the migration cycle time, but no actual data movement occurs, indicating that the system has no significant read and write operations during this cycle, and the migration cycle is extended as the new data migration cycle.

Specifically, due to the migration caused by the exhaustion of the migration cycle, there is normal data movement, and the original migration cycle remains unchanged.

Preferably, an upper limit and a lower limit are set for the data migration period. When the result of shortening the data migration cycle is lower than the lower limit value, use the lower limit value as the new data migration cycle, or, when the result of extending the data migration cycle is higher than the upper limit value, the upper limit value is used as the new data migration period.

Step 105, after the data migration period is adjusted, the two threads re-read the parameters and continue the next round of monitoring.

Preferably, after the data migration cycle is adjusted, the two threads re-read the updated data migration cycle parameters, and re-monitor whether the data migration condition is met.

Based on the same concept, the second embodiment of the present invention also provides a device for determining the timing of data migration. As shown in FIG.

The monitoring module 201 is connected with the judging module 202, and is used to start two threads, respectively monitor the space usage rate and the data migration period of the primary storage in the hierarchical storage system.

The judging module 202 is connected to the monitoring module 201 and the data migration module 203 respectively, and is used to periodically judge whether the data migration condition has been met through the dual threads.

Specifically, the data migration condition includes that the space usage rate of the primary storage in the tiered storage system exceeds a set threshold or the data migration period arrives.

Specifically, when it is judged that the space usage rate of the primary storage in the tiered storage system does not exceed the set threshold and the data migration period has not yet arrived, the data migration is not triggered, and the judgment is made after the same time interval. When it is judged that the space usage rate of the primary storage in the tiered storage system exceeds the set threshold, or when the data migration period arrives, the migration is triggered.

The data migration module 203 is used for performing the data migration.

Preferably, if the migration is triggered, the system will record which background thread triggers the migration, and select a migration object to complete the migration.

The adjustment module 204 is connected to the data migration module 203 and configured to adjust the data migration period.

Specifically, when the reason for triggering the migration is that the space usage rate of the primary storage in the tiered storage system exceeds the set threshold, that is, the migration is caused by insufficient remaining space of the primary storage, which means that there is a large amount of data written at this time. Make timely adjustments to the data location and shorten the data migration cycle as the new data migration cycle.

Specifically, the migration caused by the exhaustion of the migration cycle time, but no actual data movement occurs, indicating that the system has no significant read and write operations during this cycle, and the migration cycle is extended as the new data migration cycle.

Specifically, due to the migration caused by the exhaustion of the migration cycle, there is normal data movement, and the original migration cycle remains unchanged.

Preferably, an upper limit and a lower limit are set for the data migration period. When the result of shortening the data migration cycle is lower than the lower limit value, use the lower limit value as the new data migration cycle, or, when the result of extending the data migration cycle is higher than the upper limit value, the upper limit value is used as the new data migration period.

Preferably, the adjustment module 204 is connected to the monitoring module 201, and is used to start the two threads after the two threads re-read the parameters to monitor the space usage and data migration of any level of storage in the hierarchical storage system respectively. period, that is, return to step 101 in the first embodiment.

Preferably, after the data migration cycle is adjusted, the two threads re-read the updated data migration cycle parameters, and re-monitor whether the data migration condition is met.

Embodiments of the present invention provide a method and device for determining the timing of data migration, aiming at solving the problem of currently being unable to dynamically select a suitable timing for data migration, ensuring that the advantages of the hierarchical storage system can be brought into play, and that too many resources will not be used for internal consumption It can not only make full use of the first-level storage space in the hierarchical storage system, but also ensure that the system adjusts the data location in time, so that the access performance of the system can reach the best.

Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description in the embodiment, or can be located in one or more devices different from the embodiment according to corresponding changes. The modules in the above embodiments can be combined into one module, and can also be further split into multiple sub-modules.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is a better implementation Way. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to make a A terminal device (which may be a mobile phone, a personal computer, a server, or a network device, etc.) executes the methods described in various embodiments of the present invention.

The above is only a preferred embodiment of the present invention, it should be pointed out that, for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications can also be made. It should be regarded as the protection scope of the present invention.

Claims (6)

1. A method for determining data migration timing, comprising the steps of:

a: starting double threads, and respectively monitoring the space utilization rate and the data migration period of primary storage in the hierarchical storage system;

b: the double threads periodically judge whether the data migration condition is met, if not, the double threads wait for the same time and then judge until the data migration condition is met, and if yes, the step C is executed;

c: performing the data migration;

d: adjusting the data migration period according to the migration condition;

e: after the data migration period is adjusted, the double-thread reads parameters again and continues to monitor the next round.

2. The method for determining the data migration timing according to claim 1, wherein said periodically determining whether the data migration condition is satisfied in step B comprises:

b1: periodically judging whether the space utilization rate of primary storage in the hierarchical storage system exceeds a set threshold value or not;

and,

b2: and periodically judging whether the data migration period is reached.

3. The method of claim 2, wherein step D comprises:

d1: b1, shortening the data migration period as a new data migration period;

or,

d2: b2, if no actual data movement occurs, the data migration period is extended, and as a new data migration period, if normal data movement occurs, the original migration period is not changed.

4. The method for determining the data migration opportunity according to claim 3, wherein the step D further comprises:

d3: setting an upper limit value and a lower limit value for the data migration period;

d4: and when the result of shortening the data migration period is lower than the lower limit value, taking the lower limit value as a new data migration period, or when the result of prolonging the data migration period is higher than the upper limit value, taking the upper limit value as a new data migration period.

5. An apparatus for determining data migration opportunities, comprising:

the monitoring module is connected with the judging module and used for starting double threads and respectively monitoring the space utilization rate and the data migration period of primary storage in the hierarchical storage system;

the judging module is respectively connected with the monitoring module and the data migration module and is used for periodically judging whether the data migration condition is met or not through the double threads;

a data migration module for performing the data migration;

and the adjusting module is connected with the data migration module and used for adjusting the data migration period according to the migration condition.

6. The apparatus for determining a data migration opportunity according to claim 5, wherein the adjusting module is connected to the monitoring module, and configured to re-read the parameters for the dual thread after the adjustment of the data migration period, and continue the next round of monitoring.

Images