CN106445412A - Data volume compression ratio assessment method and system - Google Patents

Abstract

The invention discloses a method and system for evaluating the compression rate of a data volume. The method includes: setting n data sampling threads in advance, where n is an integer not less than 2; dividing the target volume into n different target volume areas, each The data sampling thread is in one-to-one correspondence with each target volume area; the corresponding target volume area is randomly sampled through each data sampling thread; each sampled data block is compressed and calculated by the preset compression algorithm; each sampled data block is correspondingly The compression calculation results are summarized to calculate the compression ratio evaluation result of the target volume. Before compressing the data volume, statistical methods are used to conduct random sampling analysis and evaluation of the compression rate of the target volume. The investigation is simple and cost-effective, ensuring the accuracy and reliability of the evaluation, reducing the occupation of system resources, and eliminating the need for long data During the compression process, the space savings of the target volume after compression can be estimated, providing data support for the user's decision whether to perform compression.

Description

Method and system for evaluating data volume compression ratio

technical field

The invention relates to the technical field of storage systems, in particular to a method and system for evaluating the compression ratio of data volumes.

Background technique

With the development of science and technology, people have entered the information age. In the information age, the most critical technology is data processing.

In the process of data processing, storage devices are often used to store data. However, with the advent of the era of big data, the amount of data is increasing, which requires improving the efficiency of the storage system. At present, common storage systems provide data compression functions, so as to occupy a large storage space during storage. The data blocks are compressed into data blocks that occupy relatively small storage space. However, data compression often takes a long time and takes up high CPU resources. For different types of data, the compression efficiency varies greatly. If some data blocks with low compression efficiency are compressed, the storage space saved is small, but it consumes a lot of time and CPU resources, which is often not worth the candle.

Therefore, how to know the space cost that can be saved after the data block is compressed, so as to judge whether the data block is worthy of data compression, is a technical problem that those skilled in the art need to solve at present.

Contents of the invention

The purpose of the present invention is to provide a method and system for evaluating the compression rate of a data volume, which can be used to know the space cost that can be saved after the data block is compressed before the data volume is compressed, so as to judge whether the data block is worth performing. Data compression provides users with decision-making basis.

In order to solve the problems of the technologies described above, the present invention provides the following technical solutions:

A method for evaluating the compression ratio of a data volume, comprising:

Pre-set n data sampling threads, n is an integer not less than 2;

Dividing the target volume into n different target volume areas, each of the data sampling threads is in one-to-one correspondence with each of the target volume areas;

performing random sampling on the corresponding target volume area through each of the data sampling threads;

Perform compression calculation on each sampled data block through a preset compression algorithm;

Summarize the corresponding compression calculation results of each sampled data block, and calculate the compression ratio evaluation result of the target volume.

Preferably, the random sampling of the corresponding target volume area by each of the data sampling threads includes:

Predetermine the size information of the data block of each sample;

The data blocks are cyclically read in by each sampling thread to the corresponding target volume area according to the determined size information of each data block.

Preferably, performing compression calculation on each sampled data block through a preset compression algorithm is:

Each sampled data block is compressed and calculated through the VIFO compression algorithm.

Preferably, summarizing the corresponding compression calculation results of each sampled data block includes:

Judging whether the total sampling amount reaches the preset lower limit threshold of the total sampling amount;

If not, then save the input data size, output data size and compression ratio of the VIFO compression algorithm when each sampling;

If so, all saved data are summarized.

Preferably, it also includes:

Judging whether the number of IO read failures of each of the data sampling threads to the corresponding target volume area exceeds a preset threshold of read failure times;

If yes, stop each data sampling thread from sampling the corresponding target volume area.

Preferably, it also includes:

Judging whether the number of empty data blocks in the sample when each of the data sampling threads randomly samples the target volume reaches a preset upper limit threshold of the number of empty data blocks;

If yes, stop each data sampling thread from sampling the corresponding target volume area.

A system for evaluating the compression ratio of data volumes, comprising: a data sampling thread module, an area division module, and a compression ratio evaluation module;

Wherein, the data sampling thread module includes n data sampling thread units, n is an integer not less than 2, and the area division module is used to divide the target volume into n different target volumes according to the number of the data sampling thread units a volume area, and each of the data sampling thread units is in one-to-one correspondence with each of the target volume areas;

The data sampling thread unit is used to randomly sample the corresponding target volume area, and perform compression calculation on each sampled data block according to a preset compression algorithm;

The compression ratio evaluation module is used to summarize the corresponding compression calculation results of each sampled data block, and calculate the compression ratio evaluation result of the target volume.

Preferably, the data sampling thread unit includes:

The sample data block size setting subunit is used to determine the size information of the data block of each sample;

The sampling function sub-unit is used to cyclically read in data blocks according to the size information of each data block, perform random sampling, and perform compression calculation on each sampled data block according to a preset compression algorithm.

Preferably, it also includes:

The first judging module is used to judge whether the total sampling amount reaches the preset lower limit threshold of the total sampling amount, and when it is determined that the total sampling amount reaches the preset lower limit threshold of the total sampling amount, control the compression rate evaluation module to perform a summary function action .

Preferably, it also includes:

The second judging module is used to judge whether the IO reading failure times of the target volume by the data sampling thread module exceeds the preset reading failure times threshold, and when the judging result is yes, control the data sampling thread module stop sampling;

The third judging module is used to judge whether the number of empty data blocks in the sample when the data sampling thread module randomly samples the target volume reaches the preset upper limit threshold of the number of empty data blocks, and when the judgment result is yes, control The data sampling thread module stops the sampling action.

Compared with the prior art, the above-mentioned technical solution has the following advantages:

The method for evaluating the compression rate of a data volume provided by the embodiment of the present invention includes: setting n data sampling threads in advance, where n is an integer not less than 2; dividing the target volume into n different target volume areas, and each data sampling thread One-to-one correspondence with each target volume area; random sampling of the corresponding target volume area through each data sampling thread; compression calculation of each sampled data block through a preset compression algorithm; corresponding compression calculation results of each sampled data block Summarize and calculate the compression ratio evaluation results for the target volume. Before compressing the data volume, the compression rate of the data volume is evaluated through the above-mentioned technical scheme, and the compression rate of the target volume is randomly sampled and analyzed using mathematical and statistical methods. The investigation is simple and cost-effective, ensuring the accuracy of the evaluation Reliability, reducing the occupation of system resources, and can estimate the space savings of the target volume after compression without a lengthy data compression process, providing data support for users to decide whether to compress.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a flow chart of a method for evaluating the compression ratio of a data volume provided by a specific embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an evaluation system for data volume compression ratio provided by a specific embodiment of the present invention.

detailed description

The core of the present invention is to provide a method and system for evaluating the compression rate of a data volume, which can be used to know the space cost that can be saved after the data block is compressed before the data volume is compressed, so as to judge whether the data block is worth performing. Data compression provides users with decision-making basis.

In order to make the above objects, features and advantages of the present invention more comprehensible, the specific implementation manners of the present invention will be described in detail below in conjunction with the accompanying drawings.

In the following description, specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways than those described here, and those skilled in the art can make similar extensions without departing from the connotation of the present invention. Accordingly, the present invention is not limited to the specific embodiments disclosed below.

Please refer to FIG. 1 , which is a flowchart of a method for evaluating the compression ratio of a data volume provided by a specific embodiment of the present invention.

A specific embodiment of the present invention provides a method for evaluating the compression rate of a data volume, including:

S11: Preset n data sampling threads, where n is an integer not less than 2;

S12: Divide the target volume into n different target volume areas, each data sampling thread corresponds to each target volume area;

S13: Randomly sample the corresponding target volume area through each data sampling thread;

S14: Perform compression calculation on each sampled data block through a preset compression algorithm;

S15: Summarize the corresponding compression calculation results of each sampled data block, and calculate the compression ratio evaluation result of the target volume.

In this embodiment, in order to improve the evaluation algorithm efficiency of the evaluation method and the accuracy of random sampling, a multi-threaded architecture is adopted, that is, n data sampling threads are preset, and the target volume is divided into different areas according to the number of threads. Each thread is independently responsible for a region for random sampling. Wherein, each target volume area is equivalent to a cluster in statistical cluster sampling, therefore, the technical solution in this embodiment has the advantages of convenient investigation and cost saving. The sampled data block is compressed and calculated by a preset compression algorithm, so that the compression rate of the sample can be calculated. In this way, the evaluation result of the compression ratio of the target volume can be obtained by summarizing the calculation results of multiple samples.

Before compressing the data volume, the compression rate of the data volume is evaluated through the above-mentioned technical scheme, and the compression rate of the target volume is randomly sampled and analyzed using mathematical and statistical methods. The investigation is simple and cost-effective, ensuring the accuracy of the evaluation Reliability, reducing the occupancy of system resources, and can estimate the space savings of the target volume after compression without a long data compression process, providing data support for the user's decision whether to compress, if the current target volume is evaluated After the compression can save a large space cost, the user can compress the target volume, otherwise, the target volume does not need to be compressed, which can avoid the occupation of CPU resources and save time.

On the basis of the above embodiments, in one embodiment of the present invention, each data sampling thread randomly samples the corresponding target volume area, including:

Predetermine the size information of the data block of each sample;

The data blocks are cyclically read in by each sampling thread to the corresponding target volume area according to the determined size information of each data block.

In this embodiment, it is preferable to set every 64KB data block as a sample, so as to ensure that the data block will not be too large or too small, so as to improve sampling efficiency.

It should be noted that a data block of other sizes may also be used as a sample, which is not limited in this embodiment, and depends on specific circumstances.

Further, performing compression calculation on each sampled data block by using a preset compression algorithm is: performing compression calculation on each sampled data block by using a VIFO compression algorithm.

In this embodiment, the VIFO compression algorithm is adopted, that is, the Variable Input Fixed Output compression algorithm. The input buffer length of this algorithm is indeterminate and the output buffer length is fixed, and more data can be compressed as much as possible until the input data buffer is empty or the output buffer is empty. The data buffer is full.

Furthermore, the corresponding compression calculation results of each sampled data block are summarized, including:

Judging whether the total sampling amount reaches the preset lower limit threshold of the total sampling amount;

If not, save the input data size, output data size and compression ratio of the VIFO compression algorithm when sampling each time;

If so, all saved data are summarized.

In this embodiment, a lower limit threshold of the total sampling amount is set, and preferably, the lower limit threshold of the total sampling amount is 3400. The setting of the lower limit threshold of the total sampling amount should not only meet the requirements of sampling statistical sample size, but also take into account the basic requirements of efficient and fast algorithms. This is because the larger the number of samples, the more accurate the evaluation results. If the error rate is small, even if the error rate is within 5%, the minimum total number of samples is 3400. Of course, according to actual needs, if the error rate is required to be smaller, then the corresponding lower limit threshold of the total number of samples can be set. When the total sampling amount does not reach the lower limit threshold of the total sampling amount, summary calculation is not performed to ensure the accuracy of the evaluation result of the compression rate.

In one embodiment of the invention, the method also includes:

Determine whether the number of IO read failures of each data sampling thread to the corresponding target volume area exceeds the preset threshold of read failure times;

If yes, stop each data sampling thread from sampling the corresponding target volume area.

When the target volume is being used by the current business, it will often cause the data sampling thread to fail to read the IO of the target volume. At this time, if the sampling evaluation is continued forcibly, it will affect the progress of the current business. Therefore, when each data sampling thread When the number of IO read failures corresponding to the target volume area exceeds the preset threshold of read failure times, the sampling operation is automatically terminated. Preferably, when the proportion of read IO failures reaches 1% (1% by default, maximum 5%), the operation is automatically terminated. This proportion refers to the ratio of the number of failures to the number of read IOs corresponding to the lower limit threshold of the total sampling amount .

Further, the method also includes:

Judging whether the number of empty data blocks in the sample when each data sampling thread randomly samples the target volume reaches the preset upper limit threshold of the number of empty data blocks;

If yes, stop each data sampling thread from sampling the corresponding target volume area.

During the sampling process, if most of the data blocks used as samples are empty data blocks, such as when the sampling ratio of empty data blocks exceeds 95%, the operation is automatically terminated. This is because when sampling the target volume to calculate the evaluation results, it is necessary to ensure the accuracy range of the evaluation results, that is, to ensure that the sampling error is within a certain range. If there are too many empty data blocks, the accuracy of the sampling results will be affected. In addition, if the empty data blocks Too many data blocks indicate that the disk is relatively empty and the available space is sufficient, even without compression and compression evaluation. Therefore, in this embodiment, the compression ratio of the target volume can be evaluated more efficiently and accurately according to the evaluation result combined with the two indicators of the accuracy range of the result.

Please refer to FIG. 2 , which is a schematic structural diagram of a system for evaluating the compression ratio of data volumes provided by a specific embodiment of the present invention.

Correspondingly, an embodiment of the present invention also provides a system for evaluating the compression ratio of data volumes, including: a data sampling thread module 1, an area division module 2, and a compression ratio evaluation module 3; wherein, the data sampling thread module 1 includes n data sampling thread units, n is an integer not less than 2, and the area division module 2 is used to divide the target volume into n different target volume areas according to the number of data sampling thread units, and each data sampling thread unit and each target volume One-to-one correspondence between areas; the data sampling thread unit is used to randomly sample the corresponding target volume area, and perform compression calculation on each sampled data block according to the preset compression algorithm; the compression rate evaluation module 3 is used to summarize each sampled data block The corresponding compression calculation results, and calculate the compression ratio evaluation results of the target volume.

In this implementation manner, the compression ratio evaluation module and the region division module can be integrated into the data sampling thread module. Before compressing the data volume, the compression rate of the data volume is evaluated through the above-mentioned technical scheme, and the compression rate of the target volume is randomly sampled and analyzed using mathematical and statistical methods. The investigation is simple and cost-effective, ensuring the accuracy of the evaluation Reliability, reducing the occupancy of system resources, and can estimate the space savings of the target volume after compression without a long data compression process, providing data support for the user's decision whether to compress, if the current target volume is evaluated After the compression can save a large space cost, the user can compress the target volume, otherwise, the target volume does not need to be compressed, which can avoid the occupation of CPU resources and save time.

In one embodiment of the present invention, the data sampling thread unit includes: a sample data block size setting subunit, used to determine the size information of the data block of each sample; a sampling function subunit, used to The size information of the data is cyclically read into data blocks, random sampling is performed, and each sampled data block is compressed and calculated according to the preset compression algorithm.

Further, the system also includes: a first judging module, configured to judge whether the total sampling amount has reached the preset lower limit threshold of the total sampling amount, and control the compression rate when it is determined that the total sampling amount reaches the preset lower limit threshold of the total sampling amount The evaluation module performs summary functional actions.

In this embodiment, the size of the data block of each sample and the lower limit threshold of the total sampling amount are set, preferably every 64KB data block is set as a sample, so as to ensure that the data block will not be too large or too small to improve Sampling efficiency. Preferably, the lower limit threshold of the total number of samples is 3400. The setting of the lower limit threshold of the total sampling amount should not only meet the requirements of sampling statistical sample size, but also take into account the basic requirements of efficient and fast algorithms. This is because the larger the number of samples, the more accurate the evaluation results. If the error rate is small, even if the error rate is within 5%, the minimum total number of samples is 3400. Of course, according to actual needs, if the error rate is required to be smaller, then the corresponding lower limit threshold of the total number of samples can be set. When the total sampling amount does not reach the lower limit threshold of the total sampling amount, summary calculation is not performed to ensure the accuracy of the evaluation result of the compression rate.

In one embodiment of the invention, the system also includes:

The second judging module is used to judge whether the IO reading failure times of the target volume by the data sampling thread module exceeds the preset reading failure times threshold, and when the judgment result is yes, the data sampling thread module is controlled to stop the sampling action;

The third judging module is used to judge whether the number of empty data blocks of the sample when the data sampling thread module randomly samples the target volume reaches the preset upper limit threshold of the number of empty data blocks, and when the judgment result is yes, control the data sampling thread module Stop the sampling action.

When the target volume is being used by the current business, it will often cause the data sampling thread to fail to read the IO of the target volume. At this time, if the sampling evaluation is continued forcibly, the current business will be affected. Therefore, the second judgment module is adopted. , when the number of IO read failures of each data sampling thread on the corresponding target volume area exceeds the preset read failure number threshold, the sampling operation is automatically terminated. Preferably, when the proportion of read IO failures reaches 1% (1% by default, maximum 5%), the operation is automatically terminated. This proportion refers to the ratio of the number of failures to the number of read IOs corresponding to the lower limit threshold of the total sampling amount . During the sampling process, if most of the data blocks used as samples are empty data blocks, such as when the sampling ratio of empty data blocks exceeds 95%, the operation is automatically terminated. This is because when sampling the target volume to calculate the evaluation results, it is necessary to ensure the accuracy range of the evaluation results, that is, to ensure that the sampling error is within a certain range. If there are too many empty data blocks, the accuracy of the sampling results will be affected. In addition, if the empty data Too many blocks means that the disk is relatively empty, there is enough free space, and compression and compression evaluation are not even necessary. Therefore, in this embodiment, the compression ratio of the target volume can be evaluated more efficiently and accurately according to the evaluation result combined with the two indicators of the accuracy range of the result.

To sum up, the method and system for evaluating the compression rate of data volumes provided by the present invention can know the space cost that can be saved after the data blocks are compressed before compressing the data volumes, and use mathematical methods and statistics The method conducts random sampling analysis on the compression rate of the target volume, and the investigation is simple and cost-effective, which ensures the accuracy and reliability of the evaluation, reduces the occupation of system resources, and saves space after compression of the target volume without a long data compression process. Estimate to provide data support for the user's decision whether to compress. If it is estimated that the current target volume can save a large space cost after compression, the user can compress the target volume; otherwise, the target volume does not need to be compressed. Compression can avoid occupying CPU resources and save time. In addition, the compression rate of the target volume can be evaluated more efficiently and accurately by using the evaluation result combined with the two indicators of the accuracy range of the result.

The method and system for evaluating the compression ratio of data volumes provided by the present invention have been introduced in detail above. In this paper, specific examples are used to illustrate the principles and implementation methods of the present invention, and the descriptions of the above embodiments are only used to help understand the present invention and its core ideas. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (10)

1. an evaluation method of data volume compression rate, is characterized in that, comprises: Pre-set n data sampling threads, n is an integer not less than 2; Dividing the target volume into n different target volume areas, each of the data sampling threads is in one-to-one correspondence with each of the target volume areas; performing random sampling on the corresponding target volume area through each of the data sampling threads; Perform compression calculation on each sampled data block through a preset compression algorithm; Summarize the corresponding compression calculation results of each sampled data block, and calculate the compression ratio evaluation result of the target volume. 2. The method according to claim 1, wherein the random sampling of the corresponding target volume area by each of the data sampling threads comprises: Predetermine the size information of the data block of each sample; The data blocks are cyclically read in by each sampling thread to the corresponding target volume area according to the determined size information of each data block. 3. The method according to claim 2, wherein the compression calculation of each sampled data block by a preset compression algorithm is: Each sampled data block is compressed and calculated through the VIFO compression algorithm. 4. The method according to claim 3, wherein said summarizing the corresponding compression calculation results of each sampled data block comprises: Judging whether the total sampling amount reaches the preset lower limit threshold of the total sampling amount; If not, then save the input data size, output data size and compression ratio of the VIFO compression algorithm when each sampling; If so, all saved data are summarized. 5. The method according to any one of claims 1 to 4, further comprising: Judging whether the number of IO read failures of each of the data sampling threads to the corresponding target volume area exceeds a preset threshold of read failure times; If yes, stop each data sampling thread from sampling the corresponding target volume area. 6. The method according to claim 5, further comprising: Judging whether the number of empty data blocks in the sample when each of the data sampling threads randomly samples the target volume reaches a preset upper limit threshold of the number of empty data blocks; If yes, stop each data sampling thread from sampling the corresponding target volume area. 7. An evaluation system for data volume compression ratio, comprising: a data sampling thread module, a region division module and a compression ratio evaluation module; Wherein, the data sampling thread module includes n data sampling thread units, n is an integer not less than 2, and the area division module is used to divide the target volume into n different target volumes according to the number of the data sampling thread units a volume area, and each of the data sampling thread units is in one-to-one correspondence with each of the target volume areas; The data sampling thread unit is used to randomly sample the corresponding target volume area, and perform compression calculation on each sampled data block according to a preset compression algorithm; The compression ratio evaluation module is used to summarize the corresponding compression calculation results of each sampled data block, and calculate the compression ratio evaluation result of the target volume. 8. The system according to claim 7, wherein the data sampling thread unit comprises: The sample data block size setting subunit is used to determine the size information of the data block of each sample; The sampling function sub-unit is used to cyclically read in data blocks according to the size information of each data block, perform random sampling, and perform compression calculation on each sampled data block according to a preset compression algorithm. 9. The system according to claim 8, further comprising: The first judging module is used to judge whether the total sampling amount has reached the preset lower limit threshold of the total sampling amount, and when it is determined that the total sampling amount has reached the preset lower limit threshold of the total sampling amount, control the compression rate evaluation module to perform a summary function action . 10. The system of claim 9, further comprising: The second judging module is used to judge whether the IO reading failure times of the target volume by the data sampling thread module exceeds the preset reading failure times threshold, and when the judging result is yes, control the data sampling thread module stop sampling; The third judging module is used to judge whether the number of empty data blocks in the sample when the data sampling thread module randomly samples the target volume reaches the preset upper limit threshold of the number of empty data blocks, and when the judgment result is yes, control The data sampling thread module stops the sampling action.