CN111367871A - An incremental synchronization method between files based on SAPCI variable-length blocks - Google Patents

Abstract

The invention discloses a method for incremental synchronization between files based on SAPCI variable-length blocks, comprising the following steps: S1, the backup server performs file block on the backup file; Each small block is encrypted and integrated separately, and the result of encryption and integration is sent to the source server; S3. In the source server, the same SAPCI algorithm is also used for the changed source file to perform file segmentation, and MD5 is also used for the source file. Each small block after the block is encrypted; S4, in the source server, by comparing the MD5 data sent from the backup server and the MD5 data after the source file is divided and calculated by itself, so as to obtain incremental data blocks and non-incremental data blocks. S5, the source server integrates the incremental data and matching data, and sends the incremental data and matching data to the backup server; S6, the backup server integrates the backup files.

Description

An incremental synchronization method between files based on SAPCI variable-length blocks

technical field

The invention belongs to the field of increments between computer files, and in particular relates to an incremental synchronization method between files based on SAPCI variable-length blocks

Background technique

With the rapid development of information technology in today's world, the data generated by digital technology is growing explosively. At the same time, in order to ensure the fault tolerance of the system, multiple backup servers are often added to the system and multiple backup files are generated. With the rapid growth of data volume , the synchronization between source files and backup files has become an important issue that cannot be ignored.

At present, the traditional algorithm for incremental synchronization between files is mainly implemented by the Rsync algorithm. The Rsync algorithm is based on file fixed-length block and rolling verification, which can accurately locate the incremental data between the source file and the backup file. However, due to the design defect of the fixed-length block algorithm, the block algorithm has very poor resistance to byte offset, and slight changes in the file will lead to the collective offset of the subsequent block boundaries of the file (Deng Xuefeng, Sun Ruizhi, Zhang Yonghan, et al. Based on Sliding block algorithm of data bitmap [J]. Computer Research and Development, 2014, 51(Suppl.): 30-38.), so Rsync algorithm needs to calculate weak check code in addition to strong check code , and rolling verification is performed on the source server side, which greatly increases the CPU computing burden of the source server and greatly increases the verification time (A. Tridgell, "Efficient algorithms for sorting and synchronization," https ://www.sama.org//~tridge/phd_thesis.pdf, accessed February, 1999.). Based on the inherent defects of the Rsync algorithm, the variable length block (CDC) came into being. The block boundary when this type of algorithm blocks the file is designed according to certain conditions, and only when the algorithm meets the corresponding conditions will it be performed. Boundary division, so slight changes in the file will not directly lead to the collective offset of a large number of subsequent block boundaries, and this provides enough possibility to apply the algorithm to incremental synchronization between files.

SUMMARY OF THE INVENTION

The purpose of the present invention is to propose an incremental synchronization method between files based on SAPCI (Adaptive-Interval Parity Check) variable-length blocks.

The object of the present invention is achieved by at least one of the following technical solutions.

A method for incremental synchronization between files based on SAPCI variable-length blocks, comprising the following steps:

S1. The backup server (Backup Server) that needs to perform incremental synchronization performs file segmentation on the backup file (Backup File) based on the SAPCI algorithm;

S2. The backup server uses the message digest algorithm MD5 (Message-Digest, MD5Algorithm) to encrypt and integrate the small blocks of the backup file respectively, and generate the MD5 list List _backup-MD5 , and finally the encrypted and integrated result is the MD5 list. List _backup-MD5 sent to the source server (Source Server);

S3. In the source server, the changed source file (Source File) is also divided into file blocks using the same SAPCI algorithm as in step S1, and MD5 is also used to encrypt each small block after the source file is divided into blocks. The encrypted result is MD5 _src-i , where src represents the encryption of each small block after the source file is divided into blocks, i=0, 1, 2, 3, 4, 5... The numbering order of the small blocks, after the encryption is completed, the encrypted result MD5 _src-i is also stored in the memory of the source server according to the numbering order to form the MD5 list List _src-MD5 , wherein this step is performed synchronously with step S1 and step S2;

S4. Obtain incremental data blocks and non-incremental data blocks in the source server;

S5, the source server integrates the incremental data and the matching data according to certain rules, and sends the incremental data and the matching data to the backup server, and this step is performed synchronously with S4;

S6. After parsing the received incremental data, the backup server integrates the backup files, and finally completes the synchronization.

Further, the SAPCI algorithm of step S1 is improved based on the interval parity block algorithm (Parity Check OfInterval, PCI algorithm), and the improved SAPCI algorithm is called the self-adaptive interval parity block algorithm (Self-Adaptive). Parity Check Of Interval), which mainly includes the following steps:

S101. Read data from the initial byte of the file;

S102, store the read data in the end of the check interval, wherein the check interval is defined as a window with a length of len;

S103, determine whether the data amount of the verification interval is equal to or not greater than len, if not, continue to execute step S102, if yes, enter step S104;

S104, calculate the total number n of bits that are 1 in each byte data in the check interval, and judge whether n is greater than or equal to the set threshold threshold, if not, then enter step S105, if yes, then determine the check interval The tail is the block boundary;

S105, slide one byte backward in the check interval, calculate the total number n of bits 1 in each byte data in the check interval again, and judge whether n is greater than the threshold, if so, it can be determined that the end of the check interval is divided into block boundary, if not, go to step S106.

S106, determine whether the distance between the tail of the current check interval and the right border of the previous block is greater than or equal to the threshold change interval d, if not, then re-enter step S104, if so, reduce the current threshold by 1, and then re-enter step S104, until a block boundary is found;

S107, after finding the block boundary, reset the current threshold to the initial value;

S108, repeat steps S102 to S107, until the end of the check interval reaches the end of the file;

S109, take the tail data block located at the end of the file that does not meet the block condition as a separate block. The block that does not meet the block condition means that there is no interval with a length of len in the data block at the end of the file, so that each byte in the interval does not exist. The total number n of 1 bits in the data is greater than or equal to the current threshold threshold;

S110. Complete the process of dividing the file data into blocks.

Further, the encryption and integration method of step S2 is: each small block after the backup file is divided into blocks is respectively MD5 encrypted, and the encryption result of each small block is MD5 _backup-i , wherein backup represents the backup file after the block is divided. Each small block is encrypted, i=0, 1, 2, 3, 4, 5..., which represents the number sequence of each small block after the backup file is divided into blocks. After the encryption is completed, only the encrypted result needs to be MD5 _{backup -i} is stored in the communication sending buffer of the backup server according to the serial number to form the MD5 list List _backup-MD5 , and the integration can be completed at this time.

Further, the data format sent by the backup server to the source server in the step S2 is:

(chunkIndex, MD5(chunk bytes))

where chunkIndex is the sequence number of the chunk after the backup server chunks the backup file based on the SAPCI algorithm, starting from 0, and chunk bytes is the chunked byte array with variable length after the backup server chunks the backup file based on the SAPCI algorithm. , MD5 (chunk bytes) is the check code after calculating the MD5 value of the block byte array. There are many ways to calculate the MD5 check code. Taking the Java programming language as an example, it can use the JDK (Java Development Kit, The digest method provided by the MessageDigest class that comes with Java development tools) can be calculated by passing the block byte array as a parameter to the digest method.

Further, the step of obtaining the incremental data block and the non-incremental data block in the step S4 is as follows:

S401. Compare the differences in MD5 data between List _backup-MD5 and List _src-MD5 , and find the MD5 data that exists in List _src-MD5 and also exists in List _backup-MD5 . The data block corresponding to the MD5 data is Non-incremental data block, then enter step S402; on the contrary, if the MD5 data exists in List _src-MD5 , but does not exist in List _backup-MD5 , the data block corresponding to the MD5 data is the incremental data block. Enter step S403;

S402, for the non-incremental data block, generate a data structure, and the data format is as follows:

(source chunk index, backup chunk index)

Among them, the source chunk index is the data source block number currently being judged, starting from 0, and the backup chunk index is the backup file data block corresponding to the MD5 corresponding to the data source block number currently being judged. Number, which is obtained from the data sent from the backup server;

S403, for the incremental data block, generate a data structure, and the data format is as follows:

(source chunk index, diff chunk bytes)

Among them, source chunk index is the data source chunk number currently being judged, and diff chunkbytes is the entire data chunk in the data source that cannot match the MD5 of the backup file;

It can be seen that, compared with the Rsync algorithm, the file increments found by the incremental synchronization method between files based on SAPCI variable-length blocks are not accurate data, but at the same time, it reduces the need for the backup server to calculate the weak checksum and send the weak checksum. To verify the code, the source server rolls the calculation consumption of the weak verification code. Therefore, the incremental synchronization method between files based on SAPCI variable-length blocks is a method that greatly reduces the CPU computing consumption in the backup server and the source server at the expense of increasing incremental data discovery.

Among them, the additional consumption of incremental data blocks mainly depends on the fact that the incremental data is discrete in each data block in the backup file. Assuming that the current source file has a 100B data increment relative to the backup file, the Rsync algorithm can be used to accurately locate the data. The incremental data size of 100B, but it needs to consume a lot of extra computational consumption for calculating weak check codes and rolling checks. For the incremental synchronization method between files based on SAPCI variable-length blocks, the increment found depends on the degree to which the current 100B incremental data is discrete from each data block of the file.

The incremental dispersion is defined as follows:

L=C/I (1)

Where L is the incremental dispersion, C is the number of blocks in the source file distributed by the incremental data, and I is the size of the incremental bytes; the incremental dispersion satisfies the following relationship:

1/I≤L≤1 (2)

When L=1/I, it means that the source file is relative to the backup file, and the incremental data occurs in one of the blocks of the backup file. The size of the incremental data block is approximately:

IC≈I+AC (3)

where AC is the average block length;

When L=1, it means that the source file is relative to the backup file, and each byte of incremental data occurs independently in different data blocks of the backup file. The size of the incremental data block is approximately:

IC≈I+AC*I. (4)

Further, the rule of step S5 is: when the data block is successfully matched in step S4, that is, the current data block non-incremental data block, the MD5 result after the encryption of each data block after the source file is divided into blocks exists in List _src-MD5 and List src-MD5 simultaneously. In List _backup-MD5 , matching data is generated at this time, the format is source chunk index, backup chunk index, where sourcechunk index is the data source chunk number currently being judged, starting from 0, and backup chunkindex is currently being judged. The block number of the backup file data block matched by the block number of the data source;

When the data block matching fails in step S4, that is, the current data block is an incremental data block, and the encrypted MD5 result of each data block after the source file is divided into blocks only exists in List _src-MD5 , but not in List _backup-MD5 . , at this time, incremental data is generated in the format of source chunk index, diff chunk bytes, where source chunk index is also the data source block number currently being judged, and diff chunk bytes is the data source that cannot match the same data block of the backup file. Byte array, i.e. incremental data block;

After the incremental data and matching data are generated, the data results are stored in the communication sending cache of the source server according to the block number sequence of the data source to complete the integration, and the integration results are sent to the backup server according to the block number sequence of the data source. .

Further, the data format sent in the step S5 is as follows:

1) If the current data block is successfully matched, the matching data is transmitted at this time, and the sending format is as follows:

(source chunk index, backup chunk index)

2) If the current data block fails to match, the incremental data is transmitted at this time, and the sending format is as follows:

(source chunk index, diff chunk bytes).

Further, the total time consumption of steps S1 to S6 is defined as follows:

T _CDC =max{T _SB +T _ST +T _CB ,T _CS +T _SS }+T _ICT +T _M (5)

Among them, T _CDC is the total time spent, T _SB is the time taken by the backup server to calculate the strong check code MD5, T _ST is the time taken by the backup server to transmit the strong check code to the source server, and T _CB is the time spent on the backup server to classify the file. Block time, T _CS is the time required for the source server to divide the file into blocks, T _SS is the time required for the source server to calculate the MD5 of the strong parity code, and T _ICT is the source server comparing the source file block MD5 with the backup file block. Block MD5 and transmit the incremental data and matching data to the backup server synchronously, _TM is the time taken by the backup server to complete the incremental synchronization according to the integration of the incremental data and matching data.

Further, in the step S6, the backup server integrates the backup files according to the received incremental data and matching data as follows:

S601. Create a temporary file, and perform synchronization based on the temporary file;

S602, judge the received data, if the current data format is:

(source chunk index, backup chunk index)

It means that the same data block in the backup file exists in the data source. In this case, you only need to write the data block of the backup file indicated in the backup chunk index to the end of the reorganized file. When writing, ensure that the index order indicated by the source chunkindex is followed. to write;

S603. If the current data format is:

(source chunk index, diff chunk bytes)

It means that the currently received incremental data block, at this time, you only need to directly write the diff chunk bytes to the end of the reorganized file. When writing, ensure that the order is written according to the index indicated by the source chunk index;

S604. Repeat steps S602 and S603 until all data returned from the source server is read. At this time, the incremental synchronization of files is completed, the original backup file is deleted, and the temporary file is renamed to the file name of the original backup file. Integration and incremental synchronization can be done.

Compared with the prior art, the present invention has the following advantages and effects:

1) The SAPCI variable length block used in the present invention is improved based on the PCI algorithm. Compared with the PCI algorithm, the SAPCI algorithm can greatly reduce the large data block (10 times or more than the average block length), thereby reducing the subsequent redundant data transmission consumption. The adaptive threshold change method adopted, because the threshold change only occurs in local data blocks, does not have much impact on the data volume of the overall file block, that is, there is no excessive computational consumption.

2) Incremental synchronization of files based on SAPCI variable-length blocks proposed by the present invention, compared with the traditional Rsync algorithm for file incremental synchronization, its incremental data transmission takes incremental data blocks as the main unit, including certain The data redundancy, but greatly reduces the calculation consumption of the backup server to calculate the weak check code and send the weak check code, and the source server to calculate the weak check code on a rolling basis. In practical applications, it has practical application significance.

3) Incremental synchronization of files based on SAPCI variable length blocks proposed by the present invention, since the source server does not need and cannot find incremental data through rolling verification, it performs data block on the file and calculates the corresponding points. The operation of block MD5 can be performed synchronously with the operation of the backup server to perform data block on the file and calculate the MD5 corresponding to the block, thus reducing the running time of this part as a whole.

Description of drawings

1 is a schematic flowchart of a method for incremental synchronization between files based on SAPCI variable-length blocks in an embodiment of the present invention;

FIG. 2 is a schematic diagram of a pseudo-code of the SAPCI algorithm in an embodiment of the present invention.

Detailed ways

The specific implementation of the present invention will be further described below with reference to the accompanying drawings and embodiments.

Example:

As shown in Figure 1, a method for incremental synchronization between files based on SAPCI variable-length blocks includes the following steps:

1) The backup server (Backup Server) that needs to perform incremental synchronization performs file partitioning on the backup file (Backup File) based on the SAPCI algorithm; wherein the workflow pseudocode of the SAPCI variable-length partitioning is shown in Figure 2, wherein the code input is file file, check interval size lenOfWindow, threshold threshold, interval change interval d, the code output is a block boundary file index list,

The SAPCI algorithm in this embodiment is improved based on an interval parity check block algorithm (Parity Check Of Interval, PCI algorithm). The improved SAPCI algorithm is called an adaptive interval parity check block algorithm (Self-Adaptive Parity Check algorithm). Of Interval), which mainly includes the following steps:

S101. Read data from the initial byte of the file;

S102, store the read data into the end of the check interval, wherein the check interval is defined as a window with a length of len, and there is no special rule for determining len, and it is generally set to 10;

S103, determine whether the data amount of the verification interval is equal to or not greater than len, if not, continue to execute step S102, if yes, enter step S104;

S104, calculate the total number n of bits that are 1 in each byte data in the check interval, and judge whether n is greater than or equal to the set threshold threshold, if not, then enter step S105, if yes, then determine the check interval The tail is the block boundary; where threshold is a set parameter, there is no special rule for the determination of threshold, and the smaller the threshold, the greater the possibility that n is greater than the threshold, so that the probability of finding the block boundary increases, and then increases The number of file blocks to reduce the average block length, and the determination of the threshold can be flexibly set according to actual needs;

S105, slide one byte backward in the check interval, calculate the total number n of bits 1 in each byte data in the check interval again, and judge whether n is greater than the threshold, if so, it can be determined that the end of the check interval is divided into block boundary, if not, go to step S106.

S106, determine whether the distance between the tail of the current check interval and the right border of the previous block is greater than or equal to the threshold change interval d, if not, then re-enter step S104, if so, reduce the current threshold by 1, and then re-enter step S104, until a block boundary is found;

S107, after finding the block boundary, reset the current threshold to the initial value;

S108, repeat steps S102 to S107, until the end of the check interval reaches the end of the file;

S109. Dissolve the data blocks located at the end of the file that do not meet the block conditions (wherein the block conditions do not meet the block conditions means that there is no interval with a length of len in the data blocks at the end of the file, so that the total number of bits 1 in each byte data in the interval is 1). The tail data block whose number n is greater than or equal to the current threshold (threshold) is regarded as a separate block;

S110. Complete the process of dividing the file data into blocks.

2) The backup server uses MD5 (Message Digest Algorithm, Message-Digest Algorithm, hereinafter referred to as MD5) to encrypt and integrate each small block after the backup file is divided into blocks. Each small block is encrypted by MD5, and the encryption result of each small block is MD5 _backup-i (where backup represents the encryption of each small block after the backup file is divided into blocks, i=0,1,2,3,4,5 ..., which represents the numbering sequence of each small block after the backup file is divided into blocks), after completing the encryption, only the encrypted result MD5 _backup-i is stored in the communication sending cache of the backup server in the order of the number to form an MD5 list (with List _backup-MD5 ), the integration can be completed at this time, and each block is numbered. For each block, the generated data format is:

(chunkIndex, MD5(chunk bytes))

where chunkIndex is the sequence number of the chunk after the backup server chunks the backup file based on the SAPCI algorithm, starting from 0, and chunk bytes is the chunked byte array with variable length after the backup server chunks the backup file based on the SAPCI algorithm. , MD5 (chunk bytes) is the check code after calculating the MD5 value of the block byte array (there are many ways to calculate the MD5 check code. Taking the Java programming language as an example, it can use the JDK (Java Development Kit, The digest method provided by the MessageDigest class that comes with Java development tools) is calculated, just pass the block byte array as a parameter to the digest method). After the file is divided into blocks, MD5 is used to encrypt each data block, and there is no need to calculate the weak check code for the data block.

3) The backup server sends the generated data List _backup-MD5 to the source server (Source Server).

4) In the source server, the changed source file (Source File) is also divided into blocks using the same SAPCI algorithm, and the parameters of the SAPCI algorithm used in the source server and the backup server are kept consistent. This step can be performed simultaneously with steps 1-3.

5) In the source server, MD5 is also used to encrypt each data block after the source file is divided into blocks. Similarly, the encrypted result of each small block is MD5 _src-i (where src represents the encryption of each small block after the source file is divided into blocks, i=0, 1, 2, 3, 4, 5..., represents the After the source file is divided into blocks, the numbering sequence of each small block), after completing the encryption, the encrypted result MD5 _src-i is also stored in the memory of the source server in the number sequence to form an MD5 list (represented by List _src-MD5 ), this step It can be performed simultaneously with the above steps 1-3.

6) After receiving the List _backup-MD5 sent from the backup server in the above step 3, and after completing the above step 5 to obtain the List _src-MD5 , the source server compares the MD5 data difference between the two lists and finds the List _src-MD5 MD5 data that exists in MD5, but does not exist in List _backup-MD5 , the data block corresponding to the MD5 data is the incremental data block. Conversely, if the MD5 data exists in List _src-MD5 , but also exists in List _backup-MD5 , the data block corresponding to the MD5 data is the non-incremental data block. Specific steps are as follows:

S401. After receiving the List _backup-MD5 sent from the backup server in step 3, and obtaining the List _src-MD5 after completing the step 5, the source server compares the MD5 data difference between the two lists and finds that the list _src-MD5 exists in the list src-MD5. , and the MD5 data also exists in the List _backup-MD5 , and the data block corresponding to the MD5 data is the non-incremental data block. At this time, it proceeds to step S402. Conversely, if there is MD5 data in List _src-MD5 , but not in List _backup-MD5 , the data block corresponding to the MD5 data is the incremental data block, and in this case, step S403 is entered.

S402, at this time, the non-incremental data block is found, and a data structure is generated, and the data format is as follows:

(source chunk index, backup chunk index)

Among them, the source chunk index is the data source block number currently being judged, starting from 0, and the backup chunk index is the backup file data block corresponding to the MD5 corresponding to the data source block number currently being judged. Number, which is obtained from the data sent from the backup server;

S403, find the incremental data block at this time, generate a data structure, and the data format is as follows:

(source chunk index, diff chunk bytes)

Among them, source chunk index is the data source chunk number currently being judged, and diff chunkbytes is the entire data chunk in the data source that cannot match the MD5 of the backup file;

Among them, the additional consumption of incremental data blocks mainly depends on the discreteness of incremental data in each data block in the backup file. The incremental dispersion is defined as follows:

L=C/I (1)

Where L is the incremental dispersion, C is the number of blocks in the source file distributed by the incremental data, and I is the size of the incremental bytes; the incremental dispersion satisfies the following relationship:

1/I≤L≤1 (2)

When L=1/I, it means that the source file is relative to the backup file, and the incremental data occurs in one of the blocks of the backup file. The size of the incremental data block is approximately:

IC≈I+AC (3)

where AC is the average block length;

When L=1, it means that the source file is relative to the backup file, and each byte of incremental data occurs independently in different data blocks of the backup file. The size of the incremental data block is approximately:

IC≈I+AC*I. (4)

7) The source server sends data in different formats to the backup server according to whether the MD5 value of the current file block matches. If the MD5 value matches, it means that the corresponding data block is a non-incremental data block. The data format sent at this time is:

(source chunk index, backup chunk index)

Among them, the source chunk index is the data source block number currently being judged, starting from 0, and the backup chunk index is the backup file data block corresponding to the MD5 corresponding to the data source block number currently being judged. Number, which can be obtained from the above-mentioned data sent by the backup server to the source server.

If the MD5 value does not match, it means that the corresponding data block is an incremental data block, and the data format sent at this time is:

(source chunk index, diff chunk bytes)

The source chunk index is the number of the data source chunk currently being judged, and the diff chunkbytes is the entire data chunk in the data source that cannot match the MD5 of the backup file.

In addition, it should be noted that in order to improve the utilization efficiency of the CPU and IO of the source server and the backup server, the transmission of this part of the results can be carried out in the process of MD5 verification and comparison of the data on the source server.

8) After parsing the received incremental data, the backup server integrates the backup files. The integration process is as follows:

S801. Create a temporary file, and perform synchronization based on the temporary file;

S802, judge the received data, if the current data format is:

(source chunk index, backup chunk index)

It means that the same data block in the backup file exists in the data source. In this case, you only need to write the data block of the backup file indicated in the backup chunk index to the end of the reorganized file. When writing, ensure that the index order indicated by the source chunkindex is followed. to write;

S803. If the current data format is:

(source chunk index, diff chunk bytes)

It means that the current received is incremental data blocks. In this case, it is only necessary to directly write the diff chunk bytes to the end of the reorganized file. When writing, it is necessary to ensure that the order is written according to the index indicated by the source chunk index. Reorganize the order of the file, otherwise there may be a problem of out-of-order data blocks.

S804. Repeat steps S802 and S803 until all data returned from the source server is read. At this time, the incremental synchronization of files is completed, the original backup file is deleted, and the temporary file is renamed as the original backup file.

The invention can use SAPCI variable-length blocks to find incremental data between files, and realize incremental synchronization between files on this basis. The invention applies SAPCI variable-length blocks to incremental synchronization between files, which is different from the traditional Compared with the Rsync algorithm, at the expense of discovering more incremental data and reducing the accuracy of incremental data discovery, it greatly reduces the computing time required for incremental discovery, so that incremental synchronization between files can be achieved in a shorter time. Feasible and practical.

The description of the above embodiment is relatively specific and detailed, but only expresses a feasible implementation manner of the present invention, and does not limit the patent scope of the present invention. It should be pointed out that, under the framework of the present invention, scientific researchers and engineers in the field can add some modifications or improvements on the basis of this preferred embodiment, but these are all within the protection scope of the present invention patent. The protection scope of the patent of the present invention shall be subject to the appended claims.

Claims (9)

1. a method for incremental synchronization between files based on SAPCI variable length block, is characterized in that, comprises the following steps: S1. The backup server (Backup Server) that needs to perform incremental synchronization performs file segmentation on the backup file (Backup File) based on the SAPCI algorithm; S2. The backup server uses the message digest algorithm MD5 (Message-Digest, MD5Algorithm) to encrypt and integrate the small blocks of the backup file respectively, to generate an MD5 list List _backup-MD5 , and finally encrypt and integrate the result of the MD5 list. List _backup-MD5 sent to the source server (Source Server); S3. In the source server, the changed source file (Source File) is also divided into file blocks using the same SAPCI algorithm as in step S1, and MD5 is also used to encrypt each small block after the source file is divided into blocks. The encrypted result is MD5 _src-i , where src represents the encryption of each small block after the source file is divided into blocks, i=0, 1, 2, 3, 4, 5... The numbering order of the small blocks, after the encryption is completed, the encrypted result MD5 _src-i is also stored in the memory of the source server according to the numbering order to form the MD5 list List _src-MD5 , wherein this step is performed synchronously with step S1 and step S2; S4. Obtain incremental data blocks and non-incremental data blocks in the source server; S5, the source server integrates the incremental data and the matching data according to certain rules, and sends the incremental data and the matching data to the backup server, and this step is performed synchronously with S4; S6. After parsing the received incremental data, the backup server integrates the backup files, and finally completes the synchronization. 2. the incremental synchronization method between files based on SAPCI variable length block according to claim 1, is characterized in that, the SAPCI algorithm of step S1 is based on interval parity check block algorithm (Parity Check Of Interval, PCI algorithm ) to improve, the improved SAPCI algorithm is called the self-adaptive interval parity check block algorithm (Self-Adaptive Parity Check Of Interval), which mainly includes the following steps: S101. Read data from the initial byte of the file; S102, store the read data into the end of the check interval, wherein the check interval is defined as a window with a length of len; S103, determine whether the data amount of the verification interval is equal to or not greater than len, if not, continue to execute step S102, if yes, enter step S104; S104: Calculate the total number n of bits that are 1 in each byte data in the check interval, and determine whether n is greater than or equal to the set threshold threshold, if not, go to step S105, if so, the check interval can be determined The tail is the block boundary; S105, slide one byte backward in the check interval, calculate the total number n of bits 1 in each byte data in the check interval again, and judge whether n is greater than the threshold, if so, it can be determined that the end of the check interval is divided into block boundary, if not, go to step S106; S106, determine whether the distance between the tail of the current check interval and the right border of the previous block is greater than or equal to the threshold change interval d, if not, then re-enter step S104, if so, reduce the current threshold by 1, and then re-enter step S104, until a block boundary is found; S107, after finding the block boundary, reset the current threshold to the initial value; S108, repeat steps S102 to S107, until the end of the check interval reaches the end of the file; S109, take the tail data block located at the end of the file that does not meet the block condition as a separate block. The block that does not meet the block condition means that there is no interval with a length of len in the data block at the end of the file, so that each byte in the interval does not exist. The total number n of 1 bits in the data is greater than or equal to the current threshold threshold; S110. Complete the process of dividing the file data into blocks. 3. the incremental synchronization method between files based on the SAPCI variable length block according to claim 1, is characterized in that, the encryption of step S2 and the integration mode are: each small block after the block is carried out to the backup file respectively. MD5 encryption, and make the encryption result of each small block as MD5 _backup-i , where backup represents the encryption of each small block after the backup file is divided into blocks, i=0, 1, 2, 3, 4, 5... The number sequence of each small block after the backup file is divided into blocks, after encryption is completed, only the encrypted result MD5 _backup-i needs to be stored in the communication sending cache of the backup server according to the number sequence to form the MD5 list List _backup-MD5 . Integration can be completed. 4. the incremental synchronization method between files based on SAPCI variable length block according to claim 1, is characterized in that, in described step S2, the data format that backup server sends to source server is: (chunkIndex, MD5(chunk bytes)) where chunkIndex is the sequence number of the chunk after the backup server chunks the backup file based on the SAPCI algorithm, starting from 0, and chunk bytes is the chunk byte array of variable length after the backup server chunks the backup file based on the SAPCI algorithm. , MD5 (chunk bytes) is the check code after calculating the MD5 value of the chunked byte array. 5. the incremental synchronization method between files based on SAPCI variable length block according to claim 1, is characterized in that, the step that described step S4 obtains incremental data block and non-incremental data block is as follows: S401. Compare the differences in MD5 data between List _backup-MD5 and List _src-MD5 , and find the MD5 data that exists in List _src-MD5 and also exists in List _backup-MD5 . The data block corresponding to the MD5 data is Non-incremental data block, then enter step S402; on the contrary, if the MD5 data exists in List _src-MD5 , but does not exist in List _backup-MD5 , the data block corresponding to the MD5 data is the incremental data block. Enter step S403; S402, for the non-incremental data block, generate a data structure, and the data format is as follows: (source chunk index, backup chunk index) Among them, the source chunk index is the data source block number currently being judged, starting from 0, and the backup chunk index is the backup file data block corresponding to the MD5 corresponding to the data source block number currently being judged. Number, which is obtained from the data sent from the backup server; S403, for the incremental data block, generate a data structure, and the data format is as follows: (source chunk index, diff chunk bytes) Among them, source chunkindex is the number of the data source block currently being judged, and diff chunk bytes is the entire data block in the data source that cannot match the MD5 of the backup file; The additional consumption of incremental data blocks mainly depends on the discreteness of incremental data in each data block in the backup file. The incremental dispersion is defined as follows: L=C/I (1) Where L is the incremental dispersion, C is the number of blocks in the source file distributed by the incremental data, and I is the size of the incremental bytes; the incremental dispersion satisfies the following relationship: 1/I≤L≤1 (2) When L=1/I, it means that the source file is relative to the backup file, and the incremental data occurs in one of the blocks of the backup file. The size of the incremental data block is approximately: IC≈I+AC (3) where AC is the average block length; When L=1, it means that the source file is relative to the backup file, and each byte of incremental data occurs independently in different data blocks of the backup file. The size of the incremental data block is approximately: IC≈I+AC*I (4) 6. the incremental synchronization method between files based on SAPCI variable length block according to claim 1, is characterized in that, the rule of step S5 is: when data block matches successfully in step S4, namely current data block non-incremental Data block, the encrypted MD5 result of each data block after the source file is divided into blocks exists in both List _src-MD5 and List _backup-MD5 . At this time, matching data is generated in the format of source chunk index, backup chunk index, among which source chunk index is the block number of the currently judged data source, starting from 0, and the backup chunk index is the block number of the backup file data block matched by the currently judged data source block number; When the data block matching fails in step S4, that is, the current data block is an incremental data block, and the encrypted MD5 result of each data block after the source file is divided into blocks only exists in List _src-MD5 , but not in List _backup-MD5 . , at this time, incremental data is generated in the format of source chunk index, diff chunk bytes, where source chunk index is also the data source chunk number currently being judged, and diff chunk bytes is the data source that cannot match the same data chunk of the backup file. Byte array, i.e. incremental data block; After the incremental data and matching data are generated, the data results are stored in the communication sending cache of the source server according to the block number sequence of the data source to complete the integration, and the integration results are sent to the backup server according to the block number sequence of the data source. . 7. the incremental synchronization method between files based on SAPCI variable length block according to claim 1, is characterized in that, the data format that sends in described step S5 is as follows: 1) If the current data block is successfully matched, the matching data is transmitted at this time, and the sending format is as follows: (source chunk index, backup chunk index) 2) If the current data block fails to match, the incremental data is transmitted at this time, and the sending format is as follows: (source chunk index, diff chunk bytes). 8. The incremental synchronization method between files based on SAPCI variable length block according to claim 1, is characterized in that, the total time consumption of step S1～step S6 is defined as follows: T _CDC =max{T _SB +T _ST +T _CB ,T _CS +T _SS }+T _ICT +T _M (5) Among them, T _CDC is the total time spent, T _SB is the time taken by the backup server to calculate the strong check code MD5, T _ST is the time taken by the backup server to transmit the strong check code to the source server, and T _CB is the time spent on the backup server to classify the file. Block time, T _CS is the time required for the source server to divide the file into blocks, T _SS is the time required for the source server to calculate the MD5 of the strong parity code, and T _ICT is the source server comparing the source file block MD5 with the backup file block. Block MD5 and transmit the incremental data and matching data to the backup server synchronously, _TM is the time taken by the backup server to complete the incremental synchronization according to the integration of incremental data and matching data. 9. the incremental synchronization method between files based on SAPCI variable length block according to claim 1, is characterized in that, in described step S6, backup server according to received incremental data and matching data, to backup file The integration process is as follows: S601. Create a temporary file, and perform synchronization based on the temporary file; S602, judge the received data, if the current data format is: (source chunk index, backup chunk index) It means that the same data block in the backup file exists in the data source. In this case, it is only necessary to write the data block of the backup file indicated in the backup chunk index to the end of the reorganized file. When writing, ensure that the index indicated by the source chunk index is used. write sequentially; S603. If the current data format is: (source chunk index, diff chunk bytes) It means that the currently received incremental data block, at this time, you only need to directly write the diff chunk bytes to the end of the reorganized file. When writing, ensure that the order is written according to the index indicated by the source chunk index; S604. Repeat steps S602 and S603 until all data returned from the source server is read. At this time, the incremental synchronization of files is completed, the original backup file is deleted, and the temporary file is renamed to the file name of the original backup file. Integration and incremental synchronization can be done.

Images