CN103763368B - A kind of method of data synchronization across data center - Google Patents

Abstract

The present invention provides a method for synchronizing data across data centers. The specific implementation process is: completing data writing and log recording; synchronous scheduling and pushing; log playback to complete data synchronization; performing data access across data centers to realize Synchronous operations on asynchronous data. Compared with the prior art, this cross-data center data synchronization method can realize asynchronous data synchronization operations across data centers and improve data security; effectively utilize the IO resources inside the data center and the network between data centers Resources, strong practicability, easy to promote.

Description

A data synchronization method across data centers

technical field

The invention relates to the technical field of computer data storage, in particular to a data synchronization method across data centers.

Background technique

With the advent of the Internet era: social networks, microblogs, location services and other interactive websites for ordinary Internet users are booming, such as Google, Facebook, Twitter and domestic Renren, Weibo, etc., to hundreds of millions It provides interactive services based on the Internet and wireless networks to users of the project. Internet users all over the world carry out a variety of interactions every day and produce all kinds of data at any time. The amount of these data is several times the amount of data in the stand-alone era.

In order to store these data, Internet companies have built huge data centers all over the world, and the number of hosts in a single data center ranges from hundreds to tens of thousands. Information from Google shows that Google has dozens of data centers and over ten million servers around the world, storing massive amounts of data generated by its global users every day. The management and use of these data are huge challenges: including data reading and storage, indexing and addressing, configuration and management interfaces, data replication between data centers, etc., among which, between multiple data centers The support and research needs of data synchronization are particularly urgent.

At present, the research on massive data storage is still in its infancy, and there are still many aspects worthy of research and improvement for data synchronization methods between data centers. Taking Hbase as an example, the replication of Hbase depends on the architecture of Master/Slave , the feature of simple data replication between two data centers was added in version 0.90.0. There is no implementation of priority queues for replication tasks, and there is no unified scheduling for the load of data centers. On the other hand, the traditional cross-data center data synchronization algorithm usually uses the transmission and coverage of the entire block of data as the main method, which will occupy a large amount of network resources and IO resources.

In response to this situation, the patent invented a data synchronization method across data centers based on log playback.

Contents of the invention

The technical task of the present invention is to solve the deficiencies of the prior art and provide a data synchronization method across data centers.

The technical solution of the present invention is realized in the following manner. The specific implementation process of the cross-data center data synchronization method is as follows:

1. Complete data writing and log recording: run the log recording module in the main data center. When the main data center receives the data request from the client, the module will record the required operations in the main data center in the form of logs. Data center, this module is integrated into the business process of the main data center in an embedded or plug-in manner.

2. Synchronous scheduling and push: set the scheduling module to run in the primary data center, which is responsible for scheduling data playback operations, and activate log push and playback operations according to the load of the primary data center, the load of the backup data center, and scheduling policy information; The push operation required by the scheduling module is completed through the log push module, which runs in the primary data center and transmits the data operation log to the backup data center.

3. Log playback and complete data synchronization: The data operation execution pushed by the log push module of the primary data center is received by the log playback module. The log playback module runs in the backup data center and plays back the data operation logs in the current data center, realizing two Data synchronization in data center.

4. Perform data access across data centers to realize asynchronous data synchronization operations.

The detailed process of step 1 is as follows: the client identifies the primary data center where the customer data is located according to the local configuration, and sends all data operations to the primary data center for processing by the data nodes in the primary data; After the data center receives the client's request, it executes the client's operation according to the requested operation and content. During this process, the logging module captures the client's requested operation and related data by intercepting the request; the logging module judges the client's operation Is it necessary to modify the data in the data center? If so, the data operation needs to be used as the content of the cross-data center data synchronization operation. At this time, the log recording module will save the requested operation and related data to the master in a proprietary log format. The asynchronous log recording area of the data center, the content of this area is the content that needs to be played back across data centers.

The detailed process of the second step is as follows: firstly, the following conditions are monitored by the scheduling module running in the main data center.

1) The number of logs in the asynchronous logging area and the amount of data involved;

2) The load of the main data center, including network IO and disk IO;

3) Backup data center load, including network IO and disk IO;

When the above three meet the scheduling policy set by the configuration administrator, the log push operation is triggered, and the log push operation is performed by the log push module, which is responsible for writing the data operation logs in the asynchronous log recording area of the primary data center to the backup data center The asynchronous log execution area; when the log push module completes the cross-data center transmission of logs, it will notify the scheduling module, and then the scheduling module will drive the log playback module in the backup data center to perform log playback.

The detailed process of step 3 is: the log playback module running in the backup data center starts to execute the playback operation of the log after receiving the notification from the scheduling module, and the log playback module reads the data log stored in the asynchronous log execution area, Then decode the content of the log to obtain the operation and related data corresponding to the log, and then execute the operation again on the relevant nodes in the backup data center, so that the data in the backup data center is consistent with the data in the primary data center, and the data crossover is realized. Data center synchronization.

The client in step 4 acquires data by accessing the backup data center in the following two cases, when the client cannot connect to the primary data center; when the client can connect to the primary data center, but the primary data center is busy.

The beneficial effect that the present invention produces compared with prior art is:

A data synchronization method across data centers of the present invention can realize asynchronous data synchronization operations across data centers and improve data security; when users cannot access the main data center, they can also obtain data by accessing the standby data center; During the playback process, only the difference of the data needs to be transmitted, but not the data itself, so this method can also reduce the amount of transmitted data and reduce the bandwidth occupied by the synchronization operation between the data centers; in addition, the scheduling module in the system can be based on the data The load of the center is scheduled, and the IO resources inside the data center and the network resources between the data centers are effectively used to play the role of load balancing; it is practical and easy to promote.

Description of drawings

Accompanying drawing 1 is the schematic diagram of the realization process of the present invention.

detailed description

A cross-data center data synchronization method of the present invention will be described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, a cross-data center data synchronization method implements asynchronous data synchronization between data centers through playback of data operation logs. Its specific implementation process is:

First set up the following modules by programming:

(1) Logging module. Running in the main data center, it is responsible for recording the operations required by the request in the main data center in the form of logs when the main data center receives the data request from the client. This module is integrated into the business process of the main data center in an embedded or plug-in manner.

(2) Scheduling module. It runs in the main data center and is responsible for scheduling data playback operations. According to information such as the load of the primary data center, the load of the backup data center, and scheduling policies, the log push and playback operations are activated.

(3) Log push module. Running in the primary data center, it is responsible for executing the push operation required by the scheduling module and transferring the data operation log to the backup data center.

(4) Log playback module. Running in the backup data center, it is responsible for receiving the data operation execution pushed by the log push module of the main data center, and playing back the data operation log in the current data center to realize the data synchronization of the two data centers.

Complete the following operations through the above modules:

1. Data writing and log recording.

Under normal circumstances, the client recognizes the main data center where the customer data is located according to the local configuration, and sends all data operations, including reading, writing, deleting, etc., to the main data center and handed over to the main data center. The data nodes in the process are processed.

After receiving the request from the client, the primary data center will execute the client operation according to the requested operation and content. During this process, the logging module will capture the operation and related data requested by the client by intercepting the request.

The log recording module will judge whether the operation of the client needs to modify the data in the data center, and if so, it means that the data operation needs to be used as the content of the cross-data center data synchronization operation. At this time, the logging module will save the requested operation and related data in a proprietary log format to the asynchronous logging area of the primary data center. The content in this area is the content that needs to be played back across data centers.

2. Synchronous scheduling and log push.

The scheduling module running in the main data center will monitor the following conditions:

1) The number of logs in the asynchronous logging area and the amount of data involved.

2) The load of the main data center, including network IO and disk IO.

3) Backup the load of the data center, mainly including network IO and disk IO.

When the above three meet the scheduling policy set by the configuration administrator, the log push operation is triggered. The trigger is usually condition 1) high while condition 2) and condition 3) are low.

The log push operation is performed by the log push module, which is responsible for writing the data operation logs in the asynchronous log recording area of the primary data center to the asynchronous log execution area of the backup data center.

When the log push module completes the cross-data center transmission of logs, it will notify the scheduling module. Then the scheduling module drives the log playback module of the backup data center to perform log playback.

3. Log playback.

The log recall module running in the backup data center starts to execute the log playback operation after receiving the notification from the scheduling module.

The log return access module reads the data logs stored in the asynchronous log execution area, and then decodes the contents of the logs to obtain the corresponding operations and related data of the logs, and then executes the operations again on the relevant nodes of the backup data center to make the backup data The data in the center is consistent with the data in the main data center. This enables data synchronization across data centers.

4. Data access across data centers.

Clients may obtain data by accessing the backup data center in two cases:

First, when the client cannot connect to the primary data center. This situation could be due to a failure in the primary data center, or it could be due to a network outage between the primary data center and the client.

When this happens, the client will attempt to read from the backup datacenter, and will only be able to. In addition, because it is uncertain whether the data between the backup data center and the primary data center has been synchronized at this time, the client will display relevant prompt information to the user, informing the user that the source of the data at this time is the backup data center, and There is a data consistency problem.

Second, the client can connect to the primary data center, but the primary data center is busy.

When this happens, if the client judges that the client’s operation is a read-only operation, it will confirm to the primary data center whether the data involved in the operation has been synchronized to the backup data center. If the primary data center informs the client that the data has been synchronized, Then the client can obtain the data that the client wants to read by accessing the backup data center. At this point, the backup data center plays a load balancing role.

The data synchronization method across data centers of the present invention can realize asynchronous data synchronization operations across data centers and improve data security. When users cannot access the primary data center, they can also obtain data by accessing the backup data center. It can effectively reduce the amount of data generated by synchronization between data centers. Since only the difference of the data needs to be transmitted during the playback process without transmitting the data itself, the amount of transmitted data can be reduced and the bandwidth occupied by the synchronization operation between data centers can be reduced. It can be scheduled according to the load of the data center, effectively utilize the IO resources inside the data center and the network resources between the data centers, and play the role of load balancing.

The above description is only an embodiment of the present invention, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (3)

1. A data synchronization method across data centers, characterized in that its specific implementation process is: 1. Complete data writing and log recording: run the log recording module in the main data center. When the main data center receives the data request from the client, the module will record the required operations in the main data center in the form of logs. Data center, this module is integrated into the business process of the main data center in the form of embedded or plug-in; The detailed process of step 1 is: the client identifies the primary data center where the customer data is located according to the local configuration, and sends all data operations to the primary data center for processing by the data nodes of the primary data center; the primary data After the center receives the client's request, it executes the client's operation according to the requested operation and content. During this process, the logging module captures the client's requested operation and related data by intercepting the request; the logging module judges whether the client's operation is The data in the data center needs to be modified. If necessary, the data operation needs to be used as the content of the cross-data center data synchronization operation. At this time, the logging module saves the requested operation and related data to the master data in a proprietary log format The asynchronous logging area of the center, the content of this area is the content that needs to be played back across data centers; 2. Synchronous scheduling and push: set the scheduling module to run in the primary data center, which is responsible for scheduling data playback operations, and activate log push and playback operations according to the load of the primary data center, the load of the backup data center, and scheduling policy information; The push operation required by the scheduling module is completed through the log push module, which runs in the primary data center and transmits the data operation log to the backup data center; 3. Log playback and complete data synchronization: The data operation execution pushed by the log push module of the primary data center is received by the log playback module. The log playback module runs in the backup data center and plays back the data operation logs in the current data center, realizing two Data synchronization in the data center; The detailed process of step 3 is: the log playback module running in the backup data center starts to execute the playback operation of the log after receiving the notification from the scheduling module, and the log playback module reads the data log stored in the asynchronous log execution area, Then decode the content of the log to obtain the operation and related data corresponding to the log, and then execute the operation again on the relevant nodes in the backup data center, so that the data in the backup data center is consistent with the data in the primary data center, and the data crossover is realized. data center synchronization; 4. Perform data access across data centers to realize asynchronous data synchronization operations. 2. A method for synchronizing data across data centers according to claim 1 or 2, characterized in that: the detailed process of step 2 is: first, the following conditions are monitored by a scheduling module running in the main data center, 1) The number of logs in the asynchronous logging area and the amount of data involved; 2) The load of the main data center, including network IO and disk IO; 3) Backup data center load, including network IO and disk IO; When the above three meet the scheduling policy set by the configuration administrator, the log push operation is triggered, and the log push operation is performed by the log push module, which is responsible for writing the data operation logs in the asynchronous log recording area of the primary data center to the backup data center The asynchronous log execution area; when the log push module completes the cross-data center transmission of logs, it will notify the scheduling module, and then the scheduling module will drive the log playback module in the backup data center to perform log playback. 3. A cross-data center data synchronization method according to claim 2, characterized in that: the client in step 4 obtains data by accessing the backup data center in the following two cases, and the client cannot connect to the main data Center; clients can connect to the main data center, but the main data center is busy.