CN115048453A - Data synchronization method, device, equipment and storage medium - Google Patents

Abstract

The present application provides a method, apparatus, device and storage medium for data synchronization. The method includes: determining an active group in a group to which each slave library registered in the master library belongs; judging whether a synchronization confirmation message for a transaction to be synchronized by at least one slave library in each active group is received, and if so, submitting the pending transaction Synchronous transactions. The present application utilizes at least one slave library in each active group to realize the data synchronization of the transaction to be synchronized, and does not need to complete the data synchronization of the transaction to be synchronized in each slave library, thereby ensuring the transaction processing performance of the data synchronization. Moreover, there is at least one slave library in each active group to complete the data synchronization of the transaction to be synchronized, so that when the entire active group where the master library is located fails, the master-slave library can still be realized by using the slave libraries that complete data synchronization in other active groups switch, thereby improving the disaster recovery range of data synchronization.

Description

A method, device, device and storage medium for data synchronization

technical field

The embodiments of the present application relate to the technical field of data processing, and in particular, to a method, apparatus, device, and storage medium for data synchronization.

Background technique

In order to ensure the security of business data, a master-slave architecture is usually used to build a corresponding database cluster during data storage, so as to realize the master-slave synchronization processing of various business data, thereby improving the high availability and scalability of database storage. At present, asynchronous replication or full synchronous replication is usually used for data synchronization.

However, in asynchronous replication, the master library does not pay attention to the data synchronization of the current transaction from the slave library, and directly returns the transaction processing results to the client. When the master library fails, the slave library may not have completed data synchronization, which causes the slave library to switch to Incomplete data after the main database greatly reduces the disaster recovery effect of data synchronization. In full synchronous replication, the master database will wait for all slave databases to complete the data synchronization of the current transaction before returning the transaction processing result to the client, which greatly affects the transaction processing performance. Therefore, in order to solve the above problems, it is urgent to design a more efficient disaster recovery data synchronization scheme.

SUMMARY OF THE INVENTION

The present application provides a method, device, device and storage medium for data synchronization. The method of group synchronization is used to improve the disaster tolerance range of data synchronization on the basis of ensuring transaction processing performance.

In a first aspect, an embodiment of the present application provides a method for data synchronization, the method comprising:

Determine the active group in the group to which each slave library registered in the master library belongs;

It is judged whether a synchronization confirmation message for a transaction to be synchronized from at least one slave library in each of the active groups is received, and if so, the transaction to be synchronized is submitted.

In a second aspect, an embodiment of the present application provides an apparatus for data synchronization, the apparatus comprising:

The grouping determination module is used to determine the active group in the group to which each slave library registered in the master library belongs;

The data synchronization module is used for judging whether a synchronization confirmation message of at least one slave database in each active group is received for the transaction to be synchronized, and if so, the transaction to be synchronized is submitted.

In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

A processor and a memory, where the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method for data synchronization provided in the first aspect of the present application.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, the computer program causing a computer to execute the data synchronization method provided in the first aspect of the present application.

In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program/instruction, characterized in that, when the computer program/instruction is executed by a processor, the method for data synchronization as provided in the first aspect of the present application is implemented .

In a data synchronization method, device, device and storage medium provided by the embodiments of the present application, a group is set for each slave library registered in the master library, an active group is determined from each group, and then each slave library is received in the master library. When at least one slave library in an active group has a synchronization confirmation message of the transaction to be synchronized, the commit operation of the transaction to be synchronized is performed, so that at least one slave library in each active group is used to realize the data synchronization of the transaction to be synchronized, and there is no need to perform the data synchronization of the transaction to be synchronized in each active group. Complete the data synchronization of the transaction to be synchronized from the library to ensure the transaction processing performance of the data synchronization. Moreover, there is at least one slave library in each active group to complete the data synchronization of the transaction to be synchronized, so that when the entire active group where the master library is located fails, the master-slave library can still be realized by using the slave libraries in other active groups that complete data synchronization switch, thereby improving the disaster recovery range of data synchronization.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a flowchart of a method for data synchronization shown in an embodiment of the application;

2 is a flowchart of another data synchronization method shown in an embodiment of the application;

3 is a schematic block diagram of an apparatus for data synchronization shown in an embodiment of the application;

FIG. 4 is a schematic block diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

Considering the asynchronous replication and full synchronous replication during data synchronization, the disaster tolerance effect of data synchronization and the transaction processing performance are not compatible, the present application designs a new data synchronization scheme. The transaction to be synchronized is submitted by grouping each slave library of the master library, and after it is determined that at least one slave library in each active group in each group has completed the synchronization confirmation of the transaction to be synchronized. At this time, at least one slave library in each active group is used to realize the data synchronization of the transaction to be synchronized, and there is no need to complete the data synchronization of the transaction to be synchronized in each slave library, thereby ensuring the transaction processing performance of the data synchronization. Moreover, there is at least one slave library in each active group to complete the data synchronization of the transaction to be synchronized, so that when the entire active group where the master library is located fails, the master-slave library can still be realized by using the slave libraries in other active groups that complete data synchronization switch, thereby improving the disaster recovery range of data synchronization.

FIG. 1 is a flowchart of a data synchronization method according to an embodiment of the present application. 1, the method may specifically include the following steps:

S110: Determine the active group in the group to which each slave library registered in the master library belongs.

The transaction to be synchronized in the present application refers to various database transactions pointed to by the transaction request executed by the main database for the transaction request initiated by the user to the main database. At this time, in order to ensure data synchronization between the master and slave libraries, for any transaction processed by the master library, the slave library is required to complete the same processing once to ensure that the same transaction data exists between the master and slave libraries.

As an optional implementation solution in this application, when data synchronization is performed between the master and slave databases, after the master database completes the processing of a transaction, the transaction will be regarded as the transaction to be synchronized in this application. Moreover, the main library will generate a binary log (binlog) of the transaction to be synchronized. Furthermore, each subsequent slave library may request the master library for the binary log of the transaction to be synchronized, so as to perform the data synchronization operation of the transaction to be synchronized on the master and slave libraries.

In this application, in order to achieve the effect of compatible transaction processing performance and data synchronization disaster tolerance performance during data synchronization, that is, on the basis of improving the efficiency of the main database for transaction processing as much as possible, it can also ensure that transactions are performed in each slave database. Data synchronization on the database has a large disaster recovery range to improve the disaster recovery of data synchronization.

At this time, in order to ensure that the data synchronization has a large disaster tolerance range on the slave database, this application will set a group of information for each slave database registered in the master database. When registering each slave library to the master library, each slave library is divided into different groups in advance, so that the master library can set the corresponding data synchronization logic for the slave libraries in different groups, so that data synchronization can have the functions under the group level. Disaster recovery performance.

It should be noted that since each slave library registered in the master library may be deployed in multiple computer rooms, if a computer room fails or is blocked, the data synchronization operation of all slave libraries in the computer room for transactions may fail, and then When the master database fails, if a slave database in the computer room is switched to the master database, the data of the master database after the switch will be incomplete, which greatly reduces the disaster recovery effect of data synchronization. Therefore, in the present application, each slave library in the same computer room can be divided into the same group according to the computer room where each slave library is located, and then it is judged whether there is at least one slave library in different groups that successfully completes the data synchronization operation for the transaction, so that in the The data synchronization success of the transaction is judged at the group level to improve the disaster tolerance range during data synchronization.

It should be understood that, by configuring information such as the Internet Protocol (Internet Protocol, IP) address, user name and password of the master library on each slave library, each slave library can be connected to the master library, and each slave library can be registered. to the main library. Moreover, when each slave library registers with the master library, a data packet will be set, and the data packet contains information such as the service identifier, IP address, port, and group of the slave library. Furthermore, after receiving the registration information of each slave library, the master library can determine the group to which each slave library belongs.

As an optional implementation solution in this application, considering the data synchronization of any transaction, all the slave libraries in the same group may affect the successful data synchronization of any slave library in the group due to group failure or blocking sex. However, the slave databases in other groups can normally perform the data synchronization operation of the transaction, and the disaster recovery performance of data synchronization when the master database fails can still be guaranteed. Therefore, the present application can distinguish each group into an active group that can normally perform data synchronization and a blocked group that cannot perform data synchronization normally due to group failure or blocking.

In the present application, in order to prevent the failure or blocking of a certain group from affecting the transaction processing performance, the present application can find out each active group from all the groups to which each slave library registered in the master library belongs. Further, it is sufficient to determine the data synchronization status of the transaction with respect to the transaction from the slave library in each active group.

S120: Determine whether a synchronization confirmation message for a transaction to be synchronized from at least one slave library in each active group is received, and if so, submit the transaction to be synchronized.

After determining each active group in the group to which the slave library belongs, for any transaction to be synchronized executed by the master library, the slave library in each active group will continuously request the master library for each transaction according to the progress of its own synchronization of transaction data. A binary log generated after a transaction to be synchronized is processed by the master library, so that each slave library executes the binary log to synchronize the data of the transaction to be synchronized. After a slave library completes the data synchronization operation of the transaction to be synchronized, the slave library will reply a synchronization confirmation message for the transaction to be synchronized to the master library, so that the master library knows that the slave library has completed the data synchronization of the transaction to be synchronized. , that is, the slave library is currently consistent with the data in the master library.

As an optional implementation solution in the present application, in order to realize the disaster recovery of data synchronization at the group level as much as possible under the premise of ensuring the efficiency of transaction processing, the present application will determine whether there is at least one slave in each active group. The library has completed data synchronization for the transaction to be synchronized. Specifically, after the slave library completes the data synchronization operation of the transaction to be synchronized, it will reply to the master library a synchronization confirmation message for the transaction to be synchronized. Therefore, it is determined whether there is at least one slave library in each active group to complete the data synchronization of the to-be-synchronized transaction by judging whether the master library has received the synchronization confirmation message of at least one slave library in each active group.

At this time, if the master library receives the synchronization confirmation message of at least one slave library in each active group for the transaction to be synchronized, it means that there is at least one slave library in each active group that has completed the data synchronization of the transaction to be synchronized, that is, each active group has completed the data synchronization of the transaction to be synchronized. There will be at least one slave database in the group that is consistent with the master database data in real time. Therefore, when the master library receives the synchronization confirmation message of the transaction to be synchronized from at least one slave library in each active group, it can directly submit the to-be-synchronized transaction, thereby improving the efficiency of transaction processing as much as possible. Moreover, even when the master database fails, there will be at least one slave database in each active group that is consistent with the data of the master database in real time, so the slave database in any active group that is consistent with the data of the master database in real time can be directly switched to the master database. , to ensure the data integrity when the master-slave database is switched, so as to realize the data synchronization disaster recovery at the group level (computer room level).

Exemplarily, in this application, an acknowledgement receiving (ack_receiver) thread is set in the main library, and the ack_receiver thread is used to receive, after each slave library in any group completes the data synchronization of any transaction to be synchronized, the reply for the The synchronization confirmation message of the transaction to be synchronized is executed, and the judgment logic of whether the master library has received the synchronization confirmation message of the transaction to be synchronized by at least one slave library in each active group is executed.

In addition, considering that at the beginning of data synchronization between the master and slave databases, the data synchronization performance of each group for transactions is uncertain, so each group is used as an active group. Then, in the subsequent actual data synchronization process, by analyzing the data synchronization performance of each group for the transaction in real time, the group in the faulty or blocked state is converted from the active group to the blocked group.

Therefore, in the actual data synchronization process, the present application also determines the data synchronization performance of each active group for transactions in real time. That is, for any active group, if the master library does not receive a synchronization confirmation message for a transaction to be synchronized from any slave library in the active group after the preset synchronization duration exceeds, the active group will be converted into a blocking group. After the master library sends the binary log of the transaction to be synchronized to each slave library in any active group, it will determine the data synchronization duration of each slave library in the active group for the transaction to be synchronized. If the data synchronization duration of each slave database in the active group exceeds the preset synchronization duration, the master database has not received the synchronization confirmation message for the transaction pending synchronization from any slave database in the active group, indicating that the active database is active. The group is faulty or blocked for the data synchronization of the transaction to be synchronized, so the active group is directly converted into a blocking group. Exemplarily, the present application can directly dissolve the blocking group and release each slave library in the blocking group, so that the slave library uses asynchronous Restore normal data synchronization performance from the repository.

Further, if the blocking group is to be restored to the active group, it is necessary to determine whether the data synchronization progress of each slave library in the blocking group for the transaction to be synchronized can be maintained with the data synchronization progress of each slave library in each active group. Consistent. Therefore, if a synchronization confirmation message of any slave library is received, and the group to which the slave library belongs is a blocking group, the group synchronization point of the blocking group is determined according to the synchronization confirmation message of the slave library; the group synchronization point of the blocking group is determined. When the point is greater than or equal to the minimum group synchronization point in each active group in the main library, the blocking group is converted back to the active group.

At this time, by analyzing the data synchronization progress of each slave library for the transaction to be synchronized, a synchronization point can be set for each slave library, and the synchronization point is used to indicate the to-be-synchronized transaction that currently needs to perform data synchronization in each slave library The log offset position in the log file, for example, the synchronization point in each slave library can be file_name+file_pos. Moreover, the binary logs of each transaction to be synchronized in the present application will be sequentially arranged in the log file according to the transaction synchronization sequence.

Therefore, when the master library receives a synchronization confirmation message for a transaction to be synchronized from any slave library, it first determines whether the slave library belongs to the active group or the blocking group. If the group to which the slave library belongs is an active group, it is sufficient to directly determine whether the master library has received a synchronization confirmation message for a transaction to be synchronized by at least one slave library in each active group. However, if the group to which the slave library belongs is a blocking group, it is necessary to analyze the latest synchronization confirmation message that each slave library in the blocking group has replied to the master library. Then, according to the latest synchronization confirmation message that each slave library in the blocking group has replied to the master library, the synchronization point of each slave library in the blocking group is determined, that is, the current data synchronization progress of the transaction to be synchronized by the slave library. Further, from the synchronization points of each slave library in the blocking group, the largest synchronization point, that is, the slave library synchronization point with the fastest data synchronization progress, can be selected as the group synchronization point of the blocking group.

Meanwhile, according to the same way of determining the group synchronization point, the group synchronization point of each active group can be determined by analyzing the synchronization points of each slave library in each active group. Then, considering that the group synchronization point of each active group in the main library can represent the data synchronization progress of each active group for the transaction to be synchronized, therefore, by synchronizing the group synchronization point of the blocking group with the smallest group in each active group in the main library By comparing the points, it can be determined whether the data synchronization progress of the blocking group for the transaction to be synchronized is consistent with the data synchronization progress of the transaction to be synchronized in each active group. If the group synchronization point of the blocking group is greater than or equal to the minimum group synchronization point in each active group in the main database, it means that the data synchronization progress of the blocking group has been consistent with the data synchronization progress of each active group, so the blocking group is converted back to the active group. At this point, if the blocking group has been disbanded, the group is recreated and configured as an active group. However, if the group synchronization point of the blocking group is smaller than the minimum group synchronization point among the active groups in the main database, it means that the data synchronization progress of the blocking group has not caught up with the data synchronization progress of each active group, so continue to maintain the blocking group status.

The technical solution provided by the embodiment of the present application is to set up groups for each registered slave library in the master library, and determine an active group from each group, and then receive at least one slave library in each active group in the master library to be synchronized transactions. When the synchronization confirmation message is sent, the commit operation of the transaction to be synchronized is performed, so that at least one slave library in each active group is used to realize the data synchronization of the transaction to be synchronized, and there is no need to complete the data synchronization of the transaction to be synchronized in each slave library. Guaranteed transaction processing performance for data synchronization. Moreover, there is at least one slave library in each active group to complete the data synchronization of the transaction to be synchronized, so that when the entire active group where the master library is located fails, the master-slave library can still be realized by using the slave libraries in other active groups that complete data synchronization switch, thereby improving the disaster recovery range of data synchronization.

As an optional implementation solution in this application, in order to improve the disaster tolerance range during data synchronization on the premise of ensuring transaction processing performance, this application determines whether the master database has received at least one slave database in each active group. The specific process of the synchronization confirmation message of the synchronization transaction is explained in detail.

FIG. 2 is a flowchart of another data synchronization method according to an embodiment of the present application. 2, the method may specifically include the following steps:

S210: Determine the active group in the group to which each slave library registered in the master library belongs.

S220, when receiving a synchronization request for a transaction to be synchronized from any slave library, send a synchronization log of the transaction to be synchronized to the slave library.

After the main library executes any transaction to be synchronized, a binary log of the transaction to be synchronized will be generated in the log file as needed. At this time, in the process of data synchronization between the master and slave libraries, considering that the data synchronization performance of each slave library for the transaction to be synchronized is different, if the master library actively sends the binary log of each transaction to be synchronized to each slave library, it may There is a problem that the slave library accumulates a large number of binary logs of transactions to be synchronized due to the slow synchronization performance, which may cause errors in the data synchronization of the transactions to be synchronized by the slave library and fail. Therefore, in order to ensure the orderliness and accuracy of the data synchronization of each slave library for the transaction to be synchronized, in this application, each slave library will continuously send the data synchronization progress of the transaction to be synchronized to the master library. A sync request for a sync transaction.

Furthermore, each time the master library receives a synchronization request for a transaction to be synchronized from any slave library, the transaction identifier carried in the synchronization request can be parsed. Then, the binary log generated in the main database of the transaction to be synchronized specified by the transaction identifier is obtained as the synchronization log of the transaction to be synchronized in the present application. Further, the synchronization log of the transaction to be synchronized is sent to the slave library, so that the slave library writes the synchronization log into the relay log (Relay Log) in the slave library, thereby completing the data synchronization of the transaction to be synchronized operate.

At this time, in order to ensure the accuracy of the synchronization log sent by the master library to each slave library, the present application will configure a dump thread for each slave library in the master library. When the master library receives a synchronization request for a transaction to be synchronized from any slave library, it will use the dump thread corresponding to the slave library to send the synchronization log of the transaction to be synchronized to the slave library.

It should be understood that, in order for the master library to know the data synchronization progress of each slave library in each active group for the transaction to be synchronized, the slave library in this application will perform the data synchronization operation for a transaction to be synchronized each time. Reply to the main library with a synchronization confirmation message for the transaction to be synchronized. Furthermore, the master library judges the overall synchronization progress of each slave library in each active group with respect to the transaction to be synchronized by receiving the synchronization confirmation message of each slave library in each active group.

As an optional implementation solution in the present application, the present application mainly performs the commit operation of the to-be-synchronized transaction by judging whether there is at least one slave database in each active group to complete the data synchronization of the to-be-synchronized transaction. That is, the application mainly focuses on the first slave database in each active group that replies the synchronization confirmation message of the transaction to be synchronized to the master database, and does not need to pay attention to whether other slave databases reply the synchronization confirmation message of the transaction to be synchronized to the master database. Therefore, when sending the synchronization log of the transaction to be synchronized to any slave library, the present application will first determine the group synchronization point of the group to which the slave library belongs, so as to determine the fastest synchronization progress of the group to which the slave library belongs. Moreover, the synchronization position identifier of the transaction to be synchronized that needs to be synchronized this time by the slave library is analyzed to determine the current data synchronization progress of the slave library.

Then, by comparing the group synchronization point of the group to which the slave library belongs and the synchronization position identifier of the transaction to be synchronized this time in the slave library to determine the fastest synchronization progress of the transaction to be synchronized by the group to which the slave library belongs and the transaction to be synchronized by the slave library. The difference between the current data synchronization progress of the synchronization transaction.

At this time, the comparison results of the group synchronization point of the group to which the slave library belongs and the synchronization position identifier of the transaction to be synchronized this time by the slave library exist in the following three situations:

Situation 1: If the group synchronization point of the group to which the slave library belongs is empty, it means that each slave library in the group to which the slave library belongs has not replied to the master library any synchronization confirmation messages for transactions to be synchronized, so that the master library needs to pay attention to The data synchronization status of each slave library in the group is used to accurately determine the group synchronization point of the group subsequently. Therefore, when the master library sends the synchronization log of the transaction to be synchronized to the slave library, it will not make any additional configuration on the synchronization log, so that the slave library will normally send the synchronization log to the master library after completing the data synchronization operation of the transaction to be synchronized. Reply with a corresponding synchronization confirmation message, so that the master library can determine the group synchronization point of the group to which the slave library belongs according to the synchronization confirmation message.

Case 2: If the synchronization location identifier of the transaction to be synchronized is greater than the group synchronization point of the group to which the slave library belongs, it means that the data synchronization progress of the slave library after completing the data synchronization operation of the transaction to be synchronized will be greater than the current maximum value of the group to which the slave library belongs. Fast synchronization progress. That is, after the slave library completes the data synchronization operation of the transaction to be synchronized, the fastest synchronization progress of the group to which the slave library belongs also needs to be updated accordingly. Therefore, when the master library sends the synchronization log of the transaction to be synchronized to the slave library, it will not perform any additional configuration on the synchronization log, so that the slave library can normally send the synchronization log to the slave library after completing the data synchronization operation for the transaction to be synchronized. The main library replies with a corresponding synchronization confirmation message. Then, the master library updates the group synchronization point of the group to which the slave library belongs according to the synchronization confirmation message replied by the slave library, thereby ensuring real-time update of the fastest synchronization progress of the group to which the slave library belongs.

Scenario 3: If the synchronization location identifier of the transaction to be synchronized is less than or equal to the group synchronization point of the group to which the slave library belongs, it means that the data synchronization progress of the slave library after completing the data synchronization operation of the transaction to be synchronized will still not affect the slave library to which it belongs. The current fastest synchronization progress of the group. Therefore, this application will additionally configure the corresponding synchronization confirmation cancellation information in the synchronization log of the transaction to be synchronized, and the synchronization confirmation cancellation information indicates that the slave database does not need to reply the data to the master database after completing the data synchronization operation of the transaction to be synchronized. Synchronization confirmation message for synchronization. Then, the additionally configured synchronization log of the transaction to be synchronized is sent to the slave library.

S230: Determine the group synchronization point of each active group according to the synchronization confirmation message replied by each slave library in each active group.

After the master library sends the synchronization log of the transaction to be synchronized to each slave library based on the synchronization request of any slave library to be synchronized, it will continue to receive synchronization confirmation messages replied by each slave library after completing the data synchronization operation of the transaction to be synchronized. . At this time, for each active group, the application will obtain the synchronization confirmation messages that have been currently received by each slave library in the active group. Then, the synchronization point of each slave library in the active group is determined by judging the synchronization position identifier of the transaction to be synchronized indicated by the synchronization confirmation message recently replied by each slave library in the active group. Finally, from the synchronization points of each slave library in the active group, the synchronization point indicating the fastest synchronization progress can be selected as the group synchronization point of the active group.

As an exemplary solution in this application, in order to accurately record the synchronization of each active group for transactions to be synchronized, this application will introduce an active group (activegroup) class into the main library, and construct a GroupNode linked list in the activegroup class (array) structure. Furthermore, in the main library, various types of information of each active group can be recorded through the GroupNode linked list. For example, the GroupNode linked list of each active group may include the group name, the group synchronization point, each slave library included in the group, and the like. Wherein, each group in this application may include, but is not limited to, 256 slave libraries at most.

S240, based on the group synchronization point of each active group, select the slowest synchronization group from each active group.

Considering that the master library will have at least one slave library in each active group that has completed the data synchronization of the transaction to be synchronized, that is, there will be at least one slave library in each active group that is consistent with the master library data in real time. The transaction to be synchronized. Therefore, the present application determines the fastest synchronization progress of the active group for the transaction to be synchronized by analyzing the group synchronization point of each active group. Then, according to the fastest synchronization progress of each active group for the transaction to be synchronized, the active group with the slowest synchronization progress can be selected from each active group as the slowest synchronization group in this application, so that the slowest synchronization group can be used in the subsequent synchronization according to the slowest synchronization group. The update status of the group synchronization point is analyzed, and the overall synchronization progress of each active group for the transaction to be synchronized is analyzed.

As an exemplary solution in the present application, when various types of information of each active group are recorded in the main library through the GroupNode linked list, the present application can also construct a specific pointer (*slowest_group) for the GroupNode linked list, which points to The slowest synchronized group in the GroupNode linked list.

S250, when it is detected that the group synchronization point of the slowest group has completed an update, determine the transaction to be synchronized corresponding to the group synchronization point before the synchronization of the slowest group is updated, and submit the transaction to be synchronized.

Since the slowest synchronization group belongs to the active group with the slowest data synchronization progress for the transactions to be synchronized in each active group, it means that at least one slave library in other active groups has completed the synchronization of each to-be-synchronized group before the synchronization point of the group in the slowest synchronization group. Transactional data synchronization. Therefore, it can be considered that the group synchronization point of the slowest synchronization group represents the overall synchronization progress of each active group for the transactions to be synchronized, and each transaction to be synchronized before the group synchronization point of the slowest synchronization group has been submitted.

Therefore, by judging whether the group synchronization point of the slowest synchronization group is updated, it can be determined whether there is at least one slave database in each active group to complete the data synchronization of the transaction to be synchronized. At this time, when the master library detects that the group synchronization point of the slowest synchronization group has completed an update, it means that there is also a slave library in the slowest synchronization group that has completed the data synchronization of the transaction to be synchronized indicated by the group synchronization point before the update. operation, so that there is at least one slave library in each active group to complete the data synchronization of the transaction to be synchronized represented by the group synchronization point before the synchronization of the slowest group update. Furthermore, the main database can directly submit the to-be-synchronized transaction corresponding to the group synchronization point before the slowest synchronization group is updated.

It should be noted that, in order to ensure the accuracy of the slowest group synchronization in the process of packet data synchronization, the present application will re-analyze the updated data of each active group when the group synchronization point of the slowest group is detected to complete an update. group synchronization point, and then re-elect the current slowest synchronization group.

Exemplarily, if the group synchronization point of the group pointed to by the *slowest_group pointer is not updated, the slowest synchronization group in this application is not changed. However, if the group synchronization point of the group pointed to by the *slowest_group pointer is updated, it is necessary to traverse the GroupNode linked list. Then, according to the group synchronization point of each active group currently recorded in the GroupNode linked list, a group with the slowest synchronization is re-elected, so as to ensure that the *slowest_group pointer can always point to the slowest synchronization group in the GroupNode linked list.

The technical solution provided by the embodiment of the present application is to set up groups for each registered slave library in the master library, and determine an active group from each group, and then receive at least one slave library in each active group in the master library to be synchronized transactions. When the synchronization confirmation message is sent, the commit operation of the transaction to be synchronized is performed, so that at least one slave library in each active group is used to realize the data synchronization of the transaction to be synchronized, and there is no need to complete the data synchronization of the transaction to be synchronized in each slave library. Guaranteed transaction processing performance for data synchronization. Moreover, there is at least one slave library in each active group to complete the data synchronization of the transaction to be synchronized, so that when the entire active group where the master library is located fails, the master-slave library can still be realized by using the slave libraries in other active groups that complete data synchronization switch, thereby improving the disaster recovery range of data synchronization.

FIG. 3 is a schematic block diagram of a data synchronization apparatus according to an embodiment of the present application. As shown in FIG. 3, the apparatus 300 may include:

a grouping determination module 310, configured to determine the active group in the group to which each slave library registered in the master library belongs;

The data synchronization module 320 is configured to determine whether a synchronization confirmation message for a transaction to be synchronized is received from at least one slave database in each of the active groups, and if so, submit the transaction to be synchronized.

Further, the data synchronization module 320 can be specifically used for:

Determine the group synchronization point of each active group according to the synchronization confirmation message replied by each slave library in each active group;

Based on the group synchronization point of each active group, the slowest synchronization group is selected from each of the active groups;

When it is detected that the group synchronization point of the slowest synchronization group has completed an update, the transaction to be synchronized corresponding to the group synchronization point before the update of the slowest synchronization group is determined, and the transaction to be synchronized is submitted.

Further, the apparatus 300 for data synchronization may further include:

The slowest group update module is configured to reselect the slowest synchronization group based on the updated group synchronization point of each active group.

Further, the apparatus 300 for data synchronization may further include:

The synchronization log sending module is configured to send the synchronization log of the transaction to be synchronized to the slave library when receiving a synchronization request for the transaction to be synchronized from any slave library.

Further, the synchronization log sending module can be specifically used for:

If the group synchronization point of the group to which the slave library belongs is empty, or the synchronization position identifier of the transaction to be synchronized is greater than the group synchronization point of the group to which the slave library belongs, sending the synchronization log of the transaction to be synchronized to the slave library;

If the synchronization location identifier of the transaction to be synchronized is less than or equal to the group synchronization point of the group to which the slave library belongs, configure the corresponding synchronization confirmation cancellation information in the synchronization log of the transaction to be synchronized, and send the to-be-synchronized transaction to the slave library. Synchronization log for synchronization transactions.

Further, the apparatus 300 for data synchronization may further include:

The first group of conversion modules is used for any active group, if the master library has not received the synchronization confirmation message for the transaction to be synchronized from any slave library in the active group after exceeding the preset synchronization duration, then Convert the active group to a blocking group.

Further, the apparatus 300 for data synchronization may further include:

A group synchronization point determination module is used to determine the group synchronization point of the blocked group according to the synchronization confirmation message of the slave library if the synchronization confirmation message of any slave library is received, and the slave library belongs to a blocking group;

The second group conversion module is configured to convert the blocked group back to an active group when the group synchronization point of the blocked group is greater than or equal to the smallest group synchronization point in each active group in the main library.

In the embodiment of the present application, a group is set for each slave library registered in the master library, and an active group is determined from each group, and then the master library receives a synchronization confirmation of the transaction to be synchronized from at least one slave library in each active group message, to perform the commit operation of the transaction to be synchronized, so as to use at least one slave database in each active group to realize the data synchronization of the transaction to be synchronized, without the need to complete the data synchronization of the transaction to be synchronized in each slave database, to ensure data synchronization transaction processing performance. Moreover, there is at least one slave library in each active group to complete the data synchronization of the transaction to be synchronized, so that when the entire active group where the master library is located fails, the master-slave library can still be realized by using the slave libraries in other active groups that complete data synchronization switch, thereby improving the disaster recovery range of data synchronization.

It should be understood that the apparatus embodiments and the method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, details are not repeated here. Specifically, the apparatus 300 shown in FIG. 3 can execute the method embodiments provided in the present application, and the foregoing and other operations and/or functions of each module in the apparatus 300 are respectively to implement the corresponding processes in the various methods of the embodiments of the present application , and are not repeated here for brevity.

The apparatus 300 of the embodiments of the present application is described above from the perspective of functional modules with reference to the accompanying drawings. It should be understood that the functional modules can be implemented in the form of hardware, can also be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application may be completed by hardware integrated logic circuits in the processor and/or instructions in the form of software, and the steps of the methods disclosed in conjunction with the embodiments of the present application may be directly embodied as hardware The execution of the decoding processor is completed, or the execution is completed by a combination of hardware and software modules in the decoding processor. Optionally, the software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps in the above method embodiments in combination with its hardware.

FIG. 4 is a schematic block diagram of an electronic device 400 provided by an embodiment of the present application.

As shown in FIG. 4, the electronic device 400 may include:

A memory 410 and a processor 420, the memory 410 is used to store computer programs and transmit the program codes to the processor 420. In other words, the processor 420 may call and run a computer program from the memory 410 to implement the methods in the embodiments of the present application.

For example, the processor 420 may be configured to execute the above method embodiments according to the instructions in the computer program.

In some embodiments of the present application, the processor 420 may include, but is not limited to:

General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or Transistor logic devices, discrete hardware components, and more.

In some embodiments of the present application, the memory 410 includes but is not limited to:

Volatile memory and/or non-volatile memory. Wherein, the non-volatile memory may be Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (Erasable PROM, EPROM), Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double DataRate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) ) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

In some embodiments of the present application, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory 410 and executed by the processor 420 to complete the steps provided by the present application. method. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

As shown in Figure 4, the electronic device may further include:

A transceiver 430 , which can be connected to the processor 420 or the memory 410 .

The processor 420 can control the transceiver 430 to communicate with other devices, and specifically, can send information or data to other devices, or receive information or data sent by other devices. Transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include antennas, and the number of the antennas may be one or more.

It should be understood that various components in the electronic device are connected through a bus system, wherein the bus system includes a power bus, a control bus and a status signal bus in addition to a data bus.

The embodiments of the present application further provide a computer storage medium, on which a computer program is stored, and when the computer program is executed by a computer, the computer can execute the method of the above method embodiments. In other words, the embodiments of the present application further provide a computer program product including instructions, when the instructions are executed by a computer, the instructions cause the computer to execute the methods of the above method embodiments.

When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored on or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted over a wire from a website site, computer, server or data center (eg coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) means to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, digital video disc (DVD)), or semiconductor media (eg, solid state disk (SSD)), and the like.

Those of ordinary skill in the art can realize that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or modules, and may be in electrical, mechanical or other forms.

Modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical modules, that is, may be located in one place, or may be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. For example, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, and should cover within the scope of protection of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A method of data synchronization, comprising:

determining an active group in a group to which each registered slave library belongs in the master library;

and judging whether a synchronization confirmation message of at least one slave library to-be-synchronized transaction in each active group is received, and if so, submitting the to-be-synchronized transaction.

2. The method of claim 1, wherein the determining whether a synchronization confirmation message for at least one slave library in each of the active groups to a transaction to be synchronized is received, and if so, committing the transaction to be synchronized comprises:

determining a group synchronization point of each active group according to the synchronization confirmation message replied from each slave library in each active group;

based on the group synchronization point of each active group, selecting the slowest synchronization group from the active groups;

and when detecting that the group synchronization point of the slowest synchronization group completes one updating, determining a transaction to be synchronized corresponding to the group synchronization point before the slowest synchronization group is updated, and submitting the transaction to be synchronized.

3. The method of claim 2, wherein upon detecting that the group synchronization point of the slowest group to synchronize completes one update, the method further comprises:

and reselecting the slowest synchronization group based on the updated group synchronization point of each active group.

4. The method of claim 1, wherein after determining the active set in the group to which each slave repository registered in the master repository belongs, further comprising:

and when a synchronization request of any slave library to the transaction to be synchronized is received, sending a synchronization log of the transaction to be synchronized to the slave library.

5. The method of claim 4, wherein sending the synchronization log of the transaction to be synchronized to the slave library comprises:

if the group synchronization point of the group to which the slave library belongs is empty or the synchronization position identifier of the transaction to be synchronized is greater than the group synchronization point of the group to which the slave library belongs, sending a synchronization log of the transaction to be synchronized to the slave library;

and if the synchronization position identification of the transaction to be synchronized is less than or equal to the group synchronization point of the group to which the slave library belongs, configuring corresponding synchronization confirmation cancellation information in the synchronization log of the transaction to be synchronized, and sending the synchronization log of the transaction to be synchronized to the slave library.

6. The method of claim 1, further comprising:

and for any active group, if the master library still does not receive the synchronous confirmation message of any slave library in the active group to the transaction to be synchronized after the preset synchronous time length is exceeded, converting the active group into a blocking group.

7. The method of claim 6, further comprising:

if the synchronization confirmation message of any slave library is received and the slave library belongs to a block group, determining a group synchronization point of the block group according to the synchronization confirmation message of the slave library;

and when the group synchronization point of the blocking group is larger than or equal to the minimum group synchronization point in each active group in the master library, converting the blocking group back to the active group.

8. An apparatus for data synchronization, comprising:

the grouping determination module is used for determining active groups in the groups to which the registered slave libraries belong in the master library;

and the data synchronization module is used for judging whether a synchronization confirmation message of at least one slave library in each active group to the transaction to be synchronized is received, and if so, submitting the transaction to be synchronized.

9. An electronic device, comprising:

a processor and a memory, the memory for storing a computer program, the processor for invoking and executing the computer program stored in the memory to perform the method of data synchronization of any of claims 1-7.

10. A computer-readable storage medium for storing a computer program which causes a computer to perform the method of data synchronization according to any one of claims 1-7.

Images