CN117149907A - Data synchronization method, device, equipment and medium - Google Patents

Abstract

The invention discloses a data synchronization method, a device, equipment and a medium. The method comprises the following steps: acquiring a data synchronization task and a corresponding data synchronization rule, wherein the data synchronization rule is generated according to a selected preset rule template; when the data synchronization task and the data synchronization rule meet the synchronization pre-detection condition, acquiring data information to be synchronized from a source database corresponding to the data synchronization task; determining converted target data information according to a preset unified format and data information to be synchronized, and storing data; and synchronizing the target data information to a target database corresponding to the data synchronization task according to the data synchronization rule. Converting the data information to be synchronized through a preset unified format, storing the unified format data, and synchronizing the target data information to a target database according to the data synchronization rule. The method and the device realize universality and high efficiency of data synchronization among different types of databases, avoid manual configuration synchronization, and reduce technical requirements of data synchronization configuration personnel.

Description

Data synchronization methods, devices, equipment and media

Technical field

The present invention relates to the field of data processing technology, and in particular to data synchronization methods, devices, equipment and media.

Background technique

Under the diversified development trend of open source software and information technology lines, the types of databases in enterprises are becoming more and more abundant, resulting in the need for frequent relocation of enterprise data. The same data needs to be copied and saved multiple times in multiple data scenarios. Data copy. And during the frequent data copying and relocation process, data quality is difficult to guarantee.

Existing technical solutions convert the data format in the source database into the format corresponding to the target database, synchronize it to the target database, and perform comparisons during the synchronization process to ensure the accuracy of data synchronization.

However, the existing data synchronization solutions have poor versatility. Synchronization of multiple homogeneous and heterogeneous data requires one-to-one development of a large amount of code. Data quality is difficult to manage. The traditional data comparison solution requires full table scanning of both databases. Comparing the entire amount of data one by one, the data comparison takes a long time and the data differences are difficult to repair. In scenarios with large data volumes, the comparison time is long and it is difficult to meet users’ high consistency data quality requirements; the same data needs to be used frequently in different data usage scenarios. Moving and converting data requires a lot of time and resources for each data synchronization.

Contents of the invention

The invention provides a data synchronization method, device, equipment and medium to realize data synchronization between homogeneous and heterogeneous data sources.

According to a first aspect of the present invention, a data synchronization method is provided, including:

Obtain the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template;

When the data synchronization task and the data synchronization rules meet the synchronization pre-check condition, obtain the data information to be synchronized from the source database corresponding to the data synchronization task;

According to the preset unified format and the data information to be synchronized, determine the converted target data information and store the data;

According to the data synchronization rules, the target data information is synchronized to the target database corresponding to the data synchronization task.

According to a second aspect of the present invention, a data synchronization device is provided, including:

A rule acquisition module, used to obtain data synchronization tasks and corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template;

A first determination module, configured to obtain the data information to be synchronized from the source database corresponding to the data synchronization task when the data synchronization task and the data synchronization rule satisfy the synchronization pre-check condition;

The second determination module is used to determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized;

An information synchronization module, configured to synchronize the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules.

According to another aspect of the present invention, an electronic device is provided, the electronic device including:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the method described in any embodiment of the present invention. Data synchronization method.

According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions. The computer instructions are used to enable a processor to implement any embodiment of the present invention when executed. Data synchronization method.

The technical solution of the embodiment of the present invention is to obtain the data synchronization task and the corresponding data synchronization rule, wherein the data synchronization rule is generated according to the selected preset rule template; when the data synchronization task and the data synchronization rule satisfy the synchronization pre-check condition, Obtain the data information to be synchronized from the source database corresponding to the data synchronization task; determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized; synchronize the target data information to the data synchronization according to the data synchronization rules The target database corresponding to the task. Convert the data information to be synchronized through the preset unified format, store the unified format data, and synchronize the target data information to the target database according to the data synchronization rules. It achieves the versatility and efficiency of data synchronization between different types of databases, eliminating the need for manual configuration of synchronization and reducing the technical requirements for data synchronization configuration personnel.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the invention, nor is it intended to limit the scope of the invention. Other features of the present invention will become easily understood from the following description.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a data synchronization method provided according to Embodiment 1 of the present invention;

Figure 2 is an example flow chart of a data synchronization method provided according to Embodiment 1 of the present invention;

Figure 3 is a schematic structural diagram of a data synchronization device provided according to Embodiment 2 of the present invention;

Figure 4 is a schematic structural diagram of an electronic device implementing an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only These are some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the invention described herein are capable of being practiced in sequences other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus that encompasses a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

Embodiment 1

Figure 1 is a flow chart of a data synchronization method provided by Embodiment 1 of the present invention. This embodiment can be applied to data synchronization between multi-source homogeneous and heterogeneous databases. The method can be executed by a data synchronization device. The data synchronization device can be implemented in the form of hardware and/or software, and the data synchronization device can be configured in electronic equipment. As shown in Figure 1, the method includes:

S110. Obtain the data synchronization task and the corresponding data synchronization rules, where the data synchronization rules are generated according to the selected preset rule template.

In this embodiment, the data synchronization task can be understood as a task established by the user through a graphical interface for synchronizing data in the source database and the target database. You can create data synchronization tasks and edit data synchronization rules by dragging and clicking on the graphical interface. Users can create heterogeneous synchronization tasks through task scheduling, providing task start, task stop, task deletion, automated task start and stop, task queue queuing, etc.

In this embodiment, data synchronization rules can be understood as rules adopted when synchronizing heterogeneous data. These can include library table correspondence, field correspondence, field addition and subtraction rules, empty field filling in values, data desensitization rules, field type conversion, field value conversion rules, exception data processing rules, response rules after synchronization exceptions, etc. Users can choose from the provided preset rule templates or customize and generate corresponding data synchronization rules. Among them, the preset rule templates include templates generated for common data synchronization requirements. The processor can also manage and save the user's data synchronization rules. Users can share customized data synchronization rules to the template center.

Specifically, the processor can obtain the data synchronization task and corresponding data synchronization rules created by the user through the graphical interface.

S120. When the data synchronization task and the data synchronization rules meet the synchronization pre-check conditions, obtain the data information to be synchronized from the source database corresponding to the data synchronization task.

In this embodiment, the synchronization pre-check condition can be understood as a condition used to check the risk of task execution and the risk of impact on the source end and the target end before data synchronization. The source database can be understood as a relational database that requires data conversion. The data information to be synchronized can be understood as the data information that the source database needs to synchronize.

Specifically, after obtaining the data synchronization task and data synchronization rules, the processor can automatically check the compliance of the data synchronization rule configuration according to the set process, determine whether it is compliant according to the dependency conditions of the source database and the target end, and check the source database. The degree of impact on the database and the target end. When the preflight passes, the processor can add the data synchronization task to the task queue, add task monitoring, and load the data synchronization rules. When the data synchronization task and data synchronization rules meet the synchronization preflight conditions, the processor It can fully extract and incrementally extract the database structure, database objects and data of the source database, and provides the ability to resume transmission at breakpoints to obtain data information to be synchronized.

S130. According to the preset unified format and the data information to be synchronized, determine the converted target data information and store the data.

In this embodiment, the preset unified format can be understood as a preset standard data format, which is used to convert databases in different types of databases into data in a preset unified format. Including database table structure information, primary key information, transaction operation type, transaction execution time, data collection time, original SQL executed at the source, data information of affected rows before SQL execution, data information of affected rows after SQL execution, log information, etc. . The target data information can be understood as converted data in a preset unified format.

Specifically, the processor can convert the data information to be synchronized according to a preset unified format, and can also provide data processing such as data persistence, data encryption, and data compression to obtain the converted target data information and store the data.

S140. According to the data synchronization rules, synchronize the target data information to the target database corresponding to the data synchronization task.

What needs to be known is that users need to provide access rights to the source database, including connection, query, etc.; they need to provide access rights to the target database, including connection, query, etc.; they need to provide transaction log access to the source database, so that It can read the database transaction log and parse the transaction records in it.

In this embodiment, the target database can be understood as the database that needs to be written. It can be a relational database, a non-relational database, a distributed database, big data components and data storage, etc.

Specifically, the processor can convert the target data information into SQL statements of the target database according to the data synchronization rules, and load them into the target database in batches or real-time according to the data synchronization rules.

The technical solution of the embodiment of the present invention is to obtain the data synchronization task and the corresponding data synchronization rule, wherein the data synchronization rule is generated according to the selected preset rule template; when the data synchronization task and the data synchronization rule satisfy the synchronization pre-check condition, Obtain the data information to be synchronized from the source database corresponding to the data synchronization task; determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized; synchronize the target data information to the data synchronization according to the data synchronization rules The target database corresponding to the task. Convert the data information to be synchronized through the preset unified format, store the unified format data, and synchronize the target data information to the target database according to the data synchronization rules. It achieves the versatility and efficiency of data synchronization between different types of databases, eliminating the need for manual configuration of synchronization and reducing the technical requirements for data synchronization configuration personnel.

Further, based on the above embodiment, the steps of synchronizing the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules can be optimized as follows:

a1. According to the data synchronization rules, convert the target data information into structured query language SQL statements.

In this embodiment, SQL statements can be understood as statements used to store data.

Specifically, the processor can convert the target data information into SQL statements according to the conversion method recorded in the data synchronization rules.

b1. Synchronize the SQL statement to the target database corresponding to the data synchronization task.

Specifically, the processor can synchronously transmit the SQL statement to the corresponding target database.

According to the technical solution of the embodiment of the present invention, the data synchronization rules to be configured can be selected through simple page editing operations, complex heterogeneous data conversion is realized based on the data synchronization rules, and synchronous migration between homogeneous and heterogeneous data sources is realized. Its versatility realizes the data conversion, processing and loading capabilities required in the process of homogeneous or heterogeneous data synchronization, ensuring stable and efficient data synchronization capabilities. Acquisition of real-time or streaming data can be achieved by using simple SQL queries or low-code.

As a first optional embodiment of Embodiment 1, after writing the target data information into the target database corresponding to the data synchronization task according to the data synchronization rules, it also includes:

When the data synchronization task meets the data management conditions, data fingerprint information is generated based on the target data information and stored in the data fingerprint database.

In this embodiment, the data management conditions can be understood as conditions used for data quality verification. Data fingerprint information can be understood as a unique identifier representing data, which is used for data verification and quick query. The data fingerprint library can be understood as a library used to store data fingerprint information.

Specifically, the user can choose whether to perform data management in the provided graphical interface. When the user chooses to perform data management, the data synchronization task also includes data management instructions, and the processor can parse the data synchronization task to determine which includes When the data management instruction is issued, the data synchronization task meets the data management conditions, and the processor can generate data fingerprint information based on the row data primary key, row core field, entire row full data, etc. converted into the target data information record in the preset unified format.

In the first optional embodiment of Embodiment 1, through such settings, when the data synchronization task meets the data management conditions, the target data is automatically converted into data fingerprint information and stored in the data fingerprint database, which provides a basis for subsequent data quality management. Basically, the method of forming data fingerprint information can also reduce the data scanning time and data occupation required for data comparison.

As a second optional embodiment of Embodiment 1, based on the above embodiment, it also includes:

Obtain the database management request relative to the source database, and determine the managed target information based on the database management request.

In this embodiment, a database management request can be understood as a request for data acquisition. Target information can be understood as information determined relative to the database management request.

Specifically, the processor can obtain the database management request relative to the source database generated by the user through simple SQL, low-code or SDK components, call the preset data service and DataAPI capabilities according to the database management request, and determine the managed target information. .

Further, based on the above embodiment, the step of determining the managed target information according to the database management request can be optimized as follows:

a2. Determine the data information to be managed that matches the database management request in the stored data, where the data information to be managed is data converted according to a preset unified format.

In this embodiment, the data information to be managed can be understood as stored data that has been converted to a preset unified format, and can be stored during the synchronization process of the source database.

Specifically, the processor can search the stored data for data information to be managed that matches the database management request.

b2. Manage the data information to be managed through the preset data service or the preset data application interface to obtain the target management data.

In this embodiment, the preset data service can be understood as a unified data streaming or batch processing capability provided in the form of an application program. The default data application program interface can be understood as providing unified data streaming or batch processing capabilities in the form of an application program interface, and providing a unified data query interface to the outside world, such as DataAPI. Target management data can be understood as data that satisfies database management requests.

Specifically, the processor can provide data processing functions that meet database management requests through a preset data service or a preset data application interface, such as data report analysis and other data processing functions, manage the data information to be managed, and obtain target management data.

A second optional embodiment of the first embodiment, through such a setting, when receiving a database management request, the data in the unified format in the stored data is processed by calling the default data service or the default application interface. manage. By storing data in a unified format and providing data services, a piece of data can be used multiple times. The same data only needs one copy and migration to implement data services, which reduces the need to extract data from the source database and convert it during data services. time.

As a third optional embodiment of Embodiment 1, based on the above embodiment, it also includes:

When the data synchronization task meets the quality management conditions, quality management is performed on the source database corresponding to the data synchronization task to determine the quality management results of the source database.

In this embodiment, quality management conditions can be understood as conditions for determining data quality. For example, quality management can be performed regularly in a set period or other manner. Quality management results can be understood as the results used to determine whether the data in the source database is consistent with the synchronized data.

Specifically, the user can pre-select whether to perform quality management, and the selection result can be added to the data synchronization task through instructions. When the data synchronization task includes quality management instructions, the processor starts to execute the data synchronization task. , when the execution duration meets the quality management conditions, the processor can perform quality management on the data in the source database corresponding to the data synchronization task and the converted data, and determine the quality management results of the source database.

Furthermore, quality management can be performed on the source database corresponding to the data synchronization task, and the steps for determining the quality management results of the source database are optimized as follows:

a3. Determine the quality data to be managed in the source database.

Specifically, the processor can query the quality data to be managed that has not undergone quality management from the source database.

b3. Generate fingerprint information of each data to be managed corresponding to each quality data to be managed.

Specifically, the processor can sequentially generate data fingerprints for each quality data to be managed according to a set method, and obtain fingerprint information of each data to be managed corresponding to each quality data to be managed.

c3. Determine whether the data fingerprint information matching the data fingerprint information to be managed is included in the data fingerprint database.

What you need to know is that during the data synchronization process, the source database data will be converted into a unified format to generate data fingerprints, and the generated data fingerprints will be written into the fingerprint database to facilitate subsequent data quality through comparison of data fingerprints. manage.

Specifically, the processor can perform a comparison in the fingerprint database to determine whether it includes data fingerprint information that matches the fingerprint information of the data to be managed, thereby determining whether the data to be managed has been synchronized to the target database or whether the synchronized data fingerprint is correct. .

d3. If yes, data consistency will be regarded as the quality management result of the source database.

Specifically, if there is data fingerprint information in the fingerprint database that matches the fingerprint information of the data to be managed, that is, the data corresponding to the fingerprint information of the data to be managed has been synchronized to the target database, then the data consistency will be regarded as the quality management result of the source database. .

e3. If not, use the inconsistent fingerprint information of the data to be managed as the quality management result.

Specifically, if there is no data fingerprint information matching the fingerprint information of the data to be managed in the fingerprint database, that is, the data corresponding to the fingerprint information of the data to be managed is not synchronized to the target database or an error occurs during synchronization, then the inconsistent data to be managed will be Data fingerprint information is used as the quality management result of the source database.

In the third optional embodiment of the first embodiment, with such a setting, when the data synchronization task meets the quality management conditions, the data in the source database is converted and compared with the fingerprint information in the fingerprint database to determine the source The quality management results of the database, compared with the comparison methods in the existing technology, reduce the time and resource occupation of data scanning through the comparison of fingerprint information, quickly achieve data comparison and data repair capabilities, and satisfy users' requirements for high-quality data. Require.

Illustratively, in order to facilitate understanding of the present invention, a specific example is used to illustrate. FIG. 2 is an example flow chart of a data synchronization method provided according to Embodiment 1 of the present invention. As shown in Figure 2, steps may include:

S201. Obtain the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template;

S202. Perform task pre-check according to the data synchronization task and data synchronization rules, and determine whether the pre-check passes; if yes, jump to step S203; if not, jump to step S201;

S203. Add the data synchronization task to the task queue, add task monitoring, and load data synchronization rules;

S204. Obtain the data information to be synchronized from the source database;

S205. Convert the data information to be synchronized into target data information in a preset unified format;

S206. Write the target data information into the target database according to the data synchronization rules;

S207. Whether to enable the data quality management function; if yes, jump to step S208; if not, jump to step S201;

S208. Generate data fingerprint information according to the target data information and store it in the data fingerprint database, and store the target data information;

S209. Determine whether the data quality management conditions are met; if so, jump to step S210; if not, end;

S210. Read the data information to be managed from the source database, and generate fingerprint information of the data to be managed based on the data information to be managed;

S211. Query the fingerprint database based on the fingerprint information of the data to be managed;

S212. Determine whether there is matching data fingerprint information; if so, jump to step S209; if not, jump to step S213;

S213. Record inconsistent data and trigger the data repair function;

S214. Repair the data based on the data synchronization rules corresponding to the inconsistent data.

Embodiment 2

Figure 3 is a schematic structural diagram of a data synchronization device provided in Embodiment 2 of the present invention. As shown in FIG. 3 , the device includes: a rule acquisition module 31 , a first determination module 32 , a second determination module 33 and an information synchronization module 34 . in,

The rule acquisition module 31 is used to acquire data synchronization tasks and corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template;

The first determination module 32 is configured to obtain the data information to be synchronized from the source database corresponding to the data synchronization task when the data synchronization task and the data synchronization rule satisfy the synchronization pre-check condition;

The second determination module 33 is used to determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized;

The information synchronization module 34 is configured to synchronize the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules.

The technical solution of the embodiment of the present invention is to obtain the data synchronization task and the corresponding data synchronization rule, wherein the data synchronization rule is generated according to the selected preset rule template; when the data synchronization task and the data synchronization rule satisfy the synchronization pre-check condition, Obtain the data information to be synchronized from the source database corresponding to the data synchronization task; determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized; synchronize the target data information to the data synchronization according to the data synchronization rules The target database corresponding to the task. Convert the data information to be synchronized through the preset unified format, store the unified format data, and synchronize the target data information to the target database according to the data synchronization rules. It achieves the versatility and efficiency of data synchronization between different types of databases, eliminating the need for manual configuration of synchronization and reducing the technical requirements for data synchronization configuration personnel.

Further, the information synchronization module 34 is specifically used to:

Convert the target data information into a structured query language SQL statement according to the data synchronization rules;

Synchronize the SQL statement to the target database corresponding to the data synchronization task.

Optionally, the device further includes: a third determining module.

The third determination module is configured to write the target data information into the target database corresponding to the data synchronization task according to the data synchronization rule, and when the data synchronization task satisfies the data management condition, determine the target data information according to the data synchronization rule. The target data information is used to generate data fingerprint information and stored in the data fingerprint database.

Optionally, the device further includes: a fourth determining module.

The fourth determination module is used to obtain a database management request relative to the source database, and determine the managed target information according to the database management request.

Further, the fourth determination module is specifically used for:

Determine the data information to be managed that matches the database management request in the stored data, where the data information to be managed is data converted according to a preset unified format;

The data information to be managed is managed through a preset data service or a preset data application interface to obtain target management data.

Optionally, the device further includes: a result determination module.

The result determination module is configured to perform quality management on the source database corresponding to the data synchronization task and determine the quality management result of the source database when the data synchronization task meets quality management conditions.

Further, the result determination module is specifically used to:

Determine the quality data to be managed in the source database;

Generate fingerprint information of each data to be managed corresponding to each of the quality data to be managed;

Determine whether the data fingerprint information matching each of the data fingerprint information to be managed is included in the data fingerprint database;

If so, then data consistency is regarded as the quality management result of the source database;

If not, the inconsistent fingerprint information of the data to be managed is used as the quality management result.

The data synchronization device provided by the embodiment of the present invention can execute the data synchronization method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

Embodiment 3

FIG. 4 shows a schematic structural diagram of an electronic device 40 that can be used to implement embodiments of the present invention. Electronic devices are intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (eg, helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are examples only and are not intended to limit the implementation of the invention described and/or claimed herein.

As shown in Figure 4, the electronic device 40 includes at least one processor 41, and a memory communicatively connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., where the memory stores There is a computer program that can be executed by at least one processor. The processor 41 can perform the operation according to the computer program stored in the read-only memory (ROM) 42 or loaded from the storage unit 48 into the random access memory (RAM) 43. Perform various appropriate actions and processing. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, ROM 42 and RAM 43 are connected to each other through a bus 44. An input/output (I/O) interface 45 is also connected to bus 44 .

Multiple components in the electronic device 40 are connected to the I/O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of displays, speakers, etc.; storage unit 48, such as magnetic disk, optical disk etc.; and communication unit 49, such as network card, modem, wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information/data with other devices through computer networks such as the Internet and/or various telecommunications networks.

Processor 41 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processing processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs various methods and processes described above, such as the data synchronization method.

In some embodiments, the data synchronization method may be implemented as a computer program, which is tangibly embodied in a computer-readable storage medium, such as the storage unit 48. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 40 via the ROM 42 and/or the communication unit 49 . When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the data synchronization method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the data synchronization method in any other suitable manner (eg, by means of firmware).

Various implementations of the systems and techniques described above may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip implemented in a system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or a combination thereof. These various embodiments may include implementation in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor The processor, which may be a special purpose or general purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.

Computer programs for implementing the methods of the invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that the computer program, when executed by the processor, causes the functions/operations specified in the flowcharts and/or block diagrams to be implemented. A computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display)) for displaying information to the user monitor); and a keyboard and pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and may be provided in any form, including Acoustic input, voice input or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., A user's computer having a graphical user interface or web browser through which the user can interact with implementations of the systems and technologies described herein), or including such backend components, middleware components, or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communications network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

Computing systems may include clients and servers. Clients and servers are generally remote from each other and typically interact over a communications network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other. The server can be a cloud server, also known as cloud computing server or cloud host. It is a host product in the cloud computing service system to solve the problems of difficult management and weak business scalability in traditional physical hosts and VPS services. defect.

It should be understood that various forms of the process shown above may be used, with steps reordered, added or deleted. For example, each step described in the present invention can be executed in parallel, sequentially, or in different orders. As long as the desired results of the technical solution of the present invention can be achieved, there is no limitation here.

The above-mentioned specific embodiments do not constitute a limitation on the scope of the present invention. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions are possible depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. A data synchronization method, characterized by including: Obtain the data synchronization task and the corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template; When the data synchronization task and the data synchronization rules meet the synchronization pre-check condition, obtain the data information to be synchronized from the source database corresponding to the data synchronization task; According to the preset unified format and the data information to be synchronized, determine the converted target data information and store the data; According to the data synchronization rules, the target data information is synchronized to the target database corresponding to the data synchronization task. 2. The method according to claim 1, characterized in that, according to the data synchronization rules, synchronizing the target data information to the target database corresponding to the data synchronization task includes: Convert the target data information into a structured query language SQL statement according to the data synchronization rules; Synchronize the SQL statement to the target database corresponding to the data synchronization task. 3. The method according to claim 1, characterized in that, after writing the target data information into the target database corresponding to the data synchronization task according to the data synchronization rules, it further includes: When the data synchronization task meets the data management conditions, data fingerprint information is generated according to the target data information and stored in the data fingerprint database. 4. The method of claim 1, further comprising: Obtain a database management request relative to the source database, and determine the managed target information based on the database management request. 5. The method according to claim 4, characterized in that determining target management data according to the database management request includes: Determine the data information to be managed that matches the database management request in the stored data, where the data information to be managed is data converted according to a preset unified format; The data information to be managed is managed through a preset data service or a preset data application interface to obtain target management data. 6. The method of claim 1, further comprising: When the data synchronization task meets quality management conditions, quality management is performed on the source database corresponding to the data synchronization task, and the quality management result of the source database is determined. 7. The method according to claim 6, wherein the step of performing quality management on the source database corresponding to the data synchronization task and determining the quality management results of the source database includes: Determine the quality data to be managed in the source database; Generate fingerprint information of each data to be managed corresponding to each of the quality data to be managed; Determine whether the data fingerprint information matching each of the data fingerprint information to be managed is included in the data fingerprint database; If so, then data consistency is regarded as the quality management result of the source database; If not, the inconsistent fingerprint information of the data to be managed is used as the quality management result. 8. A data synchronization device, characterized in that it includes: A rule acquisition module, used to obtain data synchronization tasks and corresponding data synchronization rules, wherein the data synchronization rules are generated according to the selected preset rule template; A first determination module, configured to obtain the data information to be synchronized from the source database corresponding to the data synchronization task when the data synchronization task and the data synchronization rule satisfy the synchronization pre-check condition; The second determination module is used to determine the converted target data information and store the data according to the preset unified format and the data information to be synchronized; An information synchronization module, configured to synchronize the target data information to the target database corresponding to the data synchronization task according to the data synchronization rules. 9. An electronic device, characterized in that the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor, so that the at least one processor can execute any one of claims 1-7 The data synchronization method described. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the method of any one of claims 1-7 when executed by a processor. Data synchronization method.