CN112788151B

CN112788151B - Method, device and system for data synchronization

Info

Publication number: CN112788151B
Application number: CN202110111853.2A
Authority: CN
Inventors: 王吉元; 余俊; 徐明明
Original assignee: Advanced New Technologies Co Ltd
Current assignee: Advanced New Technologies Co Ltd
Priority date: 2017-09-08
Filing date: 2017-09-08
Publication date: 2024-02-06
Anticipated expiration: 2037-09-08
Also published as: CN112788151A; CN107566499B; CN107566499A

Abstract

The embodiment of the application discloses a method, a device and a system for data synchronization, wherein the method comprises the following steps: acquiring a data synchronization task, wherein the data synchronization task is used for indicating to synchronize data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain; acquiring the data to be synchronized; and writing the data to be synchronized into the second data source according to the indication of the data synchronization task.

Description

Method, device and system for data synchronization

This document is a divisional application of patent application with application number "201710807145.6", application date "2017, 09, 08, and application name" method, apparatus, and system for data synchronization ".

Technical Field

The present application relates to the field of data processing, and more particularly, to a method, apparatus, and system for data synchronization.

Background

Because of the demands of system backup, upgrading, data exchange or data sharing, etc., data synchronization needs to be performed between data sources of different computer network domains, but the current technical scheme can only achieve data synchronization between different data sources of the same computer network domain, cannot achieve data synchronization between data sources of different computer network domains, and cannot meet the data synchronization demands of users.

Therefore, a method for data synchronization is needed to overcome the above technical problems.

Disclosure of Invention

The purpose of the application is to provide a method, a device and a system for data synchronization, which can acquire data in a data source of one computer network domain, and write the acquired data into the data source of another computer network domain, so that the data synchronization among the data sources of different computer network domains is realized, and the data synchronization requirement of a user is met.

In order to solve the technical problems, the embodiment of the application is realized as follows:

in a first aspect, a method for data synchronization is provided, including:

acquiring a data synchronization task, wherein the data synchronization task is used for indicating to synchronize data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain;

acquiring the data to be synchronized;

and writing the data to be synchronized into the second data source according to the indication of the data synchronization task.

In a second aspect, a method for data synchronization is provided, including:

Directly reading the data to be synchronized;

and according to the indication of the data synchronization task, when a data pulling request sent by the proxy computer in the second computer network domain is received, sending the data to be synchronized to the proxy computer in the second computer network domain, wherein the data pulling request is used for requesting to write the data to be synchronized into the second data source, and the proxy computer in the second computer network domain is used for writing the data to be synchronized into the second data source.

In a third aspect, a method for data synchronization is provided, including:

receiving the data to be synchronized pushed by a proxy computer in the first computer network domain;

In a fourth aspect, a method for data synchronization is provided, including:

and when receiving a data pulling request sent by the proxy computer in the second computer network domain, sending the data to be synchronized to the proxy computer in the second computer network domain, wherein the data pulling request is used for requesting the data to be synchronized to be written into the second data source, and the proxy computer in the second computer network domain is used for writing the data to be synchronized into the second data source.

In a fifth aspect, there is provided a data synchronization apparatus, comprising:

the task acquisition unit is used for acquiring a data synchronization task, wherein the data synchronization task is used for indicating to synchronize data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain;

the data acquisition unit is used for directly reading the data to be synchronized;

and the data writing unit is used for sending the data to be synchronized to the proxy computers in the second computer network domain when receiving the data pulling request sent by the proxy computers in the second computer network domain according to the indication of the data synchronization task, wherein the data pulling request is used for requesting the data to be synchronized to be written into the second data source, and the proxy computers in the second computer network domain are used for writing the data to be synchronized into the second data source.

In a sixth aspect, there is provided a data synchronization apparatus, comprising:

a data acquisition unit for receiving the data to be synchronized pushed by the proxy computer in the first computer network domain;

and the data writing unit writes the data to be synchronized into the second data source according to the indication of the data synchronization task.

In a seventh aspect, there is provided a data synchronization apparatus, comprising:

In an eighth aspect, there is provided an electronic device comprising:

a processor; and

a memory arranged to store computer executable instructions that when executed perform the following operations using the processor:

directly reading the data to be synchronized;

In a ninth aspect, there is provided an electronic device, comprising:

a processor; and

In a tenth aspect, there is provided an electronic device comprising:

a processor; and

In an eleventh aspect, there is provided a computer readable medium storing one or more programs that, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:

directly reading the data to be synchronized;

In a twelfth aspect, there is provided a computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:

In a thirteenth aspect, there is provided a computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:

In a fourteenth aspect, a data synchronization system is provided, including the electronic device of the eighth aspect, the electronic device of the ninth aspect, and the electronic device of the tenth aspect.

According to the technical scheme provided by the embodiment of the application, after the data to be synchronized in the first data source in the first computer network domain is obtained, the data to be synchronized is written into the second data source in the second computer network domain according to the data synchronization task instruction, so that the data synchronization among the data sources in different computer network domains is realized, and the data synchronization requirement of a user is met.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.

FIG. 1 is a flow chart of a method of data synchronization according to one embodiment of the present application.

Fig. 2 is a flow chart of a method of data synchronization according to another embodiment of the present application.

Fig. 3 is a flow chart of a method of data synchronization according to yet another embodiment of the present application.

Fig. 4 is a flow chart of a method of data synchronization according to yet another embodiment of the present application.

Fig. 5 is a schematic diagram of a system framework according to one embodiment of the present application.

Fig. 6 is a schematic diagram of a system framework according to another embodiment of the present application.

Fig. 7 is a schematic diagram of a system framework according to yet another embodiment of the present application.

Fig. 8 is a schematic structural view of an electronic device according to one embodiment of the present application.

Fig. 9 is a schematic structural diagram of a data synchronization device according to an embodiment of the present application.

Fig. 10 is a schematic structural view of an electronic device according to another embodiment of the present application.

Fig. 11 is a schematic structural diagram of a data synchronization device according to another embodiment of the present application.

Fig. 12 is a schematic structural view of an electronic device according to still another embodiment of the present application.

Fig. 13 is a schematic structural view of a data synchronization device according to still another embodiment of the present application.

Detailed Description

In order to better understand the technical solutions in the present application, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.

For ease of understanding the present application, several important terms that will be referred to in the description of the embodiments of the present application are first presented herein:

one is a computer network domain. In embodiments of the present application, computers in a computer network domain have established trust relationships, and one computer in the domain accesses other computers in the domain, eliminating the need for permission from the accessed computer. These computers constitute a collection of computers with a security boundary. In the practical application process, one computer network domain may be a set of multiple local area networks, a set of multiple local area networks and a public network part with a certain range, or a set formed by computers with a part in a local area network and a safe trust relationship. In summary, the extension of a computer network domain, the number of computers involved, the distribution of physical locations, the complexity of the architecture of the computer system involved, and the like are related to the "domain" scope that the user delineates/designates in a dimension of security.

And secondly, a data source. In general, a data source is an entity of multiple data sets, and data within the data source may have the same data form or may have different data forms. In addition, the data sources themselves may be of various types, including but not limited to: mysql, oracle, sqlServer, DB2 Hbase and Oss.

Thirdly, a DataX architecture. The architecture is a data synchronization framework that enables efficient data synchronization between multiple data sources in the same computer network domain. Thus, in theory, the DataX framework may support data synchronization of any data source type in the same computer network domain.

Having introduced the above terminology, various embodiments of the present application are described below in connection with the accompanying drawings. Referring to FIG. 1, a method 100 of data synchronization of one embodiment of the present application is illustrated. The method may be performed by a data synchronization device. As shown in fig. 1, at S102, a data synchronization task is acquired that is used to instruct synchronizing data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain.

Optionally, in some embodiments, the acquiring data synchronization task is specifically a data synchronization task distributed by the receiving controller. In this case, the controller acquires a data synchronization task created by the user through the application interactive interface, determines a computer executing the data synchronization task according to the load condition of a computer capable of executing the data synchronization task in the network, and distributes the data synchronization task to the determined computer.

At S104, data to be synchronized is acquired.

At S106, the data to be synchronized is written into the second data source according to the indication of the synchronization task.

It should be noted that, although the above embodiment is described as the step S104 is disposed after the step S102 for convenience of description, the execution sequence of the two steps may be exchanged according to the actual situation, that is, the data to be synchronized is acquired first, and then the data synchronization task is received, so that the data to be synchronized that is currently required to be synchronized is determined according to the requirement of the data synchronization task, and then the synchronization task for the data to be synchronized is completed according to the indication of the data synchronization task.

Alternatively, as an embodiment, at S104, the data to be synchronized is encrypted data. Correspondingly, at S106, the data obtained after decrypting the data to be synchronized is written into the second data source. Thereby, the security of the data can be improved.

Optionally, as an embodiment, at S104, the compressed data is to be synchronized. Correspondingly, at S106, the data obtained after decompressing the data to be synchronized is written into the second data source. Thereby, the security of the data can be improved.

Methods of data synchronization according to embodiments of the present application are further described below in connection with more specific embodiments.

Embodiment one:

the data synchronization device performing the method 100 may be a proxy computer in the first computer network domain, as shown in fig. 2, and at S104, the acquiring data to be synchronized may specifically be: and directly reading the data to be synchronized. Correspondingly, at S106, according to the instruction of the data synchronization task, writing the data to be synchronized into the second data source may specifically be: according to the indication of the data synchronization task, when a data pulling request sent by an agent computer in a second computer network domain is received, sending data to be synchronized to the agent computer in the second computer network domain, wherein the data pulling request is used for requesting to write the data to be synchronized into the second data source, and the agent computer in the second computer network domain is used for writing the data to be synchronized into the second data source, so that the data to be synchronized in the second computer network domain is written into the second data source after the data to be synchronized is received. Therefore, the method and the device can read the data to be synchronized in the data source of the computer network domain and write the data to be synchronized into the data source of another computer network domain in a cross-domain manner.

Optionally, as an example, the user configures the proxy computer in the second computer network to send a data pull request to the proxy computer in the first computer network upon determining that the created data synchronization task is assigned to the proxy computer in the first computer network. Or the agent computers in the second computer network periodically or aperiodically send data pulling requests to the agent computers in the first computer network, and send data to be synchronized to the agent computers in the second computer network after the agent computers in the first computer network acquire the synchronization tasks.

Optionally, as an example, when receiving a data pulling request sent by a proxy computer in the second computer network domain, encrypting and/or compressing the data to be synchronized, and sending the encrypted and/or compressed data to be synchronized to the proxy computer in the second computer network domain. Correspondingly, after receiving the encrypted and/or compressed data to be synchronized, the proxy computer in the second computer network firstly decrypts and/or decompresses the encrypted and/or compressed data to be synchronized, and then writes the decrypted and/or decompressed data to be synchronized into the second data source.

Optionally, as yet another example, the user-created data synchronization task is also used to indicate a type of the second data source. In this case, if it is determined that the first data source and the second data source have the same type, for example, the type of the first data source is Mysql, and the type of the second data source is Mysql, after receiving a data pull request sent by the agent computer in the second computer network domain, the data to be synchronized is directly sent to the agent computer in the second computer network domain. If it is determined that the first data source and the second data source are different in type, for example, the first data source is Oracle in type and the second data source is Mysql in type, after converting the data to be synchronized into data conforming to the type of the second data source, after receiving a data pull request sent by an agent computer in the second computer network domain, the converted data is sent to the agent computer in the second computer network domain.

Alternatively, as yet another example, a user may configure the first data source and the second data source by creating configuration information through an application interactive interface. The configuration information comprises information related to a first data source and information related to a second data source, wherein the information related to the first data source comprises at least one of the following information: the type of the first data source, the data table in the first data source, and the field information in the first data source, the information related to the second data source including at least one of: the type of the second data source, a data table in the second data source, and field information in the second data source. Correspondingly, whether the first data source and the second data source are of the same type can be judged according to the configuration information.

Alternatively, when converting the data to be synchronized into data conforming to the type of the second data source, the data may be performed according to a pre-configured data conversion rule, where the data conversion rule is used to specify an operation to be performed to generate data meeting the type of the second data source according to the data to be synchronized, for example, multiplying the data to be synchronized by a certain pre-set constant C, so as to obtain data meeting the type of the second data source.

For example, taking the first data source as Oracle and the second data source as Mysql as an example, when converting the data to be synchronized into data satisfying the type of the second data source, converting binary large object (Binary Large Object, blob) type data in the data to be synchronized into longlob type data of Mysql, converting binary_float type data in the data to be synchronized into default type data of Mysql, converting binary_double type data in the data to be synchronized into double type data of Mysql, converting varchar2 type data in the data to be synchronized into varchar type data of Mysql, and converting nvachur 2 type data in the data to be synchronized into variable varchar type data of Mysql.

Embodiment two:

the data synchronization device performing the method 100 may be a proxy computer in the second computer network domain, as shown in fig. 3, and at S104, the acquiring data to be synchronized may specifically be: data to be synchronized pushed by a proxy computer in a first computer network domain is received. Correspondingly, at S106, according to the instruction of the data synchronization task, writing the data to be synchronized into the second data source may specifically be: and writing the received data to be synchronized sent by the proxy computer in the first computer network domain into the second data source according to the instruction of the data synchronization task. Thus, the data to be synchronized can be read from the data source of one computer network domain in a cross-domain mode, and the data to be synchronized is written into the data source of the computer network domain.

Alternatively, as one example, a user, upon determining that a created data synchronization task is assigned to a proxy computer in a second computer network domain, configures the proxy computer in the first computer network domain to push data to be synchronized to the proxy computer in the second computer network domain. Or the agent computer in the first computer network periodically or aperiodically pushes the data to be synchronized to the agent computer in the second computer network, and when the agent computer in the second computer network acquires the data synchronization task, the data to be synchronized is written into the second data source according to the indication of the data synchronization task.

Optionally, as an example, at S104, the received data to be synchronized is encrypted and/or compressed, at S106, the received data to be synchronized is decrypted and/or decompressed to obtain decrypted and/or decompressed data to be synchronized, and then the decrypted and/or decompressed data to be synchronized is written into the second data source.

Optionally, as yet another example, the user-created data synchronization task is also used to indicate a type of the first data source. In this case, if it is determined that the first data source and the second data source are of the same type, the data to be synchronized is directly written into the second data source after the data to be synchronized is received. If the types of the first data source and the second data source are different, converting the data to be synchronized into data conforming to the type of the second data source, and writing the converted data into the second data source.

Alternatively, as yet another example, the user may configure the proxy computer in the first computer network domain to convert the data to be synchronized into data conforming to the type of the second data source when the types of the first data source and the second data source are different, and then send the converted data to the proxy computer in the second data source.

Embodiment III:

the data synchronization device performing the method 100 may be a gateway device, as shown in fig. 4, and at S104, the obtaining data to be synchronized may specifically be: data to be synchronized pushed by a proxy computer in a first computer network domain is received. Correspondingly, at S106, according to the instruction of the data synchronization task, writing the data to be synchronized into the second data source may specifically be: according to the indication of the data synchronization task, when a data pulling request sent by an agent computer in a second computer network domain is received, sending data to be synchronized to the agent computer in the second computer network domain, wherein the data pulling request is used for requesting to write the data to be synchronized into the second data source, and the agent computer in the second computer network domain is used for writing the data to be synchronized into the second data source. Thereby causing the proxy computer in the second computer network domain to write the data to be synchronized into the second data source after receiving the data to be synchronized. Therefore, the method and the device can realize cross-domain reading of the data to be synchronized in the data source of one computer network domain and cross-domain writing of the data to be synchronized in the data source of the other computer network domain.

Alternatively, as an example, upon determining that the created data synchronization task is assigned to the gateway device, the user configures the proxy computer in the first computer network domain to push data to be synchronized to the gateway device and configures the proxy computer in the second computer network domain to send a data pull request to the gateway device. Or the proxy computer in the first computer network periodically or aperiodically pushes the data to be synchronized to the gateway device, and the proxy computer in the second computer network periodically or aperiodically sends the data pulling request to the gateway device.

Alternatively, as an example, the period in which the proxy computer in the first computer network domain pushes data to be synchronized to the gateway device is the same as the period in which the proxy computer in the second computer network domain sends a data pull request to the gateway device.

Alternatively, as another example, the data to be synchronized received at S106 is encrypted and/or compressed data. That is, the proxy computer in the first computer network domain encrypts and/or compresses the data to be synchronized, and then pushes the encrypted and/or compressed data to be synchronized to the gateway device.

Optionally, as yet another example, the user configures the type of the second data source for the proxy computer in the first computer network domain when triggering the proxy computer in the first computer network domain to push data to be synchronized to the gateway device. In this case, if the proxy computer in the first computer network domain determines that the types of the first data source and the second data source are different, the proxy computer in the first computer network domain converts the data to be synchronized into data conforming to the type of the second data source, and then pushes the converted data to the gateway device.

Optionally, as yet another example, the user-created data synchronization task is also used to indicate the type of the first data source and the second data source. In this case, if the gateway device determines that the first data source and the second data source are of the same type, the gateway device directly writes the data to be synchronized into the second data source after receiving the data pull request sent by the proxy computer in the second computer network domain. If the gateway device determines that the types of the first data source and the second data source are different, the gateway device converts the data to be synchronized into data conforming to the type of the second data source, and when receiving a data pulling request sent by the proxy computer in the second computer network domain, the gateway device sends the converted data to the proxy computer in the second computer network domain.

Optionally, in some embodiments, the user may configure the first data source and the second data source by creating configuration information through an application interaction interface. The configuration information comprises information related to a first data source and information related to a second data source, wherein the information related to the first data source comprises at least one of the following information: the type of the first data source, the data table in the first data source, and the field information in the first data source, the information related to the second data source including at least one of: the type of the second data source, a data table in the second data source, and field information in the second data source. Correspondingly, the proxy computer and/or the gateway device in the first computer network domain may determine whether the first data source and the second data source are of the same type based on the configuration information.

A method of data synchronization according to embodiments of the present application has been described. To facilitate an understanding of the concepts of the present application, a system framework 500 for a method for data synchronization according to one specific embodiment of the present application will be described below in conjunction with fig. 5, the system framework 500 being based on a DataX framework. The system framework 500 is capable of implementing the method 100 shown in fig. 2.

As shown in fig. 5, the system frame 500 includes: JSS interface 510, controller 520, first proxy computer 530, first data source 540, second proxy computer 550, second data source 560, first proxy computer 530 belonging to a first computer network domain, second proxy computer 550 belonging to a second computer network domain.

The JSS interface 510 is a Java interface for network security services, and provides a configuration portal for data synchronization across computer network domains for users. A user may configure configuration information related to the first data source 540 and the second data source 560 through JSS interface 510 and may also create a data synchronization task through JSS interface 510.

The controller 520 is used to distribute the data synchronization tasks acquired from the JSS interface 510 to the proxy computer 530. Optionally, there are multiple agent computers in the first computer network, and the controller 520 determines the agent computer performing the data synchronization task according to the load condition of the agent computer. It should be noted that the controller 520 described herein may correspond to a single controller or a cluster of controllers. In this application, a data synchronization task may be understood in abstraction as a Java process.

The first proxy computer 530 includes a read (Reader) module 531 and a network write (NetworkWriter) module 532. The Reader module 531 is configured to read data to be synchronized in the first data source 540, and transmit the data to be synchronized to the network writer module 532, where the network writer module 532 sends the data to be synchronized to the second proxy computer 550 when receiving a data pull request sent by the second proxy computer 550. Optionally, the NetworkWriter module 532 may send the data to be synchronized after compression and/or encryption to the second agent computer 550.

The second agent computer 550 includes a cross-domain read (reglnnetworkreader) module 551 and a write (Writer) module 552. The regin network reader module 551 is configured to send a data pulling request to the first proxy computer 530, receive data to be synchronized sent by the first proxy computer through the network Writer module 532, transmit the received data to be synchronized to the Writer module 552, and the Writer module 552 writes the received data to be synchronized into the second data source 560. Optionally, the data to be synchronized received by the ReginNetwork Reader module 551 is encrypted and/or compressed data, the ReginNetwork Reader module 551 performs decryption and/or decompression processing on the received data to be synchronized, and transmits the decrypted and/or decompressed data to the Writer module 552.

Note that, the NetworkWriter module 532 and the ReginNetwork Reader module 551 in the system frame 500 may be integrated into an existing DataX frame in the form of a DataX plug-in, and the data transmission between the modules uses Netty as an implementation method, but the implementation method of the data transmission between the modules is not limited to Netty.

A system framework 600 for a method of data synchronization according to one embodiment of the present application will be described below in conjunction with fig. 6, the system framework 600 being based on a DataX framework. The system framework 600 is capable of implementing the method 100 shown in fig. 3.

As shown in fig. 6, the system frame 600 includes: the JSS interface 610, the first proxy computer 620, the first data source 630, the controller 640, the second proxy computer 650, the second data source 660, the first proxy computer 620 belonging to a first computer network domain, the second proxy computer 650 belonging to a second computer network domain.

The JSS interface 610 and the JSS interface 510 have the same function, and the controller 640 and the controller 520 have similar functions, and are not described herein.

The first proxy computer 620 includes a read (Reader) module 621 and a cross-domain write (reglnnetworkwriter) module 622. The Reader module 621 is configured to read data to be synchronized in the first data source 630, and transmit the data to be synchronized to the regin network writer module 622, where the regin network writer module 622 pushes the synchronized data to the second agent computer 650. Optionally, the reglnnetworkwriter module 622 pushes the data to be synchronized to the second proxy computer 650 after compression and/or encryption.

The second agent computer 650 includes a network read (NetworkReader) module 651 and a write (Writer) module 652. The NetworkReader module 651 receives the data to be synchronized pushed by the first proxy computer 620, transmits the received data to be synchronized to the Writer module 652, and the Writer module 652 writes the received data to be synchronized into the second data source 660. Optionally, the data to be synchronized received by the NetworkReader module 651 module is encrypted and/or compressed data, the NetworkReader module 651 performs decryption and/or decompression processing on the received data to be synchronized, and transmits the data to be synchronized obtained by the decryption and/or decompression processing to the Writer module 652.

It should be noted that, the region Network writer module 622 and the Network Reader module 651 in the system framework 600 may be integrated into an existing DataX framework in the form of a DataX plug-in, and the data transmission between the modules uses Netty as an implementation method, but the implementation method of the data transmission between the modules is not limited to Netty.

A system framework 700 for a method of data synchronization according to one embodiment of the present application will be described below in conjunction with fig. 7, the system framework 700 being based on a DataX framework. The system framework 700 is capable of implementing the method 100 shown in fig. 4.

As shown in fig. 7, the system frame 700 includes: JSS interface 710, gateway device 720, controller 730, first proxy computer 740, first data source 750, second proxy computer 760, second data source 770, first proxy computer 740 belonging to a first computer network domain, second proxy computer 760 belonging to a second computer network domain.

The JSS interface 710 has the same function as the JSS interface 610, and will not be described here again.

The gateway device 720 is a computer that performs a data synchronization task, and the gateway device 720 includes a network read (NetworkReader) module 721 and a network write (NetworkWriter) module 722, where the NetworkReader module 721 receives data to be synchronized pushed by the first proxy computer 740, and transmits the data to be synchronized to the NetworkWriter module 722, and the NetworkWriter module 722 is configured to receive a data pull request sent by the second proxy computer 760, and send the data to be synchronized to the second proxy computer 760. Optionally, a data processing module may be disposed between the NetworkReader module 721 and the NetworkWriter module 722, where the data processing module is configured to convert the data to be synchronized according to a data conversion rule. For example, the data processing module may convert data to be synchronized into data that conforms to the type of the second data source 770 when the first data source 750 and the second data source 770 are of different types. It should be noted that the gateway device described herein may correspond to a stand-alone gateway device, and may correspond to a cluster of gateway devices.

The controller 730 is used to distribute the data synchronization tasks to the gateway device 720. Optionally, there are multiple gateway devices in the network, and the controller 730 determines the gateway device performing the data synchronization task according to the load situation of the gateway device. It should be noted that the controllers described herein may correspond to a single controller or a cluster of controllers. In this application, a data synchronization task may be understood in abstraction as a Java process.

The first agent computer 740 includes a read (Reader) module 741, a cross-domain write (ReginNetwork Writer) module 742, a metadata query (metaquery) module 743, and a web daemon (NetworkDaemon) module 744. The Reader module 741 is configured to read the data to be synchronized in the first data source 750, and transmit the data to be synchronized to the regin network writer module 742, where the regin network writer module 742 pushes the data to be synchronized to the network Reader module 721 in the gateway device 720. Optionally, the reglnnetworkwriter module 742 encrypts the data to be synchronized before pushing to the NetworkReader module 721. The metaquery module 743 is configured to obtain metadata associated with the first data source 750 and provide the metadata to the JSS interface so that a user queries and knows about information associated with the first data source 750, and the NetworkDaemon module 744 is configured to encapsulate differences in underlying network interfaces (e.g., interfaces of the cross-domain write module 742 and the metaquery module 743) to provide the same call interface to the system framework.

The second agent computer 760 includes a cross-domain read (reglnnetworkreader) module 761, a write (Writer) module 762, and a NetworkDaemon module 763. The regin network reader module 761 is configured to send a data pulling request to a network Writer module 722 in the gateway device 720, receive data to be synchronized sent by the network Writer module 722, transmit the received data to be synchronized to the Writer module 762, and the Writer module 762 writes the received data to the second data source 770. Optionally, the data to be synchronized received by the regin network reader module 761 is encrypted and/or compressed data, the regin network reader module 761 decrypts and/or decompresses the received data to be synchronized, and transmits the decrypted and/or decompressed data to the Writer module 762. The function of the NetworkDaemon module 763 is similar to that of the NetworkDaemon module 744 and will not be described in detail herein.

It should be noted that the reglnlnterworking writer module 742, the NetworkReader module 721, the NetworkWriter module 722, and the reglnterworking reader module 761 in the system framework 700 may be integrated into an existing DataX framework in the form of a DataX plug-in.

Thus, the system framework of the embodiments of the present application overcomes the data isolation limitations of different computer network domains and provides the ability to synchronize data in different computer network domains. The system framework of the embodiment of the application can also realize conversion between data of different types of data sources, so that data exchange and synchronization of the different types of data sources in different computer network domains are realized.

It should be noted that, in the embodiments of the present application, the proxy computer in the first computer network domain and the proxy computer in the second computer network domain may be regarded as gateway devices in the computer network domains.

An electronic device according to an embodiment of the present application will be described in detail below in conjunction with fig. 8. Referring to fig. 8, at the hardware level, the electronic device includes a processor, optionally including an internal bus, a network interface, a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, network interface, and memory may be interconnected by an internal bus, which may be an industry standard architecture (Industry Standard Architecture, ISA) bus, a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 8, but not only one bus or type of bus.

And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.

The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, and forms a data synchronization device on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:

directly reading the data to be synchronized;

The method performed by the data synchronization device disclosed in the embodiment shown in fig. 2 of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

The electronic device may also execute the method of fig. 2 and implement the functions of the data synchronization device in the embodiment shown in fig. 2, which is not described herein.

Of course, other implementations, such as a logic device or a combination of hardware and software, are not excluded from the electronic device of the present application, that is, the execution subject of the following processing flow is not limited to each logic unit, but may be hardware or a logic device.

The embodiments also provide a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform the method of the embodiment of fig. 2, and in particular to perform the method of:

directly reading the data to be synchronized;

Fig. 9 is a schematic structural diagram of a data synchronization device according to an embodiment of the present application. Referring to fig. 9, in a software implementation, the data synchronization device 900 may include: a task acquisition unit 901, a data acquisition unit 902, a data writing unit 903; wherein,

a task acquiring unit 901, configured to acquire a data synchronization task, where the data synchronization task is used to instruct synchronization of data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain;

a data acquisition unit 902, configured to directly read the data to be synchronized;

the data writing unit 903 is configured to send, when receiving a data pulling request sent by an agent computer in the second computer network domain according to the instruction of the data synchronization task, the data to be synchronized to the agent computer in the second computer network domain, where the data pulling request is used to request writing of the data to be synchronized into the second data source, and the agent computer in the second computer network domain is used to write the data to be synchronized into the second data source.

After the data synchronization device reads the data to be synchronized in the data source of the computer network domain, the data synchronization device pushes the data to be synchronized to the proxy computer or the gateway device in the other computer network domain, or sends the data to be synchronized to the proxy computer after receiving the data pulling request sent by the proxy computer in the other computer network domain, so that the data to be synchronized is synchronized from the data source of one computer network domain to the data source of the other computer network domain.

The data synchronization device 900 may also perform the method of the embodiment shown in fig. 2, which is not described herein.

Fig. 10 is a schematic structural view of an electronic device according to another embodiment of the present application. Referring to fig. 10, at the hardware level, the electronic device includes a processor, optionally including an internal bus, a network interface, a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA bus, PCI bus, EISA bus, or the like. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 10, but not only one bus or type of bus.

The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, and the data synchronization device is arranged on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:

The method performed by the data synchronization device disclosed in the embodiment shown in fig. 3 of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a CPU, NP, etc.; DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

The electronic device may also execute the method of fig. 3 and implement the functions of the data synchronization device in the embodiment shown in fig. 3, which is not described herein.

Of course, in addition to the software implementation, the server of the present application does not exclude other implementations, such as a logic device or a combination of software and hardware, that is, the execution subject of the following process flows is not limited to each logic unit, but may also be hardware or a logic device.

The embodiments also provide a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform the method of the embodiment of fig. 3, and in particular to perform the method of:

Fig. 11 is a schematic structural diagram of a data synchronization device according to an embodiment of the present application. Referring to fig. 11, in a software implementation, the data synchronization device 1100 may include: a task acquisition unit 1101, a data acquisition unit 1102, and a data writing unit 1103, wherein,

a task acquisition unit 1101 that acquires a data synchronization task for instructing to synchronize data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain;

a data obtaining unit 1102, configured to receive the data to be synchronized pushed by the proxy computer in the first computer network domain;

and a data writing unit 1103 for writing the data to be synchronized into the second data source according to the instruction of the data synchronization task.

According to the data synchronization device, data to be synchronized in one data source pushed by a proxy computer in one computer network domain or data to be synchronized in the data source sent by a gateway device are received, and the received data to be synchronized is written into the other data source according to the received data synchronization task instruction, so that the data to be synchronized is synchronized from the data source in one computer network domain to the data source in the other computer network domain.

Optionally, as an embodiment, the data to be synchronized acquired by the data acquisition unit 1102 is encrypted, and the data acquisition unit 1102 decrypts the data to be synchronized to obtain decrypted data; the data writing unit 1103 writes the decrypted data into the second data source.

Optionally, as an embodiment, the data to be synchronized acquired by the data acquisition unit 1102 is compressed, and the data acquisition unit 1102 decompresses the data to be synchronized to obtain decompressed data; the data writing unit 1103 writes the decompressed data into the second data source.

The data synchronization device 1100 may also perform the method of the embodiment shown in fig. 3, which is not described herein.

Fig. 12 is a schematic structural view of an electronic device according to another embodiment of the present application. Referring to fig. 12, at the hardware level, the electronic device includes a processor, optionally including an internal bus, a network interface, a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA bus, PCI bus, EISA bus, or the like. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 12, but not only one bus or type of bus.

The method performed by the data synchronization device disclosed in the embodiment shown in fig. 4 of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a CPU, NP, etc.; DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

The electronic device may also execute the method of fig. 4 and implement the functions of the data synchronization device in the embodiment shown in fig. 4, which is not described herein.

The embodiments also provide a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform the method of the embodiment of fig. 4, and in particular to perform the method of:

Fig. 13 is a schematic structural diagram of a data synchronization device according to an embodiment of the present application. Referring to fig. 13, in one software implementation, a data synchronization apparatus 1300 may include: a task acquisition unit 1301, a data acquisition unit 1302, a data transmission unit 1303, wherein,

a task acquiring unit 1301 configured to acquire a data synchronization task for indicating synchronization of data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain;

a data obtaining unit 1302, configured to receive the data to be synchronized pushed by the proxy computer in the first computer network domain;

and the data writing unit 1303 sends the data to be synchronized to the proxy computers in the second computer network domain when receiving a data pulling request sent by the proxy computers in the second computer network domain according to the instruction of the data synchronization task, where the data pulling request is used for requesting writing of the data to be synchronized into the second data source, and the proxy computers in the second computer network domain are used for writing of the data to be synchronized into the second data source.

According to the data synchronization device, data to be synchronized in one data source pushed by the agent computer in one computer network domain is received, and when a data pulling request sent by the agent computer in the other computer network domain is received, the data to be synchronized is sent to the agent computer in the other computer network domain, so that the data to be synchronized is synchronized from the data source in the one computer network domain to the data source in the other computer network domain.

Optionally, as an embodiment, the data to be synchronized is encrypted data.

Optionally, as an embodiment, the data to be synchronized is compressed data.

The data synchronization device 1300 may also perform the method of the embodiment shown in fig. 4, which is not described herein.

In summary, the foregoing description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

Claims

1. A method of data synchronization applied to a data synchronization device, the data synchronization device being a proxy computer in a first computer network domain, a proxy computer in a second computer network domain, or a gateway device, comprising:

receiving a data synchronization task distributed by a controller, wherein the data synchronization task is used for indicating to synchronize data to be synchronized of a first data source in a first computer network domain to a second data source in a second computer network domain;

acquiring the data to be synchronized;

2. The method of claim 1, further comprising:

judging whether the types of the first data source and the second data source are the same;

Wherein the writing the data to be synchronized into the second data source includes:

and writing the data to be synchronized into the second data source according to the judging result.

3. The method of claim 2, wherein writing the data to be synchronized into the second data source according to the result of the determination comprises:

if the judging result is that the types of the first data source and the second data source are the same, directly writing the data to be synchronized into the second data source;

and if the first data source and the second data source have different types as a result of the judgment, converting the data to be synchronized into data conforming to the type of the second data source, and writing the data conforming to the type of the second data source into the second data source.

4. A method according to claim 2 or 3, further comprising:

acquiring configuration information, wherein the configuration information comprises information related to the first data source and information related to the second data source, and the information related to the first data source comprises at least one of the following information: the type of the first data source, the data table in the first data source and the field information in the first data source, and the information related to the second data source includes at least one of the following information: the type of the second data source, a data table in the second data source, and field information in the second data source;

Wherein the determining whether the types of the first data source and the second data source are the same includes:

and judging whether the types of the first data source and the second data source are the same according to the configuration information.

5. A method according to any one of claims 1 to 3, the acquiring the data to be synchronized comprising:

directly reading the data to be synchronized;

and when a data pulling request sent by the proxy computer in the second computer network domain is received, sending the data to be synchronized to the proxy computer in the second computer network domain, wherein the proxy computer in the second computer network domain is used for writing the data to be synchronized into the second data source, and the data pulling request is used for requesting to write the data to be synchronized into the second data source.

6. A method according to any one of claims 1 to 3, the acquiring the data to be synchronized comprising:

And writing the received data to be synchronized sent by the proxy computer in the first computer network domain into the second data source.

7. A method according to any one of claims 1 to 3, the acquiring the data to be synchronized comprising:

the writing the data to be synchronized into the second data includes:

8. A method according to any one of claims 1 to 3, the data to be synchronized being encrypted data;

and writing the data obtained after decrypting the data to be synchronized into the second data source.

9. A method according to any one of claims 1 to 3, the data to be synchronized being compressed data;

and writing the data obtained after decompressing the data to be synchronized into the second data source.

10. The method of claim 1, wherein the controller is configured to obtain the data synchronization task created by a user through an application interactive interface.

11. A method of data synchronization for a proxy computer in a first computer network domain, comprising:

directly reading the data to be synchronized;

and sending the data to be synchronized to a proxy computer in the second computer network domain according to the indication of the data synchronization task, wherein the proxy computer in the second computer network domain is used for writing the data to be synchronized into the second data source.

12. A method of data synchronization for a proxy computer in a second computer network domain, comprising:

13. A method of data synchronization, applied to a gateway device, comprising:

14. A data synchronization apparatus, comprising:

The task acquisition unit is used for receiving a data synchronization task distributed by the controller, wherein the data synchronization task is used for indicating to synchronize data to be synchronized of a first data source in a first computer network to a second data source in a second computer network;

and the data writing unit is used for sending the data to be synchronized to the proxy computer in the second computer network domain according to the indication of the data synchronization task, wherein the proxy computer in the second computer network domain is used for writing the data to be synchronized into the second data source.

15. A data synchronization apparatus, comprising:

16. A data synchronization apparatus, comprising:

17. An electronic device, comprising:

a processor; and

Directly reading the data to be synchronized;

18. An electronic device, comprising:

a processor; and

19. An electronic device, comprising:

a processor; and

20. A computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:

directly reading the data to be synchronized;

21. A computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:

22. A computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to: