CN109359139A - Data synchronization method, system, electronic device and computer readable storage medium - Google Patents

Abstract

The embodiment of the disclosure discloses a data synchronization method, a system, an electronic device and a computer readable storage medium, wherein the method comprises the following steps: sending the data in the database to a message storage processing flow; consuming and analyzing data in the processing flow stored in the message to obtain analyzed data; and filtering the analysis data according to a preset filtering rule to obtain filtering data, and storing the filtering data into a memory. According to the scheme, data can be imported from the relational database to the non-relational database, data can be written in real time, and meanwhile data can be traced back.

Description

Data synchronization method, system, electronic device, and computer-readable storage medium

technical field

The present disclosure relates to the field of communication technologies, and in particular, to a data synchronization method, system, electronic device, and computer-readable storage medium.

Background technique

Currently, data is imported from a relational database to a non-relational database, and the programming model MapReduce (map reduction) in the open source software Sqoop is used to perform batch offline data calculations. The open source software Sqoop is very suitable for offline data manipulation tasks.

Because, during data migration, there will be a large amount of data, and the programming model MapReduce will execute for a long time, making the real-time data poor.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide a data synchronization method and system, an electronic device, and a computer-readable storage medium.

In a first aspect, an embodiment of the present disclosure provides a data synchronization method.

Specifically, the data synchronization method includes:

The process of sending the data in the database to the distributed publish-subscribe message;

Consume and parse the data in the processing flow of the distributed publish-subscribe message to obtain parsed data; and

The parsed data is filtered according to a preset filtering rule to obtain filtered data, and the filtered data is stored in a memory.

With reference to the first aspect, in a first implementation manner of the first aspect of the present disclosure, before the data in the database is sent to the processing flow of the distributed publish-subscribe message, it includes: initializing configuration information; wherein the configuration information includes: The connection configuration of the database, the configuration information of the distributed publish and subscribe messages, the parameter information of data processing, the preset filtering rules, the structure information of the table in the memory.

In combination with the first aspect and the first implementation manner of the first aspect, in the second implementation manner of the first aspect of the present disclosure, the data in the processing flow of the distributed publish-subscribe message is data in a preset format .

With reference to the first aspect, the first implementation manner of the first aspect, and the second implementation manner of the first aspect, in a third implementation manner of the first aspect of the present disclosure, when the filtering data is stored in the memory After that, it also includes: a function for storing data in the processing flow of consuming distributed publish-subscribe messages.

In a second aspect, an embodiment of the present disclosure provides a data backtracking method.

Specifically, the data backtracking method includes:

Set the parameters of the backtracking data table, and obtain the corresponding offset according to the parameters;

According to the corresponding offset, the data in the processing flow of the distributed publish-subscribe message is consumed and parsed to obtain parsed data;

The parsed data is filtered according to a preset filtering rule to obtain filtered data, and the filtered data is stored in a memory.

In a third aspect, an embodiment of the present disclosure provides a data synchronization system.

Specifically, the data synchronization system includes:

The sending module is configured to send the data in the database to the processing flow of the distributed publish-subscribe message;

A parsing module, configured to consume and parse the data in the processing flow of the distributed publish-subscribe message to obtain parsed data;

a filtering module, configured to filter the parsed data according to a preset filtering rule to obtain filtered data;

A storage module configured to store the filtering data in a memory.

With reference to the third aspect, in a first implementation manner of the third aspect of the present disclosure, the data synchronization system further includes: an initialization module configured to initialize configuration information; wherein the configuration information includes: database connection configuration, distribution configuration information, parameter information of data processing, preset filtering rules, and structure information of tables in memory.

In combination with the third aspect and the first implementation manner of the third aspect, in the second implementation manner of the third aspect of the present disclosure, the data in the processing flow of the distributed publish-subscribe message is data in a preset format .

With reference to the third aspect, the first implementation manner of the third aspect, and the second implementation manner of the third aspect, in a third implementation manner of the third aspect of the present disclosure, the storage module is configured to store consumption distribution A function of the data in the processing flow of a publish-subscribe message.

In a fourth aspect, an embodiment of the present disclosure provides a data backtracking system.

Specifically, the data backtracking system includes:

A settings module, configured to set the parameters of the backtracking data table;

a function obtaining module, configured to obtain the corresponding offset according to the parameter;

The parsing module is configured to consume and parse the data in the processing flow of the distributed publish-subscribe message to obtain the parsed data;

a filtering module, configured to filter the parsed data according to a preset filtering rule to obtain filtered data;

A storage module configured to store the filtering data in a memory.

In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor, where the memory is configured to store one or more computer instructions that support the data synchronization system to perform the data synchronization method in the first and second aspects above , the processor is configured to execute computer instructions stored in the memory. The data synchronization system may also include a communication interface for the data synchronization system to communicate with other devices or a communication network.

In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by a data synchronization system, which includes the data synchronization method used in the data synchronization system for executing the first and second aspects above. The computer instructions involved.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

The above technical solution realizes the real-time synchronization processing of data in the data table by sending the data in the database to the processing flow of the distributed publish-subscribe message, analyzing and filtering it, and storing the filtered data in the corresponding memory.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

Other features, objects and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the attached image:

1 shows a flowchart of a data synchronization method according to an embodiment of the present disclosure;

2 shows a flowchart of a data backtracking method according to an embodiment of the present disclosure;

3 shows a structural block diagram of a data synchronization system according to an embodiment of the present disclosure;

4 shows a structural block diagram of a data backtracking system according to an embodiment of the present disclosure;

FIG. 5 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a computer system suitable for implementing a data synchronization or data backtracking method according to an embodiment of the present disclosure.

Detailed ways

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Also, for the sake of clarity, parts unrelated to describing the exemplary embodiments are omitted from the drawings.

In the present disclosure, it should be understood that terms such as "comprising" or "having" are intended to indicate the presence of features, numbers, steps, acts, components, parts, or combinations thereof disclosed in this specification, and are not intended to exclude a or multiple other features, numbers, steps, acts, components, parts, or combinations thereof may exist or be added.

In addition, it should be noted that the embodiments of the present disclosure and the features of the embodiments may be combined with each other under the condition of no conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

The technical solution provided by the embodiments of the present disclosure is to parse and process the processing flow of the binary log file BinLog of the relational database management system mySQL and then send it to the distributed publish-subscribe message Kafka. store the obtained data in the table corresponding to the mySQL database. Among them, the rule engine is used to filter the parsed data according to the custom method, and return the filtering result. Among them, Kafka is a high-throughput distributed publishing message subscription system, which is used as the middleware for message storage.

FIG. 1 shows a flowchart of a data synchronization method according to an embodiment of the present disclosure. As shown in FIG. 1, the data synchronization method includes the following steps S110-S130:

In step S110, the data in the database is sent to the Kafka processing flow;

In step S120, the data in the Kafka processing flow is consumed and parsed to obtain parsed data;

In step S130, filtering the parsed data according to a preset filtering rule to obtain filtering data, and storing the filtering data in a memory.

As mentioned above, the data in the database is processed in the prior art as follows: using the open source tool Sqoop to execute tasks at regular intervals every morning, importing the data in the database from the relational database to the non-relational database in batches, and then constructing a data warehouse, Users can use the interactive Structured Query Language (SQL, Structured Query Language) to complete data query and other operations. Due to the large amount of data to be processed and the long execution time, the real-time performance of the data is poor.

Considering the above defects, in this embodiment, a data synchronization method is proposed. The method sends the data in the database to the Kafka processing flow, parses and filters the data, and stores the filtered data in the corresponding memory. Real-time synchronization of data in the data table.

Among them, before the data in the database is sent to the Kafka processing process, the binary log file BinLog needs to be parsed.

Among them, the data in the Kafka processing flow is consumed and parsed. As the middleware for message storage, the Kafka system cannot receive data sent from all databases at one time, so it can only receive the next batch of data after consuming the data in Kafka. The format of the parsed data can be Map<K,V>, which is convenient to store in the corresponding memory. The data stored in the Kafka processing flow is all data in a preset format (for example, JSON format) parsed by the DRC (Data Replication Center, data replication center) or the Canal database incremental log parsing system. Among them, JSON (JavaScript Object Notation, JavaScript Object Notation).

The preset filtering rules can be customized through functions, and the rule engine filters the data by loading the customized functions to obtain filtered data. Among them, the filter data is also in Map<K,V> format.

Wherein, storing the filtering data in the memory includes: storing the filtering data in a Kudu table or an HBase table.

In an optional implementation of this embodiment, before sending the data in the database to the Kafka processing flow, it includes: initialization configuration information; wherein the configuration information includes: database connection configuration, Kafka configuration information, and data processing parameters information, preset filtering rules, structure information of tables in memory. Among them, the configuration information is stored in the database. Among them, the parameter information of data processing is converted into the classes that need to be used in the data processing process, and by broadcasting the classes, the Excutor system can obtain the broadcast metadata information when performing tasks.

In an optional implementation manner of this embodiment, after storing the filtering data in the memory, the method further includes: a function for storing and consuming data in the Kafka processing flow. Wherein, the function is offset. The function of the function offset is to take the specified reference as the reference system, and obtain a new reference through the given offset.

FIG. 2 shows a flowchart of a data backtracking method according to an embodiment of the present disclosure. As shown in Figure 2, the data backtracking method includes the following steps S210-S230:

In step S210, the parameters of the backtracking data table are set, and according to the parameters, the corresponding offset is obtained (for example, through the function offset);

In step S220, according to the corresponding offset, the data in the Kafka processing flow is consumed and parsed to obtain parsed data;

In step S230, the parsed data is filtered according to a preset filtering rule to obtain filtered data, and the filtered data is stored in a memory.

In this embodiment, the data that needs to be queried and verified is backtracked through the stored offset, so as to ensure the accuracy and consistency of the data.

The parameters of the backtracking data table include the name of the data table to be backtracked, the backtracking time period, and the type of the table that stores the backtracking data. The parameters of the backtracking data table are inputted externally.

Wherein, obtaining the corresponding offset according to the parameter includes: finding the corresponding data table according to the name of the data table to be backtracked, and then finding the corresponding offset according to the backtracking time period. Specifically, the parameters are parsed according to the time period, and it is determined whether to backtrack by day or by hour, and obtain the configuration information of the table to be backtracked.

Among them, the parameter information of data processing is converted into the classes that need to be used in the data processing process, and by broadcasting the classes, the Excutor system can obtain the broadcast metadata information when performing tasks.

Among them, according to the corresponding offset, the data in the Kafka processing flow is consumed and parsed, and the parsed data is obtained. Specifically, the data in Kafka is read through the Spark-Kafka API, where the data is the offset data for the specified interval.

Among them, the data in the Kafka processing flow is consumed and parsed. As the middleware for message storage, the Kafka system cannot receive data sent from all databases at one time, so it can only receive the next batch of data after consuming the data in Kafka. The format of the parsed data can be Map<K,V>, which is convenient to store in the corresponding memory. The data stored in the Kafka processing process are all JSON-formatted data parsed by the DRC (Data Replication Center) or the Canal database incremental log parsing system.

The preset filtering rules can be customized through functions, and the rule engine filters the data by loading the customized functions to obtain filtered data. Among them, the filter data is also in Map<K,V> format.

Wherein, storing the filtering data in the memory includes: storing the filtering data in a Kudu table or an HBase table.

The following system embodiments of the present disclosure can be used to execute the method embodiments of the present disclosure.

FIG. 3 shows a structural block diagram of a data synchronization system according to an embodiment of the present disclosure. The system can be implemented as part or all of an electronic device through software, hardware, or a combination of the two. As shown in Figure 3, the data synchronization system includes:

The sending module 302 is configured to send the data in the database to the Kafka processing flow;

The parsing module 303 is configured to consume and parse the data in the Kafka processing flow to obtain the parsed data;

The filtering module 304 is configured to filter the parsed data according to a preset filtering rule to obtain filtered data;

The storage module 305 is configured to store the filtering data in the memory.

As mentioned above, the data in the database is processed in the prior art as follows: using the open source tool Sqoop to execute tasks at regular intervals every morning, importing the data in the database from the relational database to the non-relational database in batches, and then constructing a data warehouse, Users can use the interactive Structured Query Language (SQL, Structured Query Language) to complete data query and other operations. Due to the large amount of data to be processed and the long execution time, the real-time performance of the data is poor.

Considering the above defects, in this embodiment, a data synchronization system is proposed, which sends the data in the database to the Kafka processing flow, parses and filters the data, and stores the filtered data in the corresponding memory to realize Real-time synchronization of data in data tables. .

Among them, before the data in the database is sent to the Kafka processing process, the binary log file BinLog needs to be parsed.

Among them, the Kafka system, as the middleware for message storage, cannot receive data sent from all databases at one time, so it can only receive the next batch of data after consuming the data in Kafka. The format of the parsed data can be Map<K,V>, which is convenient to store in the corresponding memory. The data stored in the Kafka processing process are all JSON-formatted data parsed by the DRC (Data Replication Center) or the Canal database incremental log parsing system.

The preset filtering rules can be customized through functions, and the rule engine filters the data by loading the customized functions to obtain filtered data. Among them, the filter data is also in Map<K,V> format.

The storage module 305 is configured to store the filtering data in a Kudu table or an HBase table.

In an optional implementation manner of this embodiment, the data synchronization system further includes: an initialization module 301 configured to initialize configuration information; wherein the configuration information includes: database connection configuration, Kafka configuration information, data processing Parameter information, preset filtering rules, structure information of tables in memory. Among them, the configuration information is stored in the database. Among them, the parameter information of data processing is converted into the classes that need to be used in the data processing process, and by broadcasting the classes, the Excutor system can obtain the broadcast metadata information when performing tasks.

In an optional implementation manner of this embodiment, the storage module 305 is configured to store a function of consuming data in the Kafka processing flow. Wherein, the function is offset. The function of the function offset is to take the specified reference as the reference system, and obtain a new reference through the given offset.

FIG. 4 shows a structural block diagram of a data retrospective system according to an embodiment of the present disclosure. The system can be implemented as part or all of an electronic device through software, hardware, or a combination of the two. As shown in Figure 4, the data backtracking system includes:

The setting module 401 is configured to set the parameters of the retrospective data table;

The function obtaining module 402 is configured to obtain the corresponding offset according to the parameter;

The parsing module 403 is configured to consume and parse the data in the Kafka processing flow to obtain the parsed data;

The filtering module 404 is configured to filter the parsed data according to preset filtering rules to obtain filtered data;

The storage module 405 is configured to store the filtering data in the memory.

Specifically, for the specific limitations of the data retrospective system, reference may be made to the above-mentioned limitations on the data retrospective method, which will not be repeated here.

The present disclosure also discloses an electronic device. FIG. 5 shows a structural block diagram of the electronic device according to an embodiment of the present disclosure. As shown in FIG. 5 , the electronic device 500 includes a memory 501 and a processor 502; wherein,

The memory 501 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 502 to implement any of the above method steps.

FIG. 6 is a schematic structural diagram of a computer system suitable for implementing the data synchronization method according to an embodiment of the present disclosure.

As shown in FIG. 6, a computer system 600 includes a central processing unit (CPU) 601, which can be loaded into a random access memory (RAM) 603 according to a program stored in a read only memory (ROM) 602 or a program from a storage section 608 Instead, various processes in the above-described embodiments are executed. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601 , the ROM 602 and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc. ; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

In particular, according to embodiments of the present disclosure, the methods described above may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a readable medium thereof, the computer program including program code for executing the data synchronization method. In such an embodiment, the computer program may be downloaded and installed from the network via the communication section 609 and/or installed from the removable medium 611 .

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the diagram or block diagram may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function. executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units or modules involved in the embodiments of the present disclosure can be implemented in software or hardware. The described units or modules may also be provided in the processor, and the names of these units or modules do not constitute limitations to the units or modules themselves in certain circumstances.

As another aspect, the present disclosure also provides a computer-readable storage medium. The computer-readable storage medium may be a computer-readable storage medium included in the system described in the foregoing embodiments; A computer-readable storage medium that fits into a device. The computer-readable storage medium stores one or more programs used by one or more processors to perform the methods described in the present disclosure.

The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. It should be understood by those skilled in the art that the scope of the invention involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover the above technical features without departing from the inventive concept. Other technical solutions formed by any combination of its equivalent features. For example, a technical solution is formed by replacing the above-mentioned features with the technical features disclosed in the present disclosure (but not limited to) with similar functions.

Claims (10)

1. A method of data synchronization, comprising:

sending the data in the database to a processing flow of the distributed publishing and subscribing message;

consuming and analyzing the data in the distributed publish-subscribe message processing flow to obtain analyzed data; and

and filtering the analysis data according to a preset filtering rule to obtain filtering data, and storing the filtering data into a memory.

2. The method of claim 1, prior to sending the data in the database to the process flow of the distributed publish-subscribe message, comprising:

initializing configuration information;

wherein the configuration information includes: the method comprises the steps of continuous configuration of a database, configuration information of distributed publish-subscribe messages, parameter information of data processing, preset filtering rules and structure information of a table in a memory.

3. The method of claim 1, further comprising, after said storing said filtered data in memory:

and storing a function for consuming the data in the processing flow of the distributed publish-subscribe message.

4. A data backtracking method, comprising:

setting parameters of a backtracking data table, and acquiring corresponding offset according to the parameters;

according to the corresponding offset, consuming and analyzing data in the processing flow of the distributed publishing and subscribing message to obtain analyzed data; and

and filtering the analysis data according to a preset filtering rule to obtain filtering data, and storing the filtering data into a memory.

5. A data synchronization system, comprising:

the sending module is configured to send the data in the database to a processing flow of the distributed publishing and subscribing message;

the analysis module is configured to consume and analyze data in the processing flow of the distributed publish-subscribe message to obtain analysis data;

the filtering module is configured to filter the analysis data according to a preset filtering rule to obtain filtered data;

a storage module configured to store the filtered data in a memory.

6. The system of claim 5, further comprising:

an initialization module configured to initialize the configuration information;

wherein the configuration information includes: the method comprises the steps of continuous configuration of a database, configuration information of distributed publish-subscribe messages, parameter information of data processing, preset filtering rules and structure information of a table in a memory.

7. The system of claim 5, wherein the storage module is configured to store a function of data in the process flow of consuming the distributed publish-subscribe message.

8. A data backtracking system, comprising:

a setting module configured to set parameters of the backtracking data table;

a function obtaining module configured to obtain a corresponding offset according to the parameter;

the analysis module is configured to consume data in the processing flow of the distributed publish-subscribe message and analyze the data to obtain analysis data;

the filtering module is configured to filter the analysis data according to a preset filtering rule to obtain filtered data;

a storage module configured to store the filtered data in a memory.

9. An electronic device comprising a memory and a processor; wherein,

the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any of claims 1-4.

10. A computer-readable storage medium having stored thereon computer instructions, characterized in that the computer instructions, when executed by a processor, carry out the method steps of any of claims 1-4.