CN119697200A - Method, device and related equipment for synchronously transmitting data - Google Patents

Abstract

The application provides a method, a device and related equipment for synchronously transmitting data, which are used for determining target system information of subscription data and service node information for synchronously transmitting the data, determining data to be synchronously transmitted by each service node based on the service node information, configuring Disruptor Consumer consumption thread quantity of each service node, carrying out task allocation on the data to be synchronously transmitted based on the consumption thread quantity, and transmitting the allocated synchronous transmission tasks to a corresponding target system in an asynchronous parallel mode by utilizing the service node. The method is used for supporting high-efficiency synchronization of a large amount of data to a business system subscribed to data through technologies such as a high-performance memory queue, streaming inquiry, asynchronous parallel processing and the like under the condition that hardware resources are limited and message queues and other external middleware cannot be added, and the friendly support can be used for efficiently synchronizing a large amount of data to the business system subscribed to the data under the condition that memory and cpu resources occupy a small space, so that resources are saved and meanwhile, data synchronization business can be efficiently and rapidly completed.

Description

Method, device and related equipment for synchronously transmitting data

Technical Field

The present application relates to the field of computer information processing technologies, and in particular, to a method and apparatus for data synchronous transmission, and related devices.

Background

With the development of informatization, the network services provided in society gradually develop in the direction of the development of main stream technologies in the directions of micro services, distributed clusters, data integration platforms, large data platforms and other large system architectures. The application deployment of the technologies requires more and more resources, the data volume to be processed is also huge and complex, for the system with limited server resources, the task of synchronously transmitting the data with huge and different complexity is that it is difficult to ensure that massive data is synchronously transmitted to the target system subscribing to the data in time and accurately, and the situation of synchronously transmitting the data with massive and different complexity is faced to the situation of synchronously transmitting the data requirement, so that the current industry realizes the data synchronization to a plurality of subscription information systems by adding middleware, the mode needs to re-edit the communication mode of the system to a certain extent, and the efficiency of synchronously processing the data with huge and difficult format is also common.

Disclosure of Invention

Accordingly, embodiments of the present application provide a method, apparatus and related device for data synchronous transmission, so as to at least or partially solve the above-mentioned problems.

In a first aspect, an embodiment of the present application provides a method for synchronously sending data, including:

determining target system information of subscription data and service node information for synchronous data transmission, wherein the target system information and the service node information are determined by the method

The service node information comprises at least the number of service nodes and the number information of each service node, wherein the target system information comprises one or more of target system ID, target system name, target system address and target subscription data type;

Determining data to be synchronized, which are required to be synchronously transmitted by each service node, based on the service node information;

The method comprises the steps of configuring Disruptor Consumer consumption thread numbers of all service nodes, and performing task allocation on data to be synchronized based on the Disruptor Consumer consumption thread numbers;

And transmitting the data to be synchronized to a corresponding target system of the target system information in an asynchronous parallel mode by using the service node.

Optionally, in an embodiment of the present application, the determining, based on the service node information, to-be-synchronized data that needs to be synchronously sent by each service node includes:

The following SQL statement is executed to achieve batch computation of the data to be synchronized:

select from task_data_ SYNC WHERE ID% [ number of service nodes ] = [ number of nodes ],

And performing modular operation on the value of the task number of the data to be synchronized, distributing the data to be synchronized to service nodes of the serial numbers of nodes determined by batch calculation, and configuring each service node to process different data to be synchronized.

Optionally, in an embodiment of the present application, the determining, based on the service node information, to-be-synchronized data that needs to be synchronously sent by each service node includes:

And determining data to be synchronized, which are required to be synchronously transmitted by each service node, by utilizing a stream query mode based on the service node corresponding to the service node information.

Optionally, in an embodiment of the present application, the determining, by using a streaming query manner, to-be-synchronized data that needs to be synchronously sent by each service node based on the service node corresponding to the service node information includes:

When the SQL statement is executed to realize batch calculation of the data to be synchronized, adding configuration ResultSetType. FORWARD_ONLY in a Mapper interface method for querying the data SQL to be synchronized by Mybatis to open a stream query mode for determining each service

The [ HZTC-HXRJGS241001] [ HS2411340CCN ] node needs to synchronize the data to be synchronized that is sent.

Optionally, in an embodiment of the present application, the configuring the number of Disruptor Consumer consuming threads of each service node, based on the number of Disruptor Consumer consuming threads, before performing task allocation on the data to be synchronized, the method further includes:

According to the data type information of the data to be synchronized and the corresponding subscription target system information, packaging the data to be synchronized to form a data set to be synchronized in the form of a JSON character string message, wherein the packaged synchronous data set at least comprises one or more of a data id, an original data message, target system information and a synchronous mode of the data to be synchronized;

And taking the synchronous data set as target data to be synchronized, and supporting task allocation of the number to be synchronized.

Optionally, in an embodiment of the present application, the sending, by the service node, the allocated synchronous sending task to the target system of the corresponding target system information in an asynchronous parallel manner, includes:

And using an asynchronous processing function to send the data set to be synchronized in the form of the JSON character string message to a dispatcher memory queue of a corresponding service node, so that the service node is utilized to send the data to be synchronized to a target system of the corresponding target system information in an asynchronous parallel mode.

Optionally, in an embodiment of the present application, the sending, by the service node, the data to be synchronized to a target system of the corresponding target system information in an asynchronous parallel manner includes:

Analyzing the data id, the original message data, the target system address and the synchronization mode in the json character string message by DisruptorConsumer consumers identifying the json character string message of the data set to be synchronized from the dispersotor memory queue;

And according to the analysis result and the target system address, the target system corresponding to the original message data is processed.

Optionally, in an embodiment of the present application, the method further includes configuring a consumption waiting policy of the disraptor memory queue to be a blocking waiting policy.

[HZTC-HXRJGS241001][HS2411340CCN]

In a second aspect, based on the method for synchronously sending data in the first aspect of the present application, an embodiment of the present application further provides a device for synchronously sending data, including:

The system comprises a determining module, a data synchronization module and a data synchronization module, wherein the determining module is used for determining target system information of subscription data and service node information for carrying out data synchronization transmission, and the service node information at least comprises the number of service nodes and the number information of each service node;

The query module is used for determining data to be synchronized, which are required to be synchronously transmitted, of each service node based on the service node information;

the allocation module is used for configuring Disruptor Consumer consumption thread numbers of the service nodes and performing task allocation on the data to be synchronized based on the Disruptor Consumer consumption thread numbers;

And the synchronous module is used for transmitting the distributed synchronous transmission tasks to the corresponding target system of the target system information in an asynchronous parallel mode by utilizing the service node.

In a third aspect, an embodiment of the present application further provides a computer storage medium, where computer executable instructions are stored, where the computer executable instructions when executed perform any one of the methods for synchronously sending data according to the first aspect of the embodiment of the present application.

In a fourth aspect, the embodiment of the application also provides an electronic device, which comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;

The memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform any one of the methods for data synchronous transmission according to the first aspect of the embodiment of the present application.

The application provides a method, a device and related equipment for synchronously transmitting data, which are characterized in that target system information of subscription data and service node information for synchronously transmitting the data are determined, wherein the service node information at least comprises the number of service nodes and the number information of each service node, the target system information comprises one or more of target system ID, target system name, target system address and target subscription data type, the data to be synchronously transmitted by each service node is determined based on the service node information, disruptor Consumer consumption thread number of each service node is configured, and the data is based on the service node information

[ HZTC-HXRJGS241001] [ HS2411340CCN ] Disruptor Consumer consumption thread number, task allocation is carried out on the data to be synchronized; and transmitting the data to be synchronized to a corresponding target system of the target system information in an asynchronous parallel mode by using the service node. The data synchronous sending mode aims at the conditions that hardware resources are limited and external middleware such as a message queue cannot be newly added, and through technologies such as a high-performance memory queue dispeyptor, stream inquiry, asynchronous parallel processing and the like, under the condition that memory and cpu resources occupy less, the data synchronous sending mode is friendly to support efficient synchronization of a large amount of data to a business system subscribed to the data, and can efficiently and rapidly finish data synchronous business while saving resources.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.

Fig. 1 is a schematic workflow diagram of a method for synchronously sending data according to an embodiment of the present application;

Fig. 2 is a schematic structural diagram of a device for synchronously sending data according to an embodiment of the present application.

Fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to better understand the technical solutions in the embodiments of the present application, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which are derived by a person skilled in the art based on the embodiments of the present application, shall fall within the scope of protection of the embodiments of the present application.

It should be understood that the various steps recited in the method embodiments of the present application may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the application is not limited in this respect.

Embodiment 1,

The embodiment of the application provides a method for synchronously transmitting data, as shown in fig. 1, fig. 1 is the present application

[ HZTC-HXRJGS241001] [ HS2411340CCN ] A workflow schematic of a method for synchronously transmitting data provided in the embodiment comprises:

Step S101, determining target system information of subscription data and service node information for synchronous data transmission. In the embodiment of the application, the service nodes are currently available system servers, wherein the service node information at least comprises the number of the service nodes and the number information of each service node, and the target system information comprises one or more of target system ID, target system name, target system address and target subscription data type. The embodiment of the application firstly determines the target information to be synchronously transmitted and the node information supporting synchronous transmission so as to accurately support the flow of data synchronous transmission.

Step S102, based on the service node information, determining data to be synchronized, which are required to be synchronously transmitted by each service node. The embodiment of the application carries out targeted preliminary allocation of the synchronous data through the acquired service node information, so as to facilitate the timely utilization of a plurality of available nodes to process a large number of data synchronous transmission tasks and improve the efficiency of data synchronization. The embodiment of the application exemplifies the process, for example, when a plurality of data synchronization tasks need to be processed and the currently available service nodes are determined to be the nodes with the numbers of 0,1 and 2, the plurality of data synchronization tasks are distributed to the nodes with the numbers of 0,1 and 2 so as to finish preliminary distribution of data to be synchronized.

And step S103, configuring Disruptor Consumer consumption thread numbers of all service nodes, and performing task allocation on the data to be synchronized based on the Disruptor Consumer consumption thread numbers. Disrupt is a medium-high performance message queue developed by LMAX, and is mainly used for solving the problem of delay of a memory queue. Disrupter can be used as a high-performance bounded memory queue, which is well suited for both [ producer-consumer ] scenarios and [ publish-subscribe ] scenarios. Wherein producer-consumer scenario. The most commonly used scenario [ producer-consumer ] refers to a scenario that the information scenario of the memory queue is "one producer, multiple consumers", and requires sequential processing, and the data synchronous sending process described in the embodiments of the present application belongs to a typical [ producer-consumer ] scenario, and also belongs to a [ publish-subscribe ] scenario. According to the embodiment of the application, the data synchronization transmission process is carried out through the dispatcher queue, so that the data synchronization efficiency can be remarkably improved, and the working requirement of synchronously transmitting a large amount of data burst of the system can be met more friendly. In the embodiment of the present application, the Disruptor Consumer consuming threads of each service node are configured, since the number of consuming threads of the Disrupt is generally managed by the external thread pool, but not

[ HZTC-HXRJGS241001] [ HS2411340CCN ] is directly arranged inside the Disrupter framework, so that a user can better optimize the performance and resource utilization rate of the high-performance queue of the Disrupter by configuring the size of an external thread pool according to actual requirements. Thereby further improving the efficiency of the synchronous data transmission.

Optionally, in one embodiment of the present application, determining, based on the service node information, data to be synchronized that each service node needs to send synchronously includes executing the following SQL statement to implement batch calculation of the data to be synchronized:

selecting from task_data_ SYNC WHERE ID% [ number of service nodes ] = [ number of nodes ], performing modular operation on the value of the task number of the data to be synchronized, distributing the data to be synchronized to the service nodes of the number of the nodes determined by batch calculation, and configuring each service node to process different data to be synchronized. According to the embodiment of the application, through the mode, the same piece of data to be synchronized can be processed by the same service node only by the modular operation, so that the data to be synchronized processed by each service node are uniformly and uniformly distributed, and the system is more balanced in operation. And different service nodes are further configured to process different data to be processed, so that resources are more reasonably utilized, and the processing efficiency of the task to be synchronized is further ensured.

Optionally, in an optional implementation manner of the embodiment of the present application, determining, based on the service node information, data to be synchronized that needs to be synchronously transmitted by each service node includes determining, based on a service node corresponding to the service node information, data to be synchronized that needs to be synchronously transmitted by each service node by using a streaming query manner. The presently embodiment determines the data to be synchronized which are required to be synchronously transmitted by each service node in a streaming query mode, so that the problem that the memory of the system program overflows and OMM crashes due to overlarge data to be synchronized can be effectively solved, and the system is guaranteed to have better working stability and working efficiency.

Optionally, in an embodiment of the present application, determining, by using a streaming query manner, data to be synchronized that needs to be synchronously transmitted by each service node based on the service node information corresponding to the service node information includes adding a configured result set type. Resultsettype. Forward_only means that the cursor of the result set can ONLY move forward, without supporting scrolling. Since the result set can only be traversed forward, the properties

[ HZTC-HXRJGS241001] [ HS2411340CCN ] can be better, and is suitable for service application of query scenes needing to process a large amount of data. According to the embodiment of the application, a large amount of data is processed by adding the configuration resultSetType. FORWARD_ONLY in the used map interface method for Mybatis for inquiring the data sql to be synchronized, so that the operation of carrying out complex scenes such as data rollback or random access on the inquired target data, namely a result set, is reduced, and better inquiry performance is realized. Correspondingly, the method further comprises adding a configuration resultSetType. FORWARD_ONLY in a Mapper interface method for querying the data sql to be synchronized by Mybatis so as to start a streaming query mode, and the method further comprises configuring ResultHandler < T > parameter incoming callback functions to receive a result set for processing the streaming query. The method can meet more application scenes, such as the operation of the scene of the query result and the data rollback or random access, and the like, improves the scene applicability of the method for synchronously transmitting the data, and the process of realizing the query and the acquisition of the data to be synchronized by the data in the mode is simpler, so that the programmer can conveniently perform system development and maintenance.

Step S104, the service node is utilized to send the allocated synchronous sending task to the corresponding target system of the target system information in an asynchronous parallel mode. The embodiment of the application utilizes the service node in an asynchronous parallel mode, specifically utilizes the Disruptor Consumer consumption thread quantity configured in the service node to synchronously transmit the data, so that the high-performance message queue of the Disrupter is more fully utilized to support the synchronous transmission of the data, the high-efficiency synchronous service requirement on a large amount of data is stably realized under the condition of occupying less system resources, and the efficiency of synchronous transmission of the data is improved.

Optionally, in an embodiment of the present application, the number of Disruptor Consumer consuming threads of each service node is configured, and before task allocation is performed on data to be synchronized based on the number of Disruptor Consumer consuming threads, the method further includes packaging the data to be synchronized according to data type information of the data to be synchronized and corresponding subscription target system information, to form a data set to be synchronized in the form of JSON string messages, where the packaged synchronization data set at least includes one or more of a data id, an original data message, target system information, and a synchronization mode of the data to be synchronized, and the synchronization data set is used as target data to be synchronized to support task allocation on the number to be synchronized. In the embodiment of the application, the data to be synchronized is packaged to form the data set to be synchronized in the form of JSON character string message, and then the data set to be synchronized is sent to the high-performance queue for synchronous sending processing, so that the method can be more square

The synchronous transmission processing is carried out on the data to be synchronized by [ HZTC-HXRJGS241001] [ HS2411340CCN ], the pre-analysis processing such as type identification and the like is not needed to be carried out on the data to be synchronized at the stage, and the data to be synchronized at the stage is submitted to the dispeptor for high performance in the subsequent steps, so that the task allocation efficiency of the data to be synchronized at the stage is improved.

Specifically, in the embodiment of the present application, according to the data type information of the data to be synchronized and the corresponding subscription target system information, the implementation process of packaging the data to be synchronized may be performed in a callback function ResultHandler < T >, where each piece of obtained data to be synchronized is processed to implement packaging of the data. In the embodiment of the application, the use of ResultHandler can greatly improve the efficiency of data processing, when a large amount of data needs to be processed, resultHandler is used for processing the query result, so that the condition that one query or packaged result is returned completely is avoided, the occurrence probability of the condition of wasting memory resources is reduced, or the problem that memory overflows OMM collapses is avoided.

Correspondingly, in an alternative implementation manner of the embodiment of the present application, the service node is utilized to send the allocated synchronous sending task to the target system of the corresponding target system information in an asynchronous parallel manner, and the method includes using an asynchronous processing function to send the set of data to be synchronized in the form of the JSON string message to a dispepter memory queue of the corresponding service node, so that the service node is utilized to send the data to be synchronized to the target system of the corresponding target system information in an asynchronous parallel manner.

Optionally, in an embodiment of the present application, the sending the data to be synchronized to the corresponding target system of the target system information by using the service node in an asynchronous parallel manner includes identifying, by DisruptorConsumer consumers, a json string message of the data set to be synchronized from the server memory queue, parsing a data id, original message data, a target system address and a synchronization manner in the json string message, and according to the parsing result, according to the target system address, sending the target system corresponding to the original message data. The specific execution process of the data synchronization is performed by the Disrupter high-performance memory message queue, so that the synchronous transmission efficiency is improved while the accuracy and reliability of the synchronous transmission result are ensured.

Optionally, in one embodiment of the present application, the method further includes configuring a consumption wait policy of the Disrupt memory queue to be a blocking wait policy (BlockingWaitStrategy). BlockingWaitStrategy is a common waiting strategy in the dispeptor framework, which is mainly used for deciding how to wait for the producer to put the event object into the dispeptor queue.

[ HZTC-HXRJGS241001] [ HS2411340CCN ] BlockingWaitStrategy uses ReentrantLock and Condition variables to implement blocking logic to ensure that during event processing, a consumer can wait correctly until an event is available. It implements wait logic by using reentrant locks (ReentrantLock) and Condition variables (conditions). When a consumer attempts to acquire an event, if the event is not available, the consumer may be blocked until the producer places the event in a queue and notifies the consumer. This mechanism ensures thread security, which allows threads to be suspended while waiting, rather than continuously occupying CPU resources, thereby reducing the system CPU footprint.

Optionally, in an implementation manner of the embodiment of the present application, the determining the target system information of the subscription data and the service node information for performing data synchronization transmission includes periodically performing, according to a preset period, a technical step of determining the target system information of the subscription data and the service node information for performing data synchronization transmission, so as to reduce working strength of an automation system for performing the method of the embodiment of the present application while ensuring data synchronization transmission efficiency.

The application provides a data synchronous transmission method, which comprises the steps of determining target system information of subscription data and service node information for carrying out data synchronous transmission, wherein the service node information at least comprises the number of service nodes and the number information of each service node, the target system information comprises one or more of a target system ID, a target system name, a target system address and a target subscription data type, determining data to be synchronized, which are required to be synchronously transmitted, of each service node based on the service node information, configuring Disruptor Consumer consumption thread number of each service node, carrying out task allocation on the data to be synchronized based on the Disruptor Consumer consumption thread number, and transmitting the allocated synchronous transmission tasks to a target system of the corresponding target system information in an asynchronous parallel mode by utilizing the service nodes. The data synchronous sending mode aims at the conditions that hardware resources are limited and external middleware such as a message queue cannot be newly added, and through technologies such as a high-performance memory queue dispeyptor, stream inquiry, asynchronous parallel processing and the like, under the condition that memory and cpu resources occupy less, the data synchronous sending mode is friendly to support efficient synchronization of a large amount of data to a business system subscribed to the data, and can efficiently and rapidly finish data synchronous business while saving resources.

Embodiment II,

Based on the method for synchronously transmitting data provided in the first embodiment of the present application, the embodiment of the present application further provides a device for synchronously transmitting data, as shown in fig. 2, fig. 2 is a schematic structural diagram of a device 20 for synchronously transmitting data provided in the first embodiment of the present application [ HZTC-HXRJGS241001] [ HS2411340CCN ], where the device 20 for synchronously transmitting data includes:

A determining module 201, configured to determine target system information of subscription data and service node information for performing data synchronization transmission, where the service node information at least includes the number of service nodes and number information of each service node;

a query module 202, configured to determine, based on the service node information, data to be synchronized that needs to be synchronously transmitted by each service node;

the allocation module 203 is configured to configure Disruptor Consumer consumption thread numbers of the service nodes, and perform task allocation on the data to be synchronized based on the Disruptor Consumer consumption thread numbers;

And the synchronization module 204 is configured to send the allocated synchronous sending task to the target system of the corresponding target system information in an asynchronous parallel manner by using the service node.

Optionally, in one implementation of the embodiment of the present application, the query module 202 is further configured to execute the following SQL statement to implement batch computation of the data to be synchronized:

select from task_data_ SYNC WHERE ID% [ number of service nodes ] = [ number of nodes ],

And performing modular operation on the value of the task number of the data to be synchronized, distributing the data to be synchronized to service nodes of the serial numbers of nodes determined by batch calculation, and configuring each service node to process different data to be synchronized.

Optionally, in an embodiment of the present application, the query module 202 is further configured to determine, by using a streaming query manner, data to be synchronized that needs to be synchronously sent by each service node based on the service node corresponding to the service node information.

Optionally, in an embodiment of the present application, the query module 202 is further configured to add a configured resultsettype. Forward_only_only to the Mapper interface method for querying the SQL data to be synchronized by Mybatis when executing the SQL statement to implement batch computation of the data to be synchronized, so as to open a streaming query manner, and determine that each service node needs to send the data to be synchronized synchronously.

[HZTC-HXRJGS241001][HS2411340CCN]

Optionally, in an embodiment of the present application, the apparatus 20 further includes an encapsulation module (not shown in the drawing), where the encapsulation module is configured to encapsulate, before the configuring the number of Disruptor Consumer consuming threads of each service node and the task allocation is performed on the data to be synchronized based on the number of Disruptor Consumer consuming threads, the data to be synchronized according to the data type information of the data to be synchronized and the corresponding subscription target system information, to form a data set to be synchronized in the form of a JSON string message, where the encapsulated synchronization data set includes at least one or more of a data id of the data to be synchronized, an original data message, target system information, and a synchronization manner, and uses the synchronization data set as target data to be synchronized to support task allocation for the number to be synchronized.

Optionally, in an embodiment of the present application, the synchronization module 204 is further configured to send the set of data to be synchronized in the form of the JSON string message to a dispeptor memory queue of a corresponding service node by using an asynchronous processing function, so that the service node is utilized to send the data to be synchronized to a target system corresponding to the target system information in an asynchronous parallel manner.

Optionally, in an embodiment of the present application, the synchronization module 204 is further configured to identify, by DisruptorConsumer consumers, a json string message of the dataset to be synchronized from the server memory queue, parse a data id, original message data, a target system address and a synchronization manner in the json string message, and according to the parsing result, and according to the target system address, use a target system corresponding to the original message data.

Optionally, in an embodiment of the present application, the apparatus 20 further includes a configuration module (not shown in the drawing), where the configuration module is configured to configure the consumption waiting policy of the disraptor memory queue to be a blocking waiting policy.

The application provides a device for synchronously transmitting data, which comprises a setting determining module, a setting inquiring module, a setting distributing module, a setting synchronizing module and a processing module, wherein the setting determining module is used for determining target system information of subscription data and service node information for synchronously transmitting data, the service node information at least comprises the number of service nodes and the number information of each service node, the target system information comprises one or more of target system ID, target system name, target system address and target subscription data type, the setting inquiring module is used for determining data to be synchronously transmitted by each service node based on the service node information, the setting distributing module is used for configuring Disruptor Consumer consumption thread number of each service node, and the setting synchronizing module is used for treating the same based on the Disruptor Consumer consumption thread number

And utilizing the service node to send the distributed synchronous sending task to a corresponding target system of the target system information in an asynchronous parallel mode. The data synchronous sending device aims at the conditions that hardware resources are limited and external middleware such as a message queue cannot be newly added, and through technologies such as a high-performance memory queue dispeyptor, stream inquiry, asynchronous parallel processing and the like, under the condition that memory and cpu resources occupy less, the device is friendly to support efficient synchronization of a large amount of data to a business system subscribed to the data, and can efficiently and rapidly complete data synchronous business while saving resources.

Third embodiment,

The embodiment of the application also provides a storage medium, on which a computer program is stored, which when executed by a processor, implements any one of the methods for synchronous data transmission according to the embodiment of the application.

Fourth embodiment,

An embodiment of the present application further provides an electronic device, as shown in fig. 3, and fig. 3 is a schematic structural diagram of an electronic device 30 according to an embodiment of the present application, where the electronic device 30 includes:

One or more processors 301, communication interfaces 302, memory 303, and communication buses 304, the processors 301, memory 303, and communication interfaces 302 completing communication among each other through the communication buses 304;

A memory 303 for storing one or more programs;

the one or more programs, when executed by the one or more processors 301, cause the one or more processors 301 to implement any of the methods for data synchronization transmission described in the first embodiment of the present application.

Thus, the present application has been described with respect to particular embodiments of the present subject matter. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may be advantageous.

In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many of today's process flows

Improvements in [ HZTC-HXRJGS241001] [ HS2411340CCN ] have been seen as straightforward improvements in hardware circuit architecture. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable GATE ARRAY, FPGA)) is an integrated circuit whose logic functions are determined by user programming of the device. A designer programs to "integrate" a digital system layer onto a PLD without having to ask the chip manufacturer to design and fabricate application specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented with "logic compiler (logic compiler)" software, which is similar to the software compiler used in program development and writing, and the original code before being compiled is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but HDL is not just one, but a plurality of kinds, such as ABEL(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description Language)、Confluence、CUPL(Cornell University Programming Language)、HDCal、JHDL(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby Hardware Description Language), and VHDL (Very-High-SPEED INTEGRATED Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application SPECIFIC INTEGRATED Circuits (ASICs), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that it is well possible to implement the controller in a manner other than by means of a purely computer readable program code

[ HZTC-HXRJGS241001] [ HS2411340CCN ] the method steps are logically programmed to cause the controller to perform the same function in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system layers, apparatuses, modules or units set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in the same piece or pieces of software and/or hardware when implementing the present application.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.

It will be appreciated by those skilled in the art that embodiments of the application may be provided as a method, system layer, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular transactions or implement particular abstract data types. The application may also be practiced in distributed computing environments where the application is described by remote locations that are linked through a communications network

[ HZTC-HXRJGS241001] [ HS2411340CCN ] processes devices to execute transactions. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system layer embodiments, the description is relatively simple, as it is substantially similar to method embodiments, with reference to the section of the method embodiments being relevant.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims (10)

1. A method for synchronously transmitting data, comprising:

determining target system information of subscription data and service node information for synchronously transmitting the data, wherein the service node information at least comprises the number of service nodes and the number information of each service node, and the target system information comprises one or more of target system ID, target system name, target system address and target subscription data type;

Determining data to be synchronized, which are required to be synchronously transmitted by each service node, based on the service node information;

The method comprises the steps of configuring Disruptor Consumer consumption thread numbers of all service nodes, and performing task allocation on data to be synchronized based on the Disruptor Consumer consumption thread numbers;

And transmitting the data to be synchronized to a corresponding target system of the target system information in an asynchronous parallel mode by using the service node.

2. The method for synchronously sending data according to claim 1, wherein determining, based on the service node information, data to be synchronized that each service node needs to send synchronously includes executing the following SQL statement to implement batch calculation of the data to be synchronized:

select from task_data_ SYNC WHERE ID% [ number of service nodes ] = [ number of nodes ],

And performing modular operation on the value of the task number of the data to be synchronized, distributing the data to be synchronized to service nodes of the serial numbers of nodes determined by batch calculation, and configuring each service node to process different data to be synchronized.

3. The method for synchronously transmitting data according to claim 2, wherein the determining, based on the service node information, data to be synchronously transmitted by each service node includes:

And determining data to be synchronized, which are required to be synchronously transmitted by each service node, by utilizing a stream query mode based on the service node corresponding to the service node information.

4. The method for synchronously transmitting data according to claim 3, wherein the determining, by using a streaming query method, data to be synchronously transmitted by each service node according to [ HZTC-HXRJGS241001] [ HS2411340CCN ] corresponding to the service node information includes:

when the SQL statement is executed to realize batch calculation of the data to be synchronized, a configuration ResultSetType. FORWARD_ONLY is added in a Mapper interface method for inquiring the data SQL to be synchronized by Mybatis so as to start a stream inquiry mode for determining the data to be synchronized which needs to be synchronously transmitted by each service node.

5. The method for synchronously transmitting data according to claim 1, wherein said configuring Disruptor Consumer consumption threads of each service node is based on said Disruptor Consumer consumption threads, said method further comprising, before task allocation for data to be synchronized:

According to the data type information of the data to be synchronized and the corresponding subscription target system information, packaging the data to be synchronized to form a data set to be synchronized in the form of a JSON character string message, wherein the packaged synchronous data set at least comprises one or more of a data id, an original data message, target system information and a synchronous mode of the data to be synchronized;

And taking the synchronous data set as target data to be synchronized, and supporting task allocation of the number to be synchronized.

6. The method for synchronously transmitting data according to claim 5, wherein said transmitting the data to be synchronized to the corresponding target system of the target system information by using the service node in an asynchronous parallel manner with the allocated synchronous transmission task comprises:

And using an asynchronous processing function to send the data set to be synchronized in the form of the JSON character string message to a dispatcher memory queue of a corresponding service node, so that the service node is utilized to send the data to be synchronized to a target system of the corresponding target system information in an asynchronous parallel mode.

7. The method for synchronously transmitting data according to claim 6, wherein said transmitting the data to be synchronized to the corresponding target system of the target system information in an asynchronous parallel manner by using the service node comprises:

Analyzing the data id, the original message data, the target system address and the synchronization mode in the json character string message by DisruptorConsumer consumers identifying the json character string message of the data set to be synchronized from the dispersotor memory queue;

And [ HZTC-HXRJGS241001] [ HS2411340CCN ] according to the analysis result, according to the target system address, the target system corresponding to the original message data is obtained.

8. The method for synchronous transmission of data according to claim 1, further comprising configuring a consumption waiting policy of the Disrupt memory queue to be a blocking waiting policy.

9. An apparatus for synchronously transmitting data, comprising:

The system comprises a determining module, a data synchronization module and a data synchronization module, wherein the determining module is used for determining target system information of subscription data and service node information for carrying out data synchronization transmission, and the service node information at least comprises the number of service nodes and the number information of each service node;

The query module is used for determining data to be synchronized, which are required to be synchronously transmitted, of each service node based on the service node information;

the allocation module is used for configuring Disruptor Consumer consumption thread numbers of the service nodes and performing task allocation on the data to be synchronized based on the Disruptor Consumer consumption thread numbers;

And the synchronous module is used for transmitting the distributed synchronous transmission tasks to the corresponding target system of the target system information in an asynchronous parallel mode by utilizing the service node.

10. A computer storage medium having stored thereon computer executable instructions which when executed perform the method of data synchronous transmission according to any of claims 1-8.