CN112600943B - Message synchronization method of heterogeneous system in high-concurrency data state - Google Patents
Message synchronization method of heterogeneous system in high-concurrency data state Download PDFInfo
- Publication number
- CN112600943B CN112600943B CN202110222726.XA CN202110222726A CN112600943B CN 112600943 B CN112600943 B CN 112600943B CN 202110222726 A CN202110222726 A CN 202110222726A CN 112600943 B CN112600943 B CN 112600943B
- Authority
- CN
- China
- Prior art keywords
- message
- metadata information
- receiver
- synchronization
- sender
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/625—Queue scheduling characterised by scheduling criteria for service slots or service orders
- H04L47/6275—Queue scheduling characterised by scheduling criteria for service slots or service orders based on priority
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a message synchronization method of a heterogeneous system in a high concurrent data state, which comprises the steps of establishing metadata information between a message sender and a message receiver, acquiring the metadata information by network request timing polling, and storing the metadata information as local files of respective systems; the message receiver establishes a message receiving port, a service and/or a local file directory according to the metadata information; when a message sender initiates message synchronization, the message sender inquires and matches metadata information in a local file, packages and encrypts the message according to the specified metadata information, and sends and delivers the message according to a communication protocol; and after receiving the message, the message receiver processes the message according to the matched metadata information and the importance ranking priority and stores the message. The method realizes the message synchronization among heterogeneous systems, and the message synchronization among the systems is quickly, stably and correctly completed, thereby improving the message processing efficiency and the stability and the correctness of the system.
Description
Technical Field
The invention relates to the technical field of internet information, in particular to a message synchronization method of a heterogeneous system in a high concurrent data state.
Background
The heterogeneous system generally refers to systems with different technical architectures, such as a system program developed by using different languages and running on different system platforms, that is, a heterogeneous system composed of a plurality of systems with different architectures. Because of different systems and architectures, organization modes of message data and the like are different, and information synchronization and sharing are not easy to perform, while under the internet environment, a large number of messages which need to be synchronized exist, and particularly on a C-end internet platform, a large number of messages need to be frequently transmitted between different systems.
Under the circumstance, the traditional message service bus technology is developed, and the bus is designed into a central system by taking the design idea of internal hardware composition of a computer as reference, so that a uniform service entrance is provided, and functions of uniform service management, service routing, protocol conversion, data format conversion and the like are realized. This enables efficient connection of different systems.
However, the message service bus mechanism is located in the central system, and when heterogeneous systems are abundant, the message sending amount is large, and high concurrency is achieved, the message service bus mechanism is very easy to become a resource bottleneck according to the centralized design, and cannot meet the timely and efficient message synchronization. Especially in the event of concurrent peaks, if the service bus goes down, the entire messaging will be disabled.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a message synchronization method of a heterogeneous system in a high-concurrency data state, the method overcomes the defect of message synchronization of a traditional heterogeneous system bus mechanism, realizes message synchronization among heterogeneous systems, and rapidly, stably and correctly completes message synchronization among the systems under the condition of a service peak of massive user access requests, thereby improving the message processing efficiency and the system stability and correctness.
In order to solve the above technical problem, the message synchronization method of the heterogeneous system in the high concurrency data state of the present invention comprises the following steps:
establishing metadata information between a message sender and a message receiver, wherein the metadata information comprises system information, importance levels, communication information, a communication protocol, an encryption mechanism, data capacity, a load strategy and a degradation strategy;
step two, the message sender and the message receiver acquire metadata information through network request timing polling, store the metadata information as local files of respective systems, and update the changed data to the local files if the metadata information is changed;
step three, the message receiver establishes a message receiving port, a service and/or a local file directory according to the metadata information in the local file;
step four, when the message sender initiates message synchronization, the message sender inquires and matches metadata information in a local file, packages and encrypts the message according to the system, communication and protocol information conditions of the message receiver, and sends and delivers the message according to a communication protocol;
and step five, after the message receiver receives the message, processing the message according to the matched metadata information and the importance ranking priority, wherein the message processing comprises unpacking, deciphering and data CRC (cyclic redundancy check) and storing the message.
In the fifth step, when the message capacity and the concurrency are increased, the thread number of the message receiver in operation and the maximum thread number in the file are automatically increased, the thread number of the message receiver and the maximum thread number in the file are maintained by the message receiver and are dynamically adjusted according to the message capacity and the concurrency, and when the message capacity and the concurrency of the message receiver are increased, the thread number is dynamically increased, and the receiving capacity is improved; and if the maximum message capacity is reached, performing degradation strategy processing on the newly received message, wherein the degradation strategy comprises updating metadata information, amplifying the response time of message sending, and performing supplementary processing after temporarily storing the message.
Further, if the heterogeneous system changes, the metadata information is synchronously updated and is periodically synchronized to each message sender and each message receiver.
Further, the message receiver calculates the message capacity and the concurrency amount by adopting an accumulation counter or a unit time counter according to the number of system message senders and the number of received messages, and the number of the message senders and the number of the message receivers are maintained in the metadata.
Further, the message sender judges whether the message receiver reaches the maximum load according to the system load condition of the message receiver in the metadata information and the message sending response time exceeding the system load and the response time, if so, the system metadata are synchronized, the message receiver is judged to reach the maximum load according to the system load and the response time threshold, a fusing strategy is executed, the message synchronization is not carried out any more, and the message receiver is converted into a temporary message to be stored.
Further, after the message processing capability of the message receiver is recovered, the metadata information is updated, and after the metadata information is acquired by the message sender, the message synchronization is recovered.
The message synchronization method of the heterogeneous system in the high-concurrency data state adopts the technical scheme, namely the method establishes metadata information between a message sender and a message receiver, acquires the metadata information by network request timing polling and stores the metadata information as local files of respective systems; the message receiver establishes a message receiving port, a service and/or a local file directory according to the metadata information; when a message sender initiates message synchronization, the message sender performs query matching in metadata information, packages and encrypts the message, and transmits and delivers the message according to a communication protocol; and after receiving the message, the message receiver processes the message according to the matched metadata information and the importance ranking priority and stores the message. The method overcomes the defect of message synchronization of a traditional heterogeneous system bus mechanism, realizes message synchronization among heterogeneous systems, and rapidly, stably and correctly completes message synchronization among the systems under the condition of a service peak of mass user access requests, thereby improving the message processing efficiency and the system stability and correctness.
Drawings
The invention is described in further detail below with reference to the following figures and embodiments:
FIG. 1 is a schematic block diagram of a message synchronization method of a heterogeneous system in a high concurrency data state according to the present invention;
fig. 2 is a schematic diagram of metadata synchronization between a message sender and a message receiver in the method.
Detailed Description
For example, as shown in fig. 1 and fig. 2, the message synchronization method for a heterogeneous system in a high concurrency data state of the present invention includes the following steps:
establishing metadata information between a message sender and a message receiver, wherein the metadata information comprises system information, importance levels, communication information, a communication protocol, an encryption mechanism, data capacity, a load strategy and a degradation strategy; the message sender is a message sender when the message synchronization is carried out in the heterogeneous system, and the message receiver is a message receiver;
step two, the message sender and the message receiver acquire metadata information through network request timing polling, store the metadata information as local files of respective systems, and update the changed data to the local files if the metadata information is changed;
step three, the message receiver establishes a message receiving port, a service and/or a local file directory according to the metadata information, wherein the metadata information comprises system information, communication information, a communication protocol and an encryption mechanism, and the local file directory stores the received message to a local file and then is responsible for processing by other threads; the local file directory is a file directory stored by the message, and when a message receiver receives the message, the message receiver can directly process the message and also can store the message into the local file to realize the exchange of the message;
step four, when the message sender initiates message synchronization, the message sender inquires and matches the metadata information in the local file according to the information of the message receiver by searching the metadata information in the local file, packages and encrypts the message according to the system, communication and protocol information conditions of the message receiver, and sends and delivers the message according to the communication protocol; the query matching element of the message receiver information is used for loading file data into a memory by a message sender, and system information, communication information and protocol information of the message receiver are analyzed in the memory;
after receiving the message, the message receiver processes the message according to the matched metadata information and importance ranking priority, wherein the message processing comprises unpacking, decryption and data CRC (cyclic redundancy check) and saves the message; wherein, unpacking is to analyze the message body from the binary stream and deserialize into a format which can be processed by a program; the decryption is to encrypt and decrypt the message for the purpose of protecting data security, and the used encryption/decryption mode comprises encryption/decryption algorithms such as DES/3 DES/AES/RSA and the like; the data CRC check is a check on the correctness and integrity of the data by CRC 32.
In the fifth step, when the message capacity and the concurrency are increased, the thread number of the message receiver in operation and the maximum thread number in the file are automatically increased, the thread number of the message receiver and the maximum thread number in the file are maintained by the message receiver and are dynamically adjusted according to the message capacity and the concurrency, and when the message capacity and the concurrency of the message receiver are increased, the thread number is dynamically increased, and the receiving capacity is improved; if the maximum message capacity is reached, performing degradation strategy processing on the newly received message, wherein the degradation strategy comprises updating metadata information, amplifying the response time of message sending, and performing supplementary processing after temporarily storing the message;
after receiving the message, the message receiver starts a thread for processing under the default condition, and when the message capacity and the concurrency are increased, the number of the newly added threads is processed in a multi-thread mode, so that the processing capacity is improved; the number of the file threads is the number of the threads for storing the file, when the message needs to be stored, a message receiver can start one thread for storing by default and store the thread as one file, when the message capacity and the concurrency are increased, the number of the added threads is stored into a plurality of files in a multithreading mode.
Preferably, if the heterogeneous system changes, the metadata information is updated synchronously and is synchronized to each message sender and each message receiver in a timing mode.
Preferably, the message receiver calculates the message capacity and the concurrency amount by using an accumulation counter or a unit time counter according to the number of system message senders and the number of received messages. In a heterogeneous system, a message recipient may correspond to multiple message senders, the number of message senders and message recipients being maintained in the metadata.
Preferably, the message sender determines whether the message receiver has reached the maximum load according to the system load condition of the message receiver in the metadata information and the message sending response time, if so, the system metadata is synchronized, and the message receiver is judged to have reached the maximum load according to the system load and response time threshold, and then the fusing policy is executed, the message synchronization is not performed any more, and the message is converted into the temporary message storage.
Preferably, after the message processing capability of the message receiver is recovered, the metadata information is updated, and after the metadata information is acquired by the message sender, the message synchronization is recovered.
The method aims to solve a general message synchronization mechanism between heterogeneous systems, simultaneously leads to overhigh system load caused by a high-concurrency message synchronization process, ensures that the system does not crash because of overhigh load by dynamically expanding system message processing resources and synchronizing the load conditions of a message sender and a message receiver in real time, meets the large requirement that the message sender has urgent synchronous message sending, can ensure that the message receiver accurately and reliably receives messages, improves the message processing efficiency and ensures the stability and the correctness of the system.
Claims (5)
1. A message synchronization method of a heterogeneous system in a high concurrent data state is characterized by comprising the following steps:
establishing metadata information between a message sender and a message receiver, wherein the metadata information comprises system information, importance levels, communication information, a communication protocol, an encryption mechanism, data capacity, a load strategy and a degradation strategy;
step two, the message sender and the message receiver acquire metadata information through network request timing polling, store the metadata information as local files of respective systems, and update the changed data to the local files if the metadata information is changed;
step three, the message receiver establishes a message receiving port, a service and/or a local file directory according to the metadata information in the local file;
step four, when the message sender initiates message synchronization, the message sender inquires and matches metadata information in a local file, packages and encrypts the message according to the system, communication and protocol information conditions of the message receiver, and sends and delivers the message according to a communication protocol;
after receiving the message, the message receiver processes the message according to the matched metadata information and importance ranking priority, wherein the message processing comprises unpacking, decryption and data CRC (cyclic redundancy check) and saves the message;
when the message capacity and the concurrency are increased, the thread number of a message receiver in operation and the maximum thread number in a file are automatically increased, the thread number of the message receiver and the maximum thread number in the file are maintained by the message receiver and are dynamically adjusted according to the message capacity and the concurrency, and when the message capacity and the concurrency of the message receiver are increased, the thread number is dynamically increased, and the receiving capacity is improved; and if the maximum message capacity is reached, performing degradation strategy processing on the newly received message, wherein the degradation strategy comprises updating metadata information, amplifying the response time of message sending, and performing supplementary processing after temporarily storing the message.
2. The method for synchronizing messages of a heterogeneous system in a highly concurrent data state according to claim 1, wherein: if the heterogeneous system changes, the metadata information is synchronously updated and is synchronized to each message sender and each message receiver in a timing mode.
3. The message synchronization method for heterogeneous systems in high concurrency data state according to claim 1 or 2, wherein: and the message receiver calculates the message capacity and the concurrency by adopting an accumulation counter or a unit time counter according to the number of system message senders and the number of received messages, wherein the number of the message senders and the number of the message receivers are maintained in the metadata.
4. The method for message synchronization of heterogeneous systems in a highly concurrent data state according to claim 3, wherein: and the message sender synchronizes the system metadata according to the system load condition of the message receiver in the metadata information and whether the message sending response time exceeds the system load and the response time, if so, the message receiver is judged to reach the maximum load according to the system load and the response time threshold, a fusing strategy is executed, the message synchronization is not carried out any more, and the message is converted into a temporary message for storage.
5. The method for message synchronization of heterogeneous systems in high concurrency data state as claimed in claim 4, wherein: after the message processing capability of the message receiver is recovered, the metadata information is updated, and after the metadata information is acquired by the message sender, the message synchronization is recovered.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110222726.XA CN112600943B (en) | 2021-03-01 | 2021-03-01 | Message synchronization method of heterogeneous system in high-concurrency data state |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110222726.XA CN112600943B (en) | 2021-03-01 | 2021-03-01 | Message synchronization method of heterogeneous system in high-concurrency data state |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112600943A CN112600943A (en) | 2021-04-02 |
| CN112600943B true CN112600943B (en) | 2021-06-01 |
Family
ID=75207547
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110222726.XA Expired - Fee Related CN112600943B (en) | 2021-03-01 | 2021-03-01 | Message synchronization method of heterogeneous system in high-concurrency data state |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112600943B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115221543B (en) * | 2022-08-30 | 2022-11-29 | 成都瑞安信信息安全技术有限公司 | File service-based multi-file concurrent encryption and decryption method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104133674A (en) * | 2014-07-11 | 2014-11-05 | 国家电网公司 | Heterogeneous system and model synchronization method of heterogeneous system |
| CN110515910A (en) * | 2019-08-19 | 2019-11-29 | 中国建材集团财务有限公司 | Data processing method, device and computer-readable storage medium among heterogeneous systems |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105338113B (en) * | 2015-11-25 | 2018-06-26 | 北京航空航天大学 | A kind of multi-platform data interconnection system for Urban Data resource-sharing |
| CN110162571A (en) * | 2019-04-26 | 2019-08-23 | 厦门市美亚柏科信息股份有限公司 | A kind of system, method, storage medium that data among heterogeneous databases synchronize |
| US20200341855A1 (en) * | 2019-04-28 | 2020-10-29 | Synamedia | Object store specialized backup and point-in-time recovery architecture |
| CN112306567B (en) * | 2019-07-26 | 2023-07-21 | 广州虎牙科技有限公司 | Cluster management system and container management and control method |
| CN111431985B (en) * | 2020-03-18 | 2023-04-21 | 紫光云技术有限公司 | Method for synchronizing resource states among cloud service heterogeneous subsystems |
-
2021
- 2021-03-01 CN CN202110222726.XA patent/CN112600943B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104133674A (en) * | 2014-07-11 | 2014-11-05 | 国家电网公司 | Heterogeneous system and model synchronization method of heterogeneous system |
| CN110515910A (en) * | 2019-08-19 | 2019-11-29 | 中国建材集团财务有限公司 | Data processing method, device and computer-readable storage medium among heterogeneous systems |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112600943A (en) | 2021-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108449405B (en) | Event reminding method and device, computer equipment and storage medium | |
| US10432741B2 (en) | Systems and methods for guaranteeing delivery of pushed data to remote clients | |
| CN111510474B (en) | Data transmission method based on message middleware and related equipment | |
| CN111401904B (en) | Consensus method and system in alliance chain | |
| US20190207887A1 (en) | Techniques for message deduplication | |
| CN102761864A (en) | Data transmission method, system and device | |
| CN113014528A (en) | Message processing method, processing unit and virtual private network server | |
| CN111611090B (en) | Distributed message processing method and system | |
| CN104184756A (en) | Data synchronization method, device and system | |
| CN107548039B (en) | Short message retry processing method, device and system | |
| CN111831748A (en) | Data synchronization method, device and storage medium | |
| CN107231435B (en) | Data synchronization monitoring method and system | |
| US20190207873A1 (en) | Techniques for batched bulk processing | |
| US20190207899A1 (en) | Techniques for messaging conversation indexing | |
| CN110213359B (en) | Internet of vehicles networking data pushing system and method based on D2D | |
| ZA200509949B (en) | Method and device for configuring data in communication device | |
| CN112600943B (en) | Message synchronization method of heterogeneous system in high-concurrency data state | |
| CN103942324A (en) | Data real-time synchronization system and method | |
| CN110909030B (en) | Information processing method and server cluster | |
| US10673791B2 (en) | Techniques for data reads from secondary stores | |
| CN105812492B (en) | A kind of method of data synchronization and system | |
| US20040210669A1 (en) | Apparatus and method for distributing packet without IP reassembly | |
| CN112667586B (en) | Method, system, equipment and medium for synchronizing data based on stream processing | |
| CN113315875B (en) | Call information synchronization method and device, call information synchronization equipment and storage medium | |
| US20080212505A1 (en) | Method and receiving apparatus for processing arq block in wibro system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210601 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |