CN111651284B

CN111651284B - Method, device, system and storage medium for processing business data

Info

Publication number: CN111651284B
Application number: CN202010447040.6A
Authority: CN
Inventors: 杨旺明
Original assignee: Taikang Life Insurance Co ltd; Taikang Insurance Group Co Ltd
Current assignee: Taikang Life Insurance Co ltd; Taikang Insurance Group Co Ltd
Priority date: 2020-05-25
Filing date: 2020-05-25
Publication date: 2024-06-28
Anticipated expiration: 2040-05-25
Also published as: CN111651284A

Abstract

The application provides a method, a device, a system and a storage medium for processing service data, which are used for improving the reliability of transmitting the service data. The method comprises the following steps: if the first database is determined to be abnormal in the process of processing the current service data, and the second database is determined to be not abnormal in the process of processing the current service data, acquiring operation information of the second database in the process of processing the current service data, and controlling the first database to process the current service data again; if the message middleware is determined to receive the current service data and is abnormal, acquiring the current service data from the first database, and retransmitting the current service data to the message middleware; and if the abnormal times reach the preset times, acquiring the current service data from the second database, and retransmitting the current service data to the message middleware.

Description

Method, device, system and storage medium for processing business data

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, a system, and a storage medium for processing service data.

Background

In some scenarios, data transmission between multiple systems is required, for example, in the insurance industry, transmission of a large amount of service data between a customer resource system and a salesman assessment system is required, so as to assess salary of a salesman or track services of a customer.

At present, when service transmission is performed between two systems, in the transmission process of service data, due to large data volume, unstable network, abnormal hardware equipment and the like, abnormal service data processing may occur, and once the service data process is abnormal, fault may occur in service data between two service systems, resulting in lower reliability of the transmitted service data.

Disclosure of Invention

The embodiment of the application provides a method, a device, a system and a storage medium for processing service data, which are used for improving the reliability of transmitting the service data.

In a first aspect, there is provided a method of processing traffic data, the method comprising:

If the first database is determined to be abnormal in the process of processing the current service data, and the second database is determined to be not abnormal in the process of processing the current service data, acquiring operation information of the second database in the process of processing the current service data, and controlling the first database to process the current service data again according to the operation information;

If the message middleware is determined to receive the current service data and is abnormal, acquiring the current service data from the first database, and retransmitting the current service data to the message middleware; wherein, all service data which is not confirmed to be successfully transmitted are cached in the first database;

If the number of times that the message middleware re-receives the current service data from the first database and is abnormal reaches the preset number of times, acquiring the current service data from the second database, and re-sending the current service data to the message middleware; wherein the second database stores all of the business data generated by the source system.

In the embodiment of the application, when no abnormality occurs in the process of processing the service data by the second database and the process of processing the service data by the first database, the first database can be controlled to process the service data according to the operation information of processing the service data by the second database, thereby eliminating the abnormal condition of the first database, avoiding unnecessary rollback operation on the second database and improving the efficiency of processing the service data. And by adding the first database, when the service data sent to the message middleware is abnormal, the service data can be directly read in the first database and sent to the message middleware again, when the times of receiving the abnormal condition of the service data from the first database reach the preset times, the service data can be obtained from the second database again and sent to the message middleware again, and in the case of abnormal service data sending, the service data can be obtained again based on the two databases, so that the abnormal condition of service data transmission is reduced, and the reliability of processing the service data is improved.

Optionally, after resending the current service data to the message middleware, the method includes:

And if the message middleware is determined to receive the current service data sent by the source system, deleting the current service data stored in the first database.

In the embodiment of the application, after the service data sent by the message middleware receiving source system is determined, the first database can be controlled to delete the service data stored in the first database, so that the memory occupation is reduced, the unnecessary resource occupation can be avoided, and the utilization rate of the memory resource is improved to a certain extent.

Optionally, before the current service data is acquired from the second database, the method includes:

if it is determined that an abnormality occurs in the process of writing the current service data into the first database by the source system or in the process of writing the current service data into the second database by the source system, the second database is controlled to roll back to a state before writing the current service data, and the source system is instructed to reacquire the current service data.

In the embodiment of the application, when the first database or the second database is abnormal in the process of writing the service data, the abnormal condition can be eliminated by rolling back the second database, the operation logic is simple, and the consistency of the service data processed by the two databases in the service data transmission process is maintained. And after the write operation of the first database is put in the write operation of the second database, when the second database is abnormal, the second database can be directly rolled back, the write operation of the first database is not needed to be continued, and the efficiency of processing the service data is improved.

Optionally, after obtaining the current service data from the second database and resending the current service data to the message middleware, the method includes:

and if the message middleware is determined to re-receive the current service data from the second database, indicating a source system to re-send the current service data to the message middleware or a destination system.

In the embodiment of the application, when the service data received from the second database is abnormal, the source system can be instructed to directly send the service data to the target system or send the service data to the message middleware again. The method of the embodiment of the application can instruct the source system to send the service data to the message middleware or the destination system under the condition that the service data is sent abnormally for a plurality of times, thereby reducing the abnormal transmission condition of the service data through the database and further improving the reliability of processing the service data.

Optionally, the method further comprises:

If the message middleware is determined to re-receive the abnormality cause of the abnormality of the current service data from the second database, wherein the abnormality cause is the abnormality of the message middleware, first indication information is sent to the source system, so that the source system sends the current service data to the destination system through a communication component except the message middleware; the first indication information is used for indicating that the message middleware is abnormal;

If the message middleware is determined to re-receive the abnormality cause of the abnormality of the current service data from the second database as the read-write abnormality of the second database, sending second indication information to the source system so that the source system re-writes the current service data into the second database and re-sends the current service data to the message middleware; wherein the second indication information is used for indicating that the second database is abnormal.

In the embodiment of the application, when the message middleware receives the service data from the second database and is abnormal, if the abnormality is caused by the abnormality of the message middleware, the source system can be instructed to adopt other communication components to send the service data to the destination system, so that the service data can be successfully sent to the destination system, and the reliability of service data transmission is improved. If the abnormality is caused by abnormal reading and writing of the second database, the source system can be instructed to write the service data into the second database again and send the service data to the message middleware, so that the service data can be successfully stored in the second database and successfully sent to the message middleware, the destination system can successfully receive the service data, and the reliability of service data transmission is improved.

Optionally, after instructing the source system to resend the current service data to the message middleware or the destination system, the method includes:

And according to the abnormal information generated when the current service data from the second database is abnormal, the current service data is sent to the destination system by the message middleware, so that the destination system determines whether the current service data in the abnormal information is the same as the service data associated with the current service data re-received by the destination system.

In the embodiment of the application, for the abnormal service data, after the target system successfully receives the service data, the service data can be further determined to determine whether the service data in the abnormal information is consistent with the service data successfully received by the target system, thereby further ensuring the consistency in the service data transmission process and improving the reliability of the service data transmission.

In a second aspect, an apparatus for processing service data is provided, including:

The first processing unit is used for acquiring operation information in the process of processing the current service data by the second database if the first database is determined to be abnormal in the process of processing the current service data and the second database is determined to be not abnormal in the process of processing the current service data, and controlling the first database to process the current service data again according to the operation information;

The second processing unit is used for acquiring the current service data from the first database if the message middleware is determined to receive the current service data to be abnormal; wherein, all service data which is not confirmed to be successfully transmitted are cached in the first database;

a transceiver unit, configured to resend the current service data to the message middleware;

The second processing unit is further configured to acquire the current service data from the second database if it is determined that the number of times that the message middleware re-receives the current service data from the first database and that the abnormality occurs reaches a preset number of times; wherein the second database stores all service data generated by the source system;

The receiving and transmitting unit is further configured to resend the current service data to the message middleware.

Optionally, the first processing unit is further configured to:

And after the current service data is resent to the message middleware, deleting the current service data stored in the first database if the message middleware is determined to receive the current service data sent by the source system.

Optionally, the first processing unit is further configured to:

Before the current service data is acquired from the second database, if it is determined that the first database is abnormal in the process of writing the current service data by the source system, or if it is determined that the second database is abnormal in the process of writing the current service data by the source system, the second database is controlled to roll back to a state before writing the current service data, and the source system is instructed to reacquire the current service data.

Optionally, the second processing unit is further configured to: after obtaining the current service data from a second database and resending the current service data to the message middleware, determining that the message middleware resends the current service data from the second database and is abnormal;

the transceiver unit is further configured to: and the source system is instructed to resend the current service data to the message middleware or the destination system.

Optionally, the transceiver unit is specifically configured to:

If the message middleware is determined to re-receive the abnormality cause of the abnormality of the current service data from the second database, wherein the abnormality cause is the abnormality of the message middleware, first indication information is sent to the source system, so that the source system sends the current service data to the destination system through a communication component except the message middleware; the first indication information is used for indicating that the message middleware is abnormal; and

Optionally, the transceiver unit is further configured to:

after the source system is instructed to resend the current service data to the message middleware or the destination system, the current service data is sent to the destination system according to the abnormal information generated when the current service data from the second database is abnormal, so that the destination system determines whether the current service data in the abnormal information is the same as the service data associated with the current service data re-received by the destination system.

In a third aspect, a system for processing service data is provided, comprising an apparatus for processing service data according to the second aspect, and message middleware, wherein:

and the message middleware is used for acquiring the service data from the device for processing the service data and forwarding the service data to a target system.

In a fourth aspect, an apparatus for processing service data is provided, including:

A memory for storing program instructions;

And a processor for invoking program instructions stored in the memory and executing the method according to any of the first aspects in accordance with the obtained program instructions.

In a fifth aspect, there is provided a storage medium storing computer-executable instructions for causing a computer to perform the method of any one of the first aspects.

Drawings

Fig. 1 is an application scenario of a method for processing service data according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a message middleware according to an embodiment of the present application;

FIG. 3 is a diagram of an interaction process for processing service data according to an embodiment of the present application;

Fig. 4 is a schematic flow chart of a method for processing service data according to an embodiment of the present application;

Fig. 5 is a schematic structural diagram of a device for processing service data according to an embodiment of the present application;

Fig. 6 is a schematic diagram of a device for processing service data according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

In addition, in the embodiment of the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.

In order to improve the reliability of the current inter-system transmission service data, the present application provides a method for processing service data, please refer to fig. 1, which is an application scenario of the method for processing service data, and may also be represented as a device deployment diagram for processing service data. The application scenario includes a source system 101, a destination system 102, a listening device 103, a message middleware 104, a terminal device 105, a server 106, a first database 107, a second database 108, and a relay system 109.

Specifically, the source system 101 and the destination system 102 may perform transmission of service data through the listening device 103 and the message middleware 104. The monitoring device 103 determines that the source system 101 sends service data stored by a user to the message middleware 104, and if the monitoring device 103 determines that the first database 107 is abnormal in the process of processing the service data and the second database 108 is not abnormal in the process of processing the service data, the monitoring device 103 acquires operation information in the process of processing the service data by the second database 108. The monitoring device 103 controls the first database 107 to process the service data again according to the acquired operation information.

If the listening device 103 determines that the message middleware 104 did not successfully receive the service data, the listening device 103 acquires the service data stored in the first database 107 and transmits the service data acquired from the first database 107 to the message middleware 104. If the number of times of reception failure of the message middleware 104 reaches a preset number of times when the service data from the first database 107 is transmitted to the message middleware 104, the monitoring device 103 acquires the service data stored in the second database 108 and transmits the service data acquired from the second database 108 to the message middleware 104.

In the embodiment of the present application, the first database 107 is used to buffer currently transmitted service data, and when the message middleware 104 does not successfully receive the service data in the transmission process of the service data, it can be determined whether the processing procedure of the first database 107 is abnormal or the processing procedure of the second database 108 is abnormal. If the processing procedure of the first database 107 is abnormal and the processing procedure of the second database 108 is not abnormal, the processing procedure of the first database 107 can be re-performed according to the processing procedure of the second database 108, so that the abnormal condition of the first database is cleared, unnecessary rollback operation is not required to be performed on the second database, and the efficiency of processing the service data is improved.

Further, the monitoring device 103 may obtain the service data from the first database 107 and send the service data to the message middleware 104, if the service data is obtained from the first database 107, the service data is sent to the message middleware 104, and the number of times of receiving failure of the message middleware 104 reaches a preset number of times, the monitoring device 103 may obtain the service data from the second database 108 and send the service data to the message middleware 104, and in case that the service data is not obtained from the first database 107, the service data may be obtained from the second database 108, thereby improving reliability of transmitting the service data.

Where the source system 101 and the destination system 102 generally refer to systems for processing traffic, the source system refers to an end that transmits traffic data, and the destination system refers to an end that receives traffic data, but virtually any traffic system can be used as both a destination system and a source system. The source system may include a terminal device 105, a server 106, a first database 107, and a second database 108. The first database 107 and the second database 108 are used for storing data, and the first database 107 may be a non-relational database, for example, the first database 107 may be understood as an in-memory database 107 of the source system 101, such as a remote dictionary service (remote dictionary server, dis) database, and the second database 108 may be a relational database, for example, the second database 108 may be understood as a background database 107 of the source system 101. The destination system 102 may comprise a terminal device 105 and a server 106, and the destination system 102 may further comprise a first database 107 and a second database 108. The first database 107 and the second database 108 may be provided in the server 106 or may be implemented by separately provided storage devices.

The terminal device 105 may have a client for processing services, where the client is used to communicate with the server 106, and the client may be a web client, or a client installed in the terminal device 105, which is not limited specifically.

The listening device 103 may be implemented together with the message middleware 104 by the same entity device, for example, by the relay system 109 in fig. 1, or the listening device 103 may be implemented separately by another device, for example, by a server or a terminal device, which is not limited in particular.

Referring to fig. 2, a schematic structural diagram of a relay system 109 is shown, where the relay system 109 may be considered to be composed of at least one relay unit, where a relay unit may communicate through a switch 201, a route 202, and a memory 203 connected to the switch through the route, and the switch 201 and the memory 203 communicate through the route 202.

Based on the application scenario of fig. 1, a method for processing service data in the embodiment of the present application is described below.

Referring to fig. 3, a diagram of an interaction process of processing service data between a source system 101, a destination system 102, a message middleware 104, and a listening device 103 according to an embodiment of the present application is shown.

S301, the source system 101 stores the service data in the second database 108.

Specifically, the user may submit corresponding service data through the source system, which is equivalent to the source system 101 obtaining the service data, and the source system 101 writes the service data into the second database 108. The source system 101 writing the service data into the second database 108 may be that the source system 101 stores the service data into the second database 108, or may be that the source system 101 deletes the service data from the second database 108, or may be that the source system 101 updates the corresponding service data in the second database 108 with the service data, and the manner in which the source system 101 writes the service data into the second database 108 is not particularly limited. The service data that needs to be processed currently may also be referred to as current service data.

For example, the user submits a corresponding policy through the interface displayed by the terminal device 105 of the source system 101, which is equivalent to the source system 101 obtaining the policy, which is also regarded as business data.

S302, if it is determined that the second database 108 is abnormal, the second database 108 is controlled to roll back.

Specifically, for example, a service data format error, a service data type error, or a lock table may cause an abnormality in the process of writing service data into the second database 108, where when the monitoring device 103 monitors that the second database 108 is writing service data, the monitoring device 103 controls the second database 108 to roll back to a state before the source system 101 writes the service data, that is, the monitoring device 103 controls to clear an execution trace of writing the service data. The lock table may be understood as an abnormal situation generated when the database is not executing the next operation because the database is waiting for the previous operation, for example, when one user is operating a piece of data, if the user forgets to submit the data, the other user needs to modify the data. Since the last user is not submitted, another user modification operation is always in a waiting state, and thus the lock table condition may be caused.

For example, during the process of the source system 101 storing the policy of the user in the second database 108, the monitoring device 103 determines that the second database 108 is abnormal, at this time, the monitoring device 103 controls the second database 108 to return to the state before the policy of the user is stored in the second database 108.

In one possible embodiment, after listening device 103 determines that second database 108 is abnormal, and controls second database 108 to roll back, listening device 103 may instruct source system 101 to re-acquire traffic data.

Specifically, the monitoring device 103 sends indication information to the source system 101, which can be used to instruct the source system 101 to display prompt information to the user, where the prompt information is used to prompt the user that the operation is failed and needs to be performed again, and the specific prompt mode is not limited. After the user re-operates the source system 101, the source system 101 may re-acquire the service data, and the listening device 103 may re-perform S301.

S303, the source system 101 stores the service data to the first database 107.

In one possible embodiment, the source system 101 may need to convert the format of the service data into the write format of the first database 107 before storing the service data in the first database 107 in order for the destination system 102 to obtain and use the service data.

Specifically, the description will be given taking an example in which the writing format of the first database 107 is stored in the form of a key-value. The source system 101 may convert the service data into a file format required when the first database 107 writes the data, and generate a unique flag, where the unique flag is used to flag the sequence of the service data in the first database 107, the flag is used as a primary key, the converted file is a value corresponding to the primary key, and the converted file is stored in the first database 107 in a key-value form, and then other devices may acquire the file corresponding to the flag according to the flag, that is, acquire the specified service data.

For example, the user determines, on the source system (e.g., the client policy distribution system), a business relationship between the client's policy and the business person handling the policy, and the user clicks the submit button, and the client policy distribution system converts the business data into json format file and generates a unique ID tag, and writes the value corresponding to the json format file as the primary key into the first database 107 while saving the business data to the second database 108.

S304, if the listening device 103 determines that the first database 107 is abnormal, the second database 108 is controlled to roll back.

Specifically, during the process of writing the service data into the first database 107, the listening device 103 controls the second database 108 to roll back to the state before the source system 101 writes the service data in the second data 108, that is, clears the execution trace of the write operation, if the listening device 103 determines that the first database 107 is abnormal. The anomaly of the first database 107 may be an anomaly of the first database 107 caused by a network anomaly, an anomaly within the first database 107, or the like.

In one possible embodiment, after listening device 103 determines that first database 107 is abnormal and controls second database 108 to roll back, listening device 103 may instruct source system 101 to re-acquire traffic data.

Specifically, the monitoring device 103 may send indication information to the source system 101, and may be used to instruct the source system 101 to display prompt information to the user, where the prompt information is used to prompt the user that the operation is failed and that the operation needs to be performed again, and the specific prompting mode is not limited. After the user re-operates the source system 101, the source system 101 may re-acquire the service data, and the listening device 103 may write the service data into the second database 108 again, and re-perform S301 and S303.

S305, if the monitoring device 103 determines that an abnormality occurs in the process of processing the service data by the first database, and that no abnormality occurs in the process of processing the service data by the second database, the first database is controlled to process the service data again.

Specifically, if the monitoring device 103 determines that the first database 107 is abnormal during the process of processing the service data and the second database 108 is not abnormal during the process of processing the service data by the second database 108, the monitoring device 103 may obtain the operation information of the second database 108 for processing the service data, or the monitoring device 103 may obtain the operation information cached during the process of processing the service data by the first database 107, and according to the operation information, control the first database 107 to process the service data again. The operation information may include instruction information for processing service data, such as adding, deleting, modifying, and reading the service data, and may include processed service data, which is not particularly limited.

When the operation information is different, the content of controlling the first database 107 to process the service data again is also different according to the operation information, which will be described in detail below.

First kind:

When the operation information is an instruction to add service data, the processing of the service data may be controlling the first database 107 to store the service data.

Second kind:

when the operation information is an instruction to delete the service data, the processing of the service data may be controlling the first database 107 to delete the service data.

Third kind:

When the operation information is an instruction to modify the service data, the processing of the service data may be controlling the first database 107 to modify the service data.

Fourth kind:

when the operation information is an instruction to read the service data, the processing of the service data may be controlling the first database 107 to read the service data.

For example, in the process that the client policy distribution system stores the service data in the second database 108, no abnormality occurs in the second database 108, in the process that the client policy distribution system stores the service data in the first database 107, the first database 107 is abnormal, at this time, the monitoring device 103 determines that the first database 107 is abnormal, the second database 108 is not abnormal, the monitoring device 103 obtains a storage instruction of the second database 108 for the service data, and according to the storage instruction, controls the first database 107 to store the service data.

S306, the source system 101 sends service data to the message middleware 104.

Specifically, the source system 101 sends service data to the message middleware 104, and the switch 201 in the message middleware 104 receives the service data from the source system, and assigns different routes 202 for the received service data according to different service types, so as to allocate corresponding memories 203 for the service data, thereby implementing that the message middleware 104 receives the service data.

In a possible embodiment, the monitoring device 103 determines whether the message middleware 104 receives the service data successfully, and if the monitoring device 103 determines that the message middleware 104 receives the service data successfully, the monitoring device 103 controls the first database 107 to delete the stored service data.

S307, if the listening device 103 determines that the message middleware 104 receives the service data abnormally, the current service data is obtained from the first database 107, and the current service data is retransmitted to the message middleware 104.

Specifically, if the message middleware 104 receives the service data abnormally, the listening device 103 acquires the service data stored in the first database 107 and transmits the acquired service data to the message middleware 104. If the message middleware 104 receives the service data still abnormal, the monitoring device 103 continues to acquire the service data stored in the first database 107 and sends the service data to the message middleware 104, and the process is repeated until the abnormality reaches the preset number of times. The preset times may be a value preset according to an empirical value, or may be a user-defined value, which is not particularly limited.

The occurrence of an anomaly in receiving service data by the message middleware 104 may be that the message middleware 104 does not successfully write service data, or may be that the message middleware 104 is abnormal in the process of receiving service data, for example, the message middleware 104 is abnormal due to network anomaly or internal storage anomaly of the message middleware 104, so that the message middleware 104 does not receive service data.

S308, if the monitoring device 103 determines that the message middleware 104 re-receives the service data from the first database 107 for a preset number of times, the service data is acquired from the second database 108, and the service data is re-sent to the message middleware 104.

Specifically, if the number of times that the message middleware 104 receives the service data abnormally reaches the preset number of times, the monitoring apparatus 103 acquires the service data stored in the second database 108, and sends the acquired service data to the message middleware 104.

S309, if the monitoring device 103 determines that the message middleware 104 re-receives the traffic data from the first database 107 for a preset number of times, the monitoring device 103 stores the anomaly information into the second database 108.

The anomaly information may include service data of the anomaly, and may further include anomaly types of the anomaly, such as a first database 107 storage anomaly, a first database 107 network anomaly, a second database 108 data format anomaly, a second database 108 primary key collision anomaly, a second database 108 connection anomaly, or a message middleware 104 storage anomaly, and may further include the number of anomalies, and the anomaly information may further include other information, which is not particularly limited.

If the listening device 103 determines that the message middleware 104 re-receives the service data from the second database 108 and is abnormal in S310, the listening device 103 sends indication information to the source system 101, so as to instruct the source system 101 to send the service data to the message middleware 104 again, or instruct the source system 101 to send the service data to the destination system 102 directly.

Specifically, if the monitoring apparatus 103 determines that the message middleware 104 re-receives the traffic data from the second database 108 and an abnormality occurs, the monitoring apparatus 103 acquires abnormality information of the abnormality, and determines an abnormality cause of the occurrence of the abnormality according to the abnormality information. The cause of the abnormality includes two cases, which are specifically described below.

Case one:

listening device 103 determines that message middleware 104 re-receives traffic data from second database 108 that the anomaly cause of the anomaly is message middleware anomaly.

Specifically, if the listening device 103 determines that the cause of the occurrence of the abnormality is that the message middleware 104 is abnormal, the listening device 103 generates first indication information for instructing the source system 101 to transmit traffic data to the destination system 102 through communication components other than the message middleware 104, based on the abnormality information. The listening device 103 sends the first indication information to the source system 101, and the source system 101 receives the first indication information from the listening device 103. The source system 101 determines that the message middleware 104 has an abnormality in storage according to the first indication information, and cannot send service data to the destination system 102 through the message middleware 104, and then the source system 101 determines to send the service data directly to the destination system 102 through other communication components. Other communication components besides message middleware 104 may be communication interfaces, such as web interfaces, and the like.

For example, the first indication information includes the number of the abnormal information, for example 400 represents that the message middleware 104 is abnormal, 500 represents that the second database 108 is abnormal, after the source system 101 receives the first indication information from the monitoring device 103, the source system 101 determines the number of the abnormal information in the first indication information, and if the number is 400, the source system 101 determines that the message middleware 104 is abnormal; if numbered 500, the source system 101 determines that an anomaly has occurred in the second database 108.

And a second case:

listening device 103 determines that message middleware 104 re-receives traffic data from second database 108 that the exception occurred because second database 108 read-write exception.

Specifically, if the listening device 103 determines that the cause of the occurrence of the abnormality is that the second database 108 reads and writes the abnormality, the listening device 103 generates second indication information according to the abnormality information, the second indication information being used to instruct the source system 101 to rewrite the service data into the second database 108, and resend the service data to the message middleware 104. The second indication information may include indication information for indicating an abnormality of the second database 108, and may further include indication information for indicating a specific cause of the abnormality, such as an abnormality of the connection of the second database 108, an abnormality of a primary key collision of the second database 108, or an abnormality of a business data format.

After generating the second instruction information, the listening device 103 transmits the second instruction information to the source system 101, and the source system 101 receives the second instruction information from the listening device 103. The source system 101 determines that the second database 108 is abnormal according to the second indication information, and further determines the cause of the abnormality of the second database 108.

If the source system 101 determines that the cause of the anomaly in the second database 108 is an anomaly in the connection of the second database 108, then the source system 101 reestablishes the connection with the second database 108, and after the connection is successful, the service data is again written into the second database 108.

If the source system 101 determines that the cause of the abnormality of the second database 108 is an abnormality of the second database 108 primary key collision, then the source system 101 determines that the abnormal service data is currently stored, and in the second database 108, there is the same service data as the service data primary key, that is, primary key collision, then the source system 101 controls the second database 108 to delete the service data stored in the second database 108 that conflicts with the service data primary key that is currently abnormal, and rewrites the service data that is currently abnormal into the second database 108.

If the source system 101 determines that the reason for the abnormality of the second database 108 is that the data format of the service data is abnormal, the source system 101 determines that the data format of the service data in which the abnormality occurs is different from the data format required by the second database 108, and then the source system 101 adjusts the data format of the service data in which the abnormality occurs according to the storage format when the service data is stored in the second database 108, and rewrites the adjusted service data into the second database 108.

For example, the second indication information includes an abnormal information number of the second database 108, for example 501 represents an abnormal connection of the second database 108, 502 represents an abnormal collision of the primary key of the second database 108 or 503 represents an abnormal format of the service data, after the source system 101 receives the second indication information from the monitoring device 103, the source system 101 determines the abnormal information number in the second indication information, and if the number is 501, the source system 101 determines that the second database 108 is abnormal; if numbered 502, the source system 101 determines that the primary key conflict of the second database 108 is abnormal; if numbered 503, the source system 101 determines that the traffic data format is abnormal.

S311, if the listening device 103 determines that the message middleware 104 receives the service data successfully, the first database 107 is controlled to delete the stored service data.

S312, the destination system 102 obtains the service data from the message middleware 104.

Specifically, message broker 104 receives a request from destination system 102 to obtain traffic data, which may include a specified route 202, message broker 104 may determine memory 203 based on the specified route 202, thereby determining traffic data in memory 203, message broker 104 sends the traffic data to destination system 102, and destination system 102 receives the traffic data from message broker 104. Message middleware 104 may actively send traffic data to destination system 102 after obtaining new traffic data, or destination system 102 may periodically query message middleware 104 to obtain new traffic data.

S313, if the monitoring device 103 determines that the destination system 102 receives the service data successfully, the destination system 102 sends the indication information for indicating that the service data is received successfully to the message middleware 104.

S314, if the monitoring device 103 determines that the message middleware 104 successfully sends the service data, the monitoring device 103 controls the message middleware 104 to delete the stored service data.

Specifically, the message middleware 104 receives the indication information from the destination system 102, so that the monitoring device 103 determines that the destination system 102 successfully acquires the service data from the message middleware 104. After the listening device 103 determines that the message middleware 104 successfully transmits the service data, the message middleware 104 deletes the stored service data.

In one possible embodiment, if listening device 103 determines that message middleware 104 failed to send the traffic data, destination system 102 continues to obtain the traffic data from message middleware 104.

Specifically, if the message middleware 104 receives not the information indicating that the service data is successfully received from the destination system 102 but the request for acquiring the service data from the destination system 102 after sending the service data to the destination system 102, the monitoring apparatus 103 determines that the message middleware 104 fails to send the service data, and the message middleware 104 continues to send the service data to the destination system 102.

In one possible embodiment, if the number of times the message middleware 104 fails to send the service data reaches the preset number of times, the failure information is stored in the second database 108 of the destination system 102, where the failure information may include the service data that fails to send, and may also include a cause of the abnormality in sending, such as a network abnormality or a hardware device abnormality, etc.

S315, the destination system 102 receives the service data corresponding to the anomaly information from the source system 101, and determines whether the service data corresponding to the anomaly information is consistent with the service data received by the destination system 102 again.

Specifically, to further improve the reliability of data transmission, if the message middleware 104 receives the service data abnormally, after the monitoring module 103 determines that the destination system 102 receives the service data successfully, it may further determine whether the service data received by the destination system 102 is correct. If the message middleware 104 receives the service data from the second database 108 again and is abnormal, the monitoring module 103 obtains the corresponding abnormal information, and then after determining that the destination system 102 receives the service data again successfully, the monitoring module 103 may convert the service data into a preset transmission format, such as a file format specified by a file transfer protocol (FILE TRANSFER protocol, FTP), or a JSON file format, and send a file corresponding to the abnormal information to the destination system 102, and after receiving the file corresponding to the abnormal information, the destination system 102 may parse the received file according to the preset transmission format to obtain the abnormal information. The destination system 102 may determine whether the traffic data corresponding to the anomaly information is consistent with the traffic data received by the destination system 102 again. The anomaly information may include service data processed when an anomaly occurs, and may also include time of service data submission, and the like. The anomaly information may be directly transmitted to the destination system 102 by the listening module 103, or the source system 101 may be notified by the listening module 103, and transmitted to the destination system 102 by the source system 101, with no limitation on the specific transmission manner.

Whether the service data corresponding to the abnormal information is consistent with the service data successfully re-received by the destination system 102 or not may be that each item of data in the service data is consistent, if there is an inconsistent item of data, the service data corresponding to the abnormal information is considered to be inconsistent with the service data successfully re-received by the destination system 102.

For example, in the insurance industry, since the policy service transaction data involves promotion and assessment and salary calculation of a policy agent (i.e., a salesman), secondary development of customer resources of the policy agent or an enterprise, and the like, the importance of the service data is high, and thus, in the process of transmitting the service data, it is necessary to ensure the reliability of data transmission. When the interception module 103 determines that the traffic data is abnormal during the storage of the second database 108 or the message middleware 104 or during the data transmission between the second database 108 and the message middleware 104, the interception module 103 may further determine whether the received traffic data is accurate after determining that the destination system 102 successfully receives the traffic data. The monitoring module 103 sends the service data corresponding to the abnormal information to the destination system 102, and the destination system 102 determines whether the service data corresponding to the abnormal information is consistent with the re-received service data.

Taking service data as policy service transaction data, wherein the service data comprises three data of a customer name, a policy service handled by the customer and a policy agent name as an example, if the policy service transaction data corresponding to abnormal information is consistent with the policy service transaction data which is re-received successfully by the destination system 102, the policy service transaction data corresponding to the abnormal information can be considered to be consistent with the policy service transaction data which is re-received successfully by the destination system 102; if there is a data inconsistency, such as a different name of the policy agent, the policy service transaction data corresponding to the exception information is deemed inconsistent with the policy service transaction data that was re-received successfully by the destination system 102.

In a possible embodiment, the anomaly information may include a time of submitting the service data, and then the monitoring device 103 may send, to the destination system 102, anomaly information of the service data that is abnormal in the service data submitted in the period, with a preset period of time as a period, and after the destination system 102 receives the service data that is abnormal in the period, query, in a local storage of the destination system 102, for example, in the second database 108 of the destination system 102, the corresponding service data in the period, and compare each service data separately to determine whether there is inconsistent service data.

Each service data and the corresponding service data submitting time can be converted into an FTP file or a JSON file in a preset transmission format, or each service data and the corresponding service data submitting time can be converted into one file data in an FTP file or a JSON file in a preset transmission format. The preset transmission format is, for example, a first behavior service data submitting time of the FTP file, service data corresponding to the second behavior service data submitting time, or the above period of the first behavior of the FTP file, service data corresponding to the period of the second behavior, and a submitting time of the service data for each service data suffix, etc.

In one possible embodiment, if the service data corresponding to the anomaly information is consistent with the service data successfully re-received by the destination system 102, the destination system 102 does not further process the service data successfully re-received.

Specifically, if the service data corresponding to the anomaly information is consistent with the service data successfully re-received by the destination system 102, the destination system 102 does not further process the service data successfully re-received. The destination system 102 may send a prompt message to the source system 101 to prompt the source system 101 that the exception information has been processed, and after receiving the prompt message, the source system 101 may mark the corresponding exception information stored in the second database 108 to indicate that the exception information has been processed.

S316, if the service data corresponding to the anomaly information is inconsistent with the service data received by the destination system 102 again, the destination system 102 updates the service data received again successfully according to the service data in the anomaly information.

Specifically, if the service data corresponding to the anomaly information is inconsistent with the service data that is successfully re-received by the destination system 102, the destination system 102 deletes the service data that is successfully re-received and is corresponding to the service data in the anomaly information stored locally, and stores the service data in the anomaly information, so as to ensure that the service data corresponding to the anomaly information is consistent with the service data that is successfully re-received by the destination system 102. The local storage is, for example, a second database 108 of the destination system 102, or the like.

As an example, S301 to S304, S309 to S316 are optional parts.

As an example, S304 and S305 may perform only one of the steps.

In order to more clearly illustrate the process of processing service data in fig. 3, a method for processing service data in the embodiment of the present application is specifically illustrated below.

Referring to fig. 4, a flow chart for processing service data is shown. In the embodiment of the present application, the source system 101 is taken as a customer policy distribution system in the insurance industry, the destination system 102 is taken as a service operator checking system, and the service data is taken as an example of the distribution relationship between the customer policy and the service operator processing the policy.

The user determines the business relation between the client's policy and the business personnel processing the policy in the interface of the client's policy distribution system, which is equivalent to the client's policy distribution system obtaining business data.

S401, the client policy distribution system stores the business relationship as business data in the first database 107 and the second database 108 of the client policy distribution system, respectively.

The client policy distribution system sends the business data to the message broker 104S 402.

S403, the service operator checking system acquires service data.

Specifically, when service data is needed to be used, the service data is acquired from the message middleware 104 by the service operator checking system, and the service operator checking system can check the service operator according to the acquired service data.

S404, the listening device 103 determines whether the sending of the service data to the message middleware 104 is successful.

Specifically, during the process of sending the service data to the message middleware 104 by the client policy distribution system, the monitoring device 103 determines whether the service data is sent successfully.

If the listening device 103 determines that the service data is successfully sent to the message middleware 104, the listening module determines that the first database 107 deletes the stored service data S405.

S406, if the listening device 103 determines that the service data is sent abnormally to the message middleware 104, the listening device 103 acquires the service data from the first database 107 and sends the service data to the message middleware 104.

S407, if the monitoring apparatus 103 determines that the traffic data transmission abnormality from the first database 107 reaches the preset number of times, the monitoring apparatus 103 acquires the traffic data from the second database 108 and transmits the traffic data to the message middleware 104.

S408, if the monitoring apparatus 103 determines that the number of times of transmission abnormality of the traffic data reaches the preset number of times, the monitoring apparatus 103 stores the abnormality information into the second database 108.

S409, if the listening device 103 determines that the traffic data from the second database 108 is abnormally transmitted, the listening module 103 instructs the source system 101 to retransmit the traffic data to the message middleware 104 or the destination system 102.

If the message middleware 104 sends service data to the service assessment system successfully, the listening device 103 determines that the message middleware 104 deletes the stored service data S410.

S411, if the destination system 102 successfully receives the service data, the source system 101 sends exception information to the destination system 102, and the destination system 102 determines, according to the received exception information, whether the service data in the exception information is consistent with the service data successfully received by the destination system.

S412, if the service data in the anomaly information is inconsistent with the service data successfully received by the destination system, the destination system 102 uses the service data in the anomaly information to cover the service data successfully received by the destination system.

S413, if the message middleware 104 transmits the service data to the service assessment system as abnormal, the monitoring apparatus 103 determines that the message middleware 104 retransmits the service data, and stores the abnormal information into the background database of the service assessment system.

Based on the same inventive concept, the embodiment of the application provides a device for processing service data, which can realize the functions corresponding to the method for processing service data. This device corresponds to the listening device 103 discussed previously. Referring to fig. 5, the apparatus includes a first processing unit 501, a second processing unit 502, and a transceiver unit 503, where:

The first processing unit 501 is configured to, if it is determined that the first database is abnormal in the process of processing the current service data, and it is determined that the second database is not abnormal in the process of processing the current service data, obtain operation information in the process of processing the second database with respect to the current service data, and control the first database to process the current service data again according to the operation information;

A second processing unit 502, configured to acquire current service data from the first database if it is determined that the message middleware receives the current service data and is abnormal; wherein, all service data which is not confirmed to be successfully transmitted are cached in the first database;

a transceiver unit 503, configured to resend current service data to the message middleware;

The second processing unit 502 is further configured to obtain the current service data from the second database if it is determined that the number of times that the message middleware re-receives the current service data from the first database is abnormal reaches a preset number of times; wherein the second database stores all service data generated by the active system;

The transceiving unit 503 is further configured to retransmit the current service data to the message middleware.

In a possible embodiment, the first processing unit 501 is further configured to:

after retransmitting the current service data to the message middleware, if it is determined that the message middleware receives the current service data transmitted by the source system, deleting the current service data stored in the first database.

Before the current service data is obtained from the second database, if the first database is abnormal in the process of writing the current service data by the source system or the second database is abnormal in the process of writing the current service data by the source system, the second database is controlled to roll back to a state before the current service data is written, and the source system is instructed to reacquire the current service data.

In a possible embodiment, the second processing unit 502 is further configured to: after obtaining the current service data from the second database and retransmitting the current service data to the message middleware, determining that the message middleware receives the current service data from the second database again and is abnormal;

the transceiving unit 503 is further configured to: the source system is instructed to retransmit the current traffic data to the message middleware or destination system.

In a possible embodiment, the transceiver unit 503 is specifically configured to:

If the message middleware is determined to re-receive the abnormality cause of the abnormality of the current service data from the second database, wherein the abnormality cause is the abnormality of the message middleware, the first indication information is sent to the source system, so that the source system sends the current service data to the destination system through a communication component except the message middleware; the first indication information is used for indicating that the message middleware is abnormal; and

If the message middleware is determined to re-receive the abnormality cause of the abnormality of the current service data from the second database as the read-write abnormality of the second database, sending second indication information to the source system so that the source system re-writes the current service data into the second database and re-sends the current service data to the message middleware; the second indication information is used for indicating that the second database is abnormal.

In a possible embodiment, the transceiving unit 503 is further configured to:

After the source system is instructed to resend the current service data to the message middleware or the destination system, the current service data is sent to the destination system according to the abnormal information generated when the current service data from the second database is abnormal, so that the destination system determines whether the current service data in the abnormal information is the same as the service data associated with the current service data received again by the destination system.

Based on the same inventive concept, an embodiment of the present application provides a system for processing service data, which is equivalent to the transit system 109 discussed above, and includes the device for processing service data and the message middleware 104 discussed above, where:

message middleware 104 for obtaining service data from the device for processing service data and forwarding the service data to the destination system.

In one possible embodiment, the system further comprises:

the message middleware comprises a plurality of transfer units, wherein each transfer unit is used for forwarding service data of a preset type; wherein, the service data types forwarded by different transfer units are different.

Specifically, during the transmission process of the source system 101 and the destination system 102, service data in the transmission process of the source system 101 and the destination system 102 can be divided into several types, when the destination system 102 sends the service data, corresponding transfer units can be designated, and different transfer units process the service data between the source system 101 and the destination system 102, so as to implement a multithreading concurrent mode to process the service data, and improve the processing efficiency of the service data.

Based on the same inventive concept, an embodiment of the present application provides an apparatus for processing service data, which is equivalent to the listening apparatus 103 discussed above, please refer to fig. 6, and the apparatus includes:

At least one processor 601, and a memory 602 connected to the at least one processor 601, a specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application, and in fig. 6, the processor 601 and the memory 602 are connected through a bus 600 as an example. Bus 600 is shown in bold lines in fig. 6, and the manner in which the other components are connected is illustrated schematically and not by way of limitation. The bus 600 may be divided into an address bus, a data bus, a control bus, etc., and is represented by only one thick line in fig. 6 for convenience of representation, but does not represent only one bus or one type of bus. Alternatively, the processor 601 may be referred to as a controller 601, and the names are not limited.

In the embodiment of the present application, the memory 602 stores instructions executable by the at least one processor 601, and the at least one processor 601 may perform the method for processing service data as described above by executing the instructions stored in the memory 602. The processor 601 may implement the functions of the respective modules in the apparatus shown in fig. 5.

The processor 601 is a control center of the device, and can connect various parts of the whole device by various interfaces and lines, and various functions of the device and process data by running or executing instructions stored in the memory 602 and calling data stored in the memory 602.

In one possible embodiment, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor primarily processes operating systems, user interfaces, application programs, and the like, and the modem processor primarily processes wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 601. In some embodiments, processor 601 and memory 602 may be implemented on the same chip, or they may be implemented separately on separate chips in some embodiments.

The processor 601 may be a general-purpose processor, such as a Central Processing Unit (CPU), a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic blocks disclosed in the embodiments of the present application.

The memory 602 is a non-volatile computer readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The Memory 602 may include at least one type of storage medium, which may include, for example, flash Memory, hard disk, multimedia card, card Memory, random access Memory (Random Access Memory, RAM), static random access Memory (Static Random Access Memory, SRAM), programmable Read-Only Memory (Programmable Read Only Memory, PROM), read-Only Memory (ROM), charged erasable programmable Read-Only Memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory, EEPROM), magnetic Memory, magnetic disk, optical disk, and the like. Memory 602 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 602 in embodiments of the present application may also be circuitry or any other device capable of performing storage functions for storing program instructions and/or data.

Based on the same inventive concept, embodiments of the present application also provide a storage medium storing computer instructions that, when run on a computer, cause the computer to perform the method of processing traffic data as previously discussed.

In some possible embodiments, aspects of the method of controlling an apparatus provided by the present application may also be implemented in the form of a program product comprising program code for causing an apparatus to carry out the steps in the method of processing traffic data according to the various exemplary embodiments of the application as described in the present specification when the program product is run on the apparatus.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A method of processing traffic data, comprising:

If it is determined that the first database is abnormal in the process of processing the current service data and it is determined that the second database is not abnormal in the process of processing the current service data, obtaining operation information cached in the first database when the first database is abnormal in the process of processing the current service data, and controlling the first database to process the current service data again according to the operation information, wherein the operation information comprises instruction information for processing the service data;

If the number of times that the message middleware re-receives the current service data from the first database and is abnormal reaches the preset number of times, acquiring the current service data from the second database, and re-sending the current service data to the message middleware; wherein the second database stores all service data generated by the active system.

2. The method of claim 1, comprising, after retransmitting the current service data to the message middleware:

3. The method of claim 1, comprising, prior to retrieving the current business data from a second database:

4. The method of claim 1, wherein after retrieving the current service data from a second database and resending the current service data to the message broker, comprising:

5. The method of claim 4, wherein if it is determined that the message middleware re-receives the current traffic data from the second database for an exception, instructing a source system to re-send the current traffic data to the message middleware or a destination system comprises:

6. The method according to claim 4 or 5, characterized in that after instructing a source system to resend current traffic data to the message middleware or destination system, it comprises:

7. An apparatus for processing traffic data, comprising:

The first processing unit is used for acquiring operation information cached when the first database is abnormal in the process of processing the current service data according to the operation information if the first database is abnormal in the process of processing the current service data and the second database is not abnormal in the process of processing the current service data, and controlling the first database to process the current service data again according to the operation information, wherein the operation information comprises instruction information for processing the service data;

The second processing unit is further configured to acquire the current service data from the second database if it is determined that the number of times that the message middleware re-receives the current service data from the first database and that the abnormality occurs reaches a preset number of times; wherein the second database stores all service data generated by the active system;

8. A system for processing traffic data, comprising the apparatus for processing traffic data according to claim 7, and message middleware; wherein:

9. An apparatus for processing traffic data, comprising:

A memory for storing program instructions;

A processor for invoking program instructions stored in said memory and for performing the method according to any of claims 1-6 in accordance with the obtained program instructions.

10. A storage medium storing computer-executable instructions for causing a computer to perform the method of any one of claims 1 to 6.