CN113886500A - A data processing method, device, server and storage medium - Google Patents
A data processing method, device, server and storage medium Download PDFInfo
- Publication number
- CN113886500A CN113886500A CN202111223304.0A CN202111223304A CN113886500A CN 113886500 A CN113886500 A CN 113886500A CN 202111223304 A CN202111223304 A CN 202111223304A CN 113886500 A CN113886500 A CN 113886500A
- Authority
- CN
- China
- Prior art keywords
- database
- data information
- data
- read
- written
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 99
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000004590 computer program Methods 0.000 claims description 5
- 230000003139 buffering effect Effects 0.000 claims 1
- 238000004891 communication Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 230000001360 synchronised effect Effects 0.000 description 9
- 230000009286 beneficial effect Effects 0.000 description 8
- 230000004044 response Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure relates to a data processing method, a data processing device, a server and a storage medium, and relates to the technical field of data processing. The embodiment of the disclosure at least solves the problem that in the related art, when the master database fails, the data consistency of the master database and the read databases and the performance of the master database cannot be simultaneously maintained. The method comprises the following steps: receiving a write request message, wherein the write request message is used for requesting to write first data information; writing first data information into a master database and at least one first database; the first database is used for caching data information which is requested to be written into the main database; and if the main database fails and the first data information is not successfully written into the main database, writing the first data information into a second database, and adding the first data information which is not successfully written into at least one first database into a preset list.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method, an apparatus, a server, and a storage medium.
Background
In a payment scene, after a user successfully pays, state data such as commodity inventory, account balance, order details and the like need to be timely and accurately stored, wherein consistency of the stored data in each database corresponding to the state data is ensured, and the state data is a storage target of the state data.
In the related art, two databases, a main database and a read database, are provided to hold status type data. If the main database is not in fault (normal state), the main database is responsible for reading and writing, and the read database is responsible for backing up the data of the main database. If the main database fails, the read-write request can be switched to the read database so as to ensure that the read-write request can be executed, thereby realizing high availability of the database.
However, if the read database replicates the data of the master database by using the asynchronous replication technology, under the condition that the master database fails, due to the delay of data synchronization, the data of the master database and the data of the read database may be inconsistent, and switching the read-write request to the read database will cause the read-write service to be affected. If the read database replicates the data of the main database by using the synchronous replication technology, the consistency of the main database and the data of the read database can be ensured, and the read-write service of the read database can be normally performed under the condition of the failure of the main database, but the performance of the main database can be seriously reduced.
Disclosure of Invention
The present disclosure provides a data processing method, apparatus, server, and storage medium to at least solve the problem in the related art that, when a master database fails, maintaining data consistency of the master database and a read database, and performance of the master database cannot be simultaneously achieved. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, there is provided a data processing method, including: receiving a write request message, wherein the write request message is used for requesting to write first data information; writing first data information into a master database and at least one first database; the first database is used for caching data information which is requested to be written into the main database; and if the main database fails and the first data information is not successfully written into the main database, writing the first data information into a second database, and adding the first data information which is not successfully written into at least one first database into a preset list.
Optionally, the data processing method further includes: receiving a read request message, wherein the read request message is used for requesting to read second data information; if the main database fails and the preset list does not comprise the second data information, whether the second database comprises the second data information is inquired; if the second database does not comprise the second data information, inquiring whether the first database comprises the second data information; and if the first database does not comprise the second data information, inquiring whether the read database comprises the second data information or not, wherein the read database is used for backing up the data information in the main database.
Optionally, the data processing method further includes: after the fault of the main database is relieved, writing the data information in the second database into the main database; and backing up the data information written into the second database in the main database to the read database.
Optionally, the data processing method further includes: and deleting the data information which is successfully written into the main database in the second database.
Optionally, the data processing method further includes: after receiving the write request message, if the first data information is successfully written into the main database, backing up the first data information in the main database to the read database.
Optionally, the data processing method further includes: and after receiving the read request message, if the master database is normal, querying second data information from the master database.
Optionally, the "writing the first data information into the at least one first database" includes: pulling first data information requested to be written by the write request message to at least one first database through the message processing component; the "adding the first data information that is not successfully written into the at least one first database into the preset list" includes: and determining that the first data information which is not successfully pulled to the at least one first database is obtained through the message processing component, and adding the determined first data information into a preset list.
According to a second aspect of the embodiments of the present disclosure, there is provided a data processing apparatus including: a receiving unit, a writing unit and a processing unit; the device comprises a receiving unit, a sending unit and a receiving unit, wherein the receiving unit is used for receiving a write request message which is used for requesting to write first data information; the writing unit is used for writing the first data information into the main database and the at least one first database; the first database is used for caching data information which is requested to be written into the main database; and the processing unit is used for writing the first data information into the second database and adding the first data information which is not successfully written into at least one first database into a preset list if the main database fails and the first data information is not successfully written into the main database.
Optionally, the apparatus further comprises: a sending unit and a query unit; the receiving unit is further used for receiving a reading request message, and the reading request message is used for requesting to read the second data information; the query unit is used for querying whether the second database comprises the second data information or not if the main database fails and the preset list does not comprise the second data information; the query unit is further used for querying whether the first database comprises the second data information or not if the second database does not comprise the second data information; and the query unit is further used for querying whether the read database comprises the second data information or not if the first database does not comprise the second data information, and the read database is used for backing up the data information in the main database.
Optionally, the apparatus further comprises: a backup unit; the writing unit is also used for writing the data information in the second database into the main database after the fault of the main database is relieved; and the backup unit is used for backing up the data information written into the second database in the main database to the read database.
Optionally, the apparatus further comprises: a deletion unit; and the deleting unit is also used for deleting the data information which is successfully written into the main database in the second database.
Optionally, the apparatus further comprises: and the backup unit is further used for backing up the first data information in the main database to the read database if the first data information is successfully written into the main database after the write request message is received.
Optionally, the apparatus further comprises: and the query unit is further used for querying the second data information from the master database after the read request message is received and if the master database is normal.
Optionally, the processing unit is specifically configured to: pulling first data information requested to be written by the write request message to at least one first database through the message processing component; and determining that the first data information which is not successfully pulled to the at least one first database is obtained through the message processing component, and adding the determined first data information into a preset list.
According to a third aspect of the embodiments of the present disclosure, there is provided a server, including: a processor, a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data processing method as provided by the first aspect and any one of its possible design forms.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, wherein instructions of the computer-readable storage medium, when executed by a processor of a server, enable the server to perform the data processing method as provided by the first aspect and any one of its possible designs.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising computer instructions which, when run on a server, cause the server to perform the data processing method as provided by the first aspect and any one of its possible designs.
The technical scheme provided by the disclosure at least brings the following beneficial effects: and after receiving the write request message, writing first data information requested to be written by the write request message into the main database and the at least one first database, if the main database fails and the first data information is not successfully written into the main database, writing the first data information into the second database, and adding the first data information which is not successfully written into the at least one first database into a preset list. Therefore, if the main database fails, the data stored in at least one of the first database and the second database can store the data which is not synchronized in the main database to the read database, so that the written data is not lost under the condition that the main database fails, the first database, the second database and the like can normally respond to the read-write request, and the read-write service is not influenced. Since the data requested to be written is not lost even in the event of a failure of the master database, the data in the master database does not have to be synchronized to the read database in real time, but may be asynchronously or semi-synchronized to the read database, thereby not affecting the performance of the master database.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
FIG. 1 is a block diagram illustrating a data processing system in accordance with an exemplary embodiment;
FIG. 2 is one of the flow diagrams of a data processing method according to an exemplary embodiment;
FIG. 3 is a second flowchart illustrating a data processing method according to an exemplary embodiment;
FIG. 4 is a third flowchart illustration of a method of data processing according to an exemplary embodiment;
FIG. 5 is one of the object relationship diagrams of a method of data processing shown in accordance with an exemplary embodiment;
FIG. 6 is a second schematic diagram illustrating object relationships for a method of data processing according to an exemplary embodiment;
FIG. 7 is a third diagram illustrating object relationships for a method of data processing in accordance with an illustrative embodiment;
FIG. 8 is a block diagram illustrating a data processing apparatus in accordance with an exemplary embodiment;
fig. 9 is a schematic diagram illustrating a configuration of a server according to an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
In addition, in the description of the embodiments of the present disclosure, "/" indicates an OR meaning, for example, A/B may indicate A or B, unless otherwise specified. "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present disclosure, "a plurality" means two or more than two.
The data processing method provided by the embodiment of the disclosure can be applied to a data processing system. Fig. 1 shows a schematic representation of a configuration of the data processing system. As shown in fig. 1, a data processing system 10 is used to solve the problem in the related art how to achieve data consistency after switching between a primary database and a backup database when the primary database fails without affecting the performance of the primary database. The storage system 10 includes a data processing apparatus 11 and a server 12. The data processing device 11 is connected to a server 12. The data processing device 11 and the server 12 may be connected by a wired manner or a wireless manner, which is not limited in the embodiment of the present disclosure.
The data processing apparatus 11 may be configured to perform data interaction with the server 12, for example, the data processing apparatus 11 may receive a write request message from the server 12, where the write request message is used to write the first data information. The databases, such as the first database, the second database, the main database, the preset list, etc., may be integrated into the server 12, or may be provided separately from the server 12. For example, the data processing apparatus 11 may write the first data information to a first database in the server 12. Alternatively, databases such as the first database, the second database, the master database, the preset list, and the like may be integrated in the data processing apparatus 11, or may be provided separately from the data processing apparatus 11. Alternatively, the data processing device 11 may be integrated in the server 12. Alternatively, the data processing device 11 may be other electronic devices than the server 12, such as a client.
The data processing apparatus 11 may also perform the data processing method in the embodiments of the present disclosure, for example, receive a write request message, the write request message being used to request writing of the first data information; writing first data information into a master database and at least one first database; the first database is used for caching data information which is requested to be written into the main database; and if the main database fails and the first data information is not successfully written into the main database, writing the first data information into a second database, and adding the first data information which is not successfully written into at least one first database into a preset list.
It should be noted that the first data information related to the embodiment of the present disclosure may include types such as video, audio, graphics, web page links, or text, for example, the first data information may be status data such as a commodity inventory, an account balance, and order details, and the embodiment of the present disclosure is not limited specifically here.
It should be noted that the data processing device 11 and the server 12 may be independent devices or may be integrated into the same device, and the present invention is not limited to this.
When the data processing device 11 and the server 12 are integrated in the same device, the communication mode between the data processing device 11 and the server 12 is communication between internal modules of the device. In this case, the communication flow between the two is the same as "the communication flow between the data processing apparatus 11 and the server 12 is independent of each other".
In the following embodiments provided by the present invention, the present invention is described by taking an example in which the data processing apparatus 11 and the server 12 are provided independently of each other.
In practical applications, the data processing method provided by the embodiment of the present invention may be applied to a data processing apparatus, and may also be applied to a server.
As shown in fig. 2, the data processing method provided by the embodiment of the present disclosure includes the following steps 201 to 203.
In the embodiment of the present disclosure, the write request message is used to request to write the first data information. For example, the first data information may be account balance information, and the write request message is used to request to write the account balance information. It is understood that the write request message also includes a data source, a transmission path, check data, a file size, and the like. The write request message carries the specific information in a preset message sending format.
In the embodiment of the present disclosure, the at least one first database means that the number of first databases is greater than or equal to 1. It should be noted that, the data processing apparatus further includes a message center and a message processing component, after receiving the write request message, the write request message (also may be understood as a transaction message) is delivered to the message center, and then the message processing component pulls the write request message from the message center, and writes the first data information in the write request message into the at least one first database. It is understood that each of the at least one first database successfully writes the write request message, indicating that the write request message was successfully consumed.
It should be noted that the first database has a limited storage space, and generally, data stored for a long time is periodically cleared, so that the first database is used for backing up a write request message received in the near future. When the number of the first databases is larger than 1, the first data information can be backed up for multiple times to ensure that the data is not lost.
Illustratively, the first database may be a remote dictionary server (Redis) cache database.
Optionally, in this embodiment of the present disclosure, the data processing apparatus further includes: and searching whether the main database has a fault or not according to a preset period. It should be noted that, the data processing apparatus may include an interface detection component, and whether the master database fails is detected by using an interface detection method.
In the embodiment of the present disclosure, the preset list is used to store the first data information that is not successfully written into the at least one first database. The data information which is not successfully written into the at least one first database can be understood as data information which is not successfully consumed, and the data information which is successfully written into the at least one first database can be understood as data information which is successfully consumed.
The data processing device may receive a plurality of write request information, and for first data information corresponding to each write request information, if the first data information is not successfully written into the at least one first database, the first data information is placed in a preset list. In this way, the plurality of data information corresponding to the plurality of write request information and not successfully written into the at least one first database are all put into the preset list.
In the embodiment of the present disclosure, the data processing apparatus may further receive other request messages before receiving the write request message and after receiving the write request message, and in writing the write request message and the other request messages into the first database, the write request message and the other request messages need to be processed step by step according to an actual device operating state and a network state, and thus, in the case where the master database fails, there may be first data information that is not written into the first database.
In the embodiment of the present disclosure, the second database is a failover (failover) database, and is enabled only when the primary database fails to transmit data, so as to temporarily store data that the primary database cannot temporarily store. It will be appreciated that after the failure of the primary database is resolved, the data stored in the secondary database is progressively switched back into the primary database.
In the embodiment of the present disclosure, in case that the primary database does not fail, the data processing apparatus controls the second database not to accept any request message, and the second database is an empty database, wherein the request message may include but is not limited to a read request message, a write request message, a delete request message, a sort request message, and a deduplication request message.
Optionally, after receiving the write request message, if the first data information is successfully written into the master database, the first data information in the master database is backed up to the read database.
In the embodiment of the present disclosure, if the master database is normal, the first data information can be successfully written into the master database.
In the embodiment of the present disclosure, if the master database is normal, the data information in the master database is continuously backed up to the read database. After the first data information is successfully written into the main database, the first data information is also backed up to the read database.
The technical scheme provided by the disclosure at least brings the following beneficial effects: when the master database is in a normal state without failure, data is written into the master database, so that the efficiency of processing write request messages is improved, and data backup (also referred to as data synchronization) is performed in an asynchronous or semi-synchronous mode to ensure that the data consistency between the read database and the master database can be maintained.
In the embodiment of the disclosure, when the master database fails, the first data information corresponding to the write request message outside the preset list is already written into the master database, for the first data information, which has been written into the at least one first database, it may be stored only in the master database, if the read database has synchronized the first data information then the first data information is stored in both the master database and the read database, if the read database does not synchronize the first data information then it is stored in either the first database or the second database, namely, the read database, the first database and the second database can store the full amount of data, so that the request information can be responded in the first database, the second database and the read database after the main database fails, the smooth switching of the databases is realized, and meanwhile, the condition that the data is inconsistent or lost is avoided. Therefore, at least one first database and at least one second database are added in addition to the master database and the read database, and the first data information requested to be written by the write request message can be saved in real time regardless of whether the master database fails or not.
It should be noted that, because at least one first database can store part of the first data information, when the data synchronization is performed on the master database by the read database, an asynchronous synchronization mode can be adopted, so that the data synchronization process and the process of processing the write request message do not occupy network resources, thereby improving the efficiency of processing the write request message, i.e. ensuring that the performance of the master database is not affected; a semi-synchronous mode can be adopted, the success rate of data synchronization is ensured, and meanwhile, a certain synchronous delay exists, so that the success rate of data synchronization and the synchronous delay are balanced, and the efficiency of processing the write request message is improved.
The technical scheme provided by the disclosure at least brings the following beneficial effects: and after receiving the write request message, writing first data information requested to be written by the write request message into the main database and the at least one first database, if the main database fails and the first data information is not successfully written into the main database, writing the first data information into the second database, and adding the first data information which is not successfully written into the at least one first database into a preset list. Therefore, if the main database fails, the data stored in at least one of the first database and the second database can store the data which is not synchronized in the main database to the read database, so that the written data is not lost under the condition that the main database fails, the first database, the second database and the like can normally respond to the read-write request, and the read-write service is not influenced. Since the data requested to be written is not lost even in the event of a failure of the master database, the data in the master database does not have to be synchronized to the read database in real time, but may be asynchronously or semi-synchronized to the read database, thereby not affecting the performance of the master database.
Optionally, the data processing method provided in the embodiments of the present disclosure may further be capable of receiving and processing a read request message, and specifically includes, after receiving the read request message, if the master database is normal, querying the second data information from the master database.
In the embodiment of the present disclosure, if the master database is normal, it indicates that the master database stores all written data, and therefore, the second data information can be obtained by querying from the master database.
The technical scheme provided by the disclosure at least brings the following beneficial effects: in the normal state that the main database has no fault, the main database comprises all data, so that the second data information can be directly inquired from the main database without switching the database, the response time of switching the database can be shortened, and the inquiry speed is improved.
Optionally, as shown in fig. 3, the data processing method provided by the embodiment of the present disclosure can also receive and process a read request message, and the processing procedure of the data processing method can be implemented through step 301 to step 305.
In the embodiment of the disclosure, the read request message is used to request to read the second data information, and the failure response message is used to indicate that the reading of the second data information fails. It is understood that the read request message further includes an Identity (ID) of a user sending the read request message, an original source of the second data information, a data identification of the second data information, a sending path, check data, a file size, and the like. The second data information carries the specific information in a preset message sending format.
In the embodiment of the present disclosure, after the received read request message, when the primary database is in a failure state, according to whether the preset list includes the second data information, two different manners, i.e., step 302 and step 303 to step 305, are respectively adopted to perform processing.
In the embodiment of the present disclosure, in the case that the main database has a fault and the preset list includes the second data information, the read request message is not processed, and a failure response message is sent to the terminal that sent the read request message, so as to prompt the user that the corresponding data information cannot be acquired. The high availability of the whole service in the whole data processing process is realized through the unavailability of a small part of data set in the preset list.
It should be noted that, if the second data information belongs to the preset list, it is determined that the second data information is not stored in the first database or the master database, that is, when the read request message is processed, the second data information is not uniformly stored in each database, that is, the information stored in the first database, the second database, the master database, and the read database is inconsistent. Because the second data information is status data which changes in real time according to user operation, if the contents of the data source, the second data information and the like included in the read request message do not realize the consistency of the storage messages of the plurality of databases, the read request message is not processed in order to ensure the accuracy of the written or read data.
In the embodiment of the disclosure, the preset list can store the first data information, and meanwhile, the preset list can also store the user ID corresponding to the first data information, and by searching the user ID corresponding to the second data information, the search range can be reduced, so that whether the second data information is included in the preset list can be determined quickly.
In the embodiment of the present disclosure, in a case that the main database fails and the preset list does not include the second data information, the data processing apparatus processes the read request message. And in the processing process, sequentially searching the databases for storing the second data information according to the sequence of the second database, the first database and the read database.
It is understood that the data processing apparatus sequentially changes the second database, the first database, and the read database from small to large according to the purpose of use of each of the databases described above by the amount of data stored in the default database. First, the second database is used for temporary data stored when the first database fails, and the amount of data stored in the second database may be minimized. Secondly, the first database usually needs to be preset with a storage space, and data is stored in all storage positions of the preset storage space in a first-in first-out mode in the caching process, so that the amount of data stored in the first database is larger than that stored in the second database. Finally, the read database is used to synchronize the data in the master database, so the amount of data stored in the read database is the largest.
In the embodiment of the present disclosure, the data processing apparatus may search the database storing the second data information by sending the user identity ID of the read request message, the original source of the second data information, or the data identifier of the second data information.
The technical scheme provided by the disclosure at least brings the following beneficial effects: in case of a failure of the master database, if the second data information is included in the preset list, it indicates that the second data information is not successfully written in the first database or the master database, and thus the second data information cannot be read, a failure response message is sent, and a reading state of the second data information can be prompted. And under the condition that the main database fails, if the second data information is not included in the preset list, the second data information is successfully written into the first database, so that the accurate second data information can be read. In addition, in the process of reading the second data information, the second database, the first database and the read database are sequentially selected according to the sequence of less to more stored contents in the databases, and whether each database comprises the second data information is searched, so that the response of the read request information can be ensured, and the efficiency of processing the read request information can be improved.
Optionally, as shown in fig. 4, after the failure of the master database is resolved, the data processing method provided in the embodiment of the present disclosure further includes step 401 and then step 402.
In the embodiment of the present disclosure, after the failure of the primary database is resolved, the data information in the secondary database is written into the primary database, that is, the data in the secondary database is switched back to the primary database, so as to ensure that all the written data information is stored in the primary database.
The technical scheme provided by the disclosure at least brings the following beneficial effects: through data switchback, guarantee that first database can save whole data, correct the data omission that causes because of first database trouble, adopt asynchronous or semi-synchronous mode to carry out data synchronization simultaneously, under the prerequisite of guaranteeing that read database and main database carry out data synchronization, improve the efficiency of switchback.
Further alternatively, in the disclosed embodiment, after the failure of the primary database is resolved and the data information in the secondary database is written into the primary database, the data processing apparatus deletes the data information in the secondary database that has been successfully written into the primary database. It is understood that the second database may receive a partial read request message after the failure of the primary database is resolved without accepting the write request message. After all the data in the second database is switched back into the primary database, the second database no longer accepts any request messages.
The technical scheme provided by the disclosure at least brings the following beneficial effects: and deleting the data in the second database, and ensuring that the second database is an empty database under the condition that the main database does not fail or the failure is relieved, so as to save the storage space.
Further optionally, in this embodiment of the disclosure, the data processing apparatus may further delete the first data information in the preset list after the failure of the master database is resolved and the second data information is written in the master database.
Optionally, in this embodiment of the present disclosure, the "writing the first data information into the at least one first database" in step 202 may be implemented in a specific manner as follows: the data processing device pulls the first data information requested to be written by the writing request message to at least one first database through the message processing component. In the step 203, the "adding the first data information that is not successfully written into the at least one first database into the preset list" may specifically be: the data processing device determines that the first data information which is not successfully pulled to the at least one first database is obtained through the message processing assembly, and adds the determined first data information to a preset list.
In the embodiment of the present disclosure, if the first data message is pulled to the first database, it indicates that the write message request corresponding to the first data message has been written, and may be deleted from the message processing component.
Further optionally, in this embodiment of the present disclosure, after the failure of the primary database is recovered each time, the first data message in the preset list is rewritten into the first database process, and the first data message is written into the first database.
Further optionally, in the embodiment of the present disclosure, after the failure of the master database is recovered each time, the preset list is directly cleared.
Further optionally, the data processing apparatus determines that the data information in the at least one first database is not successfully pulled after the last failure of the autonomous data is determined by the message processing component, and adds the determined data information to the preset list.
In the embodiment of the disclosure, at intervals of each failure of the primary database, it is determined that data information that is not successfully pulled to at least the first database after the last failure of the primary database may include one first data information requested to be written by one write request message, and may also include a plurality of first data information requested to be written by a plurality of write request messages.
The technical scheme provided by the disclosure at least brings the following beneficial effects: and recording the preset list by judging whether the data is successfully pulled to at least one first database or not, so that the preset list can accurately record the data information written into the first database.
In the embodiment of the present disclosure, as shown in fig. 5, after the message center receives the write request message, on one hand, the message processing component pulls the first data information requested to be written by the write request message to the 3 first databases, and on the other hand, when the result of the interface detection is that the master database is normal, writes the first data information into the master database, and in addition, the read database backs up the data in the master database in an asynchronous or semi-synchronous manner.
In the embodiment of the present disclosure, as shown in fig. 6, after the message center receives the write request, similarly, on one hand, the message processing component pulls the first data information requested to be written by the write request message to the 3 first databases, and on the other hand, when the result of the interface activity detection is a failure of the master database, the first data information cannot be written into the master database, and cannot be synchronized from the master database to the read database; at this time, the first data information is written into the second database. In addition, the first data information which is not written in the message center is added into a preset list. If other write request messages are received during the failure of the primary database, the data requested to be written by the write request messages is directly written into the secondary database.
In the embodiment of the present disclosure, as shown in fig. 7, after the failure of the primary database is resolved, if the result of the interface detection is that the primary database is normal, the data information requested to be written by the write request information is no longer written into the secondary database, but is written into the primary database. After the failure of the primary database is relieved, the first data information stored in the second database is written into the primary database, the second database may receive a partial read request message before the first data information stored in the second database is completely written into the primary database, and the second database may not receive any request message any more after the first data information stored in the second database is completely written into the primary database. After the failure of the main database is relieved, the read database continuously backs up the data in the main database in an asynchronous or semi-synchronous mode.
In the embodiment of the present disclosure, for the read request message, if the master database is normal (no fault occurs), the master database responds to the read request message, and of course, it may also be determined that the first database can respond to the read request message first, and in the case that the first database can respond to the read request message, the master database responds to the read request message; in the event of a failure of the primary database, the read request message is looked up and responded to in the second database, the first database and the read database.
Fig. 8 is a schematic diagram illustrating a structure of a data processing apparatus according to an exemplary embodiment. Referring to fig. 8, a data processing apparatus provided in an embodiment of the present disclosure includes: a receiving unit 81, a writing unit 82, and a processing unit 83;
a receiving unit 81, configured to receive a write request message, where the write request message is used to request to write first data information; for example, as shown in fig. 2, the receiving unit 81 may be configured to perform step 201.
A writing unit 82 for writing the first data information into the master database and the at least one first database; the first database is used for caching data information which is requested to be written into the main database; for example, as shown in fig. 2, the writing unit 82 may be configured to perform step 202.
And the processing unit 83 is configured to, if the master database fails and the first data information is not successfully written into the master database, write the first data information into the second database, and add the first data information that is not successfully written into at least one first database into a preset list. For example, as shown in fig. 2, the processing unit 83 may be configured to perform step 203.
Optionally, as shown in fig. 8, the apparatus further includes: a query unit 84;
a receiving unit 81, further configured to receive a read request message, where the read request message is used to request to read the second data information; for example, as shown in fig. 3, the receiving unit 81 may be configured to perform step 301.
The query unit 84 is configured to query whether the second database includes the second data information if the main database fails and the preset list does not include the second data information; for example, as shown in FIG. 3, the query unit 84 may be configured to perform step 303.
The querying unit 84 is further configured to query whether the first database includes the second data information if the second database does not include the second data information; for example, as shown in FIG. 3, the query unit 84 may be configured to perform step 304.
The querying unit 84 is further configured to query whether the read database includes the second data information if the first database does not include the second data information, and the read database is used for backing up the data information in the primary database. For example, as shown in FIG. 3, the query unit 84 may be configured to perform step 305.
Optionally, as shown in fig. 8, the apparatus further includes: a backup unit 85;
the writing unit 82 is further configured to write the data information in the second database into the master database after the failure of the master database is resolved; for example, as shown in fig. 4, the writing unit 82 may be configured to perform step 401.
And a backup unit 85 for backing up the data information written in the second database in the primary database to the read database. For example, as shown in FIG. 4, backup unit 85 may be used to perform step 402.
Optionally, as shown in fig. 8, the apparatus further includes: a deletion unit 86;
and the deleting unit 86 is further configured to delete the data information in the second database that has been successfully written into the primary database.
Optionally, the apparatus further comprises: the backup unit 85 is further configured to, after receiving the write request message, backup the first data information in the master database to the read database if the first data information is successfully written into the master database.
Optionally, the apparatus further comprises: the querying unit 84 is further configured to query the master database for the second data information after receiving the read request message if the master database is normal.
Optionally, the writing unit 82 is specifically configured to: pulling first data information requested to be written by the write request message to at least one first database through the message processing component;
the processing unit 83 is specifically configured to: and determining that the first data information which is not successfully pulled to the at least one first database is obtained through the message processing component, and adding the determined first data information into a preset list.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 9 is a schematic structural diagram of a server provided by the present disclosure. As in fig. 9, the server may include at least one processor 901 and a memory 903 for storing processor-executable instructions. Wherein the processor 901 is configured to execute instructions in the memory 903 to implement the data processing method in the above-described embodiments.
Additionally, the server can also include a communication bus 902 and at least one communication interface 904.
The processor 901 may be a processor (CPU), a micro-processing unit, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present disclosure.
The memory 903 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these. The memory may be self-contained and connected to the processing unit by a bus. The memory may also be integrated with the processing unit.
The memory 903 is used for storing instructions for executing the disclosed solution, and is controlled by the processor 901 to execute. The processor 901 is configured to execute instructions stored in the memory 903, thereby implementing functions in the disclosed methods.
As an example, in connection with fig. 8, the functions implemented by the receiving unit 81, the writing unit 82, and the processing unit 83 in the data processing apparatus are the same as those of the processor 901 in fig. 9.
In particular implementations, processor 901 may include one or more CPUs such as CPU0 and CPU1 in fig. 9 as an example.
In particular implementations, a server may include multiple processors, such as processor 901 and processor 907 in fig. 9, as an example. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).
In particular implementations, server 90 may also include an output device 905 and an input device 906, as one embodiment. An output device 905, which is in communication with the processor 901, may display information in a variety of ways. For example, the output device 905 may be a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display device, a Cathode Ray Tube (CRT) display device, a projector (projector), or the like. The input device 906, which is in communication with the processor 901, may accept input from a user in a variety of ways. For example, the input device 606 may be a mouse, a keyboard, a touch screen device, or a sensing device, among others.
Those skilled in the art will appreciate that the architecture shown in FIG. 9 does not constitute a limitation of a server, and may include more or fewer components than those shown, or combine certain components, or employ a different arrangement of components.
In addition, the present disclosure also provides a computer-readable storage medium, in which instructions, when executed by a processor of a server, enable the server to perform the data processing method provided as the above embodiment.
In addition, the present disclosure also provides a computer program product comprising computer instructions, which when run on a server, cause the server to execute the data processing method as provided in the above embodiments.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111223304.0A CN113886500A (en) | 2021-10-20 | 2021-10-20 | A data processing method, device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111223304.0A CN113886500A (en) | 2021-10-20 | 2021-10-20 | A data processing method, device, server and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113886500A true CN113886500A (en) | 2022-01-04 |
Family
ID=79003905
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111223304.0A Pending CN113886500A (en) | 2021-10-20 | 2021-10-20 | A data processing method, device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113886500A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115098474A (en) * | 2022-06-21 | 2022-09-23 | 马上消费金融股份有限公司 | Data processing method, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447029A (en) * | 2014-08-27 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Business processing method and equipment used when database goes wrong |
| CN105574020A (en) * | 2014-10-14 | 2016-05-11 | 阿里巴巴集团控股有限公司 | Database operation method and device |
| CN109062731A (en) * | 2018-07-16 | 2018-12-21 | 阿里巴巴集团控股有限公司 | Idempotent control method and device when a kind of database switches |
| CN109710619A (en) * | 2018-12-29 | 2019-05-03 | 中国银联股份有限公司 | Database expansion method, device and readable medium |
-
2021
- 2021-10-20 CN CN202111223304.0A patent/CN113886500A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447029A (en) * | 2014-08-27 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Business processing method and equipment used when database goes wrong |
| CN105574020A (en) * | 2014-10-14 | 2016-05-11 | 阿里巴巴集团控股有限公司 | Database operation method and device |
| CN109062731A (en) * | 2018-07-16 | 2018-12-21 | 阿里巴巴集团控股有限公司 | Idempotent control method and device when a kind of database switches |
| CN109710619A (en) * | 2018-12-29 | 2019-05-03 | 中国银联股份有限公司 | Database expansion method, device and readable medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115098474A (en) * | 2022-06-21 | 2022-09-23 | 马上消费金融股份有限公司 | Data processing method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8868492B2 (en) | Method for maximizing throughput and minimizing transactions response times on the primary system in the presence of a zero data loss standby replica | |
| US8838919B2 (en) | Controlling data lag in a replicated computer system | |
| US9753954B2 (en) | Data node fencing in a distributed file system | |
| US20200019543A1 (en) | Method, apparatus and device for updating data, and medium | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| US20090019094A1 (en) | Redirected updates on a backup server | |
| WO2022111188A1 (en) | Transaction processing method, system, apparatus, device, storage medium, and program product | |
| CN110134648A (en) | Log processing method, device, equipment, system, and computer-readable storage medium | |
| US12327032B2 (en) | Data processing method, apparatus, device and medium based on distributed storage | |
| JP5686034B2 (en) | Cluster system, synchronization control method, server device, and synchronization control program | |
| JP2009157785A (en) | Method for adding standby computer, computer and computer system | |
| US11687533B2 (en) | Centralized storage for search servers | |
| CN110784498B (en) | Personalized data disaster tolerance method and device | |
| JP5741254B2 (en) | Transmission control method, apparatus and program | |
| US11522975B2 (en) | Double persistence layer using an in-memory map | |
| CN110196680B (en) | Data processing method, device and storage medium | |
| CN113010549A (en) | Data processing method based on remote multi-active system, related equipment and storage medium | |
| WO2021057108A1 (en) | Data reading method, data writing method, and server | |
| US9330153B2 (en) | System, method, and computer readable medium that coordinates between devices using exchange of log files | |
| JP7392131B2 (en) | Switching to eventually consistent database replication | |
| CN114077518B (en) | Data snapshot method, device, equipment and storage medium | |
| CN113886500A (en) | A data processing method, device, server and storage medium | |
| CN112417043B (en) | Data processing system and method | |
| CN107404511B (en) | Method and device for replacing servers in cluster | |
| US11768741B2 (en) | Replicating changes written by a transactional virtual storage access method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |