CN114328711A - Data processing method and device, computer equipment and storage medium - Google Patents
Data processing method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN114328711A CN114328711A CN202011062848.9A CN202011062848A CN114328711A CN 114328711 A CN114328711 A CN 114328711A CN 202011062848 A CN202011062848 A CN 202011062848A CN 114328711 A CN114328711 A CN 114328711A
- Authority
- CN
- China
- Prior art keywords
- library
- slave
- master
- log
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 13
- 230000001360 synchronised effect Effects 0.000 claims abstract description 90
- 238000000034 method Methods 0.000 claims abstract description 55
- 238000004590 computer program Methods 0.000 claims description 18
- 238000013500 data storage Methods 0.000 claims description 2
- 230000008859 change Effects 0.000 abstract description 5
- 239000003795 chemical substances by application Substances 0.000 description 36
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data processing method, a data processing device, a computer device and a storage medium. The method comprises the following steps: receiving a reading request, wherein the reading request carries a session identifier; acquiring a master library synchronization log site corresponding to the session identifier; receiving slave library synchronization log sites reported by slave libraries; and when the synchronous log locus of the master library is smaller than the synchronous log locus of the slave library, sending a read request to the slave library. The method comprises the steps of storing a master library synchronous log site of each session in an agent layer, determining the offset of the request in a change log in a master library according to a session identifier in a read request, namely the synchronous log site, and the synchronous log site reported by a slave library, judging whether the current synchronous log site of the slave library exceeds the synchronous log site of the master library, reading data through the slave library if the current synchronous log site of the slave library exceeds the synchronous log site of the master library, and reading data through the master library if the current synchronous log site of the slave library exceeds the synchronous log site of the master library, otherwise, reducing the reading pressure of the master library.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, a computer device, and a storage medium.
Background
In the read-expansion scenario, the industry database generally adopts a read-write separation architecture. I.e., write requests are sent to the master library and read requests are sent to the slave library through the proxy layer. Because the master and the slave have certain synchronous delay, the read-write consistency of service access cannot be ensured, and only the final consistency can be ensured. In order to solve the problems, the service is split, the read-write request with strong consistency requirements is directly connected to the master library, and partial read-write separation of final consistency can be accepted. The direct connection of strong-consistency read-write requests to the main library in the upper mode can increase the pressure of the main library and influence the effect of read-write separation.
Disclosure of Invention
In order to solve the technical problem, the present application provides a data processing method, apparatus, computer device and storage medium.
In one embodiment, the present application provides a method of data processing, comprising:
receiving a reading request, wherein the reading request carries a session identifier;
acquiring a master library synchronization log site corresponding to the session identifier;
receiving slave library synchronization log sites reported by slave libraries;
and when the master library synchronization log locus is larger than or equal to the slave library synchronization log locus, sending the read request to the master library.
In one embodiment, the present application provides an apparatus for data processing, comprising:
a request receiving module, configured to receive a read request, where the read request carries a session identifier;
the data acquisition module is used for acquiring a master library synchronous log site corresponding to the session identifier;
the data receiving module is used for receiving slave library synchronous log sites reported by a slave library;
and the request sending module is used for sending the read request to the master library when the master library synchronous log site is greater than or equal to the slave library synchronous log site.
In one embodiment, the present application provides a method of data processing, comprising:
receiving a write request sent by an agent layer, wherein the write request carries a session identifier;
executing the write request;
and after the write request is executed, sending return information containing the master library synchronization log position to the proxy layer.
In one embodiment, the present application provides an apparatus for data processing, comprising:
the request receiving module is used for receiving a write request sent by an agent layer, wherein the write request carries a session identifier;
a request execution module for executing the write request;
and the data sending module is used for sending return information containing the master library synchronization log site to the proxy layer after the write request is executed.
In one embodiment, the present application provides a method of data processing, comprising:
receiving a reporting instruction sent by an agent layer;
reporting the slave library synchronous log sites of the slave libraries according to the reporting instruction;
receiving a read instruction sent by the agent layer, wherein the read instruction carries a session identifier and a slave library synchronization log site, and the master library synchronization log site corresponding to the session identifier is less than or equal to the slave library synchronization log site;
and returning data to the proxy layer according to the read instruction.
In one embodiment, the present application provides an apparatus for data processing, comprising:
the instruction receiving module is used for receiving a reporting instruction sent by the agent layer;
the data reporting module is used for reporting the slave library synchronization log sites of the slave library according to the reporting instruction;
the instruction receiving module is further configured to receive a read instruction sent by the agent layer, where the read instruction carries a session identifier and a slave library synchronization log site, and the master library synchronization log site corresponding to the session identifier is less than or equal to the slave library synchronization log site;
and the data return module is used for returning data to the proxy layer according to the read instruction.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above-mentioned data processing method when executing the computer program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method of data processing.
The data processing method, the data processing device, the computer equipment and the storage medium comprise the following steps: receiving a reading request, wherein the reading request carries a session identifier; acquiring a master library synchronization log site corresponding to the session identifier; receiving slave library synchronization log sites reported by slave libraries; and when the synchronous log locus of the master library is smaller than the synchronous log locus of the slave library, sending a read request to the slave library. The method comprises the steps that synchronous log sites of a master library of each session are stored in an agent layer, when a read request is received, the offset of the request in a change log in the master library, namely the synchronous log sites, and the synchronous log sites reported by a slave library are determined according to a session identifier in the read request, whether the current synchronous log sites of the slave library exceed the synchronous log sites of the master library or not is judged, if yes, data are read through the slave library, and otherwise, the data are read through the master library. And reducing the reading pressure of the main library under the condition of ensuring the consistency of database reading and writing.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a diagram of an application environment of a method of data processing in one embodiment;
FIG. 2 is a flow diagram illustrating a method of data processing in one embodiment;
FIG. 3 is a flow diagram illustrating a method of data processing in one embodiment;
FIG. 4 is a flow diagram illustrating a method of data processing in one embodiment;
FIG. 5 is a block diagram showing an arrangement for data processing in one embodiment;
FIG. 6 is a block diagram showing an arrangement for data processing in one embodiment;
FIG. 7 is a block diagram showing an arrangement for data processing in one embodiment;
FIG. 8 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
FIG. 1 is a diagram of an application environment of a method of data processing in one embodiment. Referring to fig. 1, the method of data processing is applied to a database read-write system. The database read-write system includes an agent layer 110, a master library 120, and a slave library 130. The proxy layer 110, the master library 120, and the slave library 130 are connected through a network. The agent layer 110 may be a terminal or a server, the terminal may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The master library 120 and the slave library 130 may be implemented as separate servers or as a server cluster of multiple servers.
In one embodiment, as shown in FIG. 2, a method of data processing is provided. The embodiment is mainly illustrated by applying the method to the proxy layer 110 in fig. 1. Referring to fig. 2, the data processing method specifically includes the following steps:
step S201, a read request is received.
In this embodiment, the read request carries a session identifier.
Step S202, a master library synchronization log site corresponding to the session identifier is obtained.
Step S203, receiving the slave library synchronization log sites reported by the slave library.
Step S204, when the master library synchronization log position is larger than or equal to the slave library synchronization log position, a read request is sent to the master library.
Step S205, when the synchronous log position of the master library is smaller than the synchronous log position reported by the slave library, a read request is sent to the slave library.
In particular, the master/slave library refers to a master/slave node that achieves high availability and performs data synchronization using a replication technique. The master node and the slave nodes form a database cluster. The main nodes provide service writing and reading, and generally, only one main node is arranged in one cluster; the slave node is typically used for traffic reading, and there may be multiple standby nodes in a set of clusters.
The read request is a computer instruction for reading data, and the computer instruction carries a session identifier, a data identifier for reading data, and the like. The session identification is a label for identifying the entire session. A session refers to a Context (Context) from the beginning to the end of a communication between two communicating parties. The synchronization log location is an offset (offset) in the log used to record the data changes in the database, and the location of the data is recorded by the offset in the log, such as which row of the log file the data is located in. The master library synchronous log site refers to the latest synchronous log site corresponding to the request returned by the master library, and the slave library synchronous log site refers to the synchronous log site reported by the slave library. And the synchronous log site returned by the master library corresponds to the connection identifier, and the synchronous log site reported by the slave library is the executed synchronous log site of the slave library synchronous master library data. The slave library synchronization log sites are actively reported to the agent layer from the library, when the slave library is started in the agent layer, a reporting instruction sent by the agent layer is received, and the corresponding slave library synchronization log sites are reported according to the reporting requirement in the reporting instruction.
In a specific embodiment, the latest synchronization log position of the master library is 200, the latest synchronization log position reported by the slave library is 180 (slave library synchronization log position), the master library synchronization log position corresponding to the session identifier is 150, and if the slave library synchronization log position (180) is greater than the master library synchronization log position (150), the proxy layer sends a read request to the slave library. And if the slave library synchronous log site is smaller than the master library synchronous log site, the proxy layer sends a read request to the master library.
In one embodiment, when a plurality of slave nodes exist, receiving a slave library synchronization log site reported by each slave node, judging whether the slave library synchronization log site of each slave node is larger than a master library synchronization log site corresponding to a session identifier, and if the slave library synchronization log site of each slave node is smaller than the master library synchronization log site, sending a read request to a master library. And if the slave library synchronization log locus of one slave node is the same as the master library synchronization log locus, sending a read request to the slave library of which the slave library synchronization log locus is larger than the master library synchronization log locus. If the slave node comprises a slave node A, a slave node B and a slave node C, wherein the slave library synchronous log position of the slave node A is larger than the master library synchronous log position, the read request is sent to the slave node A.
The data processing method comprises the following steps: receiving a reading request, wherein the reading request carries a session identifier; acquiring a master library synchronization log site corresponding to the session identifier; receiving slave library synchronization log sites reported by slave libraries; and when the synchronous log locus of the master library is smaller than the synchronous log locus reported by the slave library, sending a read request to the slave library. The method comprises the steps that synchronous log sites of a master library of each session are stored in an agent layer, when a read request is received, the offset of the request in a change log in the master library, namely the synchronous log sites, and the synchronous log sites reported by a slave library are determined according to a session identifier in the read request, whether the current synchronous log sites of the slave library exceed the synchronous log sites of the master library or not is judged, if yes, data are read through the slave library, and otherwise, the data are read through the master library. And reducing the reading pressure of the main library under the condition of ensuring the consistency of database reading and writing.
In one embodiment, before obtaining the master library synchronization log site corresponding to the session identifier, the method further includes: receiving a write request, wherein the write request carries a session identifier; sending a write request to a master library; and receiving return information returned by the master library, wherein the return information is returned after the write request is executed, and the return information comprises master library synchronous log sites.
In particular, a write request is for computer instructions to write data, through which the data is written to the master library. Write instructions include, but are not limited to, updates and additions of data, and the like. And sending a write request to the master library, and updating the data in the database by the master library according to the write request. After finishing data updating, the master library returns corresponding information to the agent layer, namely, returns information, wherein the returned information comprises a mark of successful writing and a master library synchronization log site corresponding to the session. And storing the master library synchronization log site and the corresponding relation between the master library synchronization log site and the session identifier. And expanding the returned information value when returning the information, namely increasing the master library synchronous log site corresponding to the write request.
In one embodiment, after storing the corresponding relationship between the master library synchronization log location and the session identifier, the method further includes: and when the master library synchronous log site corresponding to the session identifier is received again, updating the corresponding relation between the master library synchronous log site and the session identifier.
Specifically, if the agent layer receives a data update request corresponding to the session identifier, the data update request is sent to the master library, the master library returns return information corresponding to the session identifier after completing data update, and the corresponding relationship between the master library synchronization log site and the session identifier is updated according to the session identifier in the return information. The latest master library synchronization log location that is about to be returned replaces the previous master library synchronization log location. If the latest master library synchronization log position returned by the session identifier is 300 and the previous master library synchronization log position corresponding to the session identifier is 150, the master library synchronization log position corresponding to the session identifier is updated to 300.
In one embodiment, before receiving the slave library synchronization log site reported from the library, the method comprises: and sending a timed report instruction to the slave library so that the slave library reports the synchronous log sites of the slave library at a fixed time.
Specifically, the timed report instruction is used for indicating the slave library, and the slave library synchronization log sites are reported in a timed manner according to the time information in the timed report instruction, if the timed report time can be customized, and if the timed report time is 10ms, the slave library synchronization log sites are uploaded once every 10 ms.
In an embodiment, the data processing method further includes: acquiring current time information; acquiring time information of a slave library synchronization log site which is reported from a slave library and is received the last time with the current time information to obtain first time information; calculating a difference value between the current time information and the first time information; and when the current difference value is larger than the preset difference value, sending the timed reporting instruction to the slave library again.
Specifically, the current time information is time information for describing the current time. The first time information refers to the time information of the slave library synchronization log position reported from the slave library the latest time from the current time information, namely the time information of the slave library synchronization log position reported from the slave library the last time. And calculating a difference value between the current time information and the first time information, namely time difference, judging whether the time difference is larger than a preset difference value, if so, indicating that the slave library does not report data for a long time, and determining whether the slave library normally operates or not or determining whether a network has a problem. And sending the timing reporting instruction again, and determining whether the slave library is abnormal or the network is abnormal according to the returned information of the sent timing reporting instruction.
In one embodiment, the slave library comprises a plurality of slave nodes, a slave library synchronization log site for receiving packets on each slave node of the slave library; and when the master library synchronization log site is smaller than the slave library synchronization log site of the current slave node, sending a read request to the current slave node.
Specifically, when the slave library includes a plurality of slave nodes, each slave node backs up different data, or a plurality of slave nodes synchronize the same data, and the synchronization progress of each slave node is inconsistent due to differences in resources and networks of each slave node. And if the slave library synchronous log site of the current slave node is greater than or equal to the master library synchronous log site, sending a read request to the current slave node. And if the slave library synchronous log position of the plurality of slave nodes is larger than or equal to the master library synchronous log position, sending a reading request to any one target slave node, wherein the slave library synchronous log position of the target slave node is larger than or equal to the master library synchronous log position.
In one embodiment, the maximum value is screened out from the slave library synchronization log sites reported by each cluster node, whether the maximum value is smaller than the master library synchronization log site or not is judged, and if the maximum value is smaller than the master library synchronization log site, a read request is sent to the master library. And if the maximum value is larger than or equal to the master library synchronous log site, sending a read request to the slave node corresponding to the maximum value. The judgment process that a plurality of slave nodes need to be judged one by one can be avoided by directly judging the maximum value.
In one embodiment, as shown in fig. 3, the embodiment is mainly illustrated by applying the method to the master library 120 in fig. 1. Referring to fig. 3, the data processing method specifically includes the following steps:
step S301, receiving a write request sent by the proxy layer.
In this embodiment, the write request carries a session identifier.
In step S302, a write request is executed.
Step S303, after the write request is executed, sending the return information including the synchronization log location of the master library to the proxy layer.
In particular, a write request is for computer instructions to write data, through which the data is written to the master library. Write instructions include, but are not limited to, updates and additions of data, and the like. And sending a write request to the master library, and updating the data in the database by the master library according to the write request. After finishing data updating, the master library returns corresponding information to the agent layer, namely, returns information, wherein the returned information comprises a mark of successful writing and a master library synchronization log site corresponding to the session. And storing the master library synchronization log site and the corresponding relation between the master library synchronization log site and the session identifier. And expanding the returned information value when returning the information, namely increasing the master library synchronous log site corresponding to the write request. After the master library executes the write request, adding master library synchronous log sites into return information, storing the master library synchronous log sites by an agent layer, storing master library synchronous log sites of each session by the agent layer, determining the offset of the request in a change log in the master library according to a session identifier in the read request when the read request is received, namely the synchronous log sites, and the synchronous log sites reported by the slave library, judging whether the current synchronous log sites of the slave library exceed the master library synchronous log sites, if so, reading data through the slave library, otherwise, reading the data through the master library. And reducing the reading pressure of the main library under the condition of ensuring the consistency of database reading and writing.
In one embodiment, as shown in fig. 4, the present embodiment is mainly illustrated by applying the method to the slave library 130 in fig. 1. Referring to fig. 4, the data processing method specifically includes the following steps:
step S401, a reporting instruction sent by the agent layer is received.
And step S402, reporting the slave library synchronization log sites of the slave libraries according to the reporting instruction.
Step S403, receiving a read command sent by the proxy layer.
In this embodiment, the read instruction carries a session identifier and a slave library synchronization log location, and the master library synchronization log location corresponding to the session identifier is less than or equal to the slave library synchronization log location.
Step S404, returning data to the proxy layer according to the read command.
Specifically, the reporting instruction is used for indicating the slave library and reporting the slave library synchronization log site according to the reporting instruction. The reporting time of the reporting instruction can be customized according to the requirement. And the slave library synchronization log site actively reports the report instruction to the agent layer after receiving the report instruction. When the slave library receives the read request sent by the agent layer, the slave library synchronization log position of the slave library is larger than or equal to the master library synchronization log position. And the slave library acquires corresponding data according to the reading instruction and returns the data to the proxy layer. The method comprises the steps that synchronous log sites of a master library of each session are stored in an agent layer, when a read request is received, the offset of the request in a change log in the master library, namely the synchronous log sites, and the synchronous log sites reported by a slave library are determined according to a session identifier in the read request, whether the current synchronous log sites of the slave library exceed the synchronous log sites of the master library or not is judged, if yes, data are read through the slave library, and otherwise, the data are read through the master library. And reducing the reading pressure of the main library under the condition of ensuring the consistency of database reading and writing.
In a specific embodiment, the data processing method includes:
the proxy layer maintains a gtid (global event id) corresponding to the latest write on each user session granularity, that is, a synchronization log site P _ session corresponding to the session write request:
when the database returns the write-in result to the agent layer, expanding the return value and increasing the synchronous log sites corresponding to the requests; and the agent layer updates the synchronous log position P _ session corresponding to the session in the memory according to the synchronous log position.
The agent layer obtains a synchronization site P _ slave of the slave node of the database in an event mode:
when the agent layer is started, sending a command for reporting a synchronization site P _ slave to the slave library;
the Slave reports the synchronization site P _ Slave of the node to the agent layer at regular time;
when the slave does not report the synchronization site for more than n cycles (multiple of 10), the agent layer issues a command of the synchronization site to the slave layer to detect the running availability of the slave.
The agent layer realizes read-write consistency at the combination session site and the slave node site:
when the proxy layer receives a read request from a certain session, a slave node with a synchronization site P _ slave > -P _ session is selected according to the session site P _ session, and the read request is sent to the slave node.
And when all the slave nodes are not satisfied, sending the read request to the master node where the master library is located.
The event mode has less increase on the expense of the agent layer, and reduces the expense of the interaction mode of the event on the system. By the data reading mode, the splitting of the service can be reduced, and the adaptation cost of the service is reduced.
Fig. 2-4 are flow diagrams illustrating a method of data processing according to an embodiment, respectively. It should be understood that although the various steps in the flow charts of fig. 2-4 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps of each of the flowcharts of fig. 2-4 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 5, there is provided an apparatus 200 for data processing, comprising:
a request receiving module 201, configured to receive a read request, where the read request carries a session identifier.
And a data obtaining module 202, configured to obtain a master library synchronization log location corresponding to the session identifier.
And the data receiving module 203 is configured to receive the slave library synchronization log sites reported by the slave libraries.
A request sending module 204, configured to send a read request to the master library when the master library synchronization log location is greater than or equal to the slave library synchronization log location.
The request sending module 204 is further configured to send a read request to the slave library when the synchronization log position of the master library is smaller than the synchronization log position reported by the slave library.
In an embodiment, the apparatus 200 for data processing further includes:
the request receiving module 201 is further configured to receive a write request, where the write request carries a session identifier.
The request sending module 204 is further configured to send a write request to the master library;
the data receiving module 203 is further configured to receive return information returned by the master library, where the return information is information returned after the master library executes the write request, and the return information includes a master library synchronization log site.
In an embodiment, the apparatus 200 for data processing further includes:
and the data storage module is used for storing the corresponding relation between the synchronization log sites of the master library and the session identification.
In an embodiment, the apparatus 200 for data processing further includes:
and the data updating module is used for updating the corresponding relation between the master library synchronization log site and the session identifier when the master library synchronization log site corresponding to the session identifier is received again.
In an embodiment, the apparatus 200 for data processing further includes:
and the report instruction sending module is used for sending a timed report instruction to the slave library so as to ensure that the slave library reports the slave library synchronous log sites at a timed mode.
In an embodiment, the apparatus 200 for data processing further includes:
and the time information acquisition module is used for acquiring the current time information, acquiring the time information of the slave library synchronization log site which is reported from the slave library and is received the latest time with the current time information, and acquiring the first time information.
And the difference value calculating module is used for calculating the difference value between the current time information and the first time information.
The reporting instruction sending module is further used for sending the timing reporting instruction to the slave library again when the current difference value is larger than the preset difference value.
In one embodiment, the slave library includes a plurality of slave nodes,
the data receiving module 203 is specifically configured to receive slave library synchronization log sites of packets on each slave node of the slave library.
The request sending module 204 is specifically configured to send a read request to the current slave node when the master library synchronization log location is smaller than the slave library synchronization log location of the current slave node.
In one embodiment, as shown in fig. 6, there is provided an apparatus 300 for data processing, applied to a master library, the apparatus comprising:
a request receiving module 301, configured to receive a write request sent by an agent layer, where the write request carries a session identifier;
a request execution module 302 for executing the write request;
and the data sending module 303 is configured to send return information including a master library synchronization log location to the proxy layer after the write request is executed.
In one embodiment, as shown in fig. 7, there is provided an apparatus 400 for data processing, applied to a master library, the apparatus comprising:
an instruction receiving module 401, configured to receive a reporting instruction sent by an agent layer;
a data reporting module 402, configured to report a slave library synchronization log site of a slave library according to a reporting instruction;
the instruction receiving module 401 is further configured to receive a read instruction sent by the agent layer, where the read instruction carries a session identifier and a slave library synchronization log site, and the master library synchronization log site corresponding to the session identifier is less than or equal to the slave library synchronization log site;
and a data returning module 403, configured to return data to the proxy layer according to the read instruction.
FIG. 8 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may specifically be the proxy layer 110 (or master library 120/slave library 130) in fig. 1. As shown in fig. 8, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen connected via a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program which, when executed by the processor, causes the processor to implement the method of data processing. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a method of data processing. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, the data processing apparatus provided in the present application may be implemented in the form of a computer program, and the computer program may be run on a computer device as shown in fig. 8. The memory of the computer device may store therein various program modules constituting the data processing apparatus, such as a request receiving module 201, a data acquiring module 202, a data receiving module 203, and a request transmitting module 204 shown in fig. 5. Or the request receiving module 301, the request executing module 302 and the data sending module 303 shown in fig. 6. Or, the instruction receiving module 401, the data reporting module 402, and the data returning module 403 shown in fig. 7. The computer program constituted by the respective program modules causes the processor to execute the steps in the method of data processing of the respective embodiments of the present application described in the present specification.
For example, the computer device shown in fig. 8 may perform receiving, by the request receiving module 201 in the apparatus 200 for data processing shown in fig. 5, a read request, where the read request carries a session identifier. The computer device may perform the acquisition of the master library synchronization log site corresponding to the session identification through the data acquisition module 202. The computer device may perform receiving the slave library synchronization log sites reported from the library through the data receiving module 203. The computer device may perform sending a read request to the master library via the request sending module 204 when the master library synchronization log location is greater than or equal to the slave library synchronization log location.
For example, the computer device shown in fig. 8 may execute, by the request receiving module 301 in the apparatus 300 for data processing shown in fig. 6, a write request sent by the receiving agent layer, where the write request carries a session identifier; the computer device may execute the write request through the request execution module 302. After the computer device executes the write request through the data sending module 303, the computer device sends the return information containing the master library synchronization log location to the proxy layer.
For example, the computer device shown in fig. 8 may instruct the receiving module 401 to execute the reporting instruction sent by the receiving agent layer in the data processing apparatus 400 shown in fig. 7. The computer device can report the slave library synchronization log position of the slave library according to the report instruction through the data reporting module 402. The computer device may further execute, by using the instruction receiving module 401, a read instruction sent by the receiving agent layer, where the read instruction carries a session identifier and a slave library synchronization log site, and the master library synchronization log site corresponding to the session identifier is less than or equal to the slave library synchronization log site. The computer device may perform the return of data to the proxy layer according to the read instruction through the data return module 403.
In one embodiment, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any of the embodiments of the method of data processing described above when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of any of the embodiments of the method of data processing described above.
In one embodiment, a computer program product or computer program is provided that includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps of any of the embodiments of the method of data processing described above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (20)
1. A method of data processing, the method comprising:
receiving a reading request, wherein the reading request carries a session identifier;
acquiring a master library synchronization log site corresponding to the session identifier;
receiving slave library synchronization log sites reported by slave libraries;
when the master library synchronization log locus is larger than or equal to the slave library synchronization log locus, sending the read request to a master library;
and when the synchronous log site of the master library is smaller than the synchronous log site reported by the slave library, sending the read request to the slave library.
2. The method of claim 1, wherein before the obtaining the master library synchronization log site corresponding to the session identifier, further comprising:
receiving a write request, wherein the write request carries the session identifier;
sending the write request to a master library;
and receiving return information returned by the master library, wherein the return information is returned by the master library after the write request is executed, and the return information comprises the master library synchronous log locus.
3. The method according to claim 2, wherein after receiving the return information returned from the master library, further comprising:
and storing the corresponding relation between the synchronization log locus of the master library and the session identifier.
4. The method of claim 3, wherein after saving the correspondence between the master repository synchronization log location and the session identifier, further comprising:
and when the master library synchronous log site corresponding to the session identifier is received again, updating the corresponding relation between the master library synchronous log site and the session identifier.
5. The method of claim 1, wherein prior to receiving the slave library synchronization log site reported from the library, comprising:
and sending a timed report instruction to the slave library so that the slave library reports the slave library synchronous log sites at a timed time.
6. The method of claim 5, further comprising:
acquiring current time information;
acquiring time information of a slave library synchronization log site reported by the slave library most recently received with the current time information to obtain first time information;
calculating a difference value between the current time information and the first time information;
and when the current difference value is larger than a preset difference value, the timing reporting instruction is sent to the slave library again.
7. The method of claim 1, wherein the slave library comprises a plurality of slave nodes,
the receiving of the slave library synchronization log sites reported by the slave libraries comprises: receiving the slave pool synchronization log sites for packets on respective slave nodes of the slave pool;
when the master library synchronization log locus is smaller than the slave library synchronization log locus, sending a read request to the slave library, including: and when the master library synchronization log site is smaller than the slave library synchronization log site of the current slave node, sending the read request to the current slave node.
8. A method of data processing, applied to a master library, the method comprising:
receiving a write request sent by an agent layer, wherein the write request carries a session identifier;
executing the write request;
and after the write request is executed, sending return information containing the master library synchronization log position to the proxy layer.
9. A method of data processing, applied to a slave library, the method comprising:
receiving a reporting instruction sent by an agent layer;
reporting the slave library synchronous log sites of the slave libraries according to the reporting instruction;
receiving a read instruction sent by the agent layer, wherein the read instruction carries a session identifier and a slave library synchronization log site, and the master library synchronization log site corresponding to the session identifier is less than or equal to the slave library synchronization log site;
and returning data to the proxy layer according to the read instruction.
10. An apparatus for data processing, the apparatus comprising:
a request receiving module, configured to receive a read request, where the read request carries a session identifier;
the data acquisition module is used for acquiring a master library synchronous log site corresponding to the session identifier;
the data receiving module is used for receiving slave library synchronous log sites reported by a slave library;
the request sending module is used for sending the read request to the master library when the master library synchronous log site is larger than or equal to the slave library synchronous log site;
the request sending module is further configured to send the read request to the slave library.
11. The apparatus of claim 10,
the request receiving module is further configured to receive a write request, where the write request carries the session identifier;
the request sending module is also used for sending the writing request to a master library;
the data receiving module is further configured to receive return information returned by the master library, where the return information is information returned by the master library after the write request is executed, and the return information includes the master library synchronization log site.
12. The apparatus of claim 11, further comprising:
and the data storage module is used for storing the corresponding relation between the synchronization log locus of the master library and the session identifier.
13. The apparatus of claim 12, further comprising:
and the data updating module is used for updating the corresponding relation between the master library synchronous log site and the session identifier when the master library synchronous log site corresponding to the session identifier is received again.
14. The apparatus of claim 10, further comprising:
and the reporting instruction sending module is used for sending a timed reporting instruction to the slave library so as to ensure that the slave library reports the slave library synchronous log sites at a fixed time.
15. The apparatus of claim 14, further comprising:
the time information acquisition module is used for acquiring current time information, acquiring time information of a slave library synchronization log site reported by the slave library and received the latest time with the current time information, and acquiring first time information;
a difference value calculating module, configured to calculate a difference value between the current time information and the first time information;
the reporting instruction sending module is further configured to send the timed reporting instruction to the slave library again when the current difference value is greater than a preset difference value.
16. The apparatus of claim 10, wherein the slave library comprises a plurality of slave nodes,
the data receiving module is specifically configured to receive the slave library synchronization log sites of packets on each slave node of the slave library;
the request sending module is specifically configured to send the read request to the current slave node when the master library synchronization log site is smaller than the slave library synchronization log site of the current slave node.
17. An apparatus for data processing, applied to a master library, the apparatus comprising:
the request receiving module is used for receiving a write request sent by an agent layer, wherein the write request carries a session identifier;
a request execution module for executing the write request;
and the data sending module is used for sending return information containing the master library synchronization log site to the proxy layer after the write request is executed.
18. An apparatus for data processing, applied to a slave library, the apparatus comprising:
the instruction receiving module is used for receiving a reporting instruction sent by the agent layer;
the data reporting module is used for reporting the slave library synchronization log sites of the slave library according to the reporting instruction;
the instruction receiving module is further configured to receive a read instruction sent by the agent layer, where the read instruction carries a session identifier and a slave library synchronization log site, and the master library synchronization log site corresponding to the session identifier is less than or equal to the slave library synchronization log site;
and the data return module is used for returning data to the proxy layer according to the read instruction.
19. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 9 are implemented when the computer program is executed by the processor.
20. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 9.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011062848.9A CN114328711A (en) | 2020-09-30 | 2020-09-30 | Data processing method and device, computer equipment and storage medium |
| PCT/CN2021/094599 WO2022068220A1 (en) | 2020-09-30 | 2021-05-19 | Data processing method and apparatus, computer device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011062848.9A CN114328711A (en) | 2020-09-30 | 2020-09-30 | Data processing method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114328711A true CN114328711A (en) | 2022-04-12 |
Family
ID=80950919
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011062848.9A Pending CN114328711A (en) | 2020-09-30 | 2020-09-30 | Data processing method and device, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114328711A (en) |
| WO (1) | WO2022068220A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117009146B (en) * | 2023-09-28 | 2024-03-08 | 金篆信科有限责任公司 | Data synchronization method, device, computer equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090030986A1 (en) * | 2007-07-27 | 2009-01-29 | Twinstrata, Inc. | System and method for remote asynchronous data replication |
| US7702698B1 (en) * | 2005-03-01 | 2010-04-20 | Yahoo! Inc. | Database replication across different database platforms |
| WO2010106991A1 (en) * | 2009-03-19 | 2010-09-23 | 株式会社ココリンク | Method and system for data replication management |
| CN102591964A (en) * | 2011-12-30 | 2012-07-18 | 北京新媒传信科技有限公司 | Implementation method and device for data reading-writing splitting system |
| CN103020243A (en) * | 2012-12-18 | 2013-04-03 | 新浪网技术(中国)有限公司 | Method and device for master-slave replication structure replacement of database |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6862593B2 (en) * | 2002-05-17 | 2005-03-01 | Sun Microsystems, Inc. | Separation of database transactions |
| CN104391873A (en) * | 2014-10-29 | 2015-03-04 | 上海达梦数据库有限公司 | Database operation separation method and database operation separation system |
| CN104376127B (en) * | 2014-12-12 | 2018-08-03 | 北京奇虎科技有限公司 | A kind of data manipulation method and device |
| CN105912628B (en) * | 2016-04-07 | 2019-05-28 | 北京奇虎科技有限公司 | The synchronous method and device of master-slave database |
| CN108038157A (en) * | 2017-12-05 | 2018-05-15 | 北京星河星云信息技术有限公司 | Master-slave database reading/writing method, system, storage device and server |
| CN108200157B (en) * | 2017-12-29 | 2020-12-25 | 北京奇虎科技有限公司 | Log synchronization method and device for triggering rollback by master node |
| CN110781214A (en) * | 2019-09-26 | 2020-02-11 | 金蝶软件（中国）有限公司 | Database reading and writing method and device, computer equipment and storage medium |
-
2020
- 2020-09-30 CN CN202011062848.9A patent/CN114328711A/en active Pending
-
2021
- 2021-05-19 WO PCT/CN2021/094599 patent/WO2022068220A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7702698B1 (en) * | 2005-03-01 | 2010-04-20 | Yahoo! Inc. | Database replication across different database platforms |
| US20090030986A1 (en) * | 2007-07-27 | 2009-01-29 | Twinstrata, Inc. | System and method for remote asynchronous data replication |
| WO2010106991A1 (en) * | 2009-03-19 | 2010-09-23 | 株式会社ココリンク | Method and system for data replication management |
| CN102591964A (en) * | 2011-12-30 | 2012-07-18 | 北京新媒传信科技有限公司 | Implementation method and device for data reading-writing splitting system |
| CN103020243A (en) * | 2012-12-18 | 2013-04-03 | 新浪网技术(中国)有限公司 | Method and device for master-slave replication structure replacement of database |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022068220A1 (en) | 2022-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106874328B (en) | Data processing method and equipment | |
| CN110781214A (en) | Database reading and writing method and device, computer equipment and storage medium | |
| CN111880956B (en) | Data synchronization method and device | |
| CN108319617B (en) | Method and device for determining master-slave difference of database and switching control method and device | |
| CN110213326B (en) | Method, system, equipment and storage medium for capacity expansion of metadata node cluster | |
| CN105338078A (en) | Data storage method and device used for storing system | |
| CN110555041A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN111198845B (en) | Data migration method, readable storage medium and computing device | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN112698926B (en) | Data processing method, device, equipment, storage medium and system | |
| CN111124475A (en) | Method for storage management, electronic device and computer-readable storage medium | |
| CN111291062B (en) | Data synchronous writing method and device, computer equipment and storage medium | |
| CN110740155A (en) | Request processing method and device in distributed system | |
| CN111090701B (en) | Service request processing method, device, readable storage medium and computer equipment | |
| CN112423281B (en) | Wireless module upgrading method, device, computer equipment and storage medium | |
| CN115221156A (en) | Database cluster capacity expansion method and device, computer equipment and storage medium | |
| CN112527561A (en) | Data backup method and device based on Internet of things cloud storage | |
| CN114328711A (en) | Data processing method and device, computer equipment and storage medium | |
| CN114756293A (en) | Service processing method, device, computer equipment and storage medium | |
| CN109542872B (en) | Data reading method and device, computer equipment and storage medium | |
| CN112860494A (en) | Data center switching method and related equipment thereof | |
| CN112711606A (en) | Database access method and device, computer equipment and storage medium | |
| CN112069152A (en) | Database cluster upgrading method, device, equipment and storage medium | |
| CN117435569A (en) | Dynamic capacity expansion method, device, equipment, medium and program product for cache system | |
| CN116719791A (en) | Information processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |