CN115203334A

CN115203334A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN115203334A
Application number: CN202210929616.1A
Authority: CN
Inventors: 武文斌; 刘静; 黄海鹏; 傅兵
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2022-08-03
Filing date: 2022-08-03
Publication date: 2022-10-18

Abstract

The present disclosure provides a data processing method, an apparatus, an electronic device and a storage medium, which can be used in the technical field of data processing or the financial field, the method comprising: determining a storage node corresponding to an operation request according to the operation request acquired from a client, wherein the storage node comprises a master node and M slave nodes, and M is a positive integer; processing the operation request and the waiting instruction request according to the real-time available states of the M slave nodes to generate a merging request, wherein the real-time available states of the M slave nodes are acquired in real time through a monitoring component communicated with a registration center; according to the merging request, establishing a dynamic connection relation between the agent end and the storage node, wherein the dynamic connection relation is released when a preset condition is met; and sending the merging request to the storage node according to the dynamic connection relation, receiving and forwarding a response result generated based on an execution result of the storage node by the proxy, wherein the execution result is generated by the master node and the slave nodes of the storage node according to the merging request.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

With the development of the internet, in order to ensure that data has a high availability characteristic during operation, data needs to be processed in a synchronous manner and the like, in a data structure server, a data synchronization policy of a master node and a slave node is asynchronous replication, when abnormal conditions such as hardware failure, network interruption and the like occur in a storage layer, the master node triggers high availability switching, and the slave node becomes the master node to replace the master node with problems. In the prior art, data between the master node and the slave node is asynchronously copied, data synchronization of the master node and the slave node can not be completely realized by the data in the asynchronous copying process, and further, when scenes such as second killing, robbery and the like are carried out, the master node and the slave node are switched when the master node and the slave node are not completely synchronized, so that the problems of data error and the like caused by operation request loss are caused.

Disclosure of Invention

In view of this, the present disclosure provides a data processing method, an apparatus, an electronic device, and a storage medium, which at least partially solve the problem of data loss caused after switching between a master node and a slave node, and can ensure consistency of data after switching between the master node and the slave node, and improve high availability of the storage node.

A first aspect of the present disclosure provides a data processing method, including: determining a storage node corresponding to an operation request according to the operation request obtained from a client, wherein the storage node comprises a main node and M slave nodes, and M is a positive integer; processing the operation request and the waiting instruction request according to the real-time available states of the M slave nodes to generate a merging request, wherein the real-time available states of the M slave nodes are acquired in real time through a monitoring component communicated with a registration center; according to the merging request, establishing a dynamic connection relation between the agent end and the storage node, wherein the dynamic connection relation is released when a preset condition is met; and sending the merging request to a storage node according to the dynamic connection relation, and receiving and forwarding a response result generated based on an execution result of the storage node by the proxy, wherein the execution result is generated by a main node and a slave node of the storage node according to the merging request.

In some exemplary embodiments of the present disclosure, the method further comprises: and acquiring the availability status of each slave node in the M slave nodes acquired by the monitoring component in real time from the registry before processing the operation request and the instruction waiting request according to the real-time availability statuses of the M slave nodes to generate a combined request.

In some exemplary embodiments of the present disclosure, the method further comprises: the method comprises the steps that before a storage node corresponding to an operation request is determined according to the operation request obtained from a client, the operation request is obtained from the client; after receiving an operation request acquired from a client, storing client information and request data associated with the operation request to a target queue.

In some exemplary embodiments of the present disclosure, the determining, according to an operation request obtained from a client, a storage node corresponding to the operation request includes: analyzing an operation request acquired from a client according to a preset algorithm to determine target data; and determining a storage node corresponding to the operation request according to the target data.

In some exemplary embodiments of the disclosure, the processing the operation request and the wait instruction request according to the real-time availability status of the M slave nodes, and generating a merge request includes: counting the number of available slave nodes according to the acquired available state of each node in the M slave nodes; and when the number of the available slave nodes is larger than a set threshold value, processing the operation request and the waiting instruction request to generate a combined request.

In some exemplary embodiments of the present disclosure, the preset condition includes: the time for establishing the dynamic connection relation exceeds the set time; and the slave node generates an execution result according to the merging request.

In some exemplary embodiments of the present disclosure, the method further comprises: and when the dynamic connection relation is released, deleting the request queue associated with the dynamic connection relation, and releasing the stored memory information associated with the dynamic connection relation.

In some exemplary embodiments of the present disclosure, the receiving and forwarding, by the proxy, a response result generated based on an execution result of the storage node includes: the agent end receives a first execution result, and the first execution result is generated by the main node according to the operation request in the combination request; the agent end receives a second execution result, and the second execution result is generated by the slave node synchronizing the data of the master node to M slave nodes according to the waiting instruction request in the merging request; and the agent terminal generates a response result according to the received first execution result and the second execution result and forwards the response result to the client terminal.

In a second aspect of the present disclosure, there is provided a data processing apparatus comprising: the determining module is configured to determine a storage node corresponding to an operation request according to the operation request acquired from a client, wherein the storage node comprises a master node and M slave nodes, and M is a positive integer; the generation module is configured to process the operation request and the waiting instruction request according to the real-time available states of the M slave nodes, and generate a combination request, wherein the real-time available states of the M slave nodes are acquired in real time through a monitoring component communicated with a registration center; the processing module is configured to establish a dynamic connection relation between the agent end and the storage node according to the merging request, and the dynamic connection relation is released when a preset condition is met; and the receiving and forwarding module is configured to send the merging request to the storage node according to the dynamic connection relation, the proxy receives and forwards a response result generated based on an execution result of the storage node, and the execution result is generated by a master node and a slave node of the storage node according to the merging request.

In some exemplary embodiments of the present disclosure, the data processing apparatus further includes an obtaining module, configured to obtain, from the registry, the availability status of each slave node in the M slave nodes obtained by the monitoring component in real time before generating a merge request by processing the operation request and the wait instruction request according to the real-time availability statuses of the M slave nodes.

In some exemplary embodiments of the present disclosure, the data processing apparatus further comprises a storage module configured to: the method comprises the steps that before a storage node corresponding to an operation request is determined according to the operation request obtained from a client, the operation request is obtained from the client; after receiving an operation request acquired from a client, storing client information and request data associated with the operation request to a target queue.

In some exemplary embodiments of the present disclosure, the determining module comprises a determining submodule configured to: analyzing the operation request acquired from the client according to a preset algorithm to determine target data; and determining a storage node corresponding to the operation request according to the target data.

In some exemplary embodiments of the present disclosure, the generation module comprises a generation submodule configured to: counting the number of available slave nodes according to the acquired available state of each node in the M slave nodes; and when the number of the available slave nodes is greater than a set threshold value, processing the operation request and the waiting instruction request to generate a combined request.

In some exemplary embodiments of the present disclosure, the data processing apparatus further comprises a deletion release module configured to: and when the dynamic connection relation is released, deleting the request queue associated with the dynamic connection relation, and releasing the stored memory information associated with the dynamic connection relation.

In some exemplary embodiments of the present disclosure, the receiving and forwarding module further includes a receiving and forwarding sub-module configured to: the agent end receives a first execution result, and the first execution result is generated by the main node according to the operation request in the combination request; the agent end receives a second execution result, and the second execution result is generated by synchronizing the data of the master node to M slave nodes according to the waiting instruction request in the merging request; and the agent terminal generates a response result according to the received first execution result and the second execution result and forwards the response result to the client terminal.

In a third aspect of the present disclosure, there is provided an electronic device, including: one or more processors; a storage device for storing executable instructions that, when executed by the processor, implement the method according to the above.

In a fourth aspect of the disclosure, a computer-readable storage medium is provided having executable instructions stored thereon which, when executed by a processor, implement a method according to the above.

A fifth aspect of the disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements a method according to the above.

According to the embodiment of the disclosure, the data processing method can be applied to a proxy end, the operation request and the waiting instruction request from the client end are processed according to the acquired real-time available state of the slave node to generate the merging request, the merging request is sent to the storage node according to the established dynamic connection relation, and the response result generated based on the execution result of the storage node is received. On the other hand, as the execution result is that the master node and the slave node of the storage node are generated according to the merging request, and each merging request corresponds to one dynamic connection relation, the data consistency of the master node and the slave node can be ensured during operation, and the dynamic connection relation is adopted, so that the operation request is not blocked, and the data processing efficiency is improved.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

fig. 1 schematically shows an exemplary system architecture to which a data processing method may be applied according to an embodiment of the present disclosure;

FIG. 2 schematically shows a flow diagram of a data processing method according to an embodiment of the present disclosure;

fig. 3 schematically shows a flow chart of a data processing method in acquiring an available state of a slave node according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow diagram of a data processing method at operation 400, according to an embodiment of the present disclosure;

fig. 5 schematically shows a flowchart of a data processing method according to an embodiment of the present disclosure at operation S210;

fig. 6 schematically shows a flowchart of a data processing method according to an embodiment of the present disclosure at operation S220;

FIG. 7 schematically shows a flow chart of a data processing method at dynamic connection release time according to an embodiment of the present disclosure;

FIG. 8 schematically illustrates a flow diagram of a data processing method at operation 600, according to an embodiment of the present disclosure;

FIG. 9 is a schematic diagram illustrating an implementation of a data processing method according to an embodiment of the disclosure;

fig. 10 schematically shows a response result of a wait instruction request in a merge request of a data processing method according to an embodiment of the present disclosure;

fig. 11 schematically shows a communication relationship diagram of an agent side, a registry and a monitoring component in a data processing method according to an embodiment of the present disclosure;

FIG. 12 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

fig. 13 schematically shows a block diagram of an electronic device for implementing a data processing method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that these descriptions are illustrative only and are not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B, or C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B, or C" would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.). The terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features.

In the embodiment of the present disclosure, the term "wait for instruction request" refers to a request for waiting for completion of execution of other instructions, and the command waits until all operation requests (write commands) of the master node of the currently connected storage node are transmitted to a specified number of slave nodes, and returns 0 if data synchronization of the master node and the slave nodes is not completed within a timeout time. Illustratively, the wait instruction request may be a wait request.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, necessary security measures are taken, and the commonness and the customs are not violated. In the technical scheme of the disclosure, the operations of obtaining, storing, applying and the like of the related user personal information all obtain the authorization of the user.

In the prior art, after receiving an operation request from a client, a storage node operates a master node, and directly returns a response result of a master ground point based on the operation request to the client after the master node completes operation. When the master node in the storage node cannot be used, the slave node is directly used as a standby master node to realize the function of the master node, however, due to the asynchronous replication strategy adopted between the master node and the slave node, when the master node is switched to the slave node, the operation request is lost with a certain probability. In addition, if the synchronization status of the slave node is to be waited for, and a response result of synchronization completion is returned to the client after the synchronization of the slave node is completed, when the slave node is damaged or cannot be connected, the client cannot receive the response result, the waiting time is consumed, the availability of the slave node cannot be ensured, and a large amount of connection resources are occupied due to the corresponding result of waiting for synchronization completion, so that the efficiency of data transmission is reduced.

In order to solve the above problems in the related art, embodiments of the present disclosure provide a data processing method, an apparatus, an electronic device, and a storage medium, which may be applied to a proxy end, to implement real-time acquisition of an available state of a slave node, and simultaneously send a merge request including an operation request and a wait instruction request to a storage node based on the available state of the slave node, so that the storage node may complete a data synchronization process of a master node and the slave node, ensure data consistency of the master node and the slave node, reduce occupation of connection resources, and effectively improve efficiency of data transmission. The data processing method comprises the following steps: determining a storage node corresponding to an operation request according to the operation request acquired from a client, wherein the storage node comprises a master node and M slave nodes, and M is a positive integer; processing the operation request and the waiting instruction request according to the real-time available states of the M slave nodes to generate a merging request, wherein the real-time available states of the M slave nodes are acquired in real time through a monitoring component communicated with a registration center; according to the merging request, establishing a dynamic connection relation between the agent end and the storage node, and removing the dynamic connection relation when the dynamic connection relation meets a preset condition; and sending the merging request to the storage node according to the dynamic connection relation, receiving and forwarding a response result generated based on an execution result of the storage node by the proxy, wherein the execution result is generated by the master node and the slave node of the storage node according to the merging request.

According to the embodiment of the disclosure, the data processing method can be applied to a proxy end, a merging request is generated by processing an operation request and a waiting instruction request from a client according to the acquired real-time available state of a slave node, the merging request is sent to a storage node according to the established dynamic connection relation, and a response result generated based on the execution result of the storage node is received. On the other hand, as the execution result is that the master node and the slave node of the storage node are generated according to the merging request, and each merging request corresponds to one dynamic connection relation, the data consistency of the master node and the slave node can be ensured during operation, and the dynamic connection relation is adopted, so that the operation request is ensured not to be blocked, and the data processing efficiency is improved.

Fig. 1 schematically shows an exemplary system architecture to which the data processing method may be applied, according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios. It should be noted that the data processing method, the data processing apparatus, the electronic device, and the readable storage medium provided in the embodiments of the present disclosure may be used in related aspects in the data processing technology field and the financial field, and may also be used in various fields other than the financial field.

As shown in fig. 1, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, a network 103, a proxy server 104, and a database server 105.

The network 103 serves as a medium for providing communication links between the

terminal devices

101, 102 and the proxy server 104 and the database server 105. Network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

Users may use the

terminal devices

101, 102 to interact with the proxy server 104 and the database server 105 over the network 103 to receive or transmit data, instructions, etc. The

terminal devices

101 and 102 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, etc. (for example only), and the

terminal devices

101 and 102 are configured to obtain data from the database server 105 or send data requests to the database server 105.

The

terminal devices

101, 102 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The proxy server 104 may provide an intermediate service between the

terminal devices

101, 102 and the database server 105.

The database server 105 may be a server that provides various database services, such as a database server that provides data support for data of websites browsed by users using the terminal devices 101, 102 (for example only). The database server may analyze and the like the received data such as the user request, and feed back the data (e.g., the web page, information, or data obtained or generated according to the user request) in the database server to the terminal device according to the user request.

It should be noted that the data processing method provided by the embodiment of the present disclosure may be generally executed by the proxy server 104, and accordingly, the data processing apparatus provided by the embodiment of the present disclosure may be generally disposed in the proxy server 104. The data processing method provided by the embodiment of the present disclosure may also be performed by other proxy servers different from the proxy server 104 and capable of communicating with the

terminal devices

101 and 102 and/or the database server 105. Accordingly, the data processing apparatus provided by the embodiment of the present disclosure may also be disposed in other proxy servers different from the

terminal devices

101 and 102 and capable of communicating with the

terminal devices

101 and 102 and/or the database server 105.

It should be understood that the number of terminal devices, network and proxy servers, database servers in fig. 1 are merely illustrative. There may be any number of terminal devices, network and proxy servers, database servers, as desired for implementation.

Fig. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure. The data processing method of the embodiment of the present disclosure may be applied to a Proxy, for example, a Proxy module Proxy. As shown in fig. 2, the flow 200 of the data processing method of the embodiment of the present disclosure includes operations S210 to S240.

In operation S210, according to an operation request obtained from a client, a storage node corresponding to the operation request is determined, where the storage node includes a master node and M slave nodes, and M is a positive integer.

In the embodiment of the present disclosure, the client sends a request packet to the proxy, where the request packet includes an operation request, for example, a write request, that is, an operation request for writing corresponding data into the storage node. After receiving the operation request from the client, the agent performs an operation such as parsing on the operation request, thereby determining a storage node corresponding to the operation request.

Each operation request has a corresponding storage node, and by determining the corresponding storage node, the operation can be effectively performed on the master node and the slave node in the storage nodes. For example, the method includes determining the available states of the master node and the slave node, or performing operations on the master node and the slave node according to the operation requests, or obtaining the response results of the corresponding master node and the slave node, and so on.

In the embodiment of the present disclosure, each storage node has a master node and one or more slave nodes serving as backup nodes of the master node, and after the master node fails (e.g., is damaged or unavailable), the slave nodes are switched to the master node to replace the master node to implement corresponding functions, thereby ensuring high availability performance of the system.

According to the embodiment of the disclosure, the master node and the M slave nodes of the storage node can be switched, so that when the client sends the operation request to the storage node, the high availability of the storage node is ensured, and the data consistency in the data processing process is improved.

In operation S220, the operation request and the wait instruction request are processed according to the real-time availability statuses of the M slave nodes, which are obtained in real time by the monitoring component in communication with the registry, to generate a merge request.

In the embodiment of the disclosure, the real-time available states of M slave nodes of the storage node are acquired in real time through the monitoring component, and the operation request and the instruction waiting request are processed according to the real-time available states of the M slave nodes.

Illustratively, the real-time available states of the slave nodes comprise available states and unavailable states, and when the number of the slave nodes of which the real-time available states are available in the M slave nodes is greater than or equal to 1, the operation requests and the waiting instruction requests are processed to generate the merging requests.

And when the number of slave nodes which are available in the real-time available state in the M slave nodes is less than 1, processing the operation request and the waiting instruction request, and directly sending the operation request to the master node of the storage node.

In the embodiment of the disclosure, a wait instruction request (e.g., wait request) is used to synchronize data written by the master node to the slave node, and after the slave node completes synchronization, the result of synchronization is fed back to the proxy side, and the final result is fed back to the client side by the proxy side, so that the client side can perform further operation according to the fed-back result. When at least one slave node is available, a waiting instruction request needs to be sent to the slave node so as to obtain the data synchronization result of the slave master node of the slave node. When the number of available slave nodes is 0, the data synchronization result of the slave nodes does not need to be acquired at this time, and the execution result of the operation request sent by the master node based on the client of the storage node is directly fed back to the client. The efficiency of data processing can be effectively improved without waiting.

In an embodiment of the present disclosure, the wait instruction request is sent by the agent to the storage node, and is used to determine a synchronization result of synchronizing data from the master node to the slave node in the storage node. For example, when the storage node includes at least one available slave node, the master node feeds back a result of the operation request to the agent side based on the operation request, the slave node requests the master node to synchronize data to the slave node based on the wait instruction, and after the synchronization is completed, feeds back a synchronization result to the agent side. Therefore, the data consistency of the slave node and the master node can be ensured, namely, after the slave node completes the data synchronization from the master node to the slave node based on the operation request, the corresponding result is fed back to the client.

In operation S230, a dynamic connection relationship between the agent and the storage node is established according to the merge request, and the dynamic connection relationship is released when a preset condition is satisfied.

In an embodiment of the present disclosure, when a merge request is generated, it is indicated that there are available slave nodes included in the storage node. When there is an available node, a synchronization result of synchronizing data from the master node to the slave node needs to be obtained, and for each operation request, a data synchronization result between the master node and the slave node corresponding to the operation request needs to be obtained. In this regard, according to the present disclosure, by establishing a dynamic connection relationship between the agent and the storage node, after the master node feeds back an execution result to the agent based on the operation request of the client, the master node simultaneously feeds back an execution result of the slave node based on the operation request and the instruction waiting request to the agent through the same dynamic connection relationship, that is, a data synchronization result between the slave node and the master node is fed back to the agent. And after the agent end acquires the execution result of the main node and the synchronization result of the slave node, the dynamic connection relation is released, so that the occupation condition of system resources is reduced.

For example, the operation request of the client may be a write request, for example, when M slave nodes have one or more available slave nodes, a merge request is generated according to the operation request (e.g., write request) and the request for waiting for an instruction, and a dynamic connection relationship between the proxy and the storage node is established, that is, a connection is newly established according to each merge request, and the concurrency is improved by increasing the number of connections, so as to achieve a processing capability with higher throughput. Because the newly-established connection is a dynamic connection relation, the dynamic connection relation is released when the preset condition is met.

In the embodiment of the present disclosure, the preset condition includes that the time for establishing the dynamic connection relationship exceeds the set time; and the slave node generates an execution result according to the combination request.

For example, the dynamic connection relationship is periodically detected and monitored, when the time for establishing the dynamic connection relationship exceeds the set time, it indicates that the establishment of the dynamic connection between the agent and the storage node cannot be completed within the set time, and at this time, the establishment process of the dynamic connection relationship is released, or the dynamic connection relationship is released.

For another example, when the slave node generates an execution result based on the merge request, that is, when the slave node synchronizes data of the master node to the slave node based on the wait instruction request in the merge request, and generates an execution result based on the result of the synchronization, the dynamic connection relationship is released. The dynamic connection relation is released, so that the occupation condition of system resources is reduced.

In operation S240, the merge request is sent to the storage node according to the dynamic connection relationship, and the agent receives and forwards a response result generated based on an execution result of the storage node, where the execution result is generated by the master node and the slave node of the storage node according to the merge request.

In the embodiment of the disclosure, the merging request is sent to the storage node based on the established dynamic connection relationship, and the agent receives a response result generated by the storage node according to the execution result of the merging request based on the dynamic connection relationship.

Illustratively, after the merge request is sent to the storage node, the master node in the storage node generates one execution result based on the operation request in the merge request, and the slave node synchronizes data in the master node to the slave node based on the wait instruction request and generates another execution result. And processing the two execution results to generate a response result, wherein the response result is used for feeding back to the client.

Fig. 3 schematically shows a flow chart of a data processing method in acquiring an available state of a slave node according to an embodiment of the present disclosure.

As shown in fig. 3, the data processing method of the embodiment of the present disclosure further includes operation S300. In operation S300, the availability status of each of the M slave nodes obtained by the monitoring component in real time is obtained from the registry before the operation request and the wait instruction request are processed to generate the merge request according to the real-time availability statuses of the M slave nodes.

In the embodiment of the disclosure, the monitoring component monitors the available state of the slave node in the storage node, and updates the available state of the slave node in the registry in real time, so that the agent end can acquire the available state of each slave node from the registry in real time. Further, whether to generate a merge request may be determined according to the available state of the slave node, and when receiving the response result, the number of execution results obtained by the slave node based on the merge request may be determined, for example, when the number of the slave nodes is N (N is a positive integer less than or equal to M), the number of the execution results returned by the slave node is N, and it may be considered that the process of performing data synchronization to the N slave nodes by the master node is completed.

Fig. 4 schematically illustrates a flow diagram of a data processing method at operation 400 according to an embodiment of the present disclosure.

As shown in fig. 4, the data processing method according to the embodiment of the present disclosure further includes a process 400, where the process 400 is an operation before determining a storage node corresponding to an operation request according to the operation request acquired from the client, and the process 400 includes operations S410 to S420.

In operation S410, an operation request is acquired from a client.

In an embodiment of the present disclosure, the operation request may be, for example, a write request, and the storage node may write data to the primary node based on the write request.

In operation S420, after receiving an operation request acquired from a client, client information and request data associated with the operation request are stored in a target queue.

According to the embodiment of the disclosure, the client information and the request data associated with the operation request are stored in the target queue, so that the associated request data can be conveniently processed after the storage node receives the operation request. And feeding back the response result to the client based on the client information after receiving the response result.

In the embodiment of the disclosure, since the operation request and the wait instruction request of the client are generated as the merge request, the agent receives two response results of the storage node based on the merge request, that is, a response result based on the operation request and a response result based on the wait instruction request. According to the principle of queue pop at the agent end, after a response result is received each time, a client request object (for example, request data) is popped from the queue, and after a merge request is generated, the client request object (for example, request data) is repeatedly added into the queue.

Fig. 5 schematically shows a flowchart of a data processing method according to an embodiment of the present disclosure at operation S210.

As shown in fig. 5, operation S210 includes operations S211 through S212.

In operation S211, the operation request obtained from the client is parsed according to a preset algorithm, and the target data is determined.

In the embodiment of the disclosure, after an operation request of a client is received, the obtained operation request is analyzed. For example, the predetermined algorithm may be a routing algorithm, differentiated according to a plurality of characteristics. And analyzing the operation request to obtain a key value in the operation request, wherein the key value is the target data.

In operation S212, a storage node corresponding to the operation request is determined according to the target data.

For example, according to the determined key value, a storage node corresponding to the operation request is calculated and determined, and the storage node is provided with a master node and M slave nodes.

Fig. 6 schematically shows a flowchart of a data processing method according to an embodiment of the present disclosure in operation S220.

As shown in fig. 6, operation S220 includes operations S221 through S222.

In operation S221, the number of available slave nodes is counted according to the acquired available state of each of the M slave nodes.

In the embodiment of the disclosure, the availability status of the slave node includes available and unavailable, when the availability status of the slave node is available, the number of the slave nodes is counted, and when the availability status of the slave node is unavailable, the number of the slave nodes is not counted.

According to the embodiment of the disclosure, the reliability in the storage node can be effectively improved by counting the number of available slave nodes, for example, when no available node exists in the slave nodes in the storage node, maintenance can be reminded, and the problem of data loss or service interruption when a master node fails is prevented. In an embodiment of the present disclosure, by counting the number of available slave nodes, the number of slave node execution results when the receiving storage node is based on the merge request may also be determined. For example, when there are 2 available nodes, the number of execution results of the receiving slave nodes based on the merge request is 2, and when the number of the receiving slave nodes is less than 2, it indicates that the slave nodes do not execute the merge request, and a wait operation may be performed, so as to ensure that each available slave node can execute the merge request.

In operation S222, when the number of available slave nodes is greater than the set threshold, the operation request and the wait instruction request are processed to generate a merge request.

In the embodiment of the present disclosure, the set threshold may be a positive integer greater than 0, for example, when the set threshold is 0, and the number of available nodes is 1, the merge request is generated for processing the operation request and the wait instruction request.

In alternative embodiments, the set threshold may be other suitable values.

Fig. 7 schematically shows a flowchart of a data processing method at the time of dynamic connection relation release according to an embodiment of the present disclosure.

As shown in fig. 7, the data processing method of the embodiment of the present disclosure further includes operation S500.

In operation S500, when the dynamic connection relation is released, the request queue associated with the dynamic connection relation is deleted, and the stored memory information associated with the dynamic connection relation is released.

According to an embodiment of the present disclosure, at dynamic connection relationship establishment, client information and request data associated with the operation request are stored to a target queue, which may be a request queue associated with the dynamic connection relationship, according to the above. When the dynamic connection relation is released, the data in the queue can not be executed. And deleting the request queue associated with the dynamic connection relation at the moment, and releasing the stored memory information associated with the dynamic connection relation. The occupation of memory resources can be effectively saved, and the problem of memory leakage caused by no response of the memory layer is avoided.

Fig. 8 schematically illustrates a flow diagram of a data processing method at operation 600, according to an embodiment of the present disclosure.

In the embodiment of the present disclosure, in operation S240, sending the merge request to the storage node according to the dynamic connection relationship, the process 600 is further included: the proxy receives and forwards a response result generated based on the execution result of the storage node. The process 600 includes operations S610 to S630.

In operation S610, the agent receives a first execution result, where the first execution result is generated by the master node according to the operation request in the merge request.

In the embodiment of the disclosure, after the merge request is sent to the storage node, the master node in the storage node executes according to the operation request in the merge request, and generates a first execution result, for example, if the first execution result is OK, it indicates that the master node executes successfully based on the operation request. The agent end receives the first execution result.

In operation S620, the agent receives a second execution result, where the second execution result is generated by the slave node synchronizing data of the master node to the M slave nodes according to the wait instruction request in the merge request.

In the embodiment of the present disclosure, after the slave node in the storage node generates the first execution result at the master node, the slave node synchronizes the data of the master node to the M slave nodes according to the wait instruction request in the merge request, and generates the second execution result. In this embodiment, the second execution result indicates that the data of the master node has been synchronized to all the nodes of the M slave nodes, and in an alternative embodiment, the second execution result may indicate that the data of the master node is synchronized to at least one of the M slave nodes.

In operation S630, the agent generates a response result according to the received first execution result and the second execution result, and forwards the response result to the client.

In the embodiment of the disclosure, after receiving the first execution result and the second execution result of the storage node, the agent generates a response result, where the response result may indicate an execution operation request of a master node in the storage node, and in this embodiment, the response result also indicates that data of the master node is all synchronized to the slave node, and after receiving the response result, the client may perform other operations, where data of the master node and data of the slave node are kept consistent, and when the master node fails and is switched to the slave node, there is no data loss, so that data consistency of the master node and the slave node is achieved.

Fig. 9 schematically shows an implementation process diagram of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 9, the data processing method is executed by a client 701, an agent 702, and a storage node 703.

The client 701 sends an operation request 704 to the proxy 702, and the operation request 704 may be, for example, a write operation. After receiving an operation request 704 from a client 701, an agent 702 determines a storage node 703 corresponding to the operation request 704, where the storage node 703 includes a master node and M slave nodes. The agent 702 obtains the real-time availability statuses of the M slave nodes in real time through a monitoring component in communication with the registry, and processes the operation request 704 and the wait instruction request 705 based on the real-time availability statuses of the M slave nodes to generate a merge request 706.

After the merge request 706 is generated, a dynamic connection relationship between the agent 702 and the storage node 703 is established according to the merge request, the merge request 706 is sent to the storage node through the dynamic connection relationship, and an execution result of the storage node 703 based on the merge request 706 is returned based on the dynamic connection relationship.

The master node in the storage node 703 generates a first execution result 707 based on the operation request in the merge request, and the slave node in the storage node 703 synchronizes data of the master node to the slave node based on the wait instruction request in the merge request to generate a second execution result 708.

The agent generates a response result 709 according to the received first execution result 707 and the second execution result 708, and forwards the response result to the client 701.

According to the embodiment of the disclosure, the availability of the slave nodes in the storage nodes can be ensured by acquiring the availability states of the slave nodes in real time, and the problem that the operation requests of the master node and the slave nodes fail or are delayed due to the fact that the waiting instruction requests of the slave nodes cannot respond when the slave nodes are unavailable is avoided. By generating the merge request, network transmission resource consumption of the operation request can be reduced, and the data processing speed can be increased. The agent terminal generates a response request according to the received first execution result and the second execution result, and forwards the response request to the client terminal, so that the consistency of the data of the main node and the slave node is ensured, and the dynamic connection relation is adopted, so that the operation request is ensured not to be blocked, and the data processing efficiency is improved.

Fig. 10 schematically shows a response result of a wait instruction request in a merge request of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 10, the slave node responds based on the wait instruction request in the merge request, for example, when all data of the master node are synchronized to the slave node, the response is successful, and the response gives the successful result of the operation request to the client. And when the data of the master node is not synchronized to the slave node, the response is unsuccessful, and whether the operation request based on the combination request of the master node is successful is further judged. When the master node succeeds in responding to the operation request (such as a write request), the error result is customized to the client, the master node completes the operation request, and the slave node does not complete the synchronization of the data of the master node to the slave node. When the master node is unsuccessful in responding to the operation request (e.g., write request) from the client, the master node does not complete the operation request and the slave node does not play the data synchronization.

Fig. 11 schematically shows a communication relationship diagram of an agent side, a registry and a monitoring component in the data processing method according to the embodiment of the disclosure.

As shown in fig. 11, the monitoring component monitors the master node and the slave node in the storage node in real time, specifically, obtains the available state of the slave node in real time, and updates the available state of the slave node to the registry, and after receiving the operation request from the client, the agent obtains the available state of the slave node in the storage node corresponding to the operation request in real time through the registry. The number of the slave nodes to be synchronized can be determined according to the available states of the slave nodes, so that the execution result generated by the storage node based on the instruction waiting request in the merging request is received based on the number of the available slave nodes, and the response to the synchronization state of the slave nodes is accurately realized.

Fig. 12 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

As shown in fig. 12, the data processing apparatus 800 includes a determining module 810, a generating module 820, a processing module 830, and a receiving and forwarding module 840.

The determining module 810 is configured to determine, according to an operation request obtained from a client, a storage node corresponding to the operation request, where the storage node includes a master node and M slave nodes, and M is a positive integer. In an embodiment, the determining module 810 may be configured to perform the operation S210 described above, which is not described herein again.

A generating module 820 configured to process the operation request and the wait instruction request according to real-time availability statuses of M slave nodes, which are obtained in real time by a monitoring component in communication with the registry, and generate a merge request. In an embodiment, the generating module 820 may be configured to perform the operation S220 described above, which is not described herein again.

The processing module 830 is configured to establish a dynamic connection relationship between the agent and the storage node according to the merge request, and the dynamic connection relationship is released when a preset condition is met. In an embodiment, the processing module 830 may be configured to perform the operation S230 described above, and is not described herein again.

The receiving and forwarding module 840 is configured to send the merge request to the storage node according to the dynamic connection relationship, and the agent receives and forwards a response result generated based on an execution result of the storage node, where the execution result is generated by the master node and the slave node of the storage node according to the merge request. In an embodiment, the receiving and forwarding module 840 may be configured to perform the operation S240 described above, which is not described herein again.

In some exemplary embodiments of the disclosure, the determining module comprises a determining submodule configured to: analyzing the operation request acquired from the client according to a preset algorithm to determine target data; and determining a storage node corresponding to the operation request according to the target data.

In some exemplary embodiments of the disclosure, the generation module comprises a generation submodule configured to: counting the number of available slave nodes according to the acquired available state of each node in the M slave nodes; and when the number of the available slave nodes is larger than a set threshold value, processing the operation request and the waiting instruction request to generate a combined request.

In some exemplary embodiments of the present disclosure, the receiving and forwarding module further comprises a receiving and forwarding sub-module configured to: the agent end receives a first execution result, and the first execution result is generated by the main node according to the operation request in the combination request; the agent end receives a second execution result, and the second execution result is generated by synchronizing the data of the master node to M slave nodes according to the waiting instruction request in the merging request; and the agent terminal generates a response result according to the received first execution result and the second execution result and forwards the response result to the client terminal.

According to the embodiment of the present disclosure, any multiple modules of the determining module 810, the generating module 820, the processing module 830 and the receiving and forwarding module 840 may be combined into one module to be implemented, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the determining module 810, the generating module 820, the processing module 830, and the receiving and forwarding module 840 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented in any one of three implementations of software, hardware, and firmware, or in a suitable combination of any of them. Alternatively, at least one of the determining module 810, the generating module 820, the processing module 830, the receiving and forwarding module 840 may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

Fig. 13 schematically shows a block diagram of an electronic device for implementing a data processing method according to an embodiment of the present disclosure. The electronic device shown in fig. 13 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 13, an electronic apparatus 900 according to an embodiment of the present disclosure includes a processor 901 which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage portion 908 into a Random Access Memory (RAM) 903. Processor 901 may comprise, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 901 may also include on-board memory for caching purposes. The processor 901 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 903, various programs and data necessary for the operation of the electronic apparatus 900 are stored. The processor 901, ROM 902, and RAM 903 are connected to each other by a bus 904. The processor 901 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 900 may also include input/output (I/O) interface 905, input/output (I/O) interface 905 also connected to bus 904, according to an embodiment of the present disclosure. The electronic device 900 may also include one or more of the following components connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. A drive 910 is also connected to the I/O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary so that a computer program read out therefrom is mounted into the storage section 908 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be embodied in the device/apparatus/system described in the above embodiments; or may exist alone without being assembled into the device/apparatus/system. The above-mentioned computer-readable storage medium carries one or more programs which, when executed, implement the data processing method according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 902 and/or the RAM 903 described above and/or one or more memories other than the ROM 902 and the RAM 903.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. When the computer program product runs in a computer system, the program code is used for causing the computer system to realize the data processing method provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 901. The above described systems, devices, modules, units, etc. may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, and downloaded and installed through the communication section 909 and/or installed from the removable medium 911. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 909 and/or installed from the removable medium 911. The computer program, when executed by the processor 901, performs the above-described functions defined in the system of the embodiment of the present disclosure. The above described systems, devices, apparatuses, modules, units, etc. may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the disclosure, and these alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A method of data processing, comprising:

determining a storage node corresponding to an operation request according to the operation request obtained from a client, wherein the storage node comprises a main node and M slave nodes, and M is a positive integer;

processing the operation request and the waiting instruction request according to the real-time available states of the M slave nodes to generate a merging request, wherein the real-time available states of the M slave nodes are acquired in real time through a monitoring component communicated with a registration center;

establishing a dynamic connection relation between the agent end and the storage node according to the merging request, wherein the dynamic connection relation is released when a preset condition is met;

and sending the merging request to a storage node according to the dynamic connection relation, and receiving and forwarding a response result generated based on an execution result of the storage node by the agent, wherein the execution result is generated by a main node and a slave node of the storage node according to the merging request.

2. The method of claim 1, further comprising:

before the operation request and the waiting instruction request are processed according to the real-time available states of the M slave nodes to generate a combined request,

obtaining, from the registry, the availability status of each of the M slave nodes obtained in real-time by the monitoring component.

3. The method of claim 1, further comprising:

before determining a storage node corresponding to an operation request according to the operation request acquired from a client,

acquiring an operation request from a client;

after receiving an operation request acquired from a client, storing client information and request data associated with the operation request to a target queue.

4. The method of claim 1, wherein,

the determining, according to an operation request acquired from a client, a storage node corresponding to the operation request includes:

analyzing the operation request acquired from the client according to a preset algorithm to determine target data;

and determining a storage node corresponding to the operation request according to the target data.

5. The method of claim 1, wherein,

the processing the operation request and the instruction waiting request according to the real-time available states of the M slave nodes, and generating a merge request includes:

counting the number of available slave nodes according to the acquired available state of each node in the M slave nodes;

and when the number of the available slave nodes is larger than a set threshold value, processing the operation request and the waiting instruction request to generate a combined request.

6. The method of claim 1, wherein the preset conditions include:

the time for establishing the dynamic connection relation exceeds the set time; and

and the slave node generates an execution result according to the combination request.

7. The method of claim 5, further comprising:

and when the dynamic connection relation is released, deleting the request queue associated with the dynamic connection relation, and releasing the stored memory information associated with the dynamic connection relation.

8. The method of claim 1, wherein,

the proxy receives and forwards a response result generated based on the execution result of the storage node, and the response result comprises:

the agent end receives a first execution result, and the first execution result is generated by the main node according to the operation request in the combination request;

the agent end receives a second execution result, and the second execution result is generated by synchronizing the data of the master node to M slave nodes according to the waiting instruction request in the merging request;

and the agent terminal generates a response result according to the received first execution result and the second execution result and forwards the response result to the client terminal.

9. A data processing apparatus comprising:

the determining module is configured to determine a storage node corresponding to an operation request according to the operation request acquired from a client, wherein the storage node comprises a master node and M slave nodes, and M is a positive integer;

the generation module is configured to process the operation request and the waiting instruction request according to the real-time available states of the M slave nodes, and generate a combination request, wherein the real-time available states of the M slave nodes are acquired in real time through a monitoring component communicated with a registration center;

the processing module is configured to establish a dynamic connection relation between the agent end and the storage node according to the merging request, and the dynamic connection relation is released when a preset condition is met;

and the receiving and forwarding module is configured to send the merging request to the storage node according to the dynamic connection relation, the proxy receives and forwards a response result generated based on an execution result of the storage node, and the execution result is generated by a master node and a slave node of the storage node according to the merging request.

10. An electronic device, comprising:

one or more processors;

storage means for storing executable instructions which, when executed by the processor, implement the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, implement the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.