Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments obtained based on the embodiments in the disclosure belong to the protection scope of the disclosure.
In view of the above-mentioned technical problems that the existing local cache processing method is likely to cause data asynchronism between clients and has poor disaster tolerance capability, the present disclosure provides a local cache processing method, an apparatus, a device, and a computer-readable storage medium.
It should be noted that the present disclosure provides local caching processes, apparatuses, devices and computer readable storage media that can be utilized in various local caching processing scenarios.
In practical applications, the cache data in each client in the cluster is basically inconsistent due to request load balancing, which is very disadvantageous for some specific scenarios. For example, after a user cache is deleted in a client, the user information cannot be synchronized to all local caches in the cluster, and sometimes the user information cannot be completely cleaned from the local caches until the expiration time is reached. In addition, the local caches in the two clients in the cluster cannot be used for mutual operation because the local caches cannot sense the existence of the other client.
In the process of solving the technical problem, the inventors found through research that, in order to achieve synchronization of the cache between the clients in the cluster, a connection relationship between the clients in the cluster can be established. After the connection is established, when any client involves data change, a data synchronization request can be sent, so that other clients perform data synchronization operation according to the data synchronization request, and the data in the updated local cache can be guaranteed to be consistent with the data in the local cache of the target client.
The noun explains:
local caching: a caching component that can run on a client.
P2P communication: point-to-point communication, a communication method for removing servers, a node holding a communication list can access a remote node through an address in the list.
Netty: network communication assembly realized based on Java.
Mina: a network communication component.
Hashmap: hash table, a hash structure that can store data. Generally acting as a local cache, non-thread-safe.
guava cache: the local caching component of guava.
Cache penetration: the cache has no data, and the data is required to be sent to the database and then written back to the cache.
Buffering avalanche: a large number of requests penetrate directly through the cache to the database, causing the database to crash. Thereby triggering a process in which the entire cluster gradually crashes.
Fig. 1 is a schematic diagram of a network architecture based on the present disclosure, and as shown in fig. 1, the network architecture based on the present disclosure at least includes: the system comprises a server 1 and a plurality of clients 2, wherein a local cache processing device is arranged in the server, and the local cache processing device can be written by adopting languages such as C/C + +, Java, Shell or Python.
Fig. 2 is a schematic flow chart of a local cache processing method according to an embodiment of the present disclosure, and as shown in fig. 2, the method includes:
step 101, establishing a communication connection with each client in the cluster.
The execution subject of this embodiment is a local cache processing device, and the local cache processing device may be coupled to a server.
In order to achieve the synchronization of the data in the local cache of each client in the cluster, when any local cache performs data update, the data update operation may be performed according to the content synchronization of the local cache update. Therefore, a communication connection needs to be established with each client in the cluster first, so that the sending and receiving of the data synchronization request can be realized subsequently.
In particular, a communication connection may be established with each client in the cluster via a point-to-point communication technique. Each local cache in the cluster stores the connection addresses of other local caches, so that sensing and data transmission can be conveniently carried out. The communication capacity of the local cache is given by utilizing the bottom-layer netty component, and strong cluster sensing and control functions can be realized through the communication capacity and a proper communication protocol.
Step 102, obtaining a data synchronization request sent by any client in the cluster, where the data synchronization request includes a cache to be synchronized and identification information of a target client that initiated the data synchronization request.
In this embodiment, after establishing a communication connection with each client in the cluster, information interaction with each client may be performed. Specifically, when any client has a data update, in order to ensure that the data in other local caches is kept synchronized, a data synchronization request may be sent, where the data synchronization request may include identification information of a target client that currently has the data update and a cache to be synchronized corresponding to the client. Accordingly, the local cache processing device may obtain the data synchronization request sent by the target client.
And 103, updating the data in the local cache according to the cache to be synchronized in the data synchronization request, so that the updated data in the local cache is consistent with the data in the local cache of the target client.
In this embodiment, after the data synchronization request is acquired, the data in the local cache may be updated according to the cache to be synchronized in the data synchronization request.
Specifically, the data update of the target client may be data writing, accordingly, the cache to be synchronized may be content that needs to be written into the local cache, and after the data synchronization request is obtained, the content that needs to be cached in synchronization may be written into the local cache. The data update of the target client may be a data deletion operation, the content in the cache to be synchronized may be identification information of the cache to be deleted, and after the data synchronization request is obtained, the identification information of the cache to be deleted may perform a deletion operation on the cache to be deleted.
Accordingly, after the data in the local cache is updated according to the data synchronization request, the updated data in the local cache is consistent with the data in the local cache of the target client. So that the synchronicity of the data in the local caches of the clients can be guaranteed in any case.
Fig. 3 is a schematic diagram of connection of local caches provided in the embodiment of the present disclosure, and as shown in fig. 3, each local cache is in communication connection with other local caches in a cluster, so that information interaction can be achieved.
In the local cache processing method provided in this embodiment, a communication connection is established with each client in the cluster, a data synchronization request sent by each client is obtained, and data synchronization operation is performed according to the data synchronization request, so that it is ensured that data in the updated local cache is consistent with data in the local cache of the target client, the problem of local cache data in each client being asynchronous is effectively avoided, and the disaster tolerance capability is strong.
Fig. 4 is a schematic flow chart of a local cache processing method according to a second embodiment of the present disclosure, and on the basis of the first embodiment, as shown in fig. 4, after step 101, the method further includes:
step 201, obtaining status update information sent by any client in the cluster, where the status update information includes identification information of the client sending the status update information and a target status of the client sending the status update information.
Step 202, updating the state of the client sending the state updating information in a cache list pre-stored in a local cache to the target state according to the state updating information.
In this embodiment, since the operating status of each client in the cluster may be different, for example, an individual client may go offline due to some emergency. Therefore, a cache list including the connection address and the operation state of each local cache in the cluster may be stored in the local cache in advance.
In order to accurately determine the operating state of each client, monitoring operation can be performed in real time to obtain state update information sent by any client in the cluster, wherein the state update information includes identification information of the client sending the state update information and a target state of the client sending the state update information. The target status may specifically include an online status and an offline status.
After the state update information is acquired, the state of the client sending the state update information in the pre-stored cache list may be updated to the target state according to the state information.
For example, in practical applications, the local cache processing device may listen to the status update information sent by the clients in the cluster in real time. Specifically, the status update information may be an online information, and the online information may be a custom online protocol of 0x01 ip. If the local cache processing device receives the online information, the state of the client sending the state updating information in the cache list can be updated to be the online state. The status update information may be offline information, and the online information may specifically be 0x02 ip offline protocol. When receiving the offline message, the other clients set the client sending the status update information in the respective cache list to be in the offline status. Subsequent synchronized data operations will not be sent to the client that is offline.
According to the local cache processing method provided by the embodiment, the state updating information sent by each client is monitored, and the state of the client in the pre-stored cache list is updated according to the state updating instruction, so that the running state of each client can be accurately determined, data synchronization operation can be performed according to the running state, and the accuracy of data synchronization is improved.
Further, on the basis of any of the above embodiments, after the step 101, the method further includes:
and acquiring heartbeat packets sent by each client according to a preset first time interval, and updating the machine heartbeat activity detection counts of the clients sending the heartbeat packets in a pre-stored cache list according to the heartbeat packets.
In this embodiment, each client in the cluster may send a heartbeat packet to the cluster according to a preset first time interval to prove that the client is in a alive state. Wherein the first time interval may be one minute. Correspondingly, the local cache processing device can monitor whether a client sends a heartbeat packet in real time, and update the machine heartbeat activity detection count of the client sending the heartbeat packet in the pre-stored cache list according to the heartbeat packet.
Further, on the basis of any of the above embodiments, the method further includes:
and for each client in the cluster, if the heartbeat packet of the client is not received within a preset second time interval, updating the state of the client which does not send the heartbeat packet in a cache list prestored in a local cache to be an offline state.
In this embodiment, for each client in the cluster, if the heartbeat packet is not sent to the cluster beyond a preset second time interval, the client may be determined to process a dropped state, where the preset second time interval may be five minutes. Therefore, for each client in the cluster, if the local cache processing device does not receive the heartbeat packet of the client beyond the preset second time interval, the state of the client which does not send the heartbeat packet in the cache list pre-stored in the local cache is updated to be an offline state. Correspondingly, if the state of the client is judged to be the offline state, the client will not be sent to the offline client when a subsequent client performs data updating.
Different from the existing local cache processing method, when the local cache is hung for various reasons, a user is required to capture the exception, and then the cache is reestablished to perform data flash. For low concurrency applications, this data flush may cause cache breakthrough without much impact, but for high concurrency applications, this cache breakthrough will place great pressure on the underlying services, even causing cache avalanches. According to the local cache processing method provided by the embodiment, the state of each client is updated according to the time interval of sending the heartbeat packet by each client, so that the state of each client can be automatically and accurately determined, the disaster tolerance capability of the local cache is improved, and cache avalanche is avoided.
According to the local cache processing method provided by the embodiment, the heartbeat packets sent by each client are monitored, and the state of each client is updated according to the time interval of the heartbeat packets sent by each client, so that the state of each client can be accurately determined, and data synchronization operation can be performed according to the state, so that the accuracy of data synchronization is improved.
Fig. 5 is a schematic flow chart of a local cache processing method according to a third embodiment of the present disclosure, and on the basis of any one of the foregoing embodiments, as shown in fig. 5, after step 101, the method further includes:
step 301, obtaining a pause interaction request sent by any client in the cluster, where the pause interaction request includes identification information of the client sending the pause interaction request.
And step 302, stopping information interaction with the client sending the interaction suspending request according to the interaction suspending request.
In this embodiment, when the network traffic of the whole cluster is larger than a preset threshold and is not rich in bandwidth, the client may send a request for suspending interaction. The pause of network interaction does not affect the function of the local cache, and the local cache can still be normally used, only if no network is operated. And after the network flow is reduced, the network can be opened again.
Accordingly, the local cache processing device may listen to a pause interaction request sent by any client in the cluster, where the pause interaction request includes identification information of the client sending the pause interaction request. And further, according to the interaction suspending request, stopping information interaction with the client sending the interaction suspending request.
According to the local cache processing method provided by the embodiment, when the overall network flow of the cluster is larger than the preset threshold value and is larger than a preset threshold value and the bandwidth is not abundant, the information interaction with the client sending the interaction suspending request is stopped according to the interaction suspending request, so that the bandwidth pressure caused by the information interaction can be avoided, and the use efficiency of the local cache is improved.
Fig. 6 is a schematic structural diagram of a local cache processing apparatus according to a fourth embodiment of the present disclosure, and as shown in fig. 6, the apparatus includes: a connection module 41, a synchronization request obtaining module 42 and a synchronization module 43, wherein the connection module 41 is configured to establish a communication connection with each client in the cluster. A synchronization request obtaining module 42, configured to obtain a data synchronization request sent by any client in the cluster, where the data synchronization request includes a cache to be synchronized and identification information of a target client that initiates the data synchronization request. And a synchronization module 43, configured to perform an update operation on the data in the local cache according to the cache to be synchronized in the data synchronization request, so that the updated data in the local cache is consistent with the data in the local cache of the target client.
Specifically, on the basis of the fourth embodiment, the connection module is configured to:
a communication connection is established with each client in the cluster via a point-to-point communication technique.
The local cache processing apparatus provided in this embodiment establishes a communication connection with each client in the cluster, obtains a data synchronization request sent by each client, and performs data synchronization operation according to the data synchronization request, so as to ensure that data in the updated local cache is consistent with data in the local cache of the target client, thereby effectively avoiding the problem of local cache data in each client being asynchronous, and having a strong disaster tolerance capability.
Further, on the basis of the fourth embodiment, the apparatus further includes: the system comprises an update information acquisition module and an update module, wherein the update information acquisition module is used for acquiring state update information sent by any client in the cluster, and the state update information comprises identification information of the client sending the state update information and a target state of the client sending the state update information. And the updating module is used for updating the state of the client which sends the state updating information in a cache list pre-stored in a local cache into the target state according to the state updating information.
Further, on the basis of the fourth embodiment, the target status includes an online status and a offline status.
Further, on the basis of any one of the above embodiments, the apparatus further includes: and a heartbeat packet acquisition module. The heartbeat packet acquisition module is used for acquiring heartbeat packets sent by the clients according to a preset first time interval, and updating the machine heartbeat sounding counts of the clients sending the heartbeat packets in a pre-stored cache list according to the heartbeat packets.
Further, on the basis of any of the above embodiments, the method further includes: and the state updating module is used for updating the state of the client which does not send the heartbeat packet in the cache list prestored in the local cache to an offline state if the heartbeat packet of the client is not received in the cache list exceeding a preset second time interval for each client in the cluster.
Further, on the basis of any one of the above embodiments, the apparatus further includes: the system comprises an acquisition module and a control module, wherein the acquisition module is used for acquiring a pause interaction request sent by any client in the cluster, and the pause interaction request comprises identification information of the client sending the pause interaction request. And the control module is used for stopping information interaction with the client sending the pause interaction request according to the pause interaction request.
Further, on the basis of any of the above embodiments, the suspend interaction request is sent by the client when detecting that the current network traffic is greater than a preset threshold.
Fig. 7 is a schematic structural diagram of a local cache processing device according to a fifth embodiment of the present disclosure, and as shown in fig. 7, the device includes: a processing component 502, a memory 504, a power component 506, and a communication component 508.
The processing component 502 generally controls overall operation of the device 500, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 502 may include one or more processors 510 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 502 can include one or more modules that facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate interaction between the multimedia component 505 and the processing component 502.
The memory 504 is configured to store various types of data to support operation at the device 500. Examples of such data include instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 504 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component 506 provides power to the various components of the device 500. The power components 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 500.
The communication component 508 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 508 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 508 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 500 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 504 comprising instructions, executable by the processor 520 of the device 500 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
A non-transitory computer-readable storage medium, in which instructions, when executed by a processor of a terminal device, enable the terminal device to perform the above-described local cache processing method of the terminal device.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present disclosure, and not for limiting the same; while the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.