CN118568114B

CN118568114B - Cache refreshing method, device, equipment, computer program product and storage medium

Info

Publication number: CN118568114B
Application number: CN202411035782.2A
Authority: CN
Inventors: 郭立民; 孔维亭; 郭涛
Original assignee: Inspur Jinan data Technology Co ltd
Current assignee: Inspur Jinan data Technology Co ltd
Priority date: 2024-07-31
Filing date: 2024-07-31
Publication date: 2024-10-11
Anticipated expiration: 2044-07-31
Also published as: CN118568114A

Abstract

The invention discloses a cache refreshing method, a device, equipment, a computer program product and a storage medium, belongs to the technical field of cache, and is used for updating a cache according to a read-write operation log of a database, so that the problems of low cache refreshing efficiency and user business invasion are solved. Firstly, characteristic information of a read-write operation event is determined based on a real-time read-write operation log of a database, data of the read-write operation event is converted into a serialized array, then a key value pair set corresponding to the read-write operation event is generated according to table structure information of a target database table to which the read-write operation event belongs and the serialized array of the read-write operation event, and then refreshing operation corresponding to the event type of the read-write operation event can be executed on cache data corresponding to the read-write operation event based on the key value pair set.

Description

Cache refreshing method, device, equipment, computer program product and storage medium

Technical Field

The present invention relates to the field of cache technologies, and in particular, to a cache refreshing method, apparatus, device, computer program product, and storage medium.

Background

With the expansion of service scale, the access pressure of the database is continuously increased, the high concurrency and low delay service demands cannot be met by the database only, for the caching technology, the common data can be stored by the cache to improve the read-write performance, in order to enable related services to be performed normally, the data consistency between the database and the cache needs to be ensured, however, the related technology lacks a mature method, so that the data consistency between the database and the cache is difficult to realize efficiently, and user services possibly need to be invaded.

Therefore, how to provide a solution to the above technical problem is a problem that a person skilled in the art needs to solve at present.

Disclosure of Invention

The invention aims to provide a cache refreshing method, a device, equipment, a computer program product and a storage medium, wherein cache data can be refreshed through a database real-time read-write operation log, so that cache refreshing efficiency is improved, and user business invasion is avoided.

In order to solve the above technical problems, the present invention provides a cache refreshing method, which is applied to a computer system, and includes:

determining characteristic information of a read-write operation event for the read-write operation event in a real-time read-write operation log of a database, and converting data of the read-write operation event into a serialization array, wherein the characteristic information comprises a table name and an event type of a database table to which the characteristic information belongs;

If the database table to which the read-write operation event belongs to a pre-designated target database table, generating a key value pair set corresponding to the read-write operation event according to the serialized array of the read-write operation event and the table structure information of the target database table, wherein the key value pair set comprises a plurality of key value pairs, and any key value pair comprises: a single field value and a field name corresponding to the single field value in the read-write operation event;

determining cache data corresponding to the read-write operation event in a cache according to the primary key information of the read-write operation event in the affiliated database table;

And executing refreshing operation corresponding to the event type of the read-write operation event on the cache data corresponding to the read-write operation event based on the key value pair set of the read-write operation event.

In another aspect, the cache refreshing method further includes:

Responding to a cache reset instruction, and blocking refreshing operation of the first thread;

after the cache is emptied, writing the data of each target database table in the database into the cache;

Unblocking a refresh operation for the first thread;

the first thread is a thread executing a cache refreshing operation, and the cache refreshing operation refers to: and executing refreshing operation corresponding to the event type of the read-write operation event on the cache data corresponding to the read-write operation event based on the key value pair set of the read-write operation event.

On the other hand, in response to the cache reset instruction, blocking the refresh operation of the first thread includes:

Responding to a buffer reset instruction, and sending a first numerical value signal quantity to a buffer so as to block the refreshing operation of a first thread after the buffer recognizes the first numerical value signal quantity;

unblocking a refresh operation for the first thread includes:

And sending the second numerical semaphore to the cache so that the cache can unblock the refreshing operation of the first thread after identifying the second numerical semaphore.

On the other hand, the cache reset instruction comprises a first reset instruction and/or a second reset instruction;

the first reset instruction is: a reset instruction automatically generated at a preset time of day;

The second reset instruction is: and a reset instruction received through a human-computer interface.

On the other hand, the cache and the database are both communicated to preset interface software, so that each application service can call data from the cache or the database through the preset interface software;

the preset interface software comprises a software development kit.

In another aspect, the cache refreshing method further includes:

After the read request sent by the application service through the preset interface software hits the cache, writing the hit target cache data into a second message queue;

judging whether database data corresponding to the target cache data exists in the database or not;

If not, deleting the target cache data from the cache;

if so, judging whether the target cache data is consistent with database data corresponding to the target cache data in the database;

And if the target cache data are inconsistent, updating the target cache data in the cache by taking the database data as a reference.

On the other hand, any node applied in a computer system;

The cache refreshing method further comprises the following steps:

responding to a heartbeat signal generated every other preset period, and increasing preset duration for expiration time of service identifiers corresponding to the service identifiers in a service identifier sequence, wherein the service identifier sequence comprises service identifiers of a plurality of nodes which are orderly arranged, and the service identifiers with expiration time not more than zero are in an expiration state;

For a read-write operation event in a read-write operation log of a database in real time, determining characteristic information of the read-write operation event, and converting data of the read-write operation event into a serialization array comprises:

judging whether the service identifier corresponding to the service identifier is in the first position in the service identifier sequence or not;

if the data is in the first position, judging the data as a main copy, determining characteristic information of a read-write operation event for the read-write operation event in a real-time read-write operation log of a database, and converting the data of the read-write operation event into a serialized array.

On the other hand, for the read-write operation event in the real-time read-write operation log of the database, determining the characteristic information of the read-write operation event, and converting the data of the read-write operation event into a serialized array comprises:

Requesting a real-time read-write operation log in a database according to the identity of a slave node of the database;

determining the event type of the read-write operation event through the data of the read-write operation event in the read-write operation log;

Determining the table name of a database table to which the read-write operation event belongs through a table mapping event positioned before the read-write operation event in the read-write operation log;

converting the data of the read-write operation event into a serialization array;

The event types of the read-write operation event comprise a deletion event, an insertion event and an update event.

On the other hand, based on the key value pair set of the read-write operation event, executing the refresh operation corresponding to the event type of the read-write operation event on the cache data corresponding to the read-write operation event includes:

if the event type of the read-write operation event is a deletion event, deleting the cache data corresponding to the read-write operation event;

If the event type of the read-write operation event is an insertion event, writing the key value pair set of the read-write operation event and the primary key information into a cache as cache data;

If the event type of the read-write operation event is an update event, overlaying the key value pair set of the read-write operation event and the primary key information to cache data corresponding to the read-write operation event in a cache;

When the write cache operation corresponding to the write event fails, adding the primary key information of the write event with the write cache failure into a first message queue so that an asynchronous thread deletes cache data corresponding to the primary key information in the first message queue, wherein the write event comprises an insertion event and an update event.

On the other hand, if the database table to which the read-write operation event belongs to a pre-designated target database table, generating the key value pair set corresponding to the read-write operation event according to the serialized array of the read-write operation event and the table structure information of the target database table includes:

If the database table to which the read-write operation event belongs to a pre-designated target database table, determining a field name sequence from table structure information of the target database table to which the read-write operation event belongs, wherein the field name sequence comprises a plurality of field names which are orderly arranged, the field names are field names in the target database table to which the read-write operation event belongs, and the sequence of the field names in the field name sequence is equal to the sequence of the field names in each field name of the target database table to which the field names belong;

if the read-write operation event is a deletion event, generating a first key value pair set corresponding to the deletion event according to the serialized data corresponding to the deletion event and the field name sequence, wherein the serialized data corresponding to the deletion event is one row of data in a database table deleted by the deletion event, the first key value pair set comprises a plurality of key value pairs, and each key value pair comprises a single field value and a field name corresponding to the single field value deleted by the deletion event;

If the read-write operation event is an insertion event, generating a second key value pair set corresponding to the insertion event according to the serialized data corresponding to the insertion event and the field name sequence, wherein the serialized data corresponding to the insertion event is one line of data inserted by the insertion event into a database table, the second key value pair set comprises a plurality of key value pairs, and each key value pair comprises a single field value and a field name corresponding to the single field value inserted by the insertion event;

If the read-write operation event is an update event, generating a third key value pair set and a fourth key value pair set corresponding to the update event according to the serialized data and the field name sequence corresponding to the update event, wherein the serialized data corresponding to the update event is target row data updated in a database table by the update event, the third key value pair set and the fourth key value pair set both comprise a plurality of key value pairs, and any key value pair in the third key value pair set comprises: a single field value and a corresponding field name of the target line data before being updated, wherein any key value pair in the fourth key value pair set comprises: and the single field value and the corresponding field name of the updated target line data.

On the other hand, if the database table to which the read-write operation event belongs to a pre-designated target database table, after generating the key value pair set corresponding to the read-write operation event according to the serialized array of the read-write operation event and the table structure information of the target database table, the cache refreshing method further comprises:

constructing a storage object with a preset structure based on the characteristic information of the read-write operation event and a key value pair set corresponding to the read-write operation event;

The storage object comprises a first storage position and a second storage position, wherein the first storage position is used for storing the characteristic information, the second storage position comprises a front data storage position and a rear data storage position, the front data storage position is used for storing a key value pair set corresponding to data before a read-write operation event occurs, and the rear data storage position is used for storing a key value pair set corresponding to data after the read-write operation event occurs;

according to the primary key information of the read-write operation event in the affiliated database table, determining cache data corresponding to the read-write operation event in a cache comprises the following steps:

and determining cache data corresponding to the read-write operation event in the cache according to the primary key information in the database table corresponding to the table name in the first storage position in the storage object.

In order to solve the technical problem, the present invention further provides a cache refreshing device, including:

The first determining module is used for determining characteristic information of the read-write operation event for the read-write operation event in the real-time read-write operation log of the database and converting the data of the read-write operation event into a serialization array, wherein the characteristic information comprises a table name and an event type of a database table to which the characteristic information belongs;

The first generation module is configured to generate a set of key value pairs corresponding to the read-write operation event according to the serialized array of the read-write operation event and table structure information of the target database table if the database table to which the read-write operation event belongs to a pre-specified target database table, where the set of key value pairs includes a plurality of key value pairs, and any one of the key value pairs includes: a single field value and a field name corresponding to the single field value in the read-write operation event;

The second determining module is used for determining cache data corresponding to the read-write operation event in the cache according to the primary key information of the read-write operation event in the affiliated database table;

And the first refreshing module is used for executing refreshing operation corresponding to the event type of the read-write operation event on the cache data corresponding to the read-write operation event based on the key value pair set of the read-write operation event.

In order to solve the above technical problem, the present invention further provides a cache refreshing device, including:

a memory for storing a computer program;

and the processor is used for realizing the steps of the cache refreshing method when executing the computer program.

To solve the above technical problem, the present invention also provides a computer program product, which includes a computer program/instruction, and the computer program/instruction implements the steps of the cache refreshing method as described above when executed by a processor.

To solve the above technical problem, the present invention further provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the cache refreshing method as described above.

The beneficial effects are that: the invention provides a cache refreshing method, which considers that the latest state of database data is reflected in a real-time read-write operation log of a database, and access to the read-write operation log avoids invading user business, so that the characteristic information of read-write operation events can be determined firstly based on the real-time read-write operation log of the database, the data of the read-write operation events are converted into a serialization array, then a set of key value pairs corresponding to the read-write operation events is generated according to the table structure information of a target database table to which the read-write operation events belong and the serialization array of the read-write operation events, and then refreshing operation corresponding to the event types of the read-write operation events can be executed on the cache data corresponding to the read-write operation events based on the set of key value pairs of the read-write operation events.

The invention also provides a cache refreshing device, equipment, a computer program product and a computer readable storage medium, which have the same beneficial effects as the cache refreshing method.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following description will briefly explain the related art and the drawings required to be used in the embodiments, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person of ordinary skill in the art.

FIG. 1 is a schematic flow chart of a cache refreshing method provided by the invention;

FIG. 2 is a schematic diagram of a data query method according to the present invention;

Fig. 3 is a schematic diagram of a buffer error correction method according to the present invention;

FIG. 4 is a schematic diagram of a primary copy determination method according to the present invention;

FIG. 5 is a flowchart illustrating another cache refreshing method according to the present invention;

FIG. 6 is a flowchart illustrating a buffer refreshing method according to another embodiment of the present invention;

FIG. 7 is a schematic diagram of a cache refreshing device according to the present invention;

FIG. 8 is a schematic diagram of a cache refreshing device according to the present invention;

fig. 9 is a schematic structural diagram of a computer readable storage medium according to the present invention.

Detailed Description

The core of the invention is to provide a cache refreshing method, a device, equipment, a computer program product and a storage medium, wherein cache data can be refreshed through a database real-time read-write operation log, so that cache refreshing efficiency is improved, and user business invasion is avoided.

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 1, fig. 1 is a flow chart of a cache refreshing method provided by the present invention, where the cache refreshing method is applied to a computer system, and includes:

s101: determining characteristic information of a read-write operation event for the read-write operation event in a real-time read-write operation log of a database, and converting data of the read-write operation event into a serialization array, wherein the characteristic information comprises a table name and an event type of a database table to which the characteristic information belongs;

specifically, in consideration of the technical problems in the background art, in combination with consideration of (1) the latest state of database data is reflected in the real-time read-write operation log of the database, (2) the access to the read-write operation log avoids invading user service, so that in the embodiment of the invention, the real-time data in the database is to be updated based on the real-time read-write operation log of the database and synchronized into the cache, so that the consistency of the database and the cache is realized efficiently, and the access to the read-write operation log of the database avoids invading user service, thereby improving user experience.

The read-write operation log refers to a log generated by the database based on read-write operation events, the read-write operation events refer to events for performing read-write operation on data in the database, and event types of the read-write operation events may include various types, such as a deletion event, an insertion event, an update event, and the like, which are not limited herein.

Specifically, in order to smoothly refresh cache data in a cache based on a read-write operation event, a database table to which the read-write operation event belongs and an event type need to be known, and data of the read-write operation event need to be acquired, where the data may refer to data of the database before the occurrence of the read-write operation event and data after the occurrence of the read-write operation event, where it is considered that each field value of a row of data to be operated in the database table may be included in the read-write operation event, each field name corresponding to each field value is not included, and each field value in the data of the read-write operation event has a sequence, which may play an important role in determining the field name corresponding to the field value, so in the embodiment of the present invention, the data of the read-write operation event may be converted into a serialized array, and both serialized data and feature information may serve as a data base of a subsequent step.

For a better explanation of embodiments of the present invention, please refer to the following table 1:

Table 1: database table

Identification mark	A	B	C	D
					1	2	3	4	5

Specifically, in table 1, each item in the first row in table 1 is a field name, each item in the second row is a field value corresponding to the field name, for example, the field value corresponding to the identifier is 1, where the identifier may also be referred to as an ID, which is generally regarded as a primary key of a database table, in the database table, a row including a plurality of rows of field values may be included except for the row where the field name is located, and an object generally processed by a single read/write operation event is a single row of field values in the database table.

S102: if the database table to which the read-write operation event belongs to a pre-designated target database table, generating a key value pair set corresponding to the read-write operation event according to the serialized array of the read-write operation event and the table structure information of the target database table, wherein the key value pair set comprises a plurality of key value pairs, and any key value pair comprises: a single field value in a read-write operation event and a field name corresponding to the single field value;

specifically, considering that the read-write operation event in the read-write operation log may relate to any database table, but not all database tables need to be updated into the cache, the target database table may be preset in the embodiment of the present invention, so in the embodiment of the present invention, the subsequent operation may be performed when the database table to which the read-write operation event belongs to the pre-specified target database table.

Specifically, as stated above, since the serialized array obtained by converting the data in the read-write operation event includes each field value but does not include each field name, which hinders refreshing of the cached data, in the embodiment of the present invention, a set of key value pairs corresponding to the read-write operation event may be generated according to the serialized array of the read-write operation event and the table structure information of the target database table, that is, a corresponding relationship is established between the field values in the serialized array and the field names to which the field values belong, so as to obtain a set of key value pairs composed of each key value pair, and use the key value pair set as a data base of a subsequent step.

The data is organized in the form of key value pairs, so that the data processing efficiency can be improved.

In addition, it should be noted that, in the embodiment of the present invention, the table structure information of each target database table may be stored in a specific column of a specific database table in the database in advance, and then after the computer system is operated, the table structure information of each target database table may be obtained from the specific column of the specific database table in the database through a read operation, so as to be used for subsequent standby, where the core content of the table structure information of the target database table is: the table name, the field name of each column of the target database table, and the position of the column in the table, of course, may also include other information, for example, the mode in which the table is located, the data type of each column, and the like, which are not limited herein.

S103: determining cache data corresponding to the read-write operation event in the cache according to the primary key information of the read-write operation event in the affiliated database table;

Specifically, in order to determine which row in which database table the database data related to the read-write operation event belongs to, and these information may be reflected in the primary key information of the database table to which the read-write operation event belongs, so the embodiment of the present invention may determine, according to the primary key information of the read-write operation event in the database table to which the read-write operation event belongs, cache data corresponding to the read-write operation event in a cache, so as to perform a cache refresh operation in a subsequent step.

S104: and executing refreshing operation corresponding to the event type of the read-write operation event on the cache data corresponding to the read-write operation event based on the key value pair set of the read-write operation event.

Specifically, since the key value pair set corresponding to the read-write operation event and the cache data corresponding to the read-write operation event have been determined in the foregoing steps, and the fact that the specific cache refreshing operation is related to the event type of the read-write operation event is considered, in the embodiment of the present invention, the refreshing operation corresponding to the event type of the read-write operation event can be performed on the cache data corresponding to the read-write operation event based on the key value pair set of the read-write operation event.

The invention provides a cache refreshing method, which considers that the latest state of database data is reflected in a real-time read-write operation log of a database, and access to the read-write operation log avoids invading user business, so that the characteristic information of read-write operation events can be determined firstly based on the real-time read-write operation log of the database, the data of the read-write operation events are converted into a serialization array, then a set of key value pairs corresponding to the read-write operation events is generated according to the table structure information of a target database table to which the read-write operation events belong and the serialization array of the read-write operation events, and then refreshing operation corresponding to the event types of the read-write operation events can be executed on the cache data corresponding to the read-write operation events based on the set of key value pairs of the read-write operation events.

Based on the above embodiments:

as an optional embodiment, the cache refreshing method further includes:

Unblocking a refresh operation for the first thread;

the first thread is a thread executing a cache refreshing operation, wherein the cache refreshing operation refers to: and executing refreshing operation corresponding to the event type of the read-write operation event on the cache data corresponding to the read-write operation event based on the key value pair set of the read-write operation event.

Specifically, considering the problem that data in a database in a computer system may be inconsistent with cache for a long time due to a cache refresh mechanism failure or the like, or before the cache refresh method in the previous embodiment is started, each target database table in the database has history data, which cannot be written into the cache based on the cache refresh method in the previous embodiment, so in order to further improve the data consistency between the database and the cache, in the embodiment of the invention, the refresh operation of the first thread is blocked in response to the cache reset instruction, and the blocked action can avoid updating cache time sequence conflict with the action of the first thread, so that the database is inconsistent with the cache data; while in blocking, the cache may be emptied and the data of each target database table in the database written into the cache, which may write all required field values of the target database table at that time into the cache, including the aforementioned history data.

It should be noted that, in the embodiment of the present invention, for each target database table, a field name required in the target database table may be determined, so as to avoid writing the field name and the field value thereof that are not required in the target database table into the cache.

Specifically, after the cache is emptied, after the data of each target database table in the database is written into the cache, the blocking of the refreshing operation of the first thread can be released, and after the blocking is released, the read-write operation log in the database can be accessed back in the period that the first thread is blocked, so that the first thread can refresh the cache by the read-write operation log in the database in the period that the first thread is blocked, and the data consistency of the database and the cache is further improved.

As an alternative embodiment, in response to the cache reset instruction, blocking the refresh operation of the first thread includes:

unblocking the refresh operation for the first thread includes:

And sending the second numerical semaphore to the cache so that the cache unblocks the refresh operation for the first thread after identifying the second numerical semaphore.

In particular, considering the semaphore represented by the value, compared with a distributed lock or other forms of thread blocking modes, the thread perception can be reduced, so that the efficiency of the cache refreshing work is further improved.

The first value and the second value may be set autonomously, for example, the first value may be one of 1 and 0, the second value may be the other of 1 and 0, etc., which is not limited herein.

Of course, the first thread may be blocked and contact blocked in other ways besides the blocking manner, and embodiments of the present invention are not limited herein.

As an alternative embodiment, the cache reset instruction comprises a first reset instruction and/or a second reset instruction;

Specifically, the reset instruction automatically generated at the preset time in one day triggers the reset, so that forced consistency operation of the database and the cache can be performed every day, the problem of inconsistency between the database and the cache can not last for more than one day, and the reset instruction received through the human-computer interface can enable the cache reset operation to have autonomy and flexibility.

Of course, the generation mode of the cache reset instruction may be any other generation mode besides the two generation modes, and the embodiment of the present invention is not limited herein.

As an alternative embodiment, the cache and the database are both connected to the preset interface software, so that each application service can call data from the cache or the database through the preset interface software;

the preset interface software comprises a software development kit.

Specifically, by providing unified preset interface software for each application service, and the preset interface software can be connected to the database and the cache, in this case, any application service can search target data from the database and the cache through the preset interface software, if the result of the query is not found, it represents that the target data does not exist in the computer system, and the development burden of each application service can be reduced.

The software development kit tool (SDK, software Development Kit) has the advantage of simplicity and easiness, and of course, the preset interface software can be of various types besides the SDK, and the embodiment of the invention is not limited herein.

Specifically, the setting of the query logic can be performed based on the preset interface software, for example, the query logic can be set to query the target data from the cache first, and if the query is not completed, the cache can be penetrated to query the target data from the database.

For better explaining the embodiments of the present invention, please refer to fig. 2, fig. 2 is a schematic diagram of a data query method provided by the present invention, where each application service may access a cache and a database through preset interface software, may read the cache, or may penetrate the cache to read the database.

As an optional embodiment, the cache refreshing method further includes:

if not, deleting the target cache data from the cache;

And if the data are inconsistent, updating the target cache data in the cache by taking the database data as a reference.

Specifically, in order to further ensure the consistency of the database and the cache data, considering that when the application service hits the cache through the preset interface software, consistency check can be performed on the hit cache data, so that the cache is refreshed in time, so in the embodiment of the invention, after the application service hits the cache through the read request sent by the preset interface software, the hit target cache data is written into the second message queue, then whether the database data corresponding to the target cache data exists in the database is judged, if the database data does not exist, the target cache data can be deleted from the cache, and if the database data does not exist, whether the target cache data is consistent with the database data corresponding to the target cache data in the database can be judged, and if the database data does not exist, the target cache data in the cache can be updated by taking the database data as a reference.

For better explaining the embodiments of the present invention, please refer to fig. 3, fig. 3 is a schematic diagram of a cache error correction method provided by the present invention, after a hit cache of an application service is applied, the hit target cache data may be written into a second message queue, a cache error corrector (virtual module) may monitor a message in the second message queue, read database data corresponding to the target cache data in the database by reading the database, and then compare the cache data with data in the database, and if the cache data is inconsistent, update the target cache data in the cache with the database data as a reference.

As an alternative embodiment, applied to any node in a computer system;

The cache refreshing method further comprises the following steps:

Responding to a heartbeat signal generated every other preset period, and increasing preset duration for expiration time of service identifiers corresponding to the service identifiers in the service identifier sequence, wherein the service identifier sequence comprises service identifiers of a plurality of nodes which are orderly arranged, and the service identifiers with expiration time not more than zero are in an expiration state;

For the read-write operation event in the real-time read-write operation log of the database, determining the characteristic information of the read-write operation event and converting the data of the read-write operation event into a serialized array comprises:

if the data is in the first position, the data is judged to be the main copy, and for the read-write operation event in the read-write operation log of the database in real time, the characteristic information of the read-write operation event is determined, and the data of the read-write operation event is converted into a serialization array.

Specifically, in order to promote high availability of the cache refreshing mechanism, the cache refreshing method in the embodiment of the invention can be applied to any node in a computer system, each node is provided with a program (i.e. a copy) of the cache refreshing method, and then each step of the cache refreshing method is executed by a master copy in each copy.

Specifically, each node can determine whether the node belongs to the primary copy through the service identification sequence, the service identification sequence comprises service identifications of a plurality of nodes which are orderly arranged, the service identifications with expiration time not longer than zero are in an invalid state, each node can respond to heartbeat signals generated every preset period to increase the expiration time of the service identifications corresponding to the node in the service identification sequence by preset time length, in this case, when a certain node cannot generate a heartbeat signal or cannot respond to the heartbeat signal due to failure, the service identifications belonging to the failure node in the service identification sequence can expire, the service identification of the last one position of the service identifications of the failure node advances one position along with the failure of the service identifications of the failure node, and the copy corresponding to the first service identification in the service identification sequence is identified as the primary copy.

Of course, there may be a plurality of ways to determine the primary copy in a healthy state other than the above ways, and embodiments of the present invention are not limited herein.

For better explaining the embodiments of the present invention, please refer to fig. 4, fig. 4 is a schematic diagram of a primary copy determining method provided by the present invention, where the first to third auxiliary books are connected with a service identifier sequence, and may delay the service identifier thereof according to a heartbeat signal.

As an optional embodiment, for a read-write operation event in a real-time read-write operation log of a database, determining feature information of the read-write operation event, and converting data of the read-write operation event into a serialized array includes:

Requesting real-time read-write operation logs in the database according to the identities of slave nodes of the database;

Specifically, considering that the slave node serving as the database can smoothly inquire the read-write operation log from the database, the computer system in the embodiment of the invention can request the real-time read-write operation log in the database according to the identity of the slave node of the database, then determine the event type of the read-write operation event according to the data of the read-write operation event in the read-write operation log, and determine the table name of the database table to which the read-write operation event belongs according to the table mapping event before the read-write operation event in the read-write operation log, and finally convert the data of the read-write operation event into a serialization array.

Specifically, taking a computer system as a cloud computing platform, taking a MySQL (relational database management system) database as an example, caching as a Redis (Remote Dictionary Server, remote dictionary service) cache, a thread in the computer system can disguise itself as a slave node of the MySQL database, and an I/O (Input/Output) thread is started to initiate a dump request to a MySQL master node. The master node starts a Binlog dump thread and pushes a Binlog (logic log) log (including a read-write operation log) to the I/O thread. Therefore, the Binlog log change of the master node can be monitored in real time based on the master-slave protocol of MySQL.

Of course, the computer system, database, and cache may be of various types other than those specifically described above, and embodiments of the present invention are not limited in this regard.

As an optional embodiment, based on the set of key value pairs of the read-write operation event, performing a refresh operation corresponding to an event type of the read-write operation event on the cache data corresponding to the read-write operation event includes:

If the event type of the read-write operation event is an insertion event, writing a key value pair set and main key information of the read-write operation event into a cache as cache data;

if the event type of the read-write operation event is an update event, covering the key value pair set and the primary key information of the read-write operation event to cache data corresponding to the read-write operation event in a cache;

When the write cache operation corresponding to the write event fails, adding the primary key information of the write event with the write cache failure into the first message queue so that the asynchronous thread deletes the cache data corresponding to the primary key information in the first message queue in the cache, wherein the write event comprises an insertion event and an update event.

Specifically, the first half of the embodiment of the present invention introduces a specific process of "performing a refresh operation corresponding to an event type of a read-write operation event on cached data corresponding to the read-write operation event" based on a set of key value pairs of the read-write operation event, and the last section introduces a processing manner for "write cache operation failure": in order to avoid that the cache data associated with the write-in event in the first message queue is covered during the retry period, so that the cache data is in error, in the embodiment of the invention, the primary key information of the write-in event with the cache failure can be added into the first message queue, and the corresponding cache data of the primary key information in the first message queue in the cache is deleted through an asynchronous thread, so that the data consistency of the database and the cache is further improved.

For better explanation of the embodiments of the present invention, please refer to fig. 5, fig. 5 is a schematic flow chart of another cache refreshing method provided by the present invention, a listening resolver may perform listening and resolving (i.e. a process of obtaining feature information and serialization array) on a read-write operation log in a database, a master copy may refresh a cache through a single thread, write a write cache failure event into a second message queue, and delete cache data corresponding to the write cache failure event for the write cache failure event in the second message queue.

As an optional embodiment, if the database table to which the read-write operation event belongs to a pre-designated target database table, generating the key value pair set corresponding to the read-write operation event according to the serialized array of the read-write operation event and the table structure information of the target database table includes:

If the database table to which the read-write operation event belongs to a pre-designated target database table, determining a field name sequence from table structure information of the target database table to which the read-write operation event belongs, wherein the field name sequence comprises a plurality of orderly arranged field names, the field names are field names in the target database table to which the read-write operation event belongs, and the order of the field names in the field name sequence is equal to the order of the field names in each field name of the target database table to which the field names belong;

If the read-write operation event is a deletion event, generating a first key value pair set corresponding to the deletion event according to the serialized data and the field name sequence corresponding to the deletion event, wherein the serialized data corresponding to the deletion event is one row of data in a database table deleted by the deletion event, the first key value pair set comprises a plurality of key value pairs, and each key value pair comprises a single field value deleted by the deletion event and a field name corresponding to the single field value;

if the read-write operation event is an insertion event, generating a second key value pair set corresponding to the insertion event according to the serialized data and the field name sequence corresponding to the insertion event, wherein the serialized data corresponding to the insertion event is one row of data inserted into the database table by the insertion event, the second key value pair set comprises a plurality of key value pairs, and each key value pair comprises a single field value and a field name corresponding to the single field value inserted by the insertion event;

If the read-write operation event is an update event, generating a third key value pair set and a fourth key value pair set corresponding to the update event according to the serialized data and the field name sequence corresponding to the update event, wherein the serialized data corresponding to the update event is the target row data updated in the database table by the update event, the third key value pair set and the fourth key value pair set both comprise a plurality of key value pairs, and any key value pair in the third key value pair set comprises: the single field value and its corresponding field name before the target line data is updated, and any key value pair in the fourth key value pair set includes: the single field value after the target line data is updated and its corresponding field name.

Specifically, considering that the data of different types of read-write operation events have differences, the embodiment of the invention introduces the generating process of key value pair sets of different types of read-write operation events, and the main difference is that the data of the delete event comprises a field value of row data before the delete event, the data of the insert event comprises a field value of row data after the insert event, and the data of the update event comprises two sets of field values of row data before and after the update event, so that after determining a field name sequence of a database table to which the event belongs, the embodiment of the invention can construct different key value pair sets for the event data according to the difference of the event types.

As an optional embodiment, if the database table to which the read-write operation event belongs to a pre-designated target database table, after generating the key value pair set corresponding to the read-write operation event according to the serialized array of the read-write operation event and the table structure information of the target database table, the cache refreshing method further includes:

Constructing a storage object with a preset structure based on the characteristic information of the read-write operation event and the key value pair set corresponding to the read-write operation event;

The storage object comprises a first storage position and a second storage position, the first storage position is used for storing characteristic information, the second storage position comprises a front data storage position and a rear data storage position, the front data storage position is used for storing a key value pair set corresponding to data before a read-write operation event occurs, and the rear data storage position is used for storing a key value pair set corresponding to data after the read-write operation event occurs;

according to the primary key information of the read-write operation event in the affiliated database table, determining cache data corresponding to the read-write operation event in the cache comprises the following steps:

Specifically, in order to enable a key value pair sequence of a read-write operation event of each event type to have a unified storage architecture, in the embodiment of the present invention, a storage object with a preset structure may be constructed based on feature information of the read-write operation event and a key value pair set corresponding to the read-write operation event, where the storage object includes a first storage location and a second storage location, the first storage location is used for storing the feature information, the second storage location includes a front data storage location and a rear data storage location, the front data storage location is used for storing a key value pair set corresponding to data before the read-write operation event occurs, and the rear data storage location is used for storing a key value pair set corresponding to the data after the read-write operation event occurs, that is, for a deletion event, the key value pair sequence is stored in the front data storage location, for an insertion event, and for an update event, the front data storage location is used for storing a key value pair set corresponding to the data before the read operation event occurs, and the rear data storage location is used for storing a key value pair set corresponding to the read-write operation event.

In addition, according to the primary key information of the read-write operation event in the affiliated database table, determining the cache data corresponding to the read-write operation event in the cache may include:

And acquiring characteristic information and serialized data in the storage object, determining a database table corresponding to the table name, further determining the field name of a main key in the database table corresponding to the table name, taking the first field value in the serialized data of the storage object as the field value of the main key, thereby obtaining main key information, and further determining which row of the database table the read-write operation event belongs to, and further determining the cache data in the cache through the main key information.

For better explanation of the embodiments of the present invention, please refer to fig. 6, fig. 6 is a flow chart of another cache refreshing method provided by the present invention, and fig. 6 is a global diagram of the cache refreshing method combined from fig. 2 to fig. 5, so that the description of fig. 2 to fig. 5 can be referred to, and the embodiments of the present invention will not be repeated here.

Referring to fig. 7, fig. 7 is a schematic structural diagram of a cache refreshing device according to the present invention, where the cache refreshing device includes:

The first determining module 71 is configured to determine, for a read-write operation event in a read-write operation log of the database in real time, feature information of the read-write operation event, and convert data of the read-write operation event into a serialized array, where the feature information includes a table name and an event type of a table of the database to which the feature information belongs;

The first generating module 72 is configured to generate, if the database table to which the read-write operation event belongs to a pre-specified target database table, a set of key-value pairs corresponding to the read-write operation event according to the serialized array of the read-write operation event and table structure information of the target database table, where the set of key-value pairs includes a plurality of key-value pairs, and any one key-value pair includes: a single field value in a read-write operation event and a field name corresponding to the single field value;

A second determining module 73, configured to determine cache data corresponding to the read-write operation event in the cache according to the primary key information of the read-write operation event in the affiliated database table;

the first refresh module 74 is configured to perform a refresh operation corresponding to an event type of the read/write operation event on the cached data corresponding to the read/write operation event based on the set of key value pairs of the read/write operation event.

As an alternative embodiment, the cache refreshing device further includes:

The first blocking module is used for responding to the cache reset instruction and blocking the refreshing operation of the first thread;

The first action module is used for writing the data of each target database table in the database into the cache after the cache is emptied;

A second blocking module for unblocking a refresh operation for the first thread;

As an alternative embodiment, the first blocking module is specifically configured to:

The second blocking module is specifically configured to:

the preset interface software comprises a software development kit.

As an alternative embodiment, the cache refreshing device further includes:

the second action module is used for writing the hit target cache data into a second message queue after the read request sent by the application service through the preset interface software hits the cache;

The first judging module is used for judging whether the database data corresponding to the target cache data exists in the database, if not, triggering the first deleting module, and if so, triggering the second judging module;

The first deleting module is used for deleting the target cache data from the cache;

The second judging module is used for judging whether the target cache data is consistent with the database data corresponding to the target cache data in the database, and triggering the third action module if the target cache data is inconsistent with the database data corresponding to the target cache data in the database;

And the third action module is used for updating the target cache data in the cache by taking the database data as a reference.

As an optional embodiment, the cache refreshing device is applied to any node in the computer system;

the cache refreshing device further comprises:

a fourth action module, configured to increase a preset duration for an expiration time of a service identifier corresponding to the fourth action module in the service identifier sequence in response to a heartbeat signal generated every a preset period, where the service identifier sequence includes service identifiers of a plurality of nodes in an orderly arrangement, and the service identifier whose expiration time is not greater than zero is in an expiration state;

The first determination module 71 includes:

the third judging module is used for judging whether the service identifier corresponding to the third judging module is at the first position in the service identifier sequence, and triggering the fifth action module if the third judging module is at the first position;

and the fifth action module is used for judging the self as a main copy, determining characteristic information of the read-write operation event for the read-write operation event in the real-time read-write operation log of the database, and converting data of the read-write operation event into a serialized array.

As an alternative embodiment, the first determining module 71 includes:

The sixth action module is used for requesting real-time read-write operation logs in the database according to the identities of the slave nodes of the database;

The third determining module is used for determining the event type of the read-write operation event through the data of the read-write operation event in the read-write operation log;

The fourth determining module is used for determining the table name of the database table to which the read-write operation event belongs through the table mapping event positioned before the read-write operation event in the read-write operation log;

The conversion module is used for converting the data of the read-write operation event into a serialization array;

As an alternative embodiment, the first refresh module 74 includes:

The first refreshing sub-module is used for deleting the cache data corresponding to the read-write operation event if the event type of the read-write operation event is a deletion event;

The second refreshing sub-module is used for writing the key value pair set of the read-write operation event and the primary key information into the cache as cache data if the event type of the read-write operation event is an insertion event;

the third refreshing sub-module is used for covering the key value pair set of the read-write operation event and the main key information to the cache data corresponding to the read-write operation event in the cache if the event type of the read-write operation event is an update event;

And the seventh action module is used for adding the primary key information of the write-in event with the write-in event failing to the first message queue when the write-in operation corresponding to the write-in event fails, so that the asynchronous thread deletes the cache data corresponding to the primary key information in the first message queue in the cache, wherein the write-in event comprises an insertion event and an update event.

As an alternative embodiment, the first generation module 72 includes:

A first generation sub-module, configured to determine a field name sequence from table structure information of a target database table to which the read-write operation event belongs, if the database table to which the read-write operation event belongs to a pre-specified target database table, the field name sequence comprises a plurality of field names which are orderly arranged, wherein the field names are field names in a target database table to which a read-write operation event belongs, and the order of the field names in the field name sequence is equal to the order of the field names in each field name of the target database table to which the field names belong;

The second generation sub-module is used for generating a first key value pair set corresponding to the deletion event according to the serialized data and the field name sequence corresponding to the deletion event if the read-write operation event is the deletion event, wherein the serialized data corresponding to the deletion event is one row of data in the database table deleted by the deletion event, the first key value pair set comprises a plurality of key value pairs, and each key value pair comprises a single field value deleted by the deletion event and a field name corresponding to the single field value;

a third generating sub-module, configured to generate, if the read-write operation event is an insertion event, a second key value pair set corresponding to the insertion event according to the serialized data and the field name sequence corresponding to the insertion event, where the serialized data corresponding to the insertion event is a row of data inserted by the insertion event into the database table, the second key value pair set includes a plurality of key value pairs, and each key value pair includes a single field value and a field name corresponding to the single field value inserted by the insertion event;

A fourth generation sub-module, configured to generate, if the read-write operation event is an update event, a third key value pair set and a fourth key value pair set corresponding to the update event according to the serialized data and the field name sequence corresponding to the update event, where the serialized data corresponding to the update event is target row data updated in the database table by the update event, the third key value pair set and the fourth key value pair set each include a plurality of key value pairs, and any one key value pair in the third key value pair set includes: the single field value and its corresponding field name before the target line data is updated, and any key value pair in the fourth key value pair set includes: the single field value after the target line data is updated and its corresponding field name.

As an alternative embodiment, the cache refreshing device further includes:

The eighth action module is used for constructing a storage object with a preset structure based on the characteristic information of the read-write operation event and the key value pair set corresponding to the read-write operation event;

The second determining module 73 is specifically configured to:

For the description of the cache refreshing device provided in the embodiment of the present invention, reference is made to the foregoing embodiments of the cache refreshing method, and the embodiments of the present invention are not repeated herein.

Referring to fig. 8, fig. 8 is a schematic structural diagram of a cache refreshing device according to the present invention, where the cache refreshing device includes:

A memory 81 for storing a computer program;

A processor 82 for implementing the steps of the cache refresh method in the previous embodiment when executing a computer program.

The present invention also provides a computer program product comprising computer programs/instructions which when executed by a processor implement the steps of the cache flushing method as in the previous embodiments.

For the description of the computer program product provided by the embodiment of the present invention, reference is made to the foregoing embodiments of the cache refreshing method, and the embodiments of the present invention are not repeated herein.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a computer readable storage medium provided in the present invention, and a computer program 92 is stored on the computer readable storage medium 91, and the steps of the cache refreshing method in the foregoing embodiments are implemented when the computer program 92 is executed by a processor.

For the description of the computer readable storage medium provided in the embodiment of the present invention, reference is made to the foregoing embodiments of the cache refreshing method, and the embodiments of the present invention are not repeated here.

In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section. It should also be noted that in this specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cache refreshing method, applied to a computer system, comprising:

2. The cache refreshing method according to claim 1, characterized in that the cache refreshing method further comprises:

Unblocking a refresh operation for the first thread;

3. The cache flushing method of claim 2, wherein blocking the flushing operation of the first thread in response to the cache reset instruction comprises:

unblocking a refresh operation for the first thread includes:

4. The cache refreshing method according to claim 2, wherein the cache reset instruction comprises a first reset instruction and/or a second reset instruction;

5. The cache refreshing method according to claim 1, wherein the cache and the database are both connected to a preset interface software, so that each application service can call data from the cache or the database through the preset interface software;

the preset interface software comprises a software development kit.

6. The cache refreshing method according to claim 5, further comprising:

If not, deleting the target cache data from the cache;

7. The cache refreshing method according to claim 1, applied to any node in a computer system;

The cache refreshing method further comprises the following steps:

8. The cache refreshing method according to claim 1, wherein for a read-write operation event in a read-write operation log of a database in real time, determining characteristic information of the read-write operation event, and converting data of the read-write operation event into a serialized array comprises:

9. The cache refreshing method according to claim 1, wherein performing a refresh operation corresponding to an event type of the read-write operation event on cache data corresponding to the read-write operation event based on a set of key value pairs of the read-write operation event comprises:

10. The cache refreshing method according to any one of claims 1 to 9, wherein if the database table to which the read-write operation event belongs to a pre-specified target database table, generating a set of key value pairs corresponding to the read-write operation event according to the serialized array of the read-write operation event and table structure information of the target database table includes:

11. The method of claim 10, wherein if the database table to which the read-write operation event belongs to a pre-designated target database table, after generating the set of key value pairs corresponding to the read-write operation event according to the serialized array of the read-write operation event and table structure information of the target database table, the method further comprises:

12. A cache refreshing apparatus, comprising:

13. A cache refreshing apparatus, comprising:

a memory for storing a computer program;

processor for implementing the steps of the cache refreshing method according to any of claims 1 to 11 when executing said computer program.

14. A computer program product comprising computer programs/instructions which when executed by a processor implement the steps of the cache flushing method of any of claims 1 to 11.

15. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the cache refresh method according to any of claims 1 to 11.