CN113127443B - Method and device for updating cache data - Google Patents
Method and device for updating cache data Download PDFInfo
- Publication number
- CN113127443B CN113127443B CN202010037735.7A CN202010037735A CN113127443B CN 113127443 B CN113127443 B CN 113127443B CN 202010037735 A CN202010037735 A CN 202010037735A CN 113127443 B CN113127443 B CN 113127443B
- Authority
- CN
- China
- Prior art keywords
- task
- cache data
- updating
- abnormal
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 230000002159 abnormal effect Effects 0.000 claims description 112
- 238000011084 recovery Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 17
- 230000006870 function Effects 0.000 description 8
- 230000004048 modification Effects 0.000 description 8
- 238000012986 modification Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000013502 data validation Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 239000003607 modifier Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 241000533950 Leucojum Species 0.000 description 1
- 235000010627 Phaseolus vulgaris Nutrition 0.000 description 1
- 244000046052 Phaseolus vulgaris Species 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for updating cache data, and relates to the technical field of computers. One embodiment of the method comprises the following steps: receiving an update request, and generating a parent task for updating the cache data to each cluster according to the update request; the update request carries information of each cluster and service identification of the cache data, and the parent task comprises a plurality of subtasks; the cache data are acquired and respectively updated to the clusters corresponding to the subtasks; and respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data. The embodiment can solve the technical problem of data consistency among cache clusters.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and apparatus for updating cache data.
Background
In order to carry massive order processing requests, caching techniques (such as Redis) have become a scaled technical solution. However, as the service scale grows, there is often a situation that a single cache cluster cannot bear more pressure. For example, client applications expand machines without limit, resulting in a corresponding increase in the number of cache client connections; the slave examples of the cache clusters are excessively mounted, the data replication quantity between the master and the slave is large, and the cluster pressure is increased due to factors such as overlong replication links. To solve this problem, the possible solutions are: the whole cache clusters are copied into a plurality of cache clusters, client applications are distributed in a vertical grouping mode, and different client applications are connected with different cache clusters, so that pressure sharing is completed. But this solution presents a problem: if these client applications require strong data consistency among multiple cache clusters, and require strict data validation time consistency. Otherwise, different client applications may cause inconsistent computing results due to inconsistent data of multiple cache clusters, resulting in online problems and loss of client experience. At present, there are two main ways to update cache data: the single-thread serial updating cache data and the multi-thread parallel updating cache data.
When the same cache data needs to be written into a plurality of cache clusters, the following problems exist no matter the cache data is updated by adopting a single thread serial mode or the cache data is updated by adopting a multi-thread parallel mode:
1) Data consistency among multiple clusters cannot be guaranteed: and because of the reasons of cluster faults or network faults, partial cluster data refreshing is successful, and partial cluster data refreshing fails.
2) The data validation time among a plurality of clusters cannot achieve synchronous validation: the buffer data is updated by adopting a single line Cheng Chuanhang, and the scheme can not realize the synchronous effect of the data; the cache data is updated in parallel by adopting multiple threads, and due to various factors such as uncontrollable scheduling of the threads by an operating system bottom layer, pause of JVM GC (garbage collection mechanism), no strict consistency of network request response time and the like, the synchronous effectiveness of the data among a plurality of clusters can not be ensured.
Disclosure of Invention
In view of this, the embodiments of the present invention provide a method and apparatus for updating cache data, so as to solve the technical problem of data consistency between cache clusters.
To achieve the above object, according to one aspect of the embodiments of the present invention, there is provided a method for updating cache data, including:
Receiving an update request, and generating a parent task for updating the cache data to each cluster according to the update request; the update request carries information of each cluster and service identification of the cache data, and the parent task comprises a plurality of subtasks;
the cache data are acquired and respectively updated to the clusters corresponding to the subtasks;
And respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data.
Optionally, generating a parent task for updating the cache data to each cluster according to the update request includes:
Generating a task registry and updating the cache data to the father task of each cluster according to the update request;
registering the father task and the corresponding subtasks in the task registry;
The number of the subtasks is the same as that of the clusters, and the task registry stores the information of the father task, the information of each subtask corresponding to the father task, the task state of each subtask and the version information of the cache data.
Optionally, obtaining the cache data, and updating the cache data to the clusters corresponding to the subtasks respectively includes:
and for each subtask, acquiring the cache data, updating the cache data into a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Optionally, sending a version switching request to each cluster respectively includes:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
If yes, respectively sending a version switching request to each cluster;
if not, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table; and the abnormal task list stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Optionally, after the abnormal task is saved in the abnormal task table, the method further includes:
Scanning the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
Optionally, the task metadata includes a task name, a reusable component name for executing the task, and a method for executing the task;
scanning the abnormal task table, re-executing the abnormal task in the abnormal task table according to a recovery strategy, and comprising the following steps:
Scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism to call a method for executing the task so as to execute the abnormal task.
Optionally, the version information of the cached data includes service identification and timestamp information of the cached data.
In addition, according to another aspect of an embodiment of the present invention, there is provided an apparatus for updating cache data, including:
The registration module is used for receiving an update request, and generating a parent task for updating the cache data to each cluster according to the update request; the update request carries information of each cluster and service identification of the cache data, and the parent task comprises a plurality of subtasks;
the updating module is used for acquiring the cache data and respectively updating the cache data into the clusters corresponding to the subtasks;
And the switching module is used for respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data.
Optionally, the registration module is further configured to:
Generating a task registry and updating the cache data to the father task of each cluster according to the update request;
registering the father task and the corresponding subtasks in the task registry;
The number of the subtasks is the same as that of the clusters, and the task registry stores the information of the father task, the information of each subtask corresponding to the father task, the task state of each subtask and the version information of the cache data.
Optionally, the updating module is further configured to:
and for each subtask, acquiring the cache data, updating the cache data into a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Optionally, the switching module is further configured to:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
If yes, respectively sending a version switching request to each cluster;
if not, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table; and the abnormal task list stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Optionally, the method further comprises:
the recovery module is used for scanning the abnormal task table after the abnormal task is stored in the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
Optionally, the task metadata includes a task name, a reusable component name for executing the task, and a method for executing the task;
The recovery module is further configured to:
Scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism to call a method for executing the task so as to execute the abnormal task.
Optionally, the version information of the cached data includes service identification and timestamp information of the cached data.
According to another aspect of an embodiment of the present invention, there is also provided an electronic device including:
One or more processors;
Storage means for storing one or more programs,
The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the methods of any of the embodiments described above.
According to another aspect of an embodiment of the present invention, there is also provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any of the embodiments described above.
One embodiment of the above invention has the following advantages or benefits: because the technical means of respectively updating the cache data into the clusters corresponding to each subtask and respectively sending the version switching request to each cluster is adopted, the technical problem of data consistency among the cache clusters in the prior art is solved. In the embodiment of the invention, the cache cluster can simultaneously retain the historical effective version data, so that the online service can be rapidly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem can be rapidly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization and effectiveness among a plurality of cache clusters, avoid the generation of online problems and bring better user experience. Meanwhile, the bottleneck that the cache cluster cannot be expanded in one step is broken, and the problem that an online service system cannot be expanded is solved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main flow of a method for updating cache data according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main flow of a method for updating cache data according to one exemplary embodiment of the present invention;
FIG. 3 is a schematic diagram of the main flow of a method for updating cache data according to another exemplary embodiment of the present invention;
FIG. 4 is a schematic diagram of a framework for implementing a method of updating cached data according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of main modules of an apparatus for updating cache data according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
Fig. 7 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a method for updating cache data according to an embodiment of the present invention. As an embodiment of the present invention, as shown in fig. 1, the method for updating cache data may include:
step 101, receiving an update request, and generating a parent task for updating the cache data to each cluster according to the update request.
When the data in the cache cluster needs to be updated, the update request can be initiated by a user or by a timing task. After receiving the update request of the cache data, generating a parent task for updating the cache data to each cache cluster (such as a Redis cluster) according to the update request. The update request carries information of each cluster and service identification of the cache data, and the parent task comprises a plurality of subtasks. Thus, by parsing the update request, the service identifier of the cache data and the information (such as cluster ID and the like) of each cluster can be obtained. Version information (such as version number) of the cache data can be further generated according to the service identifier and the current timestamp of the cache data, so that the cache data of different versions can be distinguished conveniently, and the cache data of each version can be switched and managed conveniently.
Alternatively, the version number of the cache data may be generated using "service identification_server system time", such as route_2018.08.01-11:00:00, with each update of the data yielding a version number. The service identifier is a unique service identifier of the registration task table and is used for distinguishing different services. The server system time is the server system time when the update request is received, and the date format is for convenience of preview and quick switching of the effective version if necessary.
Optionally, generating a parent task for updating the cache data to each cluster according to the update request includes: generating a task registry and updating the cache data to the father task of each cluster according to the update request; registering the parent task and the corresponding subtasks thereof in the task registry. The number of the subtasks is the same as that of the clusters, and the task content of each subtask is to update the cache data into the cache cluster corresponding to the subtask. For example, the task content of the subtask 1 is to update the cache data into the cache cluster 1, the task content of the subtask 2 is to update the cache data into the cache cluster 2, and the task content of the subtask 3 is to update the cache data into the cache cluster 3, so that each cache cluster has the cache data of the version, and the subsequent synchronous version switching is facilitated.
Optionally, the task registry stores information of the parent task, information of each subtask corresponding to the parent task, task states of each subtask, and version information of the cache data. Specifically, the task registry may store parent task codes, child task codes, service identifiers, current effective version numbers, version numbers after modification, task states, and the like, and may also store creator, creation time, modifier, modification time, and the like information. When the parent task is generated, the same number of subtasks are automatically generated according to the number of clusters, and the subtask codes can be generated by adopting snowflake algorithm.
Step 102, obtaining the cache data, and respectively updating the cache data into the clusters corresponding to the subtasks.
Corresponding cache data can be obtained from the database according to the version number after change recorded in the task registry, and then the cache data are respectively updated into the cache clusters (such as the cache cluster 1, the cache cluster 2, the cache cluster 3 and the like) corresponding to the subtasks. During the update process, the updated cache data is not effective, so that the data consistency is not affected whether the update is serial or parallel.
Optionally, step 102 includes, for each subtask, acquiring the cache data, updating the cache data to a cluster corresponding to the subtask, and updating a task state of the subtask in the task registry to be successful. After finishing data updating of a certain cache cluster, the task state of the subtask corresponding to the cache cluster is updated to be successful in a registered task table so as to conveniently inquire the task state of each subtask, thereby ensuring the data consistency among the clusters when the versions are switched.
And step 103, respectively sending version switching requests to the clusters.
And after the data of each cache cluster is updated, respectively sending a version switching request to each cache cluster, wherein the version switching request carries the version information of the cache data. After receiving the version switching request, the cache cluster analyzes the version switching request to obtain version information of the cache data, and switches the currently effective version into a version number carried in the version switching request, so that the cache data of the version can be synchronously effective in each cluster, and the data consistency among the clusters is ensured. It should be noted that multiple versions of cached data may be stored in a cache cluster, e.g. versions that have been last few available on-line: 1.2, 3, …, 10, then the switch between these several versions can be made arbitrarily fast, so that any version can be switched quickly when there is a problem with the in-line cache data.
Optionally, sending a version switching request to each cluster respectively includes: scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful; if yes, respectively sending a version switching request to each cluster; if not, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table. And the abnormal task list stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Detecting the task state of each subtask in the registered task list is the root guarantee of realizing the data consistency among clusters, and if the task states of all subtasks are successful, distributing the latest effective version number to each cache cluster; if one subtask fails to be executed, generating an abnormal task corresponding to the subtask, and storing the abnormal task into an abnormal task table. The abnormal task may be completed by initiating a task restoration step.
According to the various embodiments described above, it can be seen that the present invention solves the technical problem of data consistency between cache clusters in the prior art by means of updating the cache data into clusters corresponding to each subtask, and sending version switching requests to each cluster. In the embodiment of the invention, the cache cluster can simultaneously retain the historical effective version data, so that the online service can be rapidly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem can be rapidly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization and effectiveness among a plurality of cache clusters, avoid the generation of online problems and bring better user experience. Meanwhile, the bottleneck that the cache cluster cannot be expanded in one step is broken, and the problem that an online service system cannot be expanded is solved.
FIG. 2 is a schematic diagram of the main flow of a method for updating cache data according to one embodiment of the present invention. As still another embodiment of the present invention, the method for updating cache data includes the steps of:
step 201, receiving and analyzing the update request to obtain the information of each cluster and the service identifier of the cache data.
When the data in the cache cluster needs to be updated, the update request can be initiated by a user or by a timing task. After receiving the update request, analyzing the update request to obtain the information of each cluster, the version information of the cache data and the service identification.
Step 202, judging whether the service has a task in execution according to the service identification; if yes, ending; if not, go to step 203.
Specifically, whether the service has an executing task can be determined by querying whether the generated task registry has an identifier of the service. If there is a task being executed, it is indicated that the cached data of the service is currently being updated, and thus a version conflict occurs, which may be resolved by the task version conflict solution, or may be directly ended.
Step 203, generating a task registry and updating the cache data to the parent task of each cluster.
If no executing task exists, indicating that no version conflict exists, generating a task registry according to the parsed information and updating the cache data to the parent task of each cluster. Wherein the parent task includes a plurality of subtasks, and the number of subtasks is the same as the number of clusters, each cluster corresponding to one subtask. Meanwhile, version information of the cache data can be generated according to the service identification and the current timestamp of the cache data.
And 204, registering the father task and the subtasks corresponding to the father task in the task registry.
And the task registry stores the information of the father task, the information of each subtask corresponding to the father task, the task state of each subtask and the version information of the cache data so as to detect the task state of each subtask.
Step 205, obtaining the cache data, and updating the cache data to the clusters corresponding to the subtasks respectively.
Corresponding cache data can be obtained from the database according to the version number after change recorded in the task registry, and then the cache data are respectively updated into the cache clusters corresponding to the subtasks. And for each subtask, acquiring the cache data, updating the cache data into a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Step 206, scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful; if yes, go to step 207; if not, go to step 208.
And step 207, respectively sending version switching requests to the clusters.
And if the task state of each subtask is successful, sending a version switching request to each cluster, wherein the version switching request carries the version information of the cache data. After receiving the version switching request, the cache cluster analyzes the version switching request to obtain version information of the cache data, and switches the currently effective version into a version number carried in the version switching request, so that the cache data of the version can be synchronously effective in each cluster, and the data consistency among the clusters is ensured.
And step 208, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table.
If one subtask fails to be executed, generating an abnormal task corresponding to the subtask, and storing the abnormal task into an abnormal task table. The abnormal task may be completed by initiating a task restoration step. And the abnormal task list stores the information, retry times, retry strategies, task states and task metadata of the subtasks.
In addition, in the embodiment of the present invention, the method for updating the cache data has been described in detail in the above description, and thus the description thereof will not be repeated here.
FIG. 3 is a schematic diagram of the main flow of a method for updating cache data according to another exemplary embodiment of the present invention. As another embodiment of the present invention, the method for updating cache data may further include the steps of:
Step 301, the task state in the abnormal task table is scanned as an abnormal task to be recovered.
The abnormal task list can be scanned at regular time by the regular task, so that the abnormal task with the task state to be recovered is obtained. Wherein, the abnormal task list stores information (such as task codes) of the subtasks, retry times, retry strategies, task states and task metadata. The task metadata includes a task Name, a reusable component Name (Bean Name) for executing the task, and a method of executing the task. The abnormal task table can also store information such as creator, creation time, modifier, modification time and the like.
Step 302, parsing task metadata of the abnormal task.
And reading the metadata of the abnormal task from the abnormal task table, and obtaining the task name of the abnormal task, the reusable component name of the execution task and the method for executing the task by analyzing the task metadata of the abnormal task.
In step 303, a reusable component for executing the task is generated through a reflection mechanism to call a method for executing the task, so as to execute the abnormal task.
Step 304, judging whether the abnormal task is successfully executed; if yes, go to step 305; if not, go to step 306.
Step 305, updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful, and updating the task state of the subtask corresponding to the abnormal task in the abnormal task table to be recovered.
Step 306, adding 1 to the retry number of the subtasks corresponding to the abnormal tasks in the abnormal task table.
In addition, in another embodiment of the present invention, the method for updating the cache data is described in detail in the above description, so that the description is not repeated here.
Fig. 4 is a schematic diagram of a framework for implementing a method for updating cache data according to an embodiment of the present invention. As shown in fig. 4, the version control module is responsible for providing functions of task registration, version number generation, and version conflict detection. Specifically, the version control module receives and analyzes the update request to obtain information of each cluster, version information of the cache data and service identification; then judging whether the service has a task in execution or not according to the service identification; if yes, ending; if not, generating a task registry, updating the cache data to the parent task of each cluster, and registering the parent task and the corresponding subtasks (such as 831 subtask, 832 subtask, 833 subtask and the like) into the task registry. Wherein the registration task table may be stored in a change control library.
The cache updating module can acquire corresponding cache data from the database according to the version number after change recorded in the task registry, and then respectively update the cache data into cache clusters (such as a cache cluster 1, a cache cluster 2, a cache cluster 3 and the like) corresponding to each subtask; and after the cache data is updated, updating the task state of the subtask in the task registry to be successful.
The task state detection module scans the task state of each subtask in the task registry and judges whether the task state of each subtask is successful; if yes, respectively sending a version switching request to each cluster; if not, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table.
The abnormal task recovery module is responsible for scanning the abnormal task table at regular time and re-executing the abnormal task according to the recovery strategy. Specifically, the abnormal task recovery module scans the abnormal task table at regular time, so as to obtain the abnormal task with the task state to be recovered; analyzing task metadata of the abnormal task; generating a reusable component for executing the task through a reflection mechanism to call a method for executing the task, thereby executing the abnormal task; if the execution is successful, the task state of the subtask corresponding to the abnormal task in the task registry is updated to be successful, and the task state of the subtask corresponding to the abnormal task in the abnormal task table is updated to be recovered; if the execution is unsuccessful, adding 1 to the retry number of the subtasks corresponding to the abnormal tasks in the abnormal task table.
The cache version management module is a WEB end of the whole framework and can provide functions of cache version inquiry, manual rollback of the cache version, historical version clearing strategy configuration and the like.
The framework provided by the embodiment of the invention enables each application to be accessed with smaller modification cost through the abstraction of the underlying data model and the mutual cooperation among the modules, and the infinite copy of the cache cluster is replaced, so that the infinite expansibility in theory is brought, and the cache cluster decoupling among the applications is used.
Since multiple versions of cache data are stored in the cache cluster, the interface for the client to read the cache needs to be adapted. In order to be transparent to the upper layer service, a proxy class of the read cache interface can be generated by adopting an interceptor method (a bottom layer principle: byte code modification), and the latest generation version number is woven in the proxy class.
Fig. 5 is a schematic diagram of main modules of an apparatus for updating cache data according to an embodiment of the present invention, and as shown in fig. 5, the apparatus 500 for updating cache data includes a registration module 501, an update module 502, and a switching module 503. The registration module 501 is configured to receive an update request, and generate a parent task for updating the cache data to each cluster according to the update request; the update request carries information of each cluster and service identification of the cache data, and the parent task comprises a plurality of subtasks; the updating module 502 is configured to obtain the cache data, and update the cache data to clusters corresponding to the subtasks respectively; the switching module 503 is configured to send a version switching request to each cluster, where the version switching request carries version information of the cached data.
Optionally, the registration module 501 is further configured to:
Generating a task registry and updating the cache data to the father task of each cluster according to the update request;
registering the father task and the corresponding subtasks in the task registry;
The number of the subtasks is the same as that of the clusters, and the task registry stores the information of the father task, the information of each subtask corresponding to the father task, the task state of each subtask and the version information of the cache data.
Optionally, the updating module 502 is further configured to:
and for each subtask, acquiring the cache data, updating the cache data into a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
Optionally, the switching module 503 is further configured to:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
If yes, respectively sending a version switching request to each cluster;
if not, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table; and the abnormal task list stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
Optionally, the method further comprises:
the recovery module is used for scanning the abnormal task table after the abnormal task is stored in the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
Optionally, the task metadata includes a task name, a reusable component name for executing the task, and a method for executing the task;
The recovery module is further configured to:
Scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism to call a method for executing the task so as to execute the abnormal task.
Optionally, the version information of the cached data includes service identification and timestamp information of the cached data.
According to the various embodiments described above, it can be seen that the present invention solves the technical problem of data consistency between cache clusters in the prior art by means of updating the cache data into clusters corresponding to each subtask, and sending version switching requests to each cluster. In the embodiment of the invention, the cache cluster can simultaneously retain the historical effective version data, so that the online service can be rapidly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem can be rapidly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization and effectiveness among a plurality of cache clusters, avoid the generation of online problems and bring better user experience. Meanwhile, the bottleneck that the cache cluster cannot be expanded in one step is broken, and the problem that an online service system cannot be expanded is solved.
In addition, the specific implementation of the apparatus for updating cache data according to the present invention is described in detail in the method for updating cache data, so that the description is not repeated here.
Fig. 6 illustrates an exemplary system architecture 600 to which a method of updating cached data or an apparatus of updating cached data according to an embodiment of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 is used as a medium to provide communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 605 via the network 604 using the terminal devices 601, 602, 603 to receive or send messages, etc. Various communication client applications such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 601, 602, 603.
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 605 may be a server providing various services, such as a background management server (by way of example only) providing support for shopping-type websites browsed by users using terminal devices 601, 602, 603. The background management server may analyze and process the received data such as the article information query request, and feedback the processing result (e.g., the target push information, the article information—only an example) to the terminal device.
It should be noted that, the method for updating the cache data provided in the embodiment of the present invention is generally executed by the server 605, and accordingly, the device for updating the cache data is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, ROM 702, and RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a registration module, an update module, and a switch module, where the names of the modules do not constitute a limitation on the module itself in some cases.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: receiving an update request, and generating a parent task for updating the cache data to each cluster according to the update request; the update request carries information of each cluster and service identification of the cache data, and the parent task comprises a plurality of subtasks; the cache data are acquired and respectively updated to the clusters corresponding to the subtasks; and respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data.
According to the technical scheme of the embodiment of the invention, the technical means of respectively updating the cache data into the clusters corresponding to each subtask and respectively sending the version switching request to each cluster are adopted, so that the technical problem of data consistency among the cache clusters in the prior art is solved. In the embodiment of the invention, the cache cluster can simultaneously retain the historical effective version data, so that the online service can be rapidly guided to the correct data only by performing a switching operation on the version number of the data, and the online problem can be rapidly avoided. Therefore, the embodiment of the invention can effectively avoid the problems of data consistency and data synchronization and effectiveness among a plurality of cache clusters, avoid the generation of online problems and bring better user experience. Meanwhile, the bottleneck that the cache cluster cannot be expanded in one step is broken, and the problem that an online service system cannot be expanded is solved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (9)
1. A method of updating cached data, comprising:
receiving an update request, and generating a parent task for updating the cache data to each cluster according to the update request; the update request carries information of each cluster and service identification of the cache data, the parent task comprises a plurality of subtasks, and the number of the subtasks is the same as that of the clusters;
the cache data are acquired and respectively updated to the clusters corresponding to the subtasks;
Respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data;
generating a parent task for updating the cache data to each cluster according to the update request, wherein the parent task comprises the following steps:
Generating a task registry and updating the cache data to the father task of each cluster according to the update request;
registering the father task and the corresponding subtasks in the task registry;
And the task registry stores the information of the father task, the information of each subtask corresponding to the father task, the task state of each subtask and the version information of the cache data.
2. The method of claim 1, wherein obtaining the cache data and updating the cache data to the clusters corresponding to the respective subtasks respectively comprises:
and for each subtask, acquiring the cache data, updating the cache data into a cluster corresponding to the subtask, and updating the task state of the subtask in the task registry to be successful.
3. The method of claim 2, wherein sending version switch requests to the respective clusters, respectively, comprises:
scanning the task state of each subtask in the task registry, and judging whether the task state of each subtask is successful;
If yes, respectively sending a version switching request to each cluster;
if not, constructing an abnormal task for the failed subtask according to the task state, and storing the abnormal task into an abnormal task table; and the abnormal task list stores the information of the subtasks, the retry times, the retry strategy, the task state and the task metadata.
4. The method of claim 3, further comprising, after saving the abnormal task in an abnormal task table:
Scanning the abnormal task table, and re-executing the abnormal task in the abnormal task table according to a recovery strategy;
and updating the task state of the subtask corresponding to the abnormal task in the task registry to be successful.
5. The method of claim 4, wherein the task metadata includes a task name, a reusable component name for executing a task, and a method of executing a task;
scanning the abnormal task table, re-executing the abnormal task in the abnormal task table according to a recovery strategy, and comprising the following steps:
Scanning the task state in the abnormal task table as an abnormal task to be recovered;
and analyzing the task metadata of the abnormal task, and generating a reusable component for task execution through a reflection mechanism to call a method for executing the task so as to execute the abnormal task.
6. The method of claim 1, wherein the version information of the cached data includes service identification and timestamp information of the cached data.
7. An apparatus for updating cached data, comprising:
The registration module is used for receiving an update request, and generating a parent task for updating the cache data to each cluster according to the update request; the update request carries information of each cluster and service identification of the cache data, the parent task comprises a plurality of subtasks, and the number of the subtasks is the same as that of the clusters;
the updating module is used for acquiring the cache data and respectively updating the cache data into the clusters corresponding to the subtasks;
the switching module is used for respectively sending version switching requests to each cluster, wherein the version switching requests carry version information of the cache data;
The registration module is further configured to:
Generating a task registry and updating the cache data to the father task of each cluster according to the update request;
registering the father task and the corresponding subtasks in the task registry;
And the task registry stores the information of the father task, the information of each subtask corresponding to the father task, the task state of each subtask and the version information of the cache data.
8. An electronic device, comprising:
One or more processors;
Storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.
9. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010037735.7A CN113127443B (en) | 2020-01-14 | 2020-01-14 | Method and device for updating cache data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010037735.7A CN113127443B (en) | 2020-01-14 | 2020-01-14 | Method and device for updating cache data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113127443A CN113127443A (en) | 2021-07-16 |
| CN113127443B true CN113127443B (en) | 2024-06-18 |
Family
ID=76771137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010037735.7A Active CN113127443B (en) | 2020-01-14 | 2020-01-14 | Method and device for updating cache data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127443B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114745438B (en) * | 2022-03-22 | 2024-02-09 | 北京乐我无限科技有限责任公司 | Cache data processing method, device and system of multi-data center and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107315629A (en) * | 2017-06-14 | 2017-11-03 | 北京小米移动软件有限公司 | Task processing method, device and storage medium |
| CN110175159A (en) * | 2019-05-29 | 2019-08-27 | 京东数字科技控股有限公司 | Method of data synchronization and system for object storage cluster |
| CN110442605A (en) * | 2019-07-11 | 2019-11-12 | 阿里巴巴集团控股有限公司 | The buffer memory management method and device of server |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102790788B (en) * | 2011-05-23 | 2014-12-10 | 同济大学 | Grid resource management system |
| US8966457B2 (en) * | 2011-11-15 | 2015-02-24 | Global Supercomputing Corporation | Method and system for converting a single-threaded software program into an application-specific supercomputer |
| US9600500B1 (en) * | 2013-06-21 | 2017-03-21 | Amazon Technologies, Inc. | Single phase transaction commits for distributed database transactions |
| CN104331255B (en) * | 2014-11-17 | 2018-04-17 | 中国科学院声学研究所 | A kind of stream data read method based on embedded file system |
| CN106533719A (en) * | 2015-09-10 | 2017-03-22 | 中兴通讯股份有限公司 | Method and system for realizing cluster deployment |
| CN107133234B (en) * | 2016-02-29 | 2021-04-27 | 创新先进技术有限公司 | Method, device and system for updating cache data |
| CN109582684B (en) * | 2018-11-30 | 2021-11-09 | 深圳市乐信信息服务有限公司 | Local cache data updating method and device, server and storage medium |
| CN109451065B (en) * | 2018-12-26 | 2021-06-01 | 中电福富信息科技有限公司 | Soft load balancing and shunting automation system and operation method thereof |
-
2020
- 2020-01-14 CN CN202010037735.7A patent/CN113127443B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107315629A (en) * | 2017-06-14 | 2017-11-03 | 北京小米移动软件有限公司 | Task processing method, device and storage medium |
| CN110175159A (en) * | 2019-05-29 | 2019-08-27 | 京东数字科技控股有限公司 | Method of data synchronization and system for object storage cluster |
| CN110442605A (en) * | 2019-07-11 | 2019-11-12 | 阿里巴巴集团控股有限公司 | The buffer memory management method and device of server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113127443A (en) | 2021-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108984170B (en) | H5 page multi-language rendering method and device | |
| US9721246B2 (en) | Synchronization patterns for mobile applications | |
| CN110019350A (en) | Data query method and apparatus based on configuration information | |
| CN111694857B (en) | Method, device, electronic equipment and computer readable medium for storing resource data | |
| JP5682996B2 (en) | Client program, terminal, server device, server program, system and method | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN111143382A (en) | Data processing method, system and computer readable storage medium | |
| CN110727468A (en) | Method and apparatus for managing algorithm models | |
| CN112948498A (en) | Method and device for generating global identification of distributed system | |
| CN110162334B (en) | Code management method, device and storage medium | |
| CN104809199A (en) | Database synchronization method and device | |
| CN111782235B (en) | Data upgrading and inquiring method and device | |
| CN113779082B (en) | Method and device for updating data | |
| CN111753226A (en) | A page loading method and device | |
| US20210149709A1 (en) | Method and apparatus for processing transaction | |
| CN113127443B (en) | Method and device for updating cache data | |
| CN113760860B (en) | Data reading method and device | |
| CN112463887B (en) | A data processing method, device, equipment and storage medium | |
| CN112463251B (en) | A method and device for hot publishing of uliweb framework configuration | |
| US10728323B2 (en) | Method and apparatus for operating infrastructure layer in cloud computing architecture | |
| CN113076343B (en) | Data query method, device, equipment and storage medium | |
| CN114281786B (en) | File processing method, device, electronic equipment and storage medium | |
| CN115454666A (en) | Data synchronization method and device among message queue clusters | |
| CN110309157A (en) | A data synchronization update method and device | |
| CN115167769A (en) | Method, device and system for writing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |