CN110795495B - Data processing method, device, electronic equipment and computer readable medium - Google Patents
Data processing method, device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN110795495B CN110795495B CN201810783276.XA CN201810783276A CN110795495B CN 110795495 B CN110795495 B CN 110795495B CN 201810783276 A CN201810783276 A CN 201810783276A CN 110795495 B CN110795495 B CN 110795495B
- Authority
- CN
- China
- Prior art keywords
- data
- full
- version number
- incremental
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a data processing method, a data processing device, electronic equipment and a computer readable medium, and belongs to the technical field of Internet. The method comprises the following steps: the data center acquires full data or incremental data from the database; synchronizing the full data and the corresponding version number to the cloud service so that the cloud service stores the full data and the version number; and synchronizing the incremental data and the version number corresponding to the incremental data to the client through a remote procedure call protocol (RPC) service. The client side only needs to read the incremental data from the data center by means of the version number of the current existing data, the phenomenon that the database links are occupied due to the fact that the number of the client side is too large or the requested data volume is too large can be prevented, only the data center accesses the database and acquires the incremental data in the whole process, the client side does not directly access the database any more, when the access volume of the client side is too large, effective and accurate data service can be provided through the capacity-expanding data center, and complex and expensive data capacity-expanding means are avoided.
Description
Technical Field
The present disclosure relates generally to the field of internet technology, and in particular, to a data processing method, apparatus, electronic device, and computer readable medium.
Background
At present, with the increase of users, the concurrency of service requests for accessing the database is increased, and the requirement on the access speed is also increased, whether the database is accessed through service or through unified cache middleware, the network overhead cannot be avoided, and thus the network overhead is increased.
In order to solve the problem of network overhead, the required data can be directly stored in the memory of the service, so that the requirement of the access speed can be well met, and the concurrent processing speed can be improved. However, the required data is directly stored in the memory of the service, and various problems, such as consistency problems, are inevitably generated.
Therefore, there is still a need for improvement in the technical solutions of the prior art.
The above information disclosed in the background section is only for enhancement of understanding of the background of the disclosure and therefore it may include information that does not form the prior art that is already known to a person of ordinary skill in the art.
Disclosure of Invention
The present disclosure provides a data processing method, apparatus, electronic device, and computer readable medium, which solve the above technical problems.
Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.
According to an aspect of the present disclosure, there is provided a data processing method including:
the data center acquires full data or incremental data from a database;
synchronizing the full data and the corresponding version number to a cloud service so that the cloud service stores the full data and the version number;
And synchronizing the incremental data and the version numbers corresponding to the incremental data to the client through a remote procedure call protocol (RPC) service.
In one embodiment of the present disclosure, the data center obtaining full or incremental data from a database includes:
judging whether the version number exists in the data center, and obtaining a judging result to be existence or nonexistence;
If the judging result is that the total data does not exist, the total data is firstly obtained from the database, and the total data and the corresponding version number are stored in a memory of a data center;
if the judgment result is that the version number exists, acquiring corresponding incremental data from the database according to the version number, and storing the incremental data in a memory of a data center;
wherein the version number is a timestamp of the full or incremental data obtained from the database.
In one embodiment of the present disclosure, after the data center obtains the full amount of data from the database, the method further includes:
storing the full data stored in the memory of the data center into a local file of the data center in combination with a corresponding version number;
and checking the full-volume data, and synchronizing the full-volume data and the corresponding version number to the cloud service if the check is passed.
In one embodiment of the present disclosure, after the data center obtains the incremental data from the database, the method further includes:
storing the incremental data stored in the memory of the data center into a local file of the data center;
and combining the original full-volume data stored in the local file of the data center with the incremental data and the corresponding version number to form new full-volume data, and storing the new full-volume data into the local file of the data center.
In one embodiment of the present disclosure, synchronizing the full amount of data and version numbers to a cloud service includes:
And the data center synchronizes the full-volume data acquired for the first time with a corresponding version number or the new full-volume data with the corresponding version number to the cloud service.
In one embodiment of the present disclosure, after synchronizing the incremental data and the version number corresponding to the incremental data to the client by the remote procedure call protocol RPC service, the method further includes:
the client acquires the full data and the corresponding version number from the cloud service;
checking the full data and the full data stored in the local file of the client, and acquiring new full data according to a checking result;
and storing the new full data and the corresponding version number in a local file of the client.
In one embodiment of the present disclosure, further comprising:
If the client is restarted, the client tries to load the full data and the corresponding version number stored in the local file of the client after restarting; if loading fails, the client acquires full-volume data and corresponding version numbers from the cloud service, acquires incremental data from the data center through the RPS service, and stores the incremental data in a memory of the client.
According to still another aspect of the present disclosure, there is provided a data processing apparatus including:
a data acquisition module configured to acquire full-scale data or incremental data from a database by the data center;
A full-volume synchronization module configured to synchronize the full-volume data and the corresponding version number to a cloud service so that the cloud service stores the full-volume data and the version number;
And the increment synchronization module is configured to synchronize the increment data and the version number corresponding to the increment data to the client through a remote procedure call protocol (RPC) service.
According to yet another aspect of the present disclosure, there is provided an electronic device including a processor; a memory storing instructions for the processor to control the method steps as described above.
According to another aspect of the present disclosure, there is provided a computer readable medium having stored thereon computer executable instructions which when executed by a processor implement the method steps as described above.
According to the data processing method, the device, the electronic equipment and the computer readable medium provided by the embodiment of the disclosure, on one hand, the data is firstly read from the database by adding the data center, then the full data and the incremental data are respectively treated, namely, the full data is synchronized to the cloud service, and the incremental data is synchronized to the client, so that the client only needs to read the incremental data from the data center by means of the version number of the current existing data, the phenomenon that the database link is full due to the excessive number of the client or the excessive request data can be prevented, only the data center accesses the database and acquires the incremental data in the whole process, the client can not directly access the database any more, and effective and accurate data service can be provided through the capacity expansion data center when the access amount of the client is excessive, and the complicated and expensive data capacity expansion means are avoided. On the other hand, as the full-volume data and the corresponding version number are stored in the cloud service, the problem that the full-volume data is inconsistent with the database data, which is easily caused by the fact that the client directly writes the full-volume data into the local file of the client, can be avoided; and the cloud service can be used for conveniently managing the whole data, so that the problems can be rapidly checked and the error data can be rapidly repaired. The scheme does not pass through hardware or middleware, and can reduce network overhead.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.
Fig. 1 is a system architecture diagram of a data processing method and apparatus according to an embodiment of the present disclosure.
Fig. 2 shows a flow chart of a data processing method provided in an embodiment of the present disclosure.
Fig. 3 is a flowchart illustrating step S210 in fig. 2 according to an embodiment of the disclosure.
Fig. 4 is a flowchart illustrating step S230 in fig. 2 according to an embodiment of the disclosure.
FIG. 5 is a flow diagram of data associated with a system architecture in an embodiment of the disclosure.
Fig. 6 shows an overall flowchart of data processing steps in an embodiment of the present disclosure.
Fig. 7 shows a schematic diagram of a data processing apparatus provided in another embodiment of the present disclosure.
Fig. 8 is a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.
Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
The present invention will be further described in detail below with reference to specific embodiments and with reference to the accompanying drawings, in order to make the objects, technical solutions and advantages of the present invention more apparent.
In a related embodiment of the disclosure, for the unified scheme of the distributed memory, each client may be directly connected to the database to perform data synchronization, then perform persistent full-size data locally, and then perform incremental data synchronization through the database. This solution. If all the clients are directly connected with the database to update the local memory, firstly, if the clients are gradually increased, the database becomes a bottleneck for expanding services, and if the number of the clients is excessive or the request data volume is excessive, the database links are fully occupied, so that the number of the loaded clients is limited. In addition, since each service is directly connected with the database and then the local file is made persistent, each service needs to store a local full-volume file, and then the service is loaded from the local, so that the problem is that if data are inconsistent, the data are difficult to check, the full-volume files of each service need to be downloaded and compared, and many problems are brought to data restoration.
In addition, the method can be implemented by hardware or middleware (such as Redis and the like). The consistency of each service memory is realized through hardware, for example, a data bus is realized, the cost is relatively high, and if the capacity expansion and the capacity contraction are carried out, the data bus is required to be simultaneously accessed or cut out, and the circuit is complex; in addition, there is a strong dependence on hardware, and if the data bus goes wrong, various inconsistency problems can be caused. The middleware is used for unified storage, network overhead is inevitably generated, and the performance is affected.
Based on the above, it is necessary to provide a unified memory scheme of a distributed system, which can effectively and accurately update the memory and ensure the consistency of memory data in the distributed system on the premise of not affecting the service performance.
Fig. 1 is a system architecture diagram of a data processing method and apparatus according to an embodiment of the present disclosure.
As shown in fig. 1, system architecture 100 may include a database 101, a data center 102, cloud services 103, and clients 104. In addition, the system also includes a network for providing a medium for communication links between the database and the data center, between the data center and the cloud service, between the data center and the client, and between the cloud service and the client. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others. Wherein in the system architecture 100 the data center 102 and clients 104 are each formed in clusters, i.e., data center service clusters and client clusters.
Based on the system architecture described above, fig. 2 shows a flowchart of a data processing method provided in an embodiment of the present disclosure, where the method is used in a data center in the system architecture shown in fig. 1, and includes the following steps:
As shown in fig. 2, in step S210, the data center acquires full-amount data or incremental data from a database.
As shown in fig. 2, in step S220, the full amount data and the corresponding version number are synchronized to a cloud service, so that the cloud service stores the full amount data and the version number.
As shown in fig. 2, in step S230, the incremental data and the version number corresponding to the incremental data are synchronized to the client through a remote procedure call protocol (Remote Procedure Call Protocol, abbreviated as RPC) service.
According to the data processing method provided by the embodiment of the disclosure, the data is firstly read from the database through the data center, then the full-volume data and the incremental data are respectively treated, namely, the full-volume data is synchronized to the cloud service, and the incremental data is synchronized to the client, so that the client only needs to read the incremental data from the data center by means of the version number of the current existing data, the phenomenon that the database links are occupied due to too many clients or too large request data volume can be prevented, only the data center accesses the database and acquires the incremental data in the whole process, the client does not directly access the database any more, and when the access amount of the client is too large, effective and accurate data service can be provided through the capacity expansion data center, and a complex and expensive data capacity expansion means is avoided.
The data processing method provided in the present disclosure is described in detail below with reference to the flowchart shown in fig. 2, and specifically as follows:
in step S210, the data center acquires full-volume data or incremental data from a database.
In one embodiment of the present disclosure, unlike the prior art, the data is not obtained by the client accessing the database directly, but rather by the data center accessing the database.
Fig. 3 shows a flow chart of data acquisition by a data center, comprising the steps of:
As shown in fig. 3, in step S310, it is determined whether or not a version number exists in the data center, and the determination result is presence or absence. If the determination result is not present, the first start is indicated, and the process goes to step S320, and if the determination result is present, the process goes to step S330.
The version number is a timestamp of acquiring the full-volume data or the incremental data from the database, and because the data is transmitted and stored among the database, the data center, the cloud service and the clients in sequence, the time inconsistency among different components in the architecture system is avoided, meanwhile, in order to avoid the time inconsistency among the data centers and the clients, the timestamp of acquiring the data from the database is used as the version number, the timestamp is unique and is generated along with the generation of the full-volume data, so that the version number and the data file can be paired, the incremental data can be acquired again from the version number node through the full-volume data file and the version number when the memory data is lost, missing data is avoided, and data recovery can be performed rapidly.
As shown in fig. 3, in step S320, the full-size data is obtained from the database, and the full-size data and the corresponding version number are stored in the memory of the data center, and then the process goes to step S340.
As shown in fig. 3, in step S330, corresponding incremental data is obtained from the database according to the version number, and the incremental data is stored in the memory of the data center.
As shown in fig. 3, in step S340, RPC service data, that is, incremental data to be synchronized to a client, is updated according to data in a memory of a data center.
Based on the steps shown in fig. 3, the data center acquires full-volume data or incremental data from the database by adopting a timing trigger mechanism, and after a task is triggered at fixed time, the data center loads the data file after the current version number in the database DB according to the version number, writes the data file into a memory of the data center, and updates the RPC service data in the memory. If no version number indicates the first start, the full data is loaded from the DB and written into the memory (i.e., the memory of the data center).
In one embodiment of the present disclosure, after the data center obtains the full amount of data from the database in step S320, the method further includes:
and storing the full data stored in the memory of the data center into a local file of the data center by adopting a timing trigger mechanism and combining the corresponding version number. And then, checking the full data, and synchronizing the full data and the corresponding version number to the cloud service if the check is passed.
The verification may be performed by using a hash algorithm (e.g., MD 5), specifically, comparing the current MD5 value of the local file with the standard MD5 value to verify whether it is correct and complete, so as to ensure that the latest full-size data and the corresponding version number are written into the cloud service.
In one embodiment of the present disclosure, after the data center obtains the incremental data from the database in step S330, the method further includes:
Firstly, storing the increment data stored in the memory of the data center into a local file of the data center; and secondly, combining the original full-volume data stored in the local file of the data center with the incremental data and the corresponding version number to form new full-volume data, and storing the new full-volume data into the local file of the data center.
In step S220, the full amount of data and the corresponding version number are synchronized to a cloud service, so that the cloud service stores the full amount of data and the version number.
In one embodiment of the present disclosure, synchronizing the full amount of data and version numbers to a cloud service includes:
And the data center synchronizes the full-volume data acquired for the first time with a corresponding version number or the new full-volume data with the corresponding version number to the cloud service.
In step S230, the incremental data and the version number corresponding to the incremental data are synchronized to the client by the remote procedure call protocol RPC service.
In one embodiment of the present disclosure, fig. 4 shows a step flow after step S230 synchronizes the incremental data and the version number corresponding to the incremental data to the client through the remote procedure call protocol RPC service in the present embodiment, including the following steps:
As shown in fig. 4, in step S410, the client obtains the full-size data and the corresponding version number from the cloud service.
As shown in fig. 4, in step S420, the full-volume data and the full-volume data stored in the local file of the client are verified, and new full-volume data is obtained according to the verification result.
As shown in fig. 4, in step S430, the new full-size data and the corresponding version number are stored in the local file of the client.
Based on the steps shown in fig. 4, the latest full-size data and the corresponding version number are always stored in the local file of the client. And the client accesses the data center through the RPC range according to the starting time in the memory or the version number after the last increment updating, and updates the increment data in the local memory. In addition, the client side acquires the full-volume data and the corresponding version number from the cloud service at regular time, updates the local full-volume data file and the version file, and ensures that the full-volume file can be normally loaded from the local if the cloud service is abnormal in the next starting.
In one embodiment of the present disclosure, after step S230, further includes:
If the client is restarted, the client tries to load the full data and the corresponding version number stored in the local file of the client after restarting; if loading fails, the client acquires full-volume data and a corresponding version number from the cloud service, acquires incremental data from the data center through an RPC service, and stores the incremental data in a memory of the client; and if the loading is successful, directly loading the full data and the corresponding version number stored in the local file of the client.
In this way, under the condition that the client normally works without restarting, the client acquires the full data and stores the full data in the local file of the client through accessing the cloud service, and acquires the incremental data from the data center through the RPC service and stores the incremental data in the memory of the client. If the client is restarted due to reasons, firstly loading the full data and the corresponding version number in the local file of the client after restarting, if recording fails, directly loading from the cloud service, and regularly acquiring incremental data from a data center through the RPC service to update the memory of the client, thereby ensuring the data consistency of each client in the client cluster.
Based on the foregoing, fig. 5 is a schematic diagram of a data flow of a system architecture, and details of a data processing method in an embodiment of the disclosure are described below:
as shown in fig. 5, in step S501, the cache of the data update data center is acquired at regular time.
As shown in fig. 5, in step S502, the full-size data and the version number are timing-synchronized.
As shown in fig. 5, in step S503, the RPC service synchronizes the incremental data.
As shown in fig. 5, in step S504, the full amount of data and the version number are acquired at regular time.
Through the system architecture and the corresponding data stream transmission direction shown in fig. 5, in the system architecture, the data center can avoid the bottleneck that the DB becomes a cluster to expand capacity, and prevent the database link from being full due to excessive number of clients and excessive request data volume. Through the data center, DB access by only the service of the data center is realized, and the client side does not directly access the DB any more, so that free capacity expansion of the client side is realized. The cloud service is used as a persistent data warehouse of the full-volume data file and version number in the system architecture, so that the problem that the full-volume data file is inconsistent easily caused by the fact that the client directly writes the full-volume data file into the local of the client is avoided. The cloud service is used for uniformly managing the total data files, so that the data problem in the memory can be conveniently and rapidly checked, and meanwhile, the purpose of rapidly repairing the memory data in each service can be achieved by repairing the data of the cloud service. The client is a server, the client needs to acquire full data and incremental data, wherein the full data is acquired by accessing a cloud service, a local file is written under the condition of no restarting, and the incremental data in the memory of the server is updated in an increment mode by accessing an RPC service, so that the data consistency of each server is ensured.
Fig. 6 shows an overall flow chart of the above data processing steps, including the steps of:
as shown in fig. 6, in step S601, the data center acquires data from the DB, including full-volume data file generation started for the first time and subsequent incremental data acquisition.
As shown in fig. 6, in step S602, the data center writes the data in the memory into the local file at regular time, and guarantees that the full amount of data is successfully written into the cloud service according to the MD5 check.
As shown in fig. 6, in step S603, the data center provides RPC service of incremental data, and provides service for clients.
As shown in fig. 6, in step S604, the client cluster starts loading the full amount of data from the cloud service for the first time, writes the local file, and regularly accesses the RPC service of the data center to update the data in the memory of the client.
As shown in fig. 6, in step S605, the client service acquires the full-size data from the cloud service at regular time, performs a comparison check with MD5 of the local file, and writes the latest full-size data and the corresponding version number into the local file.
As shown in fig. 6, in step S606, if the client is restarted, the local full quantity and the corresponding version number are loaded first, if the acquisition fails, the loading is directly performed from the cloud service, and then the memory of the client is updated through the RPC service at regular time.
In addition, on the basis of the step flow shown in fig. 6, if an abnormality occurs in a data center, if an abnormality occurs in a certain data center service in the data center service cluster, other data centers can normally provide RPC service and timing update of the full-size file; under the condition that all the data center servers are abnormal, the client side only can not normally update incremental data within a certain time, and after the data center is restored, the incremental RPC service can be accurately and continuously provided according to the full data file and version number in the cloud service, and data omission does not occur. Under the condition that the cloud service is abnormal, the client can continuously access the data center at regular time through the full data in the local file and the corresponding version number to provide the memory service; the data center can continue to provide incremental data service, only the full-volume data cannot be updated temporarily, after the cloud service is restored, the data center can continue to write the latest full-volume data and the corresponding version number into the cloud service, so that the client can continue to update the local full-volume data and the corresponding version number, and the data cannot be omitted.
It should be noted that, in the system architecture, cloud service and RPC service call implementation can be customized, and version number and full data pairing is used for performing memory service, so that the problem that memory data consistency is difficult to locate and repair is solved, dependence on unified memory hardware and network overhead of cache middleware are avoided, and a unified memory scheme in a distributed system with a robust fail over mechanism is provided.
In summary, according to the data processing method provided by the embodiment of the disclosure, on one hand, by adding the data center to read data from the database, and then treating the full amount of data and the incremental data respectively, that is, synchronizing the full amount of data to the cloud service and synchronizing the incremental data to the client, the client only needs to read the incremental data from the data center by means of the version number of the current existing data, so that the phenomenon that the database link is full due to excessive number of the client or excessive request data can be prevented, only the data center accesses the database and acquires the incremental data in the whole process, the client does not directly access the database any more, and when the access amount of the client is excessive, effective and accurate data service can be provided through the capacity expansion data center, and a complex and expensive data capacity expansion means is avoided. On the other hand, the problem that the full-volume data and the database data are inconsistent due to the fact that the full-volume data and the corresponding version number are stored in the cloud service and the client side directly writes the full-volume data into the local file of the client side can be avoided; and the cloud service can be used for conveniently managing the whole data, so that the problems can be rapidly checked and the error data can be rapidly repaired. The scheme does not pass through hardware or middleware, and can reduce network overhead.
Fig. 7 shows a schematic diagram of a data processing apparatus provided in another embodiment of the disclosure, as shown in fig. 7, the apparatus 700 includes: a data acquisition module 710, a full synchronization module 720, and an incremental synchronization module 730.
The data acquisition module 710 is configured for the data center to acquire full or incremental data from a database; the full-volume synchronization module 720 is configured to synchronize the full-volume data and the corresponding version number to a cloud service, so that the cloud service stores the full-volume data and the version number; the delta synchronization module 730 is configured to synchronize the delta data and the version number corresponding to the delta data to the client through a remote procedure call protocol RPC service.
The functions of each module in the apparatus are described in the above method embodiments, and are not described herein.
In summary, according to the data processing device provided by the embodiment of the disclosure, on one hand, by adding the data center to read data from the database, and then treating the full amount of data and the incremental data respectively, that is, synchronizing the full amount of data to the cloud service and synchronizing the incremental data to the client, the client only needs to read the incremental data from the data center by means of the version number of the current existing data, so that the phenomenon that the database link is full due to excessive number of the client or excessive request data can be prevented, only the data center accesses the database and acquires the incremental data in the whole process, the client does not directly access the database any more, and when the access amount of the client is excessive, effective and accurate data service can be provided through the capacity expansion data center, and a complex and expensive data capacity expansion means is avoided. On the other hand, the problem that the full-volume data and the database data are inconsistent due to the fact that the full-volume data and the corresponding version number are stored in the cloud service and the client side directly writes the full-volume data into the local file of the client side can be avoided; and the cloud service can be used for conveniently managing the whole data, so that the problems can be rapidly checked and the error data can be rapidly repaired. The scheme does not pass through hardware or middleware, and can reduce network overhead.
In another aspect, the present disclosure also provides an electronic device, including a processor and a memory, the memory storing operating instructions for the processor to control:
The data center acquires full data or incremental data from a database; synchronizing the full data and the corresponding version number to a cloud service so that the cloud service stores the full data and the version number; and synchronizing the incremental data and the version numbers corresponding to the incremental data to the client through a remote procedure call protocol (RPC) service.
Referring now to FIG. 8, there is illustrated a schematic diagram of a computer system 800 suitable for use in implementing an electronic device of an embodiment of the present application. The electronic device shown in fig. 8 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments of the application.
As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage section 807 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, mouse, etc.; an output portion 807 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 808 including a hard disk or the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. The drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as needed, so that a computer program read out therefrom is mounted into the storage portion 808 as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 809, and/or installed from the removable media 811. The above-described functions defined in the system of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 801.
The computer readable medium shown in the present application may be a computer readable signal medium or a computer readable medium, or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium that is not a computer-readable medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present application may be implemented in software or in hardware. The described units may also be provided in a processor, for example, described as: a processor includes a transmitting unit, an acquiring unit, a determining unit, and a first processing unit. The names of these units do not constitute a limitation on the unit itself in some cases, and for example, the transmitting unit may also be described as "a unit that transmits a picture acquisition request to a connected server".
In another aspect, the present disclosure also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise the method steps of:
The data center acquires full data or incremental data from a database; synchronizing the full data and the corresponding version number to a cloud service so that the cloud service stores the full data and the version number; and synchronizing the incremental data and the version numbers corresponding to the incremental data to the client through a remote procedure call protocol (RPC) service.
It should be clearly understood that the present disclosure describes how to make and use particular examples, but the principles of the present disclosure are not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.
Exemplary embodiments of the present disclosure are specifically illustrated and described above. It is to be understood that this disclosure is not limited to the particular arrangements, instrumentalities and methods of implementation described herein; on the contrary, the disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. A method of data processing, comprising:
the data center acquires full data or incremental data from a database, wherein the data center and the database are connected through a network;
Synchronizing the full data and the corresponding version number to a cloud service so that the cloud service stores the full data and the version number;
Synchronizing the incremental data and version numbers corresponding to the incremental data to a client through a remote procedure call protocol (RPC) service;
wherein the data center obtaining full or incremental data from the database comprises:
judging whether the version number exists in the data center, and obtaining a judging result to be existence or nonexistence;
If the judging result is that the total data does not exist, the total data is firstly obtained from the database, and the total data and the corresponding version number are stored in a memory of a data center;
if the judgment result is that the version number exists, acquiring corresponding incremental data from the database according to the version number, and storing the incremental data in a memory of a data center;
wherein the version number is a timestamp of the full or incremental data obtained from the database.
2. The data processing method according to claim 1, wherein after the data center acquires the full amount of data from the database, further comprising:
storing the full data stored in the memory of the data center into a local file of the data center in combination with a corresponding version number;
and checking the full-volume data, and synchronizing the full-volume data and the corresponding version number to the cloud service if the check is passed.
3. The data processing method of claim 1, wherein after the data center obtains the incremental data from the database, further comprising:
storing the incremental data stored in the memory of the data center into a local file of the data center;
And combining the original full-volume data stored in the local file of the data center with the incremental data and the corresponding version number to form new full-volume data, and storing the new full-volume data into the local file of the data center.
4. A data processing method according to claim 3, wherein synchronizing the full amount of data and version number to a cloud service comprises:
And the data center synchronizes the full-volume data acquired for the first time with a corresponding version number or the new full-volume data with the corresponding version number to the cloud service.
5. The data processing method of claim 2, wherein synchronizing the full amount of data and version numbers to a cloud service comprises:
And the data center synchronizes the full-volume data acquired for the first time with a corresponding version number or the new full-volume data with the corresponding version number to the cloud service.
6. The data processing method according to claim 1, wherein after synchronizing the incremental data and the version number corresponding to the incremental data to the client by the remote procedure call protocol RPC service, further comprising:
the client acquires the full data and the corresponding version number from the cloud service;
Checking the full data and the full data stored in the local file of the client, and acquiring new full data according to a checking result;
and storing the new full data and the corresponding version number in a local file of the client.
7. The data processing method of claim 6, further comprising:
If the client is restarted, the client tries to load the full data and the corresponding version number stored in the local file of the client after restarting; if loading fails, the client acquires full-volume data and corresponding version numbers from the cloud service, acquires incremental data from the data center through the RPS service, and stores the incremental data in a memory of the client.
8. A data processing apparatus, comprising:
The data acquisition module is configured to acquire full-quantity data or incremental data from a database by a data center, and the data center is connected with the database through a network; wherein the data center obtaining full or incremental data from the database comprises:
judging whether the version number exists in the data center, and obtaining a judging result to be existence or nonexistence;
If the judging result is that the total data does not exist, the total data is firstly obtained from the database, and the total data and the corresponding version number are stored in a memory of a data center;
if the judgment result is that the version number exists, acquiring corresponding incremental data from the database according to the version number, and storing the incremental data in a memory of a data center;
Wherein the version number is a timestamp of acquiring full data or incremental data from the database;
a full-volume synchronization module configured to synchronize the full-volume data and the corresponding version number to a cloud service so that the cloud service stores the full-volume data and the version number;
and the increment synchronization module is configured to synchronize the increment data and the version number corresponding to the increment data to the client through a remote procedure call protocol (RPC) service.
9. An electronic device, comprising:
A processor;
Memory storing method steps for the processor to control any one of claims 1-7.
10. A computer readable medium having stored thereon computer executable instructions, which when executed by a processor implement the method steps of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810783276.XA CN110795495B (en) | 2018-07-17 | 2018-07-17 | Data processing method, device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810783276.XA CN110795495B (en) | 2018-07-17 | 2018-07-17 | Data processing method, device, electronic equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110795495A CN110795495A (en) | 2020-02-14 |
| CN110795495B true CN110795495B (en) | 2024-09-20 |
Family
ID=69424883
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810783276.XA Active CN110795495B (en) | 2018-07-17 | 2018-07-17 | Data processing method, device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110795495B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112487028A (en) * | 2020-11-09 | 2021-03-12 | 杭州电魂网络科技股份有限公司 | Method, system, electronic device and storage medium for server data storage |
| CN112685427B (en) * | 2021-01-25 | 2024-03-26 | 拉卡拉支付股份有限公司 | Data access method, device, electronic equipment and storage medium |
| CN113961538A (en) * | 2021-09-23 | 2022-01-21 | 畅移(上海)信息科技有限公司 | Method and system for supporting automatic data synchronization of multi-source heterogeneous database |
| CN114579888B (en) * | 2022-04-26 | 2022-08-30 | 支付宝(杭州)信息技术有限公司 | Methods, systems, and non-transitory computer-readable media for knowledge-graph data construction |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335170A (en) * | 2014-06-05 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Distributed system and incremental data updating method |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7953946B2 (en) * | 2007-04-16 | 2011-05-31 | Microsoft Corporation | Controlled anticipation in creating a shadow copy |
| CN103678494B (en) * | 2013-11-15 | 2018-09-11 | 北京奇虎科技有限公司 | Client synchronization services the method and device of end data |
| CN103647816A (en) * | 2013-12-03 | 2014-03-19 | 北京奇虎科技有限公司 | Method and device for upgrading application software |
| CN104023085A (en) * | 2014-06-25 | 2014-09-03 | 武汉大学 | Security cloud storage system based on increment synchronization |
| CN105763554A (en) * | 2016-03-28 | 2016-07-13 | 努比亚技术有限公司 | Network detection method, client, and network detection system |
| CN107748790B (en) * | 2017-11-01 | 2021-09-10 | 北京奇艺世纪科技有限公司 | Online service system, data loading method, device and equipment |
| CN107819870A (en) * | 2017-11-21 | 2018-03-20 | 广州视睿电子科技有限公司 | Incremental data pulling method and device, storage medium, terminal equipment and server |
-
2018
- 2018-07-17 CN CN201810783276.XA patent/CN110795495B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335170A (en) * | 2014-06-05 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Distributed system and incremental data updating method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110795495A (en) | 2020-02-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110795495B (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN109522363B (en) | Cloud platform synchronization method, system, equipment and storage medium based on block chain | |
| CN109614439B (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN113626416A (en) | Data verification method and device, computing equipment and storage medium | |
| CN112948409A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113448971A (en) | Data updating method based on distributed system, computing node and storage medium | |
| CN111338834B (en) | Data storage method and device | |
| CN113779082B (en) | Method and device for updating data | |
| CN106909427A (en) | A kind of Oftware updating method, device and software update data dissemination method, system | |
| CN114840562B (en) | Distributed caching method and device for business data, electronic equipment and storage medium | |
| CN111177260A (en) | Database remote copying method and device and electronic equipment | |
| CN113163023B (en) | Domain name management method, device, medium and electronic equipment | |
| CN113783916B (en) | Information synchronization method and device | |
| CN114444719A (en) | Model updating method, apparatus, storage medium and electronic device | |
| CN115687503A (en) | Method, device and equipment for synchronizing data among databases and storage medium | |
| CN112948377A (en) | Data storage method and device, electronic equipment and computer readable medium | |
| CN113760862B (en) | A method, device, equipment and storage medium for incremental data breakpoint resume transmission | |
| CN115167769A (en) | Method, device and system for writing data | |
| CN114253924A (en) | Synchronization method, synchronization equipment and storage medium | |
| CN108196979B (en) | Data backup method and device | |
| CN113761056B (en) | A distributed service processing method, device and system | |
| CN118916430B (en) | Data processing method, device, equipment, storage medium and program product | |
| CN105765908A (en) | Method, client and system for multi-site automatic update | |
| CN112235332A (en) | Read-write switching method and device for cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |