CN112632184A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112632184A CN112632184A CN202011483608.6A CN202011483608A CN112632184A CN 112632184 A CN112632184 A CN 112632184A CN 202011483608 A CN202011483608 A CN 202011483608A CN 112632184 A CN112632184 A CN 112632184A
- Authority
- CN
- China
- Prior art keywords
- data
- message system
- distributed message
- database
- backup database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 37
- 230000008569 process Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 9
- 230000036961 partial effect Effects 0.000 claims description 2
- 230000002411 adverse Effects 0.000 abstract description 7
- 230000000694 effects Effects 0.000 abstract description 6
- 238000004891 communication Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000002829 reductive effect Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Retry When Errors Occur (AREA)
Abstract
The embodiment of the disclosure provides a data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: when the first database fails to write, sending the first data which fails to write to the distributed message system, reading part of data in the first data from the distributed message system in batches, and writing the read data into the backup database; and writing the data stored in the backup database into the first database at preset time intervals, and deleting the successfully written data from the backup database until the data is successfully written. Therefore, the embodiment of the disclosure can reduce the risk of data loss to the greatest extent, and further reduce adverse effects brought to business implementation.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
Database (DB), also called data management system, can be regarded as an electronic file cabinet, i.e. a place for storing electronic files, and a user can add, intercept, update, delete, etc. to the data in the files. MySQL is an associative database management system. Wherein the associative database stores data in different tables instead of putting all data in one large repository, thereby increasing data processing speed and flexibility. Thus, currently, data of applications is typically MySQL-based DB storage.
However, although the availability of the current MySQL-based DB is high, the DB may still be unavailable for a short time or a long time due to network jitter, DB failure, and the like, resulting in a loss of data writing. And the loss of data has a relatively large impact on the traffic of some comparison cores.
As described above, in the prior art, there is a problem that data may be lost when there is a network jitter or a DB failure, which adversely affects the implementation of a service.
Disclosure of Invention
In order to solve the technical problems described in the background art, embodiments of the present disclosure provide a data processing method, an apparatus, an electronic device, and a storage medium, and a technical solution of the present disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, there is provided a data processing method, the method including:
when the first database fails to write, sending the first data which fails to write to a distributed message system;
reading part of data in the first data from the distributed message system in batches, and writing the read data into a backup database;
and writing the data stored in the backup database into the first database at preset time intervals, and deleting the successfully written data from the backup database until the data is successfully written.
Optionally, the method further includes:
in the process of reading part of data in the first data from the distributed message system in batches and writing the read data into the backup database, when the backup database fails to write, resetting a compensation parameter of the distributed message system according to identification information of second data in the distributed message system, wherein the second data is the data which fails to write into the backup database, and the value of the compensation parameter represents a data starting position when the data in the distributed message system is read;
and reading the second data from the distributed message system according to the reset compensation parameters, and rewriting the second data into the backup database.
Optionally, the sending the first data with the write failure to the distributed message system includes:
and when the data volume of the first data is larger than a preset threshold value, sending the first data to a distributed message system.
Optionally, the method further includes:
and when the data volume of the first data is less than or equal to the preset threshold, directly storing the first data into the backup database.
According to a second aspect of embodiments of the present disclosure, there is provided a data processing apparatus, the apparatus comprising:
the sending module is configured to send the first data with write failure to the distributed message system when the first database has write failure;
the first backup module is configured to read partial data in the first data from the distributed message system in batches and write the read data into a backup database;
and the processing module is configured to write the data stored in the backup database into the first database at preset time intervals, and delete the successfully written data from the backup database until the writing is successful.
Optionally, the apparatus further comprises:
the resetting module is configured to reset a compensation parameter of the distributed message system according to identification information of second data in the distributed message system when the backup database fails to write, wherein the second data is the data which fails to write into the backup database, and a value of the compensation parameter indicates a data starting position when the data in the distributed message system is read;
a second backup module configured to read the second data from the distributed message system according to the reset compensation parameter and rewrite the second data into the backup database.
Optionally, when the sending module sends the first data with the write failure to the distributed message system, the sending module is specifically configured to:
and when the data volume of the first data is larger than a preset threshold value, sending the first data to a distributed message system.
Optionally, the apparatus further comprises:
the third backup module is configured to directly store the first data into the backup database when the data volume of the first data is less than or equal to the preset threshold.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the data processing method provided by the present disclosure.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, wherein instructions of the storage medium, when executed by a processor of an electronic device, cause the electronic device to implement a data processing method provided by the present disclosure.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising computer programs/instructions which, when executed by a processor, implement the data processing method provided by the present disclosure.
According to the technical scheme provided by the embodiment of the disclosure, when the first database fails to be written, the first data which fails to be written is sent to the distributed message system, so that part of data in the first data can be read from the distributed message system in batches, the read data is written into the backup database, the data stored in the backup database is written into the first database at preset time intervals, and the data which succeeds in writing is deleted from the backup database until the writing succeeds. Therefore, according to the technical scheme provided by the embodiment of the disclosure, a backup database is pre-established, so that when a write failure occurs in a first database, the data with the write failure can be stored in the backup database one by one in batches, and then the data with the write failure can be repeatedly tried to be written into the first database again, and the data with the write failure is deleted from the backup database until the write failure occurs.
Therefore, according to the technical scheme provided by the embodiment of the disclosure, no matter in case of network jitter or when writing fails due to long-term unavailability of a database failure, the data with writing failures can be timely stored in the backup database, and the data stored in the backup database is repeatedly tried to be written into the database with writing failures until the writing succeeds, so that the risk of data loss can be reduced to the greatest extent, and adverse effects on service implementation are reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
FIG. 1 is a flow diagram illustrating a method of data processing in accordance with an exemplary embodiment;
FIG. 2 is a flow diagram illustrating another data processing method in accordance with an exemplary embodiment;
FIG. 3 is a schematic diagram illustrating a data processing method according to an exemplary embodiment;
FIG. 4 is a block diagram illustrating a data processing apparatus in accordance with an exemplary embodiment;
FIG. 5 is a block diagram illustrating an electronic device in accordance with an exemplary embodiment;
FIG. 6 is a block diagram illustrating another electronic device in accordance with an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The method aims to solve the problem that data loss caused by network jitter and DB faults in the prior art brings adverse effects to service implementation. The embodiment of the disclosure provides a data processing method, a device, a system, an electronic device and a storage medium.
According to a first aspect of embodiments of the present disclosure, there is provided a data processing method, as shown in fig. 1, the data processing method may include the steps of:
step 101: and when the first database fails to write, sending the first data which fails to write to the distributed message system.
The first Database may be a data management system (DB), which may be regarded as an electronic file cabinet, that is, a place where the electronic file is stored, and the user may perform operations such as adding, intercepting, updating, and deleting on the data in the file.
Step 102: and reading part of the first data from the distributed message system in batches, and writing the read data into a backup database.
The distributed message system may be Kafka. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site, and is widely used with horizontal scalability and high throughput.
In addition, the distributed message system allows for smooth consumption, that is, less data is allowed to be read each time, and therefore, in the embodiment of the present disclosure, when a write failure occurs in the first database, the data with the write failure may be sent to the distributed message system, so that data may be read from the distributed message system in batches to write the read data to the backup database. For example, the first data that fails this writing includes 100 pieces of data, the 100 pieces of data may be sent to the distributed message system, so that every 10 pieces are taken as one batch, that is, 10 pieces are read from the distributed message system at a time to be stored in the backup database.
Step 103: and writing the data stored in the backup database into the first database at preset time intervals, and deleting the successfully written data from the backup database until the data is successfully written.
As shown in step 103, in the embodiment of the present disclosure, the data stored in the backup database is periodically written to the first database until the writing is successful. That is, each time the preset time interval is reached, the data stored in the backup database at this time is written into the first database until the writing is successful.
As can be seen from the above, according to the technical scheme provided by the embodiment of the present disclosure, when a write failure occurs in the first database, the first data with the write failure is sent to the distributed message system, so that part of data in the first data can be read from the distributed message system in batches, the read data is written into the backup database, the data stored in the backup database is written into the first database at preset time intervals, and until the write is successful, the data with the write success is deleted from the backup database. Therefore, according to the technical scheme provided by the embodiment of the disclosure, a backup database is pre-established, so that when a write failure occurs in a first database, the data with the write failure can be stored in the backup database one by one in batches, and then the data with the write failure can be repeatedly tried to be written into the first database again, and the data with the write failure is deleted from the backup database until the write failure occurs.
Therefore, according to the technical scheme provided by the embodiment of the disclosure, no matter in case of network jitter or when writing fails due to long-term unavailability of a database failure, the data with writing failures can be timely stored in the backup database, and the data stored in the backup database is repeatedly tried to be written into the database with writing failures until the writing succeeds, so that the risk of data loss can be reduced to the greatest extent, and adverse effects on service implementation are reduced.
It should be noted that, the execution order of the steps 102 and 103 may be that the step 102 is executed first, the step 103 is executed later, or the steps 102 and 103 are executed in parallel.
Wherein, when the steps 102 and 103 are executed in parallel, the process of reading the data from the distributed message system in batches and writing the read data to the backup database and the process of periodically writing the data in the backup database to the first database are executed in parallel. In this case, the data is read from the distributed message system and stored in the backup database on a batch-by-batch basis, and the part of the data successfully written is deleted from the backup database after each successful writing of the data in the backup database into the first database, so that the data can be added and subtracted in the backup database in parallel, and the storage pressure of the backup database can be relieved. Therefore, the backup database can meet the use requirement by setting a smaller storage space.
According to a second aspect of the embodiments of the present disclosure, there is provided a data processing method, as shown in fig. 2, the data processing method may include the steps of:
step 201: and when the first database fails to write, sending the first data which fails to write to the distributed message system.
The first database can be a DB which can be regarded as an electronic file cabinet, namely a place for storing electronic files, and a user can perform operations such as adding, intercepting, updating, deleting and the like on data in the files.
Step 202: and reading part of the first data from the distributed message system in batches, and writing the read data into the backup database.
The distributed message system may be Kafka. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site, and is widely used with horizontal scalability and high throughput.
In addition, the distributed message system allows for smooth consumption, that is, less data is allowed to be read each time, and therefore, in the embodiment of the present disclosure, when a write failure occurs in the first database, the data with the write failure may be sent to the distributed message system, so that data may be read from the distributed message system in batches to write the read data to the backup database. For example, the first data that fails this writing includes 100 pieces of data, the 100 pieces of data may be sent to the distributed message system, so that every 10 pieces are taken as one batch, that is, 10 pieces are read from the distributed message system at a time to be stored in the backup database.
Step 203: and writing the data stored in the backup database into the first database at preset time intervals, and deleting the successfully written data from the backup database until the data is successfully written.
As shown in step 203, in the embodiment of the present disclosure, the data stored in the backup database is periodically written into the first database until the writing is successful. That is, each time the preset time interval is reached, the data stored in the backup database at this time is written into the first database until the writing is successful.
It should be noted that, the execution sequence of the step 202 and the step 203 may be that the step 202 is executed first, the step 203 is executed later, or the step 202 and the step 203 are executed in parallel.
Wherein, when the step 202 and the step 203 are executed in parallel, the process of reading the data from the distributed message system in batches and writing the read data to the backup database is executed in parallel with the process of periodically writing the data in the backup database to the first database. In this case, the data is read from the distributed message system and stored in the backup database on a batch-by-batch basis, and the part of the data successfully written is deleted from the backup database after each successful writing of the data in the backup database into the first database, so that the data can be added and subtracted in the backup database in parallel, and the storage pressure of the backup database can be relieved. Therefore, the backup database can meet the use requirement by setting a smaller storage space.
Step 204: in the process of reading part of data in the first data from the distributed message system in batches and writing the read data into the backup database, when the backup database fails to write, resetting the compensation parameters of the distributed message system according to the identification information of the second data in the distributed message system.
And the second data is data which fails to be written into the backup database, and the value of the compensation parameter represents a data initial position when the data in the distributed message system is read. Therefore, in the embodiment of the present disclosure, the compensation parameter of the distributed message system is reset according to the identification information of the second data in the distributed message system, that is, the value of the compensation parameter of the distributed message system is modified to the position of the second data in the distributed message system, so that when the data in the distributed message system is read following the reset compensation parameter, reading can be started from the second data.
Step 205: and reading the second data from the distributed message system according to the reset compensation parameters, and rewriting the second data into the backup database.
In the process of reading data from the distributed message system and writing the read data into the backup database, network jitter, database failure, and the like may occur, so that data read from the distributed message system at a certain time cannot be successfully written into the backup database. In this case, in the embodiment of the present disclosure, the identification information of the data that fails to be written into the backup database in the distributed message system is recorded, so that the compensation parameter in the distributed message system is reset according to the identification information, so that the data can still be read from the part of data next time, and the read data is rewritten into the backup database.
Specifically, when the distributed message system is Kafka, data is read according to a value of a compensation parameter (offset), and when data read from Kafka at a certain time cannot be successfully written into the backup database, identification information of data in Kafka, which has failed in writing into the backup database at this time, may be recorded, so that the offset is reset according to the identification information, so that the data can be read from data indicated by the identification information directly according to the reset offset at the next time, and the read data is rewritten into the backup database.
Therefore, when the backup database fails to write, the embodiment of the disclosure may record the identification information of the data that fails to write in this time in the distributed message system, so that the data can be read from the part of data in the following process and written into the backup database again, thereby further reducing the risk of data loss.
Optionally, the sending the first data with the write failure to the distributed message system includes:
and when the data volume of the first data is larger than a preset threshold value, sending the first data to a distributed message system.
Therefore, in the embodiment of the disclosure, when the first data amount is greater than the preset threshold, the first data may be divided into a plurality of batches, and the batches are stored in the backup database one by one, so as to reduce the processing procedure when the batch processing is unnecessary, and further increase the data processing speed.
Optionally, the method further includes:
and when the data volume of the first data is less than or equal to the preset threshold, directly storing the first data into the backup database.
Therefore, when the data volume of the first data is smaller than or equal to the preset threshold, the first data can be directly stored in the backup database, so as to further improve the data processing speed.
In summary, the principle of the specific implementation of the data processing method according to the embodiment of the present disclosure is shown in fig. 3, which is specifically described as follows:
in the process of writing data into the original DB by the client, if the original DB has write failure, the data with write failure is sent to Kafka, then the data in Kafka is read in batches, and the read data is written into the pre-established backup DB.
During the process of reading the data in the Kafka in batches and writing the read data into the backup DB, a round-robin task is started to synchronize the data in the backup DB into the original DB to be written before at preset time intervals, if the writing is successful, the data which is written successfully is deleted from the backup DB, and if the writing is failed, the failed data is retained, and the writing is tried again next time until the writing is successful.
In addition, in the process of writing the backup DB, if the backup DB may also have write failure, the offset of Kakfa may be reset, so that reading can be started from the data that failed this time next time.
In addition, the backup DB is only used when the original DB data fails to be written, and there is less possibility that a large amount of data fails to be written, and thus, the backup DB can be provided with a smaller space. While Kafka allows for gentle consumption, i.e., allows for reading less data multiple times, Kafka may be introduced in order to further address the pressure of a large number of data write failures on the backup DB. That is, in the embodiment of the present disclosure, the data peak of the transient write to the backup DB can be alleviated by Kafka.
In summary, the embodiment of the present disclosure introduces the message queue and the backup DB mechanism, so as to reduce the risk of data loss to the greatest extent, and further reduce adverse effects on service implementation.
According to a third aspect of the embodiments of the present disclosure, there is provided a data processing apparatus, as shown in fig. 4, the data processing apparatus 400 may include:
a sending module 401, configured to send, when a write failure occurs in the first database, the first data with the write failure to the distributed message system;
a first backup module 402 configured to read a part of the first data from the distributed message system in batches and write the read data into a backup database;
the processing module 403 is configured to write the data stored in the backup database into the first database at preset time intervals, and delete the successfully written data from the backup database until the writing is successful.
Optionally, the apparatus further comprises:
a resetting module 404, configured to, in the process of reading part of the first data from the distributed message system in batches and writing the read data into the backup database, when a write failure occurs in the backup database, reset a compensation parameter of the distributed message system according to identification information of second data in the distributed message system, where the second data is the data that has failed to be written into the backup database, and a value of the compensation parameter indicates a data start position when the data in the distributed message system is read;
a second backup module 405 configured to read the second data from the distributed message system according to the reset compensation parameter and rewrite the second data into the backup database.
Optionally, when the sending module 401 sends the first data with the write failure to the distributed message system, the sending module is specifically configured to:
and when the data volume of the first data is larger than a preset threshold value, sending the first data to a distributed message system.
Optionally, the apparatus further comprises:
a third backup module 406, configured to directly store the first data in the backup database when the data amount of the first data is less than or equal to the preset threshold.
The data processing apparatus provided by the embodiment of the disclosure sends the first data with failed writing to the distributed message system when the first database has failed writing, so that part of the first data can be read from the distributed message system in batches, the read data is written into the backup database, the data stored in the backup database is written into the first database at preset time intervals, and the data with successful writing is deleted from the backup database until the writing is successful. Therefore, the data processing apparatus provided in the embodiment of the present disclosure establishes a backup database in advance, so that when a write failure occurs in a first database, data with the write failure can be stored in the backup database one by one in batches, and then the data with the write failure can be tried to be written into the first database again for many times, until the data with the write failure is deleted from the backup database when the write failure occurs.
Therefore, the data processing apparatus provided in the embodiment of the present disclosure can save the data that fails to be written to the backup database in time no matter when the network is jittered or when the database fails to be available for a long time, and try to write the data stored in the backup database to the database that fails to be written again for many times until the data is successfully written, so that the risk of data loss can be reduced to the greatest extent, and adverse effects on service implementation are reduced.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
According to a fourth aspect of embodiments of the present disclosure, an electronic device is provided. Referring to fig. 5, the electronic device includes:
a processor 510;
a memory 520 for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the data processing method described above.
According to a fifth aspect of the embodiments of the present disclosure, there is also provided an electronic device. As shown in fig. 6, the electronic device 600 may be a mobile phone, a computer, a digital broadcast electronic device, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.
Referring to fig. 6, electronic device 600 may include one or more of the following components: processing component 602, memory 604, power component 606, multimedia component 608, audio component 610, input/output (I/O) interface 612, sensor component 614, and communication component 616.
The processing component 602 generally controls overall operation of the electronic device 600, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 602 can include one or more modules that facilitate interaction between the processing component 602 and other components. For example, the processing component 602 can include a multimedia module to facilitate interaction between the multimedia component 608 and the processing component 602.
The memory 604 is configured to store various types of data to support operations at the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 604 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 600 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 610 is configured to output and/or input audio signals. For example, the audio component 610 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 600 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 604 or transmitted via the communication component 616. In some embodiments, audio component 610 further includes a speaker for outputting audio signals.
The I/O interface 612 provides an interface between the processing component 602 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor component 614 includes one or more sensors for providing status assessment of various aspects of the electronic device 600. For example, the sensor component 614 may detect an open/closed state of the electronic device 600, the relative positioning of components, such as a display and keypad of the electronic device 600, the sensor component 614 may also detect a change in the position of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, orientation or acceleration/deceleration of the electronic device 600, and a change in the temperature of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 616 is configured to facilitate communications between the electronic device 600 and other devices in a wired or wireless manner. The electronic device 600 may access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 8G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 600 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the data processing methods described above.
In an exemplary embodiment, a non-transitory computer readable storage medium comprising instructions, such as the memory 604 comprising instructions, executable by the processor 620 of the electronic device 600 to perform the above-described method is also provided. Alternatively, for example, the storage medium may be a non-transitory computer-readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In yet another aspect of the present disclosure, the present disclosure also provides a computer-readable storage medium, where instructions of the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method described above.
According to yet another aspect of the embodiments of the present disclosure, there is provided a computer program product containing instructions, comprising computer programs/instructions, which when executed by a processor, implement the data processing method described above.
The data processing schemes provided herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The structure required to construct a system incorporating aspects of the present invention will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in an information extraction scheme according to embodiments of the present disclosure. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (10)
1. A method of data processing, the method comprising:
when the first database fails to write, sending the first data which fails to write to a distributed message system;
reading part of data in the first data from the distributed message system in batches, and writing the read data into a backup database;
and writing the data stored in the backup database into the first database at preset time intervals, and deleting the successfully written data from the backup database until the data is successfully written.
2. The data processing method of claim 1, wherein the method further comprises:
in the process of reading part of data in the first data from the distributed message system in batches and writing the read data into the backup database, when the backup database fails to write, resetting a compensation parameter of the distributed message system according to identification information of second data in the distributed message system, wherein the second data is the data which fails to write into the backup database, and the value of the compensation parameter represents a data starting position when the data in the distributed message system is read;
and reading the second data from the distributed message system according to the reset compensation parameters, and rewriting the second data into the backup database.
3. The data processing method of claim 1, wherein sending the first data with the write failure to a distributed message system comprises:
and when the data volume of the first data is larger than a preset threshold value, sending the first data to a distributed message system.
4. The data processing method of claim 3, wherein the method further comprises:
and when the data volume of the first data is less than or equal to the preset threshold, directly storing the first data into the backup database.
5. A data processing apparatus, characterized in that the apparatus comprises:
the sending module is configured to send the first data with write failure to the distributed message system when the first database has write failure;
the first backup module is configured to read partial data in the first data from the distributed message system in batches and write the read data into a backup database;
and the processing module is configured to write the data stored in the backup database into the first database at preset time intervals, and delete the successfully written data from the backup database until the writing is successful.
6. The data processing apparatus of claim 5, wherein the apparatus further comprises:
the resetting module is configured to reset a compensation parameter of the distributed message system according to identification information of second data in the distributed message system when the backup database fails to write, wherein the second data is the data which fails to write into the backup database, and a value of the compensation parameter indicates a data starting position when the data in the distributed message system is read;
a second backup module configured to read the second data from the distributed message system according to the reset compensation parameter and rewrite the second data into the backup database.
7. The data processing apparatus according to claim 5, wherein the sending module, when sending the first data with the write failure to the distributed message system, is specifically configured to:
and when the data volume of the first data is larger than a preset threshold value, sending the first data to a distributed message system.
8. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the data processing method of any one of claims 1 to 4.
9. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to implement the data processing method of any one of claims 1 to 4.
10. A computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions, when executed by a processor, implement the data processing method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011483608.6A CN112632184A (en) | 2020-12-15 | 2020-12-15 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011483608.6A CN112632184A (en) | 2020-12-15 | 2020-12-15 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112632184A true CN112632184A (en) | 2021-04-09 |
Family
ID=75314181
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011483608.6A Pending CN112632184A (en) | 2020-12-15 | 2020-12-15 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112632184A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114297175A (en) * | 2021-11-01 | 2022-04-08 | 浙江百应科技有限公司 | A data storage method, device and electronic device based on multiple databases |
| CN115098474A (en) * | 2022-06-21 | 2022-09-23 | 马上消费金融股份有限公司 | Data processing method, device and storage medium |
| CN115617763A (en) * | 2022-09-23 | 2023-01-17 | 中电金信软件有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103313240A (en) * | 2012-03-15 | 2013-09-18 | 中国移动通信集团公司 | Secure access method, device and system |
| CN104899506A (en) * | 2015-05-08 | 2015-09-09 | 深圳市雪球科技有限公司 | Security system implementation method based on virtual security element in trusted execution environment |
| CN105701219A (en) * | 2016-01-14 | 2016-06-22 | 北京邮电大学 | Distributed cache implementation method |
| CN106228072A (en) * | 2016-07-21 | 2016-12-14 | 恒宝股份有限公司 | A kind of general TA payment platform and method of payment |
| CN106407398A (en) * | 2016-09-21 | 2017-02-15 | 福建中金在线信息科技有限公司 | A data storage method and system |
| US20190227888A1 (en) * | 2018-01-23 | 2019-07-25 | International Business Machines Corporation | Handling node failure in multi-node data storage systems |
| CN110134525A (en) * | 2019-03-25 | 2019-08-16 | 杭州比智科技有限公司 | A kind of message compensation method and device |
| CN112035292A (en) * | 2020-07-31 | 2020-12-04 | 北京金山云网络技术有限公司 | A method, device, device and storage medium for processing abnormal data writing |
-
2020
- 2020-12-15 CN CN202011483608.6A patent/CN112632184A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103313240A (en) * | 2012-03-15 | 2013-09-18 | 中国移动通信集团公司 | Secure access method, device and system |
| CN104899506A (en) * | 2015-05-08 | 2015-09-09 | 深圳市雪球科技有限公司 | Security system implementation method based on virtual security element in trusted execution environment |
| CN105701219A (en) * | 2016-01-14 | 2016-06-22 | 北京邮电大学 | Distributed cache implementation method |
| CN106228072A (en) * | 2016-07-21 | 2016-12-14 | 恒宝股份有限公司 | A kind of general TA payment platform and method of payment |
| CN106407398A (en) * | 2016-09-21 | 2017-02-15 | 福建中金在线信息科技有限公司 | A data storage method and system |
| US20190227888A1 (en) * | 2018-01-23 | 2019-07-25 | International Business Machines Corporation | Handling node failure in multi-node data storage systems |
| CN110134525A (en) * | 2019-03-25 | 2019-08-16 | 杭州比智科技有限公司 | A kind of message compensation method and device |
| CN112035292A (en) * | 2020-07-31 | 2020-12-04 | 北京金山云网络技术有限公司 | A method, device, device and storage medium for processing abnormal data writing |
Non-Patent Citations (1)
| Title |
|---|
| 王亚玲等: "数据库系统应用分片中间件", 《计算机系统应用》, vol. 24, no. 10, pages 1 - 2 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114297175A (en) * | 2021-11-01 | 2022-04-08 | 浙江百应科技有限公司 | A data storage method, device and electronic device based on multiple databases |
| CN114297175B (en) * | 2021-11-01 | 2025-01-03 | 浙江百应科技有限公司 | A data storage method, device and electronic device based on multiple databases |
| CN115098474A (en) * | 2022-06-21 | 2022-09-23 | 马上消费金融股份有限公司 | Data processing method, device and storage medium |
| CN115617763A (en) * | 2022-09-23 | 2023-01-17 | 中电金信软件有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3188066B1 (en) | A method and an apparatus for managing an application | |
| CN105955766B (en) | Using preloading method and device | |
| JP6348989B2 (en) | Firmware restoration method, apparatus, terminal, program, and recording medium | |
| CN104951335B (en) | The processing method and processing device of application program installation kit | |
| CN106598488B (en) | Distributed data read method and device | |
| CN106534562B (en) | File deletion method and device | |
| CN106919629B (en) | Method and device for realizing information screening in group chat | |
| CN112632184A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN105930213B (en) | Using operation method and device | |
| CN112506553B (en) | Upgrading method and device for data surface container of service grid and electronic equipment | |
| CN113703937A (en) | Animation switching method and device and storage medium | |
| EP3236355A1 (en) | Method and apparatus for managing task of instant messaging application | |
| CN113596175A (en) | Scene synchronization method, terminal, server and system | |
| CN109634762B (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN108153488B (en) | Data self-adding method and device | |
| CN107391128B (en) | Method and device for monitoring virtual file object model vdom | |
| CN112506700B (en) | Conflict processing method, device, electronic equipment and storage medium | |
| CN106919679B (en) | Log replay method, device and terminal applied to distributed file system | |
| CN105824513A (en) | Message processing method and apparatus | |
| CN113467854B (en) | Application program starting method and device, electronic equipment and storage medium | |
| CN110995767B (en) | Request processing method and device | |
| CN108762944B (en) | Processing method, device, equipment and medium of business system | |
| CN115357277A (en) | Hot update method, device, electronic apparatus, storage medium and program product | |
| CN114491332A (en) | Page loading method and device, electronic equipment and storage medium | |
| CN112182027A (en) | Information query method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |