[go: up one dir, main page]

CN102110121A - Method and system for processing data - Google Patents

Method and system for processing data Download PDF

Info

Publication number
CN102110121A
CN102110121A CN2009102601736A CN200910260173A CN102110121A CN 102110121 A CN102110121 A CN 102110121A CN 2009102601736 A CN2009102601736 A CN 2009102601736A CN 200910260173 A CN200910260173 A CN 200910260173A CN 102110121 A CN102110121 A CN 102110121A
Authority
CN
China
Prior art keywords
data recording
data
database
version information
request
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN2009102601736A
Other languages
Chinese (zh)
Other versions
CN102110121B (en
Inventor
覃健祥
常国斌
张宋景
李翀
朱明君
全鑫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN200910260173.6A priority Critical patent/CN102110121B/en
Publication of CN102110121A publication Critical patent/CN102110121A/en
Priority to HK11108936.7A priority patent/HK1155236A1/en
Application granted granted Critical
Publication of CN102110121B publication Critical patent/CN102110121B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method and system for processing data. The method comprises the following steps: when a request for updating a data record in a database is obtained, newly plugging in a data record with the same object identity into the database according to the data record requesting to update; adding edition information in the data record which is newly plugged in, wherein the edition information is used for demonstrating the time sequence of the plug-in operation of the data record with the same object identity; and when the database is copied to another database, through utilizing a plurality of threads, the database copy operate is carried out. By utilizing the method and system for processing the data, the efficiency of copying the database is improved on the premise of guaranteeing the data consistency to a certain degree.

Description

A kind of data processing method and system thereof
Technical field
The application relates to the data processing technique in the communications field, relates in particular to a kind of data processing method and system thereof.
Background technology
(as MySQL, PostgreSQL Oracle) has leader follower replication (Replication) function to the main flow relational database.Suppose that A is a master server, B is from server, and the operation system based on database has been disposed " database server A " to " database server B " copy function, then increase (Insert), delete (Delete) when data go up to take place database server A, when changing (Update) operation, these operations can be synchronized to database server B, and this process is called duplicates (Replication).
Usually, people can often use increasing (Insert), delete (Delete), change (Update), look into (Select) four kinds of operations the data recording in the database when handling based on database professional.Dispose after the database replication, these operations relevant with the data recording write operation (comprise and increase, delete, change) will be synchronized to from database server by main database server in order.
But this synchronous operation must be orderly.Suppose that in recording the operation system of user's signature main database server is DB1, is DB2 from database server, and such sequence of operations has taken place on the DB1:
Scene one: ID is that 12065 user's oneself signature is set to " meddlesome dog ";
Corresponding SQL statement 1 is: update user_sign set nickname=" meddlesome dog " whereuserid=12065;
Corresponding database is operating as: the pet name that is 12065 user to ID is updated to " meddlesome dog ";
Scene two: ID is that 12065 user's oneself signature is set to " Fruitds In Summer ";
Corresponding SQL statement 2 is: update user_sign set nickname=" Fruitds In Summer " whereuserid=12065;
Corresponding database is operating as: the pet name that is 12065 user to ID is updated to " Fruitds In Summer ";
Scene three: ID is that 12065 user's oneself signature is set to " approximately in the winter time ";
Corresponding SQL statement 3 is: update user_sign set nickname=" approximately in the winter time " whereuserid=12065;
Corresponding database is operating as: the pet name that is 12065 user to ID is updated to " approximately in the winter time ".
When DB1 when DB2 carries out replicate run, being used for more SQL statement 1, SQL statement 2, the SQL statement 3 of new data records must carry out successively, the data on DB1, the DB2 just can be consistent like this.If the execution sequence on the DB2 is: SQL statement 1, SQL statement 3, SQL statement 2, then the pet name of DB1 last 12065 is " approximately in the winter time ", and the pet name of DB2 last 12065 is " Fruitds In Summer ", and the data in the master and slave database server are just inconsistent.
Because more operating in the reproduction process of new data records must be carried out the consistance that could guarantee the data in the master and slave database server according to the order of sequence, in order to guarantee the consistance of the data in the master and slave database server, replicate run between the master and slave database server is normally single-threaded to be carried out, and that it carries out efficient is lower.
The application content
The embodiment of the present application provides a kind of database operation method and system thereof, in order to solve the low problem of database replication operating efficiency in the available data treatment technology.
The technical scheme that the embodiment of the present application provides comprises:
A kind of data processing method may further comprise the steps:
After getting access to the request that the data recording in the database is upgraded, be recorded in the new identical data recording of object sign of inserting in the described database according to the request data updated, and in the new data recording of inserting, adding version information, described version information is used to show the time sequencing of the insertion operation of the data recording with same object sign;
When another database is arrived in described database replication, utilize a plurality of thread parallels to carry out the database replicate run.
A kind of data handling system comprises:
Acquisition module is used for obtaining the request that the data recording of database is upgraded;
Update processing module, be used for after getting access to the request that the data record is upgraded, be recorded in the new identical data recording of object sign of inserting in the described database according to the request data updated, and in the new data recording of inserting, adding version information, described version information is used to show the time sequencing of the insertion operation of the data recording with same object sign;
The replication processes module is used for described database replication utilizing a plurality of thread parallels to carry out the database replicate run when another database.
The foregoing description of the application, by when getting access to the request that the data recording in the database is upgraded, the new identical data recording of object sign of inserting in database, and in the new data recording of inserting, add version information, thereby being replaced with data, the Data Update operation inserts operation; When database is duplicated, utilize a plurality of thread parallels to carry out the database replicate run.On the one hand, carry out the efficient of database replicate run can raising database replication operation with multi-threaded parallel; On the other hand, because more the operation of new data records replaces with the operation of inserting data recording, like this, even cause not having time order and function to duplicate owing to using multithreading to carry out replicate run, also can not destroy master and slave data of database consistance in strict accordance with data recording insertion operation.
Description of drawings
Fig. 1 is one of structural representation of the embodiment of the present application data handling system;
Fig. 2 be the embodiment of the present application data handling system structural representation two;
Fig. 3 be the embodiment of the present application data handling system structural representation three.
Embodiment
Problem at the prior art existence, the embodiment of the present application is inserted operation by the Data Update operation is replaced with data, and use multi-threaded parallel to carry out the database replicate run, thereby guaranteeing to improve database replication efficient under the conforming prerequisite of master and slave database data.
The data handling system of the embodiment of the present application can be operated the data recording in the database, comprise: insert the operation of (Insert), deletion (Delete), inquiry (Select) data recording, the data of master data base are duplicated (Replication) to from the operation of database etc.Wherein, include a field that is used for recording version information in master data base, the database table from database, wherein record the version information of data recording.Data recording in the database table includes an identification field usually, is used for unique identity of description object, as user ID, other fields then are used to describe the attribute of this object, as address name, age etc., can also comprise the attribute of data of description record, as the version information of data record.
The various data processing request that the human-computer interaction interface that the data handling system of the embodiment of the present application can get access to the user to be provided by this data handling system is imported, as the request of: establishment, renewal, deletion or data query record, or carry out the request of database replication;
When system gets access to the request of creating data recording, generate the statement that inserts data recording, and in database, insert a data recording by carrying out this statement, be added with version information in the new data recording of inserting;
When system gets access to the request of new data records more, generate the statement that inserts data recording, and pass through to carry out this statement and in database, insert a data recording, be added with version information in the new data recording of inserting, and have and the identical object identity of request data updated record;
When system gets access to the request of data query record, according to the statement of the sign generated query data recording of the object of being inquired about, and by carrying out the content that this statement returns the data recording of up-to-date insertion in all data recording with this object identity;
When system according to database replication strategy of disposing (as replicative cycle) or the duplicate requests that gets access to, when carrying out the database replication operation, in reproduction process, can use multi-threaded parallel to carry out the database replicate run, further, can adopt the distributed multi-threaded parallel of realizing of multiserver to carry out the database replicate run.
The data handling system of the embodiment of the present application also can comprise the data recording delete function, when system gets access to the request of delete data record, generate the statement that inserts data recording, and in database, insert a data recording of ask to delete with institute and have the data recording that same object identifies by carrying out this statement, and in this data recording, add the version information that shows its insertion time sequencing, and this data recording is labeled as deletion.Accordingly, when system gets access to the data recording query requests, at first find the data recording of up-to-date insertion according to the sign of object of ask inquiry, and then judge whether this data recording is marked as deletion, if then return the Query Result that does not find, otherwise the content of data recording that will this up-to-date insertion is returned as Query Result.
Above-mentioned version information can be generated when inserting data recording by this data handling system.Described version information is used for showing that the insertion of the data recording with same object sign operates in temporal sequencing.Because the data handling system of the embodiment of the present application is converted to the renewal operation of data recording the insertion operation of data recording, like this, data recording at the same target sign, if follow-up relative users repeatedly asks the content of its data recording is upgraded, then can in database table, insert the new data recording of respective numbers.For when the data base querying, provide Query Result (promptly only returning the data recording of wherein up-to-date insertion as Query Result) accurately to the user at many data recording of same target sign, therefore need identify the version information of each data recording, with the time sequencing that shows that data recording is inserted.Described version information can be the sequence number that increases progressively successively, also can be timestamp, perhaps the character combination that other can the express time sequencing.
Because the data handling system of the embodiment of the present application is operated the renewal of data recording and delete the insertion operation that replaces with data recording, and each data recording all includes version information, therefore, when coping database, the time sequencing that does not need strictness to follow each data manipulation execution is duplicated, therefore can use that multi-thread concurrent mode and multiserver are distributed to carry out the database replication operation, thereby improve the efficient of database replication operation when guaranteeing data consistency to a certain extent.
Embodiment to the application is described in detail below in conjunction with accompanying drawing.
Referring to Fig. 1, be the structural representation of the embodiment of the present application data handling system, this system can realize by the software programming mode.This database operating system comprises: acquisition module 101, update processing module 103, replication processes module 106 can comprise in addition: insert processing module 102, deletion processing module 104, query processing module 105.
The major function of each functional module in this data handling system comprises:
Acquisition module 101: the various data processing request that the human-computer interaction interface that can get access to the user provides by this data handling system is imported, as: the request of establishment, renewal, deletion or data query record, or carry out the request of database replication, and can trigger corresponding data processing function module according to the data processing request that receives and carry out data processing operation;
Insert processing module 102: the request of the establishment data recording that can get access to according to acquisition module 101, generate the statement that inserts data recording, and in database, inserts a data recording by carrying out this statement, be added with version information in the data recording of new insertion;
Update processing module 103: the request of the more new data records that can get access to according to acquisition module 101, generate the statement that inserts data recording, and pass through to carry out this statement and in database, insert a data recording, be added with version information in the new data recording of inserting, and have and the identical object identity of request data updated record;
Deletion processing module 104: the request of the delete data record that can get access to according to acquisition module 101 generates the statement that inserts data recording, and inserts a new data recording by carrying out this statement, and this data recording is labeled as deletion;
Query processing module 105: the request of the data query record that can get access to according to acquisition module 101, the statement of generated query data recording, and return Query Result by carrying out this statement;
Replication processes module 106: the request of the coping database that can get access to according to acquisition module 101, perhaps according to the database replication strategy of disposing (as replicative cycle), carry out the database replication operation, and in reproduction process, can adopt multiserver distributed, and multi-threaded parallel is carried out the database replicate run.
The database table structure of putting down in writing the data handling system (also can be described as operation system) of user's signature with being used to shown in the table 1 is an example below, and the various flow chart of data processing of the embodiment of the present application are described in detail.This operation system that is used to put down in writing user's signature has aforesaid acquisition module 101, inserts processing module 102, update processing module 103, query processing module 105 and replication processes module 106, can also comprise deletion processing module 104.
Table 1: the structure that is used to put down in writing the database table user_sign of user's signature
userid ?signature ?revision
In the table 1, userid, signature, revision are respectively the field names of this database table, and the implication of each field is as follows:
Userid: user ID (ID);
Signature: signature contents (or being called the pet name);
Revision: the version information of sequence number form (to call version number in the following text).
When user's request registration, can input user name, password and user's signature log-on messages such as (being " meddlesome dog ") and submit log-in command to by the human-computer interaction interface that this system provides as user's signature, acquisition module 101 is submitted to insertion processing module 102 after receiving these log-on messages and log-in command, inserting processing module 102 is this user's distributing user ID (as 12065), generates data recording then and inserts statement:
INSERT?INTO?user_sign
(userid, signature, revision) VALUES (12065, " meddlesome dog ", 1)
After carrying out this statement, data recording of new insertion in the user_sign database table, the database table after the insertion data recording is as shown in table 2, wherein, owing to be the initial creation data recording, so the version number that system distributes for this data recording is 1.
Table 2: the database table user_sign that includes the initial user signature of userid=12065
userid signature revision
12065 Meddlesome dog 1
When this user will upgrade its user's signature, can login this system, the user's signature " Fruitds In Summer " after the human-computer interaction interface input that provides by this system is upgraded is selected update command then and is submitted to.The acquisition module 101 of this system receive update command and upgrade after user's signature after it is submitted to update processing module 103, update processing module 103 can get access to the user ID of this login user, and the user's signature generation data recording according to the user ID that gets access to and after upgrading is inserted statement then:
INSERT?INTO?user_sign
(userid, signature, revision) VALUES (12065, " Fruitds In Summer ", 2)
After carrying out this statement, data recording of new insertion in the user_sign database table, database table after the insertion data recording is as shown in table 3, wherein, update processing module 103 is before generating data recording insertion statement, can inquire user ID and be all data recording of 12065 and inquire wherein maximum version number, thereby then this version number be increased progressively the release train number " 2 " that obtains the data recording that to insert.Certainly, if represent version information with timestamp, it is just passable as version information then directly to write the current time in the statement that inserts data recording.
The user of table 3:userid=12065 upgrades the database table user_sign behind the user's signature for the first time
userid signature revision
12065 Meddlesome dog 1
12065 Fruitds In Summer 2
When this user will upgrade its user's signature once more, can login this system, the user's signature " approximately in the winter time " after the human-computer interaction interface input that provides by this system is upgraded is selected update command then and is submitted to.The acquisition module 101 of this system receive update command and upgrade after user's signature after it is submitted to update processing module 103, update processing module 103 generates following data recording and inserts statement:
INSERT?INTO?user_sign
(userid, signature, revision) VALUES (12065, " approximately in the winter time ", 3)
After carrying out this statement, data recording of new insertion in the user_sign database table, the database table after the insertion data recording is as shown in table 4:
The user of table 4:userid=12065 upgrades the database table user_sign behind the user's signature for the second time
userid signature revision
12065 Meddlesome dog 1
12065 Fruitds In Summer 2
12065 Approximately in the winter time 3
The query manipulation of user's signature is carried out by the system manager usually.When the system manager is 12065 by the bookkeeping interface input userid that this system provides, and after selecting querying command to submit to, the acquisition module 101 of this system is submitted to query processing module 105 after receiving this order and user ID, and query processing module 105 generates the query statement of following data recording:
SELECT?signature?FROM?user_sign
WHERE?userid=12065
ORDER?BY?revision?DESC?LIMIT?1
In this statement implementation, user's the signature of inquiry userid=12065 in user_sign database table as shown in table 4, and according to the rank order of version number's descending, and only return article one data recording after the ordering.Because " approximately in the winter time " pairing version number maximum in all data recording of userid=12065, then Query Result is " approximately in the winter time ".
If this data handling system also provides the function of delete data record, then also need deletion action is converted to the operation of inserting data recording, promptly insert a new record and this record mark is deletion.Usually undertaken by the system manager from the operation of user_sign database table delete data record.If current user_sign is as shown in table 4, when user's signature " approximately in the winter time " is imported or selected at the bookkeeping interface that the system manager provides by this system, and after selecting delete command to submit to, the acquisition module 101 of this system is submitted to deletion processing module 104 after receiving this order and user ID, and the user ID that deletion processing module 104 gets access to " approximately in the winter time " is that 12065 backs generate following data recording insertion statement:
INSERT?INTO?user_sign
(userid, signature, deleted, revision) VALUES (12065, " approximately in the winter time ", 1,4)
Wherein, the 4th, be the version number of new data recording distribution of inserting, deleted is the delete flag field, and its value is that the data recording of 1 expression this time insertion is the insertion record that generates according to removal request, shows that user ID is that 12065 user's signature data recording is deleted.
After this statement is carried out, data recording of new insertion in the user_sign database table, the database table after the insertion data recording is as shown in table 5:
Table 5: carry out the database table user_sign after user to userid=12065 deletes the user's signature record
userid signature revision deleted
12065 Meddlesome dog 1
12065 Fruitds In Summer ?2
12065 Approximately in the winter time ?3
12065 Approximately in the winter time ?4 1
At the data handling system that delete function is provided, when data query writes down, at first find the data recording of up-to-date insertion according to querying condition, judge then whether this data recording is labeled as deletion, if unmarked, otherwise return the Query Result that expression does not inquire the data of being asked for deletion then according to this data recording generated query result.For example, for the database table user_sign shown in the table 5, when the system manager imports user's signature " approximately in the winter time " by the bookkeeping interface that this system provides, and after selecting querying command to submit to, the acquisition module 101 of this system is submitted to query processing module 105 after receiving this order and user ID, query processing module 105 inquires the userid=12065 of " approximately in the winter time " by the query statement that generates, and inquire the data recording of up-to-date insertion in the data recording of userid=12065, be that version number is 4 data recording, because the deleted field value of this record is 1, then prompting " not finding ".
When this system deployment had master and slave database replication strategy, this system can carry out database replication by replication processes module 106 according to the replication strategy of being disposed (as performance period of replicate run etc.).In reproduction process, can adopt the replicate run of the distributed execution data of multi-threaded parallel and multiserver.As above in the example, in the database replication process, version number be 2 and version number be that two data recording of 3 can be synchronized to from database server with any order, 2 corresponding data records insert operation note and also can be synchronized to from database server with any order, and the data in the final master and slave database server all are consistent.
Have only field of signature in the above-mentioned user_sign database table except user ID and version information field, in actual applications, database table may comprise more field.Table 6 has provided and has included the more example of the user_sign database table of multiword section, has wherein increased age (age) field than above-mentioned user_sign database table:
Table 6: contain the more user_sign table of multiword section
userid signature age revision
12065 Meddlesome dog 18 1
12065 Fruitds In Summer 18 2
At the database table shown in the table 6, data recording when its user registers is inserted operation, and the process of the operation of deletion user record, database replication operation is identical with the corresponding flow process of foregoing description with principle.When user's request was upgraded the data record, its process was:
When this user will upgrade its user's signature, can login this system, the user's signature " approximately in the winter time " after the human-computer interaction interface input that provides by this system is upgraded is selected update command then and is submitted to.The acquisition module 101 of this system receive update command and upgrade after user's signature after it is submitted to update processing module 103, the user ID that update processing module 103 can get access to this login user is 12065, from showing, inquires about user_sign the data recording of userid=12065 then according to the user ID that gets access to, then with the respective field value of the age field value " 18 " in the data recording of user's signature " approximately in the winter time " after upgrading and the userid=12065 that inquires, and generate following data recording insertion statement as new data recording:
INSERT?INTO?user_sign
(userid, signature, age, revision) VALUES (12065, " approximately in the winter time ", 18,3)
After carrying out this statement, data recording of new insertion in the user_sign database table, the database table after the insertion data recording is as shown in table 7:
Table 7: the user_sign table behind the renewal user's signature
userid signature age revision
12065 Meddlesome dog 18 1
12065 Fruitds In Summer 18 2
12065 Approximately in the winter time 18 3
As can be seen, when generating the INSERT statement, for each field, if in the data by the update request submission this field value is arranged, then the value of this field is as the criterion with the field value of submitting in the update request, if no, then be as the criterion with the respective field value in the data recording of the latest edition that inquires.
Because the data handling system of the embodiment of the present application more operation of new data records replaces with the operation of inserting data recording, therefore in database table, can preserve redundant data recording at the same target sign, as for the user_sign database table shown in the above-mentioned table 7, at the data recording of userid=12065, can preserve the user's signature (signature) of a plurality of versions in the database table.For capacity and the performance that guarantees database, can only preserve the data recording of some in the embodiment of the present application at the data recording of same target sign, and the data recording that other are redundant deletion.The implementation of its deletion redundant data record has multiple, such as:
Mode one: regularly clear up data
Can work out a liquidation procedures in advance, regularly carry out this liquidation procedures then.This liquidation procedures can the ergodic data storehouse table, wherein redundant data is write down all or part of deletion.As, database table for above-mentioned user_sign by name, when carrying out liquidation procedures at every turn, liquidation procedures travels through the data recording in this table, find that userid is that 12065 data recording has many, therefore can be according to setting in advance, only keep the data recording of version number's maximum wherein and delete the data recording of other version numbers; Perhaps, be under 2 the situation in data recording quantity threshold value, according to beginning to keep 2 data recording wherein and the data recording of deleting other from the version number of maximum after version number's rank order from big to small.
Mode two: cleaning data when at every turn inserting data recording
Each insert data recording before or after, can ergodic data storehouse table, wherein redundant data is write down all or part of deletion.As, database table for above-mentioned user_sign by name, ask to upgrade user's signature and after inserting data recording the user at every turn, travel through the data recording in this table, find that userid is that 12065 data recording has many, therefore can be according to setting in advance, only keep the data recording of version number's maximum wherein and delete the data recording of other version numbers; Perhaps, be under 2 the situation in data recording quantity threshold value, according to beginning to keep 2 data recording wherein and the data recording of deleting other from the version number of maximum after version number's rank order from big to small.
Above-mentioned cleaning operation can be realized by corresponding functional modules, such as increase by the first cleaning processing module 107 in the data handling system of the embodiment of the present application, as shown in Figure 2, perhaps increases by the second cleaning processing module 108, as shown in Figure 3.Wherein, first clears up the function that processing module 107 can realize regularly clearing up data, and the second cleaning processing module 108 can be cleared up data at every turn when inserting data recording.
Above-mentioned dual mode is cleared up when inserting data recording at every turn and can be improved the degree of coupling of inserting operation by contrast, reduces its response speed, therefore, in actual applications, can preferably adopt and regularly clear up data, like this, can obtain better reliability and shorter response time.
The embodiment of the present application is particularly useful for the data redundancy requirement not high, but the application of the professional aspect of the internet content that concurrency is had relatively high expectations.Aspect type of database, the embodiment of the present application is particularly useful for adopting the database of key-value (being key-value pair) storage, puts down in writing the database of the operation system of user's signature as above-mentioned being used to.
The database operating system of the embodiment of the present application also can provide following function except that above-mentioned functions: when the data on the main database server are damaged, can be recovered by the data from database server; When main database server is delayed machine, can take over the work of main database server from database server, the efficient of the replicate run between the master and slave database server is high more, and the real-time property from database server is just good more; From database server can with main database server load sharing together, realize load balancing.Replicate run efficient between the master and slave database server is high more, and the consistance of the data on the master and slave database server is just good more, and the effect of above-mentioned functions is also good more.
Need to prove; though it is above-mentioned in the data of description processing procedure; be with Fig. 1; Fig. 2 or data handling system shown in Figure 3 are according to describing; but those skilled in the art should be understood that; Fig. 1; the functional module dividing mode of Fig. 2 and data handling system shown in Figure 3 only is a kind of in numerous functional module dividing mode; as long as data handling system can realize the data handling procedure that the embodiment of the invention is described; the function that possesses embodiment of the invention data handling system; its partition functionality module how no matter then all should be within protection scope of the present invention.
In sum, upgrade operation owing to use data to insert the operation alternate data in the embodiment of the present application, thereby do not upgrade action type in the data manipulation type, therefore there be not substituting and covering of data, as long as these insert statement and finally are synchronized to from database server, just can guarantee that the data in the master and slave database server are consistent, the sequencing that statement is carried out does not influence for data consistency, this just makes that database replication can multi-threaded parallel, even distributed execution, thereby can increase substantially execution efficient, eliminate and block.
The application is that reference is described according to the process flow diagram and/or the block scheme of method, equipment (system) and the computer program of the embodiment of the present application.Should understand can be by the flow process in each flow process in computer program instructions realization flow figure and/or the block scheme and/or square frame and process flow diagram and/or the block scheme and/or the combination of square frame.Can provide these computer program instructions to the processor of multi-purpose computer, special purpose computer, Embedded Processor or other programmable data processing device to produce a machine, make the instruction of carrying out by the processor of computing machine or other programmable data processing device produce to be used for the device of the function that is implemented in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame appointments.
These computer program instructions also can be stored in energy vectoring computer or the computer-readable memory of other programmable data processing device with ad hoc fashion work, make the instruction that is stored in this computer-readable memory produce the manufacture that comprises command device, this command device is implemented in the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame.
These computer program instructions also can be loaded on computing machine or other programmable data processing device, make on computing machine or other programmable devices and to carry out the sequence of operations step producing computer implemented processing, thereby the instruction of carrying out on computing machine or other programmable devices is provided for being implemented in the step of the function of appointment in flow process of process flow diagram or a plurality of flow process and/or square frame of block scheme or a plurality of square frame.
Obviously, those skilled in the art can carry out various changes and modification and the spirit and scope that do not break away from the application to the application.Like this, if these of the application are revised and modification belongs within the scope of the application's claim and equivalent technologies thereof, then the application also is intended to comprise these changes and modification interior.

Claims (15)

1. a data processing method is characterized in that, may further comprise the steps:
After getting access to the request that the data recording in the database is upgraded, be recorded in the new identical data recording of object sign of inserting in the described database according to the request data updated, and in the new data recording of inserting, adding version information, described version information is used to show the time sequencing of the insertion operation of the data recording with same object sign;
When another database is arrived in described database replication, utilize a plurality of thread parallels to carry out the database replicate run.
2. data processing method as claimed in claim 1 is characterized in that, is recorded in the new identical data recording of object sign of inserting in the database according to the request data updated, may further comprise the steps:
In described database, inquire about all the identical data recording of object identity that write down with the request data updated, and determine the wherein last data recording of inserting according to the version information in the data recording;
Merge the identical data recording of generation one object sign according to the data content in the data recording of data content after the renewal of carrying in the described request and insertion at last, and the data recording of generation is inserted in the described database.
3. data processing method as claimed in claim 1, it is characterized in that, also comprise: when predefined time or cycle arrive, inquire about the data recording in the described database, if the corresponding data recording of same target sign has many, then keep the last data recording of inserting in the data recording that inquires according to version information, and with the remainder data record deletion; Or,
When predefined time or cycle arrive, inquire about the data recording in the described database, if the corresponding data recording of same target sign has many, then the data recording that inquires is sorted according to version information, when the quantity of the data recording that inquires surpasses setting threshold, then the data recording of inserting at last from the data recording after the ordering begins to keep the quantity data record of this threshold value defined, and with the remainder data record deletion.
4. data processing method as claimed in claim 1 is characterized in that, before or after inserting new data recording, also comprises:
All identical data recording of object identity in inquiry and this new data recording keep the data recording of last insertion in the data recording inquire, and remaining data recording are deleted according to version information; Or,
All identical data recording of object identity in inquiry and this new data recording, and according to version information the data recording that inquires is sorted; If the quantity of the data recording that inquires surpasses setting threshold, then the data recording of inserting at last from the data recording after the ordering begins to keep the quantity data record of this threshold value defined, and with the remainder data record deletion.
5. data processing method as claimed in claim 1 is characterized in that, also comprises:
After the request that gets access to data query record, ask the object identity of the data recording of inquiring about to inquire all data recording according to institute with this object identity; According to the version information in the data recording, the last data recording of inserting in the data recording that inquires is returned as Query Result.
6. the method for claim 1 is characterized in that, also comprises:
After getting access to the request that the data recording in the database is deleted, data according to ask deletion are recorded in the new identical data recording of object sign of inserting in the described database, in the new data recording of inserting, add version information, and this data recording is labeled as deletion.
7. method as claimed in claim 6 is characterized in that, also comprises:
After the request that gets access to data query record, ask the object identity of the data recording of inquiring about to inquire all data recording according to institute with this object identity; According to the version information in the data recording, the last data recording of inserting is marked as when deleting in the data recording that judgement inquires, and returns to show the Query Result of not searching the data recording of inquiry of asking.
8. as each described data processing method of claim 1-7, it is characterized in that described version information is sequence number or timestamp;
If described version information is a sequence number, then in the new data recording of inserting, add version information, be specially:
In described database, inquire about all the identical data recording of object identity that write down with the request data updated, and determine the wherein last data recording of inserting according to the sequence number of version information field in the data recording;
The sequence number of the version information field of this last data recording of inserting is increased progressively, and the sequence number after will increasing progressively is as the sequence number of the version information field of the data recording of described generation.
9. a data handling system is characterized in that, comprising:
Acquisition module is used for obtaining the request that the data recording of database is upgraded;
Update processing module, be used for after getting access to the request that the data record is upgraded, be recorded in the new identical data recording of object sign of inserting in the described database according to the request data updated, and in the new data recording of inserting, adding version information, described version information is used to show the time sequencing of the insertion operation of the data recording with same object sign;
The replication processes module is used for described database replication utilizing a plurality of thread parallels to carry out the database replicate run when another database.
10. data handling system as claimed in claim 9, it is characterized in that, when the new insertion of described update processing module one object identifies identical data recording, in described database, inquire about all the identical data recording of object identity that write down with the request data updated, and determine the wherein last data recording of inserting according to the version information in the data recording; Then, merge generation one data recording, and the data recording that generates is inserted in the described database according to other data contents in the data recording of data content after the renewal of carrying in the described request and insertion at last.
11. data handling system as claimed in claim 9 is characterized in that, also comprises:
First removes processing module, be used for before or after described update processing module is inserted new data recording, the inquiry with this new data recording in all identical data recording of object identity, keep the last data recording of inserting in the data recording that inquires according to version information, and with remaining data recording deletion; Or, according to version information the data recording that inquires is sorted, if the quantity of the data recording that inquires surpasses setting threshold, then the data recording of inserting at last from the data recording after the ordering begins to keep the quantity data record of this threshold value defined, and with the remainder data record deletion.
12. data handling system as claimed in claim 9 is characterized in that, also comprises:
Second removes processing module, be used for when predefined time or cycle arrive, inquire about the data recording in the described database, if the corresponding data recording of same target sign has many, then keep the last data recording of inserting in the data recording that inquires according to version information, and with remaining data recording deletion; Or inquiry has the data recording of same object sign, and according to version information the data recording that inquires is sorted; If the quantity of the data recording that inquires surpasses setting threshold, then the data recording of inserting at last from the data recording after the ordering begins to keep the quantity data record of this threshold value defined, and with the remainder data record deletion.
13. data handling system as claimed in claim 9 is characterized in that, described acquisition module is further used for, and obtains the request of data query record;
Described data handling system also comprises:
Query processing module is used for inquiring all data recording with this object identity according to the object identity in the data recording of ask inquiry after the request that gets access to the data query record; Then, according to the version information in the data recording, the last data recording of inserting in the data recording that inquires is returned as Query Result.
14. system as claimed in claim 9 is characterized in that, described acquisition module is further used for, and obtains the request of delete data record;
Described data handling system also comprises:
The deletion processing module, be used for after getting access to the request that the data recording of database is deleted, data according to ask deletion are recorded in the new identical data recording of object sign of inserting in the described database, in the new data recording of inserting, add version information, and this data recording is labeled as deletion.
15. system as claimed in claim 14 is characterized in that, described acquisition module is further used for, and obtains the request of data query record;
Described data handling system also comprises:
Query processing module is used for after the request that gets access to data query record, asks the object identity of the data recording of inquiring about to inquire all data recording with this object identity according to institute; According to the version information in the data recording, the last data recording of inserting is marked as when deleting in the data recording that judgement inquires, and returns to show the Query Result of not searching the data recording of inquiry of asking.
CN200910260173.6A 2009-12-24 2009-12-24 A kind of data processing method and system thereof Active CN102110121B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN200910260173.6A CN102110121B (en) 2009-12-24 2009-12-24 A kind of data processing method and system thereof
HK11108936.7A HK1155236A1 (en) 2009-12-24 2011-08-24 A data processing method and system thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN200910260173.6A CN102110121B (en) 2009-12-24 2009-12-24 A kind of data processing method and system thereof

Publications (2)

Publication Number Publication Date
CN102110121A true CN102110121A (en) 2011-06-29
CN102110121B CN102110121B (en) 2015-09-23

Family

ID=44174283

Family Applications (1)

Application Number Title Priority Date Filing Date
CN200910260173.6A Active CN102110121B (en) 2009-12-24 2009-12-24 A kind of data processing method and system thereof

Country Status (2)

Country Link
CN (1) CN102110121B (en)
HK (1) HK1155236A1 (en)

Cited By (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102231161A (en) * 2011-06-30 2011-11-02 北京新媒传信科技有限公司 Method for synchronously verifying and monitoring databases
CN102663045A (en) * 2012-03-29 2012-09-12 苏州阔地网络科技有限公司 Method and system for processing data information
CN103020058A (en) * 2011-09-21 2013-04-03 阿里巴巴集团控股有限公司 Multi-version data acquisition method and device
CN103092840A (en) * 2011-10-28 2013-05-08 上海邮电设计咨询研究院有限公司 Method for acquiring self-increment mass data files from multiple sources
CN103390041A (en) * 2013-07-18 2013-11-13 杭州东信北邮信息技术有限公司 Method and system for providing data service based on middleware
CN103455677A (en) * 2013-09-04 2013-12-18 广东电网公司电力调度控制中心 Environment simulation method and system
CN103744906A (en) * 2013-12-26 2014-04-23 乐视网信息技术(北京)股份有限公司 System, method and device for data synchronization
CN104239476A (en) * 2014-09-04 2014-12-24 上海天脉聚源文化传媒有限公司 Method, device and system for synchronizing databases
CN104699541A (en) * 2015-03-30 2015-06-10 北京奇虎科技有限公司 Method, device, data transmission assembly and system for synchronizing data
CN105893393A (en) * 2015-01-26 2016-08-24 阿里巴巴集团控股有限公司 Data storage method and device
CN105989195A (en) * 2015-03-23 2016-10-05 国际商业机器公司 Approach and system for processing data in database
CN106156070A (en) * 2015-03-31 2016-11-23 华为技术有限公司 A kind of querying method, Piece file mergence method and relevant apparatus
CN106326425A (en) * 2016-08-24 2017-01-11 明算科技(北京)股份有限公司 Data classification processing method and device
WO2017181430A1 (en) * 2016-04-22 2017-10-26 华为技术有限公司 Method and device for duplicating database in distributed system
CN107733957A (en) * 2016-08-12 2018-02-23 北京融聚世界网络科技有限公司 Distributed Services configure system and version number's distribution method
CN107861959A (en) * 2016-09-22 2018-03-30 阿里巴巴集团控股有限公司 Data processing method, apparatus and system
CN108073596A (en) * 2016-11-10 2018-05-25 北京国双科技有限公司 The data-erasure method and device of a kind of olap database
CN108399259A (en) * 2018-03-09 2018-08-14 深圳市汗青文化传媒有限公司 A kind of data processing method and system
WO2018203185A1 (en) * 2017-05-02 2018-11-08 International Business Machines Corporation Asynchronous data store operations
CN108804442A (en) * 2017-04-27 2018-11-13 北京京东尚科信息技术有限公司 Sequence number generation method and device
CN109144980A (en) * 2018-08-21 2019-01-04 成都四方伟业软件股份有限公司 Metadata management method, device and electronic equipment
CN109408589A (en) * 2018-09-14 2019-03-01 新华三大数据技术有限公司 Method of data synchronization and device
CN110069487A (en) * 2017-09-28 2019-07-30 北京国双科技有限公司 A kind of data processing method, apparatus and system
CN110544086A (en) * 2019-07-17 2019-12-06 金华苏夏信息技术有限公司 Non-networking selective payment method for hotel sales counter
US10540282B2 (en) 2017-05-02 2020-01-21 International Business Machines Corporation Asynchronous data store operations including selectively returning a value from cache or a value determined by an asynchronous computation
CN111367893A (en) * 2020-03-31 2020-07-03 中国建设银行股份有限公司 Method and device for database version iteration
CN111488483A (en) * 2020-04-16 2020-08-04 北京雷石天地电子技术有限公司 Method, device, terminal and non-transitory computer-readable storage medium for updating song library
CN112015819A (en) * 2020-08-31 2020-12-01 杭州欧若数网科技有限公司 Data updating method, device, equipment and medium for distributed graph database
CN112114839A (en) * 2019-06-20 2020-12-22 上海安吉星信息服务有限公司 Method and system for rapid upgrade of standby environment
CN113287099A (en) * 2019-01-23 2021-08-20 株式会社斯凯拉 System with tamper detection
CN113468153A (en) * 2020-12-01 2021-10-01 王跃 Medical data management system based on big data
WO2022111733A1 (en) * 2020-11-30 2022-06-02 百果园技术(新加坡)有限公司 Message processing method and apparatus, and electronic device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060038260A1 (en) * 2003-07-01 2006-02-23 Renesas Technology Corp Semiconductor wafer and method of manufacturing semiconductor device
CN1856785A (en) * 2004-03-29 2006-11-01 微软公司 Systems and methods for versioning based triggers
CN101408864A (en) * 2007-10-09 2009-04-15 群联电子股份有限公司 Data protection method for power failure and controller using the same
CN101506766A (en) * 2005-05-10 2009-08-12 微软公司 Database corruption recovery systems and methods

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060038260A1 (en) * 2003-07-01 2006-02-23 Renesas Technology Corp Semiconductor wafer and method of manufacturing semiconductor device
CN1856785A (en) * 2004-03-29 2006-11-01 微软公司 Systems and methods for versioning based triggers
CN101506766A (en) * 2005-05-10 2009-08-12 微软公司 Database corruption recovery systems and methods
CN101408864A (en) * 2007-10-09 2009-04-15 群联电子股份有限公司 Data protection method for power failure and controller using the same

Cited By (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102231161A (en) * 2011-06-30 2011-11-02 北京新媒传信科技有限公司 Method for synchronously verifying and monitoring databases
CN103020058B (en) * 2011-09-21 2016-07-06 阿里巴巴集团控股有限公司 A kind of multi-version data acquisition method and device
CN103020058A (en) * 2011-09-21 2013-04-03 阿里巴巴集团控股有限公司 Multi-version data acquisition method and device
CN103092840B (en) * 2011-10-28 2015-09-16 上海邮电设计咨询研究院有限公司 Multi-source is from increasing massive data files real-time collecting method
CN103092840A (en) * 2011-10-28 2013-05-08 上海邮电设计咨询研究院有限公司 Method for acquiring self-increment mass data files from multiple sources
CN102663045A (en) * 2012-03-29 2012-09-12 苏州阔地网络科技有限公司 Method and system for processing data information
CN103390041A (en) * 2013-07-18 2013-11-13 杭州东信北邮信息技术有限公司 Method and system for providing data service based on middleware
CN103390041B (en) * 2013-07-18 2016-05-04 杭州东信北邮信息技术有限公司 A kind of method and system that data, services is provided based on middleware
CN103455677A (en) * 2013-09-04 2013-12-18 广东电网公司电力调度控制中心 Environment simulation method and system
CN103455677B (en) * 2013-09-04 2017-06-09 广东电网公司电力调度控制中心 Environmental simulation method and system
CN103744906A (en) * 2013-12-26 2014-04-23 乐视网信息技术(北京)股份有限公司 System, method and device for data synchronization
CN104239476A (en) * 2014-09-04 2014-12-24 上海天脉聚源文化传媒有限公司 Method, device and system for synchronizing databases
CN104239476B (en) * 2014-09-04 2018-09-25 上海天脉聚源文化传媒有限公司 A kind of method, apparatus and system of database synchronization
CN105893393B (en) * 2015-01-26 2019-11-05 阿里巴巴集团控股有限公司 Data save method and device
CN105893393A (en) * 2015-01-26 2016-08-24 阿里巴巴集团控股有限公司 Data storage method and device
US10262013B2 (en) 2015-03-23 2019-04-16 International Business Machines Corporation Efficient full delete operations
CN105989195A (en) * 2015-03-23 2016-10-05 国际商业机器公司 Approach and system for processing data in database
CN105989195B (en) * 2015-03-23 2019-09-13 国际商业机器公司 For handling the method and system of database data
CN104699541B (en) * 2015-03-30 2018-07-10 北京奇虎科技有限公司 Method, apparatus, data transfer components and the system of synchrodata
CN104699541A (en) * 2015-03-30 2015-06-10 北京奇虎科技有限公司 Method, device, data transmission assembly and system for synchronizing data
CN106156070A (en) * 2015-03-31 2016-11-23 华为技术有限公司 A kind of querying method, Piece file mergence method and relevant apparatus
CN106156070B (en) * 2015-03-31 2019-07-12 华为技术有限公司 A kind of querying method, file mergences method and relevant apparatus
US11093522B2 (en) 2016-04-22 2021-08-17 Huawei Technologies Co., Ltd. Database replication method and apparatus for distributed system
WO2017181430A1 (en) * 2016-04-22 2017-10-26 华为技术有限公司 Method and device for duplicating database in distributed system
CN107733957A (en) * 2016-08-12 2018-02-23 北京融聚世界网络科技有限公司 Distributed Services configure system and version number's distribution method
CN107733957B (en) * 2016-08-12 2020-10-16 北京融聚世界网络科技有限公司 Distributed service configuration system and version number distribution method
CN106326425A (en) * 2016-08-24 2017-01-11 明算科技(北京)股份有限公司 Data classification processing method and device
CN106326425B (en) * 2016-08-24 2019-11-05 明算科技(北京)股份有限公司 Data classification treating method and apparatus
CN107861959A (en) * 2016-09-22 2018-03-30 阿里巴巴集团控股有限公司 Data processing method, apparatus and system
CN108073596A (en) * 2016-11-10 2018-05-25 北京国双科技有限公司 The data-erasure method and device of a kind of olap database
CN108073596B (en) * 2016-11-10 2020-08-14 北京国双科技有限公司 Data deletion method and device for OLAP database
CN108804442A (en) * 2017-04-27 2018-11-13 北京京东尚科信息技术有限公司 Sequence number generation method and device
CN108804442B (en) * 2017-04-27 2022-06-07 北京京东尚科信息技术有限公司 Serial number generation method and device
US10540282B2 (en) 2017-05-02 2020-01-21 International Business Machines Corporation Asynchronous data store operations including selectively returning a value from cache or a value determined by an asynchronous computation
US11157406B2 (en) 2017-05-02 2021-10-26 International Business Machines Corporation Methods for providing data values using asynchronous operations and querying a plurality of servers
WO2018203185A1 (en) * 2017-05-02 2018-11-08 International Business Machines Corporation Asynchronous data store operations
CN110069487A (en) * 2017-09-28 2019-07-30 北京国双科技有限公司 A kind of data processing method, apparatus and system
CN108399259A (en) * 2018-03-09 2018-08-14 深圳市汗青文化传媒有限公司 A kind of data processing method and system
CN109144980A (en) * 2018-08-21 2019-01-04 成都四方伟业软件股份有限公司 Metadata management method, device and electronic equipment
CN109408589A (en) * 2018-09-14 2019-03-01 新华三大数据技术有限公司 Method of data synchronization and device
CN109408589B (en) * 2018-09-14 2020-08-14 新华三大数据技术有限公司 Data synchronization method and device
CN113287099A (en) * 2019-01-23 2021-08-20 株式会社斯凯拉 System with tamper detection
CN113287099B (en) * 2019-01-23 2024-05-28 株式会社斯凯拉 System with tamper detection
CN112114839A (en) * 2019-06-20 2020-12-22 上海安吉星信息服务有限公司 Method and system for rapid upgrade of standby environment
CN110544086A (en) * 2019-07-17 2019-12-06 金华苏夏信息技术有限公司 Non-networking selective payment method for hotel sales counter
CN111367893A (en) * 2020-03-31 2020-07-03 中国建设银行股份有限公司 Method and device for database version iteration
CN111488483A (en) * 2020-04-16 2020-08-04 北京雷石天地电子技术有限公司 Method, device, terminal and non-transitory computer-readable storage medium for updating song library
CN111488483B (en) * 2020-04-16 2023-10-24 北京雷石天地电子技术有限公司 Method, device, terminal and non-transitory computer readable storage medium for updating a library
CN112015819A (en) * 2020-08-31 2020-12-01 杭州欧若数网科技有限公司 Data updating method, device, equipment and medium for distributed graph database
WO2022111733A1 (en) * 2020-11-30 2022-06-02 百果园技术(新加坡)有限公司 Message processing method and apparatus, and electronic device
CN113468153A (en) * 2020-12-01 2021-10-01 王跃 Medical data management system based on big data

Also Published As

Publication number Publication date
CN102110121B (en) 2015-09-23
HK1155236A1 (en) 2012-05-11

Similar Documents

Publication Publication Date Title
CN102110121A (en) Method and system for processing data
JP6553822B2 (en) Dividing and moving ranges in distributed systems
US7958088B2 (en) Dynamic data reorganization to accommodate growth across replicated databases
KR20200093569A (en) Resolving violations in client synchronization
CN111881223B (en) Data management method, device, system and storage medium
CN107077495A (en) High performance transaction in data base management system
CN101989301B (en) A Method of Index Maintenance Supporting Multiple Data Sources
US7523141B2 (en) Synchronization operations involving entity identifiers
CN105630860A (en) Database system with transaction control block index
CN102831120A (en) Data processing method and system
US10296542B2 (en) Integration database framework
CN103714090A (en) Multi-index based database transaction processing method and database
JP5799812B2 (en) Data allocation / calculation system, data allocation / calculation method, master device, and data allocation method
CN104216955A (en) Methods and devices for operating data and managing events and distributed system
AU2018290753B2 (en) Systems and methods of creation and deletion of tenants within a database
CN107870954A (en) A distributed database extension method and device
CN104881466A (en) Method and device for processing data fragments and deleting garbage files
KR100529661B1 (en) Object integrated management system
CN104410666B (en) The method and system of isomerism storage resources management are realized under cloud computing
CN102202087A (en) Method for identifying storage equipment and system thereof
US9569461B2 (en) Distributed data authority system
CN103365987A (en) Clustered database system and data processing method based on shared-disk framework
CN103678577A (en) Method and device for updating data
CN106933555A (en) A kind of data framework system
CN105550342B (en) A kind of data processing method of the distributed data base of all-transparent

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1155236

Country of ref document: HK

C14 Grant of patent or utility model
GR01 Patent grant
REG Reference to a national code

Ref country code: HK

Ref legal event code: GR

Ref document number: 1155236

Country of ref document: HK