CN104572737A - Data storage aiding method and system - Google Patents
Data storage aiding method and system Download PDFInfo
- Publication number
- CN104572737A CN104572737A CN201310503949.9A CN201310503949A CN104572737A CN 104572737 A CN104572737 A CN 104572737A CN 201310503949 A CN201310503949 A CN 201310503949A CN 104572737 A CN104572737 A CN 104572737A
- Authority
- CN
- China
- Prior art keywords
- data
- ldb
- mdb
- task processing
- processing threads
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage aiding method and system. A data storage system is in a three-level structure of a DB (database), an LDB (lasting database) and an MDB (memory database). When an application system sends out a search request, the MDB is searched first; if the MDB is missed, searched data is returned by searching the LDB, and searched data in the LDB are written back into the MDB; if the LDB is missed, searched data are returned by searching the DB, and the searched data in the DB are written back into the LDB. The method includes: judging whether or not data in the DB are modified; if yes, modifying corresponding data in the LDB, invalidating the corresponding data in the MDB so that the MDB is searched when the application system sends out the search request for the modified data, and after the MDB is missed, returning the modified data by searching the LDB. The data storage aiding method and system has the advantages that read request processing stress s of the database can be decreased and responding efficiency of a write request can be increased.
Description
Technical field
The application relates to technical field of data storage, particularly relates to data and stores householder method and system.
Background technology
In some application or service system, usually need some relevant data to store in a database, and process accordingly according to the needs of management.In E-commerce transaction platform, seller user by the information of business object (such as commodity, service etc.), can comprise inventory information etc., is published in transaction platform.In transaction platform, need the setting according to seller user, the inventory information of conservation object in a database, and, when buyer user produces buying behavior for certain object, need to reduce the stockpile number of business object, other users when inquiring about the stockpile number of this business object, then can demonstrate up-to-date stockpile number.
But, in concurrent at height, large request amount situation, can strengthen suddenly the read-write requests amount of database, be easy to the limit reaching database processing ability, the partial function of system can be caused so normally to use.Therefore, database becomes the bottleneck of system for restricting fast development gradually.
Solution general at present adopts the two-level memory framework of buffer memory (MDB)+database (DB) substantially, wherein, MDB is arranged in internal memory, read or write speed is far longer than DB, TPS when wall scroll record size is 512 byte reaches 70,000, response time RT<2ms.See Fig. 1, in the two-level memory framework of MDB+DB, when a query manipulation comes interim, common way is:
1.0: first inquire about in MDB, if hit in MDB, then directly return Query Result;
1.1: if miss in MDB, then go to inquire about DB;
1.2: the result inquired in DB is written back in MDB.
When a write operation requests arrives, common way is:
2.0: first the data in DB are modified;
2.1: data corresponding in MDB are carried out invalidation, such as, carries out the operations such as deletion.
Such as, suppose to produce an inquiry request in certain application, corresponding query object is data A, and now, first inquiry request arrives MDB, if there is this record in MDB, then directly to return from MDB, otherwise this inquiry request can be sent to DB, after DB returns Query Result, Query Result is written in MDB.Like this, follow-up when having this data A of other application queries again, then directly to return from MDB, DB need not be inquired about again.
There is again the amendment of applying and creating data A if follow-up, then can modify to data A in DB, and the record about this data A in MDB is deleted; Now, if there are other application to initiate inquiry to data A again, then cannot inquire in MDB, but be sent in DB and inquire about, obtain Query Result in DB after, be then written in MDB.The consistance of data between MDB and DB can be ensured like this, and then ensure the correctness of Query Result.
Visible, this two-level memory structure decreases the inquiry request directly to DB, is conducive to reducing the response time applied when writing DB.But at least there is following problem: according to the characteristic of MDB itself, data are all kept in internal memory, and limit due to the size of internal memory, data space can not dilatation arbitrarily, this makes to store restriction, and easily due to a variety of causes (as server resets etc.) obliterated data, data integrity can not be guaranteed.In addition, when write request is a lot, because the method can empty data corresponding in MDB; therefore; the hit rate of read request after making in MDB sharply declines, thus makes greatly to reduce the protection of DB, and then the response efficiency of DB to write operation requests is also decreased.
Summary of the invention
This application provides data and store householder method and system, the read request processing pressure of database D B can be reduced, improve the response efficiency to write operation requests.
This application provides following scheme:
A kind of data store householder method, data-storage system adopts database D B, perdurable data storehouse LDB, memory database MDB tertiary structure, when application system sends inquiry request, first inquire about in MDB, if MDB is miss, then return by inquiry LDB the data inquired, and by the write back data that inquires in LDB to MDB; As LDB is also miss, then return by inquiry DB the data inquired, and by the write back data that inquires in DB to LDB; Described method comprises:
Judge whether the data in DB are revised;
If, then data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation, so that when application system sends the inquiry request to the described data revised, first inquire about in MDB, after discovery MDB is miss, then the data revised described in being returned by inquiry LDB.
A kind of data store backup system, data-storage system adopts database D B, perdurable data storehouse LDB, memory database MDB tertiary structure, when application system sends inquiry request, first inquire about in MDB, if MDB is miss, then return by inquiry LDB the data inquired, and by the write back data that inquires in LDB to MDB; As LDB is also miss, then return by inquiry DB the data inquired, and by the write back data that inquires in DB to LDB; Described backup system comprises:
Judging unit, for judging whether the data in DB are revised;
Amendment and invalidation unit, if the judged result for described judging unit is yes, then data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation, so that when application system sends the inquiry request to the described data revised, first inquire about in MDB, after discovery MDB is miss, then the data revised described in being returned by inquiry LDB.
According to the specific embodiment that the application provides, this application discloses following technique effect:
Pass through the embodiment of the present application; data-storage system can adopt DB, LDB, MDB tertiary storage structure; and when there is write operation requests; the data consistency between LDB and DB can be ensured, simultaneously again because LDB has persistence, therefore; when read operation request can not be hit in MDB; generally can hit at LDB layer, thus avoid a large amount of read operation request to be sent to DB, more be conducive to the protection to DB.Simultaneously; because MDB, DB still exist; therefore; MDB can be used and read and write the highest advantage of speed; and the advantage that DB supports affairs and complex query can be used; in a word, while being equivalent to make the advantage of whole storage system when retaining DB, MDB two-level memory, the protection to DB is strengthened.And for DB, be interactional between read operation request and write operation requests, when the read operation request quantity arriving DB reduces, the write operation response time of DB also can be reduced, and is therefore conducive to the performance improving storage system on the whole.
Wherein, when specifically corresponding in LDB data are modified, the mode of asynchronous amendment can be adopted, can avoid like this affecting the response time of DB to write operation, and be also conducive to making the sequencing to the difference amendment of same data produces according to amendment carry out serial processing, when a large amount of write operation requests is concurrent, also ensure the data consistency of LDB and DB.
Certainly, the arbitrary product implementing the application might not need to reach above-described all advantages simultaneously.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present application or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment below, apparently, accompanying drawing in the following describes is only some embodiments of the application, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is storage organization schematic diagram of the prior art;
Fig. 2 is the process flow diagram of the method that the embodiment of the present application provides;
Fig. 3 is the storage organization schematic diagram that the embodiment of the present application provides;
Fig. 4 is the schematic diagram of the system that the embodiment of the present application provides.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present application, be clearly and completely described the technical scheme in the embodiment of the present application, obviously, described embodiment is only some embodiments of the present application, instead of whole embodiments.Based on the embodiment in the application, the every other embodiment that those of ordinary skill in the art obtain, all belongs to the scope of the application's protection.
In the embodiment of the present application, in order to better protect DB, propose a kind of multistage storage organization, that is, increase one deck and store between DB and the MDB of prior art, this layer stores and adopts LDB to realize.Then by ensureing the consistance of data between LDB and DB, make the inquiry request of miss MDB, great majority can obtain Query Result in LDB, thus reduce the inquiry request to DB, better play the effect of protection DB.
Wherein, LDB is a kind of perdurable data storehouse (also claiming persistence buffer memory), and its read-write speed is much larger than DB, and according to statistics, TPS when wall scroll record size is 512 byte can reach 50,000, response time RT<=2ms.Visible, its read-write efficiency will lower than MDB, and for DB, it does not support affairs, does not support complex query, and dirigibility is inadequate, but with regard to single read-write requests, read-write efficiency contrast DB has very large advantage, and data security, be persistent storage.
Therefore, in the embodiment of the present application, employ DB-LDB-MDB tertiary storage structure, directly do not replace MDB or DB, make to store use and can use respective advantage.First, because MDB is located immediately in internal memory, its read-write speed is the highest, is therefore retained; DB can support the inquiry of affairs and complexity, is therefore also retained.Like this, in the storage architecture of the embodiment of the present application, for read operation request, first request arrives MDB, if MDB hit, is then directly returned the data inquired by MDB, otherwise read operation request arrives LDB, if LDB hit, then returned the data inquired by LDB, and by the write back data that inquires in LDB in MDB; If LDB does not also hit, read operation request just can be sent to DB, and by the write back data that inquires in DB to LDB.That is, the data in MDB are the inquiry request according to application, write-back from LDB, data in LDB are the inquiry request according to application, write-back from DB.Certainly, in actual applications, when data inquiry request is all miss in MDB and LDB, the data inquired in DB also can be written back in LDB and MDB simultaneously, like this, the request having these data of inquiry next time more directly in MDB, just can obtain Query Result.
That is, in an initial condition, data are only kept in DB, and along with the generation of read operation request, the data be read can progressively be write in LDB and MDB; When producing write operation requests in application system, after data in DB are revised, the data in LDB are also modified, and data corresponding in MDB are then disabled, new read request cannot get Query Result from MDB, but in most cases can get Query Result from LDB; When serve the situation such as to restart time, the data in MDB can be lost, but the data in LDB can not be lost.
When specific implementation, the embodiment of the present application also needs to provide an auxiliary system, to solve the problem of data consistency between DB, LDB, MDB tri-grades.Here it should be noted that, about the Data Consistency between these three grades, basic starting point is, data between LDB and DB are consistent as far as possible, also be, after having carried out the data in DB according to write operation requests revising, data corresponding in LDB are also modified, and by data invalid corresponding in MDB.Like this, when producing the request of these data of inquiry in application system, can not hit in MDB, and next inquiry request can be sent to LDB, because data corresponding in LDB are revised, therefore, just can provide Query Result from LDB layer, and not need to inquire about in DB again.Just the specific implementation of this backup system is described in detail below.
See Fig. 2, the embodiment of the present application provide firstly a kind of data and stores householder method, and the method can comprise the following steps:
S201: judge whether the data in DB are revised;
S202: if, then data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation, so that when application system sends the inquiry request to the described data revised, first inquire about in MDB, after discovery MDB is miss, then the data revised described in being returned by inquiry LDB.
Wherein, specifically corresponding in LDB data modify and to MDB in corresponding data when carrying out invalid, can realize by calling corresponding interface.
During specific implementation, synchronous vacations can be carried out to the data in LDB.That is, storage system is after the write operation requests receiving an application system transmission, (comprise the renewal of data, deletion, newly-built etc.) can be modified to the data in DB in a thread, when synchronous vacations LDB, can directly in this thread to LDB in corresponding data modify, return again after the data modification success in LDB.When the concurrent quantity of write operation requests is little, the mode of this synchronous vacations can realize the data consistency in LDB and DB more timely.But in the concurrent large numbers of situation of write request, then the implementation of concurrent modification LDB data may cause phenomenons such as makeing mistakes.Such as, when height is concurrent, the repeatedly amendment about same data may be produced simultaneously, now, if synchronously modified to the data in LDB in the thread of amendment DB data, then may make mistakes or extend the response time of system.As, suppose that the data be modified in DB are stockpile numbers of certain merchandise items, suppose that the quantity before revising is 100, certain moment receives first write request, need to reduce 1, now, the data in DB are modified to 99, during data in synchronous vacations LDB, also the stockpile number of correspondence is revised as 99 by 100; But, just after receiving first write request, receive again a write request, the stockpile number of this merchandise items is reduced 2 by request again, now, thread for first write request may not yet return, need after the thread of wait first write request returns, second write request could be processed, otherwise, if it is just direct to the data modification in DB and LDB not wait for that first thread returns, will make mistakes.Therefore, in the embodiment of the present application, the implementation of asynchronous amendment LDB data can also be adopted.
That is, for revising the thread of data in DB, just can return after having revised the data in DB, and about the amendment of corresponding data in LDB, then can carry out in another one thread.For ease of describing, this other thread can be called " asynchronous task processing threads ".That is, the data modification in LDB and the data modification in DB are not that Complete Synchronization carries out, but after the data modification in DB completes, then by asynchronous task processing threads, data corresponding in LDB are modified.Like this, the thread revising DB data can be made to return more timely, improve the speed of response of write request.Certainly, when asynchronous task processing threads is modified to the data in LDB, the order that can be modified according to data in DB is carried out.
During specific implementation, in order to the order that can be modified according to data in DB is to revise data corresponding in LDB, the change of the journal file (the Binlog daily record etc. of such as Mysql database) of DB can be monitored, such as, the timing mode such as to check can be adopted to monitor, when finding to produce new record in journal file, can extract and carry out resolving (in the every bar record in journal file, preserving the information such as action type), whether the data just can judged in DB by analysis result there occurs amendment, and, the time that in journal file, each bar record produces, reflect the sequencing that the data in DB are modified, therefore, in asynchronous task processing threads, just can be recorded according to each bar the sequencing be created in journal file, data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation.
In actual applications, in order to improve the ability tackling high complications, multiple asynchronous task processing threads can be created in storage system, the operation of amendment LDB corresponding data can be assigned on different asynchronous task processing threads and perform by backup system, be conducive to like this improve treatment effeciency, reduce from DB data be modified to LDB data modification complete between time delay.Certainly, for the different amendment records that same data produce, then can be assigned on same asynchronous task processing threads and perform, perform to the time sequencing that asynchronous task processing threads can be made like this can be created according to different amendment record the retouching operation to corresponding data in LDB, avoid makeing mistakes.
During specific implementation, because each data in DB have unique identification usually, such as, for these data of the merchandise items in E-commerce transaction platform, generally can carry out unique identification respectively with ItemID, this ItemID is generally digital ID, is also namely all to represent with string number; In addition, each asynchronous task processing threads also can have respective numeral number, such as, comprises and is numbered 0,1,2 ... asynchronous task processing threads, etc.Therefore, process to the different amendment records for same data are assigned on same asynchronous task processing threads, modulus can be add up to asynchronous task processing threads, modular arithmetic is asked to the unique identification of the data be modified, using be numbered ask modular arithmetic result asynchronous task processing threads as described target asynchronous task processing threads.Such as, when extracting Article 1 data modification record from journal file, first the ID of the data be modified can therefrom be got, be such as 0002, suppose that asynchronous task processing threads adds up to 100, then, after carrying out asking modular arithmetic, the result obtained is 2, therefore, just this data modification record can be distributed to the asynchronous task processing threads being numbered 2; Next, if the ID that Article 2 data modification record shows the data be modified also is 0002, then this data amendment record also can be assigned to the asynchronous task processing threads being numbered 2, by that analogy.So just can ensure all to be assigned to same asynchronous task processing threads for the different amendment records of same data, this asynchronous task processing threads then can according to the generation sequencing of different amendment record, data corresponding in LDB are performed to the amendment of serial, avoid occurring mistake, cause the data in LDB and the data in DB inconsistent.
Certainly, it should be noted that, due in the embodiment of the present application, asynchronous carrying out to the amendment of data in the amendment of data in LDB and DB, therefore, from DB, data modification is accomplished to data modification in LDB and completes and generally can produce certain time delay, this time delay be generally Millisecond other, therefore, generally can not concrete operation system be had an impact.
It should be noted that in addition, in the embodiment of the present application, from DB data modification be accomplished to data modification in LDB complete between time delay, related with the asynchronous task processing threads quantity created in system, when the task amount of each asynchronous task processing threads is all more saturated, this time delay may be made to increase.Therefore, a time threshold can be pre-set, and in backup system, can also to each bar record from be created to journal file asynchronous amendment in LDB complete between the time interval add up, if the time interval exceedes this preset time threshold, then warning information can be sent, like this, just can determine to carry out dilatation process the need of to asynchronous task processing threads according to this warning information further, so-called dilatation process, namely according to the speed of asynchronous task processing threads process journal file, and the increment etc. recorded in journal file, create more asynchronous task processing threads, with reduce from daily record produce asynchronous amendment LDB complete between time delay.
In order to understand the technical scheme that the embodiment of the present application proposes better, below by the storage organization schematic diagram in the embodiment of the present application and corresponding reading and writing data sequential chart, the embodiment of the present application is described in detail.
See Fig. 3, it is the storage organization schematic diagram in the embodiment of the present application, and visible whole storage system is divided into DB, LDB, MDB tertiary structure, and wherein, MDB is in the superiors, is still in internal memory, and LDB mediates layer.When application system produces read operation request, concrete sequential chart can as shown in Fig. 3 1.0 ~ 1.4.Wherein:
1.0: when receiving read operation request, first inquire about MDB;
1.1: if MDB hit, then return Query Result directly to application system, if not hit in MDB, then inquire about LDB;
1.2; If LDB hits, then return Query Result to application system, and Query Result is written back to MDB; If LDB is miss, then inquire about DB;
1.3: the Query Result obtained in DB is written back to LDB;
1.4: the Query Result obtained in DB is written back to MDB.
And when application system produces write operation requests, see the step 2.0 in Fig. 3, also, directly performing the operation of the data in amendment DB, after DB has revised, corresponding thread can return.
Then, in asynchronous task processing threads, perform 2.1 in Fig. 3: asynchronous amendment is carried out to data corresponding in LDB, and 2.2: invalidation is carried out to data corresponding in MDB.
In a word; in the embodiment of the present application; data-storage system can adopt DB, LDB, MDB tertiary storage structure, and when there is write operation requests, can ensure the data consistency between LDB and DB; simultaneously again because LDB has the feature of persistence; therefore, when read operation request can not be hit in MDB, generally can hit at LDB layer; thus avoid a large amount of read operation request to be sent to DB, be more conducive to the protection to DB.Simultaneously; because MDB, DB still exist; therefore; MDB can be used and read and write the highest advantage of speed; and the advantage that DB supports affairs and complex query can be used; in a word, while being equivalent to make the advantage of whole storage system when retaining DB, MDB two-level memory, the protection to DB is strengthened.And for DB, be interactional between read operation request and write operation requests, when the read operation request quantity arriving DB reduces, the write operation response time of DB also can be reduced, and is therefore conducive to the performance improving storage system on the whole.
To store householder method corresponding with the data that the embodiment of the present application provides, the embodiment of the present application additionally provides a kind of data and stores backup system, same, concrete data-storage system adopts database D B, perdurable data storehouse LDB, memory database MDB tertiary structure, when application system sends inquiry request, first inquire about in MDB, if MDB is miss, then return by inquiry LDB the data inquired, and by the write back data that inquires in LDB to MDB; As LDB is also miss, then return by inquiry DB the data inquired, and by the write back data that inquires in DB to LDB; See Fig. 4, this backup system can comprise with lower unit:
Judging unit 401, for judging whether the data in DB are revised;
Amendment and invalidation unit 402, if the judged result for described judging unit is yes, then data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation, so that when application system sends the inquiry request to the described data revised, first inquire about in MDB, after discovery MDB is miss, then the data revised described in being returned by inquiry LDB.
Wherein, during specific implementation, amendment and invalidation unit 402 can comprise:
Synchronous process subelement, in the thread for revising the data in DB, carries out synchronous vacations to data corresponding in LDB, and data corresponding in MDB is carried out invalidation.
Or under another kind of implementation, namely return after being completed by the data modification in DB for the thread revising the data in DB, described amendment and invalidation unit 402 can comprise:
Asynchronous process subelement, in asynchronous task processing threads, carries out asynchronous amendment to data corresponding in LDB, and data corresponding in MDB is carried out invalidation.
When asynchronous process, judging unit 401 specifically can comprise:
Journal file monitors subelement, for monitoring the change of the journal file of DB;
Read and resolve subelement, for when producing new record in described journal file, from described journal file, reading the record wherein produced and resolve;
Judge subelement, for judging according to analysis result whether the data in DB are revised.
When according to the record in journal file to LDB in corresponding data modify, the sequencing that can be recorded in journal file according to each bar processes, and accordingly, asynchronous process subelement specifically may be used for:
In asynchronous task processing threads, be recorded in the sequencing be created in journal file according to each bar, data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation.
Wherein, in order to improve the speed of asynchronous amendment, the asynchronous task processing threads in system may be multiple, and now, in order to avoid the different amendment records of same data are by different asynchronous task processing threads process, described asynchronous process subelement specifically may be used for:
According to the unique identification of the data be modified in DB, determine the target asynchronous task processing threads revising corresponding data in LDB, the difference amendment task for same data is sent to same target asynchronous task processing threads, and in same target asynchronous task processing threads, the time order and function order produced according to amendment task, modifies to data corresponding in LDB.
During specific implementation, described unique identification is the ID being modified data, described ID numeral, and described asynchronous task processing threads has respective numeral number, and described asynchronous process subelement specifically can comprise:
Asking modular arithmetic subelement, for ading up to modulus with asynchronous task processing threads, modular arithmetic being asked to the unique identification of the data be modified, using be numbered ask modular arithmetic result asynchronous task processing threads as described target asynchronous task processing threads.
In addition, this backup system can also comprise:
Time interval computing unit, is calculated by the time interval that asynchronous amendment completes for being modified the data being accomplished to correspondence in LDB to the data in DB;
Alarm Unit, for when the described time interval exceedes preset time threshold, sends warning information, to carry out dilatation process according to described warning information to asynchronous task processing threads.
When application system sends inquiry request, if MDB and LDB is all miss, then return by inquiry DB the data inquired, and by the write back data that inquires in DB to LDB and MDB.
In a word; in the embodiment of the present application; data-storage system can adopt DB, LDB, MDB tertiary storage structure, and when there is write operation requests, can ensure the data consistency between LDB and DB; simultaneously again because LDB has the feature of persistence; therefore, when read operation request can not be hit in MDB, generally can hit at LDB layer; thus avoid a large amount of read operation request to be sent to DB, be more conducive to the protection to DB.Meanwhile, because MDB, DB still exist, therefore; MDB can be used and read and write the highest advantage of speed; and the advantage that DB supports affairs and complex query can being used, while being equivalent to make the advantage of whole storage system when retaining DB, MDB two-level memory, strengthening the protection to DB.And for DB, be interactional between read operation request and write operation requests, when the read operation request quantity arriving DB reduces, the write operation response time of DB also can be reduced, and is therefore conducive to the performance improving storage system on the whole.
As seen through the above description of the embodiments, those skilled in the art can be well understood to the mode that the application can add required general hardware platform by software and realizes.Based on such understanding, the technical scheme of the application can embody with the form of software product the part that prior art contributes in essence in other words, this computer software product can be stored in storage medium, as ROM/RAM, magnetic disc, CD etc., comprising some instructions in order to make a computer equipment (can be personal computer, server, or the network equipment etc.) perform the method described in some part of each embodiment of the application or embodiment.
Each embodiment in this instructions all adopts the mode of going forward one by one to describe, between each embodiment identical similar part mutually see, what each embodiment stressed is the difference with other embodiments.Especially, for system or system embodiment, because it is substantially similar to embodiment of the method, so describe fairly simple, relevant part illustrates see the part of embodiment of the method.System described above and system embodiment are only schematic, the wherein said unit illustrated as separating component or can may not be and physically separates, parts as unit display can be or may not be physical location, namely can be positioned at a place, or also can be distributed in multiple network element.Some or all of module wherein can be selected according to the actual needs to realize the object of the present embodiment scheme.Those of ordinary skill in the art, when not paying creative work, are namely appreciated that and implement.
The storage system householder method above the application provided and system, be described in detail, apply specific case herein to set forth the principle of the application and embodiment, the explanation of above embodiment is just for helping method and the core concept thereof of understanding the application; Meanwhile, for one of ordinary skill in the art, according to the thought of the application, all will change in specific embodiments and applications.In sum, this description should not be construed as the restriction to the application.
Claims (10)
1. data store householder method, it is characterized in that, data-storage system adopts database D B, perdurable data storehouse LDB, memory database MDB tertiary structure, when application system sends inquiry request, first inquire about in MDB, if MDB is miss, then return by inquiry LDB the data inquired, and by the write back data that inquires in LDB to MDB; As LDB is also miss, then return by inquiry DB the data inquired, and by the write back data that inquires in DB to LDB; Described method comprises:
Judge whether the data in DB are revised;
If, then data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation, so that when application system sends the inquiry request to the described data revised, first inquire about in MDB, after discovery MDB is miss, then the data revised described in being returned by inquiry LDB.
2. method according to claim 1, is characterized in that, describedly modifies to data corresponding in LDB, and data corresponding in MDB are carried out invalidation, comprising:
In the thread for revising the data in DB, synchronous vacations is carried out to data corresponding in LDB, and data corresponding in MDB are carried out invalidation.
3. method according to claim 1, it is characterized in that, namely return after the data modification in DB is completed for the thread revising the data in DB, described data corresponding in LDB to be modified, and data corresponding in MDB are carried out invalidation, comprising:
In asynchronous task processing threads, asynchronous amendment is carried out to data corresponding in LDB, and data corresponding in MDB are carried out invalidation.
4. method according to claim 3, is characterized in that, whether the data in the described DB of judgement are revised, and comprising:
Monitor the change of the journal file of DB;
When producing new record in described journal file, from described journal file, reading the record wherein produced and resolve;
Whether revise according to the data that analysis result judges in DB.
5. method according to claim 4, is characterized in that, described in asynchronous task processing threads, carries out asynchronous amendment to data corresponding in LDB, and data corresponding in MDB are carried out invalidation, comprising:
In asynchronous task processing threads, be recorded in the sequencing be created in journal file according to each bar, data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation.
6. method according to claim 3, is characterized in that, when described asynchronous task processing threads is multiple, described in asynchronous task processing threads, asynchronous amendment is carried out to data corresponding in LDB, and data corresponding in MDB is carried out invalidation, comprising:
According to the unique identification of the data be modified in DB, determine the target asynchronous task processing threads revising corresponding data in LDB, the difference amendment task for same data is sent to same target asynchronous task processing threads, and in same target asynchronous task processing threads, the time order and function order produced according to amendment task, modifies to data corresponding in LDB.
7. method according to claim 6, it is characterized in that, described unique identification is the ID being modified data, described ID numeral, described asynchronous task processing threads has respective numeral number, the unique identification of the data that described basis is modified, determines the target asynchronous task processing threads revising corresponding data in LDB, comprising:
Add up to modulus with asynchronous task processing threads, modular arithmetic asked to the unique identification of the data be modified, using be numbered ask modular arithmetic result asynchronous task processing threads as described target asynchronous task processing threads.
8. method according to claim 3, is characterized in that, also comprises:
Be modified to the data in DB the data being accomplished to correspondence in LDB to be calculated by the time interval that asynchronous amendment completes;
When the described time interval exceedes preset time threshold, send warning information, to carry out dilatation process according to described warning information to asynchronous task processing threads.
9. the method according to any one of claim 1 to 8, is characterized in that, when application system sends inquiry request, if MDB and LDB is all miss, then returns by inquiry DB the data inquired, and by the write back data that inquires in DB to LDB and MDB.
10. data store backup system, it is characterized in that, data-storage system adopts database D B, perdurable data storehouse LDB, memory database MDB tertiary structure, when application system sends inquiry request, first inquire about in MDB, if MDB is miss, then return by inquiry LDB the data inquired, and by the write back data that inquires in LDB to MDB; As LDB is also miss, then return by inquiry DB the data inquired, and by the write back data that inquires in DB to LDB; Described backup system comprises:
Judging unit, for judging whether the data in DB are revised;
Amendment and invalidation unit, for if, then data corresponding in LDB are modified, and data corresponding in MDB are carried out invalidation, so that when application system sends the inquiry request to the described data revised, first inquire about in MDB, after discovery MDB is miss, then the data revised described in being returned by inquiry LDB.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310503949.9A CN104572737B (en) | 2013-10-23 | 2013-10-23 | Data storage householder method and system |
| HK15106191.7A HK1205801A1 (en) | 2013-10-23 | 2015-06-30 | Auxiliary method for storing data and system thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310503949.9A CN104572737B (en) | 2013-10-23 | 2013-10-23 | Data storage householder method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104572737A true CN104572737A (en) | 2015-04-29 |
| CN104572737B CN104572737B (en) | 2018-01-30 |
Family
ID=53088821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310503949.9A Active CN104572737B (en) | 2013-10-23 | 2013-10-23 | Data storage householder method and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104572737B (en) |
| HK (1) | HK1205801A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107783728A (en) * | 2016-08-31 | 2018-03-09 | 百度在线网络技术（北京）有限公司 | Date storage method, device and equipment |
| CN108959284A (en) * | 2017-05-17 | 2018-12-07 | 中国移动通信集团重庆有限公司 | Data processing method and device in columnar database |
| CN110765076A (en) * | 2019-10-25 | 2020-02-07 | 北京奇艺世纪科技有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN112632117A (en) * | 2020-12-30 | 2021-04-09 | 广州华多网络科技有限公司 | Method and device for processing serial number data, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6701359B1 (en) * | 1998-07-28 | 2004-03-02 | International Business Machines Corp. | Apparatus and method for managing a multi-threaded persistent agent management information base including a managed object instance cache |
| US6941295B2 (en) * | 1999-08-12 | 2005-09-06 | International Business Machines Corporation | Data access system |
| CN101510209A (en) * | 2009-03-30 | 2009-08-19 | 北京金山软件有限公司 | Method, system and server for implementing real time search |
| CN102542034A (en) * | 2011-12-23 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Database interface-based result set cache method |
| CN102880557A (en) * | 2012-09-13 | 2013-01-16 | 浙江金大科技有限公司 | Multistage distribution type high-speed cache of heterogeneous data source |
-
2013
- 2013-10-23 CN CN201310503949.9A patent/CN104572737B/en active Active
-
2015
- 2015-06-30 HK HK15106191.7A patent/HK1205801A1/en unknown
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6701359B1 (en) * | 1998-07-28 | 2004-03-02 | International Business Machines Corp. | Apparatus and method for managing a multi-threaded persistent agent management information base including a managed object instance cache |
| US6941295B2 (en) * | 1999-08-12 | 2005-09-06 | International Business Machines Corporation | Data access system |
| CN101510209A (en) * | 2009-03-30 | 2009-08-19 | 北京金山软件有限公司 | Method, system and server for implementing real time search |
| CN102542034A (en) * | 2011-12-23 | 2012-07-04 | 北京人大金仓信息技术股份有限公司 | Database interface-based result set cache method |
| CN102880557A (en) * | 2012-09-13 | 2013-01-16 | 浙江金大科技有限公司 | Multistage distribution type high-speed cache of heterogeneous data source |
Non-Patent Citations (2)
| Title |
|---|
| 曾超宇 等: ""Redis在高速缓存系统中的应用"", 《微型机与应用》 * |
| 杨艳 等: ""内存数据库在高速缓存方面的应用"", 《现代电信科技》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107783728A (en) * | 2016-08-31 | 2018-03-09 | 百度在线网络技术（北京）有限公司 | Date storage method, device and equipment |
| CN108959284A (en) * | 2017-05-17 | 2018-12-07 | 中国移动通信集团重庆有限公司 | Data processing method and device in columnar database |
| CN108959284B (en) * | 2017-05-17 | 2020-12-08 | 中国移动通信集团重庆有限公司 | Data processing method and device in column database |
| CN110765076A (en) * | 2019-10-25 | 2020-02-07 | 北京奇艺世纪科技有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN112632117A (en) * | 2020-12-30 | 2021-04-09 | 广州华多网络科技有限公司 | Method and device for processing serial number data, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104572737B (en) | 2018-01-30 |
| HK1205801A1 (en) | 2015-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108459826B (en) | Method and device for processing IO (input/output) request | |
| CN103870492B (en) | A kind of date storage method and device based on key row sequence | |
| CN102722449B (en) | Key-Value local storage method and system based on solid state disk (SSD) | |
| CN108009008A (en) | Data processing method and system, electronic equipment | |
| US10275481B2 (en) | Updating of in-memory synopsis metadata for inserts in database table | |
| CN103617097B (en) | File access pattern method and device | |
| CN104424219B (en) | A kind of management method and device of data file | |
| CN102867070A (en) | Method for updating cache of key-value distributed memory system | |
| US9128969B2 (en) | Columnwise storage of point data | |
| US11221999B2 (en) | Database key compression | |
| US11249968B2 (en) | Large object containers with size criteria for storing mid-sized large objects | |
| CN107391544A (en) | Processing method, device, equipment and the computer storage media of column data storage | |
| CN104572737A (en) | Data storage aiding method and system | |
| CN102968456B (en) | A kind of raster data reading and processing method and device | |
| KR20210076828A (en) | Key value device and block interface emulation method for the same | |
| US10747773B2 (en) | Database management system, computer, and database management method | |
| CN104461384A (en) | Data write-in method and storage device | |
| CN108897822A (en) | A kind of data-updating method, device, equipment and readable storage medium storing program for executing | |
| CN109086462A (en) | The management method of metadata in a kind of distributed file system | |
| KR101806394B1 (en) | A data processing method having a structure of the cache index specified to the transaction in a mobile environment dbms | |
| CN102073690A (en) | Memory database supporting historical Key information | |
| CN107133183A (en) | A kind of cache data access method and system based on TCMU Virtual Block Devices | |
| CN107704603A (en) | A kind of method and device for realizing read and write abruption | |
| CN104978283A (en) | Memory access control method and device | |
| CN108021562A (en) | Deposit method, apparatus and distributed file system applied to distributed file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1205801 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: GR Ref document number: 1205801 Country of ref document: HK |