CN102495834A - Incremental data cleaning method based on memory mapping - Google Patents
Incremental data cleaning method based on memory mapping Download PDFInfo
- Publication number
- CN102495834A CN102495834A CN2011103169145A CN201110316914A CN102495834A CN 102495834 A CN102495834 A CN 102495834A CN 2011103169145 A CN2011103169145 A CN 2011103169145A CN 201110316914 A CN201110316914 A CN 201110316914A CN 102495834 A CN102495834 A CN 102495834A
- Authority
- CN
- China
- Prior art keywords
- node
- fragment
- data
- auspj
- tree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an incremental data cleaning method based on memory mapping. Data of a data source to be cleaned are loaded into a memory, and a memory MDB (meta database)-tree is built by the aid of a HASH list; quick data inquiry is completed by the aid of a top-down strategy, and operations of data insertion, deletion and the like are completed on the basis; views in the memory mapping are standardized into AUSPJ (aggregation, union, selection, projection and join) sections and D (differential) sections, and incremental maintenance of nodes is realized by the aid of relational algebraic operation; and finally data cleaning operation is realized by the aid of a result collection merging algorithmic method. The incremental data cleaning method can be widely applied to industries and fields with high data quantities and data quality requirements, such as education, telecommunication, governments and the like.
Description
Technical field
The invention belongs to computerized information analysis and data processing field, be specifically related to a kind of incremental data cleaning method based on memory mapping.
Background technology
When setting up an infosystem,, can not guarantee in all cases that the quality of institute's store data can both satisfy user's requirement even carried out good design and planning.User's typing mistake, combination of enterprise and corporate environment in time passing and change, these all can influence the quality of institute's store data.Therefore, be necessary to represent the quality of data with metadata.The consistance of data (consistency), correctness (correctness), integrality (completeness) and four indexs of minimality (minimality) are used for the degree of declarative data.The data cleansing process is exactly to solve identical concept is had different method for expressing, when integrated a plurality of data source, eliminates problems such as pattern conflict and duplicated records.
Nowadays many this cleaning meanss and data cleansing technology is arranged on the market; But great majority are that data are placed on the external disk; In case and data pollute to some extent must whole table in addition the entire database data warehouse clean totally, so just cause cleaning efficiency very low.In order to overcome the drawback that traditional data is cleaned; The present invention has designed a kind of brand-new data cleansing method; Make full use of the fireballing advantage of internal storage data library inquiry, increase and deletion; The data that will operate are written into internal memory and set up MDB-tree index, solve problems such as buffer memory sensitivity and TLB inefficacy, improve the Data Matching rate.The present invention simultaneously uses the incremental data cleaning method, utilizes relational algebra operation to realize the incremental maintenance of node, thereby avoids full dose attended operation consuming time.Prior art involved in the present invention comprises:
1, memory database (MDB) index technology
Tradition is based on the relational database system (DRDB) of disk; Because MDL disk resident; Issued transaction often relates to the disk I operation; The optimization aim of its architecture Design is how to reduce the number of times of read-write disk, is difficult to satisfy the demand of following based on network application system to the high-performance data access ability.The data of memory database (MDB) forever reside in the internal memory.MDB has the access efficiency higher than DRDB when visit data.Along with DRAM price constantly reduces, memory size enlarges, and the increasing database of storage becomes feasible in internal memory.
The index structure that DRDB adopts mainly is the B/B+ tree, and its design object is exactly to reduce IO number of visit data in magnetic disk.And in MDB, having adopted a kind of new index structure T tree, its design object is to reduce memory cost and cpu instruction number.The T tree is that it is the binary tree that an a kind of node comprises a plurality of elements by Adelson-Velskii-Landis tree and the development of B tree.Owing to be binary tree; The T tree has kept the high-level efficiency of Adelson-Velskii-Landis tree binary chop; Simultaneously a T node comprises a plurality of elements, as the B tree, the full level of each node remain on half-full and full up between; Move usually and only need carry out inserting and delete caused data at an intranodal, reduced for the balance that keeps tree the rotary manipulation that must carry out.Because all in internal memory, it is right in a T tree node, need as B sets, not deposit N index key assignments-pointer for index and data, only need deposit the pointer that points to respective record corresponding field in the internal memory, the storage of variable-length field no longer is a problem in the index like this.But along with the understanding of processor cache effect is deepened gradually, how the data access technology hot research direction of MDB makes good use of processor cache in access process, make performance can obtain largest optimization.And the buffer memory utilization factor of T tree is very poor, has restricted the excavation of performance potential on the contrary.Thereby people have developed the CSS tree, the MDB-tree of using among the responsive index technology of buffer memorys such as CSB+ tree and the present invention in succession.
2, data increment maintenance technology
In to data depot data cleaning process, need two kinds of ETL processes: full dose ETL process and increment ETL process, full dose ETL process is used for the initialization of data warehouse, and increment ETL process then is used for constantly new data being written into data warehouse.And the method for using the Materialized View incremental maintenance can utilize full dose ETL process to give birth to increment ETL process from movable property.But Materialized View proposes in order to improve the data warehouse search efficiency at first, and incremental maintenance is for effective Materialized View that upgrades, thereby the research that has the Materialized View incremental maintenance concentrates on data warehouse environment.Data in the data warehouse are all pressed the theme tissue; Use polymerization, selection, projection, connection computing in the inquiry in a large number; And seldom use difference operation, but often using difference operation in the ETL process rejects unwanted data, this be with data warehouse environment under Materialized View safeguard different places.
The data increment maintenance technology is at first standardized to the relational algebra operation node in the ETL process; Respectively the relational algebra operation node after the standardization is carried out incremental maintenance then; Mainly consider in the present invention 6 kinds of relational algebra operations (polymerized alpha, select σ, projection π, connection
and U, poor-), standard be AUSPJ (polymerization, also, selection, projection with connect) combination of fragment and D (poor) fragment.Provide the incremental maintenance method of these two kinds of fragments simultaneously.
3, result set merger technology
About the merger of result set is one of the basis that can normally use of this method.When returning Query Result, in fact the result set with all a plurality of thread enquiry modules carries out merger, before all Query Results arrive the homophony module, need do elementary ordering.Arrive after the homophony module through merger the result set of each module is carried out a merger.
Summary of the invention
In order to overcome the above-mentioned shortcoming of prior art; The invention provides a kind of incremental data cleaning method, made full use of the big advantage of computer nowadays internal memory, data to be cleaned are loaded into internal memory based on memory mapping; Set up MDB-tree index structure efficiently; Improve data access speed greatly, utilize the increment cleaning way, improve data cleansing efficient.
The technical solution adopted for the present invention to solve the technical problems is: a kind of incremental data cleaning method based on memory mapping comprises the steps:
The first step, at first data are loaded into internal memory, make up memory database MDB-tree index structure, after having set up MDB-tree index structure, utilize the crash rate of crash rate and the TLB of secondary cache to set up the cost model this tree construction is assessed;
Second goes on foot, utilizes the MDB-tree to carry out inquiry, insertion and the deletion action of data;
The 3rd step, with memory database Materialized View standard turn to AUSPJ (polymerization, also, selection, projection with connect) fragment and D (poor) fragment, the incremental maintenance of employing relational algebra operation realization node;
The 4th step, employing result set conflation algorithm are realized the data cleansing operation.
In said evaluation process:
1) calculate the quantity that the cache of the leaf node of MDB-tree lost efficacy by following formula:
Wherein: n is available node number, and q is the number of data item to be matched, and h is the height of tree, n
cBe the quantity of the Cache Line of a node leap, u is the average utilization of node or barrel chain;
2) calculate the number that TLB lost efficacy by following formula:
In the query manipulation process of said data, the inter-node search algorithm is carried out as follows:
1) according to node pointer of importing and input key assignments; P.key in the node of this node pointed and input key assignments key are compared; Obtain the scope of key between this junction area; Judge that then this interval points to whether the pointer of child node is empty, if for empty then in the child node of this node, continue to search; If be sky then represent that this node is a leaf node;
2) use the hash function that key is done the hash computing and obtain the hash address; According to searching in the HASH table of hash address in leaf node; If do not find then search failure, if a plurality of index are arranged, then to search the key value successively in the barrel chain that is head pointer of this HASH address; If have then the index address that this key value is corresponding returns, if do not exist then search failure.
In the insertion of said data, deletion action process, the pairing leaf node of KEY that first inquiry will be inserted or delete is after leaf node is positioned; For inserting operation; Judge whether this node has enough spaces to hold new index entry, if insufficient space then divides leaf node; For deletion action, if delete after this index entry, there are not other index entries in this leaf node, then readjusts the MDB-tree.
Said employing relational algebra operation realizes that the incremental maintenance of node comprises the steps:
1) an ETL process specification that includes five kinds of fundamental relation computings and polymerization computing is turned to the combination of AUSPJ fragment and D fragment; The standardization result of ETL process is: a D fragment only comprises single difference operation; An AUSPJ fragment comprises five kinds of fundamental relation computings; And must satisfy one of following four kinds of situation: if a) the polymerization computing is arranged, then the polymerization computing must be in the top of AUSPJ fragment; B) the AUSPJ fragment is followed in D fragment back; C) the AUSPJ fragment is followed and is being connected the computing back, and union is then in the top of AUSPJ fragment; D) the AUSPJ fragment is a uppermost fragment in the ETL process;
2) two kinds of fragments are realized incremental maintenance:
Incremental maintenance to the AUSPJ fragment: must safeguard following three kinds of Materialized Views: all input relations that a) connect computing; B) output relation of project and union, and this relation will increase count attribute or allow to preserve repeating data; C) output relation of polymerization computing;
Incremental maintenance to the D fragment: its input relation must be a Materialized View, adopts SRA method or BRA method to realize.
Said BRA method is meant that for Materialized View C=A-B if satisfy one of following three kinds of situation, then C is from safeguarding:
A) A has only deletion increment and B to have only the insertion increment;
B) A has only deletion increment and B not to have increment;
C) A does not have increment and B to have only the insertion increment.
Said result set conflation algorithm is meant according to the result of each thread inquiry, and sub-result set is merged, and finally becomes complete Query Result.
Compared with prior art, good effect of the present invention is: the present invention all is written into internal memory with data and sets up memory database, sets up MDB-tree index structure, improves data search, deletion, insertion speed and efficient greatly.Help reducing data cleansing process design cost.It is big to can be applicable to data volume, industry and the field high to the data quality requirements.
Description of drawings
The present invention will explain through example and with reference to the mode of accompanying drawing, wherein:
Fig. 1 is a MDB-tree index structure;
Fig. 2 is a MDB-tree search index process flow diagram;
Fig. 3 is the definition figure of AUSPJ fragment;
Fig. 4 is the AUSPJ segment of incremental maintenance;
Fig. 5 is the parallel query merger process as a result synoptic diagram of 8 threads.
Embodiment
A kind of incremental data cleaning method based on memory mapping comprises the steps:
The first step, database all is written into internal memory, selects need carry out in the memory database table of index, the key assignments that needs the row of index in this table is set up MDB-set index structure:
At first be to set up similar B+ tree: set up rule according to the B+ tree key word is inserted in this tree, each node is not stored data, a storage key, and each intranodal key word is in order and comprise the pointer of number of keyword+1 a sensing child node.
All key words are after having set up the B+ tree; Leaf node is filled with the HASH table: the hash value of each key word being selected the hash function calculation key word of low calculated amount; Then key word and corresponding data are recorded in address in the internal memory and are put in the HASH table and go, just this hash result is stored in the barrel chain afterbody that starts with this hash address if the HASH address produces conflict.Set threshold values and will cause the leaf node division when the length of hash barrel chain surpasses.
As shown in Figure 1, MDB-tree index structure comprises two-layer, and ground floor is the same as common B+ tree, and it is the tree structure of a multilayer multiplexing.The second layer is each leaf node of this tree.Each leaf node is the mode through HASH all, data item key in the database is hashed in this HASH hash table leaf node go.
After having set up MDB-tree index structure, utilizing the crash rate of crash rate and the TLB of secondary cache to set up the cost model assesses this tree construction.
The quantity that the cache of the leaf node of in evaluation process, setting by following formula calculating MDB-lost efficacy:
Wherein: n is available node number, and q is the number of data item to be matched, and h is the height of tree, n
cBe the quantity of the Cache Line of a node leap.U is the average utilization of node or barrel chain.
Calculate the number that TLB lost efficacy by following formula:
Second goes on foot, utilizes the MDB-tree to carry out inquiry, insertion and the deletion action of data:
1) query manipulation of data: search algorithm is substantially similar with the B+ tree, is top-downly to begin to search according to KEY the path of child node from root node, in leaf node, searches index entry at last.Obtaining under the pointer situation of certain node, the inter-node search algorithm is carried out as follows:
A) according to node pointer of importing and input key assignments; P.key in the node of this node pointed and input key assignments key are compared; Obtain the scope of key between this junction area; Judge that then this interval points to whether the pointer of child node is empty, if for empty then in the child node of this node, continue to search; If be sky then represent that this node is a leaf node;
B) use the hash function that key is done the hash computing and obtain the hash address; According to searching in the HASH table of hash address in leaf node, if do not find then search failure, if a plurality of index are arranged; Then to search the key value successively in the barrel chain that is head pointer of this HASH address; If have then the index address that this key value is corresponding returns, if do not exist then search failure, it is as shown in Figure 2 to search flow process;
2) insertion of data, deletion action:
The pairing leaf node of KEY that needs inquiry to insert or to delete after leaf node is positioned, for inserting operation, needs to judge whether this node has enough spaces to hold new index entry, if insufficient space needs the division leaf node; For deletion action,, need readjust the MDB-tree so if there are not other index entries in this leaf node after needing to judge this index entry of deletion.
The 3rd step, with memory database Materialized View standard turn to AUSPJ (polymerization, also, selection, projection with connect) fragment and D (poor) fragment, the incremental maintenance of employing relational algebra operation realization node:
Adopt relational algebra operation to realize that the incremental maintenance method of node comprises the steps:
1) an ETL process specification that includes 5 kinds of fundamental relation computings (select σ, projection π, connection
and U, poor-) and polymerization computing α is turned to the combination of two kinds of fragments (AUSPJ fragment and D fragment); The standardization result of ETL process is: a D fragment only comprises single difference operation; An AUSPJ fragment comprises other 5 kinds of computings; And with
series arrangement; An AUSPJ fragment might not comprise all 5 kinds of computings, but must satisfy one of following four kinds of situation:
If a) polymerization computing α is arranged, then polymerization computing α must be in the top of AUSPJ fragment;
B) the AUSPJ fragment is followed in D fragment back;
C) the AUSPJ fragment is followed and is being connected computing
at the back, and union U is then in the top of AUSPJ fragment;
D) the AUSPJ fragment is a uppermost fragment in the ETL process, promptly on it, has no other fragments.
This normalization method is based on following equality:
With moving on the U: π
A(R ∪ S)=π
A(R) ∪ π
A(S), σ
C(R ∪ S)=σ
C(R) ∪ σ
C(S)
With moving on the π:
With moving on the σ:
It should be noted that; The equality of AUSPJ fragment of being used for standardizing does not use equality
with moving on the U; This is will be divided into 2 or more because handle the single connection computing in back like this, can reduce execution efficient.
2) two kinds of fragments are realized incremental maintenance:
Incremental maintenance to the AUSPJ fragment: must safeguard following three kinds of Materialized Views: all input relations that a) connect computing; B) output relation of project and union, and this relation will increase count attribute or allow to preserve repeating data; C) output relation of polymerization computing.For the ease of understanding, explain with an example.
Consider the complete AUSPJ fragment of a shape like
; Wherein G representes the packet attributes set; B representes to participate in the community set of polymerization computing; And Ri has increment Delta Ri and
(1≤i≤3) arbitrarily, and the definition of then whole AUSPJ fragment is as shown in Figure 3.
Introduce the incremental maintenance method that how to realize below:
At first be to connect computing
According to following formula, obtain input and concern R
1, R
2Need materialization:
Selection computing σ afterwards
C, according to following formula, do not need any relation of materialization:
ΔR
0＝σ
C(ΔR
1)
Concern Rpu after the public same increase count attribute of the project π of followed and union U.And before the polymerization computing α that Rpu is sent into the back, need a project π to remove count attribute; And because the usage count attribute has guaranteed that the tuple in the relation after the projection is inevitable unique; According to following formula, do not need any relation of materialization, directly output data is sent into polymerization computing α:
ΔR
0＝π
A(ΔR
1)
The output relation of polymerization computing α needs materialization, and if polymerization computing α asks maximum, minimum value, then need to import to concern materialization.The AUSPJ segment that is used for incremental maintenance is as shown in Figure 4.
Incremental maintenance to the D fragment: its input relation must be a Materialized View, generally adopts SRA (split relation access) method or BRA (base relation access) method to realize; The BRA method refers to for Materialized View C=A-B, if satisfy one of following three kinds of situation, then C is from safeguarding:
A) A has only deletion increment and B to have only the insertion increment;
B) A has only deletion increment and B not to have increment;
C) A does not have increment and B to have only the insertion increment.
The 4th step, employing result set conflation algorithm are realized the data cleansing operation:
Result according to each thread inquiry merges sub-result set, finally becomes complete Query Result.About the merger of result set is one of the basis that can normally use of this method.When returning Query Result, in fact the result set with all a plurality of thread enquiry modules carries out merger, before all Query Results arrive the homophony module, need do elementary ordering.Arrive after the homophony module through merger the result set of each module is carried out a merger.Fig. 5 is the parallel query merger process as a result synoptic diagram of 8 thread inquiries.
Claims (7)
1. the incremental data cleaning method based on memory mapping is characterized in that: comprise the steps:
The first step, at first data are loaded into internal memory, make up memory database MDB-tree index structure, after having set up MDB-tree index structure, utilize the crash rate of crash rate and the TLB of secondary cache to set up the cost model this tree construction is assessed.
Second goes on foot, utilizes the MDB-tree to carry out inquiry, insertion and the deletion action of data;
The 3rd step, with memory database Materialized View standard turn to AUSPJ (polymerization, also, selection, projection with connect) fragment and D (poor) fragment, the incremental maintenance of employing relational algebra operation realization node;
The 4th step, employing result set conflation algorithm are realized the data cleansing operation.
2. the incremental data cleaning method based on memory mapping according to claim 1 is characterized in that: in said evaluation process:
1) calculate the quantity that the cache of the leaf node of MDB-tree lost efficacy by following formula:
Wherein: n is available node number, and q is the number of data item to be matched, and h is the height of tree, n
cBe the quantity of the Cache Line of a node leap, u is the average utilization of node or barrel chain;
2) calculate the number that TLB lost efficacy by following formula:
3. the incremental data cleaning method based on memory mapping according to claim 1 is characterized in that: in the query manipulation process of said data, the inter-node search algorithm is carried out as follows:
1) according to node pointer of importing and input key assignments; P.key in the node of this node pointed and input key assignments key are compared; Obtain the scope of key between this junction area; Judge that then this interval points to whether the pointer of child node is empty, if for empty then in the child node of this node, continue to search; If be sky then represent that this node is a leaf node;
2) use the hash function that key is done the hash computing and obtain the hash address; According to searching in the HASH table of hash address in leaf node; If do not find then search failure, if a plurality of index are arranged, then to search the key value successively in the barrel chain that is head pointer of this HASH address; If have then the index address that this key value is corresponding returns, if do not exist then search failure.
4. the incremental data cleaning method based on memory mapping according to claim 1; It is characterized in that: in the insertion of said data, deletion action process, the pairing leaf node of KEY that first inquiry will be inserted or delete is after leaf node is positioned; For inserting operation; Judge whether this node has enough spaces to hold new index entry, if insufficient space then divides leaf node; For deletion action, if delete after this index entry, there are not other index entries in this leaf node, then readjusts the MDB-tree.
5. the incremental data cleaning method based on memory mapping according to claim 1 is characterized in that: said employing relational algebra operation realizes that the incremental maintenance of node comprises the steps:
1) an ETL process specification that includes five kinds of fundamental relation computings and polymerization computing is turned to the combination of AUSPJ fragment and D fragment; The standardization result of ETL process is: a D fragment only comprises single difference operation; An AUSPJ fragment comprises five kinds of fundamental relation computings; And must satisfy one of following four kinds of situation: if a) the polymerization computing is arranged, then the polymerization computing must be in the top of AUSPJ fragment; B) the AUSPJ fragment is followed in D fragment back; C) the AUSPJ fragment is followed and is being connected the computing back, and union is then in the top of AUSPJ fragment; D) the AUSPJ fragment is a uppermost fragment in the ETL process;
2) two kinds of fragments are realized incremental maintenance:
Incremental maintenance to the AUSPJ fragment: must safeguard following three kinds of Materialized Views: all input relations that a) connect computing; B) output relation of project and union, and this relation will increase count attribute or allow to preserve repeating data; C) output relation of polymerization computing;
Incremental maintenance to the D fragment: its input relation must be a Materialized View, adopts SRA method or BRA method to realize.
6. the incremental data cleaning method based on memory mapping according to claim 5 is characterized in that: said BRA method is meant that for Materialized View C=A-B if satisfy one of following three kinds of situation, then C is from safeguarding:
A) A has only deletion increment and B to have only the insertion increment;
B) A has only deletion increment and B not to have increment;
C) A does not have increment and B to have only the insertion increment.
7. the incremental data cleaning method based on memory mapping according to claim 1 is characterized in that: said result set conflation algorithm is meant according to the result of each thread inquiry, and sub-result set is merged, and finally becomes complete Query Result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103169145A CN102495834A (en) | 2011-10-18 | 2011-10-18 | Incremental data cleaning method based on memory mapping |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103169145A CN102495834A (en) | 2011-10-18 | 2011-10-18 | Incremental data cleaning method based on memory mapping |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102495834A true CN102495834A (en) | 2012-06-13 |
Family
ID=46187659
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2011103169145A Pending CN102495834A (en) | 2011-10-18 | 2011-10-18 | Incremental data cleaning method based on memory mapping |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102495834A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104090954A (en) * | 2014-07-04 | 2014-10-08 | 用友软件股份有限公司 | Connecting method and system of read-only tables |
| CN105488174A (en) * | 2015-12-01 | 2016-04-13 | 北京浩瀚深度信息技术股份有限公司 | Multi-distribution-type data source management method and system based on BS (Browser/Server) structure |
| CN106339270A (en) * | 2016-08-26 | 2017-01-18 | 华为技术有限公司 | Method and device for verifying data |
| CN106844790A (en) * | 2017-03-22 | 2017-06-13 | 中国人民解放军国防科学技术大学 | A kind of quick Temporal query method for user's group analysis in social networks |
| CN107729365A (en) * | 2017-09-07 | 2018-02-23 | 中国科学院自动化研究所 | Embodied spillover and optimization method and device |
| CN110019221A (en) * | 2017-12-18 | 2019-07-16 | 厦门本能管家科技有限公司 | A kind of memory mapping type Database Systems |
| CN110019477A (en) * | 2017-12-27 | 2019-07-16 | 航天信息股份有限公司 | A kind of method and system carrying out big data processing using HIVE backup table |
| CN113641685A (en) * | 2021-10-18 | 2021-11-12 | 中国民用航空总局第二研究所 | Data processing system for guiding aircraft |
| CN114943021A (en) * | 2022-07-20 | 2022-08-26 | 之江实验室 | A TB-level incremental data screening method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101102577A (en) * | 2006-07-07 | 2008-01-09 | 中兴通讯股份有限公司 | Incremental synchronization method for data in tables of frontground and background database of wireless communication base station system |
| CN101242356A (en) * | 2007-12-06 | 2008-08-13 | 中兴通讯股份有限公司 | Realization method and IPTV system for memory database in IPTV system |
| US20100115332A1 (en) * | 2008-11-06 | 2010-05-06 | Weimin Zheng | Virtual machine-based on-demand parallel disaster recovery system and the method thereof |
-
2011
- 2011-10-18 CN CN2011103169145A patent/CN102495834A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101102577A (en) * | 2006-07-07 | 2008-01-09 | 中兴通讯股份有限公司 | Incremental synchronization method for data in tables of frontground and background database of wireless communication base station system |
| CN101242356A (en) * | 2007-12-06 | 2008-08-13 | 中兴通讯股份有限公司 | Realization method and IPTV system for memory database in IPTV system |
| US20100115332A1 (en) * | 2008-11-06 | 2010-05-06 | Weimin Zheng | Virtual machine-based on-demand parallel disaster recovery system and the method thereof |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104090954B (en) * | 2014-07-04 | 2019-02-05 | 用友网络科技股份有限公司 | The connection method of meter reading and the connection system of meter reading |
| CN104090954A (en) * | 2014-07-04 | 2014-10-08 | 用友软件股份有限公司 | Connecting method and system of read-only tables |
| CN105488174A (en) * | 2015-12-01 | 2016-04-13 | 北京浩瀚深度信息技术股份有限公司 | Multi-distribution-type data source management method and system based on BS (Browser/Server) structure |
| CN105488174B (en) * | 2015-12-01 | 2018-05-04 | 北京浩瀚深度信息技术股份有限公司 | More distributed data power supply management methods and system based on BS structures |
| CN106339270A (en) * | 2016-08-26 | 2017-01-18 | 华为技术有限公司 | Method and device for verifying data |
| CN106844790A (en) * | 2017-03-22 | 2017-06-13 | 中国人民解放军国防科学技术大学 | A kind of quick Temporal query method for user's group analysis in social networks |
| CN106844790B (en) * | 2017-03-22 | 2020-03-31 | 中国人民解放军国防科学技术大学 | Rapid temporal query method for user group analysis in social network |
| CN107729365A (en) * | 2017-09-07 | 2018-02-23 | 中国科学院自动化研究所 | Embodied spillover and optimization method and device |
| CN107729365B (en) * | 2017-09-07 | 2020-02-07 | 中国科学院自动化研究所 | Materialized view selection and optimization method and device |
| CN110019221B (en) * | 2017-12-18 | 2022-07-19 | 本无链科技(深圳)有限公司 | Memory mapping type database system |
| CN110019221A (en) * | 2017-12-18 | 2019-07-16 | 厦门本能管家科技有限公司 | A kind of memory mapping type Database Systems |
| CN110019477A (en) * | 2017-12-27 | 2019-07-16 | 航天信息股份有限公司 | A kind of method and system carrying out big data processing using HIVE backup table |
| CN113641685B (en) * | 2021-10-18 | 2022-04-08 | 中国民用航空总局第二研究所 | Data processing system for guiding aircraft |
| CN113641685A (en) * | 2021-10-18 | 2021-11-12 | 中国民用航空总局第二研究所 | Data processing system for guiding aircraft |
| CN114943021A (en) * | 2022-07-20 | 2022-08-26 | 之江实验室 | A TB-level incremental data screening method and device |
| US11789639B1 (en) | 2022-07-20 | 2023-10-17 | Zhejiang Lab | Method and apparatus for screening TB-scale incremental data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102495834A (en) | Incremental data cleaning method based on memory mapping | |
| CN101916261B (en) | Data partitioning method for distributed parallel database system | |
| Bonnet et al. | Reduce, you say: What nosql can do for data aggregation and bi in large repositories | |
| US10846278B2 (en) | Dynamic updates to a semantic database using fine-grain locking | |
| Song et al. | HaoLap: A Hadoop based OLAP system for big data | |
| CN103366015B (en) | A kind of OLAP data based on Hadoop stores and querying method | |
| CN103425772A (en) | Method for searching massive data with multi-dimensional information | |
| CN103678589A (en) | Database kernel query optimization method based on equivalence class | |
| CN102411632B (en) | Chain table-based memory database page type storage method | |
| Ptiček et al. | Big data and new data warehousing approaches | |
| Cahsai et al. | Scaling k-nearest neighbours queries (the right way) | |
| CN113485638B (en) | Access optimization system for massive astronomical data | |
| Bhagat et al. | Comparative study of row and column oriented database | |
| Pivert | NoSQL data models: trends and challenges | |
| Safaee et al. | A distributed B+ Tree indexing method for processing range queries over streaming data | |
| Jürgens | Index structures for data warehouses | |
| Imran et al. | NODW Framework for Data Warehousing-A NoSQL Big Data Perspective | |
| Zhang et al. | Improving performance by creating a native join-index for OLAP | |
| Han et al. | PI-Join: Efficiently processing join queries on massive data | |
| Chen et al. | Large vector spatial data storage and query processing using Clickhouse | |
| Damasceno et al. | Tinycubes: A modular technology for interactive visual analysis of historical and continuously updated spatiotemporal data | |
| Zaboli | Parallel Olap on Multi/Many-Core and Cloud Platforms | |
| Stolze et al. | Beyond Big Data—The Ocient Hyperscale Data Warehouse | |
| Dourhri et al. | Methodology for modeling a column-oriented database with Cassandra | |
| KATRAWI et al. | Enhanced Late-Straggler Algorithm With On-Demand Etl For Big Data Retrieval |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent of invention or patent application | ||
| CB02 | Change of applicant information |
Address after: 610054 information industry building, 159 East Ring Road, Chengdu, Chenghua District, Sichuan Applicant after: Chengdu Comsys Information Technology Co., Ltd. Address before: 610054 information industry building, 159 East Ring Road, Chengdu, Chenghua District, Sichuan Applicant before: Uestc Comsys Information Co., Ltd. |
|
| CB03 | Change of inventor or designer information |
Inventor after: Tang Xuefei Inventor after: Tang Xuefeng Inventor after: Chen Ke Inventor after: Wang Hailiang Inventor before: Tang Xuefei Inventor before: Chen Ke Inventor before: Wang Hailiang Inventor before: Li Yinghong |
|
| COR | Change of bibliographic data |
Free format text: CORRECT: APPLICANT; FROM: CHENGDU KANGSAI INFORMATION TECHNOLOGY CO., LTD. OF UESTC TO: CHENGDU COMSYS INFORMATION TECHNOLOGY CO., LTD. Free format text: CORRECT: INVENTOR; FROM: TANG XUEFEI CHEN KE WANG HAILIANG LI YINGHONG TO: TANG XUEFEI TANG XUEFENGCHEN KE WANG HAILIANG |
|
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120613 |