CN106293960B - A kind of method and system for realizing data conversion based on strom and memory grid - Google Patents
A kind of method and system for realizing data conversion based on strom and memory grid Download PDFInfo
- Publication number
- CN106293960B CN106293960B CN201610599068.5A CN201610599068A CN106293960B CN 106293960 B CN106293960 B CN 106293960B CN 201610599068 A CN201610599068 A CN 201610599068A CN 106293960 B CN106293960 B CN 106293960B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- memory grid
- processing
- transmitting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention provides a kind of method for realizing data conversion based on strom and memory grid, a plurality of transmitting nodes, processing node, memory grid node and a preheating that storm need to be provided and more new node, and each processing node corresponds to a transmitting node and a memory grid node, which comprises acquisition data are into message queue；One processing node is bound with a memory grid node, realizes shared memory space and synchronous averaging, closing；After preheating receives load request of data with more new node, all processing nodes is called to obtain the first data deposit memory grid node to be matched from associated data source；Transmitting node reads the second data to be matched in message queue to processing node；Processing node obtains the first data from memory grid node again and is matched, and the data after matching are sent to corresponding output node.The present invention also provides a kind of systems for realizing data conversion based on strom and memory grid, improve data-handling efficiency.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to one kind to realize that data turn based on strom and memory grid
The method and system changed.
Background technique
As data to be treated in the unit time are more and more, be treated as in real time many mechanisms face pair it is primary
Challenge, and storm is as a kind of low delay, stream calculation frame highly fault tolerant is widely used in more and more projects and software
In system, strom reads data from message queue, is sent to processing node one by one by transmitting node and carries out data conversion behaviour
Make, data transformation operations mainly carry out corresponding operating according to preset business, operation, the warp such as generally convert or abandon
Data after crossing data transformation operations are retransmited to output node.When storm cooperates other traditional matched data sources (such as redis)
When carrying out data transformation operations, since storm processing speed is fast, when carrying out Data Matching, traditional data in data transformation operations
Source is difficult to meet the high speed processing ability of storm low delay, will cause the excessively high then initiation data product of storm processing node delay
The problem of pressure, and when storm cooperates other traditional matched data sources (such as redis) to carry out data conversions, the change of matched data
It is more very complicated, substantially reduce data-handling efficiency.
Summary of the invention
One of the technical problem to be solved in the present invention is to provide a kind of based on strom and memory grid realization data turn
The method changed reduces data and overstocks, improves data-handling efficiency.
One of the technical problem to be solved in the present invention is achieved in that a kind of based on strom and memory grid realization number
According to the method for conversion, need to provide a plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes of storm with
And a preheating and more new node, and an each processing node correspondence one transmitting node and memory grid node, the side
Method includes the following steps:
Step 10, acquisition data, and collected data are entered into message team in the form of data flow in order one by one
In column；
Step 20 binds each processing node and a memory grid node, the processing node after binding with it is interior
Grid node shared memory space is deposited, and realizes synchronous averaging and close synchronously；
Step 30, before carrying out conversion operation, after the preheating receives load request of data with more new node, call all
Processing node obtains the first data to be matched from associated data source, and first data are stored in the memory grid respectively bound
In node；
Step 40 reads the second data to be matched by the transmitting node from message queue, and with data flow
Form is emitted to processing node one by one；
It is obtained in step 50, the memory grid node bound by the second data that the processing node will receive with itself
The first data got are matched, and the data that matching is completed are sent to corresponding output node and are further processed.
Further, the method also includes step 60, when the associated data source change need to update when, pause
Storm topology, transmitting node stop transmitting data, after waiting pending datas to update, then step 30 are executed, by updated first data
After being stored in memory grid node, service topology is restarted, completes online updating operation.
Further, " in the memory grid node for respectively binding first data deposit " tool in the step 30
Body is, the first data that each processing node will acquire are stored in the memory grid node of itself binding, and with
HashMap<key, value>form stored.
Further, the step 50 specifically: by it is described processing node receive transmitting node transmitting come second
After data, handled one by one, obtained from the memory grid node of the processing node binding HashMap of storage <
Key, value > format data are matched, and matched data are sent to corresponding output node one by one and are carried out in next step
Processing.
Further, the associated data source includes structured document, relevant database, non-relational database or disappears
Cease queue.
The second technical problem to be solved by the present invention is to provide a kind of based on strom and memory grid realization data turn
The system changed reduces data and overstocks, improves data-handling efficiency.
The second technical problem to be solved by the present invention is achieved in that a kind of based on strom and memory grid realization number
According to the system of conversion, need to provide a plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes of storm with
And a preheating and more new node, and an each processing node correspondence one transmitting node and memory grid node, the system
System includes data acquisition module, node binding module, data acquisition module, data transmission module and data conversion module:
The data acquisition module, for acquiring data, and by collected data one by one in order with the shape of data flow
Formula is entered into message queue；
The node binding module, for binding each processing node and a memory grid node, after binding
Processing node and memory grid nodes sharing memory headroom, and realize synchronous averaging and close synchronously；
The data acquisition module, for before carrying out conversion operation, the preheating to receive load data with more new node
After request, all processing nodes is called to obtain the first data to be matched from associated data source, first data is stored in each
From in the memory grid node of binding；
The data transmission module, for reading the second number to be matched from message queue by the transmitting node
According to, and it is emitted to processing node one by one in the form of data flow；
The data conversion module, the second data for will be received from data transmission module by the processing node
It is matched with the first data for getting in the memory grid node of itself binding, and data that matching is completed is sent to pair
The output node answered is further processed.
Further, the system also includes an online updating modules, need for changing in the associated data source
When updating, pause storm topology, transmitting node stops transmitting data, after waiting pending datas to update, then executes data acquisition mould
Block after updated first data are stored in memory grid node, restarts service topology, completes online updating operation.
Further, " first data are stored in the memory grid section respectively bound in the data acquisition module
In point " specifically, the first data that each processing node will acquire are stored in the memory grid node of itself binding, and
With HashMap<key, value>form stored.
Further, the data conversion module specifically: for receiving transmitting node hair by the processing node
It after penetrating the second data come, is handled one by one, obtains storage from the memory grid node of the processing node binding
HashMap<key, value>format data are matched, and by matched data be sent to one by one corresponding output node into
Row is handled in next step.
Further, the associated data source includes structured document, relevant database, non-relational database or disappears
Cease queue.
The present invention has the advantage that
1, storm and memory grid are bound, and associated data can be stored in memory grid in advance, reduced because from outside
The time of associated data waste is obtained, the delay of Data Matching operation is reduced, data is not will cause and overstocks, improves data processing effect
Rate；
2, the feature fast using memory grid node read or write speed, gives full play to the high speed processing ability of storm low delay,
Improve data conversion efficiency；
3, the present invention utilizes the high fault tolerance of storm, guarantees the high availability of storm cluster and memory grid cluster；
4, the present invention will need matched data to be stored in memory grid, can lead in not traffic affecting
It crosses preheating and more new node carries out online updating.
Detailed description of the invention
The present invention is further illustrated in conjunction with the embodiments with reference to the accompanying drawings.
Fig. 1 is a kind of method flow diagram that data conversion is realized based on strom and memory grid of the present invention.
Fig. 2 is a kind of system logic block diagram that data conversion is realized based on strom and memory grid of the present invention.
Fig. 3 is a kind of schematic illustration that data conversion is realized based on strom and memory grid of the present invention.
Specific embodiment
As shown in figures 1 and 3, a method of data conversion is realized based on strom and memory grid, storm need to be provided
A plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and it is every
The one corresponding transmitting node of processing node and a memory grid node, described method includes following steps:
Step 10, acquisition data, and collected data are entered into message team in the form of data flow in order one by one
In column；
Step 20 binds each processing node and a memory grid node, the processing node after binding with it is interior
Grid node shared memory space is deposited, and realizes synchronous averaging and close synchronously；
Step 30, before carrying out conversion operation, after the preheating receives load request of data with more new node, call all
Processing node obtains the first data to be matched from associated data source, and first data are stored in the memory grid respectively bound
In node, the associated data source includes structured document, relevant database, non-relational database or message queue, institute
It states " in the memory grid node for respectively binding first data deposit " specifically, each processing node will acquire
The first data be stored in the memory grid node of itself binding, and with HashMap<key, value>form stored；
Step 40 reads the second data to be matched by the transmitting node from message queue, and with data flow
Form is emitted to processing node one by one；
It is obtained in step 50, the memory grid node bound by the second data that the processing node will receive with itself
The first data got are matched, and the data that matching is completed are sent to corresponding output node and are further processed,
The step 50 specifically: by it is described processing node receive transmitting node transmitting come the second data after, located one by one
Reason obtains HashMap<key of storage, value>format data from the memory grid node of the processing node binding
It is matched, and matched data is sent to corresponding output node one by one and are further processed.
The method also includes step 60, when the associated data source change need to update when, pause storm open up
It flutters, transmitting node stops transmitting data, after waiting pending datas to update, then executes step 30, will be in the deposit of updated first data
After depositing grid node, service topology is restarted, completes online updating operation.
As shown in Figures 2 and 3, a kind of system for realizing data conversion based on strom and memory grid, need to provide storm
A plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and it is every
The one corresponding transmitting node of processing node and a memory grid node, the system comprises data acquisition module, nodes to tie up
Cover half block, data acquisition module, data transmission module and data conversion module:
The data acquisition module, for acquiring data, and by collected data one by one in order with the shape of data flow
Formula is entered into message queue；
The node binding module, for binding each processing node and a memory grid node, after binding
Processing node and memory grid nodes sharing memory headroom, and realize synchronous averaging and close synchronously；
The data acquisition module, for before carrying out conversion operation, the preheating to receive load data with more new node
After request, all processing nodes is called to obtain the first data to be matched from associated data source, first data is stored in each
It is described specific " in the memory grid node for respectively binding first data deposit " from the memory grid node of binding
For the first data that each processing node will acquire are stored in the memory grid node of itself binding, and with HashMap
<key, value>form stored；
The data transmission module, for reading the second number to be matched from message queue by the transmitting node
According to, and it is emitted to processing node one by one in the form of data flow；
The data conversion module, the second data for will be received from data transmission module by the processing node
It is matched with the first data for getting in the memory grid node of itself binding, and data that matching is completed is sent to pair
The output node answered is further processed；The data conversion module specifically: for being received by the processing node
It after the second data that transmitting node transmitting comes, is handled one by one, from the memory grid node of the processing node binding
HashMap<key of storage is obtained, value>format data are matched, and matched data are sent to one by one corresponding
Output node is further processed.
The system also includes an online updating module, for the associated data source change need to update when,
Suspend storm topology, transmitting node stops transmitting data, after waiting pending datas to update, then data acquisition module executed, after update
The first data deposit memory grid node after, restart service topology, complete online updating operation.
Below with reference to a specific embodiment, the present invention will be further described:
It include a plurality of transmitting nodes, a plurality of processing nodes, a plurality of memory networks in storm cluster referring again to Fig. 3
Lattice node, a plurality of output nodes and a preheating and more new node, and each processing node corresponding transmitting node, one
A memory grid node and an output node；
Before data transformation operations, the second data to be converted are stored in message queue in the form of data flow in order one by one
In, it is deposited in message queue, waits the data in transmitting node transmitting queue；Each processing node and a memory grid node
It is bound, is shared to realize with block memory headroom, while starting, close and restarting；Preheating receives load with more new node
After matched data request, all processing node is called to obtain associated data source (such as structured document, relevant database, non-
Relevant database or message queue etc.) in the first data to be matched be deposited into the memory grid node with oneself binding,
And with HashMap<key, value>form storage；
When carrying out conversion operation, transmitting node is read from message queue needs matched second data, one by one with data
The form of stream is emitted to processing node, after the second data flow into processing node, handles node from the memory network bound with itself
According to HashMap<key of storage in lattice, value>formatted data is matched, and obtains value result according to predetermined
The business of different demands is completed the operation such as to convert or abandon, and the data for matching completion flow into corresponding output node one by one and carry out down
The processing of one step.
When occur matched data variation need to update when, pause storm topology, transmitting node stop transmitting data, wait
After data update, transmitting node is read from message queue again needs matched data to give processing node, preheating and more new node
Updated first data deposit memory grid node is obtained again, restarts service topology, completes online updating operation.
Associated data is stored in memory grid section in advance by binding storm and memory grid node by the present invention
In point, node to be processed is waited to call directly, reduced because reducing Data Matching operation from the external time for obtaining associated data and wasting
Delay, not will cause data overstock, improve data-handling efficiency；The high fault tolerance for utilizing storm simultaneously, guarantees storm collection
The high availability of group and memory grid cluster；Furthermore it is possible to matched data will be needed to be stored in memory grid, so that not
In the case of traffic affecting, online updating is carried out by preheating and more new node.
Although specific embodiments of the present invention have been described above, those familiar with the art should be managed
Solution, we are merely exemplary described specific embodiment, rather than for the restriction to the scope of the present invention, it is familiar with this
The technical staff in field should be covered of the invention according to modification and variation equivalent made by spirit of the invention
In scope of the claimed protection.
Claims (8)
1. a kind of method for realizing data conversion based on strom and memory grid, it is characterised in that: the plural number of storm need to be provided
A transmitting node, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and each processing
The corresponding transmitting node of node and a memory grid node, described method includes following steps:
Step 10, acquisition data, and collected data are entered into message queue in the form of data flow in order one by one；
Step 20 binds each processing node and a memory grid node, processing node and memory network after binding
Lattice nodes sharing memory headroom, and realize synchronous averaging and close synchronously；
Step 30, before carrying out conversion operation, it is described preheating with more new node receive load request of data after, call all processing
Node obtains the first data to be matched from associated data source, and first data are stored in the memory grid node respectively bound
In；
Step 40 reads the second data to be matched by the transmitting node from message queue, and in the form of data flow
It is emitted to processing node one by one；
It is got in step 50, the memory grid node bound by the second data that the processing node will receive with itself
The first data matched, and the data that matching is completed are sent to corresponding output node and are further processed；
The step 50 specifically: by it is described processing node receive transmitting node transmitting come the second data after, one by one into
Row processing obtains HashMap<key of storage, value>format from the memory grid node of the processing node binding
Data are matched, and matched data are sent to corresponding output node one by one and are further processed.
2. a kind of method for realizing data conversion based on strom and memory grid according to claim 1, feature exist
In: the method also includes step 60, when the associated data source change need to update when, pause storm topology, transmitting
Node stops transmitting data, after waiting pending datas to update, then executes step 30, and updated first data are stored in memory grid section
After point, service topology is restarted, completes online updating operation.
3. a kind of method for realizing data conversion based on strom and memory grid according to claim 1, feature exist
In: " in the memory grid node for respectively binding first data deposit " in the step 30 is specifically, each described
The first data that will acquire of processing node are stored in the memory grid node of itself binding, and with HashMap<key, and value>
Form stored.
4. a kind of method for realizing data conversion based on strom and memory grid according to claim 1, feature exist
In: the associated data source includes structured document, relevant database, non-relational database or message queue.
5. a kind of system for realizing data conversion based on strom and memory grid, it is characterised in that: the plural number of storm need to be provided
A transmitting node, a plurality of processing nodes, a plurality of memory grid nodes and a preheating and more new node, and each processing
The corresponding transmitting node of node and a memory grid node, the system comprises data acquisition module, node binding module,
Data acquisition module, data transmission module and data conversion module:
The data acquisition module is recorded in the form of data flow in order one by one for acquiring data, and by collected data
Enter into message queue；
The node binding module, for each processing node and a memory grid node to be bound, the place after binding
Node and memory grid nodes sharing memory headroom are managed, and realizes synchronous averaging and close synchronously；
The data acquisition module, for before carrying out conversion operation, the preheating to receive load request of data with more new node
Afterwards, it calls all processing nodes to obtain the first data to be matched from associated data source, first data deposit is respectively tied up
In fixed memory grid node；
The data transmission module, for the second data to be matched to be read from message queue by the transmitting node, and
It is emitted to processing node one by one in the form of data flow；
The data conversion module, for by second data that will be received from data transmission module of processing node with from
The first data for getting are matched in the memory grid node of body binding, and will data that matching is completed be sent to it is corresponding
Output node is further processed；
The data conversion module specifically: for by it is described processing node receive transmitting node transmitting come the second data
Afterwards, it is handled one by one, HashMap < key of storage is obtained from the memory grid node of the processing node binding,
Value > format data are matched, and matched data are sent to corresponding output node one by one and are carried out at next step
Reason.
6. a kind of system for realizing data conversion based on strom and memory grid according to claim 5, feature exist
In: the system also includes an online updating module, for the associated data source change need to update when, pause
Storm topology, transmitting node stop transmitting data and then execute data acquisition module after waiting pending datas to update, by updated the
After one data are stored in memory grid node, service topology is restarted, completes online updating operation.
7. a kind of system for realizing data conversion based on strom and memory grid according to claim 5, feature exist
In: " in the memory grid node for respectively binding first data deposit " in the data acquisition module is specifically, every
One first data that will acquire of processing node are stored in the memory grid node of itself binding, and with HashMap < key,
Value > form stored.
8. a kind of system for realizing data conversion based on strom and memory grid according to claim 5, feature exist
In: the associated data source includes structured document, relevant database, non-relational database or message queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610599068.5A CN106293960B (en) | 2016-07-27 | 2016-07-27 | A kind of method and system for realizing data conversion based on strom and memory grid |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610599068.5A CN106293960B (en) | 2016-07-27 | 2016-07-27 | A kind of method and system for realizing data conversion based on strom and memory grid |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106293960A CN106293960A (en) | 2017-01-04 |
| CN106293960B true CN106293960B (en) | 2019-10-25 |
Family
ID=57652542
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610599068.5A Active CN106293960B (en) | 2016-07-27 | 2016-07-27 | A kind of method and system for realizing data conversion based on strom and memory grid |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106293960B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897131B (en) * | 2017-02-22 | 2020-05-29 | 浪潮(北京)电子信息产业有限公司 | Parallel computing method and device for astronomical software Griding |
| CN111931066B (en) * | 2020-09-11 | 2021-09-07 | 四川新网银行股份有限公司 | Real-time recommendation system design method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103201740A (en) * | 2010-11-15 | 2013-07-10 | 萨思学会有限公司 | Grid computing system alongside a distributed database architecture |
| CN103473318A (en) * | 2013-09-12 | 2013-12-25 | 中国科学院软件研究所 | Distributed transaction security method for memory data grid |
| CN103488699A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Data processing device and method based on stored data grids |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2416878B (en) * | 2004-08-06 | 2008-05-14 | Univ Surrey | Resource management in grid computing |
-
2016
- 2016-07-27 CN CN201610599068.5A patent/CN106293960B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103201740A (en) * | 2010-11-15 | 2013-07-10 | 萨思学会有限公司 | Grid computing system alongside a distributed database architecture |
| CN103488699A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Data processing device and method based on stored data grids |
| CN103473318A (en) * | 2013-09-12 | 2013-12-25 | 中国科学院软件研究所 | Distributed transaction security method for memory data grid |
Non-Patent Citations (1)
| Title |
|---|
| 内存网格调度方法分析;卢俊文 等;《厦门理工学院学报》;20150630;第23卷(第3期);第75-79页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106293960A (en) | 2017-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102169500B (en) | Dynamic service flow display method and device | |
| CN109753531A (en) | A kind of big data statistical method, system, computer equipment and storage medium | |
| CN107391719A (en) | Distributed stream data processing method and system in a kind of cloud environment | |
| CN112148455B (en) | Task processing method, device and medium | |
| CN105577446B (en) | Lightweight embedded network management system and method | |
| CN105190543A (en) | Reachability-based coordination for cyclic dataflow | |
| CN105721543A (en) | Cross-protocol distribution-type cloud storage system based on operation control unit and data management method | |
| CN101719876B (en) | Network communication dispatching method and system | |
| CN107220310A (en) | A kind of database data management system, method and device | |
| CN102346775A (en) | Method for synchronizing multiple heterogeneous source databases based on log | |
| CN111445236A (en) | Block chain transaction verification method, equipment and storage medium | |
| CN106293960B (en) | A kind of method and system for realizing data conversion based on strom and memory grid | |
| WO2016045367A1 (en) | Multi-data-source data fusion method and device | |
| CN106375480A (en) | Electric energy data real-time acquisition system and method based on distributed system | |
| CN107391596B (en) | A distribution network massive data fusion method and device | |
| CN110226159A (en) | Best-effort traffic library facility | |
| CN202068449U (en) | Data exchange platform used for multistage data exchange | |
| CN107454009B (en) | Low bandwidth overhead traffic scheduling scheme for offline scenarios in data centers | |
| CN103810197A (en) | Hadoop-based data processing method and system | |
| CN105338107A (en) | Stronghold operation synchronous management system and stronghold operation synchronous management method | |
| CN101692737B (en) | Light weight data synchronization system and method | |
| CN107451211B (en) | A kind of download system based on RabbitMQ and MongoDB | |
| CN118708567A (en) | A real-time computing system for Internet of Vehicles big data | |
| CN118550643A (en) | An edge cluster data processing and interaction system | |
| CN106354493A (en) | Development mode implementation method for solving traditional software development pain points |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Wufeng Street software Avenue Gulou District of Fuzhou city in Fujian province 350000 No. 89 Building No. 12 Applicant after: China Electric fufu Mdt InfoTech Ltd Address before: Wufeng Street software Avenue Gulou District of Fuzhou city in Fujian province 350000 No. 89 Building No. 12 Applicant before: Fujian Fushitong Information Software Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |