CN108134745A - Distribution type data collection method and device - Google Patents
Distribution type data collection method and device Download PDFInfo
- Publication number
- CN108134745A CN108134745A CN201711374664.4A CN201711374664A CN108134745A CN 108134745 A CN108134745 A CN 108134745A CN 201711374664 A CN201711374664 A CN 201711374664A CN 108134745 A CN108134745 A CN 108134745A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- preset
- message data
- oriented middleware
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000013480 data collection Methods 0.000 title claims abstract description 20
- 238000012544 monitoring process Methods 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims abstract description 21
- 230000008569 process Effects 0.000 claims description 5
- 238000001514 detection method Methods 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 abstract description 8
- 238000005516 engineering process Methods 0.000 abstract description 8
- 230000000903 blocking effect Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 210000004209 hair Anatomy 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000005303 weighing Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a kind of distribution type data collection method and device, method includes：Acquire generated message data, and collected message data is sent to preset preposition message queue, the message data in above-mentioned preposition message queue is distributed to preset each message-oriented middleware according to preset data distribution mode using preset load equalizer, then the sub- message data in each message-oriented middleware is parsed, the sub- message data after parsing is preserved to the preset database handled by read and write abruption.Compared to existing technologies, the present invention can decompose the transmission pressure of message data, above-mentioned database first passes through read and write abruption processing in advance, there can be higher data processing performance, it will appear middleware blocking when gathered data is larger so as to solve track traffic signal equipment automatic monitoring system in the prior art, data processing not in time, the technical issues of leading to the poor-performing of whole system.
Description
Technical field
The present invention relates to technical field of data processing more particularly to a kind of distribution type data collection methods and device.
Background technology
With the fast development of rail traffic, computer technology is rationally applied to rail by the quick raising of computer technology
Road field of traffic has increasingly becomed the trend of regulation transport development.
For the maintenance mode of various equipment in the signal system of rail traffic, just gradually by artificial detection, periodic maintenance
Monitor and distribute automatically the Mode change of repair work order in real time towards package.It is but this to change the monitoring number for needing magnanimity
According to, and in track traffic signal equipment automatic monitoring system, various gathered datas generally carry out data forwarding by middleware, this
Kind scheme can meet system requirements, but then will appear middleware when gathered data is larger and block up when data volume is smaller
Plug, the data processing various problems such as not in time influence the performance of entire automatic checkout system.
Invention content
It is a primary object of the present invention to provide a kind of distribution type data collection method and device, it is intended to solve existing rail
Road traffic signals equipment automatic monitoring system will appear middleware blocking when gathered data is larger, and data processing not in time, is led
The technical issues of causing the poor-performing of whole system.
To achieve the above object, first aspect present invention provides a kind of distribution type data collection method, and this method includes：
Generated message data is acquired, and collected message data is sent to preset preposition message queue；
Using preset load equalizer by the message data in the preposition message queue according to preset data distribution
Mode is distributed to preset each message-oriented middleware；
The sub- message data in each message-oriented middleware is parsed, the sub- message data after parsing is preserved to preset
The database handled by read and write abruption.
Optionally, it is described using preset load equalizer by the message data in the preposition message queue according to default
Data distribution mode the step of being distributed to preset each message-oriented middleware before further include：
Monitor whether the preposition message queue receives the message data；
When the preposition message queue receives the message data, then perform and described utilize preset load equalizer
Message data in the preposition message queue is distributed to according to preset data distribution mode among preset each message
The step of part.
Optionally, it is described using preset load equalizer by the message data in the preposition message queue according to default
Data distribution mode the step of being distributed to preset each message-oriented middleware include：
Detect the receivable maximum message data amount of each message-oriented middleware；
Based on each receivable maximum message data amount of message-oriented middleware, using the load equalizer by described in
Message data in preposition message queue is distributed to each message-oriented middleware.
Optionally, it is further included before the step of sub- message data in the parsing each message-oriented middleware：
It monitors in each message-oriented middleware with the presence or absence of at least one message for having new sub- message data generation
Between part；
If there are at least one message-oriented middleware for having new sub- message data generation in each message-oriented middleware,
Perform the step of parsing the sub- message data in each message-oriented middleware.
Optionally, the sub- message data in parsing each message-oriented middleware, by the sub- message data after parsing
Preserve to it is preset by read and write abruption handle database the step of include：
The sub- message data in each message-oriented middleware is parsed, the sub- message data after parsing is cached to preset
In memory；
Every preset time interval, the sub- message data cached in the memory is preserved to the database.
To achieve the above object, second aspect of the present invention provides a kind of distributed data acquisition device, which includes：
Acquisition module, for acquiring generated message data, and by collected message data be sent to it is preset before
Put message queue；
Sending module, for utilizing preset load equalizer by the message data in the preposition message queue according to pre-
If data distribution mode be distributed to preset each message-oriented middleware；
Memory module, for parsing the sub- message data in each message-oriented middleware, by the sub- message count after parsing
According to preservation to the preset database handled by read and write abruption.
Optionally, described device further includes：
First monitoring modular, for monitoring whether the preposition message queue receives the message data, before described
When putting message queue and receiving the message data, then the sending module is performed.
Optionally, the sending module includes：
Detection module, for detecting the receivable maximum message data amount of each message-oriented middleware；
Data distribution module for being based on the receivable maximum message data amount of each message-oriented middleware, utilizes institute
It states load equalizer and the message data in the preposition message queue is distributed to each message-oriented middleware.
Optionally, described device further includes：
Second monitoring modular has new sub- message for monitoring in each message-oriented middleware with the presence or absence of at least one
The message-oriented middleware of data generation, if having what new sub- message data generated there are at least one in each message-oriented middleware
Message-oriented middleware then performs the memory module.
Optionally, the memory module includes：
For parsing the sub- message data in each message-oriented middleware, the son after parsing is disappeared for data resolution module
It ceases in data buffer storage to preset memory；
Data memory module, for every preset time interval, the sub- message data cached in the memory to be protected
It deposits to the database.
A kind of distribution type data collection method provided by the present invention, including：Generated message data is acquired, and will be adopted
The message data collected is sent to preset preposition message queue, using preset load equalizer by above-mentioned preposition message queue
In message data be distributed to preset each message-oriented middleware according to preset data distribution mode, then parse each message
Sub- message data in middleware preserves the sub- message data after parsing to the preset data handled by read and write abruption
Library.Compared to existing technologies, collected message data is distributed to multiple disappear by the present invention after load balance process
Middleware is ceased, then parses each message-oriented middleware neutron message data respectively again, and the sub- message data after parsing is preserved
To database, the transmission pressure of message data is decomposed with this, above-mentioned database first passes through read and write abruption processing in advance, can have compared with
High data processing performance, so as to solve in the prior art track traffic signal equipment automatic monitoring system in gathered data
Will appear middleware blocking when larger, data processing not in time, the technical issues of leading to the poor-performing of whole system.
Description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention, for those skilled in the art, without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 is the flow diagram of distribution type data collection method in first embodiment of the invention；
Fig. 2 is the flow diagram of distribution type data collection method in second embodiment of the invention；
Fig. 3 is the flow diagram of distribution type data collection method in third embodiment of the invention；
Fig. 4 is the program module schematic diagram of distributed data acquisition device in fourth embodiment of the invention；
Fig. 5 is the program module schematic diagram of distributed data acquisition device in fifth embodiment of the invention；
Fig. 6 is the program module schematic diagram of distributed data acquisition device in sixth embodiment of the invention.
Specific embodiment
In order to make the invention's purpose, features and advantages of the invention more obvious and easy to understand, below in conjunction with the present invention
The technical solution in the embodiment of the present invention is clearly and completely described in attached drawing in embodiment, it is clear that described reality
It is only part of the embodiment of the present invention to apply example, and not all embodiments.Based on the embodiments of the present invention, people in the art
Member's all other embodiments obtained without making creative work, shall fall within the protection scope of the present invention.
Since in the prior art, track traffic signal equipment automatic monitoring system can go out when collected data are larger
Existing middleware blocks, so as to there are data processing not in time, the technical issues of the poor-performing of whole system.
In order to solve the above-mentioned technical problem, the embodiment of the present invention proposes a kind of distribution type data collection method, and this method will
Collected message data is distributed to multiple message-oriented middlewares after load balance process, then parses each message respectively again
Middleware neutron message data, and the sub- message data after parsing is preserved to database, the biography of message data is decomposed with this
Defeated pressure in addition, above-mentioned database first passes through read and write abruption processing in advance, therefore can have higher data processing performance.
Specifically, referring to Fig. 1, Fig. 1 is the flow signal of distribution type data collection method in first embodiment of the invention
Scheme, in the embodiment of the present invention, above-mentioned distribution type data collection method includes：
Step 101, the generated message data of acquisition, and collected message data is sent to preset preposition message
Queue；
In the embodiment of the present invention, first pass through preset data acquisition equipment acquisition trajectory traffic signals equipment and monitor automatically and be
Generated message data in system, and collected message data is sent to preset preposition message queue.
Wherein, " message " refers to the data unit in two intercomputer transmission, which can be very simple, such as only
Include text-string；Can also be more complicated, such as comprising embedded object.Above-mentioned message data is sent to preposition message queue
In, wherein, " message queue " refers to preserve the container of message in the transmission process of message.Message queue manager is by message
Intermediary is served as when being relayed to its target from its source.The main purpose of queue is to provide routing and ensures the biography of message
It passs；If recipient is unavailable when sending message, message queue meeting reservation message, until can successfully transmit the message.
Step 102, using preset load equalizer by the message data in the preposition message queue according to preset
Data distribution mode is distributed to preset each message-oriented middleware；
In the embodiment of the present invention, software architecture can be pre-designed, in a manner of the data distribution that service logic is met to provide,
The software architecture can be realized based on Spring frames.
Wherein, preset data distribution mode can be used the message data in above-mentioned preposition message queue is equal by loading
Weighing apparatus is sent to multiple message-oriented middlewares.Message-oriented middleware, that is, the ActiveMQ belongs to one kind and increases income messaging bus.
Sub- message data in step 103, parsing each message-oriented middleware, the sub- message data after parsing is preserved
To the preset database handled by read and write abruption.
In the embodiment of the present invention, when above-mentioned each message-oriented middleware receives the sub- message count of above-mentioned load equalizer distribution
According to the content included in the sub- message data later, parsed in each message-oriented middleware, then all the elements parsed are protected
There are preset databases.
Wherein, above-mentioned sub- message data is distributed to the above-mentioned message data of each message-oriented middleware for above-mentioned load equalizer
In a part, be equivalent to and above-mentioned message data split into above-mentioned each sub- message data.
Wherein, above-mentioned database needs to first pass through read and write abruption processing in advance, and the performance of the database is improved with this.
The distribution type data collection method that the above embodiment of the present invention is provided includes：Generated message data is acquired,
And collected message data is sent to preset preposition message queue, preposition disappeared above-mentioned using preset load equalizer
Message data in breath queue is distributed to preset each message-oriented middleware according to preset data distribution mode, and then parsing is each
Sub- message data after parsing is preserved to preset and is handled by read and write abruption by the sub- message data in a message-oriented middleware
Database.Compared to existing technologies, collected message data is distributed to more by the present invention after load balance process
A message-oriented middleware, then parses each message-oriented middleware neutron message data respectively again, and by the sub- message data after parsing
It preserves to database, the transmission pressure of message data is decomposed with this, above-mentioned database first passes through read and write abruption processing, can have in advance
There is higher data processing performance, acquired so as to solve track traffic signal equipment automatic monitoring system in the prior art
Will appear middleware blocking when data are larger, data processing not in time, the technical issues of leading to the poor-performing of whole system.
Further, based on first embodiment of the invention, with reference to Fig. 2, Fig. 2 is distributed number in second embodiment of the invention
According to the flow diagram of acquisition method, in the embodiment of the present invention, above-mentioned distribution type data collection method includes：
Step 201, the generated message data of acquisition, and collected message data is sent to preset preposition message
Queue；
In the embodiment of the present invention, above-mentioned steps 201 and the described content base of step 101 in first embodiment of the invention
This is consistent, please refers to first embodiment of the invention, details are not described herein.
Whether step 202, the monitoring preposition message queue receive the message data, if the preposition message queue
The message data is received, then performs step 203, otherwise continues to monitor whether above-mentioned preposition message queue receives above-mentioned disappear
Cease data；
In the embodiment of the present invention, monitor whether above-mentioned preposition message queue receives message data in real time, that is, real
When monitor whether to collect new message data.When monitoring that above-mentioned preposition message queue receives new message data, then
The message data received is distributed to preset each message-oriented middleware.
Step 203, using preset load equalizer by the message data in the preposition message queue according to preset
Data distribution mode is distributed to preset each message-oriented middleware；
In the embodiment of the present invention, 203 described technical solution of above-mentioned steps may include following steps：
Step a：Detect the receivable maximum message data amount of each message-oriented middleware；
Step b：Based on each receivable maximum message data amount of message-oriented middleware, the load equalizer is utilized
Message data in the preposition message queue is distributed to each message-oriented middleware.
Specifically, in the embodiment of the present invention, can according to the receivable maximum message data amount of each message-oriented middleware come
Distribute above-mentioned message data, so as to ensure that the load of each message-oriented middleware difference is not too large.For example, can preferentially will be above-mentioned
Message data is distributed to the larger message-oriented middleware of receivable maximum message data amount.
In addition, above-mentioned data distribution mode further includes random distribution, distributes by field, broadcast distribution, specifies distribution etc..
Sub- message data in step 204, parsing each message-oriented middleware, the sub- message data after parsing is preserved
To the preset database handled by read and write abruption.
In the embodiment of the present invention, above-mentioned steps 204 and the described content base of step 103 in first embodiment of the invention
This is consistent, please refers to first embodiment of the invention, details are not described herein.
The distribution type data collection method that the embodiment of the present invention is provided, by monitoring whether above-mentioned preposition message queue connects
Above-mentioned message data is received, when above-mentioned preposition message queue receives above-mentioned message data, then utilizes preset load balancing
Message data in above-mentioned preposition message queue is distributed to according to preset data distribution mode in preset each message by device
Between part, so as to effectively prevent above-mentioned preposition message queue from the situation of data congestion occur.
Further, based on second embodiment of the invention, with reference to Fig. 3, Fig. 3 is distributed number in third embodiment of the invention
According to the flow diagram of acquisition method, in the embodiment of the present invention, above-mentioned distribution type data collection method includes：
Step 301, the generated message data of acquisition, and collected message data is sent to preset preposition message
Queue；
Whether step 302, the monitoring preposition message queue receive the message data, if the preposition message queue
The message data is received, then performs step 303, otherwise continues to monitor whether above-mentioned preposition message queue receives above-mentioned disappear
Cease data；
Step 303, using preset load equalizer by the message data in the preposition message queue according to preset
Data distribution mode is distributed to preset each message-oriented middleware；
In the embodiment of the present invention, above-mentioned steps 301 to step 303 and the step 201 in second embodiment of the invention to step
Rapid 203 described content is basically identical, please refers to second embodiment of the invention, details are not described herein.
There is new sub- message data generation with the presence or absence of at least one in step 304, monitoring each message-oriented middleware
Message-oriented middleware, if in each message-oriented middleware there are at least one message for having new sub- message data generation among
Part, then perform step 305, and otherwise continue to monitor in above-mentioned each message-oriented middleware has new sub- message with the presence or absence of at least one
The message-oriented middleware of data generation.
In the embodiment of the present invention, among any one or the multiple message monitored in above-mentioned each message-oriented middleware
Part has received new sub- message data, then performs step 305.
Sub- message data in step 305, parsing each message-oriented middleware, the sub- message data after parsing is preserved
To the preset database handled by read and write abruption.
In the embodiment of the present invention, 305 described technical solution of above-mentioned steps may include following steps：
Step A：Parse the sub- message data in each message-oriented middleware, by the sub- message data after parsing cache to
In preset memory；
Step B：Every preset time interval, the sub- message data cached in the memory is preserved to the data
Library.
In the embodiment of the present invention, it is to be understood that since the speed of service of caching is faster than memory, therefore will be after above-mentioned parsing
Sub- message data cache to preset memory, effectively help system hardware can quickly run.
The distribution type data collection method that the embodiment of the present invention is provided is by monitoring in above-mentioned each message-oriented middleware
It is no there are at least one message-oriented middleware for having new sub- message data generation, if in the presence of parsing among above-mentioned each message
Sub- message data in part, and the sub- message data after parsing is preserved to above-mentioned database, so as to effectively prevent
It states each message-oriented middleware and the situation of data congestion occurs.
In order to solve the above-mentioned technical problem, the embodiment of the present invention also proposes a kind of distributed data acquisition device, specifically,
Referring to Fig. 4, Fig. 4 is the program module schematic diagram of distributed data acquisition device in fourth embodiment of the invention, the present invention is real
It applies in example, above-mentioned distributed data acquisition device includes：
Acquisition module 401 for acquiring generated message data, and collected message data is sent to preset
Preposition message queue；
In the embodiment of the present invention, first pass through in 401 acquisition trajectory traffic signals equipment automatic monitoring system of acquisition module
The message data of generation, and collected message data is sent to preset preposition message queue.
Wherein, " message " refers to the data unit in two intercomputer transmission, which can be very simple, such as only
Include text-string；Can also be more complicated, such as comprising embedded object.Above-mentioned message data is sent to preposition message queue
In, wherein, " message queue " refers to preserve the container of message in the transmission process of message.Message queue manager is by message
Intermediary is served as when being relayed to its target from its source.The main purpose of queue is to provide routing and ensures the biography of message
It passs；If recipient is unavailable when sending message, message queue meeting reservation message, until can successfully transmit the message.
Sending module 402, for being pressed the message data in the preposition message queue using preset load equalizer
Preset each message-oriented middleware is distributed to according to preset data distribution mode；
In the embodiment of the present invention, software architecture can be pre-designed, in a manner of the data distribution that service logic is met to provide,
The software architecture can be realized based on Spring frames.
Wherein, preset data distribution mode can be used the message data in above-mentioned preposition message queue is equal by loading
Weighing apparatus is sent to multiple message-oriented middlewares.Message-oriented middleware, that is, the ActiveMQ belongs to one kind and increases income messaging bus.
Memory module 403, for parsing the sub- message data in each message-oriented middleware, by the sub- message after parsing
Data are preserved to the preset database handled by read and write abruption.
In the embodiment of the present invention, when above-mentioned each message-oriented middleware receives the sub- message count of above-mentioned load equalizer distribution
According to the content included in the sub- message data later, parsed in each message-oriented middleware, then all the elements parsed are protected
There are preset databases.
Wherein, above-mentioned sub- message data is distributed to the above-mentioned message data of each message-oriented middleware for above-mentioned load equalizer
In a part, be equivalent to and above-mentioned message data split into above-mentioned each sub- message data.
Wherein, above-mentioned database needs to first pass through read and write abruption processing in advance, and the performance of the database is improved with this.
The distributed data acquisition device that the above embodiment of the present invention is provided includes：Acquisition module 401, for acquiring
The message data of generation, and collected message data is sent to preset preposition message queue；Sending module 402, is used for
The message data in above-mentioned preposition message queue is distributed according to preset data distribution mode using preset load equalizer
To preset each message-oriented middleware；Memory module 403 for parsing the sub- message data in each message-oriented middleware, will solve
Sub- message data after analysis is preserved to the preset database handled by read and write abruption.Compared to existing technologies, this hair
It is bright that collected message data is distributed to multiple message-oriented middlewares after load balance process, it then parses respectively again each
Message-oriented middleware neutron message data, and the sub- message data after parsing is preserved to database decomposes message data with this
Transmission pressure, above-mentioned database first pass through in advance read and write abruption processing, can have higher data processing performance, so as to solve
Certainly track traffic signal equipment automatic monitoring system will appear middleware blocking, data when gathered data is larger in the prior art
The technical issues of handling not in time, leading to the poor-performing of whole system.
Further, based on fourth embodiment of the invention, with reference to Fig. 5, Fig. 5 is distributed number in fifth embodiment of the invention
According to the program module schematic diagram of harvester, in the embodiment of the present invention, above-mentioned distributed data acquisition device includes：
Acquisition module 401 for acquiring generated message data, and collected message data is sent to preset
Preposition message queue；
First monitoring modular 501, for monitoring whether the preposition message queue receives the message data, if described
Preposition message queue receives the message data, then performs step 203, otherwise continues whether to monitor above-mentioned preposition message queue
Receive above-mentioned message data；
In the embodiment of the present invention, monitor whether above-mentioned preposition message queue receives message data in real time, that is, real
When monitor whether to collect new message data.When monitoring that above-mentioned preposition message queue receives new message data, then
The message data received is distributed to preset each message-oriented middleware.
Sending module 402, for being pressed the message data in the preposition message queue using preset load equalizer
Preset each message-oriented middleware is distributed to according to preset data distribution mode；
In the embodiment of the present invention, above-mentioned sending module 402 includes：
Detection module, for detecting the receivable maximum message data amount of each message-oriented middleware；
Data distribution module for being based on the receivable maximum message data amount of each message-oriented middleware, utilizes institute
It states load equalizer and the message data in the preposition message queue is distributed to each message-oriented middleware.
Specifically, in the embodiment of the present invention, can according to the receivable maximum message data amount of each message-oriented middleware come
Distribute above-mentioned message data, so as to ensure that the load of each message-oriented middleware difference is not too large.For example, can preferentially will be above-mentioned
Message data is distributed to the larger message-oriented middleware of receivable maximum message data amount.
In addition, above-mentioned data distribution mode further includes random distribution, distributes by field, broadcast distribution, specifies distribution etc..
Memory module 403, for parsing the sub- message data in each message-oriented middleware, by the sub- message after parsing
Data are preserved to the preset database handled by read and write abruption.
The distributed data acquisition device that the embodiment of the present invention is provided, by monitoring whether above-mentioned preposition message queue connects
Above-mentioned message data is received, when above-mentioned preposition message queue receives above-mentioned message data, then utilizes preset load balancing
Message data in above-mentioned preposition message queue is distributed to according to preset data distribution mode in preset each message by device
Between part, so as to effectively prevent above-mentioned preposition message queue from the situation of data congestion occur.
Further, based on fifth embodiment of the invention, with reference to Fig. 6, Fig. 6 is distributed number in sixth embodiment of the invention
According to the program module schematic diagram of harvester, in the embodiment of the present invention, above-mentioned distributed data acquisition device includes：
Acquisition module 401 for acquiring generated message data, and collected message data is sent to preset
Preposition message queue；
First monitoring modular 501, for monitoring whether the preposition message queue receives the message data, if described
Preposition message queue receives the message data, then performs sending module 402, otherwise continues to monitor above-mentioned preposition message queue
Whether above-mentioned message data is received；
Sending module 402, for being pressed the message data in the preposition message queue using preset load equalizer
Preset each message-oriented middleware is distributed to according to preset data distribution mode；
Second monitoring modular 601 has new son for monitoring in each message-oriented middleware with the presence or absence of at least one
The message-oriented middleware of message data generation, if having new sub- message data life there are at least one in each message-oriented middleware
Into message-oriented middleware, then perform memory module 403, otherwise continue to monitor in above-mentioned each message-oriented middleware with the presence or absence of at least
One message-oriented middleware for having new sub- message data generation.
In the embodiment of the present invention, among any one or the multiple message monitored in above-mentioned each message-oriented middleware
Part has received new sub- message data, then continues to execute memory module 403.
Memory module 403, for parsing the sub- message data in each message-oriented middleware, by the sub- message after parsing
Data are preserved to the preset database handled by read and write abruption.
In the embodiment of the present invention, above-mentioned memory module 403 includes：
For parsing the sub- message data in each message-oriented middleware, the son after parsing is disappeared for data resolution module
It ceases in data buffer storage to preset memory；
Data memory module, for every preset time interval, the sub- message data cached in the memory to be protected
It deposits to the database.
In the embodiment of the present invention, it is to be understood that since the speed of service of caching is faster than memory, therefore will be after above-mentioned parsing
Sub- message data cache to preset memory, effectively help system hardware can quickly run.
The distributed data acquisition device that the embodiment of the present invention is provided is by monitoring in above-mentioned each message-oriented middleware
It is no there are at least one message-oriented middleware for having new sub- message data generation, if in the presence of parsing among above-mentioned each message
Sub- message data in part, and the sub- message data after parsing is preserved to above-mentioned database, so as to effectively prevent
It states each message-oriented middleware and the situation of data congestion occurs.
In several embodiments provided herein, it should be understood that disclosed device and method can pass through it
Its mode is realized.For example, the apparatus embodiments described above are merely exemplary, for example, the division of the module, only
Only a kind of division of logic function can have other dividing mode in actual implementation, such as multiple module or components can be tied
It closes or is desirably integrated into another system or some features can be ignored or does not perform.Another point, it is shown or discussed
Mutual coupling, direct-coupling or communication connection can be the INDIRECT COUPLING or logical by some interfaces, device or module
Letter connection can be electrical, machinery or other forms.
The module illustrated as separating component may or may not be physically separate, be shown as module
The component shown may or may not be physical module, you can be located at a place or can also be distributed to multiple
On network module.Some or all of module therein can be selected according to the actual needs to realize the mesh of this embodiment scheme
's.
In addition, each function module in each embodiment of the present invention can be integrated in a processing module, it can also
That modules are individually physically present, can also two or more modules be integrated in a module.Above-mentioned integrated mould
The form that hardware had both may be used in block is realized, can also be realized in the form of software function module.
If the integrated module is realized in the form of software function module and is independent product sale or uses
When, it can be stored in a computer read/write memory medium.Based on such understanding, technical scheme of the present invention is substantially
The part to contribute in other words to the prior art or all or part of the technical solution can be in the form of software products
It embodies, which is stored in a storage medium, is used including some instructions so that a computer
Equipment (can be personal computer, server or the network equipment etc.) performs the complete of each embodiment the method for the present invention
Portion or part steps.And aforementioned storage medium includes：USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only
Memory), random access memory (RAM, Random Access Memory), magnetic disc or CD etc. are various can store journey
The medium of sequence code.
It should be noted that for aforementioned each method embodiment, describe, therefore it is all expressed as a series of for simplicity
Combination of actions, but those skilled in the art should know, the present invention is not limited by described sequence of movement because
According to the present invention, certain steps may be used other sequences or be carried out at the same time.Secondly, those skilled in the art should also know
It knows, embodiment described in this description belongs to preferred embodiment, and involved action and module might not all be this hairs
Necessary to bright.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, there is no the portion being described in detail in some embodiment
Point, it may refer to the associated description of other embodiments.
It is above the description to a kind of distribution type data collection method provided by the present invention and device, for this field
The thought of technical staff, according to the present invention embodiment, there will be changes, comprehensive in specific embodiments and applications
On, the content of the present specification should not be construed as limiting the invention.
Claims (10)
1. a kind of distribution type data collection method, which is characterized in that the method includes：
Generated message data is acquired, and collected message data is sent to preset preposition message queue；
Using preset load equalizer by the message data in the preposition message queue according to preset data distribution mode
It is distributed to preset each message-oriented middleware；
The sub- message data in each message-oriented middleware is parsed, the sub- message data after parsing is preserved to preset process
The database of read and write abruption processing.
2. according to the method described in claim 1, it is characterized in that, described preposition disappeared using preset load equalizer by described
Before the step of message data in breath queue is distributed to preset each message-oriented middleware according to preset data distribution mode
It further includes：
Monitor whether the preposition message queue receives the message data；
When the preposition message queue receives the message data, then perform it is described using preset load equalizer by institute
The message data stated in preposition message queue is distributed to preset each message-oriented middleware according to preset data distribution mode
Step.
3. according to the method described in claim 1, it is characterized in that, described preposition disappeared using preset load equalizer by described
The step of message data in breath queue is distributed to preset each message-oriented middleware according to preset data distribution mode includes：
Detect the receivable maximum message data amount of each message-oriented middleware；
It, will be described preposition using the load equalizer based on each receivable maximum message data amount of message-oriented middleware
Message data in message queue is distributed to each message-oriented middleware.
4. the according to the method described in claim 1, it is characterized in that, sub- message parsed in each message-oriented middleware
It is further included before the step of data：
It monitors in each message-oriented middleware with the presence or absence of at least one message-oriented middleware for having new sub- message data generation；
If it is performed in each message-oriented middleware there are at least one message-oriented middleware for having new sub- message data generation
The step of parsing the sub- message data in each message-oriented middleware.
5. method according to any one of claims 1 to 4, which is characterized in that among parsing each message
Sub- message data in part preserves the sub- message data after parsing to the step of the preset database handled by read and write abruption
Suddenly include：
The sub- message data in each message-oriented middleware is parsed, the sub- message data after parsing is cached to preset storage
In device；
Every preset time interval, the sub- message data cached in the memory is preserved to the database.
6. a kind of distributed data acquisition device, which is characterized in that described device includes：
Collected message data for acquiring generated message data, and is sent to preset preposition disappear by acquisition module
Cease queue；
Sending module, for utilizing preset load equalizer by the message data in the preposition message queue according to preset
Data distribution mode is distributed to preset each message-oriented middleware；
For parsing the sub- message data in each message-oriented middleware, the sub- message data after parsing is protected for memory module
It deposits to the preset database handled by read and write abruption.
7. device according to claim 6, which is characterized in that described device further includes：
First monitoring modular for monitoring whether the preposition message queue receives the message data, preposition disappears when described
When breath queue receives the message data, then the sending module is performed.
8. device according to claim 6, which is characterized in that the sending module includes：
Detection module, for detecting the receivable maximum message data amount of each message-oriented middleware；
Data distribution module for being based on the receivable maximum message data amount of each message-oriented middleware, is born using described
It carries balanced device and the message data in the preposition message queue is distributed to each message-oriented middleware.
9. device according to claim 6, which is characterized in that described to further include：
Second monitoring modular has new sub- message data for monitoring in each message-oriented middleware with the presence or absence of at least one
The message-oriented middleware of generation, if there are at least one message for having new sub- message data generation in each message-oriented middleware
Middleware then performs the memory module.
10. according to the device described in claim 6 to 9 any one, which is characterized in that the memory module includes：
Data resolution module, for parsing the sub- message data in each message-oriented middleware, by the sub- message count after parsing
According in caching to preset memory；
Data memory module, for every preset time interval, by the sub- message data cached in the memory preserve to
The database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711374664.4A CN108134745A (en) | 2017-12-19 | 2017-12-19 | Distribution type data collection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711374664.4A CN108134745A (en) | 2017-12-19 | 2017-12-19 | Distribution type data collection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108134745A true CN108134745A (en) | 2018-06-08 |
Family
ID=62390614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711374664.4A Pending CN108134745A (en) | 2017-12-19 | 2017-12-19 | Distribution type data collection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108134745A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108900631A (en) * | 2018-07-27 | 2018-11-27 | 阿里巴巴集团控股有限公司 | A kind of message distribution method, device and distributed system |
| CN110708312A (en) * | 2019-09-30 | 2020-01-17 | 交控科技股份有限公司 | Method and system for message transmission in ATS and ATS |
| CN112597371A (en) * | 2020-12-25 | 2021-04-02 | 牧原食品股份有限公司 | Data acquisition system, method and device based on message middleware |
| CN113609199A (en) * | 2021-07-27 | 2021-11-05 | 远景智能国际私人投资有限公司 | Database system, server, and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103699692A (en) * | 2014-01-11 | 2014-04-02 | 樊建 | Internet-of-things access platform data management method |
| CN105530137A (en) * | 2014-09-28 | 2016-04-27 | 中国银联股份有限公司 | Flow data analysis method and flow data analysis system |
| CN106453482A (en) * | 2016-08-05 | 2017-02-22 | 成都卡德智能科技有限公司 | Internet of things middleware system and Internet of things system |
| CN106557561A (en) * | 2016-11-16 | 2017-04-05 | 贵州大学 | Magnanimity sensing data storage system and method based on HBase |
-
2017
- 2017-12-19 CN CN201711374664.4A patent/CN108134745A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103699692A (en) * | 2014-01-11 | 2014-04-02 | 樊建 | Internet-of-things access platform data management method |
| CN105530137A (en) * | 2014-09-28 | 2016-04-27 | 中国银联股份有限公司 | Flow data analysis method and flow data analysis system |
| CN106453482A (en) * | 2016-08-05 | 2017-02-22 | 成都卡德智能科技有限公司 | Internet of things middleware system and Internet of things system |
| CN106557561A (en) * | 2016-11-16 | 2017-04-05 | 贵州大学 | Magnanimity sensing data storage system and method based on HBase |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108900631A (en) * | 2018-07-27 | 2018-11-27 | 阿里巴巴集团控股有限公司 | A kind of message distribution method, device and distributed system |
| CN110708312A (en) * | 2019-09-30 | 2020-01-17 | 交控科技股份有限公司 | Method and system for message transmission in ATS and ATS |
| CN112597371A (en) * | 2020-12-25 | 2021-04-02 | 牧原食品股份有限公司 | Data acquisition system, method and device based on message middleware |
| CN113609199A (en) * | 2021-07-27 | 2021-11-05 | 远景智能国际私人投资有限公司 | Database system, server, and storage medium |
| CN113609199B (en) * | 2021-07-27 | 2023-09-12 | 远景智能国际私人投资有限公司 | Database system, server, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108134745A (en) | Distribution type data collection method and device | |
| CN102469033B (en) | Message subscription system and message sending method | |
| WO2015090241A1 (en) | Method for monitoring business operations data storage, and related device and system | |
| CN109491873B (en) | Cache monitoring method, medium, device and computing equipment | |
| CN110719215B (en) | Flow information acquisition method and device of virtual network | |
| CN102937984B (en) | A kind of collect the system of data, client and method | |
| US20190065534A1 (en) | Method and device for data deduplication | |
| CN103067884A (en) | Method and device for alarming information processing | |
| CN114091704A (en) | Alarm suppression method and device | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| WO2016192568A1 (en) | Multi-point monitoring alarm method and system | |
| CN105897549A (en) | Message sending and receiving methods and message sending and receiving devices | |
| CN108134746B (en) | Method and device for processing rail transit data | |
| CN108255659A (en) | A kind of application program capacity monitoring method and its system | |
| CN110275815A (en) | A system abnormal alarm processing method and device | |
| CN109710502B (en) | Log transmission method, device and storage medium | |
| CN113055493B (en) | Data packet processing method, device, system, scheduling device and storage medium | |
| CN112330910B (en) | Information processing method and device for fire early warning, storage medium and processor | |
| CN114490280A (en) | Log processing method, device, equipment and medium | |
| CN104142941A (en) | Mass positioning data storing method | |
| CN110351273B (en) | Method, device and system for network tracking long chain attack | |
| CN109376137A (en) | A file processing method and device | |
| CN109284257B (en) | Log writing method and device, electronic equipment and storage medium | |
| CN105187490B (en) | A kind of transfer processing method of internet of things data | |
| CN110347518A (en) | Message treatment method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180608 |