CN108399186A - A kind of collecting method and device - Google Patents
A kind of collecting method and device Download PDFInfo
- Publication number
- CN108399186A CN108399186A CN201810030892.8A CN201810030892A CN108399186A CN 108399186 A CN108399186 A CN 108399186A CN 201810030892 A CN201810030892 A CN 201810030892A CN 108399186 A CN108399186 A CN 108399186A
- Authority
- CN
- China
- Prior art keywords
- data
- journal file
- daily record
- record data
- reading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An embodiment of the present invention provides a kind of collecting method and devices, including：The change message of journal file in monitoring service system；Judge whether the daily record data amount in data queue is more than predetermined threshold value；If it is not, then reading the daily record data of setting quantity from journal file according to the reading subscript of journal file in journal file inventory, the daily record data of reading is sent to the reading subscript of journal file in data queue and Update log lists of documents；When it includes ending mark to determine the daily record data read not, the step of whether the daily record data amount judged in data queue is more than predetermined threshold value is returned to, until the daily record data read includes ending mark.Due to just gathered data and the data of acquisition being put into data queue from journal file when data queue's memory does not reach the upper limit, and the daily record data of preset data is only acquired every time, lead to data loss problem so as to avoid the data of acquisition are inserted into fully loaded data queue, optimizes data acquisition.
Description
Technical field
The present embodiments relate to technical field of data processing more particularly to a kind of collecting methods and device.
Background technology
The big data epoch, we in addition to will from relevant database the various structural datas of real-time synchronization, it is also necessary to from
Acquisition applications daily record, system log carry out analyzing processing in operation system, this partial data is not only complicated and quantity is huge
Greatly.Data are periodically only backuped to network attached storage (Network Attached Storage, abbreviation in the prior art
) etc. NAS in storage devices, data investigation problem is only just routed up when system breaks down.Now we by daily record data into
Also it is included in model after row cleaning and carries out analysis calculating, and result is included in main business and carries out assessment prediction, this let us logarithm
According to timeliness and integrality have higher requirement, need to obtain in real time, and cannot lack.
For real-time data collection, in the prior art by from local log read data, then it is sent to data/address bus progress
The write-in of data would generally increase a transmit queue during transmission, cache data to be sent, in an asynchronous manner will
The step of acquisition and transmission, detaches, and buffers the influence of surge pressure, improves the stability and efficiency of system.But in transmit queue
It is stored to up to the upper limit and will result in loss of data when collector can not carry out data insertion.In addition transmit queue is from acquisition process
Inside application memory creates, if process surprisingly terminates, the data to be sent in memory will lose.
Invention content
A kind of collecting method of offer of the embodiment of the present invention and device, for solving in transmit queue in the prior art
The problem of data are easily lost when queuing memory reaches the upper limit and process exception terminates.
An embodiment of the present invention provides a kind of collecting methods, including：
The change message of journal file in monitoring service system；
Judge whether the daily record data amount in data queue is more than predetermined threshold value；
It is set if it is not, then being read from the journal file according to the reading subscript of journal file described in journal file inventory
The daily record data of reading is sent to the data queue and updates institute in the journal file inventory by the daily record data of fixed number amount
The reading subscript of journal file is stated, the reading subscript is for identifying the journal file last reading in preceding primary reading
Position；
When it includes ending mark to determine the daily record data read not, the daily record data judged in the data queue is returned
The step of whether amount is more than predetermined threshold value, until the daily record data read includes ending mark.
Optionally, the reading subscript according to journal file described in journal file inventory is read from the journal file
Take setting quantity daily record data further include after the daily record data of reading is sent to the data queue：
Daily record data in the data queue is sent to data-storage system；
When receiving the reception success message of the data-storage system feedback, extremely by the journal file list synchronization
Non-volatile memory device.
Optionally, further include：
Monitor task handling duration, the task handling duration are the daily record that setting quantity is read from the journal file
The daily record data of reading is extremely sent to the duration used in the data queue by data；
When determining that the task handling duration is more than first threshold, daily record data will be read from the journal file
Speed is reduced to the first preset value；It, will be from the journal file when determining that the task handling duration is less than second threshold
The speed for reading daily record data is promoted to the second preset value.
Optionally, judge the daily record in data queue after the change message of journal file in the monitoring service system
Whether data volume is more than before predetermined threshold value, further includes：
The change message is stored to notification queue；
The change message is obtained from the notification queue；
Determine that the change type of the change message is modification.
Before the return judges the step of whether the daily record data amount in the data queue is more than predetermined threshold value, also wrap
It includes：
When it includes ending mark to determine the daily record data read not, the change message is stored again to the notice
Queue.
Optionally, further include：
When it is newly-built to determine the change type, the note of the journal file is added in the journal file inventory
Record, the record include filename, read subscript；
When it is renaming to determine the change type, the file of journal file described in the journal file inventory is updated
Name；
When it is to delete to determine the change type, the record of journal file described in the journal file inventory is deleted.
An embodiment of the present invention provides a kind of data acquisition devices, including：
Module is monitored, the change message for journal file in monitoring service system；
Processing module, for judging whether the daily record data amount in data queue is more than predetermined threshold value；If it is not, then according to day
The reading subscript of journal file described in will lists of documents reads the daily record data of setting quantity from the journal file, will read
The daily record data taken is sent to the data queue and updates the reading subscript of journal file described in the journal file inventory,
The reading subscript last reads position for identifying the journal file in preceding primary reading；Determining the daily record read
When data do not include ending mark, the step whether the daily record data amount in the data queue that judges is more than predetermined threshold value is returned
Suddenly, until the daily record data read includes ending mark.
Optionally, the processing module is additionally operable to：
Setting quantity is read from the journal file according to the reading subscript of journal file described in journal file inventory
Daily record data, after the daily record data of reading is sent to the data queue, by the daily record data in the data queue
It is sent to data-storage system；
When receiving the reception success message of the data-storage system feedback, extremely by the journal file list synchronization
Non-volatile memory device.
Optionally, the processing module is additionally operable to：
Monitor task handling duration, the task handling duration are the daily record that setting quantity is read from the journal file
The daily record data of reading is extremely sent to the duration used in the data queue by data；
When determining that the task handling duration is more than first threshold, daily record data will be read from the journal file
Speed is reduced to the first preset value；It, will be from the journal file when determining that the task handling duration is less than second threshold
The speed for reading daily record data is promoted to the second preset value.
Optionally, the processing module is additionally operable to：
In monitoring service system after the change message of journal file, judge whether the daily record data amount in data queue is big
Before predetermined threshold value, the change message is stored to notification queue；
The change message is obtained from the notification queue；
Determine that the change type of the change message is modification.
Before the return judges the step of whether the daily record data amount in the data queue is more than predetermined threshold value, also wrap
It includes：
When it includes ending mark to determine the daily record data read not, the change message is stored again to the notice
Queue.
Optionally, the processing module is additionally operable to：
When it is newly-built to determine the change type, the note of the journal file is added in the journal file inventory
Record, the record include filename, read subscript；
When it is renaming to determine the change type, the file of journal file described in the journal file inventory is updated
Name；
When it is to delete to determine the change type, the record of journal file described in the journal file inventory is deleted.
An embodiment of the present invention provides a kind of computing devices, including：
Memory, for storing program instruction；
Processor executes any of the above-described for calling the program instruction stored in the memory according to the program of acquisition
Method described in.
An embodiment of the present invention provides a kind of computer readable storage medium, the computer-readable recording medium storage has
Computer executable instructions, the computer executable instructions are for making computer execute method described in any one of the above embodiments.
To sum up, an embodiment of the present invention provides a kind of collecting method and devices, including：Daily record in monitoring service system
The change message of file；Judge whether the daily record data amount in data queue is more than predetermined threshold value；If it is not, then according to journal file
The reading subscript of journal file described in inventory reads the daily record data of setting quantity from the journal file, by the day of reading
Will data are sent to the data queue and update the reading subscript of journal file described in the journal file inventory, the reading
It removes mark and last reads position in preceding primary reading for identifying the journal file；Determining the daily record data of reading not
When being identified including ending, the step of whether the daily record data amount judged in the data queue is more than predetermined threshold value is returned, until
The daily record data of reading includes ending mark.After the change message of journal file in listening to operation system,
Before reading daily record data in journal file, the EMS memory occupation situation of data queue is judged.When data queue's memory does not arrive
Up to just gathered data and the data of acquisition data queue is put into from journal file when the upper limit, when middle memory reaches in data team
When the upper limit, stop the gathered data from journal file, is led so as to avoid the data of acquisition are inserted into fully loaded data queue
Cause data loss problem.When due to acquiring daily record data from journal file, the daily record data of preset data is only acquired every time, and
And can judge data queue's EMS memory occupation situation before acquisition every time, rather than the daily record data changed in journal file is complete
Portion is sent to data queue after reading, and to advanced optimize data acquisition, avoids the loss of data of acquisition.
Description of the drawings
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the invention, for this
For the those of ordinary skill in field, without having to pay creative labor, it can also be obtained according to these attached drawings
His attached drawing.
Fig. 1 is a kind of system architecture schematic diagram that the embodiment of the present invention is applicable in；
Fig. 2 is a kind of flow diagram of collecting method provided in an embodiment of the present invention；
Fig. 3 is a kind of structural schematic diagram of data acquisition device provided in an embodiment of the present invention；
Fig. 4 is the structural schematic diagram of another data acquisition device provided in an embodiment of the present invention.
Specific implementation mode
In order to make the purpose of the present invention, technical solution and advantageous effect be more clearly understood, below in conjunction with attached drawing and implementation
Example, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only used to explain this hair
It is bright, it is not intended to limit the present invention.
Fig. 1 illustrates a kind of applicable system architecture schematic diagram of the embodiment of the present invention, as shown in Figure 1, of the invention
The applicable system architecture of embodiment includes operation system 101, data acquisition device 102, configuration management center 103, data cleansing
Equipment 104, data/address bus 105 and data consumption equipment 106.Wherein data acquisition device 102 can be located at operation system 101
In, it can also be independently of operation system 101.One or more paths in 102 real-time monitoring service system 101 of data acquisition device
Under meet default prefix, default suffix, fuzzy matching rule multiple files.File is read when file changes.Configuration
Administrative center 103 connect with data acquisition device 102, and the configuration file and data for preserving data acquisition device 102 are adopted
The status information of 102 gathered data of acquisition means.Configuration management center 103 preserves data using Nonvolatile memory, therefore works as data
When acquisition process surprisingly terminates, the data that configuration management center 103 preserves will not lose.Data acquisition device 102 is from business system
In system 101 after gathered data, data/address bus 105 is sent data to.Total access of the data/address bus 105 as operation system 101,
Not single data that can receive the acquisition of data acquisition device 102, can also receive the data of other forms, such as from relevant database
Synchronous data, or the data from third party's acquisition.In specific implementation, using Kafka, (Kafka is that height gulps down to data/address bus 105
The distributed post for the amount of spitting subscribes to message system) message queue structure.Data/address bus 105 supports the life of a variety of different data accesses
Production person also supports a variety of consumers such as real-time, offline.Data cleansing equipment 104 is connect with data/address bus 105, is used for data
Data taking-up in bus 105 is sorted, is cleaned, structuring conversion and isomeric data merge, and then places data into number again
In different themes according to bus 105, subscribes to and use for real-time, off-line system.Specifically, data cleansing equipment 104 is one
Application cluster, being capable of the nearly source data of parallel processing.Data/address bus 105 is connect with data consumption equipment 106, data consumption equipment
106 data after obtaining cleaning in data/address bus 105 handle the data obtained, such as data according to specific application scenarios
Consumer device 106 can be data-storage system, and backup preservation is carried out after the data after obtaining cleaning in bus.Data consumption
Equipment 106 or monitoring device, obtain data, then according to the data monitoring business system of acquisition from bus 105 in real time
The working condition etc. of system 101.105 real-time recorded data consumer device 106 of bus reads the position of data, for example works as data consumption
Equipment 106 sends feedback message to data/address bus 104, data/address bus from reading data in data/address bus 105 and after completing consumption
104 update the data the position that consumer device 106 reads data according to feedback message.When data consumption equipment 106 is from data/address bus
Abnormal end when reading data in 105 but not completing consumption, data/address bus 104 will not update the data the reading of consumer device 106
The position for evidence of fetching, after abnormal ending, data consumption equipment 106 read the position of data from data/address bus 105 according to last time
Read data.
Based on system architecture shown in FIG. 1, as shown in Fig. 2, an embodiment of the present invention provides a kind of collecting method, it should
Flow can be executed by data acquisition device, be included the following steps：
Step S201, the change message of journal file in monitoring service system.
Step S202, judges whether the daily record data amount in data queue is more than predetermined threshold value.
Step S203, if it is not, then being read from journal file according to the reading subscript of journal file in journal file inventory
Set the daily record data of quantity.
The daily record data of reading is sent to journal file in data queue and Update log lists of documents by step S204
Read subscript.
Step S205 returns to the day judged in data queue when it includes ending mark to determine the daily record data read not
The step of whether will data volume is more than predetermined threshold value, until the daily record data read includes ending mark.
Specifically, in step S201 and step S202, in monitoring service system after the change message of journal file,
It can directly judge whether the daily record data amount in data queue is more than predetermined threshold value.Change message can also be stored to notice team
Then row obtain change message from notification queue, then judge whether the daily record data amount in data queue is more than predetermined threshold value.Tool
Body, when predetermined threshold value can as the case may be, judge the EMS memory occupation situation of data queue, it is also not limited to judge data team
Whether the daily record data amount in row is more than predetermined threshold value, can also judge whether memory accounting is more than predetermined threshold value in data queue
Deng.In specific implementation, when using notification queue caching change message when, in listening to operation system the change of journal file disappear
After breath, change message can be put into filename+change type in notification queue, for example it is newly-built to listen to change message 1
After journal file A, change message 1 is put into notification queue in the form of journal file A+ is newly-built；Change is for another example listened to disappear
After breath 2 is modification journal file B, change message 2 is put into notification queue in the form of journal file B+ modifications.Change message
Change type includes at least newly-built, modification, renaming and deletion.It should be noted that be not to be directed to all change types,
It will judge whether the daily record data amount in data queue is more than predetermined threshold value, only determine the change type of the change message
When to change, it can just judge whether the daily record data amount in data queue is more than predetermined threshold value.Further, the embodiment of the present invention
Middle operation system not merely only includes journal file, when needing to acquire the data in other files, skill that the present invention is implemented
Art scheme is equally applicable.
In step S203, journal file inventory includes the record for the journal file for needing to acquire, such as journal file
Filename, read subscript, timestamp etc., wherein reading subscript for identifying last reading of the journal file in preceding primary reading
Fetch bit is set.It is the 21st row to last read position when primary reading journal file A before such as, then is designated as under the reading of journal file A
21st row.When the daily record data amount in data queue that determines is not more than predetermined threshold value, according to daily record text in journal file inventory
The reading subscript of part reads the daily record data of setting data.When the daily record data amount in data queue that determines is more than predetermined threshold value
When, change message is stored again to notification queue.Setting quantity determine as the case may be, can with fixed line number, such as
It is fixed to read 50 rows every time；It can also be the line number of mechanical periodicity, for example read 50 rows for the first time, read 30 rows etc. second.
In addition, when change type is newly-built, renaming and deletes, Update log lists of documents are needed.When change type is newly-built,
The record of journal file is added in journal file inventory, record includes filename, reading subscript etc..For example become in notification queue
When more message is journal file A+ newly-built, record is added in journal file inventory：It is designated as 1 under journal file A --- reading.
When determining that change type is renaming, the filename of Update log lists of documents journal file.For example change disappears in notification queue
When breath is journal file B+ renamed as journal file C, being recorded as journal file B is found in journal file inventory：Day
Will file B --- 21 rows are designated as under reading, the record modification by journal file B is：It is designated as 21 under journal file C --- reading.
When it is to delete to determine change type, the record of journal file manifest log file is deleted.For example message is changed in notification queue
When being deleted for journal file D+, being recorded as journal file D is found in journal file inventory：Journal file D --- it reads
Under be designated as 51 rows, then delete journal file inventory in journal file D this record.
Due to the change message of journal file in real-time monitoring service system, then according to change information updating journal file
Inventory, therefore when aiming at the switching of daily zero day, the handoff procedure of journal file can be listened in real time and according to tune the case where switching
The reading of whole daily record data prevents the loss of data generated in daily record handoff procedure.It is illustrated below with specific example,
For example setting the journal file on January 4th, 2018 as journal file M, when the January in 2018 of zero on the 5th, journal file M is switched to
Then journal file N has created the journal file M on January 5th, 2018, and daily output January 5 in 2018 is put into journal file M
Raw daily record data.In daily record handoff procedure, the change message listened to is journal file M --- renamed as journal file
N, journal file M --- newly-built, journal file M --- modification, it is then clear according to the change message modification journal file listened to
It is single, it finishes, is remained to according to daily record text in journal file inventory if the daily record data of journal file N is not read when zero switches
The reading subscript of part N continues to read daily record data, until comprising ending mark in the daily record data read, to prevent zero
Loss of data caused by daily record data has not been read when point daily record switching.
In step S204 and step S205, end of the ending mark for identifying the daily record data in journal file.
It includes when ending up mark, illustrating that the daily record data in journal file has been read not yet, then to determine the daily record data read not
Change message is stored again to notification queue.Then according to the change message in notification queue, judge in data queue again
Whether daily record data amount is more than predetermined threshold value, if it is not, then according to the reading subscript of journal file in journal file inventory from daily record
The daily record data of reading is sent to data queue and Update log lists of documents by the daily record data that setting quantity is read in file
The reading subscript of middle journal file, recycles successively, until the daily record data of reading includes ending mark.
After the change message of journal file in listening to operation system, daily record number is being read from journal file
According to the EMS memory occupation situation for before, judging data queue.When data queue's memory does not reach the upper limit just from journal file
The data of acquisition are simultaneously put into data queue by gathered data, when middle memory reaches the upper limit in data team, are stopped from journal file
Middle gathered data leads to data loss problem so as to avoid the data of acquisition are inserted into fully loaded data queue.Due to from
When acquiring daily record data in journal file, the daily record data of preset data is only acquired every time, and can sentence before acquisition every time
Disconnected data queue EMS memory occupation situation, rather than it is sent to data team after the daily record data changed in journal file is all read
Row, to advanced optimize data acquisition, avoid the loss of data of acquisition.
Optionally, setting quantity is read from journal file according to the reading subscript of journal file in journal file inventory
After the daily record data of reading is sent to the data queue, the daily record data in data queue is sent to for daily record data
Data-storage system.When receiving the reception success message of data-storage system feedback, by journal file list synchronization to non-
Volatile storage devices.In specific implementation, data-storage system is non-easy for storing the daily record data acquired from operation system
The property lost storage device can be configuration management center, can also be data-storage system etc..When data acquisition process exception terminates
When, the journal file inventory in non-volatile memory device will not be lost, therefore after data acquisition process resumption is normal, according to
The journal file inventory preserved in non-volatile memory device can determine that the reading subscript of each journal file, re-read daily record
Daily record data in file, the process that avoids surprisingly terminate and caused by loss of data.Furthermore it is also possible to timing acquisition data team
The snapshot of row and notification queue, and the snapshot of acquisition is also preserved into non-volatile memory device, therefore acquire process in data
After restoring normal, journal file inventory and the snapshot of data queue and notification queue in nonvolatile memory can be combined to determine
The reading subscript of journal file.Knowing for capable of more preparing so is not read when data acquisition abnormity and is successfully counted without sending
According to decreasing the data of repeated acquisition.For data acquisition process resumption it is normal after, according to snapshot and journal file inventory
Data duplication phenomenon caused by gathered data, the embodiment of the present invention provide following solution：Unique mark is increased to every data
Know, the key message that unique mark can be according in data adds timestamp the cryptographic Hash of generation.Then in the data cleansing stage
When, re-scheduling is carried out according to the unique mark of data, so as to effectively avoid the Data duplication of acquisition.
Optionally, during log data acquisition, task handling duration is monitored in real time, task handling duration is from daily record
The daily record data of reading is extremely sent to the duration used in data queue by the daily record data that setting quantity is read in file.In determination
When task handling duration is more than first threshold, the speed that daily record data is read from journal file is reduced to the first preset value；
When determining that task handling duration is less than second threshold, it is pre- that the speed that daily record data is read from journal file is promoted to second
If value.Wherein first threshold and second threshold can form task and take baseline, and it is at operation system is conventional that task, which takes baseline,
When reason business, gathered data averagely takes range from operation system.First preset value and the second predetermined threshold value are according to specific
Situation determines, for example when determining that task handling duration is more than first threshold, halves the daily record data read from journal file
Measure and be sent to the daily record data amount of data queue.For example it when determining that task handling duration is less than second threshold, doubles from day
The daily record data amount read in will file and the daily record data amount for being sent to data queue.By monitoring task handling duration in real time
The busy extent for determining operation system illustrates that operation system current business is busy when task handling duration is more than first threshold,
In order to not influence the normal business processing of operation system, the speed that daily record data is read from journal file is reduced, to whole
The occupancy to operation system resource is reduced, ensure that the normal business processing of operation system.
Embodiment in order to preferably explain the present invention describes the embodiment of the present invention below by specific implement scene and provides
A kind of collecting method flow, which can be executed by data acquisition device.It includes prison to set data acquisition device
Device 1021, notification queue 1022, processing unit 1023, data queue 1024, transmitter 1025 are listened, it is specific as shown in Figure 3.It monitors
Device 1021 obtains the change message of journal file in operation system by receiving the notifications of operation system, then according to daily record
The change message of file determines that change type, wherein change type include newly-built, modification, renaming and deletion.Monitor 1021
With the format of filename+change type notification queue 1022 is sent to by message is changed.Processing unit 1023 is from notification queue
Change message is obtained in 1022, different disposal step is then executed according to change type different in change message, wherein handling
Unit 1023 is multithreading, being capable of parallel processing for the change message of multiple and different change types.Below with specific example
Above-mentioned four kinds of change types are illustrated：
Change type is newly-built：Setting change message 1 is that journal file A+ is created, and processing unit 1023 is knowing that change disappears
After breath 1, the record of journal file A is added in journal file inventory, for example record can be journal file A --- read subscript
It is 1.When operation system is put into the daily record data of generation in newly-built journal file, monitor will listen to change type and be
The change message of modification.
Change type is renaming：Setting change message 2 is journal file C+ renamed as journal file E, processing unit
1023 know change message 2 after, the record of query log files C in journal file inventory, if inquiry know be recorded as：
It is designated as 51 under journal file C --- reading.It is by the record modification of journal file C in journal file inventory then：Journal file
It is designated as 51 under E --- reading.When operation system is put into the daily record data of generation in the journal file of renaming, monitor will
Listen to the change message that change type is modification.
Change type is to delete：Setting change message 3 is that journal file D+ is deleted, and processing unit 1023 is knowing that change disappears
After breath 3, the record of query log files D and then deleted in journal file inventory.
Change type is modification：Setting change message 4 is journal file B+ modifications, Sino-Japan by query log files inventory
The record of will file B is known is designated as 21 under the reading of journal file B.Processing unit 1023 judges number after knowing change message 4
Whether it is more than predetermined threshold value according to daily record data amount in queue 1024, if it is not, then reading 50 rows since the 21st row of journal file B
Then 50 row daily record datas of reading are sent to data queue 1024 and by journal file in journal file inventory by daily record data
The reading subscript of B is updated to 71.Further, processing unit 1023 judges whether the 50 row daily record datas read include ending mark
Know, if including, terminates this data acquisition session.Otherwise, change message 4 is placed again into notification queue 1022.Processing is single
Member 1023 judges whether daily record data amount is more than predetermined threshold value in data queue 1024, if not again after knowing change message 4
More than predetermined threshold value, then 50 row daily record datas are read since the 71st row of journal file B, then by 50 row daily record numbers of reading
According to being sent to data queue 1024 and the reading subscript of journal file B in journal file inventory be updated to 121, and so on,
Until the daily record data that processing unit 1023 is read from journal file B includes ending mark.In addition, judging data queue
When whether daily record data amount is more than predetermined threshold value in 1024, if daily record data amount is more than predetermined threshold value in data queue 1024,
Change message 4 is placed again into notification queue 1022.Further, after data queue 1024 receives the daily record data read, lead to
It crosses transmitter 1025 and the daily record data of reading is sent to data/address bus.Processor 1023 is receiving connecing for data/address bus feedback
When receiving success message, by journal file list synchronization to configuration management center.In addition processor 1023 is appointed in processing data acquisition
When business, task handling duration is monitored in real time, task handling duration is that processor 1023 reads setting quantity from journal file
The daily record data of reading is extremely sent to the duration used in data queue by daily record data.Processor 1023 is when determining task processing
When long baseline time-consuming more than task, it will halve and read daily record data amount from journal file and be sent to the day of data queue 1024
Will data volume；When determining that task handling duration takes baseline less than task, will double to read daily record data from journal file
Measure and be sent to the daily record data amount of data queue 1024.
To sum up, an embodiment of the present invention provides a kind of collecting method and devices, including：Daily record in monitoring service system
The change message of file；Judge whether the daily record data amount in data queue is more than predetermined threshold value；If it is not, then according to journal file
The reading subscript of journal file described in inventory reads the daily record data of setting quantity from the journal file, by the day of reading
Will data are sent to the data queue and update the reading subscript of journal file described in the journal file inventory, the reading
It removes mark and last reads position in preceding primary reading for identifying the journal file；Determining the daily record data of reading not
When being identified including ending, the step of whether the daily record data amount judged in the data queue is more than predetermined threshold value is returned, until
The daily record data of reading includes ending mark.After the change message of journal file in listening to operation system,
Before reading daily record data in journal file, the EMS memory occupation situation of data queue is judged.When data queue's memory does not arrive
Up to just gathered data and the data of acquisition data queue is put into from journal file when the upper limit, when middle memory reaches in data team
When the upper limit, stop the gathered data from journal file, is led so as to avoid the data of acquisition are inserted into fully loaded data queue
Cause data loss problem.When due to acquiring daily record data from journal file, the daily record data of preset data is only acquired every time, and
And can judge data queue's EMS memory occupation situation before acquisition every time, rather than the daily record data changed in journal file is complete
Portion is sent to data queue after reading, and to advanced optimize data acquisition, avoids the loss of data of acquisition.
Based on the same technical idea, an embodiment of the present invention provides a kind of data acquisition devices, as shown in figure 4, the dress
The flow of collecting method can be executed by setting, including：
Module 401 is monitored, the change message for journal file in monitoring service system；
Processing module 402, for judging whether the daily record data amount in data queue is more than predetermined threshold value；If it is not, then root
The daily record data of setting quantity is read from the journal file according to the reading subscript of journal file described in journal file inventory,
The daily record data of reading is sent to the data queue and updates the reading of journal file described in the journal file inventory
Subscript, the reading subscript last read position for identifying the journal file in preceding primary reading；It is read determining
Daily record data when not including ending mark, return and judge daily record data amount in the data queue whether more than predetermined threshold value
The step of, until the daily record data read includes ending mark.
It should be noted that monitoring the function of module 401 and monitor 1021 in above-described embodiment in the embodiment of the present invention
It is identical, processing module and notification queue 1022, processing unit 1023, data queue 1024, transmitter 1025 in above-described embodiment
Function it is identical.
Optionally, the processing module 402 is additionally operable to：
Setting quantity is read from the journal file according to the reading subscript of journal file described in journal file inventory
Daily record data, after the daily record data of reading is sent to the data queue, by the daily record data in the data queue
It is sent to data-storage system；
When receiving the reception success message of the data-storage system feedback, extremely by the journal file list synchronization
Non-volatile memory device.
Optionally, the processing module 402 is additionally operable to：
Monitor task handling duration, the task handling duration are the daily record that setting quantity is read from the journal file
The daily record data of reading is extremely sent to the duration used in the data queue by data；
When determining that the task handling duration is more than first threshold, daily record data will be read from the journal file
Speed is reduced to the first preset value；It, will be from the journal file when determining that the task handling duration is less than second threshold
The speed for reading daily record data is promoted to the second preset value.
Optionally, the processing module 402 is additionally operable to：
In monitoring service system after the change message of journal file, judge whether the daily record data amount in data queue is big
Before predetermined threshold value, the change message is stored to notification queue；
The change message is obtained from the notification queue；
Determine that the change type of the change message is modification.
Before the return judges the step of whether the daily record data amount in the data queue is more than predetermined threshold value, also wrap
It includes：
When it includes ending mark to determine the daily record data read not, the change message is stored again to the notice
Queue.
Optionally, the processing module 402 is additionally operable to：
When it is newly-built to determine the change type, the note of the journal file is added in the journal file inventory
Record, the record include filename, read subscript；
When it is renaming to determine the change type, the file of journal file described in the journal file inventory is updated
Name；
When it is to delete to determine the change type, the record of journal file described in the journal file inventory is deleted.
An embodiment of the present invention provides a kind of computing device, which is specifically as follows desktop computer, portable
Computer, smart mobile phone, tablet computer, personal digital assistant (Personal Digital Assistant, PDA) etc..The meter
It may include central processing unit (Center Processing Unit, CPU), memory, input-output apparatus etc. to calculate equipment,
Input equipment may include keyboard, mouse, touch screen etc., and output equipment may include display equipment, such as liquid crystal display
(Liquid Crystal Display, LCD), cathode-ray tube (Cathode Ray Tube, CRT) etc..
Memory may include read-only memory (ROM) and random access memory (RAM), and be deposited to processor offer
The program instruction and data stored in reservoir.In embodiments of the present invention, memory can be used for storing collecting method
Program instruction；
Processor executes data acquisition for calling the program instruction stored in the memory according to the program of acquisition
Method.
An embodiment of the present invention provides a kind of computer storage media, the computer-readable recording medium storage has calculating
Machine executable instruction, the computer executable instructions are for making the computer implemented data acquisition method.
It should be understood by those skilled in the art that, the embodiment of the present invention can be provided as method or computer program product.
Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the present invention
Form.It is deposited moreover, the present invention can be used to can be used in the computer that one or more wherein includes computer usable program code
The shape for the computer program product implemented on storage media (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
Formula.
The present invention be with reference to according to the method for the embodiment of the present invention, the flow of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that can be realized by computer program instructions every first-class in flowchart and/or the block diagram
The combination of flow and/or box in journey and/or box and flowchart and/or the block diagram.These computer programs can be provided
Instruct the processor of all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine so that the instruction executed by computer or the processor of other programmable data processing devices is generated for real
The device for the function of being specified in present one flow of flow chart or one box of multiple flows and/or block diagram or multiple boxes.
These computer program instructions, which may also be stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that instruction generation stored in the computer readable memory includes referring to
Enable the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one box of block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device so that count
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, in computer or
The instruction executed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic
Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as
It selects embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
God and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to include these modifications and variations.
Claims (12)
1. a kind of collecting method, which is characterized in that including：
The change message of journal file in monitoring service system；
Judge whether the daily record data amount in data queue is more than predetermined threshold value；
If it is not, setting number is then read from the journal file according to the reading subscript of journal file described in journal file inventory
The daily record data of reading is sent to the data queue and updates day described in the journal file inventory by the daily record data of amount
The reading subscript of will file, the reading subscript last read position for identifying the journal file in preceding primary reading
It sets；
When it includes ending mark to determine the daily record data read not, return judges that the daily record data amount in the data queue is
No the step of being more than predetermined threshold value, until the daily record data read includes ending mark.
2. the method as described in claim 1, which is characterized in that the reading according to journal file described in journal file inventory
The daily record data that mark reads setting quantity from the journal file is removed, the daily record data of reading is sent to the data team
After row, further include：
Daily record data in the data queue is sent to data-storage system；
When receiving the reception success message of data-storage system feedback, by the journal file list synchronization to it is non-easily
The property lost storage device.
3. the method as described in claim 1, which is characterized in that further include：
Monitor task handling duration, the task handling duration are the daily record data that setting quantity is read from the journal file
The daily record data of reading is extremely sent to the duration used in the data queue；
When determining that the task handling duration is more than first threshold, the speed of daily record data will be read from the journal file
It is reduced to the first preset value；When determining that the task handling duration is less than second threshold, will be read from the journal file
The speed of daily record data is promoted to the second preset value.
4. the method as described in claims 1 to 3 is any, which is characterized in that the change of journal file in the monitoring service system
After more message, judge whether the daily record data amount in data queue is more than before predetermined threshold value, further includes：
The change message is stored to notification queue；
The change message is obtained from the notification queue；
Determine that the change type of the change message is modification；
Before the return judges the step of whether the daily record data amount in the data queue is more than predetermined threshold value, further include：
When it includes ending mark to determine the daily record data read not, the change message is stored again to the notice team
Row.
5. method as claimed in claim 4, which is characterized in that further include：
When it is newly-built to determine the change type, the record of the journal file, institute are added in the journal file inventory
Record is stated to include filename, read subscript；
When it is renaming to determine the change type, the filename of journal file described in the journal file inventory is updated；
When it is to delete to determine the change type, the record of journal file described in the journal file inventory is deleted.
6. a kind of data acquisition device, which is characterized in that including：
Module is monitored, the change message for journal file in monitoring service system；
Processing module, for judging whether the daily record data amount in data queue is more than predetermined threshold value；If it is not, then according to daily record text
The reading subscript of journal file described in part inventory reads the daily record data of setting quantity from the journal file, by reading
Daily record data is sent to the data queue and updates the reading subscript of journal file described in the journal file inventory, described
It reads subscript and last reads position in preceding primary reading for identifying the journal file；Determining the daily record data read
When not including ending mark, the step of whether the daily record data amount judged in the data queue is more than predetermined threshold value is returned to, directly
Daily record data to reading includes ending mark.
7. device as claimed in claim 6, which is characterized in that the processing module is additionally operable to：
The day of setting quantity is read from the journal file according to the reading subscript of journal file described in journal file inventory
After the daily record data of reading is sent to the data queue, the daily record data in the data queue is sent for will data
To data-storage system；
When receiving the reception success message of data-storage system feedback, by the journal file list synchronization to it is non-easily
The property lost storage device.
8. device as claimed in claim 6, which is characterized in that the processing module is additionally operable to：
Monitor task handling duration, the task handling duration are the daily record data that setting quantity is read from the journal file
The daily record data of reading is extremely sent to the duration used in the data queue；
When determining that the task handling duration is more than first threshold, the speed of daily record data will be read from the journal file
It is reduced to the first preset value；When determining that the task handling duration is less than second threshold, will be read from the journal file
The speed of daily record data is promoted to the second preset value.
9. the device as described in claim 6 to 8 is any, which is characterized in that the processing module is additionally operable to：
In monitoring service system after the change message of journal file, it is pre- to judge whether the daily record data amount in data queue is more than
If before threshold value, the change message is stored to notification queue；
The change message is obtained from the notification queue；
Determine that the change type of the change message is modification；
Before the return judges the step of whether the daily record data amount in the data queue is more than predetermined threshold value, further include：
When it includes ending mark to determine the daily record data read not, the change message is stored again to the notice team
Row.
10. device as claimed in claim 9, which is characterized in that the processing module is additionally operable to：
When it is newly-built to determine the change type, the record of the journal file, institute are added in the journal file inventory
Record is stated to include filename, read subscript；
When it is renaming to determine the change type, the filename of journal file described in the journal file inventory is updated；
When it is to delete to determine the change type, the record of journal file described in the journal file inventory is deleted.
11. a kind of computing device, which is characterized in that including：
Memory, for storing program instruction；
Processor, for calling the program instruction stored in the memory, according to acquisition program execute as claim 1 to
5 any one of them methods.
12. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer can
It executes instruction, the computer executable instructions are for making computer execute such as method described in any one of claim 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810030892.8A CN108399186A (en) | 2018-01-12 | 2018-01-12 | A kind of collecting method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810030892.8A CN108399186A (en) | 2018-01-12 | 2018-01-12 | A kind of collecting method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108399186A true CN108399186A (en) | 2018-08-14 |
Family
ID=63094961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810030892.8A Pending CN108399186A (en) | 2018-01-12 | 2018-01-12 | A kind of collecting method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108399186A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109688207A (en) * | 2018-12-11 | 2019-04-26 | 北京云中融信网络科技有限公司 | Log transmission method, apparatus and server |
| CN110263011A (en) * | 2019-06-21 | 2019-09-20 | 深圳前海微众银行股份有限公司 | A kind of log data acquisition method and device |
| CN110719233A (en) * | 2019-10-11 | 2020-01-21 | 北京百度网讯科技有限公司 | Method and device for sending information |
| CN110955653A (en) * | 2018-09-26 | 2020-04-03 | 阿里巴巴集团控股有限公司 | Data page processing method, device, equipment and computer system |
| CN111240952A (en) * | 2020-02-28 | 2020-06-05 | 北京百度网讯科技有限公司 | Log collection method and device |
| CN112084265A (en) * | 2020-09-11 | 2020-12-15 | 北京金山云网络技术有限公司 | Log data synchronization method and device, server and storage medium |
| CN112231288A (en) * | 2020-09-30 | 2021-01-15 | 北京金山云网络技术有限公司 | Log storage method and device and medium |
| CN112306827A (en) * | 2020-03-25 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Log collection device, method and computer readable storage medium |
| CN112698789A (en) * | 2020-12-29 | 2021-04-23 | 广州鼎甲计算机科技有限公司 | Data caching method, device, equipment and storage medium |
| CN113111137A (en) * | 2021-04-30 | 2021-07-13 | 深圳壹账通智能科技有限公司 | Distributed system log real-time display method, device, equipment and storage medium |
| CN113918358A (en) * | 2021-09-17 | 2022-01-11 | 远景智能国际私人投资有限公司 | Log sending method and device and log management system |
| CN114401289A (en) * | 2021-12-31 | 2022-04-26 | 深圳市麦谷科技有限公司 | Task batch uploading method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102447633A (en) * | 2011-12-29 | 2012-05-09 | 北京亿赞普网络技术有限公司 | Log transmission method and system |
| CN102467516A (en) * | 2010-11-04 | 2012-05-23 | 北京北方微电子基地设备工艺研究中心有限责任公司 | Log recording method, device and system in equipment control process |
| CN104283719A (en) * | 2014-10-28 | 2015-01-14 | 北京国双科技有限公司 | Log processing method and device and server |
| CN104572416A (en) * | 2014-12-29 | 2015-04-29 | 北京锐安科技有限公司 | Operation and maintenance data processing method and device |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
-
2018
- 2018-01-12 CN CN201810030892.8A patent/CN108399186A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102467516A (en) * | 2010-11-04 | 2012-05-23 | 北京北方微电子基地设备工艺研究中心有限责任公司 | Log recording method, device and system in equipment control process |
| CN102447633A (en) * | 2011-12-29 | 2012-05-09 | 北京亿赞普网络技术有限公司 | Log transmission method and system |
| CN104283719A (en) * | 2014-10-28 | 2015-01-14 | 北京国双科技有限公司 | Log processing method and device and server |
| CN104572416A (en) * | 2014-12-29 | 2015-04-29 | 北京锐安科技有限公司 | Operation and maintenance data processing method and device |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
Non-Patent Citations (1)
| Title |
|---|
| D_R_Y: ""Linux如何通过命令查看日志文件的某几行(中间几行或最后几行)"", 《HTTPS://WWW.CNBLOGS.COM/QIAOYANLIN/P/6706085.HTML》 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955653A (en) * | 2018-09-26 | 2020-04-03 | 阿里巴巴集团控股有限公司 | Data page processing method, device, equipment and computer system |
| CN109688207A (en) * | 2018-12-11 | 2019-04-26 | 北京云中融信网络科技有限公司 | Log transmission method, apparatus and server |
| CN110263011A (en) * | 2019-06-21 | 2019-09-20 | 深圳前海微众银行股份有限公司 | A kind of log data acquisition method and device |
| CN110719233A (en) * | 2019-10-11 | 2020-01-21 | 北京百度网讯科技有限公司 | Method and device for sending information |
| CN110719233B (en) * | 2019-10-11 | 2023-10-31 | 北京百度网讯科技有限公司 | Method and device for transmitting information |
| CN111240952B (en) * | 2020-02-28 | 2023-07-21 | 北京百度网讯科技有限公司 | Log acquisition method and device |
| CN111240952A (en) * | 2020-02-28 | 2020-06-05 | 北京百度网讯科技有限公司 | Log collection method and device |
| CN112306827A (en) * | 2020-03-25 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Log collection device, method and computer readable storage medium |
| CN112084265A (en) * | 2020-09-11 | 2020-12-15 | 北京金山云网络技术有限公司 | Log data synchronization method and device, server and storage medium |
| CN112231288A (en) * | 2020-09-30 | 2021-01-15 | 北京金山云网络技术有限公司 | Log storage method and device and medium |
| CN112698789B (en) * | 2020-12-29 | 2022-03-15 | 广州鼎甲计算机科技有限公司 | Data caching method, device, equipment and storage medium |
| CN112698789A (en) * | 2020-12-29 | 2021-04-23 | 广州鼎甲计算机科技有限公司 | Data caching method, device, equipment and storage medium |
| CN113111137A (en) * | 2021-04-30 | 2021-07-13 | 深圳壹账通智能科技有限公司 | Distributed system log real-time display method, device, equipment and storage medium |
| CN113918358A (en) * | 2021-09-17 | 2022-01-11 | 远景智能国际私人投资有限公司 | Log sending method and device and log management system |
| CN114401289A (en) * | 2021-12-31 | 2022-04-26 | 深圳市麦谷科技有限公司 | Task batch uploading method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108399186A (en) | A kind of collecting method and device | |
| CN110502494B (en) | Log processing method and device, computer equipment and storage medium | |
| CN106815254B (en) | Data processing method and device | |
| CN108090079B (en) | Chat record processing method and device of instant chat tool | |
| CN114091704B (en) | Alarm suppression method and device | |
| CN101557427A (en) | Method for providing diffluent information and realizing the diffluence of clients, system and server thereof | |
| CN110334072A (en) | A kind of distributed file system, file updating method and device | |
| CN114223189B (en) | Duration statistics method, device, electronic device and computer readable medium | |
| CN108763044A (en) | A kind of log processing method and device | |
| CN107193869A (en) | The method and device of data access in industrial Internet of Things cloud platform | |
| CN112214390A (en) | Test case generation method, device, system, equipment and medium | |
| CN110471945A (en) | Processing method, system, computer equipment and the storage medium of alive data | |
| CN112765152B (en) | Method and apparatus for merging data tables | |
| CN109857802A (en) | Daily record data synchronous method, device, equipment and computer readable storage medium | |
| CN110321364A (en) | Transaction data querying method, device and the terminal of credit card management system | |
| CN110427134B (en) | Data processing method, device and equipment for instant messaging and readable storage medium | |
| CN109450977A (en) | Data cached push, update method and device | |
| CN106844744B (en) | Click model application method and device and search system | |
| CN110245129A (en) | Distributed global data deduplication method and device | |
| CN111722973B (en) | Event timeout monitoring method, system and storage medium | |
| CN109143878B (en) | Remote signaling message and SOE message processing method for multi-channel acquisition channel | |
| US9798626B2 (en) | Implementing change data capture by interpreting published events as a database recovery log | |
| CN114971714A (en) | Accurate customer operation method based on big data label and computer equipment | |
| CN107656999B (en) | Call history tracing method and device, electronic equipment and storage medium | |
| CN110534136A (en) | The method and apparatus of recording |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180814 |
|
| RJ01 | Rejection of invention patent application after publication |