CN103164511B - Method for storage event log to automatically filter repeated jitter data - Google Patents
Method for storage event log to automatically filter repeated jitter data Download PDFInfo
- Publication number
- CN103164511B CN103164511B CN201310055768.4A CN201310055768A CN103164511B CN 103164511 B CN103164511 B CN 103164511B CN 201310055768 A CN201310055768 A CN 201310055768A CN 103164511 B CN103164511 B CN 103164511B
- Authority
- CN
- China
- Prior art keywords
- event log
- record
- data
- layer
- shake
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 32
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000001914 filtration Methods 0.000 claims abstract description 14
- 238000004458 analytical method Methods 0.000 claims abstract description 11
- 238000012216 screening Methods 0.000 claims abstract description 4
- 230000014759 maintenance of location Effects 0.000 claims description 3
- 230000008878 coupling Effects 0.000 claims description 2
- 238000010168 coupling process Methods 0.000 claims description 2
- 238000005859 coupling reaction Methods 0.000 claims description 2
- 230000001934 delay Effects 0.000 claims 1
- 230000002159 abnormal effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 206010044565 Tremor Diseases 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a method for a storage event log to automatically filter repeated jitter data. The method comprises the steps: setting two layers of filtering, screening a normal event log record through a first layer of filtering when a record (the event log record) which waits to be analyzed from a message queue, sending the normal event log record into a buffer memory, and saving the time of comparison one by one; and enabling remaining records after the first layer of filtering to abandon the jitter data through a second layer of filtering, reducing redundancy of the data, and sending an event log record filtered through the second layer of filtering to a nonvolatile storage medium, wherein the remaining records are taken out from the massage queue and wait to be analyzed. After a period of preset time, the data in the record buffer memory are sequentially stored into corresponding storage areas according to types. According to the method, in the analysis stage of the formed event log, an effective jitter data filter algorithm is added, a traditional data ergodic comparison algorithm is abandoned, the jitter data are efficiently filtered, and redundant data are reduced.
Description
Technical field
The present invention relates to shaking the detection of data and filtering, specifically a kind of storage event log automatic fitration repeats to tremble
The method of dynamic data.
Background technology
Current telecommunications equipment develops towards high speed, efficient direction, continual operation for a long time, occurs unavoidably and expects not
The special status arriving, perhaps some abnormal phenomenas flash across, do not realized by people in time, but accumulate through long,
More or less system can be impacted, so self recording apparatus abnormal information becomes particularly important, but huge data
Amount brings great inconvenience to the later stage analysis of causes, how the storage more significant of effective and reasonable planning abnormal information.By
This, event log standardization storage is born.We are classified these abnormal informations one by one as event log recorder
Storage forms event log, increases timestamp simultaneously, and huge data is effectively managed.
Through actual test checking, although any one event log recorder can record moment, a large amount of weights are piled up
The data of multiple redundancy, that is, repeat to shake data, may be also referred to simply as shaking data, this not only threatens to device storage capacity,
It has been greatly reduced the efficiency of positioning problems and the analysis of causes.
Store will be through three as shown in figure 1, brand-new event log recorder is from producing to for event log storing process
The individual stage：Event log recorder inbound message queue waits, and analyzes event log recorder, and storage event log recorder forms event
Daily record.Event log recorder by analysis, can insert the appropriate location of storage medium.But the event log recorder of shake in a large number
The data redundancy being brought, not only valueless at all to case study, return storage medium and bring serious burden.So-called shake data
The identical recordings repeating in intervals.If simple by record between multilevel iudge one by one its be whether
Shake data, not only takes, and frequent visit hardware store medium also can reduce hardware longevity, is eventually delayed system
Overall operation.
Content of the invention
For defect present in prior art, it is an object of the invention to provide a kind of storage event log automatic fitration
The method repeating to shake data, in the analysis phase forming event log, effectively shakes data filtering algorithm by increasing, gives up
Abandon traditional data traversal comparison algorithm, high efficiency filter shakes data, reduce redundant data.
For reaching object above, the present invention adopts the technical scheme that：
A kind of storage event log automatic fitration repeat shake data method it is characterised in that：Setting two-layer filters, when
When taking out the record that waiting is analyzed from message queue, normal event log recorder is selected by ground floor sieves, and will
Normal event log recorder sends into record buffer memory, saves the time compared one by one；Then to remaining after ground floor filters
The waiting analysis taken out from message queue record, then filtered by the second layer and give up to fall shake data, minimizing data
Redundancy, will send into non-volatile memory medium by remaining event log recorder after the second layer filters；
When, after one section of timing made an appointment, the data in record buffer memory being stored successively by type corresponding
Memory block.
On the basis of technique scheme, event log recorder increases field mark shake number of times and shake duration,
When giving up to fall shake data by second layer filtration, calculate shake number of times and shake duration and write corresponding event log
In the corresponding field of record.
On the basis of technique scheme, time array is set, preserves the final time of all types of event log storages,
In this, as ground floor filtercondition；
Increase an interim record buffer memory, second layer filtercondition is used as by the traversal screening of small range, traversal is adopted
With mode from back to front, reduce number of comparisons.
On the basis of technique scheme, in order to avoid interim record buffer memory is overflowed, realize obtaining event note in real time
Record, enables house dog, the interim record buffer memory of Timing Synchronization is to non-volatile memory medium.
Storage event log automatic fitration of the present invention repeats the method shaking data, is forming dividing of event log
In the analysis stage, effectively shake data filtering algorithm by increasing, discard traditional data traversal comparison algorithm, high efficiency filter is shaken
Data, reduces redundant data.Have the advantages that：
One, algorithm can shake data with high efficiency filter, without manual intervention, uses manpower and material resources sparingly.
Two, reduce redundant data, increase memory space, provide facility for later data analysis.
Three, algorithm clear layer understands it is easy to later maintenance and extension.
Brief description
The present invention has drawings described below：
Fig. 1：Event log storing process,
Fig. 2：Analysis phase two-layer filters,
Fig. 3：Shake data algorithm flow process.
Specific embodiment
Below in conjunction with accompanying drawing, the present invention is described in further detail.
As shown in Fig. 2 storage event log automatic fitration of the present invention repeats the method shaking data, two-layer is set
Filter, when from message queue（Also known as event log queue）The middle record taking out waiting analysis（Event log recorder）When, lead to
Cross ground floor sieves and select normal event log recorder, and normal event log recorder is sent into record buffer memory, save
The time compared one by one；Then the record to the waiting analysis taken out from message queue remaining after ground floor filters,
Filtered by the second layer again and give up to fall shake data, reduce the redundancy of data, will after the second layer filters remaining event
Log recording sends into non-volatile memory medium（Store corresponding memory block by type）.
When through one section of timing made an appointment（This function is realized by house dog）Afterwards, you can by record buffer memory
Data store corresponding memory block successively by type（Refer to the memory block in non-volatile memory medium）.
The data that ground floor filters needs " feeding record buffer memory ", and the data that the second layer filters then " is sent into non-volatile
Storage medium ", this is two kinds of different storage modes, and ground floor filters the record buffer memory being directly stored in internal memory, is to deposit at a high speed
Storage and be interim storage, primarily to carrying out second layer filtration, the second layer filter be storage in storage medium（I.e. FRAM）,
The all accurate recordings of content in FRAM are stored after two layers are filtered.
For differentiation " normal event log recorder is selected by ground floor sieves ", " filtered by the second layer and give up
Fall to shake data " and " remaining event log recorder after the second layer filters ", now it is illustrated with：We set
Shake interval time is 2s.So in 2s, the record only receiving as " selects normal thing by ground floor sieves
Part log recording ", and the identical record of a plurality of content receiving（It is assumed to be 10）Then think including " by second
Layer filters gives up to fall shake data ", for this plurality of record, finally store the record in FRAM and only have one, this is i.e.
" remaining event log recorder after the second layer filters ".I.e.：Through two layers filter remaining namely correctly record, just
It is final this record retaining.Remaining 9 is shake, is eventually abandoned.
On the basis of technique scheme, event log recorder increases field mark shake number of times and shake duration,
When giving up to fall shake data by second layer filtration, calculate shake number of times and shake duration and write corresponding event log
In the corresponding field of record.
On the basis of technique scheme, time array is set, preserves the final time of all types of event log storages,
In this, as ground floor filtercondition；
Increase an interim record buffer memory, by the traversal screening of small range（Data traversal comparison algorithm）As second
Layer filtercondition, traversal using by the way of from back to front, minimizing number of comparisons.
On the basis of technique scheme, in order to avoid interim record buffer memory is overflowed, realize obtaining event note in real time
Record, enables house dog, the interim record buffer memory of Timing Synchronization is to non-volatile memory medium.
As shown in figure 3, the concrete steps for an embodiment adopting the present invention program：
The first step, for process recording in message queue and generate event log event log module software in,
Newly-increased time array T [N], N is the type sum of event log recorder, and the reception of storage N class event log recorder is last respectively
Time, in this, as ground floor filtercondition.
Second step, arranges one piece of blotter buffer zone in internal memory, this interim record buffer memory preserved between the time of specifying
The final time receiving every interior most 200 receiving effectively record and every identical recordings.Specified time interval
Determination can select according to actual needs, for example：1s, 2s, 3s, are usually no more than 5s.
3rd step, starts task, successively from message queue（Also known as event log queue）Head of the queue takes out event log note
Record.
4th step, ground floor filter, compare current record receive the time be followed by most with corresponding types record in time array
Between the time receiving, exceed specified time interval and then save secondary filter, directly this record is stored in blotter caching.Otherwise continue etc.
Treat that the second layer filters.
5th step, the second layer filters, from back to front 200 blotters of traversal caching successively, find coupling record and when
Between interval less than specify time interval, judge its be shake record, abandon, be otherwise stored in blotter caching.
6th step, waits house dog timing to arrive, and each bar logout write in caching blotter successively is deposited
Storage media.
The scheme that the present invention is given has been successfully applied to fonst1000, fonst3000, fonst4000, fonst5000
Etc. in numerous systems.
The content not being described in detail in this specification belongs to prior art known to professional and technical personnel in the field.
Claims (3)
1. a kind of storage event log automatic fitration repeat shake data method it is characterised in that：Setting two-layer filter, when from
When taking out the record that waiting is analyzed in message queue, normal event log recorder is selected by ground floor sieves, and will just
Normal event log recorder sends into record buffer memory, saves the time compared one by one；Then to remaining after ground floor filters
The record of the waiting analysis taken out from message queue, then give up to fall shake data by second layer filtration, reduce the superfluous of data
Remaining, will send into non-volatile memory medium by remaining event log recorder after the second layer filters；
When, after one section of timing made an appointment, the data in record buffer memory being stored successively by type and deposits accordingly
Storage area；
Wherein, time array is set, preserves the final time of all types of event log storages, in this, as ground floor filtering rod
Part；Increase an interim record buffer memory, second layer filtercondition is used as by the traversal screening of small range, traversal is using from rear
Mode forward, reduces number of comparisons；
Specifically, ground floor filters, when comparing current record and receiving that in time and time array, corresponding types record finally receives
Between, exceed specified time interval and then save secondary filter, update corresponding time array, directly this record is stored in blotter and delays
Deposit, otherwise continue waiting for the second layer and filter, wherein it is intended that time interval is usually no more than 5s；The second layer filter, from back to front according to
Secondary traversal blotter caching, finds coupling record and time interval is less than and specifies time interval, judges that it is shake record, throws
Abandon, be otherwise stored in blotter caching.
2. as claimed in claim 1 storage event log automatic fitration repeat shake data method it is characterised in that：Event
Log recording increases field mark shake number of times and shake duration, when giving up to fall shake data by second layer filtration, meter
Calculate shake number of times and shake duration and write in the corresponding field of corresponding event log recorder.
3. as claimed in claim 1 storage event log automatic fitration repeat shake data method it is characterised in that：In order to
Avoid interim record buffer memory to overflow, realize obtaining logout in real time, enable house dog, the interim record buffer memory of Timing Synchronization
To non-volatile memory medium.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310055768.4A CN103164511B (en) | 2013-02-21 | 2013-02-21 | Method for storage event log to automatically filter repeated jitter data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310055768.4A CN103164511B (en) | 2013-02-21 | 2013-02-21 | Method for storage event log to automatically filter repeated jitter data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103164511A CN103164511A (en) | 2013-06-19 |
| CN103164511B true CN103164511B (en) | 2017-02-08 |
Family
ID=48587599
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310055768.4A Active CN103164511B (en) | 2013-02-21 | 2013-02-21 | Method for storage event log to automatically filter repeated jitter data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103164511B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10592308B2 (en) | 2015-04-30 | 2020-03-17 | Micro Focus Llc | Aggregation based event identification |
| CN105049260B (en) * | 2015-08-24 | 2019-03-19 | 浪潮(北京)电子信息产业有限公司 | Blog management method and device |
| CN105868340A (en) * | 2016-03-28 | 2016-08-17 | 乐视控股（北京）有限公司 | Log storage method and device |
| CN106776249B (en) * | 2016-11-28 | 2020-07-28 | 华迪计算机集团有限公司 | Method and system for processing concurrently generated service logs |
| CN109118609B (en) * | 2017-06-26 | 2020-11-13 | 深圳市道通智能航空技术有限公司 | Unmanned aerial vehicle flight log recording method and device, storage medium and computer equipment |
| CN109302300B (en) * | 2017-07-25 | 2022-03-15 | 阿里巴巴集团控股有限公司 | Data distribution and processing method, system and computer readable recording medium |
| CN110399095B (en) * | 2019-06-12 | 2022-12-02 | 厦门网宿有限公司 | Storage space statistical method and device |
| CN111143317A (en) * | 2019-12-29 | 2020-05-12 | 北京浪潮数据技术有限公司 | Log storage method, device and equipment and readable storage medium |
| CN111813765B (en) * | 2020-06-19 | 2024-04-12 | 北京金堤科技有限公司 | Method, device, electronic equipment and computer readable medium for processing abnormal data |
| CN117707439A (en) * | 2023-08-22 | 2024-03-15 | 荣耀终端有限公司 | Log printing method and related device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1949725A (en) * | 2006-11-22 | 2007-04-18 | 华为技术有限公司 | System and method for filtering journal message |
| CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101320348A (en) * | 2008-06-25 | 2008-12-10 | 中兴通讯股份有限公司 | Log function implementing method of embedded system |
-
2013
- 2013-02-21 CN CN201310055768.4A patent/CN103164511B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1949725A (en) * | 2006-11-22 | 2007-04-18 | 华为技术有限公司 | System and method for filtering journal message |
| CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103164511A (en) | 2013-06-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103164511B (en) | Method for storage event log to automatically filter repeated jitter data | |
| CN109164980B (en) | Aggregation optimization processing method for time sequence data | |
| US10616098B2 (en) | Apparatus and methods for forwarding data packets captured from a network | |
| JP2022511233A (en) | Metadata generation at the storage edge | |
| CN102158355B (en) | Log event correlation analysis method and device capable of concurrent and interrupted analysis | |
| CN201993755U (en) | Data filtration, compression and storage system of real-time database | |
| US9304692B2 (en) | Solid-state drive command grouping | |
| CN105630582B (en) | Method and system for handling conflict of receiving resources in satellite ground station | |
| US8918620B2 (en) | Storage control apparatus, storage system and method | |
| WO2020192243A1 (en) | Data storage apparatus and method, and readable storage medium | |
| CN104850480A (en) | Method and device for testing performance of hard disk of high-density storage server | |
| US11366714B2 (en) | Behavior-driven die management on solid-state drives | |
| CN109327680A (en) | Multi-path video data concurrently stores and read method | |
| CN102760168B (en) | Method and device for scanning fragmented files | |
| CN113542263B (en) | Firewall policy migration method and device | |
| US20160139823A1 (en) | Methods and systems for optimal snapshot distribution within a protection schedule | |
| CN101464790B (en) | Command scheduling device and method thereof | |
| WO2023088201A1 (en) | Method, apparatus and device for detecting abnormal vehicle | |
| CN104793903A (en) | Video data writing method, device and system based on IO sequencing | |
| CN101729421A (en) | Storage method and device based on time division multiplex | |
| CN105912439A (en) | Method, device and system for computer monitoring | |
| EP1774704A1 (en) | Improved networks statistics processing device | |
| TW202447415A (en) | System and method for scheduling requests in storage system and storage system | |
| CN105701738A (en) | Area architectural energy consumption platform data acquisition processing method and device for realizing same | |
| CN1905590A (en) | Single chip information acquiring method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |