CN103593277A - Log processing method and system - Google Patents
Log processing method and system Download PDFInfo
- Publication number
- CN103593277A CN103593277A CN201210290200.6A CN201210290200A CN103593277A CN 103593277 A CN103593277 A CN 103593277A CN 201210290200 A CN201210290200 A CN 201210290200A CN 103593277 A CN103593277 A CN 103593277A
- Authority
- CN
- China
- Prior art keywords
- script
- log processing
- format
- written
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
- Stored Programmes (AREA)
Abstract
The invention relates to a log processing method which includes that a script language interpreter is built, a log formatting processing function is directly packaged in the script language interpreter, corresponding log processing scripts are built respectively aiming at scripts of different formats to form a dynamic script base, various formatting operations in the log processing scripts are finished through the function, a log with a certain format is loaded, a corresponding log processing script is loaded from the dynamic script base according to the format of the log, and the script language interpreter interprets and executes the loaded log processing script to finish the log formatting operation and output a processing result. By means of the log processing method, logs from different services and with different formats can be efficiently processed into specific formats to be output. A log processing system is further provided.
Description
Technical field
The present invention relates to log processing technology, particularly a kind of log processing method and device based on dynamic script language.
Background technology
In existing Network, daily record is a requisite part, by the analyzing and processing to business diary, can recognize the management state of business, when server goes wrong, also can, by daily record is analyzed and found the problem, facilitate the solution of problem simultaneously.Current, journal file is all to deposit in file device or data library device with original form, in to service operation data analysis and statistics, need to carry out analyzing and processing to the daily record data file of unprocessed form, with the statistics that obtains wanting.
Journal file is collected from various object-based devices (such as main frame, fire wall, database, server etc.) by security management center device, because the journal format that various device produces is all different, the daily record of how to unify to process these different-formats becomes an important problem.In prior art, for the object-based device of every type, all correspondingly write specific format instrument and go by the contents extraction in daily record out, and be filled up in the model of a unified form and go, finally generate the daily record of consolidation form.
Log processing method existent defect of the prior art, due to the journal format for each object-based device, all need the code of writing a kind of correspondence to process, when increasing new object-based device or object-based device and changed journal format, needs are increased and revise corresponding formatter and just can accomplish correct collection and format daily record, in security management center device, need corresponding modification code, again issue, can cause like this device maintenance large, cause device upgrading frequent, and change code at every turn and all need to recompilate device.
Summary of the invention
In view of this, be necessary to provide a kind of log processing method and system, it can be processed into specific form by the journal formatting with different-format from different business efficiently and export.
, comprising: set up a script language interpreter, in this script language interpreter, directly encapsulate journal formatting processing function; Thereby set up respectively corresponding log processing script for the script of different-format, form dynamic script storehouse, in this log processing script, various format manipulations are completed by above-mentioned function; Be written into the daily record with certain format; According to the form of described daily record, from dynamic script storehouse, be written into corresponding log processing script; By described script language interpreter, explain and carry out the log processing script being written into, complete journal formatting and operate and export result.
, comprising: dynamic script storehouse, for storing the log processing script of setting up respectively for the script of different-format, in this log processing script, various format manipulations are completed by function; And script language interpreter, comprising: daily record is written into unit, for being written into the daily record with certain format; Script is written into unit, and for be written into corresponding log processing script from dynamic script storehouse according to the form of described daily record, the format manipulation in this log processing script is realized by function; Script Interpretation unit, for being interpreted into corresponding execution instruction by the Function Solution of the log processing script being written into; And logical execution units, for carrying out described execution instruction.
In above-mentioned log processing method and system: by function being directly encapsulated in to the inside of script language interpreter, thereby can realize the foundation that completes the log processing script of different-format in very succinct mode, promoted the efficiency formaing when the daily record of new business imports.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of a kind of log processing method of providing of the first embodiment.
Fig. 2 is the schematic diagram of the daily record that will format in the first embodiment.
Fig. 3 is the schematic diagram after the journal formatting that will format in the first embodiment.
Fig. 4 is the structured flowchart of a kind of log processing system of providing of the second embodiment.
Embodiment
For further setting forth the present invention, realize technological means and the effect that predetermined goal of the invention is taked, below in conjunction with accompanying drawing and preferred embodiment, to the log processing method proposing according to the present invention and embodiment, structure, feature and the effect thereof of system, be described in detail as follows.
The embodiment of the present invention provides a kind of log processing method, and it can be processed into specific form by the journal formatting with different-format from different business efficiently and export.Different forms for example refers to the difference of format character of field that daily record comprises, sequence, use etc.
In above-mentioned log processing method, first build a kind of new script interpretative code and script interpreter thereof, the script that is met this script interpretative code by script interpreter explanation is processed to complete the format of daily record.On the other hand, for the daily record with different-format from different business, build respectively corresponding log processing script from forming dynamic script storehouse.When having the daily record of new business to import, according to this script interpretative code, rebuild the log processing script that meets new importing business.When having existing business to carry out log processing again, from this dynamic script storehouse, remove corresponding script.
Above-mentioned script interpretative code key feature is as follows:
1, the syntactic structure of sectional type
Script divides three blocks to form: initialization block (begin), circulation block (do) and end block (end).Wherein begin block is placed initialization logic, its code is only carried out one time before log processing, do block is placed the logic that daily record is cleaned, formatd, when processing each journal entries (general a line corresponding to daily record text), code is all carried out one time, end district code block is placed and is finished logic, and code is carried out one time after log processing finishes.The structure of three above-mentioned blocks is as follows:
2, script functional character
Script has been realized variable assignments statement, condition judgment statement (if/else), branch's judgement statement (switch), loop statement (for/while), be convenient to control and clean, format logic, and the conventional logic that some journal formattings of encapsulation are processed, for example, network address decoding (URLdecode), coding turns (UTF8 and GBK) mutually, regular expression extracts, character string intercepting, print screen, from network address (URL), extract certain inquiry (query) parameter etc., these conventional logics are packaged into the function in dynamic script language, be equivalent to the function of C language, in script, can reach by these functions of nesting allocation the object of journal formatting.
3, the function providing
The conventional logic that journal formatting is processed is all packaged into the function in script, mainly comprises:
Print: print;
Ip2local: ip is converted to physical location, comprise the information in country, province, city;
Detect_encoding: encoding detection, automatically identify the type of coding of designated character string;
Iconv: carry out code conversion, Chinese is turned mutually between the codings such as utf8, gbk;
Parse_url: the query argument that extracts domain, port, appointment from url;
Url_decode: the URL character string that decoding has been encoded;
Log_get_field: from a character string, intercept out the block that appointment is initial, full stop is corresponding, as " [" with "] " the frame section of living, the section that double quotation marks encases;
Detect_browser: the browser type that identifies client from the bit string of web page server record.
The workflow of script interpreter
1) script rendering engine is written into after script, and script is carried out to lexical analysis, reads in after script, identifies the key word in script, identifier, and constant, operator and boundary symbol, and by mark mode, represent the word sequence that identifies.
2) script engine is on the basis of lexical analysis, according to syntax rule, word sequence is formed to all kinds of syntactical units, translate into execution instruction one by one, the execution instruction here represents as node with variable length multi-element type, form is for example: [< command identification >, < number of parameters >, < operand 1>, < operand 2>, < operand n>], for this code of $ a=37+39, can translate into:
Node001:ADD,2,37,39
Node002:ASSIGN,2,<point_to_var_a>,<point_to_node001>
Wherein operand can be node pointer, thereby can in internal memory, by tree structure, express the logic of execution.
3) the pending daily record being written into for every a line, the actuating logic of all using this tree structure to represent is processed, thereby obtains the result after journal formatting is processed.
The process flow diagram of a kind of log processing method that Fig. 1 provides for the first embodiment; As shown in Figure 1, the method comprises:
Step S110, set up a script language interpreter, direct encapsulation journal formatting processing function in this script language interpreter; Be appreciated that above-mentioned journal formatting processing refers to the functions such as aforesaid print with function.
Thereby step S120, set up respectively corresponding log processing script form dynamic script storehouse for the script of different-format, in this log processing script, various format manipulations are completed by above-mentioned function; Be appreciated that and when new business imports, in described dynamic script storehouse, set up the log processing script corresponding with the new business diary form importing.When having existing business to carry out log processing again, from this dynamic script storehouse, remove corresponding script.
Consult Fig. 2, it is the schematic diagram of the log content of a certain business derivation, according to above-mentioned definite language feature and rule, provides a log processing example script as follows:
Be appreciated that the daily record for different-format, all can set up the log processing script of similar form.
Step S130, be written into the daily record with certain format; When above-mentioned daily record is for example moved by miscellaneous service, produce, because different business depends on different systems, its developer is also not quite similar, so the form of daily record is varied.
Step S140, according to the form of described daily record, from dynamic script storehouse, be written into corresponding log processing script; The form of daily record for example can be judged according to its file suffixes name, determine, or the journal entries repeating in direct-detection journal file is determined according to the description field of journal file head.
Step S150, the log processing script being written into by above-mentioned script language interpreter explanation execution, complete journal formatting and operate and export result.Consult Fig. 3, the schematic diagram of its daily record that is Fig. 2 after above-mentioned log processing script format.
Be appreciated that the three stage structure of the script interpretative code based on above-mentioned, explain that carrying out the log processing script being written into comprises:
Explain the initialization block of carrying out described log processing script; For each record in described daily record, carry out the format block of described log processing script; And the end block of explaining the described log processing script of execution.
For each block, in concrete explanation implementation, according to lexical analysis, identify key word in described log processing script, identifier, constant, function and boundary's symbol, and by mark mode, represent the word sequence that identifies; Convert described word sequence to execution instruction; Then carry out described execution instruction.Concrete principle has been described in detail as above, does not repeat them here.
In the log processing method of the present embodiment, by function being directly encapsulated in to the inside of script language interpreter, thereby can realize the foundation that completes the log processing script of different-format in very succinct mode, promoted the efficiency formaing when the daily record of new business imports.
The structured flowchart of the log processing system that Fig. 4 provides for the second embodiment.As shown in Figure 4, log processing system 200 comprises: dynamic script storehouse 210 and script language interpreter 220.
Daily record is written into unit 221, for being written into the daily record with certain format;
Script is written into unit 222, and for be written into corresponding log processing script from dynamic script storehouse according to the form of described daily record, the format manipulation in this log processing script is realized by function;
In addition, log processing system 200 also can comprise that script sets up unit 230, for set up the log processing script corresponding with the business diary form of new importing in described dynamic script storehouse when new business imports.
About other details of log processing system 200, also can be referring to figs. 1 to Fig. 3 and associated description.
In the log processing system of the present embodiment, by function being directly encapsulated in to the inside of script language interpreter, thereby can realize the foundation that completes the log processing script of different-format in very succinct mode, promoted the efficiency formaing when the daily record of new business imports.
In addition, the embodiment of the present invention also provides a kind of computer-readable recording medium, is stored with computer executable instructions, and above-mentioned computer-readable recording medium is for example for example CD, hard disk or flash memory of nonvolatile memory.
Above-mentioned computing machine can be held instruction for allowing computing machine or similar arithmetic unit complete the various operations of above-mentioned log processing method.
The above, it is only preferred embodiment of the present invention, not the present invention is done to any pro forma restriction, although the present invention discloses as above with preferred embodiment, yet not in order to limit the present invention, any those skilled in the art, do not departing within the scope of technical solution of the present invention, when can utilizing the technology contents of above-mentioned announcement to make a little change or being modified to the equivalent embodiment of equivalent variations, in every case be not depart from technical solution of the present invention content, any brief introduction of above embodiment being done according to technical spirit of the present invention is revised, equivalent variations and modification, all still belong in the scope of technical solution of the present invention.
Claims (8)
1. a log processing method, comprising:
Set up a script language interpreter, in this script language interpreter, directly encapsulate journal formatting processing function;
Thereby set up respectively corresponding log processing script for the script of different-format, form dynamic script storehouse, in this log processing script, various format manipulations are completed by above-mentioned function;
Be written into the daily record with certain format;
According to the form of described daily record, from dynamic script storehouse, be written into corresponding log processing script;
By described script language interpreter, explain and carry out the log processing script being written into, complete journal formatting and operate and export result.
2. log processing method as claimed in claim 1, is characterized in that, explains that carrying out the log processing script being written into comprises:
Explain the initialization block of carrying out described log processing script;
For each record in described daily record, repeat the format block of described log processing script; And
Explain the end block of carrying out described log processing script.
3. log processing method as claimed in claim 1, is characterized in that, explains that carrying out the log processing script being written into comprises:
According to lexical analysis, identify key word in described log processing script, identifier, constant, function and boundary's symbol, and by mark mode, represent the word sequence that identifies;
Convert described word sequence to execution instruction; And
Carry out described execution instruction.
4. log processing method as claimed in claim 1, is characterized in that, also comprises:
When importing, new business sets up the log processing script corresponding with the new business diary form importing in described dynamic script storehouse.
5. a log processing system, comprising:
Dynamic script storehouse, for storing the log processing script of setting up respectively for the script of different-format, in this log processing script, various format manipulations are completed by function; And
Script language interpreter, comprising:
Daily record is written into unit, for being written into the daily record with certain format;
Script is written into unit, and for be written into corresponding log processing script from dynamic script storehouse according to the form of described daily record, the format manipulation in this log processing script is realized by function;
Script Interpretation unit, for being interpreted into corresponding execution instruction by the Function Solution of the log processing script being written into; And
Logical execution units, for carrying out described execution instruction.
6. log processing system as claimed in claim 5, is characterized in that, described logical execution units is used for:
Carry out the execution instruction corresponding with the initialization block of described log processing script;
For each record in described daily record, repeat the execution instruction corresponding with the format block of described log processing script; And
Carry out the execution instruction corresponding with the end block of described log processing script.
7. log processing system as claimed in claim 5, is characterized in that, described script Interpretation unit comprises:
Lexical analysis unit, for identify key word, identifier, constant, function and boundary's symbol of described log processing script according to lexical analysis, and represents the word sequence that identifies by mark mode;
Parsing unit, for generating and carry out instruction according to described word sequence analysis.
8. log processing system as claimed in claim 5, is characterized in that, also comprises:
Script is set up unit, for set up the log processing script corresponding with the new business diary form importing in described dynamic script storehouse when new business imports.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210290200.6A CN103593277A (en) | 2012-08-15 | 2012-08-15 | Log processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210290200.6A CN103593277A (en) | 2012-08-15 | 2012-08-15 | Log processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103593277A true CN103593277A (en) | 2014-02-19 |
Family
ID=50083429
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210290200.6A Pending CN103593277A (en) | 2012-08-15 | 2012-08-15 | Log processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103593277A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104994128A (en) * | 2015-05-15 | 2015-10-21 | 北京网康科技有限公司 | Data coding type identifying and transcoding method and device |
| CN105656706A (en) * | 2014-11-14 | 2016-06-08 | 北京通达无限科技有限公司 | Business data processing method and device |
| CN109062762A (en) * | 2018-07-26 | 2018-12-21 | 浙江数链科技有限公司 | Formatted log method for pushing and device |
| CN109669848A (en) * | 2017-10-16 | 2019-04-23 | 比亚迪股份有限公司 | Log generation method, application server, log server and log generating system |
| CN110049017A (en) * | 2019-03-22 | 2019-07-23 | 口碑(上海)信息技术有限公司 | Message interworking unit and method between heterogeneous platform |
| CN110858192A (en) * | 2018-08-23 | 2020-03-03 | 阿里巴巴集团控股有限公司 | Log query method and system, log checking system and query terminal |
| CN110874350A (en) * | 2019-10-14 | 2020-03-10 | 支付宝(杭州)信息技术有限公司 | Method and device for processing structured log data |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1499374A (en) * | 2002-11-04 | 2004-05-26 | 华为技术有限公司 | Automatic approach for unit testing |
| US20040254919A1 (en) * | 2003-06-13 | 2004-12-16 | Microsoft Corporation | Log parser |
| CN1670708A (en) * | 2004-03-17 | 2005-09-21 | 联想(北京)有限公司 | Management method for computer log |
| CN1877525A (en) * | 2005-05-10 | 2006-12-13 | 西加特技术有限责任公司 | Protocol scripting language for safe execution in embedded system |
| CN1889459A (en) * | 2006-07-20 | 2007-01-03 | 华为技术有限公司 | Journal formatting unit and method in safety management center system |
| CN101126981A (en) * | 2007-09-28 | 2008-02-20 | 华为技术有限公司 | Script order registration method, method and device for calling source program code |
| KR20090000278A (en) * | 2007-02-13 | 2009-01-07 | 삼성전자주식회사 | Log Analysis System and Its Log Analysis Method |
| CN101409655A (en) * | 2008-11-12 | 2009-04-15 | 中兴通讯股份有限公司 | Message-driven automation test system and implementing method |
| CN101408849A (en) * | 2008-12-01 | 2009-04-15 | 中国科学技术大学 | Method and system for compiling and executing TTCN-3 language |
| CN101470607A (en) * | 2007-12-29 | 2009-07-01 | 北京天融信网络安全技术有限公司 | Data normalization method |
| CN101958837A (en) * | 2010-09-30 | 2011-01-26 | 北京世纪互联工程技术服务有限公司 | Log processing system, log processing method, node server and center server |
| CN102023861A (en) * | 2010-12-08 | 2011-04-20 | 中兴通讯股份有限公司 | Method and system for generating test script and test suite |
| CN102065416A (en) * | 2009-11-18 | 2011-05-18 | 成都市华为赛门铁克科技有限公司 | Method, device and system for formatting logs |
-
2012
- 2012-08-15 CN CN201210290200.6A patent/CN103593277A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1499374A (en) * | 2002-11-04 | 2004-05-26 | 华为技术有限公司 | Automatic approach for unit testing |
| US20040254919A1 (en) * | 2003-06-13 | 2004-12-16 | Microsoft Corporation | Log parser |
| CN1670708A (en) * | 2004-03-17 | 2005-09-21 | 联想(北京)有限公司 | Management method for computer log |
| CN1877525A (en) * | 2005-05-10 | 2006-12-13 | 西加特技术有限责任公司 | Protocol scripting language for safe execution in embedded system |
| CN1889459A (en) * | 2006-07-20 | 2007-01-03 | 华为技术有限公司 | Journal formatting unit and method in safety management center system |
| KR20090000278A (en) * | 2007-02-13 | 2009-01-07 | 삼성전자주식회사 | Log Analysis System and Its Log Analysis Method |
| CN101126981A (en) * | 2007-09-28 | 2008-02-20 | 华为技术有限公司 | Script order registration method, method and device for calling source program code |
| CN101470607A (en) * | 2007-12-29 | 2009-07-01 | 北京天融信网络安全技术有限公司 | Data normalization method |
| CN101409655A (en) * | 2008-11-12 | 2009-04-15 | 中兴通讯股份有限公司 | Message-driven automation test system and implementing method |
| CN101408849A (en) * | 2008-12-01 | 2009-04-15 | 中国科学技术大学 | Method and system for compiling and executing TTCN-3 language |
| CN102065416A (en) * | 2009-11-18 | 2011-05-18 | 成都市华为赛门铁克科技有限公司 | Method, device and system for formatting logs |
| CN101958837A (en) * | 2010-09-30 | 2011-01-26 | 北京世纪互联工程技术服务有限公司 | Log processing system, log processing method, node server and center server |
| CN102023861A (en) * | 2010-12-08 | 2011-04-20 | 中兴通讯股份有限公司 | Method and system for generating test script and test suite |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105656706A (en) * | 2014-11-14 | 2016-06-08 | 北京通达无限科技有限公司 | Business data processing method and device |
| CN105656706B (en) * | 2014-11-14 | 2020-09-15 | 北京通达无限科技有限公司 | Service data processing method and device |
| CN104994128A (en) * | 2015-05-15 | 2015-10-21 | 北京网康科技有限公司 | Data coding type identifying and transcoding method and device |
| CN104994128B (en) * | 2015-05-15 | 2019-04-26 | 北京网康科技有限公司 | A kind of identification of data encoding type and code-transferring method and device |
| CN109669848A (en) * | 2017-10-16 | 2019-04-23 | 比亚迪股份有限公司 | Log generation method, application server, log server and log generating system |
| CN109062762A (en) * | 2018-07-26 | 2018-12-21 | 浙江数链科技有限公司 | Formatted log method for pushing and device |
| CN110858192A (en) * | 2018-08-23 | 2020-03-03 | 阿里巴巴集团控股有限公司 | Log query method and system, log checking system and query terminal |
| CN110049017A (en) * | 2019-03-22 | 2019-07-23 | 口碑(上海)信息技术有限公司 | Message interworking unit and method between heterogeneous platform |
| CN110049017B (en) * | 2019-03-22 | 2021-11-16 | 口碑(上海)信息技术有限公司 | Message intercommunication device and method between heterogeneous platforms |
| CN110874350A (en) * | 2019-10-14 | 2020-03-10 | 支付宝(杭州)信息技术有限公司 | Method and device for processing structured log data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106575166B (en) | Method for processing hand input character, splitting and merging data and processing encoding and decoding | |
| CN103593277A (en) | Log processing method and system | |
| Brown et al. | Implementation patterns for microservices architectures | |
| CN103699389B (en) | A Method for Extracting Relationships of Linux Kernel Modules Based on Compilation Options | |
| CN103473108A (en) | Java code generating method | |
| CN106528165A (en) | Code generation method and code generation system | |
| CN102609402B (en) | Device and method for generating and managing ontology model based on real-time policy | |
| CN104133772A (en) | Automatic test data generation method | |
| CN106970820A (en) | Code storage method and code storage | |
| CN104185845A (en) | System and method for providing a binary representation of a web page | |
| KR101255060B1 (en) | System and method for extracting distributed parallel entity based on mapreduce | |
| CN105867919A (en) | Front end data bidirectional binding realization method based on accessor hijack | |
| WO2021253641A1 (en) | Shading language translation method | |
| CN103345532A (en) | Method and device for extracting webpage information | |
| CN103235757B (en) | Several apparatus and method that input domain tested object is tested are made based on robotization | |
| CN116628066B (en) | Data transmission method, device, computer equipment and storage medium | |
| CN104090920A (en) | System for realizing digital content cross-terminal publishing | |
| CN104598348B (en) | A kind of method and system of the long-range external system interface performance of analysis in real time | |
| CN109491884A (en) | Code method for testing performance, device, computer equipment and medium | |
| CN106484412A (en) | A kind of two-way binding implementation method of the front end data based on accessor abduction | |
| CN111240772A (en) | Data processing method and device based on block chain and storage medium | |
| CN110377371B (en) | Style sheet system management method based on Web tag | |
| CN101617307B (en) | Encoding/decoding apparatus and method | |
| CN103677841A (en) | IETM code generation method and device based on element-level template | |
| CN106599241A (en) | Big data visual management method for GIS software |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140219 |