CN110413496A - A method of realizing that the operation data modularization of electronics license is collected - Google Patents
A method of realizing that the operation data modularization of electronics license is collected Download PDFInfo
- Publication number
- CN110413496A CN110413496A CN201910686104.5A CN201910686104A CN110413496A CN 110413496 A CN110413496 A CN 110413496A CN 201910686104 A CN201910686104 A CN 201910686104A CN 110413496 A CN110413496 A CN 110413496A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- analysis system
- module
- logging tools
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 14
- 238000004458 analytical method Methods 0.000 claims abstract description 35
- 238000013480 data collection Methods 0.000 claims abstract description 10
- 238000013500 data storage Methods 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 12
- 241000208340 Araliaceae Species 0.000 claims description 6
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 claims description 6
- 235000003140 Panax quinquefolius Nutrition 0.000 claims description 6
- 235000008434 ginseng Nutrition 0.000 claims description 6
- 230000000977 initiatory effect Effects 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 claims description 4
- 239000003550 marker Substances 0.000 claims description 3
- 230000000873 masking effect Effects 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013070 change management Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention relates to a kind of methods realizing electronics license operation data modularization and collecting, and provide an operational data collection platform；Including logging tools module, micro services card module, analysis system and data memory module；Logging tools module and micro services card module acquire data, and send analysis system for the data that the two acquires by logging tools module；Analysis system sends the data memory module for the data received and carries out data storage；Analysis system can also obtain the data of data memory module by way of manual retrieval or timed task, to show or report form statistics.The present invention is not limited by machine account, and for safety, only a few peoples have permission the machine of production environment, but analysis system of the invention easily can distribute account and permission to everybody very, and not influence the safety of production environment host.Effectively data can be counted, be analyzed；The utilization rate of data is improved, and retrieval conveniently has a unified search space.
Description
Technical field
The present invention relates to electronics license, data collection, modularization field, especially a kind of realization electronics license operation data
The method that modularization is collected.
Background technique
Operation data mainly includes system log, application log and security log.System O&M and developer can
To understand server software and hardware information by log, check the reason of mistake and mistake generation in configuration process.Often analysis
Log will be seen that the load of server, performance security, to take timely measure correction mistake.But current status is:
Electronics license platform possesses multiple applications, these have acquisition operation data using some, and some does not have.And acquire mode with
And the mode of storage is different.Technical aspect, some use System.out.println, some logback, some use
Log4j, there are also operation datas to be directly stored on mysql or mongoDB.
It is stored in different equipment in general, operation data is dispersed in.If managing tens of up to a hundred servers, but also
Operation data is being consulted using the conventional method for successively logging in every machine.It in this way can troublesome and inefficiency.So using
The operation data management of centralization.
After centralized management operation data, the statistics of operation data and retrieval become a more troublesome thing again, and one
As using Linux commands such as grep, awk and wc be able to achieve retrieval and statistics, but for requiring higher inquiry, sequence and system
Meter etc. requires and huge machine quantity still uses such method inevitably somewhat unable to do what one wishes.By being concentrated to log
Change management, the log information on all machines is collected, is gathered.Complete daily record data has very important work
With:
1) information searching: by retrieving log information, corresponding bug is positioned, solution is found out.
2) Service Diagnostic: by being counted, being analyzed to log information, understand the load and service operation shape of server
State is found out time consuming requests and is optimized etc..
3) data are analyzed: if it is the log of formatting, further data analysis can be done, count, polymerize it is significant
Information, for example according to the commodity id in request, find out TOP10 user's business interested.
4) link analysis is called.Collection system can also collect the letters such as the important informations such as user identifier, request module, initiation address
Breath.By it, entire call chain can be tracked, is also convenient for investigation problem.
Summary of the invention
In view of this, the purpose of the present invention is to propose to kind of the method for realizing electronics license operation data modularization collection, energy
It is unified that log is analyzed, is retrieved, it improves the acquisition of operation data and utilizes process.
The present invention is realized using following scheme: a method of it realizing that the operation data modularization of electronics license is collected, provides
One operational data collection platform；The operational data collection platform includes logging tools module, micro services card module, log
Analysis system and data memory module；The logging tools module and the micro services card module acquire data, and pass through institute
It states logging tools module and sends the Log Analysis System for the data that the two acquires；The Log Analysis System will receive
Data be sent to the data memory module and carry out data storage；The Log Analysis System can also by manual retrieval or
The mode of timed task obtains the data of the data memory module, to show or report form statistics；
After Log Analysis System stores data into memory module, built-in some timed tasks.These timed tasks can basis
Data in data module, by date or other dimensions generate some simple reports.Meanwhile Log Analysis System also provides
Visual m odeling technique interface, user can input the search conditions such as keyword, period on interface, search for the data that he wants
(log).
The platform carries out electronics license operation data modularization collection in accordance with the following steps:
Step S1: carry out customized log: the logging tools module is obtained from configuration center and is configured；From the marker bit in configuration
Judgement is that journal engine；Log recording then is carried out using the simple log interface passed through if local engine；If system
One engine then usage log tool model, by log content, log recording time, code row information locating for log is with json's
Format is sent to the Log Analysis System, executes step S7；
Step S2: statistical log storage: developer calls the interface of the logging tools module to carry out statistical data typing, and holds
Row step S7；
Step S3: it carries out micro services calling: if called side has used log plug-in unit, and not masking journal function, just
It can log enable function；Judge called side whether log enable；Called side starts log recording if log enable, and executes
Step S4；It is no to then follow the steps S5；
Step S4: the logging tools module is that current context generates unique identifier, to call the function of D-chain trace
Energy；If having initiated micro services calling, identifier can be transmitted to callee；By this mechanism, the logging tools module
It identifies that an operation is original initiation, or is called by third party；If then called side starts log recording and executes step
Rapid S6；Otherwise step S6 is directly executed；
Step S5: called side starts log recording, and obtains the additional log content of client transmitting, finally executes step
S6；
Step S6: the launch position of micro services is recorded, accessed interface, enters ginseng, return value, whether succeed, exception information, place
Manage the time spent, and by the launch position of the micro services, accessed interface, enter ginseng and result is sent to log services system
System executes step S7；
Step S7: by application identities, the local ip of log recording person, the port of log recording person monitoring, Context identifier, permission
Customized field of user etc. adds in log as custom content, is then sent to the log point in a batch
Analysis system；Log Analysis System parses the type of every log from batch, then store it big data search in real time
Analysis engine external member.
Further, the Log Types of parsing described in step S7 include but is not limited to customized log, statistical log
With micro services log.
Compared with prior art, the invention has the following beneficial effects:
The present invention is not limited by machine account, and for safety, only a few peoples have permission the machine of production environment, but the present invention
Log Analysis System easily can distribute account and permission to everybody very, and do not influence the safety of production environment host.
Effectively data can be counted, be analyzed；The utilization rate of data is improved, and retrieving conveniently has a unified retrieval empty
Between.
Detailed description of the invention
Fig. 1 is the flow chart of the embodiment of the present invention.
Specific embodiment
The present invention will be further described with reference to the accompanying drawings and embodiments.
It is noted that following detailed description is all illustrative, it is intended to provide further instruction to the application.Unless another
It indicates, all technical and scientific terms used herein has usual with the application person of an ordinary skill in the technical field
The identical meanings of understanding.
It should be noted that term used herein above is merely to describe specific embodiment, and be not intended to restricted root
According to the illustrative embodiments of the application.As used herein, unless the context clearly indicates otherwise, otherwise singular
Also it is intended to include plural form, additionally, it should be understood that, when in the present specification using term "comprising" and/or " packet
Include " when, indicate existing characteristics, step, operation, device, component and/or their combination.
As shown in Figure 1, a kind of method realizing electronics license operation data modularization and collecting is present embodiments provided,
One operational data collection platform is provided；The operational data collection platform includes logging tools module, micro services plug-in unit mould
Block, Log Analysis System and data memory module；The logging tools module and the micro services card module acquire data, and
The Log Analysis System is sent by the data that the two acquires by the logging tools module；The Log Analysis System will
The data received are sent to the data memory module and carry out data storage；The Log Analysis System can also be by artificial
The mode of retrieval or timed task, obtains the data of the data memory module, to show or report form statistics；
The platform carries out electronics license operation data modularization collection in accordance with the following steps:
Step S1: carry out customized log: the logging tools module is obtained from configuration center and is configured；From the marker bit in configuration
Judgement is that journal engine；Log recording then is carried out using the simple log interface passed through if local engine；If system
One engine then usage log tool model, by log content, log recording time, code row information locating for log is with json's
Format is sent to the Log Analysis System, executes step S7；
Step S2: statistical log storage: developer calls the interface of the logging tools module to carry out statistical data typing, and holds
Row step S7；
Step S3: it carries out micro services calling: if called side has used log plug-in unit, and not masking journal function, just
It can log enable function；Judge called side whether log enable；Called side starts log recording if log enable, and executes
Step S4；It is no to then follow the steps S5；
Step S4: the logging tools module is that current context generates unique identifier, to call the function of D-chain trace
Energy；If having initiated micro services calling, identifier can be transmitted to callee；By this mechanism, the logging tools module
It identifies that an operation is original initiation, or (identifier) is called by third party；If then called side starts log recording
And execute step S6；Otherwise step S6 is directly executed；
Step S5: called side starts log recording, and obtains the additional log content of client transmitting, finally executes step
S6；
Step S6: the launch position of micro services is recorded, accessed interface, enters ginseng, return value, whether succeed, exception information, place
Manage the time spent, and by the launch position of the micro services, accessed interface, enter ginseng and result is sent to log services system
System executes step S7；
Step S7: by application identities, the local ip of log recording person, the port of log recording person monitoring, Context identifier, permission
Customized field of user etc. adds in log as custom content, is then sent to the log point in a batch
Analysis system；Log Analysis System parses the type of every log from batch, then store it big data search in real time
Analysis engine external member.
In the present embodiment, the Log Types of parsing described in step S7 include but is not limited to customized log, statistics
Log and micro services log.
1) logging tools are a manual operation data sampling instruments, it can be collected log, filter.It can
The log of immobilized substance is written, it also can be written into the data of variable data item.Variable data item includes statistical data etc., mainly
For statisticalling analyze.
2) micro services plug-in unit is a plug-in unit without code intrusion for dubbo, if developer is not customized
Log content, there is no need to the exploitations of any additional code for that.Micro services plug-in unit can be in request end or called terminal automatic collection
Interface requests, abnormal log, request time-consuming, launch position etc..Default acquires log in request end, if request end is not pacified
Micro services plug-in unit is filled, or has disabled the plug-in unit, it will acquire log in server-side.And it also supports user id etc. to identify
It is transmitted in entire call chain, has both improved the quality of data, also function to the function of hawkeye tracking.
3) Log Analysis System is according to the index database on the automatic maintenance elasticsearch of configuration.When it receives data
Afterwards, data can be sorted out, is then stored in the corresponding index database of elasticsearch.And provide retrieval, analysis function
Energy.
4) data storage uses elasticsearch, it is the open source distributed search clothes based on Lucene
Business system.Its feature has: distributed, zero configuration, automatic to find, indexes auto plate separation, indexes copy mechanism, restful wind
Lattice interface, multi-data source, automatic search overhead etc..It provides the full-text search engine of a distributed multi-user ability, base
In RESTful web interface.Elasticsearch is developed with Java, and as the open-source under Apache license terms
Code publication is the second popular enterprise search engine.It is stable designed for real-time search in cloud computing, can be reached, reliably,
Quickly, easy to install and use.In elasticsearch, the data of all nodes are impartial.
Preferably, operational data collection is divided into manual data acquisition and micro services automatic data collection two by the present embodiment
Kind.Either any data can all add some additional information, such as current system before being sent to Log Analysis System
The address ip of system, associated user identifier of the data (if any) etc..Then these data are unifiedly stored to
Then ElasticSearch can unify to analyze log, retrieve.It substantially improves the acquisition of operation data and utilizes stream
Journey.
The foregoing is merely presently preferred embodiments of the present invention, all equivalent changes done according to scope of the present invention patent with
Modification, is all covered by the present invention.
Claims (2)
1. a kind of method realizing electronics license operation data modularization and collecting, it is characterised in that: provide an operational data collection
Platform；The operational data collection platform includes logging tools module, micro services card module, Log Analysis System and data
Memory module；The logging tools module and the micro services card module acquire data, and pass through the logging tools module
The Log Analysis System is sent by the data that the two acquires；The data received are sent institute by the Log Analysis System
It states data memory module and carries out data storage；The Log Analysis System can also pass through manual retrieval or the side of timed task
Formula obtains the data of the data memory module, to show or report form statistics；
The platform carries out electronics license operation data modularization collection in accordance with the following steps:
Step S1: carry out customized log: the logging tools module is obtained from configuration center and is configured；From the marker bit in configuration
Judgement is that journal engine；Log recording then is carried out using general simple log interface if local engine；If system
One engine then usage log tool model, by log content, log recording time, code row information locating for log is with json's
Format is sent to the Log Analysis System, executes step S7；
Step S2: statistical log storage: developer calls the interface of the logging tools module to carry out statistical data typing, and holds
Row step S7；
Step S3: it carries out micro services calling: if called side has used log plug-in unit, and not masking journal function, just
It can log enable function；Judge called side whether log enable；Called side starts log recording if log enable, and executes
Step S4；It is no to then follow the steps S5；
Step S4: the logging tools module is that current context generates unique identifier, to call the function of D-chain trace
Energy；If having initiated micro services calling, identifier can be transmitted to callee；By this mechanism, the logging tools module
It identifies that an operation is original initiation, or is called by third party；If then called side starts log recording and executes step
Rapid S6；Otherwise step S6 is directly executed；
Step S5: called side starts log recording, and obtains the additional log content of client transmitting, finally executes step
S6；
Step S6: the launch position of micro services is recorded, accessed interface, enters ginseng, return value, whether succeed, exception information, place
Manage the time spent, and by the launch position of the micro services, accessed interface, enter ginseng and result is sent to log services system
System executes step S7；
Step S7: by application identities, the local ip of log recording person, the port of log recording person monitoring, Context identifier, permission
Customized field of user etc. adds in log as custom content, is then sent to log analysis system in a batch
System；Log Analysis System parses the type of every log from batch, then stores up log to big data search and analysis in real time
Engine external member.
2. a kind of method realizing electronics license operation data modularization and collecting according to claim 1, it is characterised in that:
The Log Types of parsing described in step S7 include but is not limited to customized log, statistical log and micro services log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910686104.5A CN110413496B (en) | 2019-07-29 | 2019-07-29 | Method for realizing componentized collection of electronic license operation data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910686104.5A CN110413496B (en) | 2019-07-29 | 2019-07-29 | Method for realizing componentized collection of electronic license operation data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413496A true CN110413496A (en) | 2019-11-05 |
| CN110413496B CN110413496B (en) | 2022-08-19 |
Family
ID=68363640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910686104.5A Active CN110413496B (en) | 2019-07-29 | 2019-07-29 | Method for realizing componentized collection of electronic license operation data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413496B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112364189A (en) * | 2020-11-16 | 2021-02-12 | 浪潮云信息技术股份公司 | Electronic certificate application method based on ES service |
| CN113468352A (en) * | 2021-06-16 | 2021-10-01 | 浪潮软件股份有限公司 | Electronic license quality detection analysis model |
| CN115061873A (en) * | 2022-06-14 | 2022-09-16 | 启明信息技术股份有限公司 | Method and system for simplifying log query and analysis of software system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040167912A1 (en) * | 2003-02-20 | 2004-08-26 | International Business Machines Corporation | Unified logging service for distributed applications |
| US20100088354A1 (en) * | 2006-11-30 | 2010-04-08 | Alibaba Group Holding Limited | Method and System for Log File Analysis Based on Distributed Computing Network |
| US20140082184A1 (en) * | 2012-09-14 | 2014-03-20 | Alibaba Group Holding Limited | Method and system for monitoring execution of user request in distributed system |
| US20160098342A1 (en) * | 2014-10-05 | 2016-04-07 | YScope Inc. | Systems and processes for computer log analysis |
| US9626328B1 (en) * | 2013-06-19 | 2017-04-18 | EMC IP Holding Company LLC | Method and system for on-demand aggregated logging for distributed systems |
| WO2017166644A1 (en) * | 2016-03-31 | 2017-10-05 | 乐视控股（北京）有限公司 | Data acquisition method and system |
| CN109756364A (en) * | 2018-12-07 | 2019-05-14 | 成都四方伟业软件股份有限公司 | A kind of micro services performance optimization system and analysis method based on log analysis |
-
2019
- 2019-07-29 CN CN201910686104.5A patent/CN110413496B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040167912A1 (en) * | 2003-02-20 | 2004-08-26 | International Business Machines Corporation | Unified logging service for distributed applications |
| US20100088354A1 (en) * | 2006-11-30 | 2010-04-08 | Alibaba Group Holding Limited | Method and System for Log File Analysis Based on Distributed Computing Network |
| US20140082184A1 (en) * | 2012-09-14 | 2014-03-20 | Alibaba Group Holding Limited | Method and system for monitoring execution of user request in distributed system |
| US9626328B1 (en) * | 2013-06-19 | 2017-04-18 | EMC IP Holding Company LLC | Method and system for on-demand aggregated logging for distributed systems |
| US20160098342A1 (en) * | 2014-10-05 | 2016-04-07 | YScope Inc. | Systems and processes for computer log analysis |
| WO2017166644A1 (en) * | 2016-03-31 | 2017-10-05 | 乐视控股（北京）有限公司 | Data acquisition method and system |
| CN109756364A (en) * | 2018-12-07 | 2019-05-14 | 成都四方伟业软件股份有限公司 | A kind of micro services performance optimization system and analysis method based on log analysis |
Non-Patent Citations (1)
| Title |
|---|
| 黄媛媛等: "基于微服务架构的日志系统", 《电子技术与软件工程》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112364189A (en) * | 2020-11-16 | 2021-02-12 | 浪潮云信息技术股份公司 | Electronic certificate application method based on ES service |
| CN113468352A (en) * | 2021-06-16 | 2021-10-01 | 浪潮软件股份有限公司 | Electronic license quality detection analysis model |
| CN115061873A (en) * | 2022-06-14 | 2022-09-16 | 启明信息技术股份有限公司 | Method and system for simplifying log query and analysis of software system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413496B (en) | 2022-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11775343B1 (en) | Duty cycle estimation for job assignment | |
| Mayer et al. | An approach to extract the architecture of microservice-based software systems | |
| CN111581054B (en) | Log embedded point service analysis alarm system and method based on ELK | |
| Lu et al. | Stream bench: Towards benchmarking modern distributed stream computing frameworks | |
| CN103684898B (en) | It is a kind of to monitor the method and device that user's request is run in a distributed system | |
| CN107273267A (en) | Log analysis method based on elastic components | |
| US10116534B2 (en) | Systems and methods for WebSphere MQ performance metrics analysis | |
| CN101371245A (en) | Distributed capture and aggregation of dynamic application usage information | |
| US20030167180A1 (en) | System and methods for determining contract compliance | |
| CN110336863B (en) | Data reporting method and system | |
| CN107229556A (en) | Log Analysis System based on elastic components | |
| CN112256530A (en) | Log service method capable of tracking link and system thereof | |
| CN109460307B (en) | Micro-service calling tracking method and system based on log embedded point | |
| CN110413496A (en) | A method of realizing that the operation data modularization of electronics license is collected | |
| CN108509326A (en) | A kind of service state statistical method and system based on nginx daily records | |
| CN113868248A (en) | Indicator data pre-aggregation method | |
| CN108369550A (en) | Change is from not homologous data in real time | |
| KR100912177B1 (en) | User Behavior Analysis Method and System | |
| CN115333966B (en) | Topology-based Nginx log analysis method, system and equipment | |
| CN117971606A (en) | Log management system and method based on elastic search | |
| CN106326280B (en) | Data processing method, device and system | |
| US20050027574A1 (en) | Real-time activity intelligence system and method | |
| Albuquerque et al. | Logging design patterns for cloud-native applications | |
| CN114116872A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN113010483A (en) | Mass log management method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |