CN103577443B - A kind of log processing system - Google Patents
A kind of log processing system Download PDFInfo
- Publication number
- CN103577443B CN103577443B CN201210266986.8A CN201210266986A CN103577443B CN 103577443 B CN103577443 B CN 103577443B CN 201210266986 A CN201210266986 A CN 201210266986A CN 103577443 B CN103577443 B CN 103577443B
- Authority
- CN
- China
- Prior art keywords
- log
- information
- module
- configuration
- processing system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 48
- 238000012544 monitoring process Methods 0.000 claims abstract description 73
- 238000000034 method Methods 0.000 claims abstract description 10
- 230000008569 process Effects 0.000 claims abstract description 6
- 230000004048 modification Effects 0.000 claims description 10
- 238000012986 modification Methods 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 7
- 230000003044 adaptive effect Effects 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims 1
- 230000001143 conditioned effect Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 5
- 238000004140 cleaning Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention discloses a kind of log processing system, it is characterized in that, including daily record generation module, daily record monitoring service end, daily record monitor client, wherein, daily record generation module is arranged in one or more application systems, called so as to generate with the log information being pre-configured with by one or more application systems in the form of unified interface, daily record monitoring service end is configured to receive and process the log information sent from daily record generation module, and will be sent to daily record monitor client with pre-conditioned adaptable log information.
Description
Technical Field
The present invention relates to information technology, and more particularly, to a log processing system.
Background
At present, the IT system supporting enterprise business is more and more numerous, and for operation and maintenance personnel, a large amount of error reporting and positioning often increase the difficulty of monitoring thereof, and the processing efficiency of the IT system is reduced. However, each system is restricted by the factors such as platform, team, demand, etc., and the respective log module is often designed, so that the error reporting information of each system is five-fold, and unnecessary troubles are brought to operation and maintenance personnel and support personnel.
On the other hand, in IT systems, various levels of log information are often defined, such as an information level, a debugging level, a warning level, a general error level, and a fatal error level. Alternatively, many personalized log processing modes may be defined, such as a log recording mode (process level, thread level, module level, etc.), a log classification mode (time, module, size, etc.), and the like. According to the period of a software product, different log modes need to be set for the product at different stages or different use scenes of the software. In the prior art, different logging modes are often realized by compiling the same source program into different target products through different compiling options at the software product compiling stage, and this actually causes different software products at different stages.
Therefore, there is a need for a centralized, unified and dynamically adjustable log processing system.
Disclosure of Invention
According to an object of the present invention, a log processing system is provided, which includes a log generating module, a log monitoring server and a log monitoring client, wherein the log generating module is disposed in one or more application systems and is called by the one or more application systems in a form of a unified interface so as to generate log information having a pre-configuration, and the log monitoring server is configured to receive and process the log information transmitted from the log generating module and transmit the log information adapted to a preset condition to the log monitoring client.
Preferably, the log monitoring server includes a log receiving and processing module, a log sending module, and a log management module, wherein the log receiving and processing module is configured to receive log information from the log generating module and transmit the received log information to the log sending module, the log sending module filters the received log information according to a preset condition and sends the log information adapted to the preset condition to the log monitoring client, and the log management module is configured to receive a user-defined setting from the log monitoring client as a part of the preset condition.
Preferably, the log receiving and processing module is further configured to classify the received log information and record the classified log information into a database, and the log management module is further configured to send corresponding log information to the log monitoring client in response to a query request from the log monitoring client according to the database.
Preferably, the log sending module is further configured to maintain a link status with the log monitoring client.
Preferably, the log management module is further configured to clean up log information in the database.
Preferably, the log generating module sends the log information to the log monitoring server by using a TCP/UDP protocol.
Preferably, the log monitoring server communicates with the log monitoring client through an HTTP protocol.
Preferably, the log monitoring client comprises a PC and a mobile terminal.
Preferably, the log monitoring client is configured to be able to register with the log monitoring server,
the log monitoring server is configured to use the registered information as a part of preset conditions.
Preferably, the log generating module further comprises a log configuration memory mapping processing module and a logger, wherein the log configuration memory mapping processing module is configured to map the log configuration file into the memory of the application system during the initialization of the application system, and the logger generates the log information according to the configuration information in the memory.
Preferably, the log generating module further comprises a log configuration modification memory reloading module, wherein the log configuration modification memory reloading module dynamically adjusts configuration information in the memory according to changes of the log configuration file.
Preferably, the configuration information in the memory indicates the output form and content of the generated log information.
Preferably, the log generation module is further configured to generate a local log file, and to generate the local log file synchronously or asynchronously with the log information according to the configuration information in the memory.
The technical scheme of the invention realizes the unification of log information among a plurality of application systems, thereby not only not influencing the respective error reporting function of each system, but also uniformly and effectively processing the error reporting information of all systems and greatly improving the efficiency of error reporting processing. The invention has the advantages that the error reporting of the existing application system is unified from the aspect of expression and the aspect of processing, thereby not only facilitating the processing and analysis of operation and maintenance personnel and support personnel, reducing the resource investment of each system for developing a log processing module, but also being beneficial to performing later analysis on the integrated log data, and further reversely optimizing the application system. In addition, by providing error reporting interfaces of different platforms, the existing application system can be easily integrated into a unified log processing system.
The invention also has the advantages that according to the technical scheme of the invention, the software product can dynamically adjust the log mode of the product without compiling into different target products, so that the software product can be suitable for different stages or different use scenes of the software, and a plurality of sets of different target products are not required to be maintained.
Drawings
The various aspects of the present invention will become more apparent to those of ordinary skill in the art after reading the detailed description of the invention in light of the accompanying drawings. It should be understood by those skilled in the art that the drawings are designed solely for purposes of illustrating the subject technology in connection with the detailed description thereof and are not intended as a definition of the limits of the invention. Wherein,
FIG. 1 is a block diagram of a log processing system according to an embodiment of the present invention.
Fig. 2 is a block diagram of a log monitoring server according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a log processing procedure of the log processing system according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a log processing procedure of the log processing system according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
As shown in fig. 1, the log processing system includes a log generation module, a log monitoring server, and a log monitoring client. The LOG generation module may be provided in one or more application systems, and may be called by the one or more application systems in the form of a unified interface (as shown in fig. 3, the LOG API is integrated with application logic in the application system as the LOG generation module) to generate LOG information having a pre-configuration (including a specified format, a LOG mode, a record location, and the like). As shown in fig. 4, by providing a uniform interface for a plurality of application systems, the log monitoring server can uniformly collect log information of each application system without affecting the log function of each system. The log monitoring server is configured to receive and process the log information sent from the log generating module, and send the log information adapted to the preset condition to the log monitoring client. The method for sending the log information can be pushing, and the log monitoring server side can transmit the information to the log monitoring client side by adopting an active pushing method. It is understood that the log monitoring client may also actively read data from the log monitoring server, which may be referred to as "pull". By means of the log generation module, the log monitoring server does not need to analyze the log file of the related application system, and only analyzes the message information containing the log information sent from the remote place. In one embodiment, the LOG generating module sends the LOG information to the LOG monitoring server by using a TCP/UDP protocol, and as shown in fig. 3, the LOG API serving as the LOG generating module sends the LOG information in a UDP manner.
Fig. 2 is a block diagram of a log monitoring server according to an embodiment of the present invention. The log monitoring server comprises a log receiving and processing module, a log sending module and a log management module. The functions of the respective modules are described below. The log receiving processing module is configured to receive the log information from the log generating module and transmit the received log information to the log transmitting module. And the log sending module filters the received log information according to preset conditions and sends the log information adaptive to the preset conditions to the log monitoring client. The log management module is configured to receive user-defined settings from the log monitoring client as part of the preset conditions. In one embodiment, the log sending module is further configured to maintain a link state with the log monitoring client to ensure that the sending is valid. In one embodiment, the log monitoring server communicates with the log monitoring client via the HTTP protocol.
In one embodiment, the log monitoring client is configured to be capable of registering at the log monitoring server, and the log monitoring server is configured to use the registered information as part of the preset condition. As shown in fig. 4, the log monitoring server may transmit only the log information 3 to the log monitoring client according to the registered information as part of the preset condition. As described above, the log management module can be configured to receive user-customized settings from the log monitoring client as part of the preset conditions. In this embodiment, the log monitoring server may send only a portion of the log information 3 to the log monitoring client according to a user-defined setting as part of the preset conditions. As shown in fig. 3, the received log information can be distributed to the log monitoring client through conditional filtering. Therefore, the log monitoring client can realize centralized customization and acquisition of log information by registering and setting user-defined data in the log monitoring server. The log monitoring client may include a PC and a mobile terminal, but the present invention is not limited thereto, and for example, other devices capable of communicating with the log monitoring server using the HTTP protocol can also serve as the log monitoring client.
In one embodiment, the log reception processing module is further configured to classify and record the received log information into a database. For example, the log receiving and processing module can classify specific log information and record the specific log information into log tables in different databases according to parameters such as date and importance. In addition, the log management module is further configured to send corresponding log information to the log monitoring client in response to a query request from the log monitoring client according to the database, as shown in fig. 3.
In another embodiment, the log management module is further configured to clean up log information in the database. For example, alternate-day cleaning, alternate-month cleaning, cleaning by type, and the like may be performed periodically.
In another embodiment according to the present invention, the log generating module may further include a log configuration memory mapping processing module and a logger, wherein the log configuration memory mapping processing module is configured to map the log configuration file into a memory of the application system during initialization of the application system, and the logger generates the log information according to the configuration information in the memory. Therefore, the application system can realize customized log processing only by accessing the configuration information in the memory in the life cycle. As an example, the log configuration memory mapping processing module loads an analysis file (binary file generated from the configuration file) into the memory during the initialization of the application system, and initializes the log information of the corresponding application program according to the configuration information in the memory, and in the case of a local log file, may initialize the file name, file format, and the like of the local log file. It can be understood by those skilled in the art that the log information to be sent to the log monitoring server may also be initialized according to the configuration information in the memory, for example, the configuration information in the memory can indicate the output form and content of the generated log information.
In another example, the log generating module further includes a log configuration modification memory reloading module, wherein the log configuration modification memory reloading module dynamically adjusts the configuration information in the memory according to a change of the log configuration file. As an example, the log configuration modification memory reloading module may dynamically refresh the configuration file, and when the content of the configuration file is found to be changed, remap the content in the configuration file to the memory corresponding to the configuration information. This process of dynamic validation is transparent to the application system itself. For example, the log configuration modification memory reload module can be configured to parse the content of the configuration file, match the content in the configuration file with the configuration information in the memory, and if there is a place of inconsistency, modify the content of the configuration information in the memory to be consistent with the content in the configuration file.
As described above and shown in fig. 3, the log generation module is further configured to generate a local log file, and to generate the local log file synchronously or asynchronously with the log information according to the configuration information in the memory.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present invention may be implemented by combining software and a hardware platform, and of course, may be implemented by hardware entirely. Those skilled in the art will appreciate that various modifications and substitutions can be made to the specific embodiments of the present invention without departing from the spirit and scope of the invention. Such modifications and substitutions are intended to be included within the scope of the present invention as defined by the appended claims.
Claims (10)
1. The log processing system is characterized by comprising a log generation module, a log monitoring server and a log monitoring client, wherein,
the log generation module is arranged in one or more application systems and is called by the one or more application systems in the form of a uniform interface so as to generate log information with pre-configuration,
the log monitoring server is configured to receive and process the log information sent from the log generating module and push the log information adaptive to the preset conditions to the log monitoring client, wherein
The log monitoring client is configured to be able to register with the log monitoring server,
the log monitoring server is configured to use the registered information as a part of a preset condition,
the log generation module also comprises a log configuration memory mapping processing module and a logger,
wherein the log configuration memory mapping processing module is configured to map the log configuration file into the memory of the application system during the initialization of the application system, the logger generates log information according to the configuration information in the memory,
the log generation module also comprises a log configuration modification memory reloading module,
the log configuration modification memory reloading module dynamically adjusts configuration information in the memory according to changes of the log configuration file.
2. The log processing system of claim 1, wherein the log monitoring service includes a log receiving processing module, a log sending module, a log management module, wherein,
the log receiving processing module is configured to receive the log information from the log generating module and transfer the received log information to the log transmitting module,
the log sending module filters the received log information according to preset conditions and sends the log information adaptive to the preset conditions to the log monitoring client,
the log management module is configured to receive user-defined settings from the log monitoring client as part of the preset conditions.
3. The log processing system of claim 2,
the log reception processing module is further configured to classify and record the received log information into a database,
the log management module is also configured to send corresponding log information to the log monitoring client in response to a query request from the log monitoring client according to the database.
4. The log processing system of claim 3,
the log sending module is further configured to maintain a link state with the log monitoring client.
5. The log processing system of claim 4,
the log management module is also configured to clean up log information in the database.
6. The log processing system of claim 1,
and the log generation module sends the log information to a log monitoring server by adopting a TCP/UDP protocol.
7. The log processing system of claim 1,
and the log monitoring server side and the log monitoring client side are communicated through an HTTP (hyper text transport protocol).
8. The log processing system of claim 7,
the log monitoring client comprises a PC and a mobile terminal.
9. The log processing system of claim 1,
the configuration information in the memory indicates the output form and content of the generated log information.
10. The log processing system of claim 1,
the log generation module is further configured to generate a local log file, and to generate the local log file synchronously or asynchronously with the log information based on the configuration information in the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210266986.8A CN103577443B (en) | 2012-07-30 | 2012-07-30 | A kind of log processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210266986.8A CN103577443B (en) | 2012-07-30 | 2012-07-30 | A kind of log processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103577443A CN103577443A (en) | 2014-02-12 |
| CN103577443B true CN103577443B (en) | 2017-05-31 |
Family
ID=50049248
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210266986.8A Active CN103577443B (en) | 2012-07-30 | 2012-07-30 | A kind of log processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103577443B (en) |
Families Citing this family (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103856353B (en) * | 2014-03-06 | 2018-01-26 | 上海爱数信息技术股份有限公司 | A kind of business diary data access and the method and device of statistical analysis |
| CN105095358A (en) * | 2015-06-24 | 2015-11-25 | 北京京东尚科信息技术有限公司 | Method and system for acquiring database operation logs |
| CN105141448B (en) * | 2015-07-28 | 2018-10-02 | 杭州华为数字技术有限公司 | A kind of acquisition method and device of daily record |
| CN105069059A (en) * | 2015-07-28 | 2015-11-18 | 小米科技有限责任公司 | Database synchronization method and apparatus |
| CN105224440A (en) * | 2015-09-02 | 2016-01-06 | 上海斐讯数据通信技术有限公司 | A kind of log collection management method and system |
| CN105224441B (en) * | 2015-09-17 | 2018-11-20 | 杭州华为数字技术有限公司 | Virtual machine information acquisition device, method and virtual machine information maintaining method and system |
| CN105160011A (en) * | 2015-09-22 | 2015-12-16 | 浪潮(北京)电子信息产业有限公司 | Log management system and method |
| CN105119762A (en) * | 2015-09-23 | 2015-12-02 | 普元信息技术股份有限公司 | System and method of cloud platform for realizing transaction playback and transaction reworking based on logs |
| CN105553716A (en) * | 2015-12-12 | 2016-05-04 | 天津南大通用数据技术股份有限公司 | Fetch and display system of each node log under large-scale cluster |
| CN105677552A (en) * | 2016-01-04 | 2016-06-15 | 杭州亚美利嘉科技有限公司 | Robot log management method and server |
| CN105847062A (en) * | 2016-05-06 | 2016-08-10 | 汉柏科技有限公司 | Log aggregation method and device |
| CN106354503A (en) * | 2016-08-29 | 2017-01-25 | 浪潮电子信息产业股份有限公司 | An audit log analysis method for Linux |
| CN106790380A (en) * | 2016-11-21 | 2017-05-31 | 广州华多网络科技有限公司 | Data reporting method and device |
| CN107622011A (en) * | 2017-09-01 | 2018-01-23 | 惠州市德赛西威汽车电子股份有限公司 | A kind of dynamic log control method and system |
| CN109905253B (en) * | 2017-12-07 | 2022-05-17 | 航天信息股份有限公司 | Log information acquisition method and device |
| CN108170576B (en) * | 2017-12-26 | 2021-12-07 | Oppo广东移动通信有限公司 | Log processing method and device, terminal equipment and storage medium |
| CN108958652A (en) * | 2018-06-25 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method, apparatus and computer readable storage medium of record log information |
| CN109086098A (en) * | 2018-07-23 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of journal file method for cleaning and relevant apparatus |
| CN109739735B (en) * | 2018-12-04 | 2022-05-31 | 东软集团股份有限公司 | Log generation method and device |
| CN109672735A (en) * | 2018-12-20 | 2019-04-23 | 中国移动通信集团江苏有限公司 | A kind of method, device and equipment of Message Processing |
| CN110780857B (en) * | 2019-10-23 | 2024-01-30 | 杭州涂鸦信息技术有限公司 | Unified log component |
| CN111367867B (en) * | 2020-03-05 | 2023-03-21 | 腾讯云计算（北京）有限责任公司 | Log information processing method and device, electronic equipment and storage medium |
| CN113760655B (en) * | 2021-08-27 | 2024-07-26 | 中移（杭州）信息技术有限公司 | Analysis method and device of door lock log and computer readable storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1670708A (en) * | 2004-03-17 | 2005-09-21 | 联想(北京)有限公司 | Management method for computer log |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9098613B2 (en) * | 2006-06-23 | 2015-08-04 | Sap Se | Logging of application messages |
-
2012
- 2012-07-30 CN CN201210266986.8A patent/CN103577443B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1670708A (en) * | 2004-03-17 | 2005-09-21 | 联想(北京)有限公司 | Management method for computer log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103577443A (en) | 2014-02-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103577443B (en) | A kind of log processing system | |
| US11108863B2 (en) | Tag operating system | |
| CN109245931B (en) | Log management and monitoring alarm realization method of container cloud platform based on kubernets | |
| US20160094421A1 (en) | Platform for capturing, processing, storaging, and presentation of generic sensor data from remote arbitrary locations | |
| CN104283967B (en) | A kind of third party's data service system based on internet of things data acquisition | |
| CN108989358B (en) | Method for acquiring running data of textile machine based on TCP/IP protocol | |
| CN105915633B (en) | Automatic operation and maintenance system and method | |
| CN110601876A (en) | Intelligent Internet of things platform | |
| US8762931B2 (en) | Generating an encoded package profile | |
| US20190124139A1 (en) | Mobile core client architecture | |
| CN103177116A (en) | Distributed log handling and inquiring method based on two-stage index | |
| US8478714B2 (en) | Integrated distribution management system channel adapter | |
| US20080162690A1 (en) | Application Management System | |
| US10983040B2 (en) | Particle counter with integrated bootloader | |
| EP2779583A1 (en) | Telecommunication method and system | |
| US20110113014A1 (en) | Methodology of Applying Storage and Logistics Center Model to Achieve Business Data Exchange between Systems | |
| US12044611B2 (en) | Particle counter with integrated bootloader | |
| US20230259438A1 (en) | Edge-Based Data Collection System for an Observability Pipeline System | |
| CN113327060A (en) | Intelligent factory management system and method thereof | |
| CN102567517A (en) | Device and method for issuing data of database | |
| CN107463490B (en) | Cluster log centralized collection method applied to platform development | |
| CN103297477B (en) | A kind of data acquisition reporting system and data processing method and proxy server | |
| CN110398985B (en) | Distributed self-adaptive unmanned aerial vehicle measurement and control system and method | |
| JP2004220560A5 (en) | ||
| CN110795480B (en) | Traffic operation data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |