CN113468111A - Log monitoring management system and method for container cloud - Google Patents
Log monitoring management system and method for container cloud Download PDFInfo
- Publication number
- CN113468111A CN113468111A CN202110848017.2A CN202110848017A CN113468111A CN 113468111 A CN113468111 A CN 113468111A CN 202110848017 A CN202110848017 A CN 202110848017A CN 113468111 A CN113468111 A CN 113468111A
- Authority
- CN
- China
- Prior art keywords
- log
- module
- file
- container cloud
- files
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/156—Query results presentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a container cloud log monitoring and management system and a method, which relate to the technical field of log monitoring and management, and the system comprises: a container cloud module for establishing a container cloud; the log resource library module is used for temporarily storing the log file generated by the container cloud; the log collection module is used for extracting log files and sending log processing tasks to the distribution module; the distribution module is used for caching the log processing tasks and distributing the log processing tasks to the batch processing and compression module; the batch processing and compressing module is used for compressing the log file according to the distributed log processing task and attaching the log file to the packed file; the storage module is used for being matched with the batch processing and compressing module, receiving the compressed log files and storing the packed files meeting the preset requirements; the acquisition reading module is used for inquiring a target log file from the storage module according to the log inquiry condition sent by the visualization module; and the visualization module is used for displaying the query result through a visualization interface.
Description
Technical Field
The invention relates to the technical field of log monitoring management, in particular to a log monitoring management system and method of a container cloud.
Background
At present, when a problem occurs in an application or a certain node operated by a container cloud, a monitoring function is generally realized based on a promemeus + ELK system, and core components in the promemeus are Metric and Alert. Where Metric is used to illustrate that a threshold has been currently reached, Alert sets Metric to reach a certain cardinality to trigger an alarm. The basic unit of K8s is Pod, which exports logs to stdout and stderr, and when a problem occurs, the relevant logs are typically viewed by a developer at the interface or by command. If the application is suddenly hung up at this time, the related logs cannot be found, so that a log system ELK needs to be introduced to uniformly collect the logs, but if the ELK is used, switching between Kibana and Grafana is needed, and user experience is affected.
In summary, the monitoring system based on Prometheus + ELK system in the prior art has the following technical problems: the scheme adopts full-text retrieval to index the log, and has high resource occupation and higher cost. Generally, a complete set principle is adopted, namely, the functions are rich, complex operation is allowed, but the method is not suitable for a large number of simple query operation scenes at all, the efficiency is low, and resources are wasted. The operation is complicated, the usability is low, and the user experience is poor. Therefore, a technical solution for log monitoring that can overcome the above-mentioned drawbacks is needed.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a container cloud log monitoring and management system and method. The invention uses the labels same as Prometheus as indexes, and the labels can be used for inquiring the content of the log and inquiring the monitored data, thereby not only reducing the switching cost between two kinds of inquiry, but also greatly reducing the storage of the log index.
In a first aspect of an embodiment of the present invention, a log monitoring and management system for a container cloud is provided, where the system includes: the system comprises a container cloud module, a log resource library module, a log acquisition module, a distribution module, a batch processing and compression module, a storage module, an acquisition and reading module and a visualization module; wherein,
the container cloud module is used for establishing a container cloud, deploying a log collector on each node and pushing a log file to the log resource library module;
the log resource library module is used for temporarily storing the log file generated by the container cloud;
the log acquisition module is used for extracting log files from the log resource library module in a serialized mode, generating corresponding log processing tasks according to the log files, putting the corresponding log processing tasks into a task queue, and sending the log processing tasks to the distribution module;
the distribution module is used for receiving the log processing tasks pushed by the log acquisition module, caching the log processing tasks and distributing the log processing tasks to the batch processing and compression module;
the batch processing and compressing module is used for compressing the log file according to the distributed log processing task and attaching the log file to the packed file, and when the packed file meets the preset requirement, a refreshing instruction is sent to the storage module, and the packed file is refreshed to a database in the storage module;
the storage module comprises a file receiving unit and a database; the file receiving unit is used for being matched with the batch processing and compressing module and receiving the compressed log files; the database is used for storing the packed files meeting the preset requirements;
the acquisition reading module is used for inquiring a target log file from the storage module according to the log inquiry condition sent by the visualization module and returning an inquiry result to the visualization module;
and the visualization module is used for receiving the log query conditions input by the user and displaying the query result through the visualization interface.
Further, the container cloud module is specifically configured to:
and establishing a private container cloud based on Kubernetes + Docker, and pushing a log file to the log resource library module through standard output equipment and standard error output equipment.
Further, the log repository module is specifically configured to:
and setting a file acquisition API (application programming interface) interface for the log acquisition module to extract log files.
Further, the log collection module is specifically configured to:
and marking a label for the log file.
Further, the distribution module is specifically configured to:
according to the metadata of the log processing task, adopting a hash algorithm to obtain distribution data;
and distributing the log processing task to corresponding processing units in the batch processing and compression module according to the distribution data.
Further, the batch processing and compression module is specifically configured to:
and according to the distributed log processing tasks, compressing the log files and attaching the compressed log files to the chunk files, when the data volume of the chunk files reaches a set threshold or exceeds a fixed period, sending a refreshing instruction to the storage module, and refreshing the chunk files of which the data volume reaches the set threshold or exceeds a certain period to a database in the storage module.
Further, the storage module is specifically configured to:
and storing the chunk files with the data volume reaching a set threshold or exceeding a certain period through the database, and setting a storage index.
Further, the storage module further includes: the query unit is used for querying the log file according to a query instruction input by a user; the query mode comprises one or more of index query, grep query, large query and label query.
Further, the acquisition and reading module is specifically configured to:
according to log query conditions, querying a target log file from a storage module in a timing pulling mode; wherein, the log query conditions comprise: time range and query tag.
Further, the visualization module is built through a Grafana data visualization tool, and the query result is displayed through a visualization interface.
In a second aspect of the embodiments of the present invention, a container cloud log monitoring management method is provided, where the container cloud log monitoring management is implemented based on a container cloud module, a log resource library module, a log acquisition module, a distribution module, a batch processing and compression module, a storage module, an acquisition and reading module, and a visualization module; wherein,
the container cloud module establishes a container cloud, a log collector is deployed on each node, and a log file is pushed to the log resource library module;
the log resource library module temporarily stores log files generated by the container cloud;
the log acquisition module extracts log files from the log resource library module in a serialized mode, generates corresponding log processing tasks according to the log files, puts the corresponding log processing tasks into a task queue, and sends the log processing tasks to the distribution module;
the distribution module receives the log processing tasks pushed by the log acquisition module, caches the log processing tasks and distributes the log processing tasks to the batch processing and compression module;
the batch processing and compressing module compresses the log file according to the distributed log processing task and attaches the log file to the packed file, and when the packed file meets the preset requirement, a refreshing instruction is sent to the storage module to refresh the packed file to a database in the storage module;
the storage module comprises a file receiving unit and a database; the file receiving unit is matched with the batch processing and compressing module and used for receiving compressed log files; the database stores packed files meeting preset requirements;
the acquisition reading module inquires a target log file from the storage module according to the log inquiry condition sent by the visualization module and returns an inquiry result to the visualization module;
and the visualization module receives log query conditions input by a user and displays query results through a visualization interface.
In a third aspect of the embodiments of the present invention, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the processor implements a log monitoring management method for a container cloud.
In a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, and the computer program, when executed by a processor, implements a log monitoring management method for a container cloud.
In the log monitoring and management process, the compressed unstructured logs are stored, and only indexes are established for metadata, so that the system and the method for monitoring and managing the logs of the container cloud have the advantages of simplicity in operation and low cost; the logs are indexed and grouped through the tags, so that the expansion and operation efficiency of the logs is higher, and the use experience of a user is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of a log monitoring management system of a container cloud according to an embodiment of the present invention.
FIG. 2 is a block diagram of a memory module according to an embodiment of the invention.
FIG. 3 is a schematic diagram of a memory module according to another embodiment of the invention.
Fig. 4 is a schematic structural diagram of a log monitoring management system of a container cloud according to an embodiment of the present invention.
Fig. 5 is a schematic flowchart of a log monitoring management method of a container cloud according to an embodiment of the present invention.
Fig. 6 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The principles and spirit of the present invention will be described with reference to a number of exemplary embodiments. It is understood that these embodiments are given solely for the purpose of enabling those skilled in the art to better understand and to practice the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As will be appreciated by one skilled in the art, embodiments of the present invention may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
According to the embodiment of the invention, a system and a method for monitoring and managing logs of a container cloud are provided, and the technical field of log monitoring and management is related. The invention can avoid full-text indexing of the log, and stores the compressed unstructured log, and has the advantages of simple operation and low cost; by using the label same as Prometheus and grouping the log cables, the expansibility and the operation efficiency are greatly improved; metadata such as Pod tags can be automatically deleted and indexed, and compatibility with K8s is high; the method has higher native support to a visualization module Grafana.
In the embodiments of the present invention, terms to be described include:
logging: network equipment, a system, a service program and the like, wherein event records can be generated during operation; each row of the log records the description of the date, time, user and action.
A container: is a minimal image of the operating system on which applications may run.
A container cloud: the container and container schedule are run to the cloud, so that the combined service is the container cloud.
Kubernetes: for short, K8s is an orchestration management tool of a portable container generated for container service, and currently, K8s already dominates a cloud business process, and can implement functions of automated deployment, automatic expansion and contraction, maintenance, and the like of a container cluster.
Pod: pod is the basic unit in kubernets, and the container itself is not directly distributed to the host, but rather is packaged into a Pod object. A Pod typically represents a single application, consisting of one or more associated containers that all have the same lifecycle and are scheduled as a whole on the same node, the containers sharing the environment, storage volumes, and IP controls. Although there may be multiple containers in Pod, in kubernets it is the Pod that is the smallest unit to schedule, scale and share resources, manage lifecycle.
daemonSet: the daemon set is similar to the daemon set, and one Pod is deployed on each node meeting the matching condition. daemonSet ensures that each (partial) node in the cluster runs one copy of the pod, creates the pod when the node joins the cluster, and recovers the pod when the node leaves the cluster. If daemonSet is deleted, all the pods it creates are also deleted, and the pod in daemonSet covers the entire cluster.
Docker: by the open-source application container engine, developers can pack their applications and rely on a portable container to distribute the applications to popular Linux machines, and virtualization can be realized.
stdout: a standard output device; stderr: a standard error output device. Both output to the screen by default. But if the steering standard is used for outputting to the disk file, the difference between the two can be seen. stdout is output to the disk file, with stderr on the screen. By default, stdout is line buffered, and its output will be placed in a buffer, and will only be output to the screen when the line is wrapped. Whereas stderr is unbuffered and will output directly.
grep query: grep is a command in the Linux command line that is commonly used to find the content of filtered text files.
Grafana: a front-end display tool displays data in a very beautiful and direct graph; the system can be used for data monitoring and data statistics and has an alarm function.
The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the invention.
Fig. 1 is a schematic diagram of a log monitoring management system of a container cloud according to an embodiment of the present invention. As shown in fig. 1, the system includes: the system comprises a container cloud module 110, a log resource library module 120, a log acquisition module 130, a distribution module 140, a batch processing and compression module 150, a storage module 160, an acquisition and reading module 170 and a visualization module 180; wherein,
the container cloud module 110 is configured to establish a container cloud, deploy a log collector on each node, and push a log file to the log resource library module;
the log repository module 120 is configured to temporarily store a log file generated by a container cloud;
the log collection module 130 is configured to extract log files from the log resource library module in a serialized manner, generate corresponding log processing tasks according to the log files, place the corresponding log processing tasks into a task queue, and send the log processing tasks to the distribution module;
the distribution module 140 is configured to receive the log processing task pushed by the log acquisition module, cache the log processing task, and distribute the log processing task to the batch processing and compression module;
the batch processing and compressing module 150 is configured to compress the log file according to the distributed log processing task and attach the compressed log file to the packed file, and when the packed file meets a preset requirement, send a refresh instruction to the storage module and refresh the packed file to a database in the storage module;
referring to fig. 2, the storage module 160 includes a file receiving unit and a database; the file receiving unit is used for being matched with the batch processing and compressing module and receiving the compressed log files; the database is used for storing the packed files meeting the preset requirements;
the acquisition reading module 170 is configured to query the storage module for a target log file according to the log query condition sent by the visualization module, and return a query result to the visualization module;
the visualization module 180 is configured to receive a log query condition input by a user, and display a query result through a visualization interface.
In order to explain the above log monitoring management system of the container cloud more clearly, the following description is made in detail in conjunction with each module.
In this embodiment, the container cloud module 110 is specifically configured to:
and establishing a private container cloud based on Kubernetes + Docker, and pushing a log file to the log resource library module through standard output equipment and standard error output equipment.
In this embodiment, the log resource library module 120 is specifically configured to:
and setting a file acquisition API (application programming interface) interface for the log acquisition module to extract log files.
In this embodiment, the log collection module 130 is specifically configured to:
and marking a label for the log file.
In this embodiment, the distribution module 140 is specifically configured to:
according to the metadata of the log processing task, adopting a hash algorithm to obtain distribution data;
and distributing the log processing task to corresponding processing units in the batch processing and compression module according to the distribution data.
In this embodiment, the batch processing and compressing module 150 is specifically configured to:
and according to the distributed log processing tasks, compressing the log files and attaching the compressed log files to the chunk files, when the data volume of the chunk files reaches a set threshold or exceeds a fixed period, sending a refreshing instruction to the storage module, and refreshing the chunk files of which the data volume reaches the set threshold or exceeds a certain period to a database in the storage module.
In this embodiment, the storage module 160 is specifically configured to:
and storing the chunk files with the data volume reaching a set threshold or exceeding a certain period through the database, and setting a storage index.
Referring to fig. 3, the storage module 160 further includes: the query unit is used for querying the log file according to a query instruction input by a user; the query mode comprises one or more of index query, grep query, large query and label query.
In this embodiment, the collection reading module 170 is specifically configured to:
according to log query conditions, querying a target log file from a storage module in a timing pulling mode; wherein, the log query conditions comprise: time range and query tag.
In this embodiment, the visualization module 180 is built through a Grafana data visualization tool, and displays the query result through a visualization interface.
It should be noted that although several modules of the log monitoring management system of the container cloud are mentioned in the above detailed description, such partitioning is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the modules described above may be embodied in one module according to embodiments of the invention. Conversely, the features and functions of one module described above may be further divided into embodiments by a plurality of modules.
For a more clear explanation of the above container cloud log monitoring management system, a specific embodiment is described below.
Fig. 4 is a schematic diagram of an architectural relationship of a log monitoring management system of a container cloud according to an embodiment of the present invention.
As shown in fig. 4, the architecture relationship is:
a container cloud module:
establishing a private container cloud based on Kubernetes + Docker, deploying a log agent on each node of K8s, and pushing a log file to a log resource library in a stdout and stderr mode. The log Agent is a log collection process, and one Agent process is a complete data collection tool.
A log repository module:
the container cloud generated log files are temporarily stored and provided with the capability of providing a file collection API.
A log collection module:
log files are extracted in a serialization mode from the log resource library module, and label marking work is completed;
generating corresponding Job aiming at each log file, and putting the corresponding Job into a maintained Job queue; sending Job to the distribution module; for the daemoset mode for each container, collection of SDK-like articles is provided for integration.
A distribution module:
receiving Job pushed by the acquisition module;
because the write-in amount of the logs can be large, the logs cannot be written into the database when the logs are transmitted, the distribution module has a first-layer cache function, and calculates a target batch processing module according to metadata and a hash algorithm, so that Job distribution is completed, and consistency is ensured.
Batch processing and compression module:
the log is compressed and appended to chunk. Once the chunk is "full", i.e., the data reaches a certain amount or has passed a certain period, the module sends a "refresh" signal to the memory module and the full inch chunk in the memory module will be refreshed into the database.
A storage module:
the system comprises two parts, wherein the first part is a chunk area which is matched with a batch processing compression module and used for receiving compressed log files;
the second part is a database module which stores the index and the journal file after the chunk is full. The database module provides functions of index query, parallel grep query, large query, label selector and the like;
the acquisition and reading module:
and inquiring parameters such as tags according to the time range transmitted by the visualization module, and inquiring the target log file from the storage module in a timing pulling mode. The functions of large-scale query, parallel query and the like are supported, and query results are returned to the visualization module;
a visualization module:
facing to a user, providing log visualization effect display and allowing the user to input log query conditions; and sending the query conditions of the user to a collection reading module, and returning and displaying the query conditions in real time, wherein the module can be realized based on Grafana.
The invention provides a container cloud log monitoring and management system based on Loki, labels the same as Prometheus are used as indexes, the labels can be used for inquiring the content of a log and inquiring monitored data, the switching cost between two kinds of inquiry is reduced, and the storage of log indexes is greatly reduced. The invention will use the same service discovery and tag re-tagging library as Prometheus, write Promtail, run Promtail in a daemonset manner in each node in K8s, get the correct metadata of the log through Kubernetes API, and send them to Loki. Loki is the latest open source project of Grafana Labs team, and is a log aggregation system with the advantages of horizontal scalability, high availability and multiple tenants.
Compared with the existing log aggregation system, the log aggregation system has at least the following advantages:
full-text indexing of the log is avoided. The compressed unstructured log is stored, and only indexes are built on the metadata, so that the method has the advantages of being simple in operation and low in cost.
The same tag as Prometheus was used. The logs are indexed and grouped by the tags, which makes the expansion and operation of the logs more efficient.
Suitable for storing kubernets Pod logs, metadata such as Pod tags are automatically deleted and indexed.
Has higher native support to Grafana.
Having described the system of an exemplary embodiment of the present invention, a log monitoring management system of a container cloud of an exemplary embodiment of the present invention is next described with reference to fig. 5.
The implementation of the log monitoring and management method of the container cloud can be referred to the implementation of the system, and repeated details are not repeated.
Based on the same invention concept, the invention also provides a container cloud log monitoring and management method, wherein the container cloud log monitoring and management is realized based on a container cloud module, a log resource library module, a log acquisition module, a distribution module, a batch processing and compression module, a storage module, an acquisition and reading module and a visualization module.
As shown in fig. 5, the method includes:
step S1, the container cloud module establishes a container cloud, and a log collector is deployed on each node to push log files to the log resource library module;
step S2, the log resource library module temporarily stores the log file generated by the container cloud;
step S3, the log collection module extracts log files from the log resource library module in a serialization way, generates corresponding log processing tasks according to the log files, puts the corresponding log processing tasks into a task queue and sends the log processing tasks to the distribution module;
step S4, the distribution module receives the log processing task pushed by the log collection module, caches the log processing task, and distributes the log processing task to the batch processing and compression module;
step S5, the batch processing and compressing module compresses the log file and attaches the log file to the packed file according to the distributed log processing task, when the packed file meets the preset requirement, a refreshing instruction is sent to the storage module, and the packed file is refreshed to the database in the storage module;
step S6, the storage module comprises a file receiving unit and a database; the file receiving unit is matched with the batch processing and compressing module and used for receiving compressed log files; the database stores packed files meeting preset requirements;
step S7, the acquisition and reading module inquires a target log file from the storage module according to the log inquiry condition sent by the visualization module and returns the inquiry result to the visualization module;
and step S8, the visualization module receives the log query condition input by the user and displays the query result through the visualization interface.
It should be noted that although the operations of the method of the present invention have been described in the above embodiments and the accompanying drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the operations shown must be performed, to achieve the desired results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
Based on the aforementioned inventive concept, as shown in fig. 6, the present invention further provides a computer device 600, which includes a memory 610, a processor 620, and a computer program 630 stored on the memory 610 and operable on the processor 620, wherein the processor 620 implements the aforementioned log monitoring management method for the container cloud when executing the computer program 630.
Based on the foregoing inventive concept, the present invention provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the aforementioned log monitoring management method for a container cloud.
In the log monitoring and management process, the compressed unstructured logs are stored, and only indexes are established for metadata, so that the system and the method for monitoring and managing the logs of the container cloud have the advantages of simplicity in operation and low cost; the logs are indexed and grouped through the tags, so that the expansion and operation efficiency of the logs is higher, and the use experience of a user is improved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (13)
1. A container cloud log monitoring management system, the system comprising: the system comprises a container cloud module, a log resource library module, a log acquisition module, a distribution module, a batch processing and compression module, a storage module, an acquisition and reading module and a visualization module; wherein,
the container cloud module is used for establishing a container cloud, deploying a log collector on each node and pushing a log file to the log resource library module;
the log resource library module is used for temporarily storing the log file generated by the container cloud;
the log acquisition module is used for extracting log files from the log resource library module in a serialized mode, generating corresponding log processing tasks according to the log files, putting the corresponding log processing tasks into a task queue, and sending the log processing tasks to the distribution module;
the distribution module is used for receiving the log processing tasks pushed by the log acquisition module, caching the log processing tasks and distributing the log processing tasks to the batch processing and compression module;
the batch processing and compressing module is used for compressing the log file according to the distributed log processing task and attaching the log file to the packed file, and when the packed file meets the preset requirement, a refreshing instruction is sent to the storage module, and the packed file is refreshed to a database in the storage module;
the storage module comprises a file receiving unit and a database; the file receiving unit is used for being matched with the batch processing and compressing module and receiving the compressed log files; the database is used for storing the packed files meeting the preset requirements;
the acquisition reading module is used for inquiring a target log file from the storage module according to the log inquiry condition sent by the visualization module and returning an inquiry result to the visualization module;
and the visualization module is used for receiving the log query conditions input by the user and displaying the query result through the visualization interface.
2. The container cloud log monitoring and management system according to claim 1, wherein the container cloud module is specifically configured to:
and establishing a private container cloud based on Kubernetes + Docker, and pushing a log file to the log resource library module through standard output equipment and standard error output equipment.
3. The container cloud log monitoring and management system according to claim 1, wherein the log repository module is specifically configured to:
and setting a file acquisition API (application programming interface) interface for the log acquisition module to extract log files.
4. The container cloud log monitoring and management system according to claim 1, wherein the log collection module is specifically configured to:
and marking a label for the log file.
5. The container cloud log monitoring and management system according to claim 1, wherein the distribution module is specifically configured to:
according to the metadata of the log processing task, adopting a hash algorithm to obtain distribution data;
and distributing the log processing task to corresponding processing units in the batch processing and compression module according to the distribution data.
6. The container cloud log monitoring and management system according to claim 4, wherein the batch processing and compression module is specifically configured to:
and according to the distributed log processing tasks, compressing the log files and attaching the compressed log files to the chunk files, when the data volume of the chunk files reaches a set threshold or exceeds a fixed period, sending a refreshing instruction to the storage module, and refreshing the chunk files of which the data volume reaches the set threshold or exceeds a certain period to a database in the storage module.
7. The container cloud log monitoring and management system according to claim 6, wherein the storage module is specifically configured to:
and storing the chunk files with the data volume reaching a set threshold or exceeding a certain period through the database, and setting a storage index.
8. The container cloud log monitoring and management system according to claim 7, wherein the storage module further comprises: the query unit is used for querying the log file according to a query instruction input by a user; the query mode comprises one or more of index query, grep query, large query and label query.
9. The container cloud log monitoring and management system according to claim 7, wherein the collection reading module is specifically configured to:
according to log query conditions, querying a target log file from a storage module in a timing pulling mode; wherein, the log query conditions comprise: time range and query tag.
10. The container cloud log monitoring and management system according to claim 1, wherein the visualization module is built through a Grafana data visualization tool, and the query result is displayed through a visualization interface.
11. The log monitoring and management method of the container cloud is characterized in that the log monitoring and management of the container cloud is realized on the basis of a container cloud module, a log resource library module, a log acquisition module, a distribution module, a batch processing and compression module, a storage module, an acquisition and reading module and a visualization module; wherein,
the container cloud module establishes a container cloud, a log collector is deployed on each node, and a log file is pushed to the log resource library module;
the log resource library module temporarily stores log files generated by the container cloud;
the log acquisition module extracts log files from the log resource library module in a serialized mode, generates corresponding log processing tasks according to the log files, puts the corresponding log processing tasks into a task queue, and sends the log processing tasks to the distribution module;
the distribution module receives the log processing tasks pushed by the log acquisition module, caches the log processing tasks and distributes the log processing tasks to the batch processing and compression module;
the batch processing and compressing module compresses the log file according to the distributed log processing task and attaches the log file to the packed file, and when the packed file meets the preset requirement, a refreshing instruction is sent to the storage module to refresh the packed file to a database in the storage module;
the storage module comprises a file receiving unit and a database; the file receiving unit is matched with the batch processing and compressing module and used for receiving compressed log files; the database stores packed files meeting preset requirements;
the acquisition reading module inquires a target log file from the storage module according to the log inquiry condition sent by the visualization module and returns an inquiry result to the visualization module;
and the visualization module receives log query conditions input by a user and displays query results through a visualization interface.
12. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of claim 11 when executing the computer program.
13. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the method of claim 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110848017.2A CN113468111A (en) | 2021-07-27 | 2021-07-27 | Log monitoring management system and method for container cloud |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110848017.2A CN113468111A (en) | 2021-07-27 | 2021-07-27 | Log monitoring management system and method for container cloud |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113468111A true CN113468111A (en) | 2021-10-01 |
Family
ID=77882527
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110848017.2A Pending CN113468111A (en) | 2021-07-27 | 2021-07-27 | Log monitoring management system and method for container cloud |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113468111A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114756301A (en) * | 2022-04-24 | 2022-07-15 | 北京百度网讯科技有限公司 | Log processing method, device and system |
| CN114861773A (en) * | 2022-04-18 | 2022-08-05 | 深圳市欢太科技有限公司 | Model training visualization method, device and cloud platform |
| CN115361289A (en) * | 2022-08-23 | 2022-11-18 | 北京字跳网络技术有限公司 | Problem positioning method and device of business system, electronic equipment and storage medium |
| CN115757041A (en) * | 2022-11-28 | 2023-03-07 | 安超云软件有限公司 | Dynamically configurable multi-cluster log collection method and application |
| CN115934817A (en) * | 2022-11-30 | 2023-04-07 | 天翼电子商务有限公司 | Method and system for realizing log acquisition, query and storage based on cloud-native component |
| CN115941454A (en) * | 2022-11-28 | 2023-04-07 | 中国银行股份有限公司 | Method and device for log transmission and storage |
| CN116881174A (en) * | 2023-09-08 | 2023-10-13 | 富钛字节车载软件(长春)有限公司 | ECU application software closed-loop test method, device, equipment and storage medium |
| CN117149730A (en) * | 2023-08-14 | 2023-12-01 | 武汉光庭信息技术股份有限公司 | Log management system, method, equipment and storage medium based on cloud native technology |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102724063A (en) * | 2012-05-11 | 2012-10-10 | 北京邮电大学 | Log collection server, data packet delivering and log clustering methods and network |
| CN108897642A (en) * | 2018-06-27 | 2018-11-27 | 清华大学 | The optimization method and device of log mechanism in persistence transactional memory system |
| CN109542750A (en) * | 2018-11-26 | 2019-03-29 | 深圳天源迪科信息技术股份有限公司 | Distributed information log system |
| CN111209011A (en) * | 2019-12-31 | 2020-05-29 | 烽火通信科技股份有限公司 | Cross-platform container cloud automatic deployment system |
| CN111352921A (en) * | 2020-02-19 | 2020-06-30 | 中国平安人寿保险股份有限公司 | ELK-based slow query monitoring method and device, computer equipment and storage medium |
| CN112667698A (en) * | 2021-01-04 | 2021-04-16 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
-
2021
- 2021-07-27 CN CN202110848017.2A patent/CN113468111A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102724063A (en) * | 2012-05-11 | 2012-10-10 | 北京邮电大学 | Log collection server, data packet delivering and log clustering methods and network |
| CN108897642A (en) * | 2018-06-27 | 2018-11-27 | 清华大学 | The optimization method and device of log mechanism in persistence transactional memory system |
| CN109542750A (en) * | 2018-11-26 | 2019-03-29 | 深圳天源迪科信息技术股份有限公司 | Distributed information log system |
| CN111209011A (en) * | 2019-12-31 | 2020-05-29 | 烽火通信科技股份有限公司 | Cross-platform container cloud automatic deployment system |
| CN111352921A (en) * | 2020-02-19 | 2020-06-30 | 中国平安人寿保险股份有限公司 | ELK-based slow query monitoring method and device, computer equipment and storage medium |
| CN112667698A (en) * | 2021-01-04 | 2021-04-16 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114861773A (en) * | 2022-04-18 | 2022-08-05 | 深圳市欢太科技有限公司 | Model training visualization method, device and cloud platform |
| CN114756301A (en) * | 2022-04-24 | 2022-07-15 | 北京百度网讯科技有限公司 | Log processing method, device and system |
| CN114756301B (en) * | 2022-04-24 | 2023-09-01 | 北京百度网讯科技有限公司 | Log processing method, device and system |
| CN115361289A (en) * | 2022-08-23 | 2022-11-18 | 北京字跳网络技术有限公司 | Problem positioning method and device of business system, electronic equipment and storage medium |
| CN115757041A (en) * | 2022-11-28 | 2023-03-07 | 安超云软件有限公司 | Dynamically configurable multi-cluster log collection method and application |
| CN115941454A (en) * | 2022-11-28 | 2023-04-07 | 中国银行股份有限公司 | Method and device for log transmission and storage |
| CN115757041B (en) * | 2022-11-28 | 2023-11-21 | 安超云软件有限公司 | Method for collecting dynamically configurable multi-cluster logs and application |
| CN115934817A (en) * | 2022-11-30 | 2023-04-07 | 天翼电子商务有限公司 | Method and system for realizing log acquisition, query and storage based on cloud-native component |
| CN117149730A (en) * | 2023-08-14 | 2023-12-01 | 武汉光庭信息技术股份有限公司 | Log management system, method, equipment and storage medium based on cloud native technology |
| CN116881174A (en) * | 2023-09-08 | 2023-10-13 | 富钛字节车载软件(长春)有限公司 | ECU application software closed-loop test method, device, equipment and storage medium |
| CN116881174B (en) * | 2023-09-08 | 2023-11-14 | 富钛字节车载软件(长春)有限公司 | ECU application software closed-loop test method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113468111A (en) | Log monitoring management system and method for container cloud | |
| CN108170768B (en) | Database synchronization method, device and readable medium | |
| CN111459763B (en) | Cross-kubernetes cluster monitoring system and method | |
| US10990288B2 (en) | Systems and/or methods for leveraging in-memory storage in connection with the shuffle phase of MapReduce | |
| CN109739815B (en) | File processing method, system, device, equipment and storage medium | |
| US8762931B2 (en) | Generating an encoded package profile | |
| CN102375837B (en) | Data acquiring system and method | |
| US20140358977A1 (en) | Management of Intermediate Data Spills during the Shuffle Phase of a Map-Reduce Job | |
| CN107423404B (en) | Flow instance data synchronous processing method and device | |
| US10909000B2 (en) | Tagging data for automatic transfer during backups | |
| US9348855B2 (en) | Supporting big data in enterprise content management systems | |
| CN111966677A (en) | Data report processing method and device, electronic equipment and storage medium | |
| CN102779138B (en) | The hard disk access method of real time data | |
| CN102857578A (en) | File uploading method and file uploading system of network drive and network drive client | |
| CN105205143A (en) | File storage and processing method, device and system | |
| CN105045905A (en) | Log maintenance method and system based on full-text retrieval | |
| CN108829505A (en) | A kind of distributed scheduling system and method | |
| US10241716B2 (en) | Global occupancy aggregator for global garbage collection scheduling | |
| CN110532058B (en) | Management method, device and equipment of container cluster service and readable storage medium | |
| EP4231602A1 (en) | Method and apparatus for acquiring private cloud container cluster gene information | |
| US20190057028A1 (en) | Conflict Resolution and Garbage Collection in Distributed Databases | |
| Singh | Cluster-level logging of containers with containers: Logging challenges of container-based cloud deployments | |
| Zhou et al. | Sfmapreduce: An optimized mapreduce framework for small files | |
| CN110633274B (en) | Alarm management method and device | |
| CN109033196A (en) | A kind of distributed data scheduling system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |