CN115269516A - Log data acquisition management method, system, storage medium and electronic equipment - Google Patents
Log data acquisition management method, system, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN115269516A CN115269516A CN202210770816.7A CN202210770816A CN115269516A CN 115269516 A CN115269516 A CN 115269516A CN 202210770816 A CN202210770816 A CN 202210770816A CN 115269516 A CN115269516 A CN 115269516A
- Authority
- CN
- China
- Prior art keywords
- data
- log data
- load balancer
- processing
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 56
- 238000003860 storage Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 109
- 238000013480 data collection Methods 0.000 claims abstract description 30
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000012544 monitoring process Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 2
- 230000002688 persistence Effects 0.000 abstract description 10
- 230000008878 coupling Effects 0.000 abstract description 6
- 238000010168 coupling process Methods 0.000 abstract description 6
- 238000005859 coupling reaction Methods 0.000 abstract description 6
- 238000013523 data management Methods 0.000 abstract description 5
- 238000012423 maintenance Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 11
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000037406 food intake Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a log data acquisition management method, a system, a storage medium and an electronic device, comprising: acquiring log data acquired by each front-end server; receiving a data processing configuration type configured for each log data by the data processing dynamic configuration management platform; each data processing configuration type corresponds to one processing system; and on the basis of the acquisition time sequence, all the log data are sequentially sent to a target message queue, so that each log data is sequentially sent to a corresponding processing system for processing through the target message queue. The invention realizes the classified processing and the persistence according to the category of the log data by constructing the data processing configuration type, solves the coupling problem of the front end node and the back end node, and improves the data collection capability of the message queue. The data management mode adopted by the method greatly reduces the data access cost and reduces the operation and maintenance manpower.
Description
Technical Field
The invention relates to the technical field of data acquisition, in particular to a log data acquisition management method, a log data acquisition management system, a storage medium and electronic equipment.
Background
Data acquisition, also known as data acquisition, refers to the process of automatically acquiring information from analog and digital units under test, such as sensors and other devices under test. Data acquisition technologies have been widely used for real-time acquisition and summarization of online service data, log files, machine data, and the like of enterprises. At present, general log data acquisition needs to be completed by the cooperation of a front end and a back end, a data production end uses a data acquisition service to capture real-time data, a data summarization end uses a data acquisition service to receive front end data flow, data transmission needs to be carried out based on a network in the process, and the front end seriously has close coupling to the back end. The online log data to be collected are often in various types, the data returned to the data collecting end have different real-time processing logics, and simultaneously, the final flow direction of the data is closely related to the actual data along with the newly added data types, so that the log data persistence operation cannot be completed through a uniform processing flow. How to decouple the front end and the back end and realize real-time dynamic classification and processing persistence of log data is a problem which needs to be solved urgently.
Disclosure of Invention
In order to solve the technical problem, the invention provides a log data acquisition management method, a log data acquisition management system, a storage medium and electronic equipment.
The technical scheme of the log data acquisition management method is as follows:
acquiring log data acquired by each front-end server;
receiving a data processing configuration type configured for each log data by the data processing dynamic configuration management platform; each data processing configuration type corresponds to one processing system;
and on the basis of the acquisition time sequence, all the log data are sequentially sent to a target message queue, so that each log data is sequentially sent to a corresponding processing system for processing through the target message queue.
The log data acquisition management method has the following beneficial effects:
the method realizes the classification processing and the persistence according to the category of the log data by constructing the data processing configuration type, solves the coupling problem of the front end node and the rear end node, and improves the data collection capability of the message queue. The data management mode adopted by the method greatly reduces the data access cost and reduces the operation and maintenance manpower.
On the basis of the above scheme, the log data acquisition management method of the invention can be further improved as follows.
Further, the receiving log data collected by each front-end server includes:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
Further, the load balancer is: HTTP/TCP load balancer.
Further, the data processing configuration type is as follows: a data processing type, a data forwarding type, a data approval type or a data monitoring type.
Further, the target message queue is: KAFKA message queue.
The technical scheme of the log data acquisition management system is as follows:
the method comprises the following steps: the system comprises a receiving module, a distribution module and a sending module;
the receiving module is used for: acquiring log data acquired by each front-end server;
the allocation module is configured to: receiving a data processing configuration type configured for each log data by the data processing dynamic configuration management platform; each data processing configuration type corresponds to one processing system;
the sending module is used for: and on the basis of the acquisition time sequence, all the log data are sequentially sent to a target message queue, so that each log data is sequentially sent to a corresponding processing system through the target message queue for processing.
The log data acquisition management system has the beneficial effects that:
the system realizes the classified processing and the persistence according to the category of the log data by constructing the data processing configuration type, solves the coupling problem of front and back end nodes, and improves the data collection capability of the message queue. The data management mode adopted by the system greatly reduces the data access cost and reduces the operation and maintenance manpower.
On the basis of the above scheme, the log data acquisition management system of the present invention may further be improved as follows.
Further, the receiving module is specifically configured to:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
Further, the load balancer is: HTTP/TCP load balancer.
The technical scheme of the storage medium of the invention is as follows:
the storage medium stores instructions that, when read by a computer, cause the computer to perform the steps of a log data collection management method according to the present invention.
The technical scheme of the electronic equipment is as follows:
the log data collection management method comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, and is characterized in that when the processor executes the computer program, the computer is caused to execute the steps of the log data collection management method.
Drawings
Fig. 1 is a first flowchart of a log data collection management method according to an embodiment of the present invention;
fig. 2 is a second flowchart of a log data collection management method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a log data acquisition management system according to an embodiment of the present invention.
Detailed Description
As shown in fig. 1, a log data collection management method according to an embodiment of the present invention includes the following steps:
s1, acquiring log data collected by each front-end server.
Wherein, the log data refers to: business data generated by various business services are output to a log file in a log form in real time; and the service running logs generated by various online services record the processing flow information, the interface interaction information, the resource use information and the like of the current service.
The front-end server has the following functions: and the system is responsible for monitoring the log files and transmitting the read log data back to the back-end server in batch. The number of the log data collected by each front-end server is at least one.
Specifically, the back-end server obtains (receives) the log data collected by each front-end server, and obtains all the log data.
And S2, receiving the data processing configuration type configured by the data processing dynamic configuration management platform for each log data.
The log data collected by the front-end collection service (front-end server) may have a plurality of categories, each category has a corresponding data structure and data content, the log data reported by the front-end server can be constructed into a character string in the original state during collection, and meanwhile, some meta information (information such as collection time, data category identification of the log data, data size, server where the data is located, and the like) can be added to the data.
When receiving any piece of data, the back-end collection service (back-end server) firstly needs to read the meta information, obtains the data type, then processes the original data according to the logic (data processing configuration type) configured by the dynamic configuration management platform to obtain a piece of downstream available data, and writes the downstream available data into the target message queue.
It should be noted that the front end and the back end need to have a set of mapping relationships, and uniformly manage the data processing configuration types of each log data, such as a data processing type (real-time processing logic), a data forwarding type (data forwarding logic), and the like, which all need to be implemented by a dynamic configuration management platform.
Wherein each data processing configuration type corresponds to one processing system.
The dynamic configuration management platform has the following functions: is responsible for managing the dynamic configuration of the data collection service (i.e., the type of configuration for each log data transaction in the back-end server). The data processing configuration types include: a data processing type, a data forwarding type, a data approval type or a data monitoring type.
The data processing configuration types include, but are not limited to: a data processing type, a data forwarding type, a data approval type, or a data monitoring type. Each data processing configuration type corresponds to a different processing system (data processor). For example, when the data processing type of any log data is a data forwarding type, the processing system corresponding to the log data is a data forwarding processing system.
It should be noted that, in this embodiment, the processing system includes but is not limited to: data processing systems, data forwarding systems, data approval (operation approval) systems, data monitoring systems, and the like. Wherein, the data processing system is used for: and formatting the log data, and cleaning the log format, checking the field type, filtering abnormal data and the like. The data forwarding system has the following functions: different types of data often need partition persistence, and data with different usages are stored in corresponding message queues, and at the moment, forwarding dynamic update of data persistence needs to be supported. The data approval (operation approval) system has the functions of: when the data processing configuration is updated for the back-end collection service, an approval process is needed, the configuration meeting the regulations can be on-line, and the approval process supports multi-level and multi-person approval. The data monitoring system has the following functions: the log data are subjected to real-time statistics and recorded to data such as flow, size and time consumption of various types of data reported to regions and machine rooms, graphical monitoring is formed by visualization of the data, and meanwhile threshold value alarming of the data can be achieved.
Specifically, the back-end server is in butt joint with the data processing dynamic configuration management platform to obtain the data processing configuration type configured by the dynamic configuration management platform for each log data.
The number of front-end servers is plural.
And S3, based on the acquisition time sequence, sequentially sending all the log data to a target message queue, and sequentially sending each log data to a corresponding processing system for processing through the target message queue.
Specifically, the back-end server sends all log data to the target message queue in sequence according to the collection time sequence of each log data, and the target message queue sends each log data to the corresponding processing system for processing according to the first-in first-out sequence (i.e. the sequence of the collection time).
The target message queue adopted in this embodiment is: a KAFKA message queue.
It should be noted that the application fields of the technical solution of the present embodiment include, but are not limited to:
(1) enterprise-level real-time and offline warehouse construction: large-scale data acquisition information needs to be constructed, service logs of multiple areas and multiple machine rooms can be acquired and summarized, the service logs are acquired in real time and reported to a terminal in batches, data are classified and processed to enter a message queue uniformly, and finally, online/offline processing is carried out by a large data warehouse;
(2) large-scale online service alarm system: by the technical scheme of the embodiment, logs of thousands of services are collected to the terminal and are uniformly stored in the message queue, and business service monitoring and alarming are realized through the real-time data analysis engine.
Preferably, the S1 includes:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
Wherein, the load balancer is: an HTTP/TCP load balancer; the load balancer is bound on a public domain name and used for realizing the forwarding of log data.
Specifically, as shown in fig. 2, each front-end server (remote data acquisition service) acquires log data reported by each region, and establishes a connection between the remote data acquisition service and the load balancer through a public network domain name, when the remote data acquisition service transmits the log data to the public network domain name, the remote data acquisition service obtains an IP of the corresponding HTTP/TCP load balancer through DNS resolution, and at this time, the remote data acquisition service performs data transmission through the IP and the HTTP/TCP load balancer; then, sending the log data to a back-end server (back-end data collection service) through each HTTP/TCP load balancer; the back-end server can comprise a plurality of node nodes (data collection service nodes), each node receives configuration information of the data processing dynamic configuration management platform, and then sends all log data to a target message queue (KAFKA) according to the data processing configuration type of each log data, so that each log data is sent to a corresponding processing system through the target message queue for processing.
The data collection service is composed of a plurality of peer-to-peer service processes, the peer-to-peer service processes form a cluster by a plurality of services, each node in the cluster is a process capable of receiving data, and the function and the capability of each node are the same. The cluster scale is mainly planned according to the reported log data volume, and when the log data reported in real time is large, the nodes for collecting service can be transversely expanded, so that the data acquisition capacity is expanded.
The public network domain name has the following functions: and establishing network connection between the front-end server and the load balancer, and enabling the front-end server to acquire the IP of the corresponding load balancer. Specifically, in order to not directly bind an IP, a public network domain name needs to be applied, and the IP of the load balancer is found through DNS domain name resolution, and the front-end server only needs to initiate connection to the public network domain name and send log data, and does not need to know which load balancer specifically performs data exchange.
It should be noted that each front-end server collects log data in real time through a data collection node, constructs meta information for each data, and identifies information such as data type, size, time, and the like. And the back-end server selects a data processing configuration type corresponding to the data type of the meta-information according to the data type of each meta-information, and uniformly and synchronously sends all log data to the KAFKA message queue after the configuration of all log data is finished. The back-end server adopts an open source component flash to realize the collection and receiving functions of log data, expands a dynamic data interceptor, a dynamic data forwarder and a dynamic data persistence channel, issues data processing configuration to a corresponding back-end server through a data processing dynamic configuration management platform to realize the dynamic processing of real-time log data, and the data processing dynamic configuration management platform interacts with nodes of the back-end server (actual collection service) based on an HTTP interface to realize the issuing and updating of dynamic configuration.
It should be noted that a data processing dynamic configuration management platform is constructed to implement visualization of dynamic configuration of data processing, data forwarding, data persistence, and the like, and is provided with an online approval process, which supports functions of multilevel approval, gray level updating, full online, rollback, and the like, and supports functions of service monitoring, data monitoring, and the like.
According to the technical scheme, the log data is classified and processed and persisted according to the classes by constructing the data processing configuration type, the problem of coupling between front and rear end nodes is solved, and the data collection capacity of the message queue is improved. The technical scheme of the embodiment adopts a data management mode, so that the data access cost is greatly reduced, and the operation and maintenance manpower is reduced.
In a second embodiment (for example, mei qu) of the log data collection management method of the present invention, the method includes:
s10, the back-end server acquires log data acquired by each front-end server; wherein, the log data is: the method comprises the steps of collecting log data related to services in regions all over the country, wherein each log data corresponds to one service type.
S20, the back-end server receives the data processing configuration type configured by the data processing dynamic configuration management platform for each log data; each data processing configuration type corresponds to one service processing system;
and S30, based on the acquisition time sequence, sequentially sending all the log data to a target message queue, and sequentially sending each log data to a corresponding service processing system for processing through the target message queue.
Specifically, the S10 includes:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
Wherein the load balancer is: HTTP/TCP load balancer.
Wherein the data processing configuration type is as follows: a data processing type, a data forwarding type, a data approval type or a data monitoring type.
Wherein the target message queue is: a KAFKA message queue.
By the scheme of the second embodiment, the construction of the enterprise-level large data platform warehouse can be realized. Specifically, the mei-rou builds a unified data collection platform by means of flume, integrates functions of log collection, data reporting, data capturing and the like into a unified system, exposes a data reporting interface through a unified public network domain name, and supports multi-region log collection and reporting and mobile terminal sdk data reporting; and the method supports active data ingestion and can realize the end-to-end carrying capacity of the log data. Data acquired in a main and passive mode uniformly enter a system for processing and are distributed to message queues of all service lines in real time, and data persistence capability of various modes is supported.
In a third embodiment (taking thunder as an example) of a log data collection management method of the present invention, the method includes:
s100, acquiring log data acquired by each front-end server; wherein, the log data is: and the server operation data collected by each front-end server is used for service monitoring and real-time warning.
S200, receiving a data processing configuration type configured for each log data by the data processing dynamic configuration management platform; wherein each data processing configuration type corresponds to one processing system.
S300, based on the acquisition time sequence, all the log data are sequentially sent to a target message queue, and each log data is sequentially sent to a corresponding processing system through the target message queue for processing.
Specifically, the S100 includes:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
Wherein, the load balancer is: HTTP/TCP load balancer.
The data processing configuration type is mainly a data monitoring type, and is assisted by a data processing type, a data forwarding type and a data approval type.
Wherein the target message queue is: a KAFKA message queue.
Through the scheme of the third embodiment, the online service alarm system is realized. The method comprises the steps that service monitoring and real-time alarming of tens of thousands of CDN servers around the world are achieved through a distributed log data acquisition technology, server information, process running information and process logs are acquired in real time through a data grabbing tool embedded in the CDN servers, data are reported to a rear-end server in a unified mode through internal and external network domain names, then log data are distributed to a message queue according to data types, and finally real-time processing of the data is conducted by an alarming middle platform, data statistics is achieved based on a statistics engine, and therefore monitoring of the service stability of the CDN servers around the world and resource monitoring of the servers around the world are achieved.
As shown in fig. 3, a log data collection management system 200 according to an embodiment of the present invention includes: a receiving module 210, an assigning module 220, and a transmitting module 230;
the receiving module 210 is configured to: acquiring log data acquired by each front-end server;
the assignment module 220 is configured to: receiving a data processing configuration type configured for each log data by the data processing dynamic configuration management platform; each data processing configuration type corresponds to one processing system;
the sending module 230 is configured to: and on the basis of the acquisition time sequence, all the log data are sequentially sent to a target message queue, so that the target message queue sends each log data to a corresponding processing system for processing according to the acquisition time sequence.
Preferably, the receiving module 210 is specifically configured to:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
Preferably, the load balancer is: HTTP/TCP load balancer.
According to the technical scheme, the log data is classified and processed and persisted according to the classes by constructing the data processing configuration type, the problem of coupling between front and rear end nodes is solved, and the data collection capacity of the message queue is improved. The technical scheme of the embodiment adopts a data management mode, so that the data access cost is greatly reduced, and the operation and maintenance manpower is reduced.
The above steps for realizing the corresponding functions of each parameter and each module in the log data acquisition management system 200 of the present embodiment may refer to each parameter and step in the above embodiments of a log data acquisition management method, which are not described herein again.
An embodiment of the present invention provides a storage medium, including: the storage medium stores instructions, and when the computer reads the instructions, the computer is caused to execute the steps of the log data acquisition management method, which may specifically refer to each parameter and step in the above embodiment of the log data acquisition management method, and details are not described here.
Computer storage media such as: flash disks, portable hard disks, and the like.
An electronic device provided in an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and is characterized in that when the processor executes the computer program, the computer executes steps of a log data acquisition management method, which may specifically refer to each parameter and step in the above embodiment of the log data acquisition management method, and are not described herein again.
As will be appreciated by one skilled in the art, the present invention may be embodied as methods, systems, storage media, and electronic devices.
Thus, the present invention may be embodied in the form of: may be embodied entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.) or in a combination of hardware and software, and may be referred to herein generally as a "circuit," module "or" system. Furthermore, in some embodiments, the invention may also be embodied in the form of a computer program product in one or more computer-readable media having computer-readable program code embodied in the medium. Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. Although embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and not to be construed as limiting the present invention, and that changes, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (10)
1. A log data collection management method is characterized by comprising the following steps:
acquiring log data acquired by each front-end server;
receiving a data processing configuration type configured for each log data by the data processing dynamic configuration management platform; each data processing configuration type corresponds to one processing system;
and on the basis of the acquisition time sequence, all the log data are sequentially sent to a target message queue, so that each log data is sequentially sent to a corresponding processing system for processing through the target message queue.
2. The method for managing log data collection of claim 1, wherein the receiving log data collected by each front-end server comprises:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
3. The log data collection management method of claim 2, wherein the load balancer is: HTTP/TCP load balancer.
4. The log data collection management method according to claim 1, wherein the data processing configuration types are: a data processing type, a data forwarding type, a data approval type or a data monitoring type.
5. The log data collection management method according to any one of claims 1 to 4, wherein the target message queue is: a KAFKA message queue.
6. A log data collection management system, comprising: the system comprises a receiving module, a distribution module and a sending module;
the receiving module is used for: acquiring log data acquired by each front-end server;
the allocation module is configured to: receiving a data processing configuration type configured for each log data by the data processing dynamic configuration management platform; each data processing configuration type corresponds to one processing system;
the sending module is used for: and on the basis of the acquisition time sequence, all the log data are sequentially sent to a target message queue, so that each log data is sequentially sent to a corresponding processing system for processing through the target message queue.
7. The log data collection management system according to claim 6, wherein the receiving module is specifically configured to:
and acquiring all log data through at least one load balancer, wherein each front-end server acquires the IP address of the corresponding load balancer through a public network domain name respectively, and sends the log data to the corresponding load balancer based on the IP address of the corresponding load balancer.
8. The log data collection management system of claim 7, wherein the load balancer is: HTTP/TCP load balancer.
9. A storage medium having instructions stored therein, which when read by a computer, cause the computer to execute a log data collection management method according to any one of claims 1 to 5.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, causes the computer to perform a log data collection management method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210770816.7A CN115269516A (en) | 2022-06-30 | 2022-06-30 | Log data acquisition management method, system, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210770816.7A CN115269516A (en) | 2022-06-30 | 2022-06-30 | Log data acquisition management method, system, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115269516A true CN115269516A (en) | 2022-11-01 |
Family
ID=83764510
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210770816.7A Pending CN115269516A (en) | 2022-06-30 | 2022-06-30 | Log data acquisition management method, system, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115269516A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118245331A (en) * | 2024-05-27 | 2024-06-25 | 荣耀终端有限公司 | Data collection method and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352809A (en) * | 2020-03-06 | 2020-06-30 | 苏州浪潮智能科技有限公司 | A distributed alarm method, system and computer-readable storage medium |
| CN111522786A (en) * | 2020-04-21 | 2020-08-11 | 中国建设银行股份有限公司 | Log processing system and method |
| CN113497723A (en) * | 2020-03-20 | 2021-10-12 | 阿里巴巴集团控股有限公司 | Log processing method, log gateway and log processing system |
| CN114363042A (en) * | 2021-12-30 | 2022-04-15 | 爱集微咨询(厦门)有限公司 | Log analysis method, device, equipment and readable storage medium |
-
2022
- 2022-06-30 CN CN202210770816.7A patent/CN115269516A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352809A (en) * | 2020-03-06 | 2020-06-30 | 苏州浪潮智能科技有限公司 | A distributed alarm method, system and computer-readable storage medium |
| CN113497723A (en) * | 2020-03-20 | 2021-10-12 | 阿里巴巴集团控股有限公司 | Log processing method, log gateway and log processing system |
| CN111522786A (en) * | 2020-04-21 | 2020-08-11 | 中国建设银行股份有限公司 | Log processing system and method |
| CN114363042A (en) * | 2021-12-30 | 2022-04-15 | 爱集微咨询(厦门)有限公司 | Log analysis method, device, equipment and readable storage medium |
Non-Patent Citations (3)
| Title |
|---|
| 任鹏彦: "基于分布式架构的物流平台设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 张伟等: "《网站开发实训教程 第2版》", 东软电子出版社 * |
| 梅博: "面向电视用户的实时日志采集和处理系统设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118245331A (en) * | 2024-05-27 | 2024-06-25 | 荣耀终端有限公司 | Data collection method and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109714192B (en) | Monitoring method and system for monitoring cloud platform | |
| CN111158983A (en) | Integrated operation and maintenance management system | |
| CN108933798A (en) | Date storage method, storage server and system | |
| CN110928934A (en) | Data processing method and device for business analysis | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN101883009A (en) | Method, system and network management equipment for automatic collection of reliability data | |
| CN111984495A (en) | A big data monitoring method, device and storage medium | |
| CN115269516A (en) | Log data acquisition management method, system, storage medium and electronic equipment | |
| CN109800133A (en) | A kind of method, one-stop monitoring alarm platform and the system of unified monitoring alarm | |
| CN112884176A (en) | Management system and method | |
| CN113760634A (en) | A data processing method and device | |
| CN116431324A (en) | Edge system based on Kafka high concurrency data acquisition and distribution | |
| CN113704203B (en) | A log file processing method and device | |
| CN115391429A (en) | Time sequence data processing method and device based on big data cloud computing | |
| CN111026616A (en) | Information processing method and device, electronic equipment and storage medium | |
| CN112671922B (en) | Industrial internet data processing system and method | |
| CN114490053A (en) | Context awareness strategy recommendation system based on edge calculation and supervised learning method | |
| CN114417796A (en) | Dynamic report statistical method and system based on equipment sampling points | |
| CN110609761B (en) | Method and device for determining fault source, storage medium and electronic equipment | |
| CN105607983B (en) | Data exception monitoring method and device | |
| CN116094905B (en) | Full-link monitoring system | |
| CN113347036B (en) | Method and system for realizing cloud environment bypass monitoring by utilizing public cloud storage | |
| CN118113569A (en) | Log analysis method, device, equipment and storage medium of edge equipment | |
| CN112882992A (en) | Method and apparatus for displaying information | |
| CN115865680B (en) | Method, system and device for accessing, controlling and transmitting data of distributed equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221101 |
|
| RJ01 | Rejection of invention patent application after publication |