CN111740860B - Log data transmission link monitoring method and device - Google Patents
Log data transmission link monitoring method and device Download PDFInfo
- Publication number
- CN111740860B CN111740860B CN202010540717.0A CN202010540717A CN111740860B CN 111740860 B CN111740860 B CN 111740860B CN 202010540717 A CN202010540717 A CN 202010540717A CN 111740860 B CN111740860 B CN 111740860B
- Authority
- CN
- China
- Prior art keywords
- link
- log data
- node
- state information
- data transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 145
- 238000012544 monitoring process Methods 0.000 title claims abstract description 93
- 238000000034 method Methods 0.000 title claims abstract description 44
- 230000005856 abnormality Effects 0.000 claims abstract description 48
- 230000002159 abnormal effect Effects 0.000 claims abstract description 34
- 238000003860 storage Methods 0.000 claims description 55
- 238000004590 computer program Methods 0.000 claims description 16
- 230000010365 information processing Effects 0.000 claims description 2
- 238000004445 quantitative analysis Methods 0.000 abstract description 9
- 238000004891 communication Methods 0.000 description 16
- 238000009826 distribution Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 13
- 238000012806 monitoring device Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 10
- 239000000872 buffer Substances 0.000 description 7
- 238000012423 maintenance Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011002 quantification Methods 0.000 description 2
- 238000013139 quantization Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012882 sequential analysis Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000009747 swallowing Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0677—Localisation of faults
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
- H04L43/0829—Packet loss
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0888—Throughput
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application provides a log data transmission link monitoring method and device, wherein the method comprises the following steps: acquiring link state information of log data in a transmission link, wherein the link state information is obtained according to the current self state when each link node in the transmission link transmits the log data; determining a link node which accords with a preset link abnormality judgment condition according to the link state information; according to the method and the device, the whole link of log data in the distributed system can be effectively monitored, the abnormal node position is accurately positioned through quantitative analysis of the monitored data, and the monitoring efficiency and the accuracy of the log data in the distributed system are improved.
Description
Technical Field
The application relates to the technical field of computers, in particular to a log data transmission link monitoring method and device.
Background
Along with the continuous maturity and large-scale popularization and use of cloud platform technology, in order to realize the high-efficient utilization of log and the centralized unified collection storage of information mining become unique selection. In order to solve the problem of flow and storage pressure caused by centralized storage, a distributed log cache cluster, a distributed log distribution group and a distributed log storage cluster are introduced. And carrying out load balancing on the log traffic through the cache cluster, and realizing the load balancing of storage through the distribution group and the storage cluster.
Although the problems of log load and flow are solved by adding the buffer layer and the distribution layer, the log transmission link is obviously prolonged, and meanwhile, due to the design of a distributed architecture, certain monitoring work is only carried out on a transmitting end and a storage end, and the details of the receiving and transmitting state in the log transmission link cannot be effectively obtained. When log receiving and dispatching are abnormal, the specific problems are complicated to locate, operation and maintenance personnel are required to sequentially analyze and examine the receiving and dispatching links, quantification cannot be performed, the specific problems of the nodes in the links cannot be analyzed, and the problem efficiency is low and the operation and maintenance pressure is increased.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a log data transmission link monitoring method and device, which can effectively monitor the whole link of log data when the log data is transmitted in a distributed system, accurately position abnormal node positions through quantitative analysis of monitoring data, and improve the monitoring efficiency and accuracy of the log data in the distributed system.
In order to solve at least one of the above problems, the present application provides the following technical solutions:
in a first aspect, the present application provides a log data transmission link monitoring method, including:
Acquiring link state information of log data in a transmission link, wherein the link state information is obtained according to the current self state when each link node in the transmission link transmits the log data;
and determining the link node which accords with the preset link abnormality judgment condition according to the link state information.
Further, each link node obtains link state information according to the current self state when transmitting the log data, including:
the current link node monitors whether log data sent by the last link node is received or not;
if yes, the current link node determines link state information according to the current time information and the node identification information, and transmits the log data to the corresponding next link node.
Further, the determining, according to the link state information, the link node meeting the preset link abnormality determination condition includes:
determining log data loss rates of two adjacent link nodes in the transmission link according to the current time information of the link state information and the node identification information;
and if the log data loss rate is larger than a preset loss rate threshold value, judging that the corresponding link node is abnormal.
Further, the determining, according to the link state information, the link node meeting the preset link abnormality determination condition includes:
determining the log data transmission frequency of a link node corresponding to node identification information in the link state information according to the quantity of current time information in the link state information in a set time period;
and if the log data transmission frequency is greater than a preset transmission frequency threshold, judging that the corresponding link node is abnormal.
Further, the determining, according to the link state information, the link node meeting the preset link abnormality determination condition includes:
determining the log data throughput of a link node corresponding to node identification information in the link state information according to the quantity of current time information in the link state information in a set time period;
and if the log data throughput is larger than a preset throughput threshold, judging that the corresponding link node is abnormal.
In a second aspect, the present application provides a log data transmission link monitoring apparatus, including:
the system comprises a link state information acquisition module, a link state information acquisition module and a link state information processing module, wherein the link state information acquisition module is used for acquiring link state information of log data in a transmission link, wherein the link state information is acquired by each link node in the transmission link according to the current self state when the log data is transmitted;
And the link node abnormality judging module is used for determining the link node which accords with the preset link abnormality judging condition according to the link state information.
Further, the link state information acquisition module includes:
the log data receiving monitoring unit is used for monitoring whether log data sent by the last link node is received or not by the current link node;
and the link state information generating unit is used for determining link state information according to the current time information and the node identification information and transmitting the log data to the corresponding next link node if the current link node is monitored to receive the log data transmitted by the previous link node.
Further, the link node anomaly determination module includes:
a log data loss rate determining unit, configured to determine a log data loss rate of two adjacent link nodes in the transmission link according to the current time information of the link state information and the node identification information;
and the first link node abnormality judging unit is used for judging that the corresponding link node is abnormal if the log data loss rate is greater than a preset loss rate threshold value.
Further, the link node anomaly determination module includes:
A log data transmission frequency determining unit, configured to determine a log data transmission frequency of a link node corresponding to node identification information in the link state information according to the number of current time information in the link state information in a set time period;
and the second link node abnormality judging unit is used for judging that the corresponding link node is abnormal if the log data transmission frequency is greater than a preset transmission frequency threshold value.
Further, the link node anomaly determination module includes:
a log data throughput determining unit, configured to determine a log data throughput of a link node corresponding to node identification information in the link state information according to the number of current time information in the link state information in a set time period;
and the third link node abnormality judging unit is used for judging that the corresponding link node is abnormal if the log data throughput is greater than a preset throughput threshold.
In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the log data transmission link monitoring method when executing the program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the log data transmission link monitoring method.
According to the technical scheme, the method and the device for monitoring the log data transmission link are provided, when the log data is transmitted through each link node in the transmission link, the link state information is determined according to the current self state of each link node, whether the log data is abnormal in the transmission process or not and the corresponding link node for transmitting the abnormality are judged according to the link state information and a preset link abnormality judgment condition, therefore, the whole link of the log data in the transmission process of the distributed system is effectively monitored, the abnormal node position is accurately positioned through quantitative analysis of the monitored data, and the monitoring efficiency and the accuracy of the log data in the distributed system are improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a log data transmission link monitoring method in an embodiment of the present application;
FIG. 2 is a second flow chart of a log data transmission link monitoring method according to an embodiment of the present application;
FIG. 3 is a third flow chart of a log data transmission link monitoring method according to an embodiment of the present application;
FIG. 4 is a flowchart of a log data transmission link monitoring method according to an embodiment of the present application;
FIG. 5 is a flowchart of a log data transmission link monitoring method according to an embodiment of the present application;
FIG. 6 is one of the block diagrams of the log data transmission link monitoring apparatus in the embodiment of the present application;
FIG. 7 is a second block diagram of a log data transmission link monitoring device according to an embodiment of the present application;
FIG. 8 is a third block diagram of a log data transmission link monitoring device in an embodiment of the present application;
FIG. 9 is a fourth block diagram of a log data transmission link monitoring device in an embodiment of the present application;
FIG. 10 is a fifth block diagram of a log data transmission link monitoring device in an embodiment of the present application;
FIG. 11 is a flow chart illustrating log data transmission in a full link according to one embodiment of the present application;
fig. 12 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
In consideration of the fact that certain log data monitoring work is only carried out on a transmitting end and a storage end in the existing distributed architecture design, details of receiving and transmitting states in a log transmitting link cannot be effectively obtained, when log receiving and transmitting abnormality exists, specific problems are complex in positioning, operation and maintenance personnel are required to carry out sequential analysis and investigation of the receiving and transmitting links, quantification cannot be carried out, the specific problems of nodes in the links cannot be analyzed, abnormal positioning efficiency is low, operation and maintenance pressure is increased.
In order to effectively monitor all links when log data is transmitted in a distributed system, accurately position abnormal nodes through quantitative analysis of monitoring data, and improve monitoring efficiency and accuracy of the log data in the distributed system, the application provides an embodiment of a log data transmission link monitoring method, referring to fig. 1, wherein the log data transmission link monitoring method specifically comprises the following steps:
step S101: and acquiring link state information of the log data in a transmission link, wherein the link state information is obtained according to the current state of each link node in the transmission link when the log data is transmitted.
Optionally, when the log data is transmitted in the transmission link, there are more link nodes and longer transmission links, for example, a distributed log data transmission system includes at least: the system comprises a log acquisition layer, a log cache layer, a log distribution layer and a log storage layer; according to the method and the system, when each link node in the whole transmission link transmits the log data (for example, the log data can be received or sent), the corresponding link state information can be obtained according to the current self state, so that the actual state of the link node when the log data passes through the link node is quantitatively represented.
It may be understood that the generation of the link state information may be that each link node generates a part separately and stores the link state information and the part locally, or the starting node (for example, the log acquisition layer) generates a part first, and the subsequent nodes (for example, the log cache layer, the log distribution layer and the log storage layer) add the link state information of each link respectively on this basis, where the link state information may be stored locally in each link node respectively, or may be stored in a separate database uniformly.
Optionally, the link state information includes, but is not limited to: the current time information and the node identification information may be associated with the link node's own state, for example, the usage rate, the computing power, and the like.
Step S102: and determining the link node which accords with the preset link abnormality judgment condition according to the link state information.
Optionally, the preset link abnormality determination condition may be a rule condition for determining whether each link node transmitting log data is in an abnormal state from multiple dimensions (such as a loss rate, a transmission frequency, and an swallowing amount), and in other embodiments of the present application, the preset link abnormality determination condition may also be other rule conditions capable of performing a quantization analysis on the link nodes according to link state information.
It can be understood that by performing quantization analysis on link state information of each link node in the whole transmission link, a specific link node with an abnormality can be accurately located, thereby improving monitoring efficiency and accuracy of log data in a system (particularly, a distributed log transmission system).
As can be seen from the above description, the method for monitoring a log data transmission link according to the embodiments of the present application can determine link state information according to the current self state of each link node when log data is transmitted by each link node in a transmission link, and determine whether there is an abnormality in the log data transmission process and a corresponding link node transmitting the abnormality according to the link state information and a preset link abnormality determination condition, thereby effectively monitoring the whole link when the log data is transmitted in the distributed system, accurately positioning the position of the abnormal node through quantitative analysis of monitoring data, and improving the monitoring efficiency and accuracy of the log data in the distributed system.
In order to enable each link node in the transmission link to effectively monitor the transmission of log data, in an embodiment of the log data transmission link monitoring method of the present application, referring to fig. 2, each link node obtains link state information according to a current self state when transmitting the log data, and specifically may include the following steps:
Step S201: the current link node monitors whether log data sent by the previous link node is received.
Step S202: if yes, the current link node determines link state information according to the current time information and the node identification information, and transmits the log data to the corresponding next link node.
Optionally, when each link node in the whole transmission link transmits the log data (for example, the log data sent by the last link node may be received, or the log data may be sent to the next link node), a corresponding link state information can be obtained according to the current self state, so as to quantitatively characterize the actual state of the link node when the log data passes through the link node.
Optionally, the link state information includes, but is not limited to: the current time information and the node identification information may be associated with the link node's own state, for example, the usage rate, the computing power, and the like.
In a specific embodiment of the present application, taking a log collection layer as an example, the link state information may be a field SpandID, spandID =16-bit random number+timestamp that is formed by a random number and a current timestamp, and may further include node identification information of the log collection layer, for example, span id=16-bit random number+timestamp+node application name hostname.
In another specific embodiment of the present application, taking a log storage layer as an example, the link state information storage id may be storage id=span id+storage cluster name.
In order to accurately monitor the loss rate in the log data transmission process, in an embodiment of the log data transmission link monitoring method of the present application, referring to fig. 3, the step S102 may further specifically include the following:
step S301: and determining the log data loss rate of two adjacent link nodes in the transmission link according to the current time information of the link state information and the node identification information.
Step S302: and if the log data loss rate is larger than a preset loss rate threshold value, judging that the corresponding link node is abnormal.
Alternatively, according to the current time information in the link state information of each of the two adjacent link nodes, the sending number and the receiving number of the log data in a certain same time period can be determined, so as to determine the loss rate of the log data.
It will be appreciated that a piece of log data corresponds to a piece of current time information (e.g., a time stamp), how much current time information there is how much log data.
It will be appreciated that two adjacent link nodes may be determined by the node identification information.
In order to accurately monitor the transmission frequency of each link node in the log data transmission process, in an embodiment of the log data transmission link monitoring method of the present application, referring to fig. 4, the step S102 may further specifically include the following:
step S401: and determining the log data transmission frequency of the link node corresponding to the node identification information in the link state information according to the quantity of the current time information in the link state information in a set time period.
Step S402: and if the log data transmission frequency is greater than a preset transmission frequency threshold, judging that the corresponding link node is abnormal.
It will be appreciated that a piece of log data corresponds to a piece of current time information (e.g., a time stamp), how much current time information there is how much log data.
Optionally, according to the current time information (i.e. the number of log data) of a certain link node in a set time period (for example, 1 minute), the log data transmission frequency of the link node in the time period can be determined, for example, 100 log data are received per minute, 80 log data are sent per minute, so that the data transmission pressure of the link node can be known, and if the log transmission frequency is greater than a preset transmission frequency threshold (for example, 80 log data are received per minute), it can be determined that the link node is overloaded, and corresponding load balancing processing is needed.
In order to accurately monitor the throughput of each link node in the log data transmission process, in an embodiment of the log data transmission link monitoring method of the present application, referring to fig. 5, the step S102 may further specifically include the following:
step S501: and determining the log data throughput of the link node corresponding to the node identification information in the link state information according to the quantity of the current time information in the link state information in a set time period.
Step S502: and if the log data throughput is larger than a preset throughput threshold, judging that the corresponding link node is abnormal.
It will be appreciated that a piece of log data corresponds to a piece of current time information (e.g., a time stamp), how much current time information there is how much log data.
Optionally, according to the current time information (i.e. the log data) of a certain link node in a set time period (for example, 1 day), the log data throughput of the link node in the time period may be determined, taking a log buffer layer and a log storage layer as an example, for example, the log buffer layer and the log storage layer receive and store 1.2GB of log data in total on the same day, so that the data storage pressure of the link node can be known, and if the log throughput is greater than a preset throughput threshold (for example, 1.5GB is received and stored daily), it may be determined that the link node has solved the storage saturation, and a new node needs to be added for data throughput.
In order to effectively monitor all links when log data is transmitted in a distributed system, accurately locate abnormal node positions through quantitative analysis of monitoring data, and improve monitoring efficiency and accuracy of the log data in the distributed system, the application provides an embodiment of a log data transmission link monitoring device for implementing all or part of the content of the log data transmission link monitoring method, referring to fig. 6, where the log data transmission link monitoring device specifically includes the following contents:
the link state information obtaining module 10 is configured to obtain link state information of log data in a transmission link, where the link state information is obtained according to a current state of each link node in the transmission link when the log data is transmitted.
And the link node abnormality determination module 20 is configured to determine, according to the link state information, a link node that meets a preset link abnormality determination condition.
As can be seen from the above description, the log data transmission link monitoring device provided in the embodiments of the present application can determine, by using the current self state of each link node when log data is transmitted in each link node in a transmission link, link state information, and determine, according to the link state information and a preset link abnormality determination condition, whether there is an abnormality in the log data in the transmission process and a link node corresponding to the abnormality in transmission, thereby effectively monitoring all links when the log data is transmitted in a distributed system, accurately locating the position of the abnormal node by quantitatively analyzing monitoring data, and improving the monitoring efficiency and accuracy of the log data in the distributed system.
In order to enable each link node in the transmission link to effectively monitor the transmission of log data, in an embodiment of the log data transmission link monitoring apparatus of the present application, referring to fig. 7, the link state information obtaining module 10 includes:
the log data receiving monitoring unit 11 is configured to monitor whether log data sent by a previous link node is received by a current link node.
The link state information generating unit 12 is configured to determine link state information according to current time information and node identification information if it is monitored that the current link node receives log data sent by a previous link node, and transmit the log data to a corresponding next link node.
In order to accurately monitor the loss rate during the log data transmission, in an embodiment of the log data transmission link monitoring apparatus of the present application, referring to fig. 8, the link node anomaly determination module 20 includes:
a log data loss rate determining unit 21, configured to determine log data loss rates of two adjacent link nodes in the transmission link according to the current time information of the link state information and node identification information.
The first link node abnormality determination unit 22 is configured to determine that the corresponding link node is abnormal if the log data loss rate is greater than a preset loss rate threshold.
In order to accurately monitor the transmission frequency of each link node in the log data transmission process, in an embodiment of the log data transmission link monitoring device of the present application, referring to fig. 9, the link node anomaly determination module 20 includes:
a log data transmission frequency determining unit 23, configured to determine a log data transmission frequency of a link node corresponding to node identification information in the link state information according to the number of current time information in the link state information in a set time period.
And the second link node abnormality determination unit 24 is configured to determine that the corresponding link node is abnormal if the log data transmission frequency is greater than a preset transmission frequency threshold.
In order to accurately monitor the throughput of each link node in the log data transmission process, in an embodiment of the log data transmission link monitoring apparatus of the present application, referring to fig. 10, the link node anomaly determination module 20 includes:
a log data throughput determining unit 25, configured to determine a log data throughput of a link node corresponding to node identification information in the link state information according to the number of current time information in the link state information in a set time period.
And a third link node abnormality determination unit 26, configured to determine that the corresponding link node is abnormal if the log data throughput is greater than a preset throughput threshold.
In order to further explain the scheme, the present application further provides a specific application example of implementing the log data transmission link monitoring method by using the log data transmission link monitoring device, referring to fig. 11, which specifically includes the following contents: the system comprises a log acquisition layer, a log cache layer, a log distribution layer, a log storage layer and a link monitoring layer.
Specifically, the log collection layer is composed of a log collection group, each client represents each client for carrying out differentiated transmission on the log in an application dimension, and differentiation processing can be carried out on key applications and conventional applications. In fig. 11, the log collection groups are respectively arranged in each application node, and meanwhile, the logs are sent to the log cache layer for data cache in a load balancing manner.
Specifically, the log cache set layer corresponds to a log cache cluster, and is used for carrying out centralized cache on the logs sent by each log cache set group, so as to avoid data loss caused by network jitter or other sporadic conditions.
Specifically, the log distribution layer is composed of log distribution groups and is used for carrying out appointed forwarding on log data in the log cache layer and distributing the data to corresponding log storage clusters.
Specifically, the log storage layer corresponds to a log storage cluster, the log storage cluster can be built on the basis of an elastic service for distributed log storage cluster, and meanwhile, a plurality of storage clusters can be built for data backup and storage.
Specifically, the link monitoring is used for monitoring all link activities of log acquisition, caching and distribution to storage. For the log collection group, the log cache group, the log distribution group and the log storage group, different monitoring scheme strategies can be formulated for fine management, and each monitoring scheme strategy can be specifically as follows:
1, aiming at a log acquisition layer, files need to be processed and sent, so that node resource use, network state and node state need to be focused, and therefore, a monitoring strategy is as follows:
1) The node operates the resource to monitor, monitor the node resources such as CPU, memory and storage of the collection node in real time, when the storage exceeds the threshold value, produce the corresponding alarm, and send the resource information up at the same time. For performance evaluation.
2) Node connection monitoring: because data interaction with the cache layer is needed in real time, the network with the cache layer is subjected to activity detection once every 5min, and when three times of activity detection fail, an alarm is given. Indicating that the network anomaly caused a log transmission anomaly.
3) Node state monitoring: and alarming the state of the up-going process node, and alarming when the node is abnormal in transmission caused by configuration.
2, aiming at the log cache layer, all collection group node data needs to be loaded, so that the monitoring strategy which needs to pay attention to network load and node performance is as follows:
1) Request load monitoring: and (3) carrying out a multi-stage early warning mechanism on the connection condition of the cluster, wherein the cluster is in a health state when the connection is less than a set threshold value 1, and is in an available state when the connection exceeds the threshold value 1 but is less than the threshold value 2, and is in an early warning state when the connection exceeds the threshold value 2 but is less than the threshold value 3, sending early warning information to related personnel to pay attention to the cluster state, and when the connection exceeds the threshold value 4, temporarily preparing for adding and preparing and simultaneously sending related alarm information to processing personnel to carry out information confirmation and emergency execution.
2) Node operation resource monitoring: the memory and the use condition of the memory resources are monitored by the memory fragments.
3) Node state monitoring: the caching layer is composed of a plurality of load nodes of a designated channel, the states of the load nodes in the channel are monitored, and early warning is carried out to promote attention of operation and maintenance personnel when abnormality occurs to part of the load nodes. And when the load nodes in the channel are in abnormal states, alarming, and recovering the abnormal states as soon as possible.
3, aiming at a log distribution layer, the log distribution layer distributes data to a designated storage cluster from a centralized cache group according to a designated rule, wherein a specific monitoring strategy is as follows: .
1) And (3) resource monitoring: similar to acquisition monitoring.
2) The network monitoring is the same as the acquisition monitoring, and the inquiry of the storage condition of the storage cluster is increased.
3) And performing node exception processing, namely performing storage distribution according to the storage state of the storage cluster, and performing forwarding switching when the storage state of the storage cluster exceeds a threshold value.
And 4, storing an actual log aiming at a log storage layer, storing according to an application dimension, and providing a storage state co-distributor query, wherein a monitoring strategy is as follows:
1) Node storage monitoring: the monitoring of the memory and the storage is performed on each node of the storage cluster, and the real-time requirement of the storage state is high. To provide distribution node queries.
2) Cluster state monitoring: whether the cluster is monitored, whether abnormal application fragments exist or not, and the like.
3) And monitoring the link integrity, namely counting and registering link information, storing the state of a link field, and analyzing the receiving and transmitting proportion of each link node.
Optionally, sampling detection statistics is performed on the data every minute, 10000 records are selected every minute to sample, specified logs in the logs of each node are queried, and query results are returned, so that the specific sending and losing proportion of each node in the sending link can be obtained to score. And early warning analysis is carried out on the data, the monitoring granularity of the node with the lower link score is increased, and real-time feedback is carried out on the data so as to be convenient for operation and maintenance personnel to analyze.
In order to effectively monitor all links of log data when the log data is transmitted in a distributed system in terms of hardware, accurately position abnormal node positions through quantitative analysis of the monitored data, and improve monitoring efficiency and accuracy of the log data in the distributed system, the application provides an embodiment of electronic equipment for realizing all or part of contents in a monitoring method of the log data transmission link, wherein the electronic equipment specifically comprises the following contents:
a processor (processor), a memory (memory), a communication interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete communication with each other through the bus; the communication interface is used for realizing information transmission between the log data transmission link monitoring device and related equipment such as a core service system, a user terminal, a related database and the like; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, etc., and the embodiment is not limited thereto. In this embodiment, the logic controller may refer to an embodiment of the log data transmission link monitoring method and an embodiment of the log data transmission link monitoring device in the embodiment, and the contents thereof are incorporated herein, and are not repeated here.
It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Wherein, intelligent wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
In practical applications, part of the log data transmission link monitoring method may be performed on the electronic device side as described above, or all operations may be performed in the client device. Specifically, the selection may be made according to the processing capability of the client device, and restrictions of the use scenario of the user. The present application is not limited in this regard. If all operations are performed in the client device, the client device may further include a processor.
The client device may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.
Fig. 12 is a schematic block diagram of a system configuration of an electronic device 9600 of an embodiment of the present application. As shown in fig. 12, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 12 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.
In one embodiment, log data transmission link monitoring method functionality may be integrated into the central processor 9100. The central processor 9100 may be configured to perform the following control:
step S101: and acquiring link state information of the log data in a transmission link, wherein the link state information is obtained according to the current state of each link node in the transmission link when the log data is transmitted.
Step S102: and determining the link node which accords with the preset link abnormality judgment condition according to the link state information.
As can be seen from the above description, when the electronic device provided in the embodiment of the present application transmits the log data through each link node in the transmission link, the link state information is determined according to the current self state of each link node, and according to the link state information and a preset link abnormality determination condition, whether the log data has an abnormality in the transmission process and a link node corresponding to the abnormality is determined, so that the whole link of the log data when the log data is transmitted in the distributed system is effectively monitored, the abnormal node position is accurately located through quantitative analysis of the monitored data, and the monitoring efficiency and accuracy of the log data in the distributed system are improved.
In another embodiment, the log data transmission link monitoring apparatus may be configured separately from the central processor 9100, for example, the log data transmission link monitoring apparatus may be configured as a chip connected to the central processor 9100, and the log data transmission link monitoring method function is implemented by control of the central processor.
As shown in fig. 12, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 need not include all of the components shown in fig. 12; in addition, the electronic device 9600 may further include components not shown in fig. 12, and reference may be made to the related art.
As shown in fig. 12, the central processor 9100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 9100 receives inputs and controls the operation of the various components of the electronic device 9600.
The memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 9100 can execute the program stored in the memory 9140 to realize information storage or processing, and the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.
The memory 9140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, etc. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 storing application programs and function programs or a flow for executing operations of the electronic device 9600 by the central processor 9100.
The memory 9140 may also include a data store 9143, the data store 9143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. A communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of conventional mobile communication terminals.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132 to implement usual telecommunications functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100 so that sound can be recorded locally through the microphone 9132 and sound stored locally can be played through the speaker 9131.
The embodiments of the present application further provide a computer readable storage medium capable of implementing all the steps in the log data transmission link monitoring method in which the execution subject is a server or a client in the above embodiments, where the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all the steps in the log data transmission link monitoring method in which the execution subject is a server or a client in the above embodiments, for example, the processor implements the following steps when executing the computer program:
Step S101: and acquiring link state information of the log data in a transmission link, wherein the link state information is obtained according to the current state of each link node in the transmission link when the log data is transmitted.
Step S102: and determining the link node which accords with the preset link abnormality judgment condition according to the link state information.
As can be seen from the above description, when the log data is transmitted through each link node in the transmission link, the computer readable storage medium provided in the embodiment of the present application determines link state information according to the current self state of each link node, and determines whether the log data has an abnormality in the transmission process and sends the corresponding link node of the abnormality according to the link state information and a preset link abnormality determination condition, thereby effectively monitoring the whole link when the log data is transmitted in the distributed system, accurately positioning the abnormal node position through quantitative analysis of the monitored data, and improving the monitoring efficiency and accuracy of the log data in the distributed system.
It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (8)
1. A method for monitoring a log data transmission link, the method comprising:
acquiring link state information of log data in a transmission link, wherein the link state information is determined by each link node in the transmission link according to current time information and node identification information when monitoring and receiving the log data sent by a previous link node;
And determining a link node which accords with a preset link abnormality judgment condition according to the link state information, wherein the link state information comprises current time information and node identification information, determining log data loss rates of two adjacent link nodes in the transmission link according to the current time information and the node identification information, and determining whether the log data loss rate is larger than a preset loss rate threshold value or not according to the link abnormality judgment condition.
2. The method for monitoring a log data transmission link according to claim 1, wherein determining a link node that meets a preset link abnormality determination condition according to the link state information comprises:
determining the log data transmission frequency of a link node corresponding to node identification information in the link state information according to the quantity of current time information in the link state information in a set time period;
and if the log data transmission frequency is greater than a preset transmission frequency threshold, judging that the corresponding link node is abnormal.
3. The method for monitoring a log data transmission link according to claim 1, wherein determining a link node that meets a preset link abnormality determination condition according to the link state information comprises:
Determining the log data throughput of a link node corresponding to node identification information in the link state information according to the quantity of current time information in the link state information in a set time period;
and if the log data throughput is larger than a preset throughput threshold, judging that the corresponding link node is abnormal.
4. A log data transmission link monitoring apparatus, comprising:
the system comprises a link state information acquisition module, a link state information acquisition module and a link state information processing module, wherein the link state information acquisition module is used for acquiring link state information of log data in a transmission link, and the link state information is determined according to current time information and node identification information when each link node in the transmission link monitors and receives the log data sent by a previous link node;
the link node abnormality judging module is used for determining a link node meeting preset link abnormality judging conditions according to the link state information, wherein the link state information comprises current time information and node identification information, the log data loss rate of two adjacent link nodes in the transmission link is determined according to the current time information and the node identification information, and the link abnormality judging conditions comprise whether the log data loss rate is larger than a preset loss rate threshold value or not.
5. The log data transmission link monitoring apparatus as set forth in claim 4, wherein the link node abnormality determination module includes:
a log data transmission frequency determining unit, configured to determine a log data transmission frequency of a link node corresponding to node identification information in the link state information according to the number of current time information in the link state information in a set time period;
and the second link node abnormality judging unit is used for judging that the corresponding link node is abnormal if the log data transmission frequency is greater than a preset transmission frequency threshold value.
6. The log data transmission link monitoring apparatus as set forth in claim 4, wherein the link node abnormality determination module includes:
a log data throughput determining unit, configured to determine a log data throughput of a link node corresponding to node identification information in the link state information according to the number of current time information in the link state information in a set time period;
and the third link node abnormality judging unit is used for judging that the corresponding link node is abnormal if the log data throughput is greater than a preset throughput threshold.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the log data transmission link monitoring method of any one of claims 1 to 3 when the program is executed by the processor.
8. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor implements the steps of the log data transmission link monitoring method of any of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010540717.0A CN111740860B (en) | 2020-06-15 | 2020-06-15 | Log data transmission link monitoring method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010540717.0A CN111740860B (en) | 2020-06-15 | 2020-06-15 | Log data transmission link monitoring method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111740860A CN111740860A (en) | 2020-10-02 |
| CN111740860B true CN111740860B (en) | 2023-06-09 |
Family
ID=72649201
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010540717.0A Active CN111740860B (en) | 2020-06-15 | 2020-06-15 | Log data transmission link monitoring method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111740860B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463561B (en) * | 2020-11-20 | 2024-06-18 | 中国建设银行股份有限公司 | Fault positioning method, device, equipment and storage medium |
| CN113507418B (en) * | 2021-06-25 | 2022-08-19 | 中标慧安信息技术股份有限公司 | Internet of things platform communication link data transmission monitoring method |
| CN113537824B (en) * | 2021-08-03 | 2024-06-28 | 南方电网数字电网研究院有限公司 | Full-link monitoring system for power grid data |
| CN114285786B (en) * | 2021-12-24 | 2023-12-08 | 中国农业银行股份有限公司 | Construction method and device of network link library |
| CN114401207B (en) * | 2021-12-30 | 2024-03-15 | 北京首钢自动化信息技术有限公司 | Communication abnormal terminal equipment positioning method and device and electronic equipment |
| CN114760218B (en) * | 2022-04-24 | 2023-12-22 | 中国工商银行股份有限公司 | Link sampling method and related device thereof |
| CN115037645B (en) * | 2022-05-17 | 2023-03-24 | 中国兵器工业信息中心 | Full link monitoring and early warning system based on big data analysis |
| CN117808570B (en) * | 2024-02-29 | 2024-05-07 | 梅州客商银行股份有限公司 | Method and device for monitoring all links of bank distributed system and electronic equipment |
| CN118999691B (en) * | 2024-10-22 | 2024-12-27 | 深圳正实自动化设备有限公司 | Real-time status monitoring and analysis method, system and electronic equipment for solder paste printer |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109922466A (en) * | 2019-03-25 | 2019-06-21 | 深圳市联智物联网科技有限公司 | A method of realizing directional data access in multinode wireless transmitting system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101404613B (en) * | 2008-11-20 | 2011-04-20 | 华为技术有限公司 | Data protection method and device |
| CN104378223A (en) * | 2013-08-16 | 2015-02-25 | 中兴通讯股份有限公司 | Link performance testing method and device, logic processor and network processor |
| CN109412966B (en) * | 2018-12-12 | 2022-08-23 | 北京信息科技大学 | Large-scale log transmission method, device and system |
| CN110995468B (en) * | 2019-11-13 | 2022-07-26 | 上海钧正网络科技有限公司 | System fault processing method, device, equipment and storage medium of system to be analyzed |
| CN110932918B (en) * | 2019-12-26 | 2023-01-10 | 远景智能国际私人投资有限公司 | Log data acquisition method and device and storage medium |
-
2020
- 2020-06-15 CN CN202010540717.0A patent/CN111740860B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109922466A (en) * | 2019-03-25 | 2019-06-21 | 深圳市联智物联网科技有限公司 | A method of realizing directional data access in multinode wireless transmitting system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111740860A (en) | 2020-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111740860B (en) | Log data transmission link monitoring method and device | |
| CN110428325A (en) | Transaction tracking and device | |
| CN107645562A (en) | Data transmission processing method, device, equipment and system | |
| CN112540996B (en) | Service data verification method and device, electronic equipment and storage medium | |
| AU2003292527A1 (en) | Quality of service monitoring architecture, related method, network and computer program product | |
| CN111352806A (en) | Log data monitoring method and device | |
| CN106101264B (en) | Content distributing network log method for pushing, device and system | |
| CN111586126A (en) | Method, device and equipment for pre-downloading small program and storage medium | |
| CN109947821A (en) | Generation method, display methods, device, equipment and the storage medium of report information | |
| CN109660421A (en) | Method, apparatus, server and the storage medium of flexible scheduling resource | |
| CN110210980A (en) | A kind of driving behavior appraisal procedure, device and storage medium | |
| CN110275815A (en) | A system abnormal alarm processing method and device | |
| CN116257427A (en) | Heterogeneous test method, heterogeneous test system, heterogeneous test equipment and heterogeneous test storage medium for federal learning task | |
| CN111737297B (en) | Method and device for processing link aggregation call information | |
| CN106940710B (en) | Information pushing method and device | |
| CN111858525B (en) | Log tracking method, generation method, tracking device, generation device and system | |
| CN115080363B (en) | System capacity evaluation method and device based on service log | |
| CN110245120B (en) | Stream type computing system and log data processing method thereof | |
| CN104615702B (en) | Information pushing method and device | |
| CN110647070A (en) | Power environment monitoring system for super-large-scale data center | |
| CN116303072A (en) | Full-link performance test method and device | |
| CN108345650A (en) | Electronic invoice business monitors method for early warning and system in real time | |
| CN115664992A (en) | Network operation data processing method and device, electronic equipment and medium | |
| CN112766698B (en) | Application service pressure determining method and device | |
| CN112836770B (en) | KPI (kernel performance indicator) anomaly positioning analysis method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |