CN110362455B - Data processing method and data processing device - Google Patents
Data processing method and data processing device Download PDFInfo
- Publication number
- CN110362455B CN110362455B CN201910637057.5A CN201910637057A CN110362455B CN 110362455 B CN110362455 B CN 110362455B CN 201910637057 A CN201910637057 A CN 201910637057A CN 110362455 B CN110362455 B CN 110362455B
- Authority
- CN
- China
- Prior art keywords
- alarm
- data
- metadata
- monitored
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 82
- 238000003672 processing method Methods 0.000 title abstract description 9
- 238000012544 monitoring process Methods 0.000 claims abstract description 58
- 238000000034 method Methods 0.000 claims abstract description 27
- 230000015654 memory Effects 0.000 claims description 21
- 230000010365 information processing Effects 0.000 claims description 14
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000012795 verification Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Alarm Systems (AREA)
Abstract
The embodiment of the application provides a data processing method and a data processing device, which are used for monitoring data in real time, improving the timeliness of data monitoring alarm and the universality of a data monitoring system. The method in the embodiment of the application comprises the following steps: registering metadata for a data source to be monitored, wherein the metadata is used for describing information of the data source to be monitored; defining an alarm rule of the data source to be monitored according to the attribute of the metadata; monitoring the data source to be monitored; and sending alarm information when the data source to be monitored triggers the alarm rule.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a data processing method and a data processing apparatus.
Background
With the development of internet technology, big data is more and more widely applied. In the data processing process of big data, data needs to be collected firstly, and if the data of a data provider is changed suddenly, unexpected errors can occur in subsequent processing procedures. Therefore, in the big data processing, a user needs to determine whether the data source data is correct or not and whether the data source data changes or not at any time. Based on the requirements, the data provider designs a data monitoring system for monitoring the format specification, the value range and the integrity of the data source; and comparing the trend of the data source data with the historical baseline. And then the data provider feeds back an alarm event to the user in time according to the result of the data monitoring system, so that the user can find the change of the data at the first time.
One of the current data monitoring technologies is a traditional non-real-time timed triggering task; the other is a near real-time monitoring task. Both of these solutions have certain drawbacks. In the technical scheme of non-real-time timed triggering tasks, the timeliness of data monitoring results is poor, and monitoring and alarming cannot be carried out in time; in the technical scheme of the near real-time monitoring task, the customization degree of the monitoring task is high, the monitoring task is generally a monitoring scheme based on a fixed service scene, and the universality is poor.
Disclosure of Invention
The embodiment of the application provides a data processing method and a data processing device, which are used for monitoring data in real time, improving the timeliness of data monitoring alarm and the universality of a data monitoring system.
In a first aspect, an embodiment of the present application provides a data processing method, which specifically includes: the data processing device registers metadata for a data source to be monitored, wherein the metadata is used for describing information of the data source to be monitored; then the data processing device defines the alarm rule of the data source to be monitored according to the attribute of the metadata and translates the defined process into an SQL-like statement of Esper; then the data processing device monitors the data source to be monitored; and when the data source to be monitored triggers the alarm rule, the data processing device sends alarm information to the user terminal.
In the embodiment of the application, the data processing device defines the alarm rule of the data source to be monitored by using the similar SQL statement of Esper according to the metadata, so that not only can the regular verification of a general monitoring system be realized, but also more complex business logic verification rules can be realized, and an extended function is provided for complex business data monitoring, so that the timeliness of data monitoring alarm and the universality of the data monitoring system are improved.
Optionally, the metadata is an Esper event, the Esper event is abstracted to be a map type object, and the metadata is stored in the metadata management layer.
Optionally, the data source to be monitored includes a user account identifier, the metadata password code, and a timestamp, and monitoring the data source to be monitored includes:
storing the metadata and the alarm rule in a first data queue cluster in the form of control information, wherein the first data queue cluster is established based on message queue middleware;
the metadata and the alarm rules are consumed and monitored using an Esper complex event handling mechanism.
Optionally, the sending the alarm information includes:
setting a related alarm channel according to the alarm rule;
and sending the alarm information by utilizing the alarm information processing logic of the alarm channel.
Optionally, the sending the alarm information by using the alarm information processing logic of the alarm channel includes:
storing the alarm information in a second data queue cluster, wherein the second data queue cluster is established based on message queue middleware;
and consuming the alarm information and sending the alarm information by utilizing the alarm information processing logic of the alarm channel.
Optionally, the alarm information processing logic includes:
classifying the alarm information according to duration, and sending the alarm information according to the classification;
and/or the presence of a gas in the gas,
acquiring the occurrence frequency of the alarm information, and merging and sending the alarm information when the occurrence frequency exceeds a first preset threshold;
and/or the presence of a gas in the gas,
and acquiring the duration of the alarm information, and merging and sending the alarm information when the duration exceeds a second preset threshold.
In a second aspect, an embodiment of the present application provides a data processing apparatus, which includes:
in one possible implementation, the data processing apparatus includes:
the processing module is used for registering metadata for the data source to be monitored, and the metadata is used for describing the information of the data source to be monitored; defining an alarm rule of the data source to be monitored according to the attribute of the metadata;
the monitoring module is used for monitoring the data source to be monitored;
and the sending module is used for sending alarm information when the alarm rule is triggered by the data source to be monitored.
Optionally, the metadata is an Esper event, the Esper event is abstracted to be a map type object, and the metadata is stored in the metadata management layer.
Optionally, the data source to be monitored includes a user account identifier, the metadata password code, and a timestamp, and the monitoring module is specifically configured to store the metadata and the alarm rule in a first data queue cluster in the form of control information, where the first data queue cluster is established based on a message queue middleware; the metadata and the alarm rules are consumed and monitored using an Esper complex event handling mechanism.
Optionally, the sending module is specifically configured to set a relevant alarm channel according to the alarm rule; and sending the alarm information by utilizing the alarm information processing logic of the alarm channel.
Optionally, the sending module is specifically configured to store the alarm information in a second data queue cluster, where the second data queue cluster is established based on a message queue middleware; and consuming the alarm information and sending the alarm information by utilizing the alarm information processing logic of the alarm channel.
Optionally, the alarm information processing logic includes:
classifying the alarm information according to duration, and sending the alarm information according to the classification; and/or acquiring the occurrence frequency of the alarm information, and merging and sending the alarm information when the occurrence frequency exceeds a first preset threshold; and/or acquiring the duration of the alarm information, and merging and sending the alarm information when the duration exceeds a second preset threshold.
In another implementation, the data processing apparatus includes: a processor and a memory, wherein the memory has a computer readable program stored therein, and the processor is configured to execute the program in the memory to perform any of the methods described above.
In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing computer instructions for executing the method of any one of the above.
In a fourth aspect, embodiments of the present application provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any one of the above.
According to the technical scheme, the embodiment of the application has the following advantages: the data processing device defines the alarm rule of the data source to be monitored by using the similar SQL statement of the Esper according to the metadata, so that not only can the regular verification of a general monitoring system be realized, but also more complex service logic verification rules can be realized, and an extended function is provided for the complex service data processing, thereby improving the timeliness of data monitoring alarm and the universality of the data monitoring system.
Drawings
FIG. 1 is a block diagram of a data processing system in an embodiment of the present application;
FIG. 2 is a schematic diagram of an embodiment of a data processing method in an embodiment of the present application;
FIG. 3 is a schematic diagram of an embodiment of a data processing apparatus according to the present embodiment;
fig. 4 is a schematic diagram of another embodiment of a data processing apparatus in the embodiment of the present application.
Detailed Description
The embodiment of the application provides a data processing method and a data processing device, which are used for monitoring data in real time, improving the timeliness of data monitoring alarm and the universality of a data monitoring system.
The terms "first," "second," "third," "fourth," and the like in the description and claims of this application and in the above-described drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be implemented in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
With the development of internet technology, the application of big data is more and more extensive. In the data processing process of big data, data needs to be collected firstly, and if the data of a data provider is changed suddenly, unexpected errors can occur in subsequent processing procedures. Therefore, in the big data processing, a user needs to determine whether the data source data is correct or not at any time and whether the data source data is changed or not. Based on the requirements, the data provider designs a data monitoring system for monitoring the format specification, the value range and the integrity of the data source; and comparing the trend of the data source data with the historical baseline. And then the data provider feeds back an alarm event to the user in time according to the result of the data monitoring system, so that the user can find the change of the data at the first time. One of the current data processing technologies is the traditional non-real-time timing triggering task; the other is a near real-time monitoring task. Both of these solutions have certain drawbacks. In the technical scheme of non-real-time timed triggering tasks, the timeliness of data monitoring results is poor, and monitoring and alarming cannot be carried out in time; in the technical scheme of the near real-time monitoring task, the customization degree of the monitoring task is high, the monitoring task is generally a monitoring scheme based on a fixed service scene, and the universality is poor.
In order to solve the problem, the following technical scheme is provided in the embodiment of the application: the data processing device registers metadata for a data source to be monitored, wherein the metadata is used for describing information of the data source to be monitored; then the data processing device defines the alarm rule of the data source to be monitored according to the attribute of the metadata; then the data processing device monitors the data source to be monitored; and when the data source to be monitored triggers the alarm rule, the data processing device sends alarm information.
The embodiment of the application is applied to the data processing system shown in fig. 1, and specifically includes a metadata management layer, a data receiving layer, an alarm engine layer, a data queue cluster and a message channel engine layer. The metadata management layer is used for registering metadata for a data source to be monitored, defining an alarm rule according to the metadata and setting an alarm channel according to the alarm rule. The data receiving layer is used for receiving data of the data source to be monitored, wherein the data of the data source to be monitored carries attributes such as a user account identifier, a corresponding metadata password and a timestamp; the data receiving layer defines the verification data of the data source to be monitored according to the metadata of the data source to be monitored; and adding the data of the data source to be monitored into the data queue cluster. The data queue cluster is established based on message queue middleware and is used for storing data source data to be monitored and alarm event data. The alarm engine layer is used for consuming data of the data source to be monitored and the alarm rule in the data queue cluster and then generating an alarm event. Adding the alarm event data into the data queue cluster, consuming the alarm event data in the data queue cluster by the message channel engine layer, and generating alarm information; and finally, the message channel engine layer sends the alarm information.
Specifically, referring to fig. 2, an embodiment of data processing in the embodiment of the present application includes:
201. the data processing device acquires a data source to be monitored.
The data processing device monitors data to be acquired by a user, namely a data source to be monitored in the embodiment of the application.
202. The data processing device registers metadata for the data source to be monitored.
The data processing device registers metadata for the data source to be monitored. In this embodiment, the data processing apparatus defines an object (i.e., metadata) describing the data source to be monitored, registers an Esper event in the definition process, and abstracts the event into a map type object. Wherein any type of composition and nesting is supported within the map object. Such as metadata of a contract, may be specific to describing which fields are present, what type, respectively, etc.
203. And the data processing device defines the alarm rule of the data source to be monitored according to the attribute of the metadata.
After the data processing device registers metadata, the data processing device defines an alarm rule of the data source to be monitored according to the attribute of the metadata, and translates the defined process into an SQL-like statement of Esper. Therefore, regular verification of a common monitoring system can be realized, and more complex business logic verification rules can be realized.
In this embodiment, after defining the alarm rule, the data processing apparatus further needs to define an alarm channel associated with the alarm rule according to the alarm rule. That is, each alarm rule is associated with one alarm channel, when the change of data triggers an alarm, the logic for processing the alarm message is started according to the corresponding alarm rule and the setting of the alarm channel, the alarm message is graded and sent to the corresponding user terminal according to different frequencies and different modes.
204. The data processing device monitors the data source to be monitored.
And the data processing device monitors the data source to be monitored on the basis of the completion of the establishment of the alarm rule, the alarm channel and the metadata.
In this embodiment, the data processing apparatus receives the data source to be monitored at a data receiving layer. The data source to be monitored needs to have three attributes of account identification, metadata code and timestamp. Wherein the account identification is used to cause the data processing apparatus to distinguish between different data tenants. The metadata code is identification information of the metadata indicating that the current data is based on this metadata. The time stamp is a time identifier for indicating each data in the data source to be monitored.
In this embodiment, the specific manner of monitoring the data source to be monitored by the data processing apparatus may be as follows: the data processing device establishes a data queue cluster, wherein the data queue cluster is built based on Kafka message queue middleware. Meanwhile, because the metadata and the alarm rule are abstracted into control messages, the data processing device takes the control messages as partition strategies according to users and metadata types, and then the data processing device sends the control messages into the data queue cluster in a balanced manner. When the data processing device runs in a cluster multi-machine parallel mode on an alarm engine layer, the data processing device consumes original data and alarm rules of the data source to be monitored in a data queue cluster, and simultaneously adopts an Esper complex event processing function to perform streaming processing according to the alarm rules so as to perform real-time calculation and alarm monitoring. When the mode is adopted, the data processing device can also acquire historical data and the current real-time data to carry out calculation and alarm monitoring. Meanwhile, the data processing device can also monitor the metadata and the control message generated by the alarm rule in an abstract way in real time, so that a new registration event (namely a new data source to be monitored), the change of the verification rule (namely the generated new alarm rule) and the opening and closing of certain event or rule monitoring behaviors can be found in time.
205. And when the data source to be monitored triggers the alarm rule, the data processing device sends alarm information.
After the data processing device determines that the data source to be monitored triggers the alarm rule, the data processing device sends the alarm event to a data queue cluster (it is understood that the data queue cluster is the same cluster as the data queue cluster storing the data source to be monitored in the front, but belongs to different subjects in terms of logic function, such as the data queue cluster storing the alarm event and the data queue cluster storing the original data); then the data processing device consumes the alarm event in the data queue cluster when the alarm channel engine layer runs in a cluster multi-machine parallel mode, receives the alarm event, and grades the alarm event according to the logic in the alarm channel, thereby generating corresponding alarm information. It is understood that the alarm information includes an alarm event, a receiving user of the alarm information, a sending mode (such as a sending frequency) of the alarm information, and the like.
In this embodiment, the alarm channel engine layer may merge the alarm information corresponding to the alarm event according to the information of the occurrence frequency, duration, state, and the like of the alarm event; and then sent to the user. For example, the same alarm message is only sent to the terminal of the general staff when triggered for the first time, and when the message is triggered for ten minutes, the system sends the alarm message to the terminal of the manager at a higher level.
In this embodiment, after the data processing apparatus sends the alarm information, the data processing apparatus may further record a sending result.
In this embodiment, the data processing apparatus defines the alarm rule of the data source to be monitored by using the SQL-like statement of Esper according to the metadata, so that not only regular verification of a general monitoring system can be achieved, but also more complex business logic verification rules can be achieved, and an extended function is provided for complex business data processing, thereby improving the timeliness of data processing alarm and the universality of the data processing system.
The data processing method in the embodiment of the present application is described above, and the data processing apparatus in the embodiment of the present application is described below.
Specifically, referring to fig. 3, an embodiment of a data processing apparatus in an embodiment of the present application includes:
a processing module 301, configured to register metadata for a data source to be monitored, where the metadata is used to describe information of the data source to be monitored; defining an alarm rule of the data source to be monitored according to the attribute of the metadata;
a monitoring module 302, configured to monitor the data source to be monitored;
a sending module 303, configured to send alarm information when the data source to be monitored meets an alarm condition in the alarm rule.
Optionally, the metadata is an Esper event, the Esper event is abstracted as a map type object, and the metadata is stored in the metadata management layer.
Optionally, the data source to be monitored includes a user account identifier, the metadata password, and a timestamp, and the monitoring module 302 is specifically configured to store the metadata and the alarm rule in a form of control information in a first data queue cluster, where the first data queue cluster is established based on a message queue middleware; the metadata and the alarm rules are consumed and monitored using an Esper complex event handling mechanism.
Optionally, the sending module 303 is specifically configured to set an associated alarm channel according to the alarm rule; and sending the alarm information by utilizing the alarm information processing logic of the alarm channel.
Optionally, the sending module 303 is specifically configured to store the alarm information in a second data queue cluster, where the second data queue cluster is established based on a message queue middleware; and consuming the alarm information and sending the alarm information by utilizing the alarm information processing logic of the alarm channel.
Optionally, the alarm information processing logic includes:
classifying the alarm information according to duration, and sending the alarm information according to the classification; and/or acquiring the occurrence frequency of the alarm information, and merging and sending the alarm information when the occurrence frequency exceeds a first preset threshold value; and/or acquiring the duration of the alarm information, and merging and sending the alarm information when the duration exceeds a second preset threshold.
Referring to fig. 4, another embodiment of a data processing apparatus in the embodiment of the present application includes:
a transceiver 401, a processor 402, a bus 403;
the transceiver 401 is connected to the processor 402 via the bus 403;
the bus 403 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
The processor 402 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP.
The processor 402 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Referring to fig. 4, the data processing apparatus may further include a memory 404. The memory 404 may include volatile memory (volatile memory), such as random-access memory (RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a Hard Disk Drive (HDD) or a solid-state drive (SSD); the memory 404 may also comprise a combination of memories of the kind described above.
Optionally, the memory 404 may also be used for storing program instructions, and the processor 402 calls the program instructions stored in the memory 404, and may perform one or more steps in the above embodiments, or in alternative embodiments, implement the functions of the data processing apparatus in the above methods.
The processor 402 performs the following steps: registering metadata for a data source to be monitored, wherein the metadata is used for describing information of the data source to be monitored; defining an alarm rule of the data source to be monitored according to the attribute of the metadata; monitoring the data source to be monitored;
the transceiver 401 performs the following steps: and when the data source to be monitored triggers the alarm rule, sending alarm information.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, which are essential or part of the technical solutions contributing to the prior art, or all or part of the technical solutions, may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present application.
Claims (7)
1. A method of data processing, comprising:
registering metadata for a data source to be monitored, wherein the metadata is used for describing information of the data source to be monitored, the metadata is an Esper event, the Esper event is abstracted into a map type object, and the metadata is stored in a metadata management layer;
defining an alarm rule of the data source to be monitored according to the attribute of the metadata;
monitoring the data source to be monitored;
when the data source to be monitored triggers the alarm rule, sending alarm information;
the monitoring the data source to be monitored comprises:
storing the metadata and the alarm rules in a first data queue cluster in the form of control information, wherein the first data queue cluster is established based on Kafka message queue middleware;
and consuming the metadata and the alarm rule, and monitoring the metadata and the alarm rule by utilizing an Esper complex event processing mechanism.
2. The method of claim 1, wherein sending the alert message comprises:
setting a related alarm channel according to the alarm rule;
and sending the alarm information by utilizing an alarm information processing logic of the alarm channel.
3. The method of claim 2, wherein sending the alert information using the alert information processing logic of the alert channel comprises:
storing the alarm information in a second data queue cluster, wherein the second data queue cluster is established based on message queue middleware;
and consuming the alarm information and sending the alarm information by utilizing an alarm information processing logic of the alarm channel.
4. The method of claim 2 or 3, wherein the alert information processing logic comprises:
classifying the alarm information according to duration, and sending the alarm information according to the classification;
and/or the presence of a gas in the gas,
acquiring the occurrence frequency of the alarm information, and merging and sending the alarm information when the occurrence frequency exceeds a first preset threshold value;
and/or the presence of a gas in the gas,
and acquiring the duration of the alarm information, and merging and sending the alarm information when the duration exceeds a second preset threshold.
5. A data processing apparatus, characterized by comprising:
the system comprises a processing module, a monitoring module and a monitoring module, wherein the processing module is used for registering metadata for a data source to be monitored, and the metadata is used for describing information of the data source to be monitored; defining an alarm rule of the data source to be monitored according to the attribute of the metadata, wherein the metadata is an Esper event, the Esper event is abstracted to be a map type object, and the metadata is stored in a metadata management layer;
the monitoring module is used for storing the metadata and the alarm rule in a first data queue cluster in the form of control information, and the first data queue cluster is established based on Kafka message queue middleware; consuming the metadata and the alarm rule, and monitoring the metadata and the alarm rule by utilizing an Esper complex event processing mechanism;
and the sending module is used for sending alarm information when the data source to be monitored triggers the alarm rule.
6. A data processing apparatus comprising at least one processor and a memory, wherein a computer readable program is stored in the memory, and wherein the processor is configured to perform the method of any one of claims 1 to 4 by executing the program in the memory.
7. A computer-readable storage medium having stored thereon computer instructions for performing the method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910637057.5A CN110362455B (en) | 2019-07-15 | 2019-07-15 | Data processing method and data processing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910637057.5A CN110362455B (en) | 2019-07-15 | 2019-07-15 | Data processing method and data processing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110362455A CN110362455A (en) | 2019-10-22 |
| CN110362455B true CN110362455B (en) | 2022-12-20 |
Family
ID=68219160
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910637057.5A Active CN110362455B (en) | 2019-07-15 | 2019-07-15 | Data processing method and data processing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110362455B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110865921A (en) * | 2019-11-08 | 2020-03-06 | 拉扎斯网络科技（上海）有限公司 | Data monitoring method, apparatus, readable storage medium and electronic device |
| CN111143167B (en) * | 2019-12-24 | 2021-01-01 | 北京优特捷信息技术有限公司 | Alarm merging method, device, equipment and storage medium for multiple platforms |
| CN111209165B (en) * | 2020-01-05 | 2021-03-16 | 光大兴陇信托有限责任公司 | Two-stage monitoring processing method based on channel |
| CN111339142A (en) * | 2020-02-26 | 2020-06-26 | 广州信安数据有限公司 | Data monitoring response method, computer readable storage medium and data driving platform |
| CN111885119A (en) * | 2020-07-02 | 2020-11-03 | 天筑科技股份有限公司 | Data transmission method and device of monitoring unit |
| CN112597130B (en) * | 2020-12-22 | 2025-02-14 | 天聚地合(苏州)数据股份有限公司 | Data early warning method, device, electronic device and storage medium |
| CN112732531A (en) * | 2021-01-14 | 2021-04-30 | 北京世纪互联宽带数据中心有限公司 | Monitoring data processing method and device |
| CN112766975B (en) * | 2021-01-20 | 2025-06-10 | 中信银行股份有限公司 | Risk detection method and device, electronic equipment and readable storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8775389B2 (en) * | 2011-03-06 | 2014-07-08 | International Business Machines Corporation | Implementing continuous control monitoring for audit purposes using a complex event processing environment |
| CN102611565B (en) * | 2011-10-18 | 2015-07-08 | 深圳供电局有限公司 | Regular expression-based alarm correlation analysis method for monitoring system |
| CN103200039B (en) * | 2012-01-09 | 2017-01-18 | 阿里巴巴集团控股有限公司 | Data monitoring method and device |
| CN106371973B (en) * | 2016-08-31 | 2019-04-09 | 广州品唯软件有限公司 | A data processing method, device and system |
| CN107832200A (en) * | 2017-10-24 | 2018-03-23 | 平安科技（深圳）有限公司 | Alert processing method, device, computer equipment and storage medium |
| CN108234199A (en) * | 2017-12-20 | 2018-06-29 | 中国联合网络通信集团有限公司 | Monitoring method, apparatus and system based on Kafka |
-
2019
- 2019-07-15 CN CN201910637057.5A patent/CN110362455B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110362455A (en) | 2019-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110362455B (en) | Data processing method and data processing device | |
| CN111881011B (en) | Log management method, platform, server and storage medium | |
| CN108038130B (en) | Automatic false user cleaning method, device, equipment and storage medium | |
| CN112307057A (en) | Data processing method and device, electronic device, computer storage medium | |
| CN110995497A (en) | Method for unified operation and maintenance in cloud computing environment, terminal device and storage medium | |
| CN110932933B (en) | Network condition monitoring method, computing device and computer storage medium | |
| CN112737800B (en) | Service node fault positioning method, call chain generating method and server | |
| US10554701B1 (en) | Real-time call tracing in a service-oriented system | |
| CN110543512B (en) | Information synchronization method, device and system | |
| WO2022127944A2 (en) | Monitoring method and apparatus for electronic device, and electronic device | |
| CN111740868A (en) | Alarm data processing method and device and storage medium | |
| CN112948224A (en) | Data processing method, device, terminal and storage medium | |
| CN112100239A (en) | Portrait generation method and apparatus for vehicle detection device, server and readable storage medium | |
| CN110018932B (en) | A method and device for monitoring container disks | |
| CN109377391B (en) | Information tracking method, storage medium and server | |
| CN107818522B (en) | User power condition query method and terminal equipment | |
| CN111506446B (en) | Interface fault detection method and server | |
| CN109508356B (en) | Data abnormality early warning method, device, computer equipment and storage medium | |
| CN115168489A (en) | Data evidence storage method and device based on block chain | |
| TW201928747A (en) | Server and monitoring method thereof | |
| CN107957942B (en) | SQL script fault repairing method and terminal thereof | |
| CN109450700B (en) | Visual service detection method and device | |
| CN119226320B (en) | Data processing method, device, apparatus and program product | |
| CN115604668B (en) | Short message sending and pushing monitoring method, device, equipment and storage medium | |
| CN115811544A (en) | Unitized monitoring method, system, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |