CN116737500A - Intensive meteorological equipment and meteorological data whole-flow monitoring method and system - Google Patents
Intensive meteorological equipment and meteorological data whole-flow monitoring method and system Download PDFInfo
- Publication number
- CN116737500A CN116737500A CN202310734463.XA CN202310734463A CN116737500A CN 116737500 A CN116737500 A CN 116737500A CN 202310734463 A CN202310734463 A CN 202310734463A CN 116737500 A CN116737500 A CN 116737500A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- data
- equipment
- meteorological
- strategy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Alarm Systems (AREA)
Abstract
The invention provides an intensive meteorological equipment and meteorological data whole-flow monitoring method and system, wherein the method comprises the steps of S1, monitoring of local meteorological data acquisition equipment, a server and network equipment; s2, monitoring structured data and unstructured data; s3, equipment monitoring and data monitoring result analysis; s4, displaying a monitoring result; the system comprises a meteorological equipment monitoring module, a meteorological data monitoring module, a monitoring result analysis module, a monitoring result display module, a strategy library, a system management module and a log module. The advantages are that: by means of integrated monitoring of meteorological equipment, meteorological data and a meteorological service system, the problem that two leather sheets are monitored by one set of system is solved. The full-flow monitoring of the meteorological data and the monitoring of the whole service life of the meteorological equipment are realized.
Description
Technical Field
The invention relates to the technical field of meteorological equipment and data monitoring and alarming, in particular to an intensive meteorological equipment and full-flow monitoring method and system for meteorological data.
Background
Aiming at meteorological data, meteorological equipment state monitoring and the like, related researches are carried out by existing personnel, some achievements are obtained, and an operating monitoring and duty management system for meteorological data is invented, such as Chen Jingjing, and is mainly used for monitoring the meteorological data, pushing the result to the duty personnel and generating a report, but the system cannot monitor the states of data receiving, processing, storing and publishing equipment in real time; meanwhile, the system does not have a monitoring and analyzing function of locally collecting the quality condition of the data. Yang Xiaojun and the like, which mainly monitor and analyze data in real time, but cannot monitor the running condition of informationized equipment. Qin Shian and the like, which are mainly used for monitoring and displaying the operation condition of meteorological equipment and can be pushed to an on-duty person, but the system cannot monitor the data circulation condition and does not have a data quality monitoring function.
The technical scheme mainly aims at the single function of the equipment state or the data receiving processing state to carry out monitoring, and neither relates to the monitoring of an associated service system nor the monitoring of the correctness of data, and generally neither gives the stability of the service system, single equipment and single data source nor realizes the associated monitoring of the whole flow of data, the whole service life of the equipment and the whole coverage of the service system.
Because the meteorological industry is complex, the related meteorological equipment is more, the data types are more, the processing flow is complex, the related links are numerous, the number of the required informationized equipment is also more, a set of business system often relates to the flows of data receiving, processing, storing, publishing and the like, and each flow also relates to various informationized equipment, such as: the system comprises a data receiving server, a data processing server, a data storage server, a website server, a router, an exchanger and the like, wherein the difficulty of manual inspection, monitoring and maintenance is high, the efficiency is low, the existing technology is adopted to independently conduct monitoring of data or equipment, and the existing technology can be used for displaying whether the running state of a current system is normal or not, but the whole condition of a service system cannot be completely displayed, so that fault location is not facilitated, the whole service life management of the service system is also not facilitated, and whether the running of the service system in a future period is normal or not is difficult to predict.
Disclosure of Invention
The invention aims to provide an intensive meteorological equipment and a full-flow monitoring method and system for meteorological data, so as to solve the problems in the prior art.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
an intensive meteorological equipment and meteorological data whole-flow monitoring method comprises the following steps,
s1, monitoring local meteorological data acquisition equipment, a server and network equipment:
invoking a state information monitoring strategy of the local meteorological data acquisition equipment, a server monitoring strategy and a network equipment monitoring strategy to respectively compare and judge the state information of the local meteorological data acquisition equipment, each performance index of the server and the HTTP service state of the network equipment, and writing the comparison and judgment result into a database;
s2, monitoring structured data and unstructured data:
the method comprises the steps of calling a structured data monitoring strategy and an unstructured data monitoring strategy to respectively compare and check the latest time data of a database with corresponding acquisition time and the latest time data of a data catalog with corresponding acquisition time and file quantity, and writing comparison and check results into the database;
s3, analyzing the device monitoring and data monitoring results:
based on the equipment monitoring result data fed back by the S1, comprehensively analyzing the equipment monitoring state and the failure rate, and iteratively analyzing and predicting the service life of the equipment based on a neural network algorithm; based on the meteorological data monitoring result data fed back by the S2, comprehensively analyzing the stability and the failure rate of the data source; writing the related calculation result into a database;
s4, displaying monitoring results:
based on the calculation result fed back by the S3, classification and grading display of real-time monitoring of single equipment, single data, each processing link and each service system is carried out, and statistical analysis conditions of monitoring results of each equipment, each data and each service system are given out in the form of charts and the like.
Preferably, the monitoring of the local meteorological data acquisition device is in particular,
and calling a state information monitoring strategy, acquiring state information of the local meteorological data acquisition equipment based on a communication serial port of the local meteorological data acquisition equipment, comparing the communication state in the state information with the temperature of an acquisition board with corresponding set conditions in the state information monitoring strategy, determining whether the communication state is abnormal or not, and writing the comparison result into a database.
Preferably, the monitoring of the server monitoring is in particular,
starting SNMP service on the server, calling a server monitoring strategy every minute, acquiring each performance index of the server at regular time based on the SNMP service according to a preset acquisition frequency, comparing CPU (Central processing Unit) utilization rate, memory utilization rate, disk space utilization rate, network port inflow flow and network port outflow flow of the server with thresholds of each performance index specified in the server monitoring strategy respectively, and writing the comparison result into a database.
Preferably, the server monitoring policy includes a general warning policy and a serious warning policy, when any one of the performance indexes of the server exceeds a corresponding threshold value in the serious warning policy, the server is directly prompted to carry out serious warning, when all the performance indexes do not exceed the corresponding threshold value in the general warning policy, the server is prompted to carry out normal warning, otherwise, the server is prompted to carry out general warning.
Preferably, the monitoring of the network device is in particular,
and calling a network equipment monitoring policy, acquiring an HTTP address of the monitoring network equipment, initiating a request to a website, acquiring an HTTP status code, comparing the status code with a corresponding set condition in the network equipment monitoring policy, judging the HTTP service state, and writing the comparison result into a database.
Preferably, the local meteorological data acquisition equipment comprises a local automatic meteorological station and a shallow wind measurement system;
the server comprises a meteorological data receiving server, a meteorological data processing server, a meteorological data storage server and a meteorological service website publishing server;
the network equipment comprises a switch, a router and a network printer.
Preferably, the monitoring of the structured data is in particular,
invoking a structured data monitoring strategy, scanning a designated database according to designated frequency, acquiring data of the latest time of the database and corresponding acquisition time, comparing the acquisition time with the current time in the structured data monitoring strategy, and determining whether the data arrival time is accurate or not; the format, the limit value, the main change range and the space-time consistency of the latest time data of the database are respectively compared with corresponding set conditions in the structured data monitoring strategy, so that the correctness check of the latest time data of the database is realized; and the alignment is written to the database.
Preferably, the monitoring of unstructured data is in particular,
invoking an unstructured data monitoring strategy, scanning a designated data catalog according to designated frequency, acquiring the latest time data of the data catalog, corresponding acquisition time and file number, comparing the acquisition time with the current time in the unstructured data monitoring strategy, and determining whether the time from the data to the report is accurate; comparing the number of the files with corresponding set conditions in the unstructured data monitoring strategy, and determining whether the number of the files meets the conditions or not; the data format, the file size and the file type of the data when the data catalogue is up to date are respectively compared with corresponding set conditions in the unstructured data monitoring strategy, so that the correctness check of the data when the data catalogue is up to date is realized; and writing the comparison result into a database.
The invention also aims to provide an intensive meteorological equipment and meteorological data whole-flow monitoring system which is used for realizing the method of any one of the above, and the system comprises,
meteorological equipment monitoring module: invoking a state information monitoring strategy of the local meteorological data acquisition equipment, a server monitoring strategy and a network equipment monitoring strategy to respectively compare and judge the state information of the local meteorological data acquisition equipment, each performance index of the server and the HTTP service state of the network equipment, and writing the comparison and judgment result into a database;
weather data monitoring module: the method comprises the steps of calling a structured data monitoring strategy and an unstructured data monitoring strategy to respectively compare and check the latest time data of a database with corresponding acquisition time and the latest time data of a data catalog with corresponding acquisition time and file quantity, and writing comparison and check results into the database;
the monitoring result analysis module: the monitoring result data are used for receiving the monitoring result data fed back by the meteorological equipment monitoring module and the meteorological data monitoring module; comprehensively analyzing the monitoring state and the failure rate of the equipment based on the equipment monitoring result data fed back by the meteorological equipment monitoring module, and iteratively analyzing and predicting the service life of the equipment based on a neural network algorithm; based on the meteorological data monitoring result data fed back by the meteorological data monitoring module, comprehensively analyzing the stability and the failure rate of the data source; writing the related calculation result into a database;
the monitoring result display module: the system is used for receiving the calculation result fed back by the monitoring result analysis module, carrying out classification and grading display of real-time monitoring of single equipment, single data, each processing link and each service system based on the calculation result, and giving out the statistical analysis conditions of the monitoring results of each equipment, each data and each service system in the form of a chart and the like;
policy library: monitoring strategy for storing state information of local meteorological data acquisition equipment, monitoring strategy of server and monitoring strategy of network equipment 、 Structured data monitoring policies and unstructured data monitoring policies.
Preferably, the system comprises a system management module and a log module;
the system management module receives the manually entered state information monitoring strategy, server monitoring strategy and network equipment monitoring strategy of the local meteorological data acquisition equipment 、 The structured data monitoring strategy, unstructured data monitoring strategy, system account information, service system name, picture, equipment position and manager data are written into corresponding forms in a strategy library and transmitted to a meteorological equipment monitoring module, a meteorological data monitoring module, a monitoring result analysis module and a monitoring result display module; the method is also used for checking the operation log;
the log module is used for receiving the operation behaviors of personnel on the system and recording the operation behaviors.
The beneficial effects of the invention are as follows: 1. by means of integrated monitoring of meteorological equipment, meteorological data and a meteorological service system, the problem that two leather sheets are monitored by one set of system is solved. 2. The monitoring of meteorological equipment life span has been realized, includes: real-time monitoring of equipment state, real-time analysis of equipment health state, iterative analysis of equipment life, help management personnel to grasp equipment state in real time, develop updating of equipment in good time, etc. 3. The full flow monitoring of the meteorological data is realized, and the full flow monitoring comprises the following steps: circulation condition monitoring in links of data receiving, processing, storing, publishing and the like; meanwhile, the quality monitoring of the locally collected meteorological data and the correctness checking of the meteorological data received by other means are realized; the quality and effect of meteorological data monitoring are greatly improved. 4. On the basis of single equipment and single data monitoring, the service system is associated, so that the stability of the operation of the service system can be checked, the abnormal rate of the operation of single equipment or a certain type of data receiving processing flow can be checked, and the abnormal rate can be used as a basis for judging whether the operation of the service system is stable or not and whether the important attention is needed in daily life or not.
Drawings
FIG. 1 is a schematic flow chart of monitoring meteorological equipment according to an embodiment of the present invention;
FIG. 2 is a flow chart of structured weather data monitoring in accordance with an embodiment of the present invention;
FIG. 3 is a schematic flow chart of unstructured meteorological data monitoring in an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a monitoring system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the detailed description is presented by way of example only and is not intended to limit the invention.
Example 1
In this embodiment, an intensive meteorological equipment and meteorological data overall process monitoring method is provided, which includes four parts of content, including equipment monitoring, data monitoring, monitoring result analysis and monitoring result display, wherein the specific content of each part is as follows:
1. device monitoring
As shown in fig. 1, the monitoring system comprises a local meteorological data acquisition device, a server and a network device:
and calling a state information monitoring strategy of the meteorological data acquisition equipment, a server monitoring strategy and a network equipment monitoring strategy to respectively compare and judge the state information of the local meteorological data acquisition equipment, each performance index of the server and the HTTP service state of the network equipment, and writing the comparison and judgment result into a database.
1. The monitoring of the local meteorological data acquisition equipment is specifically as follows:
and calling a state information monitoring strategy of the local meteorological data acquisition equipment, acquiring state information of the local meteorological data acquisition equipment based on a communication serial port of the local meteorological data acquisition equipment, comparing the communication state in the state information with the temperature of an acquisition board with corresponding set conditions in the state information monitoring strategy, determining whether the communication state is abnormal or not, and writing the comparison result into a database.
The local meteorological data acquisition equipment mainly comprises meteorological information acquisition equipment such as a local automatic meteorological station, a shallow wind measurement system and the like.
When the communication state and the temperature of the acquisition board are normal, the communication state and the temperature of the acquisition board are normal; when one of them is abnormal, it indicates abnormality. The return value abnormality judgment of the local meteorological information acquisition equipment is as follows: if normal, the display is green, if abnormal, the display is red, and the system gives an alarm.
2. The monitoring of server monitoring specifically is:
starting SNMP service on the server, calling a server monitoring strategy every minute, acquiring each performance index of the server at regular time based on the SNMP service according to a preset acquisition frequency, comparing CPU (Central processing Unit) utilization rate, memory utilization rate, disk space utilization rate, network port inflow flow and network port outflow flow of the server with thresholds of each performance index specified in the server monitoring strategy respectively, and writing the comparison result into a database.
The server mainly comprises a meteorological data receiving server, a meteorological data processing server, a meteorological data storage server, a meteorological service website issuing server and the like.
The server monitoring strategy comprises a general warning strategy and a serious warning strategy, when any one of the server performance indexes exceeds the corresponding threshold value in the serious warning strategy, the server is directly prompted to carry out serious warning, when all the performance indexes do not exceed the corresponding threshold value in the general warning strategy, the server is prompted to carry out normal warning, and otherwise, the server is prompted to carry out general warning.
The return value abnormality judgment monitored by the server is divided into three stages of display: if the return value reaches the normal warning, the system gives a corresponding warning, and if the return value reaches the serious warning, the system gives a corresponding warning.
3. The monitoring of the network equipment specifically comprises the following steps:
and calling a network equipment monitoring policy, acquiring an HTTP address of the monitoring network equipment, initiating a request to a website, acquiring an HTTP status code, comparing the status code with a corresponding set condition in the network equipment monitoring policy, judging the HTTP service state, and writing the comparison result into a database.
The network device mainly comprises a switch, a router, a network printer and other devices with HTTP service.
Judging the state of the website according to the state code returned by the target website, and when the returned value is 200, judging that the state of the website is normal; judging that the website is abnormal when the return code is not 200 or has no return value, wherein the return value is 400 and represents an error request; returning to 404, indicating that the requested web page does not exist; when the return value is 503, it indicates that the service is not available. And normally displaying green, displaying red if abnormal, and giving an alarm if abnormal.
2. Data monitoring
As shown in fig. 2 and 3, includes monitoring of structured and unstructured data:
and calling a structured data monitoring strategy and an unstructured data monitoring strategy to respectively compare and check the latest time data of the database, the corresponding acquisition time and the latest time data of the data catalog, the corresponding acquisition time and the number of files, and writing the comparison and check result into the database.
The structured data mainly refers to related meteorological data acquired by automatic meteorological stations, shallow wind measurement systems and the like. Unstructured data mainly refers to file data in a local folder, an SMB shared directory, an FTP remote directory and other folders.
1. The specific process of the structured data monitoring is as follows:
as shown in fig. 2, a structured data monitoring strategy is called, a designated database is scanned according to a designated frequency, the latest time data of the database and the corresponding acquisition time are obtained, the acquisition time is compared with the current time in the structured data monitoring strategy, and whether the time from the data to the report is accurate or not is determined; the format, the limit value, the main change range and the space-time consistency of the latest time data of the database are respectively compared with corresponding set conditions in the structured data monitoring strategy, so that the correctness check of the latest time data of the database is realized; and the alignment is written to the database.
When the collection time is compared with the current time, when the difference time of the collection time and the current time is smaller than or equal to the set condition in the structured data monitoring strategy, the collection time is judged to be normal, otherwise, the collection time is abnormal, and an alarm is given under the abnormal condition.
When the data accuracy is checked when the database is up to date, when the format check, the limit value check, the main change range check and the space-time consistency check are consistent with the set conditions in the structured data monitoring strategy, judging that the data is normal, otherwise, judging that the data is abnormal, and giving an alarm under the abnormal condition. And normally displaying green, displaying red if abnormal, and giving an alarm if abnormal.
2. The monitoring of unstructured data is in particular,
as shown in fig. 3, invoking an unstructured data monitoring strategy, scanning a designated data directory according to a designated frequency, acquiring the latest time data of the data directory, corresponding acquisition time and file number, comparing the acquisition time with the current time in the unstructured data monitoring strategy, and determining whether the time from the data to the report is accurate; comparing the number of the files with corresponding set conditions in the unstructured data monitoring strategy, and determining whether the number of the files meets the conditions or not; the data format, the file size and the file type of the data when the data catalogue is up to date are respectively compared with corresponding set conditions in the unstructured data monitoring strategy, so that the correctness check of the data when the data catalogue is up to date is realized; and writing the comparison result into a database.
When the collection time is compared with the current time, when the difference time of the collection time and the current time is smaller than or equal to the set condition in the unstructured data monitoring strategy, the collection time is judged to be normal, otherwise, the collection time is abnormal, and an alarm is given under the abnormal condition.
And when the number of the files accords with the set conditions in the unstructured data monitoring strategy, judging that the number is normal, otherwise, giving an alarm when the number is abnormal.
When the data accuracy is checked when the data catalog is up to date, when the format check, the file size and the file type are consistent with the set conditions in the unstructured data monitoring strategy, judging that the data is normal, otherwise, judging that the data is abnormal, and giving an alarm under the abnormal condition. And normally displaying green, displaying red if abnormal, and giving an alarm if abnormal.
3. Monitoring result analysis
The method comprises the following steps of equipment monitoring and data monitoring result analysis:
comprehensively analyzing the monitoring state and failure rate of the equipment based on the equipment monitoring result data fed back by the equipment monitoring, and iteratively analyzing and predicting the service life of the equipment based on a neural network algorithm; comprehensively analyzing the stability and failure rate of the data source based on meteorological data monitoring result data fed back by data monitoring; and writing the relevant calculation result into a database.
4. Monitoring result display
Based on the calculation result of the analysis feedback of the monitoring result, the classification and grading display of the real-time monitoring of the single equipment, the single data, each processing link and each service system is carried out, and the statistical analysis conditions of the monitoring result of each equipment, each data and each service system are given out in the form of charts and the like.
As shown in fig. 4, in this embodiment, an intensive meteorological apparatus and meteorological data overall process monitoring system is provided, the system is used to implement the method described above, and the system includes,
1. meteorological equipment monitoring module: the method is used for calling a state information monitoring strategy, a server monitoring strategy and a network device monitoring strategy to respectively compare and judge the state information of the local meteorological data acquisition device, each performance index of the server and the HTTP service state of the network device, and writing the comparison and judgment result into a database.
The meteorological equipment monitoring module comprises local meteorological data acquisition equipment monitoring and meteorological informatization equipment monitoring; the local meteorological data acquisition equipment monitoring and the meteorological informatization equipment monitoring are mainly used for monitoring meteorological information acquisition equipment such as a local automatic meteorological station, a shallow wind measurement system and the like in real time; the weather information equipment monitoring comprises: the monitoring method comprises the steps of server monitoring and network equipment monitoring, wherein the server monitoring mainly monitors a meteorological data receiving server, a meteorological data processing server, a meteorological data storage server, a meteorological service website publishing server and the like, and the network equipment monitoring mainly monitors equipment systems with HTTP services such as switches, routers and network printers.
2. Weather data monitoring module: the method comprises the steps of calling a structured data monitoring strategy and an unstructured data monitoring strategy to respectively compare and check the latest time data of a database, the corresponding acquisition time and the latest time data of a data catalog, the corresponding acquisition time and the number of files, and writing the comparison and check result into the database.
The meteorological data monitoring module comprises: structured data monitoring and unstructured data monitoring, wherein structured data monitoring mainly monitors relational database data, such as: the structured data monitoring comprises the steps of monitoring data arrival conditions in real time on one hand, and monitoring the correctness (data quality) of the data on the other hand on the basis of a certain quality control algorithm on the basis of the data such as an automatic weather station, a shallow wind measurement system and the like; the unstructured data monitoring is mainly used for monitoring the report condition of files in a local folder, an SMB shared directory, an FTP remote directory and the like and the correctness of the data files.
3. The monitoring result analysis module: the monitoring result data are used for receiving the monitoring result data fed back by the meteorological equipment monitoring module and the meteorological data monitoring module; based on equipment monitoring result data fed back by the meteorological equipment monitoring module, comprehensively analyzing equipment monitoring states, failure rates and the like, and iteratively analyzing and predicting the service life of equipment based on a neural network algorithm; based on the meteorological data monitoring result data fed back by the meteorological data monitoring module, comprehensively analyzing the stability, failure rate and other conclusions of the data source; and writing the relevant calculation result into a database.
4. The monitoring result display module: the system is used for receiving the calculation result fed back by the monitoring result analysis module, carrying out classification and grading display of real-time monitoring of single equipment, single data, each processing link and each service system based on the calculation result, and giving out the statistical analysis conditions of the monitoring results of each equipment, each data and each service system in the form of a chart and the like.
5. Policy library: the method is used for storing state information monitoring policies, server monitoring policies, network equipment monitoring policies, structured data monitoring policies and unstructured data monitoring policies.
6. And a system management module: by receiving the state information monitoring strategy, server monitoring strategy and network equipment monitoring strategy of the manually entered local meteorological data acquisition equipment 、 The structured data monitoring strategy, unstructured data monitoring strategy, system account information, service system name, picture, equipment position and manager data are written into corresponding forms in a strategy library and transmitted to a meteorological equipment monitoring module, a meteorological data monitoring module, a monitoring result analysis module and a monitoring result display module; but also for oplog viewing.
7. And a log module: the system is used for receiving and recording the operation behaviors of personnel on the system.
In this embodiment, the system can implement the following functions based on the above modules:
1. monitoring and abnormal alarming of meteorological equipment. (1) implementing local meteorological data acquisition devices such as: monitoring real-time states of equipment such as an automatic weather station and the like and giving an abnormal alarm; (2) The real-time state acquisition, display and abnormal alarm of the network equipment such as the server, the storage equipment and the switch are realized; (3) And comprehensively analyzing the long-term state monitoring data set to obtain the health state index and the like of the equipment, predicting the service life of the equipment based on algorithms such as a neural network and the like, and providing support for an administrator to formulate an equipment updating and upgrading plan.
2. And (5) full-flow monitoring and abnormal alarming of meteorological data. (1) The method realizes real-time synchronous monitoring of the reporting condition and the data quality of the structured data, monitors the real-time collected data based on a data quality control algorithm on the basis of data collection monitoring, meets unqualified data of the data quality, gives an alarm in real time, can remind an on-duty person to check in time, avoids normal data collection and warehousing caused by equipment sensor and collector faults, but the situation of data errors is continuous, and mainly comprises the following steps: data collected by local meteorological equipment (such as an automatic meteorological station and the like) in real time; (2) The method realizes the real-time verification of the report condition monitoring and the data integrity, the format, the size and the like of unstructured data, effectively solves the problems of multiple types of meteorological data, large data volume, multiple links related to receiving, processing and publishing processes, multiple fault points and the like, and mainly comprises the following steps: file data of the data is received through means such as ground broadband, satellite broadcast receiving system and the like; (3) Based on monitoring data for a period of time, the stability of each data source is analyzed, stability of all the data sources and points with easy faults are given, and quantitative basis can be provided for management staff to upgrade, reform and the like on software for subsequent data receiving, processing, transmission and the like.
3. And classifying and correlating all the monitored equipment and data according to the local service system, so that the overall working condition of each service system is given based on the monitoring results of the corresponding equipment and data, and meanwhile, when the service system fails, the fault point can be rapidly positioned based on the monitoring of a single equipment and a single data source and fed back to a manager, thereby facilitating rapid fault treatment. In addition, the intelligent analysis of the data monitored by the long-term service system can give out the working stability of each service system, so that the running stability of the service system and the links easy to fail can be prejudged in advance, and support can be provided for the construction, upgrading, improvement and the like of the subsequent service system.
Example two
In this embodiment, for the structured relational database data, the data detection program obtains the arrival condition of the data by periodically querying the number of statistics to report. The specific method is to define a timing task, periodically scan and count a target data table, and acquire the number of data in a specific time period.
Taking the hour-by-hour manual observation data as an example, the latest group of data is mainly verified, and a specific sql command is "select '1' ISOK," manual observation station ' as "name," DATE_FORMAT (observeTime, '%Y-%m-%d%H:% i ') as "data time," temperature as "value," TIMESTAMPDIFF (MINUTE, observeTime, NOW ()) as "timeout time (MINUTEs)" from surfaceobservation t2 where observeTime > = DATE_ADD (NOW (), INTERVAL-1 hour) ORDER BY observeTime descLIMIT.
The monitoring period adopts the cron expression to set timing tasks, and according to the observation standard, the manual observation time is generally: the previous hour 50 minutes-10 minutes of this hour, the observation time range is 20 minutes, the observation data in this range are all in normal state, and the observation data is abnormal beyond this time period, so the manual observation data starts scanning every 11 minutes and 0 seconds, then the process is triggered every 5 minutes, and the specific cron expression is "0 11/5? ".
The data detection program of unstructured file data periodically scans the target folder, records the file name, file size, generation time, modification time and other attributes of the file under the directory, and stores the scanning result of each time in the data detection structure table. Meteorological data generally has higher regularity, unstructured file data can be analyzed according to naming rules of the file, and whether the data arrives or not is monitored.
The method is described by taking FY-2G satellite cloud picture report condition monitoring configuration as an example: the FY-2G satellite cloud picture file naming rule is "FY2G_ { yyyy_MM_dd_HH-8} _ d + _ L_PJ1_IR1.JPG".
Since the FY-2G satellite cloud image is named world time, HH acquired from the system is Beijing time, so that-8 hours are needed to be converted into world time, and in addition, the satellite cloud image at the whole time is generally received and processed at about 28 minutes per hour, so that the monitoring period is set to be 29 minutes and 0 seconds per hour to start scanning, and then is triggered once every 5 minutes, and the cron expression is' 0 29/5? ".
When monitoring a server and the like, the default configuration/etc/SNMP/snmpd.conf file of the SNMP service of the server needs to be modified:
(1) Configuration SNMP connection character string
vim/etc/snmp/snmpd.conf
The following fields are found:
#sec.name source community
com2sec notConfigUser default public
modifying public to self-defined community string
(2) Modifying permissions to view device nodes
In this document, the device information under the nodes of view system view include.1.3.6.1.2.1.1 and view system view include.1.3.6.1.2.1.25.1.1 are found, and since the device information such as the host CPU and the memory are not under these nodes, these data cannot be acquired.
It is therefore necessary to add a row before view system view include.1.3.6.1.2.1.1:
view systemview included.1
indicating that all device information under node.1 can be viewed.
(3) Modifying configuration of Process locks
Find the following three lines of content
#proc mountd
#proc ntalkd 4
#proc sendmail 10 1
And remove the "#" sign three rows ago and cancel the annotation.
(4) Modifying exechambles/scripts configuration
The "#" sign before # exec echo test/bin/echo hello world is removed and the annotation is cancelled.
(5) Modifying disk locks configuration
Find #disk/10000 line, and remove the preceding "#" sign, cancel the annotation.
(6) Modifying load average checks configuration
Find # load 12 14 14 line and remove the previous "#" sign, cancel the annotation.
(7) Restarting and verifying
1) Restarting the snmp service, the command is: service snmpd restart
2) Validating a modified configured snmp service
The idle rate of CPU can be obtained for verification by using the following commands
snmpwalk-v 2c-c public localhost 1.3.6.1.4.1.2021.11.11.0。
By adopting the technical scheme disclosed by the invention, the following beneficial effects are obtained:
the invention provides an intensive meteorological equipment and a full-flow monitoring method and system for meteorological data, which realize the problem of two-skin monitoring of data, equipment and business by integrating the monitoring of the meteorological equipment, the meteorological data and a meteorological business system. The monitoring of meteorological equipment life span has been realized, includes: real-time monitoring of equipment state, real-time analysis of equipment health state, iterative analysis of equipment life, help management personnel to grasp equipment state in real time, develop updating of equipment in good time, etc. The full flow monitoring of the meteorological data is realized, and the full flow monitoring comprises the following steps: circulation condition monitoring in links of data receiving, processing, storing, publishing and the like; meanwhile, the quality monitoring of the locally collected meteorological data and the correctness checking of the meteorological data received by other means are realized; the quality and effect of meteorological data monitoring are greatly improved. On the basis of single equipment and single data monitoring, the service system is associated, so that the stability of the operation of the service system can be checked, the abnormal rate of the operation of single equipment or a certain type of data receiving processing flow can be checked, and the abnormal rate can be used as a basis for judging whether the operation of the service system is stable or not and whether the important attention is needed in daily life or not.
The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which is also intended to be covered by the present invention.
Claims (10)
1. An intensive meteorological equipment and meteorological data whole-flow monitoring method is characterized in that: comprises the following steps of the method,
s1, monitoring local meteorological data acquisition equipment, a server and network equipment:
invoking a state information monitoring strategy of the local meteorological data acquisition equipment, a server monitoring strategy and a network equipment monitoring strategy to respectively compare and judge the state information of the local meteorological data acquisition equipment, each performance index of the server and the HTTP service state of the network equipment, and writing the comparison and judgment result into a database;
s2, monitoring structured data and unstructured data:
the method comprises the steps of calling a structured data monitoring strategy and an unstructured data monitoring strategy to respectively compare and check the latest time data of a database with corresponding acquisition time and the latest time data of a data catalog with corresponding acquisition time and file quantity, and writing comparison and check results into the database;
s3, analyzing the device monitoring and data monitoring results:
based on the equipment monitoring result data fed back by the S1, comprehensively analyzing the equipment monitoring state and the failure rate, and iteratively analyzing and predicting the service life of the equipment based on a neural network algorithm; based on the meteorological data monitoring result data fed back by the S2, comprehensively analyzing the stability and the failure rate of the data source; writing the related calculation result into a database;
s4, displaying monitoring results:
based on the calculation result fed back by the S3, classification and grading display of real-time monitoring of single equipment, single data, each processing link and each service system is carried out, and statistical analysis conditions of monitoring results of each equipment, each data and each service system are given out in the form of charts and the like.
2. The method for full-process monitoring of the intensive meteorological equipment and meteorological data according to claim 1, wherein the method comprises the following steps of: the monitoring of the local meteorological data acquisition device is in particular,
and calling a state information monitoring strategy, acquiring state information of the local meteorological data acquisition equipment based on a communication serial port of the local meteorological data acquisition equipment, comparing the communication state in the state information with the temperature of an acquisition board with corresponding set conditions in the state information monitoring strategy, determining whether the communication state is abnormal or not, and writing the comparison result into a database.
3. The method for full-process monitoring of the intensive meteorological equipment and meteorological data according to claim 1, wherein the method comprises the following steps of: the monitoring of the server monitoring is specifically,
starting SNMP service on the server, calling a server monitoring strategy every minute, acquiring each performance index of the server at regular time based on the SNMP service according to a preset acquisition frequency, comparing CPU (Central processing Unit) utilization rate, memory utilization rate, disk space utilization rate, network port inflow flow and network port outflow flow of the server with thresholds of each performance index specified in the server monitoring strategy respectively, and writing the comparison result into a database.
4. The method for full-process monitoring of intensive meteorological equipment and meteorological data according to claim 3, wherein the method comprises the following steps of: the server monitoring strategy comprises a general warning strategy and a serious warning strategy, when any one of the server performance indexes exceeds the corresponding threshold value in the serious warning strategy, the server is directly prompted to carry out serious warning, when all the performance indexes do not exceed the corresponding threshold value in the general warning strategy, the server is prompted to carry out normal warning, and otherwise, the server is prompted to carry out general warning.
5. The method for full-process monitoring of the intensive meteorological equipment and meteorological data according to claim 1, wherein the method comprises the following steps of: the monitoring of the network device is in particular,
and calling a network equipment monitoring policy, acquiring an HTTP address of the monitoring network equipment, initiating a request to a website, acquiring an HTTP status code, comparing the status code with a corresponding set condition in the network equipment monitoring policy, judging the HTTP service state, and writing the comparison result into a database.
6. The method for full-process monitoring of the intensive meteorological equipment and meteorological data according to claim 1, wherein the method comprises the following steps of: the local meteorological data acquisition equipment comprises a local automatic meteorological station and a shallow wind measurement system;
the server comprises a meteorological data receiving server, a meteorological data processing server, a meteorological data storage server and a meteorological service website publishing server;
the network equipment comprises a switch, a router and a network printer.
7. The method for full-process monitoring of the intensive meteorological equipment and meteorological data according to claim 1, wherein the method comprises the following steps of: the monitoring of the structured data is in particular,
invoking a structured data monitoring strategy, scanning a designated database according to designated frequency, acquiring data of the latest time of the database and corresponding acquisition time, comparing the acquisition time with the current time in the structured data monitoring strategy, and determining whether the data arrival time is accurate or not; the format, the limit value, the main change range and the space-time consistency of the latest time data of the database are respectively compared with corresponding set conditions in the structured data monitoring strategy, so that the correctness check of the latest time data of the database is realized; and the alignment is written to the database.
8. The method for full-process monitoring of the intensive meteorological equipment and meteorological data according to claim 1, wherein the method comprises the following steps of: the monitoring of unstructured data is in particular,
invoking an unstructured data monitoring strategy, scanning a designated data catalog according to designated frequency, acquiring the latest time data of the data catalog, corresponding acquisition time and file number, comparing the acquisition time with the current time in the unstructured data monitoring strategy, and determining whether the time from the data to the report is accurate; comparing the number of the files with corresponding set conditions in the unstructured data monitoring strategy, and determining whether the number of the files meets the conditions or not; the data format, the file size and the file type of the data when the data catalogue is up to date are respectively compared with corresponding set conditions in the unstructured data monitoring strategy, so that the correctness check of the data when the data catalogue is up to date is realized; and writing the comparison result into a database.
9. An intensive meteorological equipment and meteorological data whole-flow monitoring system is characterized in that: a system for implementing the method of any one of the preceding claims 1 to 8, the system comprising,
meteorological equipment monitoring module: invoking a state information monitoring strategy of the local meteorological data acquisition equipment, a server monitoring strategy and a network equipment monitoring strategy to respectively compare and judge the state information of the local meteorological data acquisition equipment, each performance index of the server and the HTTP service state of the network equipment, and writing the comparison and judgment result into a database;
weather data monitoring module: the method comprises the steps of calling a structured data monitoring strategy and an unstructured data monitoring strategy to respectively compare and check the latest time data of a database with corresponding acquisition time and the latest time data of a data catalog with corresponding acquisition time and file quantity, and writing comparison and check results into the database;
the monitoring result analysis module: the monitoring result data are used for receiving the monitoring result data fed back by the meteorological equipment monitoring module and the meteorological data monitoring module; comprehensively analyzing the monitoring state and the failure rate of the equipment based on the equipment monitoring result data fed back by the meteorological equipment monitoring module, and iteratively analyzing and predicting the service life of the equipment based on a neural network algorithm; based on the meteorological data monitoring result data fed back by the meteorological data monitoring module, comprehensively analyzing the stability and the failure rate of the data source; writing the related calculation result into a database;
the monitoring result display module: the system is used for receiving the calculation result fed back by the monitoring result analysis module, carrying out classification and grading display of real-time monitoring of single equipment, single data, each processing link and each service system based on the calculation result, and giving out the statistical analysis conditions of the monitoring results of each equipment, each data and each service system in the form of a chart and the like;
policy library: monitoring strategy for storing state information of local meteorological data acquisition equipment, monitoring strategy of server and monitoring strategy of network equipment 、 Structured data monitoring policies and unstructured data monitoring policies.
10. The enhanced weather plant and weather data whole-process monitoring system as claimed in claim 9, wherein: the system comprises a system management module and a log module;
the system management module receives the manually entered state information monitoring strategy, server monitoring strategy and network equipment monitoring strategy of the local meteorological data acquisition equipment 、 The structured data monitoring strategy, unstructured data monitoring strategy, system account information, service system name, picture, equipment position and manager data are written into corresponding forms in a strategy library and transmitted to a meteorological equipment monitoring module, a meteorological data monitoring module, a monitoring result analysis module and a monitoring result display module; the method is also used for checking the operation log;
the log module is used for receiving the operation behaviors of personnel on the system and recording the operation behaviors.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310734463.XA CN116737500A (en) | 2023-06-20 | 2023-06-20 | Intensive meteorological equipment and meteorological data whole-flow monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310734463.XA CN116737500A (en) | 2023-06-20 | 2023-06-20 | Intensive meteorological equipment and meteorological data whole-flow monitoring method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116737500A true CN116737500A (en) | 2023-09-12 |
Family
ID=87909465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310734463.XA Pending CN116737500A (en) | 2023-06-20 | 2023-06-20 | Intensive meteorological equipment and meteorological data whole-flow monitoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116737500A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118820218A (en) * | 2024-07-09 | 2024-10-22 | 国家卫星气象中心(国家空间天气监测预警中心) | A method and system for monitoring the entire process of meteorological satellite data |
-
2023
- 2023-06-20 CN CN202310734463.XA patent/CN116737500A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118820218A (en) * | 2024-07-09 | 2024-10-22 | 国家卫星气象中心(国家空间天气监测预警中心) | A method and system for monitoring the entire process of meteorological satellite data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110493348B (en) | Intelligent monitoring alarm system based on Internet of things | |
| CN107832196B (en) | Monitoring device and monitoring method for abnormal content of real-time log | |
| CN108667666A (en) | An intelligent operation and maintenance method and system based on visualization technology | |
| US20110191394A1 (en) | Method of processing log files in an information system, and log file processing system | |
| US8706726B2 (en) | Method and system for monitoring and analyzing tickets | |
| CN108964995A (en) | Log correlation analysis method based on time shaft event | |
| US11144376B2 (en) | Veto-based model for measuring product health | |
| CN109614283A (en) | The monitoring system of distributed experiment & measurement system | |
| CN112416872A (en) | A cloud platform log management system based on big data | |
| CN110209518A (en) | A kind of multi-data source daily record data, which is concentrated, collects storage method and device | |
| CN111259073A (en) | An intelligent judgment system for business system running status based on logs, traffic and business access | |
| CN106681300A (en) | Data clustering analysis method and system of power devices | |
| CN103295155A (en) | Security core service system monitoring method | |
| US7369967B1 (en) | System and method for monitoring and modeling system performance | |
| CN107635003A (en) | System log management method, device and system | |
| CN116737500A (en) | Intensive meteorological equipment and meteorological data whole-flow monitoring method and system | |
| CN117112656A (en) | An integrated information intelligent management system and method for scientific and technological volunteer service management | |
| CN116859804A (en) | Safety situation monitoring and early warning system for ship manufacturing workshop | |
| CN112291347A (en) | Network product service and running state monitoring system based on HTTP and method for obtaining test result | |
| CN117640432B (en) | Operation and maintenance monitoring method for distributed data center | |
| CN110995525A (en) | Router detection method based on maintenance matrix | |
| CN117914511A (en) | Security audit system based on data exchange and log analysis | |
| US20240036963A1 (en) | Multi-contextual anomaly detection | |
| CN110688521A (en) | Intelligent operation and maintenance system for video big data | |
| CN117194154A (en) | APM full-link monitoring system and method based on micro-service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |