CN118094417A - CTD data quality monitoring method and device, electronic equipment and medium - Google Patents

Abstract

The embodiment of the invention provides a CTD data quality monitoring method, a device, electronic equipment and a medium, wherein the method comprises the following steps: collecting CTD data of a water area, wherein the CTD data comprises warm salt depth data, biological environment information and other auxiliary parameter data, respectively analyzing the association degree of the biological environment information, the other auxiliary parameter data and the warm salt depth data, selecting target CTD data according to the association degree to construct a target CTD data set, performing abnormality judgment on the target CTD data by adopting an isolated forest algorithm, judging whether the CTD data of the batch is abnormal according to the outlier target CTD data, and sending abnormality alarm information when the CTD data of the batch is determined to be abnormal. The method of the invention carries out anomaly detection and identification on mass diversified CTD data, the data processing mode can efficiently and accurately identify and process the anomalies in the CTD data, improve the reliability, accuracy and analysis efficiency of CTD data quality monitoring, discover the anomalies of the marine environment in time and give an alarm to inform researchers in time.

Description

A CTD data quality monitoring method, device, electronic device and medium

Technical Field

The present invention relates to the technical field of CTD data quality control, and in particular to a CTD data quality monitoring method, a CTD data quality monitoring device, an electronic device and a computer readable medium.

Background technique

Marine environmental monitoring data not only occupies an important position in marine scientific research, but is also an indispensable data source in the fields of marine observation and forecasting, safety assurance, and marine infrastructure construction and maintenance.

The marine environment is complex and changeable, and there are multiple factors affecting the CTD monitoring process. In addition to the main monitoring data such as temperature, salinity and depth, modern CTD instruments also contain a variety of biological environmental information, auxiliary parameters, etc. These data inevitably contain a lot of errors, mistakes, missing and other "noise". Because the time span of CTD monitoring can often reach several months or years, the data volume is huge and the accuracy requirements are high, and a comprehensive and accurate understanding of the data quality is required, so data quality detection has become a crucial part of CTD data quality control.

In response to the above phenomenon, current researchers in the marine field often use a method of judging multiple ranges, screening multiple features in a set of data in turn and individually, marking feature codes in the process, and outputting the judgment results of the overall data quality after comprehensive sorting based on the feature codes.

The judgment range used in this quality control testing process is often derived from a global common data set, which cannot reflect data characteristics with high precision and accuracy. The range judgment method is only effective for a single data feature and it is difficult to verify the accuracy and validity of diversified information within a time period.

Summary of the invention

In view of the above problems, embodiments of the present invention are proposed to provide a CTD data quality monitoring method and a corresponding CTD data quality monitoring device, an electronic device and a computer-readable medium that overcome the above problems or at least partially solve the above problems.

The embodiment of the present invention discloses a CTD data quality monitoring method, the method comprising:

Collect CTD data of water areas, including temperature, salinity, depth, biological environment information, and other auxiliary parameter data;

Respectively analyzing the correlation between the biological environment information, the other auxiliary parameter data and the temperature, salinity and depth data, and selecting target CTD data according to the correlation to construct a target CTD data set;

An isolation forest algorithm is used to judge abnormality of the target CTD data;

It determines whether the current batch of CTD data is abnormal based on the outlier target CTD data and issues an abnormal alarm message when it is determined that the current batch of CTD data is abnormal.

Optionally, the temperature-salinity-depth data include conductivity, temperature, and pressure, and the steps of analyzing the correlation between the biological environment information, the other auxiliary parameter data, and the temperature-salinity-depth data, and selecting target CTD data according to the correlation to construct a target CTD data set include:

De-dimensionalizing and normalizing the CTD data to obtain pre-processed CTD data;

Taking conductivity, temperature and pressure as target reference items, the grey correlation coefficient between each CTD parameter feature in the preprocessed biological environment information and other auxiliary parameter data and each target reference item is calculated, and all CTD parameter features are arranged in descending order according to the grey correlation coefficient, and three groups of CTD parameter feature ordered sequences are obtained;

Selecting a preset number of CTD parameter features with top rankings from each group of CTD parameter feature ordered sequences;

The target CTD data set is formed by using conductivity, temperature, pressure and the preset number of CTD parameter features ranked at the top as target CTD data.

Optionally, the step of using an isolation forest algorithm to perform abnormality judgment on the target CTD data includes:

Using an isolation forest algorithm to calculate the abnormality score of each target CTD data in the target CTD data set;

The outlier target CTD data is determined according to the abnormality score of each target CTD data and a preset abnormality score threshold or an average abnormality score threshold.

Optionally, the step of using an isolation forest algorithm to calculate the abnormality score of each target CTD data in the target CTD data set includes:

S1, assuming that the total number of elements in the target CTD data set is n (n ≥ 5000), define the number of isolated trees t (1 ≤ t ≤ n), the average depth h, the maximum depth H, and the capacity k of the root node subsample set;

S2, sampling with replacement multiple times from the target CTD data set, extracting a specified number of elements in a purely random manner to construct m sub-sample sets (a1, a2...ai...am);

S3, select a subsample set ai as the root node of a tree and randomly select a CTD parameter feature P;

S4, for a single value q of the CTD parameter feature, perform a binary split on the tree, traverse all elements of the subsample set ai, and when any record value r of the CTD parameter feature P is less than or equal to the single value q, place this element in the left child node of the tree, otherwise place it in the right child node;

S5, using step S4, recursively constructing the left child node and the right child node to build a binary tree. The stopping condition is that any element in the array ai is isolated or the height of the tree is equal to the preset height h. When the bifurcation stops, the binary tree is regarded as an isolated tree.

S6: Execute S3-S5 in a loop until all sub-sample sets are executed. At this time, there are m isolated trees in total, forming an isolation forest with a capacity of m;

S7: Count the average path length of any element in the isolation forest and calculate the outlier score;

The average path length of a tree is calculated as:

in, is the average path length when the number of samples Ψ is given; /> Here x represents/> is the Euler constant, its value is 0.5772156649;

Abnormality score calculation:

Where h(x) is the depth of the node retrieved by the sample point x in the isolated tree; E(h(x)) is the expected value of h(x) of all isolated trees.

Optionally, the step of determining whether the current batch of CTD data is abnormal according to the outlier target CTD data and issuing abnormal alarm information when it is determined that the current batch of CTD data is abnormal includes:

Determine whether the number of all outlier target CTD data exceeds a preset outlier number threshold;

If the number of all outlier CTD data exceeds the preset outlier number threshold, the current batch of CTD data is determined to be abnormal and an abnormal alarm message is issued.

Optionally, the method further comprises:

Generate a visualization chart using the CTD data of the water area to intuitively display the trend and change of the CTD data; the visualization chart includes a trend chart, a bar chart and a pie chart;

Statistical analysis is performed on the CTD data of the waters to obtain index data to reflect the characteristics and laws of the CTD data; the index data includes mean value, variance, and standard deviation.

The embodiment of the present invention further discloses a CTD data quality monitoring device, the device comprising:

The acquisition module is used to collect CTD data of the water area, and the CTD data includes temperature, salinity and depth data, biological environment information, and other auxiliary parameter data;

A correlation analysis and selection module, used to analyze the correlation between the biological environment information, the other auxiliary parameter data and the temperature, salinity and depth data respectively, and select target CTD data according to the correlation to construct a target CTD data set;

An abnormality judgment module, used for performing abnormality judgment on the target CTD data by using an isolation forest algorithm;

The abnormal alarm module is used to determine whether the current batch of CTD data is abnormal based on the outlier target CTD data and issue an abnormal alarm message when it is determined that the current batch of CTD data is abnormal.

Optionally, the temperature-salinity-depth data include conductivity, temperature, and pressure, and the correlation analysis and selection module includes:

A preprocessing submodule, used for de-dimensionalizing and normalizing the CTD data to obtain preprocessed CTD data;

The correlation analysis arrangement submodule is used to calculate the grey correlation coefficient between each CTD parameter feature in the preprocessed biological environment information and other auxiliary parameter data and each target reference item, and arrange all CTD parameter features in descending order of the grey correlation coefficient to obtain three groups of CTD parameter feature ordered sequences;

A selection submodule is used to select a preset number of CTD parameter features that are ranked top from each group of CTD parameter feature ordered sequences;

The target CTD data set construction submodule is used to use conductivity, temperature, pressure and the CTD parameter features of the top ranking preset number as target CTD data to construct the target CTD data set.

Optionally, the abnormality judgment module includes:

An abnormality score calculation submodule, used to calculate the abnormality score of each target CTD data in the target CTD data set using an isolation forest algorithm;

The outlier target CTD data determination submodule is used to determine the outlier target CTD data according to the abnormality score of each target CTD data and a preset abnormality score threshold or an average abnormality score threshold.

Optionally, the abnormality score calculation submodule includes:

A definition unit, used to assume that the total number of elements in the target CTD data set is n (n≥5000), define the number of isolated trees t (1≤t≤n), the average depth h, the maximum depth H, and the capacity k of the root node subsample set;

A sub-sample set construction unit is used to extract a specified number of elements from the target CTD data set in a purely random manner by sampling with replacement multiple times to construct m sub-sample sets (a1, a2...ai...am);

The root node construction unit is used to select a subsample set ai as the root node of a tree and randomly select a CTD parameter feature P;

A binary tree construction unit is used to perform binary splitting on the tree for a single value q of the CTD parameter feature, traverse all elements of the subsample set ai, and when any record value r of the CTD parameter feature P is less than or equal to the single value q, the element is placed in the left child node of the tree, otherwise it is placed in the right child node;

An isolated tree construction unit is used to recursively construct a left child node and a right child node using a binary tree construction unit to construct a binary tree. The stopping condition is that any element in the array ai is isolated or the height of the tree is equal to a preset height h. When the bifurcation stops, the binary tree is regarded as constituting an isolated tree.

A loop execution unit is used to loop execute the root node construction unit, the binary tree construction unit, and the isolated tree construction unit until all sub-sample sets are executed. At this time, a total of m isolated trees are included, forming an isolated forest with a capacity of m;

Anomaly score calculation unit, used to count the average path length of any element in the isolation forest and calculate the anomaly score;

The average path length of a tree is calculated as:

in, is the average path length when the number of samples Ψ is given; /> Here x represents/> is the Euler constant, its value is 0.5772156649;

Abnormality score calculation:

Where h(x) is the depth of the node retrieved by the sample point x in the isolated tree; E(h(x)) is the expected value of h(x) of all isolated trees.

Optionally, the abnormal alarm information module includes:

The quantity threshold judgment submodule is used to judge whether the quantity of all outlier target CTD data exceeds the preset outlier quantity threshold;

The abnormal alarm submodule is used to determine that the batch of CTD data is abnormal and issue an abnormal alarm message if the number of all outlier CTD data exceeds a preset outlier number threshold.

Optionally, the device further comprises:

A visualization chart generation module, used to generate visualization charts using the CTD data of the water area to intuitively display the trends and changes of the CTD data; the visualization charts include trend charts, bar charts and pie charts;

The statistical analysis module is used to perform statistical analysis on the CTD data of the water area to obtain index data to reflect the characteristics and laws of the CTD data; the index data includes mean value, variance and standard deviation.

The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

The memory is used to store computer programs;

The processor is used to implement the CTD data quality monitoring method as described in the embodiment of the present invention when executing the program stored in the memory.

The embodiment of the present invention further discloses one or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute the CTD data quality monitoring method as described in the embodiment of the present invention.

The embodiments of the present invention include the following advantages:

The CTD data quality monitoring method of the embodiment of the present invention collects CTD data of the water area, and the CTD data includes temperature, salinity and depth data, biological environment information, and other auxiliary parameter data; respectively analyzes the correlation between the biological environment information, other auxiliary parameter data and temperature, salinity and depth data, and selects target CTD data according to the correlation to construct a target CTD data set; uses an isolated forest algorithm to perform abnormal judgment on the target CTD data; judges whether the current batch of CTD data is abnormal based on the outlier target CTD data and issues abnormal alarm information when it is determined that the current batch of CTD data is abnormal. The method of the present invention performs abnormality detection and identification on massive and diversified CTD data. This data processing method can efficiently and accurately identify and process abnormal situations in CTD data, improve the reliability, accuracy and analysis efficiency of CTD data quality monitoring, promptly discover abnormal situations in the marine environment, and promptly alarm and notify researchers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of the steps of a CTD data quality monitoring method provided in an embodiment of the present invention;

FIG2 is a schematic diagram of a target CTD data set A provided in an embodiment of the present invention;

3 is a structural block diagram of a CTD data quality monitoring system provided in an embodiment of the present invention;

FIG4 is a structural block diagram of a CTD data quality monitoring device provided in an embodiment of the present invention;

FIG5 is a block diagram of an electronic device provided in an embodiment of the present invention;

FIG. 6 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention.

Detailed ways

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

1, a flow chart of a method for monitoring CTD data quality provided in an embodiment of the present invention is shown, which may specifically include the following steps:

Step 101, collecting CTD data of the water area, wherein the CTD data includes temperature, salinity, depth data, biological environment information, and other auxiliary parameter data;

The CTD data quality monitoring method of the present invention performs quality monitoring on a large amount of diversified CTD data, detects and identifies abnormal data points, and promptly discovers abnormal conditions in the marine environment. In an embodiment of the present invention, a CTD sensor can be used to collect CTD data of a water area. The CTD sensor is an instrument capable of measuring water area information in real time, and it can measure CTD data of a water area in seawater at different depths. Among them, the CTD data of the water area may include temperature, salinity, depth data, biological environment information, and other auxiliary parameter data. The temperature, salinity, depth data may be basic data temperature, conductivity, and pressure. The biological environment information may include dissolved oxygen, chlorophyll, oxidation-reduction potential (ORP), photosynthetically active radiation (PAR), pH, turbidity, transmittance, voltage range, and other effective information. Other auxiliary parameter data may include pump performance parameters, monitoring time, and the like.

Step 102, respectively analyzing the correlation between the biological environment information, the other auxiliary parameter data and the temperature, salinity and depth data, and selecting target CTD data according to the correlation to construct a target CTD data set;

Under different latitudes, seasons, and ocean current conditions, the influence weights of CTD parameter features such as water temperature, salinity, depth, biological environment information, and other auxiliary parameter data are different. Therefore, a dimensionality reduction operation can be performed to increase the proportion of high-impact features and increase the efficiency and effectiveness of anomaly detection calculations.

In an embodiment of the present invention, the original data can be reduced in dimension by a feature correlation analysis method. Specifically, the correlation between the biological environment information and the temperature, salinity and depth data, as well as the correlation between other auxiliary parameter data and the temperature, salinity and depth data can be analyzed respectively. Then, according to the correlation between the biological environment information and the temperature, salinity and depth data, as well as the correlation between other auxiliary parameter data and the temperature, salinity and depth data, the target CTD data is selected to construct a target CTD data set, thereby reducing the dimension of the CTD parameter features that need to be judged, improving the calculation efficiency, and further improving the accuracy of judging the abnormal values of the CTD data.

In one embodiment of the present invention, the temperature-salinity-depth data includes conductivity, temperature, and pressure. The steps of analyzing the correlation between the biological environment information, the other auxiliary parameter data, and the temperature-salinity-depth data, and selecting target CTD data according to the correlation to construct a target CTD data set include:

De-dimensionalizing and normalizing the CTD data to obtain pre-processed CTD data;

Taking conductivity, temperature and pressure as target reference items, the grey correlation coefficient between each CTD parameter feature in the preprocessed biological environment information and other auxiliary parameter data and each target reference item is calculated, and all CTD parameter features are arranged in descending order according to the grey correlation coefficient, and three groups of CTD parameter feature ordered sequences are obtained;

Selecting a preset number of CTD parameter features with top rankings from each group of CTD parameter feature ordered sequences;

The target CTD data set is formed by using conductivity, temperature, pressure and the preset number of CTD parameter features ranked at the top as target CTD data.

After collecting CTD data, the CTD data can be preprocessed, such as de-dimensionalization and normalization, so that the preprocessed CTD data becomes a data form more suitable for processing by the isolation forest algorithm, thereby promoting the accuracy of the isolation forest algorithm in judging the abnormal value of the CTD data. Then, conductivity, temperature, and pressure can be used as target reference items, and the gray correlation coefficient of each CTD parameter feature in the preprocessed biological environment information and other auxiliary parameter data and each target reference item is calculated, and all CTD parameter features are arranged in order from high to low according to the gray correlation coefficient, and three groups of CTD parameter feature ordered sequences are obtained, and then a preset number of CTD parameter features with a high ranking are selected from each group of CTD parameter feature ordered sequences, and conductivity, temperature, pressure, and a preset number of CTD parameter features with a high ranking are used as target CTD data to form a target CTD data set. Referring to Figure 2, a schematic diagram of a target CTD data set A provided in an embodiment of the present invention is shown.

For example, conductivity, temperature, and pressure (important target parameters) are taken as target reference items, and then the gray correlation coefficient between the remaining features and each reference item is calculated, and three groups of features are generated as ordered lists of elements after sorting, and then the first 6 features with the highest correlation coefficients are taken from each of the three ordered lists. After deduplication of the 18 extracted features, the original data array A corresponding to the deduplication features is obtained. Then the original data set is sorted, the zero values therein are converted to 0, the temperature, salt and depth data features are converted into a standard floating point format, and the temperature, salt and depth feature vectors within a certain time range are sequentially extracted into the same element of array A. The array A is regarded as the total sample set for this data quality control monitoring, and the total number of elements is n (n>=5000). The present invention does not limit the number of features selected from the ordered list. After sorting the influence of the features, the first 40% of the data can also be selected as the data sequence for subsequent processing, or the first 3 features with the highest correlation coefficients can be selected, and so on.

Among them, the calculation of the gray correlation coefficient is to calculate the average absolute difference between the gray value of a specific column (col) of each CTD parameter feature and the gray value of the target reference item (ref_gray). The specific code is:

feature_gray = np.abs(normalized_data[col]-ref_gray).mean().

Step 103, using an isolation forest algorithm to perform abnormality judgment on the target CTD data;

After obtaining the target CTD data set, the isolation forest algorithm can be used to perform abnormality judgment on each target CTD data in the target CTD data set in order to determine the outlier target CTD data. The algorithm has the characteristics of strong adaptability and has no requirements on the dimension and linear characteristics of the monitoring data. It works best for CTD data with strong nonlinearity and high dimension.

In one embodiment of the present invention, the step of using the isolation forest algorithm to perform abnormality judgment on the target CTD data includes:

Using an isolation forest algorithm to calculate the abnormality score of each target CTD data in the target CTD data set;

The outlier target CTD data is determined according to the abnormality score of each target CTD data and a preset abnormality score threshold or an average abnormality score threshold.

The isolation forest algorithm is used to judge the abnormality of the target CTD data to determine whether the target CTD data is outliers. Specifically, the abnormality score of each target CTD data in the target CTD data set is calculated first, and then the target CTD data is determined to be outliers according to the abnormality score of each target CTD data and the preset abnormality score threshold or the average abnormality score threshold. Among them, the preset abnormality score threshold can be selected and set according to different application scenarios, such as marine environmental protection, marine meteorological warning, marine resource development and other different application scenarios.

In one embodiment of the present invention, the step of using the isolation forest algorithm to calculate the abnormality score of each target CTD data in the target CTD data set includes:

S1, assuming that the total number of elements in the target CTD data set is n (n ≥ 5000), define the number of isolated trees t (1 ≤ t ≤ n), the average depth h, the maximum depth H, and the capacity k of the root node subsample set;

S2, sampling with replacement multiple times from the target CTD data set, extracting a specified number of elements in a purely random manner to construct m sub-sample sets (a1, a2...ai...am);

S3, select a subsample set ai as the root node of a tree and randomly select a CTD parameter feature P;

S4, for a single value q of the CTD parameter feature, perform a binary split on the tree, traverse all elements of the subsample set ai, and when any record value r of the CTD parameter feature P is less than or equal to the single value q, place this element in the left child node of the tree, otherwise place it in the right child node;

S5, using step S4, recursively constructing the left child node and the right child node to build a binary tree. The stopping condition is that any element in the array ai is isolated or the height of the tree is equal to the preset height h. When the bifurcation stops, the binary tree is regarded as an isolated tree.

S6: Execute S3-S5 in a loop until all sub-sample sets are executed. At this time, there are m isolated trees in total, forming an isolation forest with a capacity of m;

S7: Count the average path length of any element in the isolation forest and calculate the outlier score;

The average path length of a tree is calculated as:

in, is the average value of the path length when the number of samples Ψ is given;/> Here x represents/> is the Euler constant, its value is 0.5772156649;

Abnormality score calculation:

Where h(x) is the depth of the node retrieved by the sample point x in the isolated tree; E(h(x0) is the expected value of h(x) of all isolated trees.

In an embodiment of the present invention, an isolation forest algorithm is used to calculate the abnormality score of the target CTD data. Specifically, S1: define the number of isolated trees t (1≤t≤n), the average depth h, the maximum depth H, and the capacity k of the root node sub-sample set; S2, then repeatedly sample with replacement from the target CTD data set, and extract a specified number of elements in a purely random manner to construct m sub-sample sets (a1, a2...ai...am); S3, then select a sub-sample set ai as the root node of a tree, and randomly select a CTD parameter feature P; S4, for a single value q of the CTD parameter feature, perform a binary split on the tree, and traverse the sub-sample set a For all elements in i, when any record value r of the CTD parameter feature P is less than or equal to a single value q, place this element in the left child node of the tree, otherwise place it in the right child node; S5, use S4 step to recursively construct the left child node and the right child node to build a binary tree. The stopping condition is that any element in the array ai is isolated or the height of the tree is equal to the preset height h. When the bifurcation stops, the binary tree is regarded as an isolated tree; S6: Loop through S3-S5 until all sub-sample sets are executed. At this time, there are m isolated trees in total, forming an isolated forest with a capacity of m; S7: Finally, count the average path length of any element in the isolated forest and calculate the anomaly score.

The average path length of a tree is calculated as:

in, is the average value of the path length when the number of samples Ψ is given;/> Here x represents is the Euler constant, its value is 0.5772156649;

Abnormality score calculation:

Where h(x) is the depth of the node retrieved by the sample point x in the isolated tree; E(h(x)) is the expected value of h(x) of all isolated trees.

The value range is (0,1]. The closer the value is to 1, the greater the probability of being considered an outlier. When E(h(x))→0,s→1; when/> s→0; when/> s→0.5. That is to say, the closer the abnormality score is to 1, the higher the possibility that the target CTD data is an outlier. If the abnormality scores of most target CTD data are close to 0.5, it means that there are no obvious outliers in this batch of CTD data.

Step 104 , judging whether the current batch of CTD data is abnormal based on the outlier target CTD data and issuing abnormal alarm information when it is determined that the current batch of CTD data is abnormal.

After determining which target CTD data are outliers, we can further determine whether there are any abnormalities in this batch of CTD data. If there are a large number of continuous outliers in this batch of data, it means that there are abnormalities in this batch of CTD data. Therefore, we can count the number of all outlier target CTD data and judge whether this batch of CTD data is abnormal based on the number of all outlier target CTD data. If the number of all outlier target CTD data exceeds a certain number, it can be determined that there are abnormalities in this batch of CTD data, and an abnormal alarm message can be issued to notify researchers to perform subsequent task processing in a timely manner. Otherwise, it is not necessary.

In one embodiment of the present invention, the step of determining whether the current batch of CTD data is abnormal based on the outlier target CTD data and issuing abnormal alarm information when it is determined that the current batch of CTD data is abnormal includes:

Determine whether the number of all outlier target CTD data exceeds a preset outlier number threshold;

If the number of all outlier CTD data exceeds the preset outlier number threshold, the current batch of CTD data is determined to be abnormal and an abnormal alarm message is issued.

In an embodiment of the present invention, a threshold value for the number of outliers can be preset, and whether the current batch of CTD data is abnormal can be determined by judging whether the number of all outlier target CTD data exceeds the preset threshold value for the number of outliers. If the number of all outlier CTD data exceeds the preset threshold value for the number of outliers, it can be shown that the current batch of CTD data is abnormal, so when it is determined that the number of all outlier CTD data exceeds the preset threshold value for the number of outliers, an abnormal alarm message can be issued. The threshold value for the number of outliers can be selected and set according to different application scenarios, such as for different application scenarios such as marine environmental protection, marine meteorological warning, and marine resource development.

In one embodiment of the present invention, the method further comprises:

Generate a visualization chart using the CTD data of the water area to intuitively display the trend and change of the CTD data; the visualization chart includes a trend chart, a bar chart and a pie chart;

Statistical analysis is performed on the CTD data of the waters to obtain index data to reflect the characteristics and laws of the CTD data; the index data includes mean value, variance, and standard deviation.

In an embodiment of the present invention, after collecting the CTD data of the water area, the CTD data of the water area can also be used to generate a visualization chart to intuitively display the trend and change of the CTD data. The visualization chart can include a trend chart, a bar chart, and a pie chart, which is not limited by the present invention. In addition, the CTD data of the water area can also be statistically analyzed to obtain index data to reflect the characteristics and laws of the CTD data. The index data can include the mean value, variance, and standard deviation, which is not limited by the present invention.

The technical effects of the CTD data quality monitoring method of the present invention are as follows:

1. Efficient and accurate:

The time complexity is low, and after the data threshold judgment and data dimension reduction process, the server computing load is effectively reduced.

2. Strong adaptability:

There are no necessary data features other than CTD, and no specific requirements for sensor models. The algorithm has the characteristics of strong adaptability, and has no requirements for the dimension and linear characteristics of the monitoring data. It works best for CTD data with strong nonlinearity and high dimension.

3. High precision:

The use of advanced ocean sensors and data processing technology can achieve high-precision collection and processing of multi-dimensional ocean detection data, and improve the reliability and accuracy of subsequent tasks.

4. Wide monitoring range:

It can be widely used in various marine environments and fields, such as marine environmental protection, marine meteorological warning, marine resource development, etc., and has stronger adaptability to a higher number of CTD sensors and a larger monitoring range.

5. Deal with problems in a timely manner:

Regular monitoring and notification of researchers in real time through a variety of means allows for more timely handling of environmental anomalies or device disconnections.

It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

3, a structural block diagram of a CTD data quality monitoring system provided in an embodiment of the present invention is shown, which may specifically include:

(1)CTD sensor

The CTD sensor is an instrument that can measure water area information in real time. It can collect basic data such as temperature, conductivity, depth (pressure, digiquartz [db]; temperature, ITS-90, degC; conductivity, mS/cm.....), biological environment data (oxygen, SBE 43 [mg/l]; fluorescence, Seapoint; turbudity, Seapoint [FTU]......), auxiliary parameters (pump status, time, Elapsed (Julian days)......), etc. in seawater at different depths.

(2) Data acquisition module

The data acquisition module is responsible for transmitting the data collected by the CTD sensor to the data storage module in real time for subsequent data analysis and anomaly detection.

(3) Data storage module

The data storage module is responsible for storing the original data collected by the CTD sensor and the data processed by the data processing module to non-volatile media for future data analysis and research.

(4) Data processing module

The data processing module of the present invention performs feature simplification, data cleaning, analysis, monitoring and anomaly detection on the data in the data storage module. First, the data processing module performs missing value and range detection on the original data to determine the online status of the sensor service. When the service is online, the original data is reduced in dimension by the feature correlation analysis method, and the isolation forest algorithm is used to screen and mark the abnormal points of the original CTD data.

When a preset abnormal situation is detected, such as the service is offline or a large number of abnormal values appear, the data processing module will process the message through the data service module and notify the alarm module.

(5)Data service layer module

The data service module of the present invention mainly includes three functions:

1) Handle exception messages;

2) Basic data statistical analysis functions (mean value, variance, standard deviation and other indicators);

3) Visual chart function (trend chart, bar chart, pie chart, etc.);

The data service layer module of the present invention can determine the notification type transmitted by the data processing module and call the corresponding method according to the specific situation. When a large number of continuous abnormal values appear, the data service layer module will notify the alarm module through the network interface so that the latter can promptly push the alarm information to the alarm device, WeChat public account or independent application. At the same time, the data service layer module can also use the chart library to generate a visual web page for abnormal point trends, quantities and other information, so that users can more intuitively understand the abnormal situation of the data. The design of this data service layer module can effectively improve the efficiency of data management and analysis, and ensure the integrity and accuracy of the data.

(6) Alarm module

Process alarm messages and send alarm information of abnormal situations to researchers through WeChat interface, SMS interface and other methods to take corresponding measures.

4, a structural block diagram of a CTD data quality monitoring device provided in an embodiment of the present invention is shown, which may specifically include the following modules:

The acquisition module 401 is used to collect CTD data of the water area, wherein the CTD data includes temperature, salinity, depth data, biological environment information, and other auxiliary parameter data;

A correlation analysis and selection module 402 is used to analyze the correlation between the biological environment information, the other auxiliary parameter data and the temperature, salinity and depth data respectively, and select target CTD data according to the correlation to construct a target CTD data set;

An abnormality judgment module 403 is used to use an isolation forest algorithm to perform abnormality judgment on the target CTD data;

The abnormality alarm module 404 is used to determine whether the current batch of CTD data is abnormal based on the outlier target CTD data and issue abnormality alarm information when it is determined that the current batch of CTD data is abnormal.

Optionally, the temperature-salinity-depth data include conductivity, temperature, and pressure, and the correlation analysis and selection module includes:

A preprocessing submodule, used for de-dimensionalizing and normalizing the CTD data to obtain preprocessed CTD data;

The correlation analysis arrangement submodule is used to calculate the grey correlation coefficient between each CTD parameter feature in the preprocessed biological environment information and other auxiliary parameter data and each target reference item, and arrange all CTD parameter features in descending order of the grey correlation coefficient to obtain three groups of CTD parameter feature ordered sequences;

A selection submodule is used to select a preset number of CTD parameter features that are ranked top from each group of CTD parameter feature ordered sequences;

The target CTD data set construction submodule is used to use conductivity, temperature, pressure and the CTD parameter features of the top ranking preset number as target CTD data to construct the target CTD data set.

Optionally, the abnormality judgment module includes:

An abnormality score calculation submodule, used to calculate the abnormality score of each target CTD data in the target CTD data set using an isolation forest algorithm;

The outlier target CTD data determination submodule is used to determine the outlier target CTD data according to the abnormality score of each target CTD data and a preset abnormality score threshold or an average abnormality score threshold.

Optionally, the abnormality score calculation submodule includes:

A definition unit, used to assume that the total number of elements in the target CTD data set is n (n≥5000), define the number of isolated trees t (1≤t≤n), the average depth h, the maximum depth H, and the capacity k of the root node subsample set;

A sub-sample set construction unit is used to extract a specified number of elements from the target CTD data set in a purely random manner by sampling with replacement multiple times to construct m sub-sample sets (a1, a2...ai...am);

The root node construction unit is used to select a subsample set ai as the root node of a tree and randomly select a CTD parameter feature P;

A binary tree construction unit is used to perform binary splitting on the tree for a single value q of the CTD parameter feature, traverse all elements of the subsample set ai, and when any record value r of the CTD parameter feature P is less than or equal to the single value q, the element is placed in the left child node of the tree, otherwise it is placed in the right child node;

An isolated tree construction unit is used to recursively construct a left child node and a right child node using a binary tree construction unit to construct a binary tree. The stopping condition is that any element in the array ai is isolated or the height of the tree is equal to a preset height h. When the bifurcation stops, the binary tree is regarded as constituting an isolated tree.

A loop execution unit is used to loop execute the root node construction unit, the binary tree construction unit, and the isolated tree construction unit until all sub-sample sets are executed. At this time, a total of m isolated trees are included, forming an isolated forest with a capacity of m;

Anomaly score calculation unit, used to count the average path length of any element in the isolation forest and calculate the anomaly score;

The average path length of a tree is calculated as:

in, is the average value of the path length when the number of samples Ψ is given;/> Here x represents/> is the Euler constant, its value is 0.5772156649;

Abnormality score calculation:

Where h(x) is the depth of the node retrieved by the sample point x in the isolated tree; E(h(x)) is the expected value of h(x) of all isolated trees.

Optionally, the abnormal alarm information module includes:

The quantity threshold judgment submodule is used to judge whether the quantity of all outlier target CTD data exceeds the preset outlier quantity threshold;

The abnormal alarm submodule is used to determine that the batch of CTD data is abnormal and issue an abnormal alarm message if the number of all outlier CTD data exceeds a preset outlier number threshold.

Optionally, the device further comprises:

A visualization chart generation module, used to generate visualization charts using the CTD data of the water area to intuitively display the trends and changes of the CTD data; the visualization charts include trend charts, bar charts and pie charts;

The statistical analysis module is used to perform statistical analysis on the CTD data of the water area to obtain index data to reflect the characteristics and laws of the CTD data; the index data includes mean value, variance and standard deviation.

As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

In addition, an embodiment of the present invention further provides an electronic device, as shown in FIG5 , including a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

Memory 503, used for storing computer programs;

The processor 501 is used to implement the CTD data quality monitoring method described in the above embodiment when executing the program stored in the memory 503.

The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above terminal and other devices.

The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

As shown in FIG. 6 , in another embodiment provided by the present invention, a computer-readable storage medium 601 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the CTD data quality monitoring method described in the above embodiment.

In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the CTD data quality monitoring method described in the above embodiment.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof.

It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (10)

1. A CTD data quality monitoring method, characterized in that the method comprises: Collect CTD data of water areas, including temperature, salinity, depth, biological environment information, and other auxiliary parameter data; Respectively analyzing the correlation between the biological environment information, the other auxiliary parameter data and the temperature, salinity and depth data, and selecting target CTD data according to the correlation to construct a target CTD data set; An isolation forest algorithm is used to judge abnormality of the target CTD data; It determines whether the current batch of CTD data is abnormal based on the outlier target CTD data and issues an abnormal alarm message when it is determined that the current batch of CTD data is abnormal. 2. The method according to claim 1 is characterized in that the temperature-salinity-depth data include conductivity, temperature, and pressure, and the steps of analyzing the correlation between the biological environment information, the other auxiliary parameter data and the temperature-salinity-depth data and selecting target CTD data according to the correlation to construct a target CTD data set include: De-dimensionalizing and normalizing the CTD data to obtain pre-processed CTD data; Taking conductivity, temperature and pressure as target reference items, the grey correlation coefficient between each CTD parameter feature in the preprocessed biological environment information and other auxiliary parameter data and each target reference item is calculated, and all CTD parameter features are arranged in descending order according to the grey correlation coefficient, and three groups of CTD parameter feature ordered sequences are obtained; Selecting a preset number of CTD parameter features with top rankings from each group of CTD parameter feature ordered sequences; The target CTD data set is formed by using conductivity, temperature, pressure and the preset number of CTD parameter features ranked at the top as target CTD data. 3. The method according to claim 1, characterized in that the step of using the isolation forest algorithm to perform abnormality judgment on the target CTD data comprises: Using an isolation forest algorithm to calculate the abnormality score of each target CTD data in the target CTD data set; The outlier target CTD data is determined according to the abnormality score of each target CTD data and a preset abnormality score threshold or an average abnormality score threshold. 4. The method according to claim 3, characterized in that the step of using the isolation forest algorithm to calculate the abnormality score of each target CTD data in the target CTD data set comprises: S1, assuming that the total number of elements in the target CTD data set is n (n ≥ 5000), define the number of isolated trees t (1 ≤ t ≤ n), the average depth h, the maximum depth H, and the capacity k of the root node subsample set; S2, sampling with replacement multiple times from the target CTD data set, extracting a specified number of elements in a purely random manner to construct m sub-sample sets (a1, a2...ai...am); S3, select a subsample set ai as the root node of a tree and randomly select a CTD parameter feature P; S4, for a single value q of the CTD parameter feature, perform a binary split on the tree, traverse all elements of the subsample set ai, and when any record value r of the CTD parameter feature P is less than or equal to the single value q, place this element in the left child node of the tree, otherwise place it in the right child node; S5, using step S4, recursively constructing the left child node and the right child node to build a binary tree. The stopping condition is that any element in the array ai is isolated or the height of the tree is equal to the preset height h. When the bifurcation stops, the binary tree is regarded as an isolated tree. S6: Execute S3-S5 in a loop until all sub-sample sets are executed. At this time, there are m isolated trees in total, forming an isolation forest with a capacity of m; S7: Count the average path length of any element in the isolation forest and calculate the outlier score; The average path length of a tree is calculated as: in, is the average path length when the number of samples Ψ is given; /> Here x represents is the Euler constant, its value is 0.5772156649; Abnormality score calculation: Where h(x) is the depth of the node retrieved by the sample point x in the isolated tree; E(h(x)) is the expected value of h(x) of all isolated trees. 5. The method according to claim 3 is characterized in that the step of judging whether the current batch of CTD data is abnormal according to the outlier target CTD data and issuing abnormal alarm information when it is determined that the current batch of CTD data is abnormal comprises: Determine whether the number of all outlier target CTD data exceeds a preset outlier number threshold; If the number of all outlier CTD data exceeds the preset outlier number threshold, the current batch of CTD data is determined to be abnormal and an abnormal alarm message is issued. 6. The method according to claim 1, further comprising: Generate a visualization chart using the CTD data of the water area to intuitively display the trend and change of the CTD data; the visualization chart includes a trend chart, a bar chart and a pie chart; Statistical analysis is performed on the CTD data of the waters to obtain index data to reflect the characteristics and laws of the CTD data; the index data includes mean value, variance, and standard deviation. 7. A CTD data quality monitoring device, characterized in that the device comprises: The acquisition module is used to collect CTD data of the water area, and the CTD data includes temperature, salinity and depth data, biological environment information, and other auxiliary parameter data; A correlation analysis and selection module, used to analyze the correlation between the biological environment information, the other auxiliary parameter data and the temperature, salinity and depth data respectively, and select target CTD data according to the correlation to construct a target CTD data set; An abnormality judgment module, used for performing abnormality judgment on the target CTD data by using an isolation forest algorithm; The abnormal alarm module is used to determine whether the current batch of CTD data is abnormal based on the outlier target CTD data and issue an abnormal alarm message when it is determined that the current batch of CTD data is abnormal. 8. The device according to claim 7, characterized in that the temperature-salinity-depth data include conductivity, temperature, and pressure, and the correlation analysis and selection module includes: A preprocessing submodule, used for de-dimensionalizing and normalizing the CTD data to obtain preprocessed CTD data; The correlation analysis arrangement submodule is used to calculate the grey correlation coefficient between each CTD parameter feature in the preprocessed biological environment information and other auxiliary parameter data and each target reference item, and arrange all CTD parameter features in descending order of the grey correlation coefficient to obtain three groups of CTD parameter feature ordered sequences; A selection submodule is used to select a preset number of CTD parameter features that are ranked top from each group of CTD parameter feature ordered sequences; The target CTD data set construction submodule is used to use conductivity, temperature, pressure and the CTD parameter features of the top ranking preset number as target CTD data to construct the target CTD data set. 9. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the CTD data quality monitoring method according to any one of claims 1 to 6 when executing the program stored in the memory. 10. One or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to perform the CTD data quality monitoring method according to any one of claims 1 to 6.