CN110597792A - Multi-level redundant data fusion method and device based on synchronous line loss data fusion - Google Patents

Abstract

The invention discloses a method and a device for fusing multistage redundant data based on synchronous line loss data fusion, wherein the method comprises the following steps: cleaning the power grid data by using a button tool; carrying out abnormal data identification, system clustering analysis and positive-negative correlation analysis on the cleaned power grid data to obtain an abnormal data discrimination result; and performing multi-level redundant data check correction according to the abnormal data discrimination result to finish the correction of the power grid equipment parameters, the calculation model, the topological data and the electric quantity data. According to the fusion method provided by the embodiment of the invention, multi-source data can be treated by utilizing multi-level redundant data fusion, the data quality is improved, and the use requirement is effectively met.

Description

Multi-level redundant data fusion method and device based on synchronous line loss data fusion

technical field

The invention relates to the technical field of power grids, in particular to a multi-level redundant data fusion method and device based on synchronous line loss data fusion.

Background technique

The simultaneous line loss system integrates six major business systems and three major platforms. Compared with the traditional line loss statistics, the simultaneous line loss not only requires data transmission, correlation and integration between different systems, but also needs to ensure the accuracy of the data. Therefore, abnormal data Identification and governance are critical.

Big data processing technology: use the Kettle tool to realize the extraction and conversion of big data. Kettle is an open source ETL tool that can run on Windows and Linux, and the data extraction is efficient and stable. This ETL toolset allows to manage data from different databases by providing a graphical user environment to describe what one wants to do.

Multi-level redundant data cleaning technology: Multi-level redundant data cleaning technology includes: the fusion redundant data cleaning method using sliding window and set theory, and the poor redundant data cleaning method that uses the idea of reference labels combined with light signal characteristics. In the multi-level redundant data cleaning of the power system using the mass data cleaning method, the data quality of the power system is improved.

Data redundancy processing technology: redundant data processing is mainly divided into three stages: data collection stage, data identification and comparison stage, and data integration stage. The data collection phase mainly refers to the system obtaining corresponding data from the outside, and storing and classifying the collected data for data identification; in the data identification and comparison phase, the system conducts repeatability discrimination according to the characteristics of the stored data, and adopts corresponding Algorithms are used to determine whether the collected data is unique or duplicate; the data integration stage refers to the processing of duplicate data by the system after data identification.

Cluster analysis: It is an exploratory analysis. In the process of classification, people do not need to give a classification standard in advance. Cluster analysis can start from sample data and automatically classify. Different methods used in cluster analysis often lead to different conclusions. When different researchers perform cluster analysis on the same set of data, the number of clusters obtained may not be consistent.

Correlation analysis: In regression and correlation analysis, the value of the dependent variable decreases (increases) as the value of the independent variable increases (decreases). In this case, the correlation coefficient between the dependent variable and the independent variable is negative, That is negative correlation. A positive correlation means that as the independent variable increases, the dependent variable also increases. Two variables change in the same direction. When one variable changes from large to small or from small to large, the other variable also changes from large to small or from small to large.

However, the existing synchronous line loss system integrates a variety of source-end power business systems and platforms, and the quality of data seriously affects the accuracy of line loss calculation, which needs to be improved urgently.

Contents of the invention

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

For this reason, an object of the present invention is to propose a multi-level redundant data fusion method based on synchronous line loss data fusion, which can use multi-level redundant data fusion to manage multi-source data, improve data quality, and effectively To meet the needs of use.

Another object of the present invention is to propose a multi-level redundant data fusion device based on synchronous line loss data fusion.

In order to achieve the above object, an embodiment of the present invention proposes a multi-level redundant data fusion method based on synchronous line loss data fusion, including the following steps: using the Kettle tool to clean the grid data; Perform abnormal data identification, system cluster analysis and positive and negative correlation analysis on the data to obtain abnormal data screening results; perform multi-level redundant data verification and correction according to the abnormal data screening results to complete power grid equipment parameters, calculation models, and topology data. and power data correction.

The multi-level redundant data fusion method based on synchronous line loss data fusion in the embodiment of the present invention cleans, analyzes and applies multi-level fused redundant data based on multi-level redundant data fusion, and realizes fast and automatic identification of power grid abnormal data And corrections, improve data quality, improve calculation accuracy, reduce the workload of business personnel for offline verification, and effectively meet the needs of use.

In addition, the multi-level redundant data fusion method based on synchronous line loss data fusion according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, the abnormal data identification, system cluster analysis and positive and negative correlation analysis of the cleaned multi-level fusion redundant numbers include: using big data processing methods to analyze the power grid Analyze the data to identify the first abnormal data; analyze the grid data by using the system cluster analysis method to identify the second abnormal data; analyze the grid data by using the positive and negative correlation analysis method to identify Get the third abnormal data.

Further, in an embodiment of the present invention, the method further includes: performing a transmission check on the data by using a cyclic redundancy check method.

Further, in one embodiment of the present invention, the cleaning of the multi-level fused redundant data using the Kettle tool further includes: using sliding windows and set theory to clean the fused redundant data; and/or according to reference The idea of labeling combined with light features of the signal is used to clean redundant data.

Further, in an embodiment of the present invention, the multi-level redundant data verification and correction according to the abnormal data screening results includes: using multi-level redundant data to perform the check and correction at each level of the main network, the distribution network, and the station area. Equipment parameters, power grid topology data, calculation models and collected data are verified and corrected.

In order to achieve the above object, another embodiment of the present invention proposes a multi-level redundant data fusion device based on synchronous line loss data fusion, including: a cleaning module, which is used to clean the grid data using Kettle tools; The selection module is used to perform abnormal data identification, system cluster analysis and positive and negative correlation analysis on the cleaned power grid data to obtain abnormal data screening results; the correction module is used to perform multi-level redundant data according to the abnormal data screening results Check and correct to complete the correction of grid equipment parameters, calculation models, topology data and power data.

The multi-level redundant data fusion device based on synchronous line loss data fusion in the embodiment of the present invention cleans, analyzes and applies multi-level fused redundant data based on multi-level redundant data fusion, and realizes fast and automatic identification of power grid abnormal data And corrections, improve data quality, improve calculation accuracy, reduce the workload of business personnel for offline verification, and effectively meet the needs of use.

In addition, the multi-level redundant data fusion device based on synchronous line loss data fusion according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

Further, in one embodiment of the present invention, the selection module includes: a first selection unit, configured to analyze the grid data by using a big data processing method to identify first abnormal data; a second selection unit , for analyzing the grid data by using a system clustering analysis method to identify the second abnormal data; the third selection unit is used for analyzing the grid data by using a positive and negative correlation analysis method to identify the second abnormal data Three abnormal data.

Further, in an embodiment of the present invention, it also includes: a verification module, configured to perform transmission verification on data by using a cyclic redundancy check method.

Further, in one embodiment of the present invention, the cleaning module is further used to clean redundant data fusion by using sliding window and set theory, and/or clean redundant data according to the idea of reference labels combined with light features of signals.

Furthermore, in one embodiment of the present invention, the correction module is further used to use multi-level redundant data to calibrate the main network, distribution network, equipment parameters at all levels in the station area, power grid topology data, calculation models and collected data. test correction.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

1 is a flowchart of a multi-level redundant data fusion method based on synchronous line loss data fusion according to an embodiment of the present invention;

2 is a flow chart of a multi-level redundant data fusion method based on synchronous line loss data fusion according to a specific embodiment of the present invention;

FIG. 3 is a schematic block diagram of a multi-level redundant data fusion device based on synchronous line loss data fusion according to an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

The multi-level redundant data fusion method and device based on synchronous line loss data fusion proposed according to the embodiments of the present invention will be described below with reference to the accompanying drawings. Level redundant data fusion method.

FIG. 1 is a flowchart of a multi-level redundant data fusion method based on synchronous line loss data fusion according to an embodiment of the present invention.

As shown in Figure 1, the multi-level redundant data fusion method based on synchronous line loss data fusion includes the following steps:

In step S101, the Kettle tool is used to clean the grid data.

It can be understood that, as shown in Figure 2, for redundant data cleaning, use the big data Kettle tool to carry out multi-level redundant data cleaning of equipment parameters, topology relations, and operating data on the multi-source fusion data of the line loss system in the same period, that is, Use the big data tool Kettle to collect grid data, including substations, transformers, switches, compensation equipment, transmission lines, distribution lines, distribution transformers, stations, users, operating data, graphic data, etc.

Further, in one embodiment of the present invention, using the Kettle tool to clean the multi-level fused redundant data further includes: using sliding windows and set theory to clean the fused redundant data; and/or combining signals according to the idea of reference labels Mild features are used for redundant data cleaning.

It can be understood that, as shown in Figure 2, for multi-level redundant data cleaning, the multi-level redundant data cleaning method of the power grid is realized by using the multi-source fusion redundant data cleaning method and the multi-source cross redundant data cleaning method.

Specifically, the multi-level redundant data cleaning technology: ①Using the fusion redundant data cleaning method of sliding window and set theory. First, initialize the fusion label list with the unique identifier of the original data as the key field. The nodes in the fusion label list are tuple cache queues composed of original data tuples; secondly, try to receive data streams from multiple readers and parse them. The unique identification attribute of the tuple, according to the unique identification of the tuple, determine the storage location of the tuple cache queue of the reader in the fusion tag list, and insert the original data group of the new tuple into the tuple cache queue to ensure that the time is ascending; then, according to the time The size of the sliding window and the noise threshold, perform noise filtering and single-point redundancy elimination on each tuple cache queue; the last random algorithm selects the unique identifier of any tuple in the group The unique identifier of the tuple that cleaned the data last night, Perform data cleaning. ② A poor redundant data cleaning method based on the idea of reference labels combined with the mild features of the signal. First read the affiliation relationship between the configuration file binding parameter label and the reader, create an initial cross-label list, and try to update the cross-label list from the raw data stream pushed by the reader; secondly, detect the cross-label list Whether there is a tag to be arbitrated by the reference tag, if not, filter the expired cross information tuple in the cross tag list to maintain a reasonable memory usage, if yes, detect the tuple cache in the tag group according to the set sliding window Then, within the time sliding window, the signal strength of the tag to be arbitrated corresponds to the reader that has monitored the tag; finally, the Euclidean distance between the tag to be arbitrated and the signal strength vector of the reader-writer reference tag is calculated. Using the method of minimizing the relative position similarity to arbitrate out the true reader-writer with poor redundant data, after the arbitration, the cross-redundant data is eliminated through mutual exclusion processing.

In step S102, abnormal data identification, system cluster analysis and positive and negative correlation analysis are performed on the cleaned grid data to obtain abnormal data screening results.

Among them, in one embodiment of the present invention, abnormal data identification, system cluster analysis and positive and negative correlation analysis are performed on the cleaned multi-level fusion redundant numbers, including: using big data processing methods to analyze power grid data to The first abnormal data is identified; the grid data is analyzed by using the system cluster analysis method to identify the second abnormal data; the positive and negative correlation analysis method is used to analyze the grid data to identify the third abnormal data.

Specifically, as shown in Figure 2, for abnormal data identification, the simultaneous line loss system integrates six major business systems and three major platforms, and uses big data processing technology to identify abnormal data such as missing and jumping; for cluster analysis, Use system clustering analysis method to analyze power grid data, analyze and screen abnormal data; positive and negative correlation analysis, use positive and negative correlation analysis method to analyze power grid data, analyze and screen abnormal data.

For example, use big data processing technology to identify and compare the collected data, make repeated judgments based on the characteristics of the data, use corresponding algorithms to determine the uniqueness of the collected data, and process redundant data.

Further, for the synchronous line loss system data that integrates six major business systems and three major platforms, use big data processing technology to analyze substations, transformers, switches, compensation equipment, transmission lines, distribution lines, distribution transformers, station areas, and users Screen out abnormal data such as missing equipment data, failure to collect operational data, and jumps in operational data.

Furthermore, the system cluster analysis is a traditional statistical cluster analysis method to determine that the cluster mainly includes the parameters of the same type of equipment and the operating data of the same equipment. First of all, the parameters of the same type of equipment, such as S9-250/10 transformers, are affected by the manufacturer, and parameters such as no-load loss, load loss, short-circuit voltage percentage, no-load current percentage, resistance, and reactance are distributed within a certain range. Abnormal jumps are abnormal data; secondly, there is a certain continuity in the operation data of the same equipment, and the operation data fluctuations can be accurately located through clustering.

Further, correlation analysis uses positive and negative correlation analysis to analyze power grid data, and analyzes and screens out abnormal data. Correlation is the uncertain dependence between variables. Correlation analysis is a commonly used statistical method to study the uncertain dependence between variables and their closeness. It is usually measured by correlation coefficient. Correlation coefficient is a description Statistics of the degree and direction of correlation between variables, usually expressed by r, and satisfying -1≤r≤1, given the data of variables ( _xi , y _i ), i=1, 2,..., n, sample The correlation coefficient of the data is calculated as follows:

Correlation analysis is the analysis of the degree of closeness between variables. Its task is to make a realistic interpretation of whether there is an inevitable connection between the variables, the degree of closeness of the connection, and the direction of change, and to measure the degree of closeness of their connection, and to test its effectiveness. In the process of multi-source data fusion, there is a certain amount of abnormal data. These data are intuitively far away from other data. Their existence reduces the closeness between variables. Using correlation analysis, accurately locate and correct multi-source data. abnormal data to guide the development of data governance.

In step S103, multi-level redundant data verification and correction is performed according to the abnormal data screening results, so as to complete the correction of grid equipment parameters, calculation models, topology data and power data.

Further, in one embodiment of the present invention, multi-level redundant data verification and correction is performed according to the abnormal data screening results, including: using multi-level redundant data to check the main network, distribution network, station area equipment parameters at all levels, power grid The topology data, calculation model and collected data are verified and corrected.

It can be understood that, as shown in Figure 2, for multi-level redundant data verification and correction, on the basis of redundant data cleaning and abnormal data analysis and screening, multi-level redundant data is used to verify and correct abnormal data, including Equipment parameters, topology data, calculation models, power data, etc., to improve data quality. In other words, use multi-level redundant data to verify and correct the main network, distribution network, and station area equipment parameters at all levels, power grid topology data, calculation models, and collected data, so as to realize rapid and automatic identification and correction of abnormal data in the power grid, and improve data quality. quality and improve calculation accuracy.

Further, in an embodiment of the present invention, the method of the embodiment of the present invention further includes: performing transmission verification on the data by using a cyclic redundancy check method.

In the embodiment of the present invention, a cyclic redundancy check (CRC) technology is used to implement a data transmission verification function and improve data transmission quality. Specifically, a cyclic redundancy check (CRC) technology is used to implement a data transmission verification function. The data sender and the receiver agree on a production polynomial G(X) in advance, and the generator polynomial is used as the divisor polynomial. The data bit sequence to be sent is used as the coefficient of a polynomial F(X), which is the dividend polynomial, and the coefficient of the remainder polynomial R(X) obtained by dividing the dividend polynomial F(X) by the divisor polynomial G(X) is the CRC code . The CRC code is added to the binary data bit sequence to be sent to form the send code. After receiving the sent code, the receiver also regards it as a coefficient sequence of a polynomial, and divides the polynomial with the same generator polynomial. If the remainder is zero, the transmission is error-free; otherwise, there is an error in the transmission.

To sum up, as shown in Figure 2, on the basis of multi-source data fusion of line loss in the same period, the big data Kettle tool is used to clean, analyze, and correct the multi-level fusion redundant data, and realize a method of utilizing multi-level redundant data. Multi-source data governance is a method to improve data quality. Among them, data cleaning includes: using Kettle for data collection, data identification and comparison, and data integration to realize redundant data cleaning of multi-source data; after redundant data cleaning, carry out Abnormal data identification, system cluster analysis, positive and negative correlation analysis, using big data processing technology to realize abnormal screening of massive data. According to the abnormal data screening results, the multi-level redundant data verification and correction function is used to realize the correction of power grid equipment parameters, calculation models, topology data, and power data, and fully utilize the multi-level redundant data to provide a multi-source fusion data. The governance method realizes the rapid and automatic identification and correction function of abnormal data of the power grid, improves the data quality, and improves the calculation accuracy.

According to the multi-level redundant data fusion method based on synchronous line loss data fusion according to the embodiment of the present invention, based on the multi-level redundant data fusion, the multi-level fused redundant data is cleaned, analyzed, and applied, and the abnormal data of the power grid is quickly and automatically realized. Identify and correct, improve data quality, improve calculation accuracy, reduce the workload of business personnel for offline verification, and effectively meet usage needs.

Next, a multi-level redundant data fusion device based on synchronous line loss data fusion proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.

Fig. 3 is a schematic block diagram of a multi-level redundant data fusion device based on synchronous line loss data fusion according to an embodiment of the present invention.

As shown in FIG. 3 , the multi-level redundant data fusion device 10 based on synchronous line loss data fusion includes: a cleaning module 100 , a selection module 200 and a correction module 300 .

Wherein, the cleaning module 100 is used to clean the grid data by using the Kettle tool. The selection module 200 is used to perform abnormal data identification, system cluster analysis and positive and negative correlation analysis on the cleaned power grid data to obtain abnormal data screening results. The correction module 300 is used to perform multi-level redundant data verification and correction according to abnormal data screening results, so as to complete the correction of grid equipment parameters, calculation models, topology data and power data. The fusion device 10 according to the embodiment of the present invention can use multi-level redundant data fusion to manage multi-source data, improve data quality, and effectively meet usage requirements.

Further, in an embodiment of the present invention, the selection module 200 includes: a first selection unit, a second selection unit and a third selection unit. Wherein, the first selection unit is used to analyze the power grid data using a big data processing method to identify the first abnormal data. The second selection unit is used to analyze the grid data by using the system cluster analysis method to identify the second abnormal data. The third selection unit is used to analyze the power grid data by positive and negative correlation analysis to identify the third abnormal data.

Further, in an embodiment of the present invention, the fusion device 10 of the embodiment of the present invention further includes: a verification module. Wherein, the verification module is used for performing transmission verification on data by using a cyclic redundancy check method.

Further, in an embodiment of the present invention, the cleaning module 100 is further used to clean redundant data fusion by using sliding window and set theory, and/or clean redundant data according to the idea of reference labels combined with light features of signals.

Further, in one embodiment of the present invention, the correction module 300 is further used to verify the main network, distribution network, equipment parameters at all levels in the station area, power grid topology data, calculation models and collected data by using multi-level redundant data fix.

It should be noted that the foregoing explanations for the embodiment of the multi-level redundant data fusion method based on synchronous line loss data fusion are also applicable to the multi-level redundant data fusion device based on synchronous line loss data fusion in this embodiment, here No longer.

According to the multi-level redundant data fusion device based on synchronous line loss data fusion according to the embodiment of the present invention, based on the multi-level redundant data fusion, the multi-level fused redundant data is cleaned, analyzed, and applied, and the abnormal data of the power grid is quickly and automatically realized. Identify and correct, improve data quality, improve calculation accuracy, reduce the workload of business personnel for offline verification, and effectively meet usage needs.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (10)

1. A multi-level redundant data fusion method based on synchronous line loss data fusion, is characterized in that, comprises the following steps: Use the Kettle tool to clean the grid data; Perform abnormal data identification, system cluster analysis and positive and negative correlation analysis on the cleaned power grid data to obtain abnormal data screening results; and Perform multi-level redundant data verification and correction according to the abnormal data screening results to complete the correction of grid equipment parameters, calculation models, topology data and power data. 2. The method according to claim 1, wherein said performing abnormal data identification, system cluster analysis and positive and negative correlation analysis on the multi-level fusion redundant numbers after cleaning includes: Analyzing the grid data by using a big data processing method to identify the first abnormal data; Analyzing the grid data by using a systematic clustering analysis method to identify the second abnormal data; The grid data is analyzed using a positive and negative correlation analysis method to identify the third abnormal data. 3. The method according to claim 1, further comprising: The data is transmitted and verified by using the cyclic redundancy check method. 4. The method according to claim 1, wherein said utilizing the Kettle tool to clean said multi-level fusion redundant data further comprises: Fusion redundant data cleaning using sliding windows and set theory; and/or Redundant data cleaning is carried out according to the idea of reference labels combined with light features of signals. 5. The method according to any one of claims 1-4, wherein the performing multi-level redundant data verification and correction according to the abnormal data screening results includes: Use multi-level redundant data to verify and correct the main network, distribution network, and equipment parameters at all levels in the station area, power grid topology data, calculation models, and collected data. 6. A multi-level redundant data fusion device based on synchronous line loss data fusion, characterized in that it comprises: A cleaning module, configured to clean the grid data using the Kettle tool; The selection module is used to perform abnormal data identification, system cluster analysis and positive and negative correlation analysis on the cleaned power grid data to obtain abnormal data screening results; and The correction module is used to perform multi-level redundant data verification and correction according to the abnormal data screening results, so as to complete the correction of grid equipment parameters, calculation models, topology data and power data. 7. The device according to claim 6, wherein the selection module comprises: The first selection unit is configured to use a big data processing method to analyze the grid data to identify the first abnormal data; The second selection unit is configured to analyze the grid data by using a system cluster analysis method to identify second abnormal data; The third selection unit is configured to analyze the grid data by using a positive and negative correlation analysis method to identify third abnormal data. 8. The device according to claim 6, further comprising: The verification module is used for performing transmission verification on data by using a cyclic redundancy check method. 9. The device according to claim 6, characterized in that, the cleaning module is further used to perform fusion redundant data cleaning by using sliding window and set theory, and/or perform redundancy according to the idea of reference labels combined with light features of signals Data cleaning. 10. The device according to any one of claims 6-9, characterized in that the correction module is further used to use multi-level redundant data to modify the equipment parameters and power grid topology data of the main network, distribution network, and station area , Calculation model and collected data for verification and correction.