CN115587130A - Intelligent power utilization platform system - Google Patents

Abstract

The invention relates to an intelligent power consumption platform system, comprising: a communication module, used for large-scale concurrent user access; an intelligent analysis module, used for intelligently analyzing massive power consumption data; and a data parallel computing module, used for realizing The big data mining of the parallel mining mechanism; and the data storage module for storing the data processed by the intelligent analysis module and the data parallel computing module. The system is conducive to efficient data mining of massive electricity consumption data.

Description

An intelligent power consumption platform system

technical field

The invention belongs to the field of big data computing, and in particular relates to an intelligent power utilization platform system.

Background technique

The current electricity consumption information platform relies on the traditional oracle database system, which can efficiently realize data entry, query, statistics and other functions; the communication access technology uses the traditional blocking IO method. However, existing databases cannot discover the relationships and rules existing in massive power consumption data, nor can they predict future development trends based on existing data, and the traditional system architecture cannot meet the core demands of high throughput and high concurrency under the new situation.

Contents of the invention

The purpose of the present invention is to provide an intelligent power consumption platform system, which is conducive to efficient data mining of massive power consumption data.

In order to achieve the above purpose, the technical solution adopted by the present invention is: an intelligent power utilization platform system, comprising:

Communication module for large-scale concurrent user access;

Intelligent analysis module for intelligent analysis of massive power consumption data;

A data parallel computing module for realizing big data mining based on a data parallel mining mechanism; and

The data storage module is used for storing the data processed by the intelligent analysis module and the data parallel computing module.

Further, the process of intelligent data analysis by the intelligent analysis module includes data preprocessing, data classification analysis, data cluster analysis, data correlation analysis and data integration. Extract valuable data to improve the management and service level of the entire system.

Further, the application of intelligent data analysis by the intelligent analysis module in actual business scenarios includes:

1) Establish a holiday power allocation plan; build a holiday load forecasting model based on massive user power consumption data, and predict the transfer power of the holiday start by establishing the automation coefficient and the company's holiday start probability prediction function, and further iterate to get the power company to fill the valley to achieve the goal The preferential electricity price should be set when the value is set. By setting the preferential electricity price, some industrial enterprises are encouraged to start work on holidays, in order to achieve the effect of filling the valley;

2) Establish a prediction model for the number of monthly faults in the power grid; by dividing the number of distribution transformer faults, using a high-order Markov prediction model and calculating the state transition matrix to predict the number of distribution transformer faults, the more detailed the classification, the higher the prediction efficiency;

3) Analyze the power consumption behavior of big data users; by extracting the characteristic points of the daily load characteristic curve, using the clustering SOM neural network, in the Matlab environment, perform clustering simulation on the data, output visual clustering results, and analyze the clustering results Conduct analysis to provide reference and guidance for power grid enterprises to optimize power marketing services;

4) Carry out the electricity stealing behavior analysis based on the collected data; use the ensemble empirical mode decomposition method to eliminate the fluctuation factors, and obtain the load residual trend items of a certain user and its class in any two identical time periods, and calculate the two The correlation coefficient is used to judge whether the user's electricity consumption behavior is abnormal.

Further, the data parallel computing module is realized by using a data parallel mining mechanism based on the MapReduce model, including a data extraction component, a data distribution component, a data classification component, a data aggregation component and a data persistence component, and the data classification component includes a Map component , the data aggregation component includes a Reduce component, which is used to comprehensively and quickly process the electricity consumption data, so as to realize the real-time performance of data processing.

Further, the data parallel mining mechanism classifies and aggregates the electricity consumption data according to the electricity consumption data query, combined with the specific business, and finally summarizes and persists the result data into a file or database; the data extraction component manages the data according to the data loading The loading rules configured by the controller, according to the terminal type and power supply unit conditions, the original data to be analyzed are preloaded by the database in parallel and put into the memory cache; the data loading manager monitors the status of each data set in real time, and if a certain data is found If the set fails, it will notify the corresponding original data to be reloaded; the data distribution component sorts the extracted original data, and distributes them to the Map processing component in a balanced manner; In this process, the same data set will be divided iteratively for multiple times to finally generate intermediate results; the Reduce component will perform multiple iterative calculations and merge operations on the intermediate results according to the configured business analysis rules to generate the final results, which are also stored in In the memory cache; the data persistence component is responsible for writing the final result to the database.

Further, the communication module adopts high concurrent communication technology, and its implementation method is as follows: when a connection sends a request to the server, the server regards the connection request as a request "event" and assigns the "event" to the corresponding Function processing; then put this processing function into a thread to execute, and return the thread after execution, so that one thread can process multiple events asynchronously.

Further, the communication module adopts the collaborative architecture of Zookeeper multi-service nodes, and its implementation method is: generally, the read and write requests of the client end are scheduled by the general leader, and the general leader can know whether each server is active through the scheduling mechanism, and perform The overall scheduling of load information is used to achieve load balancing of each server; the overall leader is obtained through the Zookeeper election algorithm, so even if the overall leader fails, the entire system will elect another overall leader to do the scheduling service.

Further, the communication module adopts multi-threading technology for a single service node, and its implementation method is as follows: a thread pool includes four parts: thread pool manager, worker thread, task interface, and task queue; one program can create multiple service nodes Node, a service node is a process, each process can create multiple thread pools, each thread pool is composed of multiple independent threads; thread is the smallest resource unit of the system during program running; in the communication of large-scale concurrent users , need to handle a large number of client communication connections, each connection is processed by a thread.

Compared with the prior art, the present invention has the following beneficial effects: the present invention realizes intelligent query, supervision, statistics, analysis and prediction of marketing management through data mining of massive electricity consumption data, enabling the management to timely and comprehensively understand The completion of various marketing and service indicators and business development of each unit provide decision-making basis for the company's operation and management. In actual business scenarios, the load forecasting model can be used to obtain the load conditions of enterprises and residents at different time points, so as to set different electricity prices and use economic leverage to cut peaks and fill valleys; accurate prediction of distribution transformer line failures can enable power companies to grasp future lines in a timely manner The number of failures that may occur, so we can formulate corresponding measures to minimize power outages, invest more energy to eliminate equipment defects, scientifically arrange maintenance plans, and at the same time do a good job of publicity to the society and customers to minimize social adverse effects; and Electricity companies can also scientifically arrange production plans based on the prediction results provided by power companies to reduce losses caused by power outages. Therefore, accurate prediction of the number of distribution transformer failures can also bring certain economic benefits; users can be extracted from massive load data The daily load characteristic curve and in-depth data mining analysis can accurately and timely obtain the user's electricity consumption behavior characteristics and improve the power marketing service level. The system adopted by the data parallel mining mechanism has the characteristics of scalability and stability. Using the MapReduce mode, the degree of data parallelization is high, which can greatly improve the operating efficiency of the program, and to some extent, it can complete what the serial algorithm cannot complete. It provides powerful technical support for processing massive power data.

Description of drawings

Fig. 1 is a schematic diagram of the system architecture of an embodiment of the present invention;

Fig. 2 is the architectural diagram of ZooKeeper multi-service node communication module in the embodiment of the present invention;

Fig. 3 is a technical model diagram of thread pool in the embodiment of the present invention;

FIG. 4 is an architecture diagram of a large-scale concurrent user access solution in an embodiment of the present invention;

Fig. 5 is a data flow diagram of intelligent analysis of massive power consumption data in an embodiment of the present invention;

Fig. 6 is a MapReduce work flowchart in the embodiment of the present invention;

7 is a functional diagram of a data parallel mining mechanism in an embodiment of the present invention;

Fig. 8 is a flowchart of the data parallel mining mechanism in the embodiment of the present invention.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

As shown in FIG. 1 , this embodiment provides an intelligent power utilization platform system, including a communication module, an intelligent analysis module, a data parallel computing module and a data storage module. Among them, the communication module is mainly responsible for the access of large-scale concurrent users. The intelligent analysis module is used to realize the intelligent analysis of massive power consumption data. The data parallel computing module mainly implements big data mining based on the data parallel mining mechanism. The data storage module is mainly used to store the data processed by the intelligent analysis module and the data parallel computing module. It adopts a general-purpose relational database Oracle, which is conducive to system compatibility and programmability.

The communication module applies high concurrent communication technology. The bottleneck of traditional blocking IO lies in the inability to handle too many connections. The present invention solves this problem by using a high-concurrency communication technology based on non-blocking IO. The specific implementation method is: when a connection sends a request to the server, the server regards the connection request as a request "event", and assigns the "event" to the corresponding function for processing; then puts the processing function into the thread Execute, and return the thread after execution, so that one thread can process multiple events asynchronously. This asynchronous non-blocking communication technology improves communication efficiency.

The communication module adopts the collaborative architecture of Zookeeper multi-service nodes as shown in FIG. 2 . The implementation method is as follows: general client-side read and write requests are scheduled by the general leader, and the general leader can know whether each server is active or not through the scheduling mechanism, and perform general scheduling of load information to achieve load balancing of each server; The leader is obtained through the Zookeeper election algorithm, so even if the general leader fails, the entire system will elect another general leader to provide scheduling services.

The communication module applies the multi-thread technology for a single service node as shown in FIG. 3 . The implementation method is as follows: a thread pool includes four parts: thread pool manager, worker thread, task interface, and task queue; a program can create multiple service nodes, a service node is a process, and each process can create multiple threads Pool, each thread pool is composed of multiple independent threads; thread is the smallest resource unit of the system during program running; in the communication of large-scale concurrent users, it is necessary to handle a large number of client communication connections, and each connection goes through a thread Processing, thereby improving the efficiency of program operation.

The large-scale user concurrent access solution adopts thread pool and non-blocking asynchronous communication technology, and the system function is designed in components, which greatly improves the concurrent user access capability of a single business node. The distributed mechanism is implemented with Zookeeper technology, which supports linear expansion of business nodes. The large-scale user access solution adopts a component architecture, as shown in Figure 4, each server includes a thread pool and a communication management module; the thread pool provides communication threads and business threads, the communication thread only handles communication connections, and the business thread is only responsible for business Processing, the communication thread is separated from the business thread independently, and the thread is managed by the thread pool. The abnormal execution of a single thread will not affect the execution of other threads. This thread pool can improve the stability of a single node; using non-blocking asynchronous communication mode, The receiving thread and the response thread can be separated, and the throughput of the communication channel can be greatly improved; multi-node cluster deployment is adopted, and each server has a communication management module. Through zookeeper technology, load balancing of multiple nodes can be achieved. When a server In the event of a failure, the client terminal on this server will be automatically dispatched to other servers that are working normally, thus effectively eliminating the failure of a single node. It is also possible to linearly increase the server to redistribute clients and reduce the number of client connections of a single service node.

Concrete working principle of the present invention is as follows:

This intelligent electricity consumption platform designs a method suitable for intelligent analysis of electricity consumption data and a parallel data mining mechanism by studying the massive information mining of intelligent electricity consumption and the large-scale concurrent user access system, and greatly improves the communication efficiency of the platform.

As shown in Figure 5, the process of intelligent data analysis by the intelligent analysis module includes data preprocessing, data classification analysis, data cluster analysis, data correlation analysis, and data integration. Valuable data can be extracted from electrical data characteristics to improve the management and service level of the entire system. The application of the intelligent analysis module to perform data intelligent analysis in actual business scenarios includes:

1) Establish a holiday power allocation plan; build a holiday load forecasting model based on massive user power consumption data, and predict the transfer power of the holiday start by establishing the automation coefficient and the company's holiday start probability prediction function, and further iterate to get the power company to fill the valley to achieve the goal The preferential electricity price should be set when the value is set. By setting the preferential electricity price, some industrial enterprises are encouraged to start work on holidays, in order to achieve the effect of filling the valley.

2) Establish a prediction model for the number of monthly faults in the power grid; by dividing the number of distribution transformer faults, using a high-order Markov prediction model and calculating the state transition matrix to predict the number of distribution transformer faults, the more detailed the classification, the higher the prediction efficiency.

3) Analyze the power consumption behavior of big data users; by extracting the characteristic points of the daily load characteristic curve, using the clustering SOM neural network, in the Matlab environment, perform clustering simulation on the data, output visual clustering results, and analyze the clustering results Through the analysis, it is verified that the daily load characteristic curves of public transformer users extracted by the SOM neural network clustering algorithm can better show the differences in the behavior characteristics of different types of public transformer users, and have a good clustering effect. Provide reference and guidance for enterprises to optimize power marketing services.

4) Carry out the electricity stealing behavior analysis based on the collected data; use the EEMD method (Ensemble Empirical Mode Decomposition Method) to remove the fluctuation factors, and obtain the load residual trend item of a certain user and its class in any two identical time periods , and calculate the correlation coefficient between the two to judge whether the user's electricity consumption behavior is abnormal.

The data parallel computing module is realized by using a data parallel mining mechanism based on the MapReduce model. Figure 6 shows the MapReduce workflow, which is a distributed programming model for massive data processing in large-scale groups. From the perspective of the implementation of the MapReduce framework, the MapReduce program has two components: one implements the Mapper, and the other implements the Reducer, which can finally convert large-scale data into smaller summary data.

As shown in Figures 7 and 8, the data parallel mining mechanism includes a data extraction component, a data distribution component, a data classification component, a data aggregation component and a data persistence component, the data classification component includes a Map component, and the data aggregation component Including the Reduce component, which is used to comprehensively and quickly process power consumption data to realize real-time data processing.

The data parallel mining mechanism classifies and aggregates the electricity consumption data according to the electricity consumption data query, combined with the specific business, and finally summarizes and persists the result data into a file or database; The configured loading rules, based on the terminal type, power supply unit and other conditions, the original data to be analyzed is preloaded by the Oracle database in parallel and put into the memory cache; the data loading manager monitors the status of each data set in real time, and if a certain data is found If the set fails, it will notify the corresponding original data to be reloaded; the data distribution component sorts the extracted original data, and distributes them to the Map processing component in a balanced manner; Segmentation, this process will perform multiple iterative segmentations on the same data set, and finally generate intermediate results; the Reduce component performs multiple iterative calculations, merges, and other operations on the intermediate results according to the configured business analysis rules to generate the final results, which are also saved In the memory cache area; the data persistence component is responsible for writing the final result to the Oracle database.

In practical business applications, the present invention designs a parallel computing method for abnormal detection of massive power data based on MapReduce. First, the method of initTableMapperJob is used to read the collection point data of each block on Hbase; after that, each computing node The Map task on the node analyzes the input collection point data. The Map task first uses the exception discovery algorithm to analyze the data of the collection points allocated on this node, and monitors the generation of abnormal data. When the data of a collection point is found to have one or more basic characteristics of abnormal data, it can be confirmed that the collection point is abnormal. Then, according to the basic characteristics of the abnormal data, the corresponding type of feature analysis algorithm is executed to further analyze all data abnormal characteristics of the abnormal data. Because MapReduce runs on a multi-node cluster, parallel computing is performed between different Maps at the same time. Finally, the Reduce task is executed, and the detected abnormal data and its feature set are input into the HBase table in the Reduce stage, thus completing the discovery process of abnormal data.

The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention to other forms. Any skilled person who is familiar with this profession may use the technical content disclosed above to change or modify the equivalent of equivalent changes. Example. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still belong to the protection scope of the technical solution of the present invention.

Claims (8)

1. An intelligent electricity platform system, comprising:

the communication module is used for accessing large-scale concurrent users;

the intelligent analysis module is used for intelligently analyzing mass power utilization data;

the data parallel computing module is used for realizing big data mining based on a data parallel mining mechanism; and

and the data storage module is used for storing the data processed by the intelligent analysis module and the data parallel computing module.

2. The intelligent power utilization platform system according to claim 1, wherein the intelligent analysis module performs intelligent data analysis including data preprocessing, data classification analysis, data clustering analysis, data association analysis and data integration, and extracts valuable data according to power utilization data characteristics in view of actual service scenarios to improve the management and service level of the whole system.

3. The intelligent power platform system according to claim 2, wherein the application of the intelligent analysis module to intelligent analysis of data in an actual service scenario comprises:

1) Establishing a festival and holiday power utilization allocation scheme; establishing a holiday load prediction model based on massive user power consumption data, predicting the transferred electric quantity of holiday start operation by establishing an automation number and an enterprise holiday start operation probability prediction function, further iterating to obtain a preferential power price which is set for the power enterprise to reach a target valley filling value, and encouraging part of industrial enterprises to start operation on holidays by setting the preferential power price so as to achieve the effect of valley filling;

2) Establishing a power grid monthly fault frequency prediction model; the distribution transformer fault frequency is predicted by dividing the distribution transformer fault frequency grade and utilizing a high-order Markov prediction model and calculating a state transition matrix, and the more detailed the grade division is, the higher the prediction efficiency is;

3) Analyzing the power utilization behavior of a big data user; by extracting characteristic points of a daily load characteristic curve, performing clustering simulation on data in a Matlab environment by adopting a clustering SOM neural network, outputting a visual clustering result, and analyzing the clustering result, so as to provide reference and guidance for power grid enterprise optimization power marketing service;

4) Carrying out electricity stealing behavior analysis based on the collected data; after fluctuation factors are eliminated by using a set empirical mode decomposition method, a load residual trend item of a certain user and the class to which the user belongs in any two same time periods is obtained, and a correlation coefficient of the two items is calculated to judge whether the power utilization behavior of the user has an abnormal condition.

4. The intelligent power utilization platform system according to claim 1, wherein the data parallel computing module is implemented by a data parallel mining mechanism based on a MapReduce model, and comprises a data extraction component, a data distribution component, a data classification component, a data aggregation component and a data persistence component, wherein the data classification component comprises a Map component, and the data aggregation component comprises a Reduce component and is used for comprehensively and rapidly processing power utilization data to achieve real-time performance of data processing.

5. The intelligent power utilization platform system according to claim 4, wherein the data parallel mining mechanism classifies and aggregates power utilization data according to queried power utilization data in combination with specific services, and finally summarizes and persists result data into a file or a database; the data extraction component is used for blocking and parallelly preloading original data to be analyzed by a database according to a loading rule configured by a data loading manager and terminal type and power supply unit conditions and putting the original data into a memory cache; the data loading manager monitors the state of each data set in real time, and if a certain data set is found to be invalid, the data loading manager informs to reload corresponding original data; the data distribution component sorts the extracted original data and distributes the sorted original data to the Map processing component in a balanced dispatching manner; the Map component performs smaller-granularity segmentation on the original data according to various configured business rules, and the process performs iterative segmentation on the same data set for multiple times to finally generate an intermediate result; the Reduce component performs repeated iterative computation and combination operation on the intermediate result according to the configured service analysis rule to generate a final result, and the final result is also stored in a memory cache region; the data persistence component is responsible for writing the final result to the database.

6. The intelligent power platform system according to claim 1, wherein the communication module employs a high-concurrency communication technology, and the implementation method is as follows: when a connection sends a request to a server, the server treats the connection request as a request 'event' and distributes the 'event' to the corresponding function processing; the processing function is then put into a thread for execution, and the thread is returned after execution is completed, so that one thread can asynchronously process a plurality of events.

7. The intelligent power platform system according to claim 1, wherein the communication module adopts a Zookeeper multi-service node cooperative architecture, and the implementation method thereof is as follows: generally, read-write requests of client ends are scheduled by a master leader, the master leader can know whether each server is active or not through a scheduling mechanism, and total scheduling of load information is carried out, so that load balance of each server is achieved; the total leader is obtained through a Zookeeper election algorithm, so that even if the total leader fails, the whole system can select another total leader for scheduling service.

8. The intelligent power utilization platform system according to claim 1, wherein the communication module adopts a multithreading technology for a single service node, and the realization method is as follows: one thread pool comprises a thread pool manager, a working thread, a task interface and a task queue; one program can create a plurality of service nodes, one service node is a process, each process can create a plurality of thread pools, and each thread pool is composed of a plurality of independent threads; the thread is the unit of the minimum resource of the system in the program operation; in the communication of large-scale concurrent users, communication connections of a large number of clients need to be processed, and each connection is processed through one thread.

Images