CN116991925B - Method for collecting data at high speed and storing mass data rapidly - Google Patents
Method for collecting data at high speed and storing mass data rapidly Download PDFInfo
- Publication number
- CN116991925B CN116991925B CN202310935692.8A CN202310935692A CN116991925B CN 116991925 B CN116991925 B CN 116991925B CN 202310935692 A CN202310935692 A CN 202310935692A CN 116991925 B CN116991925 B CN 116991925B
- Authority
- CN
- China
- Prior art keywords
- data
- acquisition
- time
- channel
- data packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
- G06F11/0754—Error or fault detection not based on redundancy by exceeding limits
- G06F11/0757—Error or fault detection not based on redundancy by exceeding limits by exceeding a time limit, i.e. time-out, e.g. watchdogs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- Maintenance And Management Of Digital Transmission (AREA)
Abstract
The invention provides a method for collecting data at high speed and storing mass data rapidly, and belongs to the technical field of data collection and storage. When the method is used for collecting data at high speed, the data packets of the collecting measuring points with different collecting channel types are processed according to a pre-stored data packet processing method of the collecting measuring points; generating an access equipment acquisition data packet command queue according to the processed acquisition measurement point data packet; establishing acquisition channels of different acquisition channel types, and establishing access connection between the different acquisition channel types and corresponding equipment; and sending the commands in the command queue of the access device acquisition data packet to the corresponding devices one by one. The invention can obviously improve the data acquisition speed, the time granularity can be accurate to 0.01 second, and the rapid storage of mass data can be realized.
Description
Technical Field
The invention relates to the technical field of data acquisition and storage, in particular to a method for acquiring data at high speed and storing mass data rapidly.
Background
Along with the development of modern technology, the data processing of a computer is widely applied to various industries, so that the scenes of high-speed data acquisition and mass data storage are more and more, and how to realize the high-speed data acquisition and how to store the mass data is an important problem to be solved by current research personnel.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a method for collecting data at high speed and storing mass data at high speed, wherein during the data collection, the data packets of the collection measurement points with different collection channel types are processed according to a pre-stored processing method of the data packets of the collection measurement points; generating an access equipment acquisition data packet command queue according to the processed acquisition measurement point data packet; establishing acquisition channels of different acquisition channel types, and establishing access connection between the different acquisition channel types and corresponding equipment; and sending the commands in the command queue of the access device acquisition data packet to the corresponding devices one by one. The invention can obviously improve the data acquisition speed, the time granularity can be accurate to 0.01 second, and the rapid storage of mass data can be realized.
The invention provides a method for collecting data at high speed, which is characterized by comprising the following steps:
processing the data packets of the acquisition measuring points of different acquisition channel types according to a pre-stored processing method of the data packets of the acquisition measuring points;
generating an access equipment acquisition data packet command queue according to the processed acquisition measurement point data packet;
establishing acquisition channels of different acquisition channel types, and establishing access connection between the different acquisition channel types and corresponding equipment;
transmitting commands in a command queue of the data packet acquired by the access device to corresponding devices one by one;
receiving a data packet responded by the equipment, and if the data packet is successfully received and analyzed, considering the data packet as normal data; otherwise, the exception is considered to occur, and exception processing is carried out.
Preferably, the processing of the measurement point data packets of different collection channel types according to a pre-stored collection measurement point data packet processing method specifically comprises the following steps:
combining the same acquisition channel type into one channel according to the acquisition channel type;
generating a measuring point address segment according to the same channel collecting measuring point address;
and combining the address field of the acquisition measuring point and the same data type into an acquisition measuring point data packet.
Preferably, the addresses of the collecting measuring points are recombined into a continuous address segment of the collecting measuring points according to the address ordering of the collecting measuring points of the same channel.
Preferably, merging the address field of the measuring point and the same data type into one data packet for collecting the measuring point is specifically as follows: and combining the continuous address segments of the measuring points with the same data type into a data packet, combining the access start address of the data packet and the access data length into a data packet, reducing the data access times and improving the data transmission efficiency.
Preferably, the preset number of data packets of the acquisition measuring points are combined into an acquisition data access packet, and the acquisition data access packet is used as an access device acquisition data packet command.
Preferably, the access device acquisition data packet command queue is composed of all access device acquisition data packet commands.
Preferably, the receiving device receives the data packet responded, if the data packet is successfully received, analyzes the data according to the data packet definition rule, and if the data packet is successfully analyzed, respectively assigning analysis contents and receiving time to the measuring point value and the collecting time of the corresponding collecting measuring point.
Preferably, the data acquisition of the devices with different baud rates is supported in one acquisition channel, which specifically comprises:
setting baud rates of different devices in the same acquisition channel when the acquisition channels of the different types of devices are established;
ordering the baud rates, and grouping the devices with the same baud rate together for regrouping;
the device is accessed each time to judge the baud rate, if the baud rate of the device acquisition channel is the same as that of the device acquisition channel accessed last time, the system does not switch the baud rate, and directly sends a data packet to acquire data; if the baud rate of the equipment acquisition channel of this time is different from that of the equipment acquisition channel accessed last time, the system closes the baud rate of the equipment acquisition channel accessed last time and opens the baud rate of the equipment acquisition channel accessed this time.
Preferably, normal data is not received within a stipulated time, and the group of data is regarded as abnormal data; if the normal data is not received for more than the set times, the channel is regarded as an abnormal channel, the normal acquisition channel is removed, the normal acquisition channel is put into a watchdog function for taking over, and during taking over, the watchdog function continuously tries to access the data packet or connect the channel according to a preset interval time until the data can be received and analyzed successfully, and the normal acquisition channel is put into.
Preferably, for an abnormal channel, the watchdog function makes a channel connection attempt within a prescribed time, and if the connection is unsuccessful, will wait for the next connection attempt; if the connection is successful, the data packet access is carried out within a specified time, if the data packet access can be normally received and the data can be normally analyzed, the data packet is considered to be normal and is put into the original acquisition channel, otherwise, the next data packet access attempt is continued to be waited; and for abnormal data, the watchdog function performs data packet access within a specified time, if the access data packet can be normally received and the data can be normally analyzed, the data packet is considered to be normal and is put into an original acquisition channel, otherwise, the data packet access attempt is continued to wait for the next time.
The invention also provides a method for rapidly storing mass data, which stores the data collected by any of the methods for collecting data at high speed, and comprises the following steps:
dividing the collected data into three types of real-time data, historical data and service logic processing data, wherein the service logic processing data is new data obtained after logic judgment and calculation;
injecting data of the same data type into the same data block;
the data blocks of different types are subjected to table division, library division and time division storage, and the data of the same type are stored in a time sequence data table of the same type; establishing a data sub-table and a data sub-database according to requirements;
and combining the same-time data with the same type at regular intervals according to the time-sharing data content.
Preferably, the real-time data sub-table is written into a Redis remote data dictionary;
writing the history data sub-table into a MySQL relational database;
the service logic processing data is divided into early warning data, alarm data, function calculation data and logic conversion data according to service requirements, the real-time service logic processing data is written into Redis, the calculated early warning data, alarm data, function calculation data and logic conversion data are written into MySQL, the alarm early warning is a table, and the function calculation and logic conversion are a table.
Preferably, when establishing the data sub-table, each device establishes a device time sequence data table according to different time; and classifying the service data after logic processing into a data table of the similar equipment according to the service type.
Preferably, the database is divided into a basic database and a history database, wherein the basic database comprises data for supporting the operation of the system, and the history database comprises collected data stored in real time.
Preferably, the collected data are classified according to data types and the occurrence time of the data are temporarily stored in a temporary cache file, the data in the temporary cache file are processed in different modes respectively, the temporary cache file comprises historical data and business logic data, and the data in the temporary cache file are combined and classified into the same data table according to the occurrence time and the preset time period as a unit.
Preferably, if the data in the temporary cache file is data of historical occurrence collected in real time, the data is classified and stored according to the time of year, month and day, and if the data in the temporary cache file is business logic data, the data is classified and stored according to the time of year, month and day.
Compared with the prior art, the invention has the following beneficial effects:
(1) The invention combines the data packets of the preset number of acquisition measuring points into one acquisition data access packet as an access equipment acquisition data packet command, reduces the access times, increases the data of the access packet and obviously improves the acquisition speed;
(2) The invention sorts the baud rates, gathers the equipment with the same baud rate together and regroups the equipment, reduces the switching times of the baud rates, and improves the acquisition speed;
(3) According to the invention, abnormal data and an abnormal channel are taken over by a watchdog function, normal data receiving and analyzing are not influenced, and the acquisition speed can be improved;
(4) The invention separates the data storage and the business processing functions, firstly temporarily caches the data for processing, and then stores the data, thereby greatly improving the storage efficiency of mass data;
(5) According to the invention, the real-time data acquired at high speed is written into the Redis remote data dictionary, so that real-time sharing with interface data is realized; the historical data is written into a MySQL relational database and used as data analysis query data, and the data is massive and needs to be quickly stored in the database; writing different data sub-tables into Redis and MySQL according to service requirements for other functional data;
(6) The data is stored in the temporary file firstly, the data is stored according to the occurrence time, the file content is the process data, the file content data is read again to be displayed again to be the multi-disc data, and the running process of the equipment can be deduced according to the data process;
(7) The invention adopts a high-speed acquisition mode, the time granularity of the acquired field data can be accurate to 0.01 second, and then the data is stored in the database by adopting the mass rapid storage method, and the data time granularity of the later analysis query can be accurate to 0.01 second.
Drawings
FIG. 1 is a flow chart of a method of collecting data at high speed in accordance with one embodiment of the present invention;
FIG. 2 is a flow chart of a method of fast mass data storage according to one embodiment of the invention.
Detailed Description
The following describes the present invention in detail.
The invention provides a method for collecting data at high speed, which is characterized by comprising the following steps:
processing the data packets of the acquisition measuring points of different acquisition channel types according to a pre-stored processing method of the data packets of the acquisition measuring points;
generating an access equipment acquisition data packet command queue according to the processed acquisition measurement point data packet;
establishing acquisition channels of different acquisition channel types, and establishing access connection between the different acquisition channel types and corresponding equipment;
transmitting commands in a command queue of the data packet acquired by the access device to corresponding devices one by one;
receiving a data packet responded by the equipment, and if the data packet is successfully received and analyzed, considering the data packet as normal data; otherwise, the exception is considered to occur, and exception processing is carried out.
According to a specific embodiment of the invention, the processing of the measurement point data packets of different collection channel types according to a pre-stored collection measurement point data packet processing method specifically comprises the following steps:
combining the same acquisition channel type into one channel according to the acquisition channel type;
generating a measuring point address segment according to the same channel collecting measuring point address;
and combining the address field of the acquisition measuring point and the same data type into an acquisition measuring point data packet.
According to a specific embodiment of the invention, the addresses of the acquisition measurement points are recombined into continuous address segments of the acquisition measurement points according to the address ordering of the acquisition measurement points of the same channel.
According to one embodiment of the invention, the merging of the station address field and the same data type into one collected station packet is specifically: and combining the continuous address segments of the measuring points with the same data type into a data packet, combining the access start address of the data packet and the access data length into a data packet, reducing the data access times and improving the data transmission efficiency.
According to a specific embodiment of the invention, a preset number of data packets of the acquisition measuring points are combined into an acquisition data access packet, and the acquisition data access packet is used as an access device acquisition data packet command.
According to one embodiment of the invention, the access device acquisition data packet command queue is composed of all access device acquisition data packet commands.
According to a specific embodiment of the invention, the receiving device receives the data packet responded, analyzes the data according to the data packet definition rule if the receiving is successful, and respectively assigns the analysis content and the receiving time to the measuring point value and the collecting time of the corresponding collecting measuring point if the analyzing is successful.
According to a specific embodiment of the invention, data acquisition is carried out by supporting different baud rate devices in one acquisition channel, and the method specifically comprises the following steps:
setting baud rates of different devices in the same acquisition channel when the acquisition channels of the different types of devices are established;
ordering the baud rates, and grouping the devices with the same baud rate together for regrouping;
the device is accessed each time to judge the baud rate, if the baud rate of the device acquisition channel is the same as that of the device acquisition channel accessed last time, the system does not switch the baud rate, and directly sends a data packet to acquire data; if the baud rate of the equipment acquisition channel of this time is different from that of the equipment acquisition channel accessed last time, the system closes the baud rate of the equipment acquisition channel accessed last time and opens the baud rate of the equipment acquisition channel accessed this time.
According to one embodiment of the present invention, normal data is not received within a predetermined time, and the group of data is regarded as abnormal data; if the normal data is not received for more than the set times, the channel is regarded as an abnormal channel, the normal acquisition channel is removed, the normal acquisition channel is put into a watchdog function for taking over, and during taking over, the watchdog function continuously tries to access the data packet or connect the channel according to a preset interval time until the data can be received and analyzed successfully, and the normal acquisition channel is put into.
According to one embodiment of the present invention, for an abnormal channel, the watchdog function makes a channel connection attempt within a prescribed time, and if the connection is unsuccessful, will wait for the next connection attempt; if the connection is successful, the data packet access is carried out within a specified time, if the data packet access can be normally received and the data can be normally analyzed, the data packet is considered to be normal and is put into the original acquisition channel, otherwise, the next data packet access attempt is continued to be waited; and for abnormal data, the watchdog function performs data packet access within a specified time, if the access data packet can be normally received and the data can be normally analyzed, the data packet is considered to be normal and is put into an original acquisition channel, otherwise, the data packet access attempt is continued to wait for the next time.
The invention also provides a method for rapidly storing mass data, which stores the data collected by any of the methods for collecting data at high speed, and comprises the following steps:
dividing the collected data into three types of real-time data, historical data and service logic processing data, wherein the service logic processing data is new data obtained after logic judgment and calculation;
injecting data of the same data type into the same data block;
the data blocks of different types are subjected to table division, library division and time division storage, and the data of the same type are stored in a time sequence data table of the same type; establishing a data sub-table and a data sub-database according to requirements;
and combining the same-time data with the same type at regular intervals according to the time-sharing data content.
According to one embodiment of the invention, a real-time data sub-table is written into a Redis remote data dictionary;
writing the history data sub-table into a MySQL relational database;
the service logic processing data is divided into early warning data, alarm data, function calculation data and logic conversion data according to service requirements, the real-time service logic processing data is written into Redis, the calculated early warning data, alarm data, function calculation data and logic conversion data are written into MySQL, the alarm early warning is a table, and the function calculation and logic conversion are a table.
According to one embodiment of the invention, when establishing the data sub-table, each device establishes a device time sequence data table according to different time; and classifying the service data after logic processing into a data table of the similar equipment according to the service type.
According to a specific embodiment of the invention, the database is divided into a base database and a history database, the base database comprising data supporting the operation of the system, the history database comprising collected data stored in real time.
According to a specific embodiment of the invention, collected data are classified according to data types and data occurrence time and are temporarily stored in a temporary cache file, the data in the temporary cache file are processed in different modes respectively, the temporary cache file comprises historical data and business logic data, and the data in the temporary cache file are combined and classified into the same data table according to the occurrence time and a preset time period as a unit.
According to one embodiment of the present invention, if the data in the temporary buffer file is data of history occurrence collected in real time, the data is classified and stored according to the time of year, month and day, and if the data in the temporary buffer file is business logic data, the data is classified and stored according to the time of year, month and day.
Example 1
According to one embodiment of the present invention, the high-speed data acquisition process includes the steps of:
processing the data packets of the acquisition measuring points of different acquisition channel types according to a pre-stored processing method of the data packets of the acquisition measuring points;
generating an access equipment acquisition data packet command queue according to the processed acquisition measurement point data packet;
establishing acquisition channels of different acquisition channel types, and establishing access connection between the different acquisition channel types and corresponding equipment;
transmitting commands in a command queue of the data packet acquired by the access device to corresponding devices one by one;
receiving a data packet responded by the equipment, and if the data packet is successfully received and analyzed, considering the data packet as normal data; otherwise, the exception is considered to occur, and exception processing is carried out.
Example 2
According to one embodiment of the present invention, this example differs from example 1 in that the station packet processing includes the following steps:
combining the same acquisition channel type into one channel according to the acquisition channel type;
according to the address ordering of the measurement points collected by the same channel, the address of the measurement points are recombined to generate a continuous address segment;
combining the address field of the acquisition measuring point and the same data type into an acquisition measuring point data packet;
and combining the 18 data packets of the acquisition measuring points into an acquisition data access packet serving as an access device acquisition data packet command, reducing the access times and increasing the data of the access packet.
Example 3
According to a specific embodiment of the present invention, the difference between this example and example 1 is that the different baud rate devices are rapidly collected in one collection channel, specifically implemented as follows:
establishing an equipment acquisition channel which is divided into PLC, modbus, TCP/IP and the like, wherein in the same Modbus acquisition channel, the equipment needs to set the baud rate, and the data cannot be normally acquired because the baud rate of the equipment cannot be unified;
firstly, sorting the baud rates of the acquisition equipment once, grouping the same baud rates together for regrouping, and reducing the baud rate switching times;
when processing different baud rates of the acquisition channel, each time the access device judges the baud rate, if the access device is the same as the last channel, the system does not switch the baud rate, and directly sends a data packet to acquire data. If the channel is different from the last channel, the system turns off the baud rate of the last channel and turns on the baud rate of the other channel again;
establishing a collecting channel receiving response fool-proof mechanism, regarding that normal data are not received within a stipulated time, regarding that the group of data are abnormal, regarding that the collecting channel is abnormal if the set times are exceeded or the normal data are not received, rejecting the normal collecting channel, and putting the collecting channel into a watchdog function for taking over;
the current channel is always high-speed data acquisition through eliminating the abnormal channel, the abnormal channel and the abnormal data block are uniformly processed by the watchdog takeover, and the current channel is put into the high-speed acquisition channel after the abnormal channel and the abnormal data block are normal.
Example 4
According to a specific embodiment of the present invention, the difference between this example and example 1 is that the abnormal data of a single collecting device does not affect the collecting speed of other devices, specifically implemented by the following manner:
establishing a collecting channel receiving response fool-proof mechanism, regarding that normal data are not received within a stipulated time, regarding that the group of data are abnormal, regarding that the collecting channel is abnormal if the set times are exceeded or the normal data are not received, rejecting the normal collecting channel, and putting the collecting channel into a watchdog function for taking over;
the current channel is always high-speed data acquisition through eliminating the abnormal channel, the abnormal channel and the abnormal data block are uniformly processed by the watchdog takeover, and the current channel is put into the high-speed acquisition channel after the abnormal channel and the abnormal data block are normal.
Example 5
According to one embodiment of the invention, a method for rapid mass data storage includes;
after the data is collected at high speed, the mass data needs to be quickly saved in a database, the saved data and the service processing function need to be separated to improve the saving efficiency, and the saved data is divided into three data of real-time data, historical data and service logic processing data;
real-time data: writing the data acquired at high speed into a Redis remote data dictionary to achieve real-time sharing with interface data;
historical data: writing the data acquired at high speed into a MySQL relational database as data analysis query data, wherein the data is massive and needs to be quickly stored in the database;
business logic data: the data collected at high speed is logically judged and calculated to obtain new data, and the new data can be divided into business data such as early warning data, alarm data, function calculation data, logic conversion data and the like according to business demands, and the business data is written into Redis and MySQL according to business demands for other functional data to use;
the process for quickly storing mass data comprises the following steps:
sorting and classifying mass data acquired at high speed according to data types, merging the same type of collection into the same type, and injecting the same type of collection into a stored data block to complete data sorting and collecting actions;
according to the mass data classification, the types are quickly stored in a database and a table, the data of the same type are stored in a time sequence data table of the same type, and the mass data are synchronously stored in a database in batches;
and the same-type data are combined regularly according to the time-sharing data content, and the same-type synchronous data combination of the data is completed.
Example 6
According to a specific embodiment of the present invention, this example differs from example 5 in that the separation of data collection, data preservation and business processing is achieved by:
the acquisition data is communicated with the equipment through a software protocol, various operation data of the equipment are read, and the acquisition software synchronously acquires the operation data of the equipment according to different acquisition channels established by the equipment; the collected data needs to be stored in a database at a high speed, the collection speed cannot be influenced, the collected data needs to be classified according to data types, the data is temporarily stored in a cache file, and data in the temporary file is processed by data storage software; the temporary cache file is divided into historical data and business logic data, and the stored data software processes the historical data and the business logic data in different modes respectively.
Example 7
According to one embodiment of the present invention, this example differs from example 5 in that the following sub-table and sub-library processing is performed for the data:
the data sub-table is divided according to the data type and the service data, and the time sequence data table of the equipment can be divided according to the data type, namely, each equipment is used as a data table according to different time. The service data is divided into data before logic processing and data after logic processing, and the data is classified into a data table of the same type of equipment according to the service after logic calculation.
The data sub-database is divided into a basic database and a historical database, wherein the basic database is mainly basic data for supporting system operation, and the historical database is data for collecting data and rapidly storing the data in real time.
Example 8
According to a specific embodiment of the present invention, the difference between the present embodiment and embodiment 5 is that the data is classified and collected into the storage block, and then the storage block is stored into the corresponding type of time sequence data table, the stored data is subjected to block batch processing operation instead of single data step-by-step operation, and the database separation and table is favorable for classifying and storing the data, reducing the data content of the single table, realizing the storage of tens of thousands of pieces of data per second, and improving the data storage speed and storage capacity. Because the data is stored in the temporary file, the data is stored according to the occurrence time, the file content is the process data, the file content data is read again to be displayed again to be the multi-disc data, and the running process of the equipment can be deduced according to the data process.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (11)
1. A method for high-speed data acquisition, comprising the steps of:
processing the data packets of the acquisition measuring points of different acquisition channel types according to a pre-stored processing method of the data packets of the acquisition measuring points;
generating an access equipment acquisition data packet command queue according to the processed acquisition measurement point data packet;
establishing acquisition channels of different acquisition channel types, and establishing access connection between the different acquisition channel types and corresponding equipment;
transmitting commands in a command queue of the data packet acquired by the access device to corresponding devices one by one;
receiving a data packet responded by the equipment, and if the data packet is successfully received and analyzed, considering the data packet as normal data; otherwise, the abnormality is considered to occur, and the abnormality processing is carried out;
the method for processing the measurement point data packets of different acquisition channel types according to the pre-stored acquisition measurement point data packet processing method specifically comprises the following steps:
combining the same acquisition channel type into one channel according to the acquisition channel type;
generating a measuring point address segment according to the addresses of the collecting measuring points of the same channel, specifically, according to the address ordering of the collecting measuring points of the same channel, recombining the addresses of the collecting measuring points into a continuous address segment of the collecting measuring points;
combining the address segments of the acquisition measuring points and the same data types into a data packet of the acquisition measuring points, specifically combining the continuous address segments of the measuring points according to the same data types into a data packet, and combining the access start address and the access data length of the data packet into a data packet;
supporting different baud rate devices in one acquisition channel to acquire data, specifically comprising:
setting baud rates of different devices in the same acquisition channel when the acquisition channels of the different types of devices are established;
sequencing the baud rates, grouping the equipment with the same baud rate together for regrouping, measuring the points with the same baud rate, and merging the continuous address segments of the measuring points with the same data type into a data packet;
the device is accessed each time to judge the baud rate, if the baud rate of the device acquisition channel is the same as that of the device acquisition channel accessed last time, the system does not switch the baud rate, and directly sends a data packet to acquire data; if the baud rate of the equipment acquisition channel of this time is different from that of the equipment acquisition channel accessed last time, the system closes the baud rate of the equipment acquisition channel accessed last time and opens the baud rate of the equipment acquisition channel accessed this time.
2. The method of claim 1, wherein a predetermined number of data packets of acquisition stations are combined into an acquisition data access packet as an access device acquisition data packet command.
3. The method of claim 2, wherein the access device collect data packet command queue is comprised of all access device collect data packet commands.
4. The method for collecting data at high speed according to claim 1, wherein the receiving device receives the data packet in response, analyzes the data according to the data packet definition rule if the receiving device is successful, and assigns the analysis content and the receiving time to the measurement point value and the collection time of the corresponding collection measurement point respectively if the analyzing is successful.
5. The method for high-speed data collection according to claim 1, wherein normal data is not received within a predetermined time, and is regarded as abnormal data; if the normal data is not received by the acquisition channel exceeding the set times, the channel is regarded as an abnormal channel, the normal acquisition channel is removed, the channel is put into a watchdog function for taking over, and during taking over, the watchdog function continuously tries to access the data packet or connect the channel according to a preset interval time until the data can be received and analyzed successfully, and the normal acquisition channel is put into.
6. The method for high-speed data acquisition according to claim 5, wherein for an abnormal channel, the watchdog function makes a channel connection attempt within a prescribed time, and if the connection is unsuccessful, waits for the next connection attempt; if the connection is successful, the data packet access is carried out within a specified time, if the data packet access can be normally received and the data can be normally analyzed, the data packet is considered to be normal and is put into the original acquisition channel, otherwise, the next data packet access attempt is continued to be waited; and for abnormal data, the watchdog function performs data packet access within a specified time, if the access data packet can be normally received and the data can be normally analyzed, the data packet is considered to be normal and is put into an original acquisition channel, otherwise, the data packet access attempt is continued to wait for the next time.
7. A method for fast mass data storage, characterized in that the data collected by the method for collecting data at high speed according to any one of claims 1-6 is stored, comprising the following steps:
dividing the collected data into three types of real-time data, historical data and service logic processing data, wherein the service logic processing data is new data obtained after logic judgment and calculation;
injecting data of the same data type into the same data block;
the data blocks of different types are subjected to table division, library division and time division storage, and the data of the same type are stored in a time sequence data table of the same type; establishing a data sub-table and a data sub-database according to requirements;
and combining the same-time data with the same type at regular intervals according to the time-sharing data content.
8. A method for rapid mass data storage as defined in claim 7, wherein,
writing the real-time data sub-table into a Redis remote data dictionary;
writing the history data sub-table into a MySQL relational database;
the service logic processing data is divided into early warning data, alarm data, function calculation data and logic conversion data according to service requirements, the real-time service logic processing data is written into Redis, and the calculated early warning data, alarm data, function calculation data and logic conversion data are written into MySQL.
9. The method for rapid mass data storage as defined in claim 7, wherein each device establishes a device time sequence data table according to different times when establishing the data sub-table; and classifying the service data after logic processing into a data table of the similar equipment according to the service type.
10. A method of rapid mass data storage as defined in claim 7 wherein the database is divided into a base database and a history database, the base database including data supporting system operation and the history database including collected data stored in real time.
11. The method for storing fast mass data according to claim 7, wherein the collected data is temporarily stored in a temporary buffer file according to the data type classification and the data occurrence time, the data in the temporary buffer file is processed in different ways, the temporary buffer file includes history data and service logic data, and the data in the temporary buffer file is merged and classified into the same data table according to the occurrence time and a preset time period.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310935692.8A CN116991925B (en) | 2023-07-27 | 2023-07-27 | Method for collecting data at high speed and storing mass data rapidly |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310935692.8A CN116991925B (en) | 2023-07-27 | 2023-07-27 | Method for collecting data at high speed and storing mass data rapidly |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116991925A CN116991925A (en) | 2023-11-03 |
| CN116991925B true CN116991925B (en) | 2024-03-15 |
Family
ID=88527783
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310935692.8A Active CN116991925B (en) | 2023-07-27 | 2023-07-27 | Method for collecting data at high speed and storing mass data rapidly |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116991925B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077405A (en) * | 2014-07-08 | 2014-10-01 | 国家电网公司 | Sequential type data accessing method |
| CN104345717A (en) * | 2014-10-17 | 2015-02-11 | 武汉华大优能信息有限公司 | Intelligent remote data acquisition system based on Internet of Things |
| CN111124679A (en) * | 2019-12-19 | 2020-05-08 | 南京莱斯信息技术股份有限公司 | Time-limited automatic processing method for multi-source heterogeneous mass data |
| CN113010565A (en) * | 2021-03-25 | 2021-06-22 | 腾讯科技（深圳）有限公司 | Server cluster-based server real-time data processing method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI366094B (en) * | 2007-12-28 | 2012-06-11 | Asmedia Technology Inc | Method and system of integrating data assessing commands and data accessing device thereof |
-
2023
- 2023-07-27 CN CN202310935692.8A patent/CN116991925B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077405A (en) * | 2014-07-08 | 2014-10-01 | 国家电网公司 | Sequential type data accessing method |
| CN104345717A (en) * | 2014-10-17 | 2015-02-11 | 武汉华大优能信息有限公司 | Intelligent remote data acquisition system based on Internet of Things |
| CN111124679A (en) * | 2019-12-19 | 2020-05-08 | 南京莱斯信息技术股份有限公司 | Time-limited automatic processing method for multi-source heterogeneous mass data |
| CN113010565A (en) * | 2021-03-25 | 2021-06-22 | 腾讯科技（深圳）有限公司 | Server cluster-based server real-time data processing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116991925A (en) | 2023-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115914022B (en) | Application relation analysis method, system, equipment and medium based on network traffic | |
| CN110855493A (en) | Application topological graph drawing device for mixed environment | |
| CN111127250B (en) | Power data monitoring event analysis system and method | |
| CN111080500A (en) | Spark streaming based real-time rule deployment and control early warning method and device | |
| CN108241717A (en) | A data processing method, device and system | |
| CN111008905A (en) | Method and device for high-precision measurement of beat of production equipment | |
| CN106646169A (en) | Electrical device partial discharge detection data collection cloud strategy | |
| CN115344207A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN106407072A (en) | Monitoring system of big data platform | |
| CN113364624A (en) | Mixed cloud flow acquisition method and system based on edge computing | |
| CN116126621A (en) | Task monitoring method of big data cluster and related equipment | |
| CN117573752A (en) | Big data information statistics acquisition method and system | |
| CN116991925B (en) | Method for collecting data at high speed and storing mass data rapidly | |
| CN114979186B (en) | Flow link analysis method and system based on Flink component | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN115632926A (en) | Alarm information processing method, device, equipment, storage medium and product | |
| CN104601374A (en) | Network fault processing method and device for numerical control machine tool | |
| CN118101411A (en) | Event-driven cross-channel data synchronization method and system | |
| CN114416785A (en) | Stream type enterprise big data processing method and storage medium | |
| CN103268353A (en) | Automatic response system for power grid alarm and automatic response method for power grid alarm | |
| CN110716909A (en) | Commercial system based on data analysis management | |
| CN111061790A (en) | Information acquisition system and method for customer data management | |
| CN111082962A (en) | Cloud-based terminal identification system and identification method | |
| CN115187086A (en) | Intelligent management and control system for emergency management | |
| CN115396752A (en) | Redis-based biplane data acquisition method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |