Disclosure of Invention
The invention aims to provide a distribution network FTU fixed-point file management method and a distribution network FTU fixed-point file management device, which are used for solving the problems that a file management process in the prior art occupies a large amount of memory and the file management complexity is high.
In order to achieve the purpose, the technical scheme of the invention is as follows:
the invention provides a distribution network FTU fixed point file management method, which comprises the following steps:
1) establishing a data area and a temporary file area in a memory, and storing fixed point data into the data area;
2) when the fixed point data is required to be uploaded, the fixed point data is searched in the data area, format conversion is carried out on the searched fixed point data, the fixed point data is stored in a fixed point file in the temporary file area, and the fixed point file is uploaded through the temporary file area.
According to the invention, the data area and the temporary file area are established in the memory, only the required dynamic data is stored, the occupation of the disk space is effectively reduced, the xml model operation which consumes memory resources very much is converted into the operation of the memory directly, the storage and the management of fixed point data are realized with extremely small memory overhead, the complexity and the resource consumption of file management are reduced on the premise of ensuring high performance, and the product cost is effectively reduced.
Furthermore, in order to conveniently call data in the FLASH, a file system is not used, a memory is directly operated by taking a block as a unit, magnetic disk fragments are not generated, the data area is divided into an information data block and N file data blocks, the fixed point data of each day occupies one file data block, each file data block is used for storing the fixed point data according to entries, each entry comprises the fixed point data and corresponding acquisition time, the information data block is used for storing attribute information of the file data block, the attribute information comprises the acquisition date and the number of stored entries of the current file data block, the position index of the attribute information is used as the index of each file data block, and N is more than or equal to 1.
Furthermore, when data is called and uploaded according to dates, in order to improve data transmission efficiency, when fixed point data is uploaded, indexes of file data blocks corresponding to the fixed point data are searched from the information data blocks according to the dates on which the data is uploaded, and then corresponding file data blocks are found according to the indexes.
Further, in order to improve the accuracy of data management, step 1) further includes a CRC check process, during initialization, the content of the information data block and a CRC check value are read from the data area into the memory structure, a CRC check value of the memory structure where the information data block is located is calculated, the CRC check value of the memory structure and the CRC check value in the data area are compared, if the CRC check values are consistent, the data is considered to be valid, and if the CRC check values are not consistent, the data is incorrect, and a default value is used to initialize the data area; when the content of the information data block in the memory changes, the CRC value of the memory structure is recalculated and is written into the data area together with the content of the information data block in the memory.
Further, in order to make the fixed point data monitoring process more suitable for practical application, in step 2), the corresponding fixed point data is converted into an xml format and stored in the fixed point file of the xml format in the temporary file area.
Further, in order to improve the data transmission efficiency, the fixed point data in the temporary file area is uploaded in a binary stream manner.
The invention also provides a distribution network FTU fixed point file management device, which comprises a memory, a storage interface, an access interface and a file conversion module; establishing a data area and a temporary file area in the memory; the storage interface is used for transmitting the collected fixed point data to a data area in the memory; the access interface is used for receiving a control command sent from the fixed point data; the file conversion module is used for searching the fixed point data in the data area and converting the format of the searched fixed point data when the fixed point data is required to be uploaded, and storing the fixed point data in the temporary file area; and the temporary file area uploads the fixed point file.
According to the invention, the data area and the temporary file area are established in the memory, only the required dynamic data is stored, the occupation of the disk space is effectively reduced, the xml model operation which consumes memory resources very much is converted into the operation of the memory directly, the storage and the management of fixed point data are realized with extremely small memory overhead, the complexity and the resource consumption of file management are reduced on the premise of ensuring high performance, and the product cost is effectively reduced.
Furthermore, in order to conveniently call data in the FLASH, a file system is not used, a memory is directly operated by taking a block as a unit, magnetic disk fragments are not generated, the data area is divided into an information data block and N file data blocks, the fixed point data of each day occupies one file data block, each file data block is used for storing the fixed point data according to entries, each entry comprises the fixed point data and corresponding acquisition time, the information data block is used for storing attribute information of the file data block, the attribute information comprises the acquisition date and the number of stored entries of the current file data block, the position index of the attribute information is used as the index of each file data block, and N is more than or equal to 1.
Furthermore, when data is called and uploaded according to dates, in order to improve data transmission efficiency, when fixed point data is uploaded, indexes of file data blocks corresponding to the fixed point data are searched from the information data blocks according to the dates on which the data is uploaded, and then corresponding file data blocks are found according to the indexes.
Further, in order to obtain a specific implementation form of the fixed point file management device, the memory is FLASH.
Detailed Description
Embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
The invention provides a distribution network FTU fixed point file management method and a device, wherein a data area and a temporary file area are established in a memory, and fixed point data is stored in the data area; when the fixed point data is required to be uploaded, the fixed point data is searched in the data area, format conversion is carried out on the searched fixed point data, the fixed point data is stored in a fixed point file in the temporary file area, and the fixed point file is uploaded through the temporary file area.
Method embodiment
Specifically, the distribution network FTU fixed point file management method is introduced by taking a fixed point data storage and uploading process within one day, the fixed point file is an xml format file, and the memory is FLASH. It should be noted that the description of the embodiments is provided to help understanding of the present invention, but the present invention is not limited thereto.
As shown in fig. 1, the memory is FLSAH, and a data area and a temporary file area are established in the memory; the storage interface is used for transmitting the collected fixed point data to a data area in the memory; the access interface is used for receiving a control command sent from the fixed point data; the file conversion module is used for searching the fixed point data in the data area and converting the format of the searched fixed point data when the fixed point data is required to be uploaded, and storing the fixed point data in the temporary file area; and the temporary file area uploads the fixed point file.
According to the standards of the FTU industry, it is generally required to have one fixed-point record every 15 minutes and one fixed-point file every day, and therefore, one fixed-point file contains 96(60/15 × 24) entries, and the number of fixed-point files is not less than 31. In order to guarantee data validity, one file data block is redundant, and 32 file data blocks are allocated in total.
According to the storage format of the entries in fig. 2, the structure of each entry is defined as:
TABLE 1
As can be seen from table 1, each entry occupies space 1B +3B +36B — 40B, and one file data block occupies space 40B — 96 — 3840B.
According to the storage format of the information data block in fig. 2, the structure of the information data block is defined as:
TABLE 2
Field(s)
|
Type (B)
|
Space(s)
|
Description of the invention
|
crc
|
unsigned int
|
4B
|
CRC32 value
|
sectornums
|
unsigned char
|
1B
|
Total block number of data (32)
|
lastsector
|
unsigned char
|
1B
|
Current block index
|
daystotal
|
unsigned char
|
1B
|
Maximum number of days of storage in the entire storage area
|
dayscnt
|
unsigned char
|
1B
|
Number of days of storage in whole memory
|
itemsize
|
unsigned char
|
1B
|
Entry occupied byte number
|
day_items
|
unsigned char
|
1B
|
Number of items per day (96)
|
days[32]
|
FIXPT_SECT_INFO
|
4B*32=128B
|
Block information |
Wherein, the structure definition of FIXPT _ SECT _ INFO is as follows:
TABLE 3
Field(s)
|
Type (B)
|
Space(s)
|
Description of the invention
|
itemcnt
|
unsigned char
|
1B
|
Number of entries already stored
|
date[3]
|
unsigned char
|
3B
|
Date (year, month and day) |
As can be seen from tables 2 and 3 above, the information data block occupies space of 4B +1B +4B × 32 — 138B.
Based on the above data, and since the erase operation of FLASH is performed in units of 4K, a storage space of 4KB is allocated to each of the file data block and the information data block.
When fixed point data of the file data block is converted into a fixed point file in an xml format, a DataFile root node is created, fixed point data attributes are stored as dataAttr nodes taking the DataFile as a father node, and each attribute is stored as a DI node taking the dataAttr as the father node. Each item of the fixed-point data is stored as a DataRec node taking a DataFile as a father node, the sect attribute of the DataRec node corresponds to the item number of the item, and the tm attribute is formed by combining the collection date and the collection time of the item in the file data attribute. Each fixed-point data value in the entry is stored as a DI node having DataRec as a parent node, and its attribute val is the fixed-point data value.
An example of the fixed point file in xml format stored in the temporary file area is as follows:
wherein, each child node is indented with two spaces before its father node, and each node is followed by a line feed character;
the maximum number of fixed point files in xml format is 25454B, calculated according to xml format and 96 entries in each file, so that 28KB of storage space is allocated for the temporary file area.
When the program is initialized, the content of the information data block is read into the memory, and the program is resident in the memory when running, and occupies the memory space of 138B.
In summary, the fixed point file management function only needs to occupy 4KB × 33+28KB — 160KB space and 138B memory. If at least 28KB × 31 KB 868KB is needed to directly store 31 xml format fixed point files.
By establishing the above fixed-point data storage format, the following fixed-point data storage method is adopted in this embodiment, and specifically includes the following storage processes:
1) defining the data area address as HDS _ FIXPT _ ADDR ═ 0x 1000000;
2) loading the information data block in the data area to the hdr _ info of the memory structure;
3) calculating CRC32 value val _ CRC32 of the hdr _ info except the CRC field, comparing the hdr _ info.crc with the val _ CRC32, if the contents do not accord with each other, indicating that the data is wrong, and initializing the data area of the FLASH by using a default value;
4) receiving a fixed-point data value pix _ info and a date cur _ date of an entry;
5) acquiring index information of a current storage block in a memory, comparing a date in the index with cur _ date, if the date is inconsistent and indicates that the data does not belong to the current file block, ending the fixed-point data storage of the current day, and recalculating the index of the current file block;
6) the method for calculating the index of the current file data block comprises the following steps: dividing the remainder of the total number of blocks occupied by the file data by the index +1 of the current data block, namely (hdr _ info.lastselector + 1)% 32, then erasing the original data of the data block, and simultaneously setting the entry hdr _ info.days [ hdr _ info.lastselector ]. itemcnt of the data block to 0;
7) calculating the current data storage address, wherein the method comprises the following steps: first, locating the current block head address according to the calculated file data block index hdr _ info. HDS _ CUR _ ADDR ═ HDS _ FIXPT _ ADDR +4K (1+ hdr _ info.lastsector); and then according to the number of the stored entries of the current block, positioning the write address of the current entry, namely: ITEM _ CUR _ ADDR ═ HDS _ CUR _ ADDR + hdr _ info.days [ hdr _ info.lastselector ]. itemcnt 40, where 40 is the space occupied by each entry;
8) writing the received fixed-point data value pix _ info into an address ITEM _ CUR _ ADDR according to the defined storage format;
9) the method for updating the stored days and date information in the memory comprises the following steps: if the value of hdr _ info.days [ hdr _ info.colors ector ]. itemcnt is 0, it indicates that the current region is a newly enabled data block, the date is updated to cur _ date, if the number of days already stored reaches the maximum number of days of storage, the number of days already stored is added by 1, that is, hdr _ info.days + -, 1;
10) updating the entry information of the current block in the memory, wherein the method comprises the following steps: (ii) the value of hdr _ info.days [ hdr _ info.lastselector ]. itemcnt plus 1;
11) and recalculating the crc of the memory data, and writing the hdr _ info into a data area in the FLASH.
By establishing the above fixed point file storage format, the following process is specifically included when accessing the fixed point file in this embodiment:
1) the communication service module accesses fixed point data of a specified date;
2) reading the file date of each block index in the memory, finding the index number of the file data block according to the date by traversing the file data block attribute list of the information data block, calculating the offset address of the corresponding file data block according to the index number and the initial head address to obtain the corresponding data block, comparing the date of the data block with the specified date, and if the date of the data block is consistent with the specified date, indicating that the data block is the fixed point data to be accessed;
3) reading entry data in the file data blocks in sequence, converting the entry data into an xml format through an operation interface, and storing the xml format into a fixed point file in an xml format of the temporary file area;
4) returning the initial address and the file size of the xml file in the temporary file area to the communication service module;
5) and the communication service module reads the xml file in the temporary file area in a binary stream mode through the access interface.
In the above embodiment, the storage medium FLASH is divided into two areas: a data area and a temporary file area. The data area comprises an information data block and a plurality of file data blocks, wherein one file data block comprises a plurality of entries and only stores dynamic data information required by the fixed point file. The temporary file area stores a complete fixed point file in xml format. And when the fixed point data is received, circularly writing the fixed point data into the FLASH according to the index information in the memory. When the communication service module accesses the fixed point file, the data block corresponding to the file is positioned by date, and the data in the data block is converted into fixed point data in xml format and stored in a temporary file area with fixed space size. And the communication service module reads the fixed point file in the xml format of the temporary file area by a binary stream method through the access interface.
In another embodiment, the file format in the temporary file area is not limited to xml format, and may be CSV, DAT format, or the like. The indexing method is not limited to date indexing, and position information and type information can be used as indexes. The data area is not limited to the data block storage, and can also be in the form of object storage. The memory may also be other storage media with read-write capability besides FLASH, such as solid state disk, mechanical hard disk, and other nonvolatile memories. The fixed point data is stored in the data area in a binary small end format, and the fixed point file in the temporary file area is uploaded in a binary stream format.
Device embodiment
As shown in fig. 1, the present invention further provides a distribution network FTU fixed-point file management apparatus, which includes a memory, a storage interface, an access interface, and a file conversion module; the working principle of the memory, the storage interface, the access interface and the file conversion module is the same as that described in the above method embodiment, and therefore, the description is omitted here.
According to the distribution network FTU fixed point file management method and device based on the method, a file system is not used, a memory is directly operated by taking a block as a unit, and disk fragments are not generated; during storage, the data are written into a memory according to a generation sequence, an absolute time sequence is not relied on, and data disorder caused by time jump is avoided; by only storing the dynamic data in the xml, the occupation of the disk space is effectively reduced; and the xml model operation which consumes memory resources very much is converted into the operation of a memory directly, the storage and management of fixed-point files are realized with extremely low memory overhead, the complexity and the resource consumption of file management are reduced on the premise of ensuring high performance, and the product cost is effectively reduced.