CN112650448B

CN112650448B - FPGA-based large-data-volume storage file management method

Info

Publication number: CN112650448B
Application number: CN202011521951.5A
Authority: CN
Inventors: 业瀚青; 唐遒; 李磊; 张健伟; 郭小晨; 田弘博
Original assignee: 8511 Research Institute of CASIC
Current assignee: 8511 Research Institute of CASIC
Priority date: 2020-12-21
Filing date: 2020-12-21
Publication date: 2024-04-05
Anticipated expiration: 2040-12-21
Also published as: CN112650448A

Abstract

The invention discloses a large data volume storage file management method based on an FPGA, which adopts FPGA hardware to realize a high-speed solid-state memory interface, fully utilizes the programmable advantage of the FPGA, enhances flexibility portability, and a high-speed DDR3 cache chip carried on the interface, solves the requirement of large data burst transmission, and balances the fluctuation of the data writing rate of the high-speed solid-state disk; the DSP processor receives and processes file information/solid-state memory data read-back; a high-speed eMMC memory chip is added for storing file information data, the purpose of which is to enable uninterrupted writing of large data into the solid-state memory to ensure the persistence of data writing.

Description

FPGA-based large-data-volume storage file management method

Technical Field

The invention belongs to the field of data storage, and particularly relates to a large-data-volume storage file management method based on an FPGA.

Background

With the increasing complexity and the changing of modern battlefield signal environments, new system radars, unknown signals and complex waveforms on the battlefield are layered endlessly, and meanwhile, the front-end sampling rate and the signal resolution of a receiver are improved, so that the information of the electromagnetic data of the battlefield is exploded and increased; sometimes external electromagnetic environment data cannot be analyzed in real time; the need for data storage and support for post-hoc read-back analysis continues to increase. For this reason, the demand for mass data storage and management is increasing. The earlier method adopts a computer-mounted disk array to store data, or an FPGA-mounted solid state disk to collect and directly store data, but the method has a plurality of defects, for example, the computer-mounted disk array can cause huge equipment and cannot meet the light-weight requirement; the FPGA data is directly collected and stored, and different data cannot be classified and managed, so that the singleness of data readback can be caused.

Disclosure of Invention

The invention aims to provide a large-data-volume storage file management method based on an FPGA, which meets the light-weight requirement of equipment and the diversity of readback data types. The invention utilizes the high bandwidth advantage of FPGA to complete the collection of big data, and special coding characters are input through a specific coding module, wherein the characters are used for constructing file information and then flow into a high-speed DDR cache, and when data is read from DDR, the characters are written into a rear-end fixed state hard disk; the writing of data needs to record the real-time writing sector address, writing data length, writing time, data type and file name information of the solid-state disk at the back end, and the information is formed into a file information entry to be recorded in the high-speed eMMC for storage. The number of the FPGA mounted solid-state disks can be flexibly changed according to the data throughput rate.

The technical solution for realizing the purpose of the invention is as follows: a method for managing a large data volume storage file based on an FPGA, wherein the large data volume is larger than 1GB, specifically comprises the following steps:

step 1, an FPGA receives original big data, adds special coding characters into a data stream to obtain a coded data stream, and shifts to step 2;

step 2, writing the encoded data stream into a high-speed DDR3 cache in real time, balancing the fluctuation of the writing rate of the solid-state memory, and turning to step 3;

step 3, reading the data stream coded in the step 2, detecting the coded characters, recording the current sector address, updating the data length information in the L2/L3 file information after the L2/L3 data is written, and generating the L1 file information when the storage duration is finally finished, and turning to the step 4;

step 4, writing the data stream containing the special codes in the step 3 into a solid-state memory for storage, and transferring to the step 5;

step 5, packaging and integrating the L1, L2 and L3 file information generated in the step 3, transmitting the integrated information to a DSP processor, and transferring to the step 6;

step 6, classifying and integrating the L1, L2 and L3 file information received in the step 5 in the DSP processor, planning the written eMMC storage address, starting the eMMC writing operation, and turning to the step 7;

step 7, after receiving the file information synchronization instruction, the DSP processor reports the file information stored in the eMMC to a server, and the step 8 is shifted to;

and step 8, after receiving the reading control instruction, the DSP processor forwards the instruction to the FPGA, the FPGA analyzes the reading instruction, extracts the sector starting address and the reading total length, reads corresponding data from the solid-state memory, forwards the data to the DSP processor, and forwards and reports the data to the DSP processor.

Compared with the prior art, the invention has the remarkable advantages that:

(1) The invention adopts the FPGA to realize the function of file management and realize light weight.

(2) The data read-back is simple and easy to manage, the management of the file information directly corresponds to the management of big data, and the data processing pressure of management software is greatly reduced.

Drawings

FIG. 1 is a flow chart of the data writing process of the FPGA-based mass storage file management method according to the present invention.

Fig. 2 is a flow chart of data readout of the method for managing the mass storage file based on the FPGA according to the present invention.

FIG. 3 is a schematic diagram of the structure of adding special coding characters to the data stream according to the present invention.

FIG. 4 is a diagram of DDR3 read/write state machine control according to the present invention.

Detailed Description

The invention is described in further detail below with reference to the accompanying drawings.

Referring to fig. 1 to 4, a method for managing a large data volume storage file based on an FPGA, the method includes the following steps:

and step 1, receiving original big data by the FPGA, and adding special coding characters to the data stream by utilizing a coding and packaging module.

The method comprises the following steps of adding special coding characters to an original big data stream:

step 11, firstly dividing the input large data volume into a plurality of segments, as shown in fig. 3, dividing the segment data by a user-defined decision, and finally forming single file data;

step 12, driving a data frame header A into each section, wherein the data frame header A comprises information such as creation time stamp, data type, creator, file name and the like; the frame header A contains special characters 0xC2C2C2C2 which are easy to detect, and the length of the data frame header is 1KB;

step 13, the data frame head A is followed by a big data stream, namely a data load D, and 1KB is taken as the minimum data granularity, so that zero padding is insufficient; the data load D can be driven with special characters 0xC3C3C3C3 to form independent small data segments;

and step 14, adding a data frame tail B after the data load D, wherein the data frame tail B comprises the data length of the data load D. The end of frame B contains a special character 0xD1D1 (0 xD2D2 if the last data) that is easy to detect, and the data end of frame length is 1KB;

after the data stream is encoded, a foundation is built for the rear end to realize data files while original data is reserved, 1 file is built every A+D+B, big data is divided into a plurality of files, the files can be continuous or disconnected, and the data stream is flexibly divided.

Step 2, writing the encoded data stream into a high-speed DDR3 cache in real time, balancing the fluctuation of the writing rate of the solid-state memory, and specifically comprising the following steps:

and step 21, pre-caching the encoded and packaged data through a FIFO, and adjusting the time sequence beat to be isolated from a clock.

Step 22, after the system is reset, entering an initialization state, entering detection read-write when the pre-cache FIFO data is not 0, starting to execute a DDR3 writing process when the pre-cache FIFO data amount in the step 21 is judged to be more than or equal to 1KB, transferring the data in the pre-cache FIFO to DDR3, writing 1KB, and returning to the detection write-read state.

Step 23, when the write-read state is executed, when the write is idle, detecting whether the read DDR3 FIFO can allow transmission, and if yes, starting the read executing process when the data quantity in the DDR3 is larger than 1 KB. And writing the data read out of DDR3 into a read DDR3 FIFO and outputting the data to the outside. When the reading is completed by 1KB, the method returns to the detection writing and reading state.

Step 24, when detecting the special character of 0xD2D2D2D2, returning to the initialization state when the data is read empty.

Step 3, detecting the read code character in the step 2, recording the current sector address, updating the data length information in the L2/L3 file information after the L2/L3 data is written, and finally generating the L1 file information when the storage time is finished, as shown in fig. 4, specifically comprising the following steps:

step 31, detecting special characters 0xC2C2 in the data frame header a, recording the sector address of the solid-state memory corresponding to the frame header a at this time, extracting the information such as the creation time stamp, the data type, the creator, the file name and the like in the frame header a, and pre-caching the information by using a register.

Step 32, receiving a data load D in a data frame, and counting the data quantity; if the special character 0xC3C3C3 is detected, the sector address at the moment is required to be recorded; while counting starts for each segment 0xC3C 3.

Step 33, detecting the end of the data frame B, judging whether the data frame B is a 0xD2D2 special character, if not (i.e. 0xD1D 1), returning to step 31; if yes, stopping detection; simultaneously latching the data quantity at the moment;

step 34, creation time stamp, data type, creator and file name and total amount of data captured in steps 31 and 33 generate file information L2 (corresponding to detection of 0xC2C 2) or L3 (corresponding to detection of 0xC3C 3); generating 1L 1 file information each time of storage tasks, wherein the L1 file information comprises a sector starting address, creator, file name, total data amount and other information; at the same time, the DDR3 buffered data is stored into solid state memory.

Step 4, writing the big data containing the special codes in the step 3 into a solid-state memory for storage, and specifically comprising the following steps:

step 41, according to the analysis instruction, starting writing operation on the received writing solid-state memory instruction;

step 42, writing big data into the solid-state memory;

step 43, when detecting 0xD2D2 in the data stream, starting and stopping the write operation.

And 5, packaging, integrating and transmitting the L1/L2/L3 file information generated in the step 3 to a DSP processor, wherein the method specifically comprises the following steps of:

step 51, respectively generating corresponding file information messages according to the L1/L2/L3 file information in the step 3, and adding a message frame head and frame tail mark;

step 52, generating L1/L2/L3 hierarchy names used as hierarchy relation definitions to divide the logical folders;

step 53, the message is sent to the DSP processor through the SRIO interface.

Step 6, classifying and integrating the L1/L2/L3 file information received in the step 5 in a processor, planning the written eMMC storage address, and starting the eMMC writing operation, wherein the method specifically comprises the following steps:

step 61, aiming at the L1/L2/L3 file information, caching in the DSP plug-in DDR3 in advance, and judging the logic hierarchical relationship;

step 62, aiming at the L1/L2/L3 file information, allocating the writing address of the sector of the eMMC memory;

step 63, writing the file information preprocessed in steps 61 and 62 into eMMC.

Step 7, after receiving the write control instruction, the DSP processor forwards the write control instruction to the FPGA driving module of the solid-state memory; after receiving the file information synchronization instruction, the DSP processor reports the file information stored in the eMMC to the server, and specifically comprises the following steps:

step 71, reading out the L1 level file information stored in the eMMC, and simultaneously reading all L2 level file information corresponding to the current L1 level;

and step 72, collecting and packaging the file information received in the step 71 and sending the file information to a server.

And step 8, a data reading flow. After receiving the read control instruction, the DSP processor forwards the instruction to the FPGA, the FPGA analyzes the read instruction, extracts the information such as the sector starting address, the read total length and the like, reads out the data corresponding to the solid-state memory from the read information and sends the data to the DSP processor, and the DSP processor forwards and reports the data, and the method specifically comprises the following steps of:

step 81, the management software reads the designated file, extracts the information in the corresponding file information, including the information of the sector initial address, the total length of the data and the like, integrates the information into a read instruction and sends the read instruction to the DSP processor;

step 82, after receiving the read control instruction, the DSP forwards the instruction to the FPGA through an EMIF interface;

step 83, the FPGA decodes the read instruction, extracts the initial address and the length of the sector, reads out the data corresponding to the solid-state memory from the initial address and the length of the sector, and sends the data to the DSP;

and step 84, the DSP processor receives the data and reports the data to the management software.

Claims

1. The method for managing the large data volume storage file based on the FPGA is characterized by comprising the following steps of:

step 1, an FPGA receives original big data, and adds special coding characters to a data stream to obtain the coded data stream, wherein the specific steps are as follows:

step 11, firstly dividing the input large data volume into a plurality of segments, dividing the segment data by a user-defined decision, and finally forming single file data;

step 12, driving a data frame header A into each segment, wherein the data frame header A comprises a creation time stamp, a data type, a creator and a file name; the frame header A contains special characters 0xC2C2C2C2 which are easy to detect, and the length of the data frame header is 1KB;

step 13, the data frame head A is followed by a big data stream, namely a data load D, and 1KB is taken as the minimum data granularity, so that zero padding is insufficient; the data load D is driven with special characters 0xC3C3C3C3 to form independent small data segments;

step 14, adding a data frame tail B after the data load D, wherein the data frame tail B comprises a data length of the data load D, the frame tail B comprises a special character 0xD1D1D1D1 which is easy to detect, if the data is the last data, the data frame tail B is 0xD2D2D2D2, and the data frame tail length is 1KB;

after the data stream is encoded, a foundation is built for the rear end to realize data files while original data is reserved, 1 file is built every A+D+B, and big data is divided into a plurality of files;

turning to step 2;

2. The method for managing the large data volume storage files based on the FPGA according to claim 1, wherein the large data is divided into a plurality of files, two adjacent files are continuous or disconnected, and the data flow is flexibly divided.

3. The method for managing mass storage files based on FPGA according to claim 2, wherein in step 2, the encoded data stream is written into the high-speed DDR3 cache in real time, so as to balance the fluctuation of the writing rate of the solid state memory, and specifically comprising the following steps:

step 21, pre-caching the coded and packaged data stream through a FIFO, and adjusting the time sequence beat to be isolated from a clock;

step 22, after the system is reset, entering an initialization state, entering detection read-write when the pre-cache FIFO data is not 0, starting to execute a DDR3 writing process when the pre-cache FIFO data amount in the step 21 is judged to be more than or equal to 1KB, transferring the data in the pre-cache FIFO to DDR3, writing 1KB, and returning to the detection write-read state;

step 23, when the writing and reading states are executed, when writing is idle, whether the reading DDR3 FIFO can allow transmission or not is detected, and if yes, and the data quantity in the DDR3 is more than 1KB, a reading executing process is started; writing the data read out of DDR3 into read DDR3 FIFO and outputting the data; returning to the detection writing and reading state when the reading of 1KB is completed;

4. The method for managing a mass storage file based on an FPGA according to claim 3, wherein in step 3, the data stream encoded in step 2 is read out and the encoded characters thereof are detected, and the method specifically comprises the steps of:

step 31, detecting special characters 0xC2C2C2C2 in the data frame header A, recording the sector address of the solid-state memory corresponding to the frame header A at the moment, extracting the creation time stamp, the data type, the creator and the file name in the frame header A, and pre-caching the information by using a register;

step 32, receiving a data load D in a data frame, and counting the data quantity; if the special character 0xC3C3C3 is detected, the sector address at the moment is required to be recorded; simultaneously starting counting of each segment of 0xC3C3C 3;

step 33, detecting the tail of the data frame B, judging whether the data frame B is a special character of 0xD2D2, if not, namely 0xD1D1, returning to step 31; if yes, stopping detection and simultaneously latching the data quantity at the moment;

step 34, creating a time stamp, a data type, a creator and a file name captured in steps 31 and 33, and generating file information L2 corresponding to detection of 0xC2C2 or L3 corresponding to detection of 0xC3C 3; generating 1L 1 file information each time of storage task, wherein the L1 file information comprises a sector starting address, a creator, a file name and a data total amount; at the same time, the DDR3 buffered data is stored into solid state memory.

5. The method for managing mass storage files based on FPGA of claim 4, wherein step 4 comprises writing the specially encoded data stream into a solid state memory for storage, comprising the steps of:

step 41, executing writing operation on the received writing solid-state memory instruction according to the analyzed instruction;

step 42, writing the data stream containing the special codes into the solid-state memory;

step 43, when detecting 0xD2D2 in the data stream containing special codes, executing the stop writing operation.

6. The method for managing mass storage files based on FPGA of claim 5, wherein in step 5, the L1, L2 and L3 file information generated in step 3 is packaged and integrated and sent to the DSP processor, specifically comprising the following steps:

step 51, respectively generating corresponding file information messages according to the L1, L2 and L3 file information in the step 3, and adding a message frame head and frame tail mark;

step 52, generating L1, L2 and L3 hierarchy names which are used as hierarchy relation definitions to divide the logic folders;

step 53, the message is sent to the DSP processor through the SRIO interface.

7. The method for managing large data volume storage files based on FPGA of claim 6, wherein in step 6, classification and integration are performed on the L1, L2 and L3 file information received in step 5 in a processor, the written eMMC storage address is planned, and eMMC writing operation is started, specifically comprising the following steps:

step 61, judging a logic hierarchical relationship according to the received L1, L2 and L3 file information, and writing the logic hierarchical relationship into a DSP plug-in DDR3 memory in a pre-cache manner;

step 62, aiming at the file information of L1, L2 and L3, allocating the writing address of the sector of the eMMC memory, and starting to start the writing operation of the eMMC;

step 63, writing the file information cached and preprocessed in steps 61 and 62 into eMMC.

8. The method for managing a large data volume storage file based on an FPGA of claim 7, wherein step 7 is a file synchronization operation, and the DSP processor reports the file information stored in the eMMC to the server after receiving the file information synchronization instruction, specifically comprising the steps of:

9. The method for managing the large data volume storage file based on the FPGA according to claim 8, wherein the step 8 is a data reading flow, after receiving a reading control instruction, the DSP processor forwards the instruction to the FPGA, the FPGA analyzes the reading instruction, extracts a sector starting address and a reading total length, reads data corresponding to the solid-state memory from the sector starting address and sends the data to the DSP processor, and the DSP processor forwards and reports the data, and the method specifically comprises the following steps:

step 81, the management software reads the designated file, extracts the information in the corresponding file information, including the sector initial address and the total length of the data, integrates the read instruction and sends the read instruction to the DSP processor;

step 84, the DSP processor receives the data and reports the data.