CN101252404A - FPGA-based distributed network clock synchronization system and method - Google Patents

Abstract

The invention claims to protect a small distributed network clock synchronization system based on field programmable gate array (FPGA), which relates to distributed network synchronization technology. It mainly includes PC, HUB and distributed units based on FPGA. Each distributed unit has a common network interface, and uses a fixed synchronization cycle time scheme to realize clock synchronization of small distributed networks, enabling the system to achieve synchronization at the microsecond level. The synchronization method of the present invention adopts a method similar to the IEEE1588 standard. According to the high-precision synchronization characteristics of the standard, the initial synchronization speed is improved, and a distributed system with fast initial synchronization speed and high-precision clock synchronization is realized. It serves as a reference for the working clock of the system based on the synchronous clock provided by each component unit of the distributed network system. In this way, distributed networks with high real-time requirements can be satisfied for data transmission and the like.

Description

FPGA-based distributed network clock synchronization system and method

technical field

The invention relates to computer control technology, in particular to distributed network synchronization technology.

Background technique

With the development of computer networks, the scope of distributed systems has expanded and become more dispersed, and the real-time performance between distributed control nodes connected through the network has become more and more important. The main root that affects the real-time performance of the distributed system is the clock difference between the various measurement and control devices and the transmission delay and jitter of the measurement and control data in the network. These problems are inherent in distributed systems. The key to solving these problems is time synchronization. The purpose of time synchronization is to accurately transmit the time reference point to each control point of the network. At present, the popular synchronization methods at home and abroad include: simple time synchronization protocol SNTP, networked measurement and control system precision clock synchronization protocol IEEE1588 standard, etc.

1. The simple time synchronization protocol SNTP can operate in unicast mode (point-to-point) and broadcast mode (point-to-multipoint), and can also operate in IP multicast mode. However, the time precision of SNTP depends on the conditions of the client and server networks. Using this synchronization method can achieve an accuracy of 0.5-2ms in a distributed system. In the case of flight test systems and industrial acquisition and control with high synchronization requirements, the synchronization accuracy is far from meeting the requirements.

2. The IEEE1588 standard defines a clock synchronization protocol suitable for use in measurement and automation systems. It can achieve ultra-high-precision clock synchronization at the microsecond level, which is higher than the accuracy of other network time synchronization protocols. However, the standard has very high requirements on the network, which is suitable for the working environment with only Ethernet. Moreover, when a new unit is added to the network or the network environment changes, the network needs to be re-synchronized, and it takes a long time to complete the synchronization. In a 2-unit system, 10 line delays are required to complete a synchronization, and as the number of synchronization units increases, the time to complete synchronization will increase exponentially.

With the development of computer and electronic technology, FPGA technology has been widely used and has become quite mature. FPGA has also added high-speed I/O, embedded memory, dedicated phase-locked loop (PLL), and embedded processor. There are a large number of pre-designed circuit blocks presented as IP cores.

Secondly, on-chip programmable system SOPC (System On a Programmable Chip,) is the convergence point and development direction of modern electronic technology and electronic system design. SOPC combines the respective advantages of SOC, PLD and FPGA, and integrates hard-core or soft-core CPU, OSP, memory, peripheral I/O and programmable logic. Users can use the SOPC platform to design high-speed, high-performance DSP processors or CPU processors with specific functions, so that the design of electronic systems has entered a new mode.

Using FPGA technology to achieve distributed network clock synchronization can meet the clock synchronization between networks that follow different protocols, with high synchronization accuracy, and the increase of synchronization units will not greatly increase the synchronization time. Improve network work efficiency.

Contents of the invention

The technical problem to be solved by the present invention is to design a distributed network clock synchronization system based on FPGA to realize network Quickly enter the synchronization state and high-precision clock synchronization.

The technical solution of the present invention to solve the above technical problems is to propose an FPGA-based distributed network clock synchronization system, which includes a control and processing part, a clock synchronization part, and a data transmission part.

The control and processing part includes a network center control processing part and each distributed unit control processing part, each distributed unit control processing part is a unit with FPGA as the core, and a microprocessor is embedded in the FPGA to control Each distributed unit in the network realizes real-time synchronization, regularly sends data to the distributed network or receives data sent by other distributed units.

The control and processing part of the network center is controlled and processed by the main control computer and the monitoring computer, including a computer detection part, a computer configuration part and a data packet detection part.

Computer detection part: the main control computer sends detection data to detect how many distributed units are in the network; computer configuration part: the main control computer sends configuration data to configure the functions of each distributed unit, and the definitions include working cycle, data generation time, Data generation frequency and data sending time, etc.; data packet detection part: when the system enters real-time work (starts data transmission), the main control and monitoring computer complete the monitoring of data packets.

The clock synchronization part includes a master distributed unit MASTER and multiple slave distributed units SLAVER, and its real-time control module realizes the synchronization between the master and slave distributed units. The real-time control module of the slave distributed unit calculates the network line delay Delay of each slave distributed unit according to the time interval of receiving and sending messages and the number of the slave distributed unit, and determines the MASTER unit according to the network line delay of each slave distributed unit and the clock reset time of the SLAVER unit, so that the clock of the MASTER unit and each SLAVER unit is reset to zero at the same time.

In this way, multiple terminals can be synchronized at the same time, so this is a network clock synchronization method that is more suitable for distributed systems with frequent multi-terminal synchronization cycles.

The data transmission part adopts the Ethernet transmission mode, and transmits detection, configuration data, synchronous initiation messages, etc. in the asynchronous mode, and transmits real-time data of each distributed unit in the synchronous mode.

The present invention also proposes an FPGA-based distributed network clock synchronization method, the main distributed unit sends a synchronization initiation message to the network; each slave distributed unit returns a processing time including its unit number, sending and receiving data packets The master distributed unit calculates the network line delay of each slave distributed unit according to the time interval and processing time; and then notifies the slave distributed unit of the synchronization start time according to the line delay.

The method comprises the steps of:

1. The MASTER unit sends a synchronization initiation message to the SLAVER unit;

2. The SLAVER unit receives the message from the MASTER unit and returns a synchronous response message with the unit number of the MASTER unit and the message processing time (the time between receiving the message and returning the message);

3. After the MASTER unit receives the synchronous response message, it calculates the line delay according to the time when the message is sent and received;

4. The MASTER unit sends a synchronization start packet to the SLAVER unit according to the line delay;

5. The clocks of the MASTER unit and all SLAVER units are reset to zero at the same time.

This distributed network clock synchronization system uses an FPGA-based distributed network clock to achieve high-precision synchronization of the distributed system in the network, and can meet the requirements of distributed real-time data collection and control such as flight test systems and high-precision industrial collection and control. Need, and when the initial synchronization or the network environment changes and needs to be resynchronized, it can quickly enter the synchronization state. As long as the data transmission time of each unit does not conflict, the system can accurately send the data collected at each time to the designated location, and the transmission is reliable, the anti-interference ability is strong, the stability is good, and the real-time performance and compatibility are strong. The practical application shows that the clock synchronization, data acquisition and processing of the system are realized by FPGA, which not only can realize the microsecond-level time synchronization, but also greatly reduces the time required for the initial synchronization of the distributed system. The work is stable and reliable; data communication can be completed in real time.

Description of drawings

Figure 1: Distributed system architecture

Figure 2: Architecture of each distributed unit

Figure 3: 2-unit synchronous working principle diagram

Figure 4: Schematic diagram of multi-unit synchronization

Figure 5: Flow chart of MASTER unit processing

Figure 6: SLAVER unit processing flow chart

specific implementation plan

The invention proposes an FPGA-based distributed network clock synchronization system, which includes a control and processing part, a clock synchronization part and a data transmission part.

The control and processing part includes the control processing part of the network center and the control processing part of each distributed unit. The control processing part of each distributed unit is a unit with FPGA as the core. There is a microprocessor embedded in the FPGA, and the control network Each distributed unit realizes real-time synchronization, regularly sends data to the distributed network or receives data sent by other distributed units. The control and processing part of the network center is controlled and processed by the main control computer and the monitoring computer, including a computer detection part, a computer configuration part and a data packet detection part. Computer detection part: the main control computer sends detection data to detect how many distributed units are in the network; computer configuration part: the main control computer sends configuration data to configure the functions of each distributed unit, and the definitions include working cycle, data generation time, Data generation frequency and data transmission time, etc.; data packet detection part: when the system enters real-time work and starts data transmission, the main control and monitoring computer completes the monitoring of data packets. The clock synchronization part includes a master distributed unit MASTER and multiple slave distributed units SLAVER. The network clock synchronization adopts a method similar to the IEEE1588 standard, but according to the disadvantage of slow initial synchronization speed, the clocks of each distributed unit are reset to zero at the same time. To replace the adjustment of the clock skew in the IEEE1588 standard,

In this way, multiple terminals can be synchronized at the same time, so this is a network clock synchronization method that is more suitable for distributed systems with frequent multi-terminal synchronization cycles.

The implementation of the present invention will be specifically described below with reference to the accompanying drawings and specific examples.

FIG. 1 is a structural block diagram of the distributed network clock synchronization system of the present invention. The system includes a master distributed unit MASTER and multiple (such as n) slave distributed units SLAVER, each unit is connected to the main control and monitoring computer of the control and processing part of the network center through a hub HUB, each distributed unit (including MASTER and SLAVER units) include network interfaces and FPGA boards, and the control and processing parts of distributed units with FPGA as the core are connected to HUB through network interfaces.

The functions of the MASTER unit and the SLAVER unit are the same except for the clock synchronization part, and the other functions and structures are the same. The MASTER unit is the clock synchronization initiator, and the SLAVER unit follows the clock of the MASTER unit.

The control and processing part of the network center is controlled and processed by the main control computer and the monitoring computer, and completes the functions of the computer detection part, the computer configuration part and the data packet detection part. The main control computer is responsible for sending detection data, synchronous commands (that is, exiting the control state) or sending configuration packets of each unit to detect how many distributed units there are in the network; configure the functions of each distributed unit, and define the working cycle and data generation time , data generation frequency and data transmission time, etc.; when the system enters real-time work (starts data transmission), the main control and monitoring computer will monitor the data packets in real time through the data packet detection part.

Figure 2 shows the architecture of each distributed unit. The control and processing part of each distributed unit is a unit with FPGA as the core, and SOPC soft-core processor is embedded in the FPGA board. There are system configuration, function configuration, transmission control, real-time control, UDP, acquisition and output modules. Each distributed unit in the control network realizes real-time synchronization, and regularly sends data to the distributed network or receives data sent by other distributed units.

Among them, the system configuration module, function configuration module and transmission control module all belong to the application layer module, and they and the acquisition and output module are realized by FPGA. Real-time control module and UDP module are realized by SOPC system in FPGA. The main control computer sends system configuration information and function configuration information. After receiving the above system configuration information, the distributed unit stores it in the FLASH of the distributed unit. The system configuration module of the distributed unit initializes the system working state according to the system configuration information. The information includes system synchronization period, data transmission period, data transmission length, data receiving buffer address, and data transmission time; the function configuration module determines the initial collection time, data collection rate, number of data collection channels, and data collection time of each synchronization cycle according to the function configuration information. Storage address, data output cache address and other information. When the system is powered on and initialized or the system configuration and function configuration information stored in the FLASH is changed, the FPGA actively changes the system function by reading and writing the FLASH.

The transmission control module includes an intra-station transmission control module and an inter-station transmission control module. The intra-station transmission control module stores the data collected by the acquisition module in the memory SRAM in real time, or sends the data in the SRAM to the cache address of the output module. In the real-time state, the inter-station transmission control module combines the real-time data stored in the SRAM into a UDP packet format for sending.

The real-time control module calculates the network line delay Delay of each SLAVER unit according to the time interval Adjust and the SLAVER unit number from receiving the synchronous initiation message Sync to sending the synchronous response message Sync_Responce from the slave unit, and according to the network line delay of each slave distributed unit Determine the time when the clocks of the MASTER unit and the SLAVER unit return to zero, and realize the synchronization between the master and slave distributed units.

UDP module: Real-time data transmission is realized by transmitting UDP packets, and data transmission is realized by SOPC soft-core processor through control commands. Its sending and receiving functions are sending message TransmitPacket and receiving message ReceivePacket respectively.

The synchronization control process of the distributed network clock synchronization system of the present invention will be specifically described below by taking two-unit and multi-unit systems as examples respectively.

FIG. 3 is a schematic diagram of the synchronization process of the distributed system of 2 units. The synchronization process defines three synchronization messages: a synchronization initiation message Sync, a synchronization response message Sync_Responce, and a synchronization start message Sync_Start. The system clock is generated by the crystal oscillator count of the distributed unit, for example, the system crystal oscillator frequency can be selected as 32M. The synchronization process is as follows:

First, the MASTER unit transmission control module controls the output module to send a Sync message to the SLAVER unit. After the SLAVER unit data acquisition module receives the Sync message, the transmission control module controls the output module to feed back a message containing the SLAVER unit from receiving the Sync message to the MASTER unit. The time interval for sending the Sync_Responce message is Adjust and the Sync_Responce message of the SLAVER unit unit number. When the MASTER unit receives the Sync_Responce message, determine the time Tm1 and Tm2 between the MASTER unit sending the Sync message and receiving the Sync_Responce message, and the MASTER unit real-time The control module calls the formula Delay=(Tm2-Tm1-Adjust)/2 to calculate the line delay Delay of the network according to the time interval Adjust and the time Tm1 and Tm2 of sending the Sync message and receiving the Sync_Responce message. Then at Tm3 the transmission control module of the MASTER unit controls its output module to send a Sync_Start message to the SLAVER unit, and after delaying Delay clocks, the real-time control module at Ts3 controls the clock of the MASTER unit to return to zero, and after the SLAVER unit receives the Sync_Start message, The clock is also reset to zero, and the clock control unit starts timing from 0. In this way, the MASTER unit and the SLAVER unit return to zero at the same time, realizing clock synchronization.

Using this method, the clock synchronization of the master-slave unit of the 2-unit distributed network is completed only through three line transmissions. Therefore, when the distributed system environment changes, the resynchronization time is very short, which can meet the needs of the system.

FIG. 4 is a schematic diagram of a multi-unit distributed system synchronization process.

When the distributed system is multi-unit (n), the MASTER unit transmission control module controls the output module to send Sync messages to all the SLAVER units in the system at the predetermined sending interval time, and record the sending time respectively, and each SLAVER unit passes the data After the acquisition module receives the Sync message, the transmission control module controls the output module to feed back a Sync_Responce message to the MASTER unit. The Sync_Responce message contains the time interval Adjust and the SLAVER unit from receiving the Sync message to sending the Sync_Responce message from the SLAVER unit unit number. After receiving the Sync_Responce message, the data acquisition module of the MASTER unit calculates the line delay Delay of each SLAVER unit according to the above line delay calculation formula Delay=(Tm2-Tm1-Adjust)/2. According to the line delay of each SLAVER unit, determine the time Tm4-Tmk to send the Sync_Start message to each SLAVER unit, then the transmission control module of the MASTER unit controls its output module to send the Sync_Start message to each SLAVER unit, according to the time of each SLAVER unit The line delay time determines the moment when each SLAVER unit receives the message. When the line delay of the nth SLAVER unit is reached, the real-time control module controls the clock of the MASTER unit to reset to zero. At the same time, the clock of the SLAVER unit has received the Sync_Start message, and the clock also resets to zero. Make the clock of the MASTER unit and each SLAVER unit reset to zero at the same time.

When the number of distributed units increases, it may cause conflicts between the transmission and reception of the MASTER unit, so the SLAVER unit and the MASTER unit can be synchronized in batches. After testing, a line delay in the system is about 100,000 clocks, so the SLAVER units that are synchronized at a time can be limited to a predetermined number. The main control computer sends detection data to detect how many distributed units are in the network. When the main control computer detects that the number of SLAVER units in the network exceeds the predetermined number, the MASTER unit receives the Sync_Responce message time of each SLAVER unit, and The information contained in it first calculates the line delay of the previously scheduled SLAVER unit, and then calculates the line delay of other SLAVER units. The real-time control module sends control commands to make all distributed units in the system The clock is reset to zero at the same time.

Due to the deviation of the clock crystal oscillator, when the system runs for a certain period of time, the system needs to re-synchronize once. The figure shows various times when the system clock needs to be reset to zero. For example: Tsnk=0 indicates that the kth clock of the slave unit n returns to 0.

Figure 5 shows the processing flow chart of the MASTER unit. The distributed system includes two states: synchronous state and asynchronous state.

In the asynchronous state, after the system is powered on, the MASTER unit reads the configuration information in the FLASH, and initializes the system configuration and function configuration. Then the MASTER unit broadcasts the hello packet to detect whether there is a master computer on the network. If the master computer is not detected or the feedback information is not received after waiting for a predetermined time, the system directly enters the real-time state and performs real-time data transmission; when the MASTER unit receives the master computer The response packet returned by the control computer enters the MASTER control state. When MASTER is in the control state, the main control computer performs work such as network detection, system configuration and function configuration.

Detection: The main control computer broadcasts the UnitGet package to the network to detect the distributed units in the network, and the online distributed units will return a UnitInform package to the main control computer; the main control computer determines that there are How many distributed units.

System configuration: The main control computer sends a system configuration SysConfig package to the MASTER unit, and determines the SysConfig package according to the system configuration data, including the system synchronization cycle, data sending cycle, data sending length, data receiving buffer address, and data sending time.

After the MASTER unit receives the SysConfig packet, it writes the configuration data into its FLASH, and initializes the working states of the system according to the system configuration information, including the system synchronization cycle, data sending cycle, data sending length, data receiving buffer address, and data sending time; function Configuration: The main control computer sends a FunConfig package to the MASTER unit. After receiving the FunConfig package, the MASTER unit writes the configuration data into the FLASH, and determines the initial collection time and data collection rate of each synchronization cycle of each distributed unit in the network according to the function configuration information. , data acquisition channel number, data acquisition storage address, data output buffer address and other information.

After the system completes the detection and configuration, the main control computer sends the QuitCongfig package, and the MASTER unit receives the QuitCongfig package from the main control computer, exits the control state, starts in real time, and the system enters the synchronous control process shown in Figure 4. After the system is synchronized, the data transmission in the network is realized according to the system configuration configuration information and function configuration information sent by the master computer in the control state;

If the MASTER unit does not detect the main control computer, the system will directly enter the real-time data transmission state. After the system is synchronized, the data transmission will be realized according to the system configuration and function configuration already stored in the FLASH. In the state of real-time data transmission, the system according to the synchronization cycle, data sending cycle, data sending length, data receiving buffer address, data sending time, initial collection time of each synchronization cycle, data collection rate, number of data collection channels, data collection storage address and Data output buffer address for real-time data transmission.

In the stage of real-time data transmission, the initial acquisition time of each synchronization cycle of each distributed unit is the same as the acquisition rate, so that the data acquisition time of each distributed unit is the same. In this way, the collected data of other distributed units received by a certain distributed unit within a data sending cycle are all generated at the same time, and the data collected by each distributed unit can be processed.

Figure 6 shows the processing flow chart of the SLAVER unit. When the system is in the asynchronous state, after the system is initialized, it directly enters the SLAVER control state. If the SLAVER unit does not receive the Sync packet sent by the MASTER unit, the SLAVER unit is always in the SLAVER control state. At this time, the master computer can control the SLAVER unit. Detection, system configuration and function configuration, etc., the work process is the same as that of the MASTER unit. After the MASTER unit sends a Sync packet to the SLAVER unit, the SLAVER unit follows the MASTER unit into a synchronization state (as described in FIG. 4 ). After synchronization, data transmission is realized according to the real-time status function of computer configuration in the control state; if there is no system configuration and function configuration in the control state, data transmission is realized according to the system configuration and function configuration in FLASH. In the state of real-time data transmission, the system performs real-time data transmission according to the synchronization cycle, data sending cycle, data sending length, data receiving buffer address, data sending time, data acquisition rate, number of data acquisition channels, data acquisition storage address and data output buffer address .

Define the format of various messages in the system according to the system configuration. The following describes the formats of several packets with examples. Network UDP packet definition: (44332211 is an ad hoc data packet field different from other systems)

hello packet: UDP header +44332211+90

response package: UDP header +44332211+91

UnitGet package: UDP header +44332211+99

UnitInform package: UDP header + 44332211 + 9a + unit number (the MASTER unit can be set to 00, the SLAVER unit can be set to 01-ff, and the unit number of each SLAVER unit must be unique)

SysConfig package: UDP header + 44332211 + 01 + unit number (same as UnitInform package) + FLASH address (2 bytes) + configuration data (system synchronization cycle + data sending cycle + data sending length + data sending time + data receiving buffer address )+ffffffff (configuration data end flag)

FunConfig package: UDP header + 44332211 + 01 + unit number (same as UnitInform package) + FLASH address (2 bytes) + configuration data (data acquisition rate + number of data acquisition channels + data acquisition storage address + data output cache address) + ffffffff (configuration data end flag)

Sync message: UDP header + 44332211 + 70 + unit number (same as UnitInform package)

Sync_Responce message: UDP header + 44332211 + 71 + unit number (same as UnitInform package) + Delay (4 bytes)

Sync_Start message: UDP header + 44332211 + 72 + unit number (same as UnitInform package).

Claims (7)

1. A distributed network time synchronization system based on FPGA technology, comprising control and processing part, clock synchronization part, data transmission part, it is characterized in that, described control and processing part comprise network center control processing part and each distributed unit The control processing part, the control processing part of each distributed unit takes FPGA as the core, and realizes real-time synchronization of each distributed unit in the control network, and sends and receives data at regular intervals; the control processing part of the network center: the main control computer and the monitoring computer implement control processing, The main control computer detects how many distributed units there are in the network and configures the functions of each distributed unit according to the system configuration information and function configuration information. The main control computer and the monitoring computer complete the monitoring of the data packets; the clock synchronization part includes a main Distributed unit MASTER and multiple slave distributed units SLAVER, the master distributed unit determines the clock reset time of the master distributed unit and the slave distributed unit according to the network line delay of each slave distributed unit, and realizes the master and slave distributed units Synchronization among them; the data transmission part transmits detection, configuration data and synchronization initiation messages in asynchronous mode, and transmits real-time data of each distributed unit in synchronous mode 2. The distributed network time synchronization system according to claim 1, wherein each slave distributed unit real-time control module receives and sends messages according to its processing time, the time interval of sending messages and the slave distributed unit Unit number, calculate the network line delay Delay of each slave distributed unit. 3. The distributed network time synchronization system according to claim 1, wherein the system configuration information includes system synchronization cycle, data sending cycle, data sending length, data receiving cache address, data sending time; function configuration module according to function The configuration information determines the data acquisition rate, the number of data acquisition channels, the data acquisition storage address, and the data output buffer address. 4. distributed network time synchronization system according to claim 1, is characterized in that, SOPC soft-core processor is embedded in FPGA, is provided with system configuration, function configuration, transmission control, real-time control, UDP and acquisition and output module , wherein the system configuration module, function configuration module and transmission control module are realized by FPGA; the real-time control module and UDP module are realized by SOPC in FPGA. 5. A distributed network time synchronization method based on FPGA technology, characterized in that, after the distributed unit receives the synchronous initiation message sent by the main distributed unit, the real-time control module controls the output module to return a message containing the processing time The interval and the synchronization response message of the slave distributed unit number, the master distributed unit calculates the time of each slave distributed unit according to the synchronization response message, the time Tm1 of sending the synchronization initiation message, and the time Tm2 of receiving the synchronization response message. Network line delay, the real-time control module determines the clock reset time of the master distributed unit and the slave distributed unit according to the network line delay of each slave distributed unit, and realizes the synchronization between the master and slave distributed units. 6. The distributed network time synchronization method according to claim 5, wherein the message processing time interval is the time interval between receiving the synchronization initiation message and sending the synchronization response message from the distributed unit Adjust, according to the formula Delay=(Tm2-Tm1-Adjust)/2, calculates the line delay from the distributed unit network. 7. The distributed network time synchronization method according to claim 5, wherein when the distributed units increase, the distributed units and the master distributed units are synchronized in batches.

Images