CN114445260B - FPGA-based distributed GPU communication method and device - Google Patents

Abstract

The method is applied to a communication device, the communication device comprises a first FPGA processing chip, the first FPGA processing chip receives request information sent by a GPU, the request information comprises a data storage address, data is read from the data storage address of the GPU, the data is sent to a server according to configuration information received from remote resource scheduling center equipment, or the data is sent to a second FPGA processing chip according to the configuration information, the FPGA processing chip is used as a transfer card between the GPU and the server, the coupling degree between the GPU and the server is reduced, and a two-dimensional annular network topology can be formed between the FPGA processing chip and an adjacent FPGA processing chip, so that flexibility between the GPUs is greatly improved, and the time for mutual communication between the GPUs is reduced.

Description

FPGA-based distributed GPU communication method and device

Technical field

The present application relates to the field of communication technology, and in particular to a method and device for distributed GPU communication based on FPGA.

Background technique

Graphics Processing Unit (GPU) is a dedicated graphics processing chip. Whether it was used for graphics and image processing in the early days or is now widely used in the field of AI artificial intelligence computing, it is an important computing chip. As a high-speed serial computer expansion bus standard (Peripheral Component Interconnectexpress, PCIe) device, the GPU is plugged into the data center server slot and communicates with the host server and other GPU nodes through the PCIe interface.

Due to the tight coupling between the GPU and the server through the PCIe interface, the GPU cannot run independently of the server. Communication between GPUs across nodes can only be done by connecting the network card to the switch. The communication network topology is not flexible enough, the data forwarding efficiency is low, and the communication delay Time is big.

Contents of the invention

Based on this, it is necessary to provide a method and device for distributed GPU communication based on FPGA to address the above technical problems. The FPGA processing chip serves as an adapter card between the GPU and the server, which not only reduces the coupling between the GPU and the server , it can also form a two-dimensional ring network topology with adjacent FPGA processing chips, which greatly improves the flexibility of the communication network topology, reduces the time for mutual communication between GPUs, and improves data forwarding efficiency.

In a first aspect, a method of distributed GPU communication based on FPGA is provided. The method is applied to a communication device. The communication device includes a first FPGA processing chip. The first FPGA processing chip includes a first interface, a second interface, and a first network. interface and a plurality of second network interfaces; the first interface of the first FPGA processing chip is communicatively connected to the GPU, and the second interface of the first FPGA processing chip is communicatively connected to the server; the first network interface is communicatively connected to the remote resource dispatching center device; A plurality of second network interfaces are used to communicate with the second FPGA processing chip; the method includes:

Receive request information sent by the GPU. The request information includes the address of the data storage;

Read data from the data storage address of the GPU;

Send data to the server according to the configuration information received from the remote resource dispatching center device; or send data to the second FPGA processing chip according to the configuration information.

In a possible implementation, the first FPGA processing chip also includes a data processing module. The data processing module includes a GPU direct data access unit; reading data from the data storage address of the GPU includes:

Data is read from the data storage address of the GPU through the GPU direct data access unit.

In a possible implementation, the first FPGA processing chip also includes a bridge module; before sending data to the server according to the configuration information received from the remote resource dispatching center device; or before sending data to the second FPGA processing chip according to the configuration information. , methods also include:

Receive the configuration information sent by the remote resource dispatching center device through the bridge module.

In a possible implementation, the configuration information includes first indication information; sending data to the server according to the configuration information includes:

Send data to the server according to the first instruction information.

In a possible implementation, the configuration information includes second indication information; sending data to the second FPGA processing chip according to the configuration information includes:

Send data to the second FPGA processing chip according to the second instruction information.

In a possible implementation, the data processing module also includes a computing unit, and the configuration information also includes computing rule information and target network interface information; sending data to the second FPGA processing chip according to the second instruction information includes:

According to the second instruction information, the arithmetic unit uses arithmetic rules to process the data to obtain a processing result;

Send data to the second FPGA processing chip through a target network interface among the plurality of second network interfaces.

In a second aspect, a device for distributed GPU communication based on FPGA is provided. The device includes a first FPGA processing chip. The first FPGA processing chip includes a first interface, a second interface, a first network interface and a plurality of first interfaces. Network interface; the first interface of the first FPGA processing chip is communicatively connected to the GPU, and the second interface of the first FPGA processing chip is communicatively connected to the server; the first network interface is communicatively connected to the remote resource dispatching center device; a plurality of second network interfaces Used for communication connection with the second FPGA processing chip; the first FPGA processing chip is used for:

Receive request information sent by the GPU through the first interface, where the request information includes the address of the data storage;

Read data from the data storage address of the GPU;

Send data to the server according to the configuration information received from the remote resource dispatching center device through the first network interface; or send data to the second FPGA processing chip according to the configuration information; the second network interface is used to communicate with the second FPGA processing chip Communication connection.

In a possible implementation, the first FPGA processing chip also includes a data processing module, and the data processing module includes a GPU direct data access unit; the first FPGA processing chip is specifically used for:

Data is read from the data storage address of the GPU through the GPU direct data access unit.

In a possible implementation, the first FPGA processing chip also includes a bridge module; the first FPGA processing chip is specifically used for:

Receive the configuration information sent by the remote resource dispatching center device through the bridge module.

In a possible implementation, the configuration information includes first indication information; the first FPGA processing chip is specifically used for:

Send data to the server according to the first instruction information.

The above-mentioned FPGA-based distributed GPU communication method and device, by receiving request information sent by the GPU, the request information includes a data storage address, reads data from the data storage address of the GPU, and reads data from the data storage address of the GPU according to the configuration received from the remote resource dispatching center device. Information is sent to the server, or data is sent to the second FPGA processing chip according to the configuration information. The FPGA processing chip serves as an adapter card between the GPU and the server, which not only reduces the coupling between the GPU and the server, but also reduces the server-side CPU, The overhead of resources such as memory and network can also form a two-dimensional ring network topology with adjacent FPGA processing chips, which greatly improves the flexibility between GPUs and reduces the time for mutual communication between GPUs.

Description of the drawings

Figure 1 is an application environment diagram of the FPGA-based distributed GPU method in one embodiment of the present application;

Figure 2 is a structural block diagram of the first FPGA processing chip in one embodiment;

Figure 3 is a schematic flowchart of an FPGA-based distributed GPU method in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

In the existing technology, the GPUDirect series is a GPU direct memory access technology that allows the GPU to access the memory of other sub-devices or hosts through the PCIe chipset bus system, thereby reducing unnecessary memory copies and reducing the cost of Central Processing. Unit, CPU) usage overhead to improve data transmission efficiency. Among them, GPUDirectShared Memory is a GPU direct shared memory technology of the GPUDirect series, which enables the GPU to share the memory pages of the host with other PCIe devices to achieve data communication between different PCIe devices.

GPUDirect P2P is a technology of the GPUDirect series that directly shares video memory between GPUs. It refers to two GPU devices in the same PCIe Root Complex (PCIe Root Complex, PCIe RC) domain directly accessing GPU video memory. There is no need to copy data between GPUs. Transfer to the host memory. Compared with GPU Direct Shared Memory technology, it reduces the steps of copying data from GPU memory to host memory and from host memory to GPU memory, reduces data path delay, and improves data transmission efficiency.

GPU Direct Remote Direct Memory Access (GPUDirect RDMA) technology uses RDMA related technologies, physical network cards, and transmission networks to realize direct interaction of video memory data between GPUs. This technology solves the problem of large processing delays in traditional network data transmission. , high CPU usage and other problems, the function of direct mutual access to GPU memory between different nodes is realized.

GPUDirect Shared Memory technology and GPUDirect P2P technology are both developed and implemented based on the GPU as a PCIe device under the host. Communication with other devices relies on the participation of the CPU, host memory and PCIe switching system. The device and server CPU, memory, etc. are tightly coupled through PCIe, and communication Limited to GPU within a single node. When using GPUDirect SharedMemory technology to exchange video memory data between GPUs in a single node, it needs to go through each CPU and CPU1 memory and other modules, resulting in large additional CPU overhead and large data transmission and processing delays. When GPUDirect P2P technology is used to exchange video memory data between GPUs, the direct interaction of video memory data between GPUs in the same PCIe RC domain is limited to the PCIe chipset. If the PCIe RC domain spans two CPUs, the CPU and CPU memory are also required to participate in data transmission. , causing the memory data exchange delay and CPU overhead between GPUs to still be very large.

Although GPUDirect RDMA technology uses RDMA technology to realize cross-node GPU communication problems, it requires the participation of high-performance network cards in the same PCIe domain and local server CPUs to help the GPU complete cross-node data transmission. The GPU and server still use PCIe Due to the tight coupling relationship of the connections, the GPU cannot run independently of the server. Communication between GPUs across nodes can only be done by connecting the network card to the switch. The communication network topology is not flexible enough, the data packet forwarding efficiency is low, and the communication delay is large.

In order to solve the existing technical problems, embodiments of the present application provide a method and device for distributed GPU communication based on FPGA. The following first introduces the FPGA-based distributed GPU communication method provided by the embodiment of the present application. This method is applied to the application environment shown in Figure 1. As shown in Figure 1, the communication device 100 includes multiple FPGA processing chips, each of which is Each FPGA processing chip includes multiple network interfaces. The first network interface 230 is connected to the remote resource dispatching center device through a switch. The second network interface 240 forms a 2D ring communication topology with other surrounding FPGA processing chips. The second network interface 240 The number can be adjusted according to user needs, and is not limited to 4. The FPGA processing chip can send data to the adjacent FPGA processing chip through the second network interface 240, or receive data sent by the adjacent FPGA processing chip. The first network interface and the second network interface are 100G network optical ports, enabling efficient communication between GPUs and between GPUs and remote resource dispatching center equipment.

Any one of the FPGA processing chips is defined as the first FPGA processing chip, and the FPGA processing chip connected to the first FPGA processing chip is defined as the second FPGA processing chip. The first FPGA processing chip and the second FPGA processing chip have the same structure. .

As shown in Figure 2, the first FPGA processing chip 200 includes a first interface 210, a second interface 220, a first network interface 230 and a plurality of second network interfaces 240; the first interface 210 of the first FPGA processing chip communicates with the GPU. The second interface 220 of the first FPGA processing chip 200 is connected to the server; the first network interface 230 is connected to the remote resource dispatching center device; and the plurality of second network interfaces 240 are used to communicate with the second FPGA processing chip.

The first FPGA processing chip 200 also includes an instruction configuration module 270, a routing module 280, a first transceiver module 290, and a second transceiver module 2100. The first transceiver module 290 is a RoCE transceiver module, together with the second network interface 240 and the routing module. 280 communication connection, receives the data packet from the adjacent FPGA processing chip through the RoCE protocol, parses the data packet, packages the data of the first FPGA processing chip, and sends the data packet to the adjacent FPGA processing chip.

The second transceiver module 2100 is a RoCEv2 transceiver module, communicates with the first network interface 230, the routing module 280 and the instruction configuration module 270, receives the configuration information sent by the remote resource dispatching center device to the first network interface 230 through the RoCEv2 protocol, and Parse the configuration information and distribute the parsing results to the corresponding modules. For example, send the algorithm information to the instruction configuration module 270 to complete tasks such as registration and initialization of GPU resources. At the same time, the GPU data connected to the first FPGA processing chip can also be packaged and connected to the switch through the first network interface 230 to communicate with the GPUs connected to other FPGA processing chips.

After the first FPGA processing chip is powered on, the routing module 280 communicates with the adjacent FPGA processing chip through the second network interface 240, obtains the MAC address information of the second network interface 240 of the adjacent FPGA processing chip, and saves it to the memory. in order to communicate via the RoCE protocol.

Figure 3 shows a schematic flowchart of a method for providing FPGA-based distributed GPU communication according to one embodiment of the present application. As shown in Figure 3, the method may include the following steps:

S310: Receive request information sent by the GPU, where the request information includes the data storage address.

Before data communication, the remote resource dispatching center equipment initializes the communication device, sends the data to be processed to the GPU through the server, and the GPU processes the received data, saves the processed data, and sends request information to the communication device. So that the communication device transmits the data saved by the GPU. The request information includes a data storage address so that the communication device can accurately obtain the data that the GPU needs to transmit.

The communication device receives the request information sent by the GPU through the first interface 210 of the first FPGA processing chip 200, where the first interface 210 is a PCIe interface and is connected to the PCIe interface of the GPU using Gen5x16 standard golden finger in Root Point mode.

S320: Read data from the data storage address of the GPU.

The first FPGA processing chip uses Direct Memory Access (DMA) to read data from the data storage address in the GPU memory according to the received data storage address, so that the data transmission delay is small. Improve data reading efficiency.

S330: Send data to the server according to the configuration information received from the remote resource dispatching center device; or send data to the second FPGA processing chip according to the configuration information.

The configuration information includes switching information, which determines the data transmission path. When the switching information is communicated from the GPU to the server, the first FPGA processing chip sends data to the server through the second interface 220. Since the FPGA-based FPGA is introduced between the GPU and the server The communication device of the processing chip reduces the coupling between the GPU and the server and facilitates independent pool management of the GPU. The second interface 220 is a PCIe interface in Endpoint mode, and is connected to the PCIe interface of the GPU using the Gen5x16 standard to match the communication interface mode of the GPU.

When the switching information is communication from the GPU to other FPGA processing chips, the first FPGA processing chip sends data to the second FPGA processing chip, and the GPU communicates with other FPGA processing chips through multiple network interfaces of the FPGA processing chip. The network topology of the GPU communication is more flexible. . Communicating with multiple FPGA processing chips through multiple network interfaces enables simultaneous calculation and transmission of data in multiple dimensions, reducing the number of data calculations and transmissions through the PCIe bus, reducing the time required for data update, thereby reducing data communication time overhead.

In the embodiment of this application, by receiving the request information sent by the GPU, the request information includes the data storage address, reading the data from the data storage address of the GPU, and sending the data to the server according to the configuration information received from the remote resource dispatching center device, Or send data to the second FPGA processing chip according to the configuration information. The FPGA processing chip serves as an adapter card between the GPU and the server, which not only reduces the coupling between the GPU and the server, but also reduces the consumption of server-side CPU, memory, network and other resources. Overhead, it can also form a two-dimensional ring network topology with adjacent FPGA processing chips, which greatly improves the flexibility between GPUs and reduces the time for mutual communication between GPUs.

In some embodiments, the first FPGA processing chip also includes a data processing module 250. The data processing module includes a GPU direct data access unit 251; reading data from the data storage address of the GPU includes:

The data is read from the data storage address in the GPU memory through the GPU direct data access unit 251. The first FPGA processing chip 200 directly accesses the GPU internal memory through PCIe and the GPU direct data access unit 251 to read the data to be transmitted, effectively Reduced communication latency between GPUs.

In some embodiments, the first FPGA processing chip 200 also includes a bridge module 260; before sending data to the server according to the configuration information received from the remote resource dispatching center device; or before sending data to the second FPGA processing chip according to the configuration information, Methods also include:

The configuration information sent by the remote resource dispatching center device is received through the bridge module 260.

Through the bridge module 260 connecting with the instruction configuration module 270, the first interface 210, the second interface 220 and the data processing module, the remote resource dispatching center device sends the configuration information to the first network interface 230 through the switch network, and the first network interface 230 The configuration information is parsed to obtain the switching PCIe RC signal, and the switching PCIe RC signal is sent to the instruction configuration module 270. The instruction configuration module 270 judges the switching PCIe RC signal, and sends the judgment result to the bridge module 260. The bridge module 260 determines the result according to the judgment result. You can switch the connection relationship of the GPU's PCIe bus.

In the initialization stage of the communication device, the remote resource dispatching center device sends configuration information to the first FPGA processing chip through the switch network, controls and switches the connection relationship of the PCIe bus of the GPU, determines the data transmission path, flexibly selects the transmission object of the data in the GPU, and releases the GPU. Dependency on the server host.

In some embodiments, the configuration information includes first indication information; sending data to the server according to the configuration information includes:

Send data to the server according to the first instruction information.

The instruction configuration module 270 receives the switching PCIe RC signal in the configuration information and analyzes it to determine whether the value corresponding to the switching PCIe RC signal is 1 or 0. Among them, the first indication information is 0. When the parsing result of the configuration information is 0, the GPU and the server perform data interaction, and the bridge module 260 directly sends the data to the server.

The configuration information includes second indication information; sending the data to the second FPGA processing chip according to the configuration information includes:

Send the data to the second FPGA processing chip according to the second instruction information.

The second indication information is 1. When the parsing result of the configuration information is 1, the GPU interacts with the second FPGA processing chip. The bridge module 260 first sends the data to the data processing module 250, and the data processing module 250 processes it. The processed data is sent to the first transceiver module 290, and finally sent to the second FPGA processing chip through the second network interface 240.

In some embodiments, the data processing module 250 also includes an operation unit 252, and the configuration information also includes operation rule information and target network interface information; sending data to the second FPGA processing chip according to the second instruction information includes:

According to the second instruction information, the operation module uses operation rules to process the data to obtain the processing result;

Send data to the second FPGA processing chip through a target network interface among the plurality of second network interfaces.

During the data interaction between the GPU and the second FPGA processing chip, if the receiving unit of the data processing module 250 does not receive the data to be processed sent by the FPGA processing chip corresponding to the other GPU, the instruction configuration module 270 controls the computing unit 252 according to the configuration information. Without any calculation, the data read from the GPU by the GPU direct data access unit 251 is directly sent to the sending unit 253. The sending unit 253 sends the data to the first transceiver module 290, and the first transceiver module 290 obtains it from the routing module 280. The AMC address information of the target network interface in the configuration information sends the data to other FPGA processing chips through the target network interface.

If the receiving unit of the data processing module receives data to be processed sent by FPGA processing chips corresponding to other GPUs, the instruction configuration module 270 controls the computing unit 252 to perform corresponding calculations according to the information of the computing rules, and the GPU direct data access unit 251 obtains the data from the GPU. After receiving the data, the data is sent to the computing unit 252. The computing unit 252 performs mixed calculations on the data read by the GPU and the data sent by the FPGA processing chip received through the receiving unit 254 according to the preconfigured computing rules, and sends the calculation results to The sending unit 253 sends the data to the first transceiver module 290. The first transceiver module 290 obtains the AMC address information of the target network interface in the configuration information from the routing module 280, and sends the AMC address information to other FPGA processing chips through the target network interface. The data is written by the FPGA processing chip into the corresponding GPU display memory through the GPU direct data access unit 251 to complete the update iteration of the GPU data.

In one embodiment, a device for distributed GPU communication based on FPGA is provided. The device includes a first FPGA processing chip. The first FPGA processing chip includes a first interface, a second interface, a first network interface and a plurality of The first network interface; the first interface of the first FPGA processing chip is communicatively connected to the GPU, and the second interface of the first FPGA processing chip is communicatively connected to the server; the first network interface is communicatively connected to the remote resource dispatching center device; a plurality of second The network interface is used for communication connection with the second FPGA processing chip; the first FPGA processing chip is used for:

Receive request information sent by the GPU through the first interface, where the request information includes the address of the data storage;

Read data from the data storage address of the GPU;

Send data to the server according to the configuration information received from the remote resource dispatching center device through the first network interface; or send data to the second FPGA processing chip according to the configuration information; the second network interface is used to communicate with the second FPGA processing chip Communication connection.

In the embodiment of this application, the first FPGA processing chip serves as an adapter card between the GPU and the server, which not only reduces the coupling between the GPU and the server, but also forms a two-dimensional ring network topology with adjacent FPGA processing chips. , which greatly improves the flexibility of the communication network topology, reduces the time for GPUs to communicate with each other, and improves data forwarding efficiency.

In one embodiment, the first FPGA processing chip also includes a data processing module, and the data processing module includes a GPU direct data access unit; the first FPGA processing chip is specifically used for:

Data is read from the data storage address of the GPU through the GPU direct data access unit.

In one embodiment, the first FPGA processing chip also includes a bridge module; the first FPGA processing chip is specifically used for:

Receive the configuration information sent by the remote resource dispatching center device through the bridge module.

In one embodiment, the configuration information includes first indication information; the first FPGA processing chip is specifically used for:

Send data to the server according to the first instruction information.

In some embodiments, the configuration information includes second indication information; the first FPGA processing chip is specifically used for:

Send data to the second FPGA processing chip according to the second instruction information.

In some embodiments, the data processing module also includes an operation unit, and the configuration information also includes operation rule information and target network interface information; the first FPGA processing chip is specifically used for:

According to the second instruction information, the arithmetic unit uses arithmetic rules to process the data to obtain a processing result;

Send data to the second FPGA processing chip through a target network interface among the plurality of second network interfaces.

The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims (7)

1. A method of distributed GPU communication based on FPGA, wherein the method is applied to a communication device, the communication device comprising a first FPGA processing chip comprising a first interface, a second interface, a first network interface and a plurality of second network interfaces; the first interface of the first FPGA processing chip is in communication connection with the GPU, and the second interface of the first FPGA processing chip is in communication connection with the server; the first network interface is in communication connection with remote resource scheduling center equipment; the plurality of second network interfaces are used for being in communication connection with a second FPGA processing chip; the method comprises the following steps:

receiving request information sent by the GPU, wherein the request information comprises an address of data storage;

reading data from the data storage address of the GPU;

transmitting the data to the server according to configuration information received from the remote resource scheduling center device; or sending the data to the second FPGA processing chip according to the configuration information; the configuration information includes first indication information and second indication information, the first indication information and the second indication information are switching PCIe RC signals, and sending the data to the server according to the configuration information includes:

according to the first indication information, sending the data to the server;

and sending the data to the second FPGA processing chip according to the configuration information, wherein the data comprises:

and sending the data to the second FPGA processing chip according to the second indication information.

2. The method of claim 1, wherein the first FPGA processing chip further comprises a data processing module comprising a GPU direct data access unit; the reading data from the data storage address of the GPU includes:

and reading data from the data storage address of the GPU through the GPU direct data access unit.

3. The method of claim 1 or 2, wherein the first FPGA processing chip further comprises a bridge module; transmitting the data to the server according to the configuration information received from the remote resource scheduling center device; or before sending the data to the second FPGA processing chip according to the configuration information, the method further includes:

and receiving the configuration information sent by the remote resource scheduling center equipment through the bridge module.

4. The method of claim 1, wherein the data processing module further comprises an operation unit, and the configuration information further comprises information of an operation rule and information of a target network interface; and sending the data to the second FPGA processing chip according to the second instruction information, including:

according to the second indication information, the operation unit processes the data by adopting the operation rule to obtain a processing result;

and sending the data to the second FPGA processing chip through the target network interface in the second network interfaces.

5. An FPGA-based distributed GPU communication apparatus, wherein the apparatus includes a first FPGA processing chip including a first interface, a second interface, a first network interface, and a plurality of second network interfaces; the first interface of the first FPGA processing chip is in communication connection with the GPU, and the second interface of the first FPGA processing chip is in communication connection with the server; the first network interface is in communication connection with remote resource scheduling center equipment; the plurality of second network interfaces are used for being in communication connection with a second FPGA processing chip; the first FPGA processing chip is configured to:

receiving request information sent by the GPU through the first interface, wherein the request information comprises an address of data storage;

reading data from the data storage address of the GPU;

transmitting the data to the server according to configuration information received from the remote resource scheduling center device through the first network interface; or sending the data to the second FPGA processing chip according to the configuration information; the configuration information includes first indication information and second indication information, the first indication information and the second indication information are switching PCIe RC signals, and sending the data to the server according to the configuration information includes:

according to the first indication information, sending the data to the server;

and sending the data to the second FPGA processing chip according to the configuration information, wherein the data comprises:

and sending the data to the second FPGA processing chip according to the second indication information.

6. The apparatus of claim 5, wherein the first FPGA processing chip further comprises a data processing module comprising a GPU direct data access unit; the first FPGA processing chip is specifically configured to:

and reading data from the data storage address of the GPU through the GPU direct data access unit.

7. The apparatus of claim 5 or 6, wherein the first FPGA processing chip further comprises a bridge module; the first FPGA processing chip is specifically configured to:

and receiving the configuration information sent by the remote resource scheduling center equipment through the bridge module.