Explore more resources

Altera® Design Hub

Multi Channel DMA Intel® FPGA IP

for PCI Express* Design Example
User Guide
Updated for Quartus® Prime Design Suite: 24.3

Online Version 683517

Send Feedback UG-20303 2024.11.04
 Contents

Contents

1. Introduction................................................................................................................... 4
1.1. Terms and Acronyms..............................................................................................4
1.2. MCDMA IP Modes...................................................................................................5
2. Design Example Detailed Description.............................................................................. 7
2.1. Design Example Overview.......................................................................................7
2.1.1. H-Tile MCDMA IP - Design Examples for Endpoint........................................... 7
2.1.2. P-Tile MCDMA IP - Design Examples for Endpoint........................................... 9
2.1.3. F-Tile MCDMA IP - Design Examples for Endpoint..........................................11
2.1.4. R-Tile MCDMA IP - Design Examples for Endpoint......................................... 13
2.1.5. Design Example BAR Mappings.................................................................. 15
2.2. Hardware and Software Requirements.................................................................... 16
2.3. PIO using MCDMA Bypass Mode............................................................................. 16
2.3.1. Avalon-ST PIO using MCDMA Bypass Mode.................................................. 16
2.3.2. Avalon-MM PIO using MCDMA Bypass Mode................................................. 17
2.4. Avalon-ST Packet Generate/Check..........................................................................19
2.4.1. Single-Port Avalon-ST Packet Generate/Check..............................................19
2.5. Avalon-ST Device-side Packet Loopback.................................................................. 22
2.5.1. Simulation Results................................................................................... 23
2.5.2. Hardware Test Results.............................................................................. 24
2.6. Avalon-MM DMA...................................................................................................27
2.6.1. Simulation Results................................................................................... 28
2.6.2. Hardware Test Results.............................................................................. 29
2.7. BAM_BAS Traffic Generator and Checker................................................................. 32
2.7.1. BAM_BAS Traffic Generator and Checker Example Design Register Map........... 33
2.8. External Descriptor Controller................................................................................36
2.8.1. Registers................................................................................................ 40
2.8.2. Hardware Test Results.............................................................................. 42
3. Design Example Quick Start Guide................................................................................ 43
3.1. Design Example Directory Structure....................................................................... 43
3.2. Generating the Example Design using Quartus Prime................................................ 45
3.2.1. Procedure............................................................................................... 45
3.3. Simulating the Design Example..............................................................................47
3.3.1. Testbench Overview................................................................................. 47
3.3.2. Supported Simulators............................................................................... 48
3.3.3. Example Testbench Flow for DMA Test with Avalon-ST Packet Generate/
Check Design Example..............................................................................52
3.3.4. Run the Simulation Script..........................................................................54
3.3.5. Steps to Run the Simulation...................................................................... 54
3.3.6. View the Results...................................................................................... 55
3.4. Compiling the Example Design in Quartus Prime...................................................... 55
3.5. Running the Design Example Application on a Hardware Setup...................................56
3.5.1. Program the FPGA....................................................................................56
3.5.2. Quick Start Guide.................................................................................... 57
4. Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide
Archives................................................................................................................ 108

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
2
Contents

5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design
Example User Guide...............................................................................................109

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
3
 683517 | 2024.11.04

Send Feedback

1. Introduction

1.1. Terms and Acronyms

Table 1. Acronyms
Term Definition

Avalon®-MM Avalon Memory-Mapped Interface

Avalon-ST Avalon Streaming Interface

CvP Configuration via Protocol

DMA Direct Memory Access

DPDK Data Plane Development Kit

D2H Device-to-Host

D2HDM Device-to-Host Data Mover

EOF End of a File (or packet) for streaming

GCSR General Control and Status Register

Gen1 PCIe 1.0

Gen2 PCIe 2.0

Gen3 PCIe 3.0

Gen4 PCIe 4.0

Gen5 PCIe 5.0

HIP Hard IP

HIDX Queue Head Index (pointer)

H2D Host-to-Device

H2DDM Host-to-Device Data Mover

IMMWR Immediate Write Operation

IP Intellectual Property

MCDMA Multi Channel Direct Memory Access

MRRS Maximum Read Request Size

PBA Pending Bit Array

PCIe* Peripheral Component Interconnect Express (PCI Express*)

PD Packet Descriptor

PIO Programmed Input/Output

SOF Start of a File (or packet) for streaming

continued...

© Altera Corporation. Altera, the Altera logo, the ‘a’ logo, and other Altera marks are trademarks of Altera
Corporation. Altera and Intel warrant performance of its FPGA and semiconductor products to current
specifications in accordance with Altera’s or Intel's standard warranty as applicable, but reserves the right to ISO
make changes to any products and services at any time without notice. Altera and Intel assume no 9001:2015
responsibility or liability arising out of the application or use of any information, product, or service described Registered
herein except as expressly agreed to inwriting by Altera or Intel. Altera and Intel customers are advised to
obtain the latest version of device specifications before relying on any published information and before placing
orders for products or services.
*Other names and brands may be claimed as the property of others.
1. Introduction
683517 | 2024.11.04

Term Definition

QCSR Queue Control and Status register

QID Queue Identification

TIDX Queue Tail Index (pointer)

TLP Transaction Layer Packet

UIO User Space Input/Output

VFIO Virtual Function Input/Output

1.2. MCDMA IP Modes

The following table summarizes the MCDMA IP variants, IP mode and FPGA
Development Kit board supported for design example hardware test.

Table 2. MCDMA IP Modes and FPGA Development Kit for Design Examples
MCDMA IP IP Mode FPGA Development
Kit Board for Design
PCI Express Application Data Application Clock Example Hardware
Width Frequency Test

MCDMA H-Tile Gen3 x16 512 bits 250 MHz Stratix® 10 GX H-Tile
Production FPGA
Gen3 x8 256 bits 250 MHz Development Kit
Stratix 10 MX H-Tile
Production FPGA
Development Kit

MCDMA P-Tile Gen4x16 512 bits Stratix 10 DX: Intel Stratix 10 DX

400/350/200/175 MHz FPGA Development Kit
Agilex™ 7: (Production)
500/450/400/350/250 Intel Agilex 7 FPGA F-
/225/200/175 MHz Series Development
Kit (Production 1 P-
Gen4 1x8 256 bits Stratix 10 DX: Tile and E-Tile)
Gen4 2x8 400/350/200/175 MHz
Agilex 7:
500/450/400/350/250
/225/200/175 MHz

Gen3 x16 512 bits 250 MHz

Gen3 1x8 256 bits 250 MHz

Gen3 2x8

MCDMA R-Tile Gen5 2x8, Gen4 x16 512 bits 500/475/450/425/400 Intel Agilex 7 FPGA I-
MHz Series Development
Kit (ES 2 x R-Tile and
Gen4 2x8, Gen3 x16, 512 bits 300/275/250 MHz 1 x F-Tile)
Gen3 2x8 Intel Agilex 7 FPGA I-
Series Development
Gen5 4x4, Gen4 2x8 256 bits 500/475/450/425/400
Kit (ES1 2 x R-Tile and
MHz
1 x F-Tile)
Gen3 2x8, Gen4 4x4, 256 bits 300/275/250 MHz
Gen3 4x4

Gen4 4x4 128 bits 500/475/450/425/400

MHz

Gen3 4x4 128 bits 300/275/250 MHz

continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
5
 1. Introduction
683517 | 2024.11.04

MCDMA IP IP Mode FPGA Development

Kit Board for Design
PCI Express Application Data Application Clock Example Hardware
Width Frequency Test

MCDMA F-Tile Gen4 1x16 512 bits 500/400/350/250/225 Intel Agilex 7 FPGA F-
/200/175 MHz Series Development
Kit (ES1 2 x F-Tile)
Gen4 1x8 256 bits 500/400/350/250/225
Gen4 2x8 /200/175 MHz

Gen3 1x16 512 bits 250 MHz

Gen3 1x8 256 bits 250 MHz

Gen3 2x8

Gen4 1x4 (Endpoint 128 bits 350/400/450/500 MHz

mode)

Gen3 1x4 (Endpoint 128 bits 250 MHz

mode)

Note: Intel Agilex 7 FPGA I-Series Development Kit (ES 2 x R-Tile and 1 x F-Tile) R-Tile A0
die revision supports only Gen5 2x8 / 512 bit, Gen4 2x8 / 512bits and Gen3 2x8 / 512
bits.

Note: Intel Agilex 7 FPGA I-Series Development Kit (ES1 2 x R-Tile and 1 x F-Tile) R-Tile B0
die revision supports all PCIe Hard IP Modes defined in MCDMA R-Tile row

For more information about MCDMA IP, refer to the Multi Channel DMA Intel FPGA IP
for PCI Express User Guide.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
6
 683517 | 2024.11.04

Send Feedback

2. Design Example Detailed Description

2.1. Design Example Overview

The Multi Channel DMA Intel® FPGA IP for PCI Express Design Examples demonstrate
a Multi Channel DMA solution for Stratix 10 GX/MX devices using the H-Tile PCIe Gen3
hard IP, Stratix 10 DX devices using the P-Tile PCIe Gen4 hard IP, and Agilex 7 devices
using the P-Tile or F-Tile PCIe Gen4 Hard IP or R-Tile PCIe Gen5 Hard IP and soft IP
implemented in the FPGA fabric.

You can generate the design example from the Example Designs tab of the Multi
Channel DMA for PCI Express IP Parameter Editor. The desired user interface type,
either Avalon-ST or Avalon-MM, can be chosen. You can allocate up to 2048 DMA
channels (with a maximum of 512 channels per function) when the Avalon-MM
interface type or Avalon-ST 1-port interface is selected.

2.1.1. H-Tile MCDMA IP - Design Examples for Endpoint

Table 3. H-Tile MCDMA IP - Design Examples for Endpoint
MCDMA Settings
Design Example Driver Support App Support
User Mode Interface Type

Perfq app (Custom PIO

Read Write Test, Verifying
Custom
on AVMM DMA and BAM
Test)
Multi-Channel DMA
AVMM Mcdma_test (DPDK PIO
BAM + MCDMA
Test, DPDK BAM Test and
DPDK Verifying on AVMM)
Testpmd (Testpmd - DMA
TX only)
AVMM DMA
Perfq app (Custom PIO
Read Write Test, Verifying
Custom
on AVMM DMA, BAM Test
and BAS Test)

BAM + BAS + MCDMA AVMM Mcdma_test (DPDK PIO

Test, DPDK BAM Test,
Verifying on AVMM and
DPDK DPDK BAS Test)
Testpmd (Testpmd - DMA
TX only)

Perfq app (Custom PIO

Custom Read Write Test, BAM Test
Multi-Channel DMA and DMA Loopback Test)
Device-side Packet
AVST 1 Port
Loopback BAM + MCDMA Mcdma_test (DPDK PIO
DPDK Test, DPDK BAM Test and
DPDK DMA Loopback Test)
continued...

© Altera Corporation. Altera, the Altera logo, the ‘a’ logo, and other Altera marks are trademarks of Altera
Corporation. Altera and Intel warrant performance of its FPGA and semiconductor products to current
specifications in accordance with Altera’s or Intel's standard warranty as applicable, but reserves the right to ISO
make changes to any products and services at any time without notice. Altera and Intel assume no 9001:2015
responsibility or liability arising out of the application or use of any information, product, or service described Registered
herein except as expressly agreed to inwriting by Altera or Intel. Altera and Intel customers are advised to
obtain the latest version of device specifications before relying on any published information and before placing
orders for products or services.
*Other names and brands may be claimed as the property of others.
 2. Design Example Detailed Description
683517 | 2024.11.04

MCDMA Settings
Design Example Driver Support App Support
User Mode Interface Type

Testpmd (Testpmd-DMA)

Netdev_app (Netdev - PIO)

Netdev
Ping (Netdev - DMA)

Perfq app (Custom PIO

Read Write Test, DMA
Custom
Loopback Test, BAM Test
and BAS Test)

Mcdma_test (DPDK PIO

BAM + BAS + MCDMA AVST 1 Port Test, DPDK BAM Test,
DPDK DPDK DMA Loopback Test
and DPDK BAS Test)
Testpmd (Testpmd - DMA)

Netdev_app (Netdev - PIO)

Netdev
Ping (Netdev - DMA)

Perfq app (Custom PIO

Read Write Test, BAM Test
Custom
and Packet Gen Test -
DMA)
Multi-Channel DMA
AVST 1 Port Mcdma_test (DPDK PIO
BAM + MCDMA
Test, DPDK BAM Test and
DPDK DPDK - Packet Gen/Check
DMA)
Testpmd (Testpmd - DMA)
Packet Generate/
Check
Perfq app (Custom PIO
Read Write Test, BAM Test,
Custom
Packet Gen Test - DMA and
BAS Test)
BAM + BAS + MCDMA AVST 1 Port Mcdma_test (DPDK PIO
Test, DPDK BAM Test,
DPDK DPDK DMA Loopback Test
and DPDK BAS Test)
Testpmd (Testpmd - DMA)

Perfq app (Custom PIO

Multi-Channel DMA Custom
AVMM Read Write Test)
BAM + MCDMA
AVST 1 Port Mcdma_test (DPDK PIO
BAM + BAS + MCDMA DPDK
Test)

Perfq app (Custom PIO

Custom
Read Write Test)
PIO using MQDMA
Bursting Master n/a
Bypass Mode
Mcdma_test (DPDK PIO
DPDK
Test)

Perfq app (Custom PIO

Custom
Read Write Test)
BAM + BAS n/a
Mcdma_test (DPDK PIO
DPDK
Test)

Perfq app (Custom PIO

Custom Read Write Test and BAS
Traffic Generator/ Test)
BAM + BAS n/a
Checker
Mcdma_test (DPDK PIO
DPDK
Test and DPDK BAS Test)

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
8
2. Design Example Detailed Description
683517 | 2024.11.04

Note: 1. MCDMA H-Tile design example supports SR-IOV with only 1 PF and its VFs for
simulation in these configurations.
2. Netdev driver supports 4 PFs and 1 channel per PF.

For information about supported simulators, refer to Supported Simulators on page

48.

2.1.2. P-Tile MCDMA IP - Design Examples for Endpoint

Table 4. P-Tile MCDMA IP - Design Examples for Endpoint
Design Example MCDMA Settings Driver Support App Support

User Mode Interface Type

Multi-Channel DMA AVMM Custom Perfq app (Custom PIO

BAM + MCDMA Read Write Test,
Verifying on AVMM DMA
and BAM Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test
and Verifying on AVMM)
Testpmd (Testpmd -
DMA TX only)
AVMM DMA
BAM + BAS + MCDMA AVMM Custom Perfq app (Custom PIO
Read Write Test,
Verifying on AVMM DMA,
BAM Test and BAS Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test,
Verifying on AVMM and
DPDK BAS Test)
Testpmd (Testpmd -
DMA TX only)

Multi-Channel DMA AVST 1 Port Custom Perfq app (Custom PIO

BAM + MCDMA Read Write Test, BAM
Test and DMA Loopback
Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test
and DPDK DMA
Loopback Test)
Testpmd (Testpmd-DMA)

Netdev Netdev_app (Netdev -

PIO)
Device-side Packet
Loopback Ping (Netdev - DMA)

BAM + BAS + MCDMA AVST 1 Port Custom Perfq app (Custom PIO
Read Write Test, DMA
Loopback Test, BAM Test
and BAS Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test,
DPDK DMA Loopback
Test and DPDK BAS
Test)
Testpmd (Testpmd -
DMA)
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
9
 2. Design Example Detailed Description
683517 | 2024.11.04

Design Example MCDMA Settings Driver Support App Support

User Mode Interface Type

Netdev Netdev_app (Netdev -

PIO)
Ping (Netdev - DMA)

Multi-Channel DMA AVST 1 Port Custom Perfq app (Custom PIO

BAM + MCDMA Read Write Test, BAM
Test and Packet Gen
Test - DMA)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test
and DPDK - Packet Gen/
Check DMA)
Testpmd (Testpmd -
DMA)
Packet Generate/
Check BAM + BAS + MCDMA AVST 1 Port Custom Perfq app (Custom PIO
Read Write Test, BAM
Test, Packet Gen Test -
DMA and BAS Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test,
DPDK DMA Loopback
Test and DPDK BAS
Test)
Testpmd (Testpmd -
DMA)

PIO using MQDMA Multi-Channel DMA AVMM Custom Perfq app (Custom PIO
Bypass Mode BAM + MCDMA AVST 1 Port Read Write Test)
BAM + BAS + MCDMA DPDK Mcdma_test (DPDK PIO
Test)

Bursting Master n/a Custom Perfq app (Custom PIO

Read Write Test)

DPDK Mcdma_test (DPDK PIO

Test)

BAM + BAS n/a Custom Perfq app (Custom PIO

Read Write Test)

DPDK Mcdma_test (DPDK PIO

Test)

Data Mover Only AVMM Custom Perfq app (Custom PIO

Read Write Test)

DPDK Mcdma_test (DPDK PIO

Test)

Traffic Generator/ BAM + BAS n/a Custom Perfq app (Custom PIO
Checker Read Write Test and BAS
Test)

DPDK Mcdma_test (DPDK PIO

Test and DPDK BAS
Test)

External Descriptor Data Mover Only AVMM Custom Perfq app (External
Controller Descriptor Mode
Verification)

Note: P-Tile MCDMA IP design example doesn’t support multiple physical functions and SR-
IOV for simulation.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
10
2. Design Example Detailed Description
683517 | 2024.11.04

Note: For 2x8 Hard IP modes, simulation is supported on PCIe0 only.

For information about supported simulators, refer to Supported Simulators on page

48.

2.1.3. F-Tile MCDMA IP - Design Examples for Endpoint

Table 5. F-Tile MCDMA IP - Design Examples for Endpoint
Design Example MCDMA Settings Driver Support App Support

User Mode Interface Type

AVMM DMA Multi-Channel DMA AVMM Custom Perfq app (Custom PIO
BAM + MCDMA Read Write Test, Verifying
on AVMM DMA and BAM
Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test and
Verifying on AVMM)
Testpmd (Testpmd - DMA
TX only)

BAM + BAS + MCDMA AVMM Custom Perfq app (Custom PIO

Read Write Test, Verifying
on AVMM DMA, BAM Test
and BAS Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test,
Verifying on AVMM and
DPDK BAS Test)
Testpmd (Testpmd - DMA
TX only)

Device-side Packet Multi-Channel DMA AVST 1 Port Custom Perfq app (Custom PIO
Loopback BAM + MCDMA Read Write Test, BAM Test
and DMA Loopback Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test and
DPDK DMA Loopback Test)
Testpmd (Testpmd-DMA)

Netdev Netdev_app (Netdev - PIO)

Ping (Netdev - DMA)

BAM + BAS + MCDMA AVST 1 Port Custom Perfq app (Custom PIO
Read Write Test, DMA
Loopback Test, BAM Test
and BAS Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test, DPDK
DMA Loopback Test and
DPDK BAS Test)
Testpmd (Testpmd - DMA)

Netdev Netdev_app (Netdev - PIO)

Ping (Netdev - DMA)

Packet Generate/ Multi-Channel DMA AVST 1 Port Custom Perfq app (Custom PIO
Check BAM + MCDMA Read Write Test, BAM Test
and Packet Gen Test - DMA)
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
11
 2. Design Example Detailed Description
683517 | 2024.11.04

Design Example MCDMA Settings Driver Support App Support

User Mode Interface Type

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test and
DPDK - Packet Gen/Check
DMA)
Testpmd (Testpmd - DMA)

BAM + BAS + MCDMA AVST 1 Port Custom Perfq app (Custom PIO
Read Write Test, BAM Test,
Packet Gen Test - DMA and
BAS Test)

DPDK Mcdma_test (DPDK PIO

Test, DPDK BAM Test, DPDK
DMA Loopback Test and
DPDK BAS Test)
Testpmd (Testpmd - DMA)

PIO using MQDMA Multi-Channel DMA AVMM Custom Perfq app (Custom PIO
Bypass Mode BAM + MCDMA AVST 1 Port Read Write Test)
BAM + BAS + MCDMA
DPDK Mcdma_test (DPDK PIO
Test)

Bursting Master n/a Custom Perfq app (Custom PIO

Read Write Test)

DPDK Mcdma_test (DPDK PIO

Test)

BAM + BAS n/a Custom Perfq app (Custom PIO

Read Write Test)

DPDK Mcdma_test (DPDK PIO

Test)

Data Mover Only AVMM Custom Perfq app (Custom PIO

Read Write Test)

DPDK Mcdma_test (DPDK PIO

Test)

Traffic Generator/ BAM + BAS n/a Custom Perfq app (Custom PIO
Checker Read Write Test and BAS
Test)

DPDK Mcdma_test (DPDK PIO

Test and DPDK BAS Test)

External Descriptor Data Mover Only AVMM Custom Perfq app (External
Controller Descriptor Mode
Verification)

Note: F-Tile MCDMA IP design example doesn’t support multiple physical functions and SR-
IOV for simulation.

Note: For 2x8 Hard IP modes, simulation is supported on PCIe0 only.

Note: F-Tile MCDMA IP 1x4 design example does not support simulation.

Note: F-Tile does not support simulation on the ModelSim* - Intel FPGA Edition, Questa*
Intel FPGA Edition, and Xcelium* simulators.

Note: For F-Tile System PLL reference clock requirement, refer to the Multi Channel DMA
Intel FPGA IP for PCI Express User Guide.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
12
2. Design Example Detailed Description
683517 | 2024.11.04

Note: PIPE mode simulation has been added for F-Tile.

For information about supported simulators, refer to Supported Simulators on page

48.

Refer to Table MCDMA IP Modes and FPGA Development Kit for Design Examples for
the supported Hard IP Modes that have Design Example support.

2.1.4. R-Tile MCDMA IP - Design Examples for Endpoint

Table 6. R-Tile MCDMA IP - Design Examples for Endpoint
Design Example MCDMA Settings Driver Support App Support

User Mode Interface Type

AVMM DMA Multi-Channel DMA AVMM Custom Perfq app (Custom PIO Read
BAM + MCDMA Write Test, Verifying on
AVMM DMA and BAM Test)

DPDK Mcdma_test (DPDK PIO Test,

DPDK BAM Test and Verifying
on AVMM)
Testpmd (Testpmd - DMA TX
only)

BAM + BAS + MCDMA AVMM Custom Perfq app (Custom PIO Read
Write Test, Verifying on
AVMM DMA, BAM Test and
BAS Test)

DPDK Mcdma_test (DPDK PIO Test,

DPDK BAM Test, Verifying on
AVMM and DPDK BAS Test)
Testpmd (Testpmd - DMA TX
only)

Device-side Packet Multi-Channel DMA AVST 1 Port Custom Perfq app (Custom PIO Read
Loopback BAM + MCDMA Write Test, BAM Test and
DMA Loopback Test)

DPDK Mcdma_test (DPDK PIO Test,

DPDK BAM Test and DPDK
DMA Loopback Test)
Testpmd (Testpmd-DMA)

NETDEV Netdev_app (Netdev - PIO)

Ping (Netdev - DMA)

BAM + BAS + MCDMA AVST 1 Port Custom Perfq app (Custom PIO Read
Write Test, DMA Loopback
Test, BAM Test and BAS Test)

DPDK Mcdma_test (DPDK PIO Test,

DPDK BAM Test, DPDK DMA
Loopback Test and DPDK
BAS Test)
Testpmd (Testpmd - DMA)

NETDEV Netdev_app (Netdev - PIO)

Ping (Netdev - DMA)

Packet Generate/ Multi-Channel DMA AVST 1 Port Custom Perfq app (Custom PIO Read
Check BAM + MCDMA Write Test, BAM Test and
Packet Gen Test - DMA)
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
13
 2. Design Example Detailed Description
683517 | 2024.11.04

Design Example MCDMA Settings Driver Support App Support

User Mode Interface Type

DPDK Mcdma_test (DPDK PIO Test,

DPDK BAM Test and DPDK -
Packet Gen/Check DMA)
Testpmd (Testpmd - DMA)

BAM + BAS + MCDMA AVST 1 Port Custom Perfq app (Custom PIO Read
Write Test, BAM Test, Packet
Gen Test - DMA and BAS
Test)

DPDK Mcdma_test (DPDK PIO Test,

DPDK BAM Test, DPDK DMA
Loopback Test and DPDK
BAS Test)
Testpmd (Testpmd - DMA)

PIO using MQDMA Multi-Channel DMA AVMM Custom Perfq app (Custom PIO Read
Bypass Mode BAM + MCDMA AVST 1 Port Write Test)
BAM + BAS + MCDMA
DPDK Mcdma_test (DPDK PIO Test)

Bursting Master n/a Custom Perfq app (Custom PIO Read

Write Test)

DPDK Mcdma_test (DPDK PIO Test)

BAM + BAS n/a Custom Perfq app (Custom PIO Read

Write Test)

DPDK Mcdma_test (DPDK PIO Test)

Data Mover Only AVMM Custom Perfq app (Custom PIO Read
Write Test)

DPDK Mcdma_test (DPDK PIO Test)

Traffic Generator/ BAM + BAS n/a Custom Perfq app (Custom PIO Read
Checker Write Test and BAS Test)

DPDK Mcdma_test (DPDK PIO Test

and DPDK BAS Test)

External Descriptor Data Mover Only AVMM Custom Perfq app (External
Controller Descriptor Mode Verification)

Note: R-Tile MCDMA IP PIO using Bypass Mode design example simulation is supported in
x16 and x8 topologies.. The remaining R-Tile design example simulations are not
supported.

Note: R-Tile MCDMA IP 4x4 design example does not support simulation.

Note: R-Tile does not support simulation on the Xcelium* simulator.

Note: R-Tile MCDMA IP design example doesn’t support multiple physical functions and SR-
IOV for simulation.

Note: Data Mover Only Mode is not available in R-Tile MCDMA IP x4 topology.

Note: For 2x8 Hard IP modes, simulation is supported on PCIe0 only.

Note: The R-Tile 32-bit address PIO is not supported when both the 32-bit addressing and
32-bit PIO are enabled.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
14
2. Design Example Detailed Description
683517 | 2024.11.04

Note: The R-Tile MCDMA BAM + MCDMA design example generation fails when the 32-bit
address routing is enabled.

Note: The R-Tile design example does not support PIPE mode simulations.

2.1.5. Design Example BAR Mappings

The following table shows the BAR mappings for different user modes and
configurations:

Table 7. BAR Mappings

User Mode Interface Design Example BAR Selected in BAR Selected for BAR Selected for
MCDMA IP PIO/BAM Design BAS Design
Example Example

AVMM DMA BAR0 BAR2 N/A

AVMM
PIO using Bypass
BAR0 BAR2 N/A
mode

Device-side
BAR0 BAR2 N/A
MCDMA Packet Loopback

Packet Generate/
AVST BAR0 BAR2 N/A
Check

PIO using Bypass

BAR0 BAR2 N/A
mode

AVMM DMA BAR0 BAR2 N/A

AVMM
PIO using Bypass
BAR0 BAR2 N/A
mode

Device-side
BAR0 BAR2 N/A
BAM + MCDMA Packet Loopback

Packet Generate/
AVST BAR0 BAR2 N/A
Check

PIO using Bypass

BAR0 BAR2 N/A
mode

AVMM DMA BAR0 BAR2 BAR4

AVMM
PIO using Bypass
BAR0 BAR2 BAR4
mode

Device-side
BAM + BAS + BAR0 BAR2 BAR4
Packet Loopback
MCDMA
Packet Generate/
AVST BAR0 BAR2 BAR4
Check

PIO using Bypass

BAR0 BAR2 BAR4
mode

PIO using Bypass

BAM N/A N/A BAR0 and BAR2 N/A
mode

Traffic Generator/
BAM + BAS N/A N/A BAR2 BAR0
Checker
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
15
 2. Design Example Detailed Description
683517 | 2024.11.04

User Mode Interface Design Example BAR Selected in BAR Selected for BAR Selected for
MCDMA IP PIO/BAM Design BAS Design
Example Example

PIO using Bypass

N/A BAR2 BAR0
mode

PIO using Bypass

N/A BAR2 N/A
mode
Data Mover Mode N/A
External
Descriptor N/A BAR0 N/A
Controller

2.2. Hardware and Software Requirements

• Quartus® Prime Pro Edition Software version 24.3
• Operating System: CentOS Linux 7 (Core) Kernel: 3.10.0-1160
• Operating System: Ubuntu 22.04 LTS Kernel: 5.15.0-52-generic
• Stratix 10 MX or GX FPGA Development Kit supporting H-Tile PCIe Gen3
• Stratix 10 DX Production, or Agilex 7 F-Series Production Development Kit
supporting P-Tile
• Agilex 7 F-Series F-Tile FPGA Development Kit supporting F-Tile
• Agilex 7 FPGA I-Series Development Kit (2x R-Tile and 1x F-Tile) with R-Tile die
revision A0 and B0 ES variants

For details on the design example simulation steps and running Hardware test, refer to
the Design Example Quick Start Guide .

For more information on development kits, refer to Intel FPGA Development Kits on
the Intel website.

Related Information
• Design Example Quick Start Guide on page 43
• Agilex 7 FPGA I-Series Development Kit

2.3. PIO using MCDMA Bypass Mode

2.3.1. Avalon-ST PIO using MCDMA Bypass Mode

2.3.1.1. Single-Port Avalon-ST PIO using MCDMA Bypass Mode

This design example enables Avalon-MM PIO master which bypasses the DMA path.
The Avalon-MM PIO master (rx_pio_master) allows your application to perform single,
non-bursting register read/write operation with on-chip memory. This design example
only supports PIO functionality and does not perform DMA operations. Hence, the
Avalon-ST DMA ports are not connected.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
16
2. Design Example Detailed Description
683517 | 2024.11.04

Figure 1. MCDMA IP Single Port Avalon-ST Interface PIO Example Design using MCDMA
Bypass Mode

Design Example Platform Designer System

Multi Channel DMA

for PCI Express ninit_done resetIP

rx_pio_ AVMM
master MEM_PIO

hip_serial
Host PCIe
HIP H2D h2d_st_0
DMA

D2H d2h_st_0
DMA

2.3.2. Avalon-MM PIO using MCDMA Bypass Mode

Figure 2. MCDMA IP - Avalon-MM Interface PIO Example Design using MCDMA Bypass
Mode

Design Example Platform Designer System

Multi Channel DMA

for PCI Express ninit_done resetIP

rx_pio_ AVMM
master MEM_PIO
hip_serial

Host PCIe
HIP H2D h2ddm_
DMA master

D2H d2hdm_
DMA master

This design example enables Avalon-MM PIO master which bypasses the DMA path.
The Avalon-MM PIO master allows application to perform single, non-bursting register
read/write operation with on-chip memory.

This design example only supports PIO functionality and does not perform DMA
operations (similar to the design examples in Avalon-ST PIO using MCDMA Bypass
Mode on page 16). Hence, the Avalon-MM DMA ports are not connected.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
17
 2. Design Example Detailed Description
683517 | 2024.11.04

The design example includes the Multi Channel DMA for PCI Express IP Core with the
parameters you specified and other supporting components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until the
entire FPGA fabric enters user mode.
• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMA
Avalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2.

Transfer mode option supported in test application software (perfq_app) command

line:
• PIO test: -o

For a description of which driver(s) to use with this design example, refer to Driver
Support on page 57.

Note: Metadata Support is not available in AVST Interface mode, PIO using MCDMA Bypass
Mode example design.

Note: User-FLR Interface is not available in AVMM Interface mode, PIO Using MCDMA Bypass
Mode example design.

2.3.2.1. Simulation Results

Testbench writes 4 KB of incrementing pattern to on-chip memory and read back via
Avalon-MM PIO interface. This design example testbench doesn’t simulate H2D/D2H
data movers.

Figure 3. Simulation Log

Figure 4. Simulation Waveform

2.3.2.2. Hardware Test Results

Note: The Custom Driver was used to generate the following output.

Note: P-Tile PIO test result screenshot is given below.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
18
2. Design Example Detailed Description
683517 | 2024.11.04

Figure 5. PIO Test

-o option

Note: If the R-Tile MCDMA IP or F-Tile MCDMA IP is configured with the 32-bit PIO enabled,
use the following command instead:

./perfq_app -b 0000:01:00.0 -p 2048 -l 5 -i -d 1 -c 4 -a 8

2.4. Avalon-ST Packet Generate/Check

2.4.1. Single-Port Avalon-ST Packet Generate/Check

Below is the block diagram of a packet generator design example with a single-port
Avalon Streaming interface supporting multiple channels without any interleaving. This
design example can be used with the perfq application to evaluate the functionality
and capture the MCDMA performance. In the H2D direction, the design example
checks for the received packets and software then reads the status registers to make
sure there are no errors. In the D2H direction, the design example generates the
packets and forwards them to the Host side by means of PCIe MWr.

For a description of which driver(s) to use with this design example, refer to Driver
Support on page 57.

Note: Metadata Support is not available in AVST Packet Generator/Checker design example.

2.4.1.1. Simulation Waveforms

Note: The simulation was run with MCDMA P-Tile.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
19
 2. Design Example Detailed Description
683517 | 2024.11.04

2.4.1.2. Simulation Log

Note: The simulation was run with MCDMA P-Tile.

2.4.1.3. Hardware Test Result

Note: This test was run with Agilex 7 F-Series P-Tile FPGA PCIe Gen4 x16 configuration using
Custom Driver.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
20
2. Design Example Detailed Description
683517 | 2024.11.04

Note: This test was run with Agilex 7 F-Series P-Tile FPGA PCIe Gen4 x16 configuration using
CentOS 7 platform.

[root@localhost perfq_app]# ./perfq_app -b 0000:2d:00.0 -p 32768 -l 10 -d 5 -c 8

-a 8 -r
Allocating 8 Channels...

----------------------------------------------------------------------
BDF: 0000:2d:00.0
Channels Allocated: 8
QDepth 508
Number of pages: 8
Completion mode: WB
Payload Size per descriptor: 32768 Bytes
SOF on descriptor: 1
EOF on descriptor: 1
File Size: 32768 Bytes
PKG Gen Files: 64
Rx Batch Size: 127 Descriptors
DCA: OFF
-----------------------------------------------------------------------
Thread initialization in progress ...
Thread initialization done
---------------------------------------------------------------------------------
----------------------------------
Dir #queues Time_elpsd B_trnsfrd TBW IBW MIBW
HIBW LIBW MPPS #stuck
Rx 8 00:05:000 127333248.00KB 24.28GBPS 24.28GBPS 03.04GBPS
03.04GBPS 03.04GBPS 00.80MPPS 0
---------------------------------------------------------------------------------
----------------------------------
All Threads exited
---------------------------------------------------------------------------------
----------------------------------
Dir #queues Time_elpsd B_trnsfrd TBW IBW MIBW
HIBW LIBW MPPS #stuck
Rx 8 00:10:000 251390912.00KB 23.97GBPS 23.66GBPS 02.96GBPS
02.96GBPS 02.96GBPS 00.78MPPS 0
---------------------------------------------------------------------------------
----------------------------------

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
21
 2. Design Example Detailed Description
683517 | 2024.11.04

TIME OUT while waiting for completions

Leaving...
-------------------------------OUTPUT
SUMMARY--------------------------------------------
Dir #queues Time_elpsd B_trnsfrd TBW MPPS Passed
Failed %passed
Rx 8 00:10:871 251390912.00KB 24.29GBPS 00.80MPPS 8
0 100.00%
---------------------------------------------------------------------------------
---------
--------------------------------------------------------------------------------
---------------------------------------------------------------------------------
---------
Total Bandwidth: 24.29GBPS, 0.80MPPS
---------------------------------------------------------------------------------
---------
Full Forms:
TBW: Total Bandwidth
IBW: Interval Bandwidth
MIBW: Mean Interval Bandwidth
HIBW: Highest Interval Bandwidth
LIBW: Lowest Interval Bandwidth

2.5. Avalon-ST Device-side Packet Loopback

Figure 6. Avalon-ST Device-side Packet Loopback

This design example performs H2D and D2H multi channel DMA via device-side
Avalon-ST streaming interface. With device-side packet loopback, Host to Device
(H2D) data stream is loop backed to the Host (D2H) through an external FIFO.

For H2D streaming, Multi Channel DMA sends the data to Avalon-ST loopback FIFOs
via four Avalon-ST Source ports. For D2H streaming, Multi Channel DMA receives the
data from Avalon-ST loopback FIFOs via Avalon-ST Sink ports.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
22
2. Design Example Detailed Description
683517 | 2024.11.04

In this device-side loopback example, the Host first sets up memory locations within
the Host memory. Data from the Host memory is then sent to the device-side memory
by the Multi Channel DMA for PCI Express IP via H2D DMA operations. Finally, the IP
loops this data back to the Host memory using D2H DMA operations.

In addition, the design example enables Avalon-MM PIO master which bypasses the
DMA path. It allows application to perform single, non-bursting register read/write
operation with on-chip memory block.

The design example includes the Multi Channel DMA for PCI Express IP Core with the
parameters you specified and following components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until the
entire FPGA fabric enters user mode
• GEN_CHK - Packet Generator and Checker for MCDMA. It interfaces with DUT
Avalon Streaming H2D/D2H interfaces (h2d_st_0, d2h_st_0) for DMA operation.
DUT AVMM PIO Master (rx_pio_master) performs read and write operations to the
CSR and On-chip memory.
• PIO_INTERPRETER - This maps DUT AVMM PIO Master address width to AVMM
Slave side address based on its parameter setting such as MAP_PF, MAP_VF, and
MAP_BAR.

Transfer mode options supported in test application software (perfq_app) command

line:
• PIO test: -o
• DMA test: -i (performance loopback operation where the Tx and Rx are run in two
different threads), -v (enable data validation, which will perform a data integrity
check).
— -i without -v flag displays the throughput per channel

For a description of which driver(s) to use with this design example, refer to Driver
Support on page 57.

Note: Metadata Support is available in AVST Device-side Packet Loopback design example.

2.5.1. Simulation Results

Note: The simulation was run with MCDMA P-Tile.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
23
 2. Design Example Detailed Description
683517 | 2024.11.04

Figure 7. Simulation Log

Figure 8. Simulation Waveform

2.5.2. Hardware Test Results

The Custom Driver was used to generate the following output.

Note: The same test options can be used with DPDK driver and Kernel Mode driver to
generate comparable results.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
24
2. Design Example Detailed Description
683517 | 2024.11.04

PIO Test
[root@bapvemb005t perfq_app]# ./perfq_app -b 0000:01:00.0 -o -v
PIO Write and Read Test ...
Pass

Figure 9. Performance Test: H-Tile

-i option. Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16 configuration.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
25
 2. Design Example Detailed Description
683517 | 2024.11.04

Figure 10. Performance Test: P-Tile

This test was run with Agilex 7 F-Series P-Tile PCIe Gen4 x16 configuration.

Data Validation Test

[root@localhost perfq_app]#./perfq_app -b 0000:2d:00.0 -p 8192 -d 2 -c 4 -a 4 -s

268435456 -u -v
Allocating 4 Channels...

----------------------------------------------------------------------
BDF: 0000:2d:00.0
Channels Allocated: 4
QDepth 508
Number of pages: 8
Completion mode: WB
Payload Size per descriptor: 8192 Bytes
#Descriptors per channel: 32768
SOF on descriptor: 1
EOF on descriptor: 1
File Size: 8192 Bytes
Tx Batch Size: 127 Descriptors
Rx Batch Size: 127 Descriptors
DCA: OFF
-----------------------------------------------------------------------
Thread initialization in progress ...
Thread initialization done
All Threads exited
-------------------------------OUTPUT
SUMMARY--------------------------------------------
Dir #queues Time_elpsd B_trnsfrd TBW MPPS Passed
Failed %passed
Tx 4 00:00:429 1048576.00KB 02.30GBPS 00.30MPPS 4
0 100.00%

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
26
2. Design Example Detailed Description
683517 | 2024.11.04

Rx 4 00:00:429 1048576.00KB 02.31GBPS 00.30MPPS 4

0 100.00%
---------------------------------------------------------------------------------
---------
--------------------------------------------------------------------------------
Full Forms:
TBW: Total Bandwidth
IBW: Interval Bandwidth
MIBW: Mean Interval Bandwidth
HIBW: Highest Interval Bandwidth
LIBW: Lowest Interval Bandwidth

To enable data validation using -v option, set the software flags in user/common/mk/
common.mk as follows:
• cflags += -UPERFQ_PERF
• cflags += -DPERFQ_LOAD_DATA

Note: This hardware test was run with Agilex 7 P-Tile PCIe Gen4x16 configuration.

2.6. Avalon-MM DMA

Figure 11. Avalon-MM DMA

Design Example Platform Designer System

Multi Channel DMA

for PCI Express ninit_done resetIP

rx_pio_ AVMM
master MEM_PIO
hip_serial

Host PCIe AVMM

HIP H2D h2ddm_
DMA master

MEM

D2H d2hdm_ AVMM

DMA master

This design example performs H2D and D2H multi channel DMA via Avalon-MM
memory-mapped interface. The Multi Channel DMA for PCI Express IP core provides
one Avalon-MM Write/Read Master port. You can allocate up to 2K DMA channels when
generating this example design.

This example design contains on-chip memories to support PIO and H2D/D2H DMA
operations.

For the H2D (Tx) DMA, the host populates the descriptor rings, allocates Tx packet
buffers in the host memory, and fills the Tx buffers with a predefined pattern. When
the application updates the Queue Tail Pointer register (Q_TAIL_POINTER), the
MCDMA IP starts the H2D DMA and writes received data to the on-chip memory.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
27
 2. Design Example Detailed Description
683517 | 2024.11.04

For the D2H (Rx) DMA, the host initializes the FPGA on-chip memory with a predefined
pattern. The MCDMA IP reads the packet data from the on-chip memory and transmits
it to the host memory.

For bidirectional DMA, H2D is started before D2H and then both DMAs operate
simultaneously.

In addition, the design example enables Avalon-MM PIO master which bypasses the
DMA path. It allows application to perform single, non-bursting register read/write
operation with on-chip memory block.

The design example includes the Multi Channel DMA for PCI Express IP Core with the
parameters you specified and following components:
• resetIP – Reset Release IP that holds the Multi Channel DMA in reset until the
entire FPGA fabric enters user mode
• MEM_PIO – On-chip memory for the PIO operation. Connected to the MCDMA
Avalon-MM PIO Master (rx_pio_master) port that is mapped to PCIe BAR2
• MEM – Dual port on-chip memory. One port is connected to the Avalon-MM Write
Master (h2ddm_master) and the other port to Avalon-MM Read Master
(d2hdm_master)

Transfer mode options supported in test application software (perfq_app) command

line:
• PIO test: -o
• DMA test: -t (Tx), -r (Rx)

For a description of which driver(s) to use with this design example, refer to Driver
Support on page 57.

Note: User FLR Interface is not available in AVMM DMA design example.

2.6.1. Simulation Results

The simulation below was run with P-Tile MCDMA IP.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
28
2. Design Example Detailed Description
683517 | 2024.11.04

Figure 12. Simulation Log

Figure 13. Simulation Waveform 1

The simulation was run with MCDMA P-Tile.

Figure 14. Simulation Waveform 2

The simulation was run with MCDMA P-Tile.

2.6.2. Hardware Test Results

The Custom Driver was used to generate the following output:

Figure 15. PIO Test

-o option

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
29
 2. Design Example Detailed Description
683517 | 2024.11.04

Figure 16. H2D Avalon-MM Write

-t option. Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16 configuration.

Figure 17. H2D Avalon-MM Write Agilex 7 F-Series P-Tile PCIe Gen4 x16
The following hardware test was run with Agilex 7 F-Series P-Tile PCIe Gen4 x16 configuration using Custom
Driver.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
30
2. Design Example Detailed Description
683517 | 2024.11.04

Figure 18. H2D Avalon-MM Write with Data Validation Enabled

-t -v option. Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16 configuration.

To enable data validation using -v option, set the software flags in user/common/mk/
common.mk as follows:

cflags += -UPERFQ_PERF
cflags += -DPERFQ_LOAD_DATA

Note: Hardware test with P-Tile Gen4 x16 may be added in a future release.

Figure 19. D2H Avalon-MM Read

-r option. Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16 configuration.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
31
 2. Design Example Detailed Description
683517 | 2024.11.04

Figure 20. D2H Avalon-MM Read Agilex 7 F-Series P-Tile PCIe Gen4 x16

2.7. BAM_BAS Traffic Generator and Checker

Figure 21. Traffic Generator and Checker

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
32
2. Design Example Detailed Description
683517 | 2024.11.04

This design example instantiates the Traffic Generator and Checker for MCDMA module
(BAS_TGC) that creates read and write transactions to exercise the Bursting Avalon-
MM Slave module of the Multi Channel DMA for PCI Express IP configured in BAM+BAS
mode. You can program the BAS_TGC by writing to its control registers through its
Control and Status Avalon-MM slave interface. The traffic that it generates, and the
traffic that checker expects is in a sequence of incrementing dwords.

For traffic generation, the host software allocates a block of memory in the PCIe space
and then programs one of the windows in the Address Mapper to point to the allocated
memory block. It also needs to set the start address which is the base address of the
selected space for write transactions, set the write count to the size of the block of the
allocated memory block, and set the transfer size before kick start to the traffic
generation. The number of completed transfers can be checked by reading the write
count register.

For traffic checking, the host software sets the read address to point to the start
address of the write transactions, set the read count to the size of the block of the
allocated memory block, and set the transfer size before kick start to the traffic
checker. The number of completed data check and the number of errors occurred can
be checked by reading the read count and read error count registers, respectively.

Note: MSI Interface is supported for Traffic Generator and Checker Example Design in
H/P/F/R-Tile MCDMA IP.

2.7.1. BAM_BAS Traffic Generator and Checker Example Design Register

Map
The Traffic generator and checker control and status registers are byte addresses. The
Traffic generator checker register map is mapped to BAR0 in the Example Design..

Table 8. Read start address (Offset 16’h0000)

Bit[63:0] Name R/W Default Description

[63:32] rsvd Reserved

[31:0] RAdd R/W 0 This register contains

the base addresses
that the Traffic
Checker reads from.

Table 9. Read count (Offset 16’h0008)

Bit[63:0] Name R/W Default Description

[63:32] rsvd Reserved

[31] Mode R/W 0 0: Fixed no of

transfers
1: non-stop transfers

[30:12] rsvd Reserved

[11:0] RCnt R/W 0 Write to the RCnt

registers to specify
the number of
transfers to execute.
Reading from one of
these registers returns
the number of
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
33
 2. Design Example Detailed Description
683517 | 2024.11.04

Bit[63:0] Name R/W Default Description

transfers that have

occurred since it was
last read.

Table 10. Read error count (Offset 16’h0010)

Bit[63:0] Name R/W Default Description

[63:12] rsvd Reserved

[11:0] RErr ROC 0 Reading the RErr

register returns the
number of errors
detected since the
register was last read.
A maximum of one
error is counted per
clock cycle.

Table 11. Read control (Offset 16’h0018)

Bit[63:0] Name R/W Default Description

[63:32] rsvd Reserved

[31] enable R/W 0 0: stop 1: start

[30:8] rsvd Reserved

[7:0] transfer_size 0 This register

configures the burst
length per transfer
(ideal value for x16 is
8 & x8 is 16). Zero is
not a legal value.

Table 12. Write start address (Offset 16’h0020)

Bit[63:0] Name R/W Default Description

[63:32] rsvd Reserved

[31:0] WAdd R/W 0 This register contains

the base addresses
that the Traffic
Generator writes to.

Table 13. Write count (Offset 16’h0028)

Bit[63:0] Name R/W Default Description

[63:32] rsvd Reserved

[31] Mode R/W 0 0: Fixed no of

transfers
1: non-stop transfers

[30:12] rsvd Reserved

[11:0] WCnt R/W 0 Write to the WCnt

registers to specify
the number of
transfers to execute.
Reading from one of
these registers returns
continued...

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
34
2. Design Example Detailed Description
683517 | 2024.11.04

Bit[63:0] Name R/W Default Description

the number of
transfers that have
occurred since it was
last read.

Table 14. Write error count (Offset 16’h0030)

Bit[63:0] Name R/W Default Description

[63:12] rsvd Reserved

[11:0] WErr ROC 0 Reserved (Write error

detection not available
yet). Because the
write error detection
feature is not available
yet, you cannot get a
valid number of errors
by reading the WErr
register.

Table 15. Write control (Offset 16’h0038)

Bit[63:0] Name R/W Default Description

[63:32] rsvd Reserved

[31] enable R/W 0 0: stop 1: start

[30:8] rsvd Reserved

[7:0] transfer_size 0 This register

configures the burst
length per transfer
(ideal value for x16 is
8 & x8 is 16). Zero is
not a legal value.

Table 16. Read address mapping table (Offset 16’h0100)

The Read address mapping table is 32 locations and 64bit wide each.

Bit[63:0] Name R/W Default Description

[63:0] raDM R/W 0 This register contains

the Traffic Checker
address mapping table
that maps thirty-two 1
MB regions of the
Avalon-MM memory
space into thirty-two 1
MB regions of the
PCIe address space.
The module occupies
only 32MB of the
Avalon-MM address
space, and only needs
a 25-bit wide address
bus, leaving space for
other Avalon-MM
slaves.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
35
 2. Design Example Detailed Description
683517 | 2024.11.04

Table 17. Write address mapping table (Offset 16’h0200)

The Write address mapping table is 32 locations and 64bit wide each.

Bit[63:0] Name R/W Default Description

[63:0] WAdm R/W 0 This register contains

the Traffic Generator
address mapping table
that maps thirty-two 1
MB regions of the
Avalon-MM memory
space into thirty-two 1
MB regions of the
PCIe address space.
The module occupies
only 32MB of the
Avalon-MM address
space, and only needs
a 25-bit wide address
bus, leaving space for
other Avalon-MM
slaves.

2.8. External Descriptor Controller

External descriptor controller example design supports descriptor fetching and queue
management along with MCDMA IP core in Data Mover mode of operations. Example
design supports the following features:
• Total 16 DMA channels (16 H2D queues and 16 D2H queues)
• Supports separate descriptor command queues for H2D and D2H Data Movers
• Supports Writeback (WB) as descriptor completion mechanism
• No Interrupt Support

The figure below is the high level block diagram of example design.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
36
2. Design Example Detailed Description
683517 | 2024.11.04

Figure 22. High Level Block Diagram of Example Design

Example Design Platform

Designer System ext_desc_ctrl

d2hdm_desc
d2hdm_
desc_status
h2ddm_
desc_cmpl

h2ddm_desc
h2ddm_
desc_status
DUT (MCDMAP-Tile/F-Tile) reset IP

H2D AVMM Wr
Data Mover Master
HIP DMA_MEM
Interface/
PCIe Scheduler/ D2H AVMM Rd
Host
HIP Arbiter Data Mover Master

BAM
BAM
Interpreter

Avalon Streaming Avalon Memory Mapped

The figure below is the internal representation of external descriptor controller block.

Figure 23. External Descriptor Controller Example Design

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
37
 2. Design Example Detailed Description
683517 | 2024.11.04

Global/Queue CSR: Implements the global CSR and Queue CSR registers required
for controlling the DMA operation. Read/Write access to these registers happen
through BAM interface of MCDMA IP. For details about the registers, refer to the
Registers on page 40 section.

H2D/D2H Descriptor Fetch: This block generates descriptor fetch commands on

h2ddm_desc interface, based on the context in QCSR registers.

Descriptor Completion Processing: This block processes the received descriptor

completion packets on h2ddm_desc_cmpl interface for the descriptor fetch request
send to MCDMA and stores the received descriptors in corresponding descriptor queue
buffers (H2D/D2H Descriptor Queue). These fetched descriptors get queued, and
corresponding data mover commands are sent to MCDMA IP through h2ddm_desc or
d2hdm_desc interface.

Descriptor Status Processing: This block processes the status information received
on h2ddm_desc_status and d2hdm_desc_status interface and generates appropriate
writeback commands to MCDMA IP on d2hdm_desc interface.

Descriptor Fetch operation

The descriptor fetch process is described below:

1. Software updates the Queue context registers in QCSR and update the Tail pointer
register (offset 0x10 for D2H; offset 0x90 for H2D).
2. H2D/D2H Descriptor fetch block detects Tail pointer register update and
generate descriptor fetch commands on h2ddm_desc interface of MCDMA to get
descriptors from host memory.
3. The number of descriptors to be fetched are determined by difference between
Head pointer (offset 0x18 for D2H; offset 0x98 for H2D) and Tail pointer (offset
0x10 for D2H; offset 0x90 for H2D).
4. The Head pointer register (offset 0x18 for D2H; offset 0x98 for H2D) is updated
based on number of descriptors fetched.
5. MCDMA provides received descriptor completions on h2ddm_desc_cmpl interface.
6. The received descriptors are stored on respective Queue buffers (H2D/D2H
Descriptor Queue).
7. The descriptor fetch process happens for multiple queues in a round-robin
arbitration scheme.

H2D Data movement operation

1. Descriptors from H2D Descriptor Queue are pulled out, translated to
corresponding data mover commands, and sent to MCDMA through h2ddm_desc
interface.
2. Data transfer happens from Host memory to DMA_MEM through H2D Data Mover
AVMM Write master.
3. Once data transfer is completed, MCDMA sends status on h2ddm_desc_status
interface.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
38
2. Design Example Detailed Description
683517 | 2024.11.04

4. The status information is processed by Descriptor Status Processing block to

generate appropriate writeback commands to MCDMA on d2hdm_desc interface.
5. The Completed pointer register (offset 0xA8 for H2D) is updated based on number
of descriptors processed.
6. This flow continues for all descriptors in the H2D Descriptor Queue.

D2H Data movement operation

1. Descriptors from D2H Descriptor Queue are pulled out, translated to
corresponding data mover commands, and sent to MCDMA through d2hdm_desc
interface.
2. Data transfer happens from DMA_MEM to Host memory through D2H Data Mover
AVMM Read master.
3. Once data transfer is completed, MCDMA sends status on d2hdm_desc_status
interface.
4. The status information is processed by Descriptor Status Processing block to
generate appropriate writeback commands to MCDMA on d2hdm_desc interface.
5. The Completed pointer register (offset 0x28 for D2H) is updated based on number
of descriptors processed.
6. This flow continues for all descriptors in the D2H Descriptor Queue.

The differences between main MCDMA and example design are shown in the table
below.

Table 18. Differences between main MCDMA and Example Design

Feature Main MCDMA External Descriptor Controller Example Design

DMA Channels Up to 2K Fixed 16

SRIOV Yes No

MSI-X Yes No
(MSI-X may be supported in future release)

Writeback Yes Yes

Queue CSR Yes Yes

(For details, see register tables)

MSI No No

Descriptor Link bit Yes No

(The descriptors are formed continuously and in consecutive locations and provided
with starting address in the QCSR)

The following table describes the simplified Host descriptor.

Note: Only the descriptor format is the same for D2H and H2D, and the descriptors are in a
separate ring.

Table 19. Host descriptor format

Name Width Description

SRC_ADDR[63:0] 64 Source address of allocated buffer read by DMA.

DEST_ADDR[127:64] 64 Destination address of allocated buffer written by DMA.

continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
39
 2. Design Example Detailed Description
683517 | 2024.11.04

Name Width Description

PYLD_CNT[147:128] 20 DMA payload size in bytes. Max 1MB, with 20’h0 indicating 1MB.

RSRV[159:148] 12 Reserved

DESC_IDX [175:160] 16 Unique Identifier for each descriptor, the same number will be applied to
Q_COMPLETED_POINTER.
Same as descriptor count which is used as tail pointer.
For example, descriptor count starts from 1 to 128 for 128 descriptors in a 4K page.

MSIX_EN[176] 1 Enable MSIX

WB_EN[177] 1 Enable Writeback

RSRVD [255:178] 78 Reserved

2.8.1. Registers

2.8.1.1. GCSR Registers (Base address 64’h00000)

Table 20. Revision Register (Offset 8’h00)

Bit[63:0] Name R/W Default Description

[63:0] revision R0 1 Read-only revision number.

Table 21. Soft-reset Register (Offset 8’h08)

Bit[63:0] Name R/W Default Description

[63:1] rsvd Reserved

[0:0] Soft_reset R/W 0 Soft-reset register to reset the QCSR block, Software needs to write 1 to reset and
then write 0 to un-reset.

2.8.1.2. QCSR Registers(Base address 64’h10000)

This space contains control and status registers for external descriptor controller. Host
maintains separate descriptor ring for both D2H & H2D queues. External descriptor
controller example design supports 16 channels. Descriptor context for each channel
can be updates in the following CSR registers.

Address [11:8] = Channel number (Queue ID).

Address [7] = Data Mover direction. 0 = D2H, 1 = H2D.

Address [6:3] = Register offset.

Table 22. D2H Start Address (Offset 8’h00)

Bit[63:0] Name R/W Default Description

[63:0] Start_addr R/W 0 D2H ring buffer start

address [63:0].

Table 23. D2H Buffer Size (Offset 8’h08)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] size R/W 0 D2H ring buffer size

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
40
2. Design Example Detailed Description
683517 | 2024.11.04

Table 24. D2H Tail Pointer (Offset 8’h10)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] tail_ptr R/W 0 D2H ring buffer tail

pointer, Updated by
Host SW & Read-only
for DMA HW.

Table 25. D2H Head Pointer (Offset 8’h18)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] head_ptr RO 0 D2H ring buffer head

pointer, Updated by
DMA HW & Read-only
for Host SW.

Table 26. D2H Write back Address (Offset 8’h20)

Bit[63:0] Name R/W Default Description

[63:0] wb_addr R/W 0 D2H write back

address [63:0].

Table 27. D2H Completion Pointer (Offset 8’h28)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] cmpl_ptr RO 0 D2H ring buffer

completion pointer,
Updated by DMA HW
& Read-only for Host
SW.

Table 28. H2D Start address (Offset 8’h80)

Bit[63:0] Name R/W Default Description

[63:0] Start_addr R/W 0 H2D ring buffer start

address [63:0].

Table 29. H2D Buffer Size (Offset 8’h88)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] size R/W 0 H2D ring buffer size

Table 30. H2D Tail Pointer (Offset 8’h90)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] tail_ptr R/W 0 H2D ring buffer tail

pointer, Updated by
Host SW & Read-only
for DMA HW.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
41
 2. Design Example Detailed Description
683517 | 2024.11.04

Table 31. H2D Head Pointer (Offset 8’h98)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] head_ptr RO 0 H2D ring buffer head

pointer, Updated by
DMA HW & Read-only
for Host SW.

Table 32. H2D Write back address (Offset 8’hA0)

Bit[63:0] Name R/W Default Description

[63:0] wb_addr R/W 0 H2D write back

address [63:0].

Table 33. H2D Completion Pointer (Offset 8’hA8)

Bit[63:0] Name R/W Default Description

[63:16] rsvd Reserved

[15:0] cmpl_ptr RO 0 H2D ring buffer

completion pointer,
Updated by DMA HW
& Read-only for Host
SW.

2.8.2. Hardware Test Results

The external descriptor controller in the example design is derived from the main
MCDMA. The same MCDMA driver is used to demonstrate the external descriptor
controller.

To enable the flag IFC_MCDMA_EXTERNL_DESC in the file software/user/

common/mk/common.mk, run the following command
__cflags += -DIFC_MCDMA_EXTERNL_DESC

Figure 24. P-Tile External Descriptor Controller Example Design Test

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
42
 683517 | 2024.11.04

Send Feedback

3. Design Example Quick Start Guide

Using Quartus Prime software, you can generate a design example for the Multi
Channel DMA for PCI Express (PCIe) IP core.

The generated design example reflects the parameters that you specify. The design
example automatically creates the files necessary to simulate and compile in the
Quartus Prime software. You can download the compiled design to your FPGA
Development Board. To download to custom hardware, update the Quartus Prime
Settings File (.qsf) with the correct pin assignments.

Figure 25. Design Example Development Steps

Compilation Functional
(Simulator) Simulation

Design
Example Compilation Hardware
Generation (Quartus Prime) Testing

3.1. Design Example Directory Structure

Table 34. Directory Structure
Directory / File Sub-directory / Sub-directory / Sub-directory / Sub-directory / Note
File File File File

Design example
pcie_ed.v
top-level HDL
sim
pcie_ed
<simulators> <simulation scripts> simulation
pcie_ed directory

Design example
synth pcie_ed.v
top-level HDL

<Components automatically generated by Platform Designer>

Testbench
pcie_ed_tb pcie_ed_tb sim pcie_ed_tb.v including Intel
FPGA BFM
continued...

© Altera Corporation. Altera, the Altera logo, the ‘a’ logo, and other Altera marks are trademarks of Altera
Corporation. Altera and Intel warrant performance of its FPGA and semiconductor products to current
specifications in accordance with Altera’s or Intel's standard warranty as applicable, but reserves the right to ISO
make changes to any products and services at any time without notice. Altera and Intel assume no 9001:2015
responsibility or liability arising out of the application or use of any information, product, or service described Registered
herein except as expressly agreed to inwriting by Altera or Intel. Altera and Intel customers are advised to
obtain the latest version of device specifications before relying on any published information and before placing
orders for products or services.
*Other names and brands may be claimed as the property of others.
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Directory / File Sub-directory / Sub-directory / Sub-directory / Sub-directory / Note

File File File File

Testbench
<simulation
<simulators> simulation
script>
directory

Intel FPGA BFM

ip pcie_ed_tb DUT_pcie_tb_ip
(RP)

Testbench
pcie_ed_tb.qsys Platform Designer
file

pcie_ed.ipx

drivers net

examples mcdma-test
dpdk
v20.05-rc1
dpdk patches
v21.11.2

Licenses license_bsd.txt

version.txt

common

mcdma-custom-
driver
kernel driver kmod Kernel driver
mcdma-netdev-
driver
pX_software
Licenses license_bsd.txt
where X= 0,1, 2,
3 (IP core <test application
numbers) Test Application
software>
perfq_app
cli
README Readme file

devmem

MCDMA and Pkt

include regs
Gen/Chk registers
user
common
mk

src

libmqdma <user space library files> User space library

Licenses

Readme Readme file

readme Readme file

ip pcie_ed <Design example IP components>

Quartus project
pcie_ed.qpf
file

Quartus setting
pcie_ed.qsf
file

Design example
pcie_ed.qsys Platform Designer
file

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
44
3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: Software directory is created multiple times depending on Hard IP mode selected
(1x16, 2x8 or 4x4) for Quartus Prime Pro Edition 23.3 version onwards.
• p0_software folder is generated for 1x16, 2x8 and 4x4 Hard IP modes.
• p1_software folder is generated for 2x8 Hard IP modes.
• p2_software and p3_software folders are generated for 4x4 Hard IP modes.

Note: You must use the corresponding software folder with each IP port.

3.2. Generating the Example Design using Quartus Prime

Figure 26. Design Example Generation

Specify
Start Parameter Specify IP Variation Select Initiate
Example Design and
Editor and Select Device Design Parameters Design Generation
Select Target Board

3.2.1. Procedure

1. In the Quartus Prime Pro Edition software, create a new project File → New
Project Wizard.
2. Specify the Directory, Name, and Top-Level Entity.
3. For Project Type, accept the default value, Empty project. Click Next.
4. For Add Files click Next.
5. For Family, Device & Board Settings, select Stratix 10 (GX/SX/MX/TX/DX)
or Agilex 7 F-Series or Agilex 7 I-Series and the Target Device for your
design.
Note: The selected device is only used if you select None in Step 10c below.
6. Click Finish.
7. In the IP Catalog locate and add the H-Tile Multichannel DMA Intel FPGA IP
(Stratix 10 GX/MX devices), P-Tile Multichannel DMA Intel FPGA IP (Stratix
10 DX and Agilex 7 devices), F-Tile Multichannel DMA Intel FPGA IP or R-Tile
Multichannel DMA Intel FPGA IP (Agilex 7 devices), which brings up the IP
Parameter Editor.
8. In the New IP Variant dialog box, specify a name for your IP. Click Create.
9. On the IP Settings tabs, specify the parameters for your IP variation.
10. On the Example Designs tab, make the following selections:
a. For Example Design Files, turn on the Simulation and Synthesis options.
If you do not need these simulation or synthesis files, leaving the
corresponding option(s) turned off significantly reduces the example design
generation time.
b. For Generated HDL Format, only Verilog is available in the current release.
c. For Target Development Kit, select the appropriate option.
Note: If you select None, the generated design example targets the device
specified. Otherwise, the design example uses the device on the
selected development board. If you intend to test the design in
hardware, make the appropriate pin assignments in the .qsf file.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
45
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: Appropriate pin assignments in the .qsf file must to be added before
compilation in P/F/R Tiles when Enable CVP (Intel VSEC) option is
checked, and when Target Development Kit is selected as None.
Otherwise, example design compilation in Quartus Prime throws an
error in the Fitter stage.
d. For Currently Selected Example Design, select a design example from a
pulldown menu. Available design examples depends on the User Mode and
Interface type setting in MCDMA Settings under IP Settings tab.
Available design examples for the MCDMA or BAM+MCDMA or BAM+BAS
+MCDMA modes and Avalon-ST Interface type:
• PIO using MQDMA Bypass Mode
• Packet Generate/Check
• Device-side Packet Loopback
Available design examples for the MCDMA or BAM+MCDMA or BAM+BAS
+MCDMA modes and Avalon-MM Interface type:
• PIO using MQDMA Bypass Mode
• AVMM DMA
Available design example for only BAM User mode:
• PIO using MQDMA Bypass Mode
Available design examples for BAM+BAS User mode:
• PIO using MQDMA Bypass Mode
• Traffic Generator / Checker
Available design examples for Data Mover Only User mode:
• PIO using MQDMA Bypass Mode
• External descriptor controller
11. Select Generate Example Design to create a design example that you can
simulate and download to hardware. If you target one of the Intel FPGA
development kits, the device on that board supersedes the device previously
selected in the Quartus Prime Pro Edition project if the devices are different. When
the prompt asks you to specify the directory for your example design, you can
choose to accept the default directory ./
intel_pcie_mcdma_0_example_design or choose another directory.
12. Click Close on Generate Example Design Completed message.
13. Close the IP Parameter Editor. Click File → Exit. When prompted with Save
changes?, you do not need to save the .ip. Click Don’t Save.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
46
3. Design Example Quick Start Guide
683517 | 2024.11.04

3.3. Simulating the Design Example

3.3.1. Testbench Overview

Figure 27. Testbench Platform Designer View

The design example, pcie_ed_inst, is generated with the link width you select in the
IP Parameter Editor. The Intel FPGA BFM, DUT_pcie_tb, is a Root Port BFM.

Note: The H-Tile Root Port BFM only supports up to Gen3 x8 width and downtrains x16
Endpoint to Gen3 x8. If you want to simulate x16 link width with MCMDA H-Tile
Endpoint, you can use a third-party Root Complex BFM.

The testbench uses a Root Port driver module to initiate the configuration and exercise
the target memory and DMA channel in the Endpoint. This is the module that you can
modify to vary the transactions sent to the example Endpoint design or your own
design.

The driver module path and file is as follows:

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
47
 3. Design Example Quick Start Guide
683517 | 2024.11.04

• MCDMA R-Tile testbench

Path: pcie_ed_tb/ip/pcie_ed_tb/dut_pcie_tb_ip/
intel_pcie_rtile_tbed_100/sim
File: altpcietb_bfm_rp_gen5_x16.sv
• MCDMA P-Tile testbench
Path: pcie_ed_tb/ip/pcie_ed_tb/DUT_pcie_tb_ip/
intel_pcie_ptile_tbed_100/sim
File: altpcietb_bfm_rp_gen4_x16.sv
• MCDMA F-Tile testbench
Path: pcie_ed_tb/ip/pcie_ed_tb/dut_pcie_tb_ip/
intel_pcie_ftile_tbed_100/sim
File: altpcietb_bfm_rp_gen4_x16.sv
• MCDMA H-Tile testbench
Path: pcie_ed_tb/ip/pcie_ed_tb/DUT_pcie_tb_ip/
altera_pcie_s10_tbed_191/sim
File: altpcietb_bfm_rp_gen3_x8.sv

For more information about Intel FPGA BFM, refer to the Testbench overview in the
user guides provided below:
• P-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide
• F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide
• R-Tile Avalon Stream Intel FPGA IP for PCI Express Design Example User Guide
• L- and H-tile Avalon Streaming and Single Root I/O Virtualization (SR-IOV) Intel
FPGA IP for PCI Express User Guide

3.3.2. Supported Simulators

The following tables show supported simulators for MCDMA example designs.

Note: Root Port mode MCDMA IP simulation is supported by VCS simulator only.

Note: For 2x8 Hard IP modes, example design simulation is supported on PCIe0 only.

Note: MCDMA R-Tile PIO using Bypass Mode design example simulation is supported for x16
and x8 topologies. The remaining R-Tile design example simulations are not
supported. This feature may be supported in a future release of the Quartus Prime
software.

Note: For x4 (1x4 or 2x4 or 4x4) Hard IP modes, example design simulation is not
supported.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
48
3. Design Example Quick Start Guide
683517 | 2024.11.04

Table 35. Supported Simulators for H-Tile MCDMA IP

Tile Design User Mode VCS VCS MX Xcelium QuestaSim Questa Aldec
Example * Intel FPGA Riviera Pro
Edition

PIO using Multi Yes Yes Yes Yes No No

Bypass channel
mode DMA
Bursting
Master
BAM+BAS
BAM
+MCDMA
BAM+BAS
+MCDMA

AVMM DMA Multi Yes Yes Yes Yes No No

channel
DMA
BAM
+MCDMA
BAM+BAS
+MCDMA
H-Tile
Device-side BAM + Yes Yes Yes Yes No No
Packet MCDMA
Loopback Multi
channel
DMA
BAM+BAS
+MCDMA

Packet BAM + Yes Yes Yes Yes No No

Generate/ MCDMA
Check Multi
channel
DMA
BAM+BAS
+MCDMA

Traffic BAM+BAS Yes Yes Yes Yes No No

Generator/
Checker

Note: SR-IOV simulation support is provided only for 1 physical function and its VFs.

Note: SR-IOV is not supported for simulation in BAM+BAS+MCDMA mode.

Table 36. Supported Simulators for P-Tile MCDMA IP

Tile Design User Mode VCS VCS MX Xcelium QuestaSim Questa Aldec
Example Intel FPGA Riviera Pro
Edition

PIO using Multi Yes Yes No Yes Yes No

Bypass channel
mode DMA
Bursting
Master
P-Tile
BAM+BAS
BAM
+MCDMA
Data Mover
Only
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
49
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Tile Design User Mode VCS VCS MX Xcelium QuestaSim Questa Aldec
Example Intel FPGA Riviera Pro
Edition

BAM+BAS
+MCDMA

AVMM DMA Multi Yes Yes No No No No

channel
DMA
BAM
+MCDMA
BAM+BAS
+MCDMA

Device-side BAM + Yes Yes No No No No

Packet MCDMA
Loopback Multi
channel
DMA
BAM+BAS
+MCDMA

Packet BAM + Yes Yes No No No No

Generate/ MCDMA
Check Multi
channel
DMA
BAM+BAS
+MCDMA

Traffic BAM+BAS Yes Yes No No No No

Generator/
Checker

External Data Mover Yes Yes No No No No

Descriptor Only
Controller

Note: SR-IOV is not supported in simulation

Table 37. Supported Simulators for F-Tile MCDMA IP

Tile Design User Mode VCS VCS MX Xcelium QuestaSim Questa Aldec
Example Intel FPGA Riviera Pro
Edition

PIO using Multi Yes Yes No No No No

Bypass channel
mode DMA
Bursting
Master
BAM+BAS
BAM
+MCDMA
F-Tile Data Mover
Only
BAM+BAS
+MCDMA

AVMM DMA Multi Yes Yes No No No No

channel
DMA
BAM
+MCDMA
continued...

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
50
3. Design Example Quick Start Guide
683517 | 2024.11.04

Tile Design User Mode VCS VCS MX Xcelium QuestaSim Questa Aldec
Example Intel FPGA Riviera Pro
Edition

BAM+BAS
+MCDMA

Device-side BAM + Yes Yes No No No No

Packet MCDMA
Loopback Multi
channel
DMA
BAM+BAS
+MCDMA

Packet BAM + Yes Yes No No No No

Generate/ MCDMA
Check Multi
channel
DMA
BAM+BAS
+MCDMA

Traffic BAM_BAS Yes Yes No No No No

Generator/
Checker

External Data Mover Yes Yes No No No No

Descriptor Only
Controller

Note: SR-IOV is not supported in simulation

Note: MCDMA F-Tile 1x4 design example does not support simulation.

Note: F-Tile MCDMA supports PIPE mode simulations.

Table 38. Supported Simulators for R-Tile MCDMA IP

Tile Design User Mode VCS VCS MX Xcelium QuestaSim Questa Aldec
Example Intel FPGA Riviera Pro
Edition

PIO using Multi Yes Yes No Yes Yes No

Bypass channel
mode DMA
Bursting
Master
BAM+BAS
BAM
+MCDMA
BAM + BAS
R-Tile + MCDMA
Data Mover
Only

AVMM DMA Multi No No No No No No

channel
DMA
BAM
+MCDMA
BAM + BAS
+ MCDMA
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
51
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Tile Design User Mode VCS VCS MX Xcelium QuestaSim Questa Aldec
Example Intel FPGA Riviera Pro
Edition

Device-side BAM + No No No No No No
Packet MCDMA
Loopback BAM + BAS
+ MCDMA
Multi
channel
DMA

Packet BAM + No No No No No No
Generate/ MCDMA
Check BAM + BAS
+ MCDMA
Multi
channel
DMA

Traffic BAM+BAS No No No No No No
Generator/
Checker

External Data Mover No No No No No No

Descriptor Only
Controller

Note: SR-IOV is not supported in simulation

Note: MCDMA R-Tile 4x4 PIO using Bypass Mode design example does not support
simulation.

Note: Data Mover Only Mode is not available in R-Tile MCDMA IP x4 topology.

Note: The R-Tile design example does not support PIPE mode simulations.

3.3.3. Example Testbench Flow for DMA Test with Avalon-ST Packet
Generate/Check Design Example
The DMA testbench for the Avalon-ST Packet Generate/Check design example
demonstrates the following two major tasks:
• Host-to-Device: Transferring packets stored in the host memory to the Packet
Checker in the design example user logic, where a checker module verifies the
integrity of the packet
• Device-to-Host: Packets generated from a Generator module are transferred to the
host memory where the host checks the packet integrity

Note: This testbench implements transfer of one packet with length of 4096 bytes.

The DMA testbench for the design example completes the following tasks

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
52
3. Design Example Quick Start Guide
683517 | 2024.11.04

1. Set up 4096 bytes of incrementing data pattern for testing data movement from
the host to the device and then back to the host.
2. Write the expected packet length value (4096 bytes) to the Packet Generation and
Checker in the design example user logic through the PIO. This value is used by
the Packet checker module for testing packet integrity.
3. MSI-X is enabled and configured for launching a memory write to signal the end of
each descriptor’s DMA transaction. Write-Back function is kept disabled for the
simulation.
4. Set up the H2D (Host-to-Device) queue in the Multi Channel DMA.
5. Set up three H2D descriptors in the host memory, with the source address
pointing to the incrementing data pattern locations in the host memory. The start
of packet (SOF) and end of packet (EOF) markers along with packet length are
indicated in the descriptors.
6. At the last step of the Queue programming, the Multi Channel DMA tail pointer
register is written, which triggers the Multi Channel DMA to start the H2D DMA
transaction.
7. The previous step instructs the H2D Data Mover to fetch the descriptors from the
host memory.
8. The Multi Channel DMA H2D Data Mover reads the data from the host memory and
forwards the packet to the Packet Generator and Checker through the AVST
Streaming interface.
9. The checker module receives the packet and checks for integrity by testing the
data pattern, length as expected and proper receipt of the “end of packet” marker.
If the packet is found to be proper, the good packet count is incremented by 1 else
the bad packet count is incremented.
10. The testbench does a PIO read access of the Good Packet Count and Bad Packet
Count registers and displays the test success or failure status.
11. MSI-X write commands are triggered for every description or completion which are
checked by the testbench for proper receipt.
12. Next, set up the D2H (Device-to-Host) Queue.
13. Setup three D2H descriptors in the host memory, with the destination address
pointing to a new address space in host memory which is pre-filled with all zeroes.
14. At the last step of the Queue programming, the Multi Channel DMA tail pointer
register is written, which triggers the Multi Channel DMA to start the D2H DMA
transaction.
15. The previous step instructs the H2D Data Mover to fetch the descriptors from the
host memory to start the D2H DMA transaction.
16. The Multi Channel DMA D2H Data Mover reads the incoming packet from the
Packet Generator and writes the data to the host memory according to the
descriptors fetched in the previous step.
17. MSI-X write commands are triggered for every description completion which are
checked by the testbench for proper receipt.
18. Compares the data written back to the system memory in D2H task with the
standard incrementing pattern and declare test success/failure.

The simulation reports Simulation stopped due to successful completion

if no errors occur.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
53
 3. Design Example Quick Start Guide
683517 | 2024.11.04

3.3.4. Run the Simulation Script

Figure 28. Simulation Script

Change to Run Analyze

Testbench Directory <Simulation Script> Results

1. Change to the testbench simulation directory, pcie_ed_tb/pcie_ed_tb/sim/

<simulators>.
2. Run the simulation script for the simulator of your choice. Refer to the table below.
3. Analyze the results.

3.3.5. Steps to Run the Simulation

3.3.5.1. Steps to Run the Simulation : QuestaSim / Questa Intel FPGA Edition

Simulation Directory

<example_design>/pcie_ed_tb/ pcie_ed _tb/sim/mentor/

Instructions

1. Invoke vsim (by typing vsim, which brings up a console window where you can
run the following commands).
2. set USER_DEFINED_ELAB_OPTIONS "-voptargs=\"-noprotectopt\""
3. do msim_setup.tcl

Note: Alternatively, instead of doing Steps 1 and 2, you can type: vsim -c -do
msim_setup.tcl
4. ld_debug

5. run -all

6. A successful simulation ends with the following message: "Simulation

stopped due to successful completion!"

3.3.5.2. Steps to Run the Simulation : VCS*/VCS* MX

Simulation Directory

<example_design> /pcie_ed_tb/ pcie_ed _tb/sim/synopsys/vcs

<example_design>/pcie_ed_tb/pcie_ed _tb/sim/synopsys/vcsmx

Instructions

Note: Each simulation command below is a single-line command

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
54
3. Design Example Quick Start Guide
683517 | 2024.11.04

1. H/F/P/R Tile VCS:

sh vcs_setup.sh USER_DEFINED_COMPILE_OPTIONS="" USER_DEFINED_ELAB_OPTIONS="-
xlrm\ uniq_prior_final\ +vcs+vcdpluson\ -debug_all"
USER_DEFINED_SIM_OPTIONS="" | tee simulation.log

Note: If R-Tile MCDMA is configured in PIPE Mode, use the following command
instead:
sh vcs_setup.sh USER_DEFINED_COMPILE_OPTIONS=""
USER_DEFINED_ELAB_OPTIONS="+define
+XTOR_PCIECXL_LM_SVS_SERDES_ARCHITECTURE\ +define+RTILE_PIPE_MODE"
USER_DEFINED_SIM_OPTIONS=""
| tee simulation.log

Note that this is a single-line command.

2. H/F/P/R-Tile VCS MX:
sh vcsmx_setup.sh USER_DEFINED_COMPILE_OPTIONS=""
USER_DEFINED_ELAB_OPTIONS="- xlrm\ uniq_prior_final\ +vcs+vcdpluson\ -
debug_all" USER_DEFINED_SIM_OPTIONS="" | tee simulation.log

3.3.5.3. Steps to Run the Simulation : Xcelium*

Simulation Directory

<example_design>/pcie_ed_tb/pcie_ed_tb/sim/xcelium

Instructions

1. sh xcelium_setup.sh USER_DEFINED_SIM_OPTIONS="" USER_DEFINED_ELAB_OPTIONS ="-

timescale\ 1ns/1ps\ -NOWARN\ CSINFI"

Note: The simulation command above is a single-line command

2. Xcelium* simulation command for F-Tile MCDMA IP
sh xcelium_setup.sh USER_DEFINED_VERILOG_COMPILE_OPTIONS="+define+RTLSIM\
+define+SSM_SEQUENCE\ -sv" USER_DEFINED_ELAB_OPTIONS="-warn_multiple_driver\
-timescale\ 1ns/1ps" USER_DEFINED_SIM_OPTIONS="" | tee simulation.log

Note: The simulation command above is a single-line command

3. A successful simulation ends with the following message: "Simulation
stopped due to successful completion!"

3.3.6. View the Results

To view the Simulation Logs, Simulation Waveforms and Hardware Test Results for
each design example, refer to Design Example Detailed Description on page 7.

3.4. Compiling the Example Design in Quartus Prime

To compile the example design, follow these steps:
1. Navigate to the design example directory,
intel_pcie_mcdma_0_example_design, and open the Quartus Prime project
file, pcie_ed.qpf in Quartus Prime Pro Edition software.
2. On the Processing menu, select Start Compilation (use the button circled in
green in the image below).

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
55
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Figure 29. Design Example Compilation

3.5. Running the Design Example Application on a Hardware Setup

The following list details the development kits used for testing:
• Stratix 10 GX FPGA Development Kit
• Stratix 10 MX FPGA Development Kit
• Stratix 10 DX FPGA Development Kit
• Agilex 7 F-Series FPGA Development Kit (P-Tile and E-Tile)
• Agilex 7 F-Series FPGA Development Kit (2x F-Tile)
• Agilex 7 FPGA I-Series Development Kit (2x R-Tile and 1x F-Tile) with R-Tile die
revision A0 and B0 ES variants

Note: Set the PCIe refclk switch on the board to select the common refclk.

3.5.1. Program the FPGA

Note: This section describes how to program the FPGA using the Stratix 10 Development
Board. If you are using one of the boards listed in the previous section, the name of
the development board you select to use shall apply for this section accordingly.

1. Connect a FPGA programming cable to the Stratix 10 FPGA Development Board

2. On the Tools menu, select Programmer
3. In the Programmer, click Hardware Setup and verify the Stratix 10 FPGA
Development Board is detected in Hardware Setting tab and JTAG Settings tab
4. Select Auto Detect to detect the JTAG device chain
5. Select the target FPGA device in the JTAG chain, select Change File, and select
the pcie_ed.sof
6. Select Start to start programming

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
56
3. Design Example Quick Start Guide
683517 | 2024.11.04

Figure 30. Programming Stratix 10 MX FPGA Development Board

3.5.2. Quick Start Guide

3.5.2.1. Software Test Setup

The following host configuration is used to test the functionality of the design
example:

Table 39. Operating System: CentOS Linux 7.8

Operating System CentOS Linux release 7.8

GCC Version 4.8.5

Kernel 3.10

Support added in Custom, DPDK and NetDev driver to run with Ubuntu 22.04 with
kernel 5.15.0-xx-generic

Table 40. Operating System: Ubuntu 22.04 LTS

Operating System Ubuntu 22.04 LTS

GCC Version 11.4

Kernel v5.15

3.5.2.2. Driver Support

Check the kernel version using this command:

$ uname -r

Expected output:

5.15.0-xx-generic

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
57
 3. Design Example Quick Start Guide
683517 | 2024.11.04

If this is not the kernel version in your Ubuntu 22.04 system, follow the steps below.
These steps will change the kernel from HWE to GA Linux 5.15.0-xx-generic and install
Linux headers and gcc required for the MCDMA drivers.

sudo apt install linux-image-generic

// Install GA.

// Reboot into the newly-installed older kernel.

sudo apt remove linux-generic-hwe-22.04

// Remove HWE.

sudo apt autoremove

// Clean up.

sudo apt-get install linux-headers-generic gcc

Check the gcc version using this command:

$ gcc --version

If the gcc is not gcc-11, install gcc-11 using this command:

$ sudo apt install gcc-11

To switch between the installed gcc versions, use the update-alternatives tool and
select gcc-11.

$ sudo update-alternatives --config gcc

Note: Ensure that Proxy is set. Otherwise, some of these updates do not work.

The table below summarizes the driver support for the MCDMA design example
variants. It uses the following acronyms:
• User Space I/O (UIO): A kernel base module that the PCIe device uses to expose
its resources to user space.
• Virtual Function I/O (VFIO) driver: An IOMMU/device agnostic framework for
exposing direct device access to user space in a secure, IOMMU-protected
environment.
• Data Plane Development Kit (DPDK): Consists of libraries to accelerate packet
processing workloads running on a wide variety of CPU architectures.

Note: Software directory is created multiple times depending on Hard IP mode selected
(1x16, 2x8 or 4x4) for Quartus Prime Pro Edition 23.3 version onwards.
• p0_software folder is generated for 1x16 Hard IP modes.
• p1_software folder is generated for 2x8 Hard IP modes.
• p2_software and p3_software folders are generated only for 4x4 Hard IP modes.

Note: You must use the corresponding software folder with each IP port.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
58
3. Design Example Quick Start Guide
683517 | 2024.11.04

Table 41. Driver Support for MCDMA Design Examples

Parameter Custom Driver DPDK Driver Kernel Mode Netdev Driver

Description Also known as the user This DPDK Poll Mode Driver Kernel Mode Netdev Driver
mode driver, this driver is (PMD) uses the DPDK exposes the MCDMA IP as a
created to support both UIO framework. The PMD will Network Device and enables
and VFIO base expose the device as an standard applications to
kernelmodules. This driver ethernet device. It supports perform network data
provides custom APIs and both UIO and VFIO base transfers using the Linux
can be used without kernel modules. Existing network stack.
depending on any DPDK applications can be
framework. integrated with the MCDMA
PMD.

Directory/Driver Path <example_design>/ <example_design>/ <example design>/

pX_software/user pX_software/dpdk pX_software/kernel/

SR-IOV Support Yes Yes Yes

Multi channel DMA Avalon- Yes, up to 2K Channels Yes, up to 2K Channels No

MM DMA Design Example

Multi channel DMA Avalon- Yes, up to 2K Channels Yes, up to 2K Channels No

MM DMA with SRIOV Design
Example

BAM+BAS+MCDMA Avalon- Yes, up to 2K Channels Yes, up to 2K Channels No

MM DMA Design Example

BAM+BAS+MCDMA Avalon- Yes, up to 2K Channels Yes, up to 2K Channels No

MM DMA with SR-IOV Design
Example

Multi channel DMA Avalon- Yes Yes No

MM PIO using MQDMA
Bypass Mode Design
Example

Multi channel DMA Avalon- Yes Yes Yes

ST 1-port PIO using MQDMA
Bypass Mode Design
Example

BAM+BAS+MCDMA Avalon- Yes Yes No

MM PIO using MQDMA
Bypass Mode Design
Example

BAM+BAS+MCDMA Avalon- Yes Yes No

ST 1-Port PIO using MQDMA
Bypass Mode Design
Example

Bursting Master PIO using Yes Yes No

MQDMA Bypass Mode Design
Example

Bursting Slave PIO using Yes Yes No

MQDMA Bypass Mode Design
Example

BAM+BAS PIO using MQDMA Yes Yes No

Bypass Mode Design
Example

Data Mover Only PIO using Yes No No

MQDMA Bypass Mode Design
Example
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
59
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Parameter Custom Driver DPDK Driver Kernel Mode Netdev Driver

Multi channel DMA Avalon 1- Yes Yes, 256 channels Yes, support 4 PFs, 64
port Device-side Packet channel per PF
Loopback Design Example

Multi channel DMA Avalon 1- Yes Yes, 256 channels No

port Device-side Packet
Loopback with SRIOV Design
Example

Multi channel DMA Avalon1- Yes Yes, 256 channels No

port Packet Generate/ Check
Design Example

Multi channel DMA Avalon 1- Yes No No

port Packet Generate/ Check
with SR-IOV Design Example

BAM+BAS Traffic Generator/ Yes Yes No

Checker Design Example

Data Mover Only External Yes No No

Descriptor Controller Design
Example

3.5.2.3. MCDMA Custom Driver

3.5.2.3.1. Prerequisites

Configuration Changes from BIOS

Enable the following parameters from the BIOS:

1. KVM
2. VT-d (or AMD-V for AMD processors)
3. SRIOV Enable
4. ARI (Alternative Routing ID Interpretation)

Make sure the IOMMU is enabled on the host by using the following command:

$ virt-host-validate | grep IOMMU

QEMU: Checking for device assignment IOMMU support : PASS

QEMU: Checking if IOMMU is enabled by kernel : PASS

External Packages

To run the DPDK software driver, you must install the numactl-devel package:
• CentOS:
yum install numactl-devel

• Ubuntu:
apt-get install numactl

To create a VM environment with QEMU, install the following software:

1. Use the command below to install the packages:

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
60
3. Design Example Quick Start Guide
683517 | 2024.11.04

• CentOS:
yum install qemu-kvm qemu-img libvirt virt-install libvirt-client

• Ubuntu:
apt-get install qemu qemu-kvm virt-manager
apt-get install python-is-python3 libjpeg62
apt-get install make ninja-build pkg-config libglib2.0-dev libpixman-1-
dev

Note: Install libpng using the following steps and create a softlink.
Download from here:
https://sourceforge.net/projects/libpng/files/libpng15/1.5.30/
libpng-1.5.30.tar.gz/download
Copy the downloaded file to ~/Downloads.
$ cd ~/Downloads/
$ tar -xvf libpng-1.5.30.tar.gz
$ cd libpng-1.5.30/
$ ./configure --prefix=/usr/local
$ make check
$ sudo make install
$ sudo ln -s /usr/local/lib/libpng15.so /usr/lib/
2. Download the QEMU software from the official site.

Note: For testing over VM, you need to generate the necessary qcow2 file.

Set the Boot Parameters

Follow the steps below to modify the default hugepages setting in the grub files:
1. Edit the /etc/default/grub file
Append the highlighted parameters to the GRUB_CMDLINE_LINUX line in the /etc/
default/grub file
CentOS: GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root
rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G
hugepages=40 panic=1 intel_iommu=on iommu=pt
Ubuntu: GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G
hugepages=20 intel_iommu=on iommu=pt panic=1 quiet splash
vt.handoff=7"
File contents after the edit for CentOS is shown below:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb
default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on
iommu=pt
GRUB_DISABLE_RECOVERY="true"

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
61
 3. Design Example Quick Start Guide
683517 | 2024.11.04

In the case of memory allocation failure at the time of Virtual Function creation,
add the following boot parameters:
"pci=hpbussize=10,hpmemsize=2M,nocrs,realloc=on"
To bind the device to vfio-pci and use IOMMU, enable the following parameter:
intel_iommu=on
To use UIO and not enable the IOMMU lookup, add the following parameter:
iommu=pt
To use the AMD platform and the UIO driver, add the following parameter at boot
time: iommu=soft
An example /etc/default/grub file on ubuntu after the edits can be seen
below:
root@bapvecise042:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update

# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:

# info -f grub -n 'Simple configuration'

GRUB_DEFAULT="1>2"

GRUB_TIMEOUT_STYLE=hidden

GRUB_TIMEOUT=0

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20

intel_iommu=on iommu=pt panic=1"

# Uncomment to enable BadRAM filtering, modify to suit your needs

# This works with Linux (no patch required) and with any kernel that obtains

# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)

#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)

#GRUB_TERMINAL=console

# The resolution used on graphical terminal

# note that you can use only modes which your graphic card supports via VBE

# you can see them in real GRUB with the command `vbeinfo'

#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to

Linux

#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries

#GRUB_DISABLE_RECOVERY="true"

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
62
3. Design Example Quick Start Guide
683517 | 2024.11.04

# Uncomment to get a beep at grub start

#GRUB_INIT_TUNE="480 440 1"

2. Generate GRUB configuration files.

To check whether the boot system is legacy or EFI-based, check the existence of
the following file:
$ls -al /sys/firmware/efi
If this file is present, the boot system is EFI-based. Otherwise, it is a legacy
system.
a. In case of a legacy system, execute the following command:
$ grub2-mkconfig -o /boot/grub2/grub.cfg
b. In case of an EFI-based system, execute the following command:
$ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
c. In case of Ubuntu, execute the following command:

grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
or
grub2-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
or
sudo grub update

3. Reboot the system.

4. Verify the changes above:
$ cat /proc/cmdline
5. Set the huge pages:
$ echo 40 > /proc/sys/vm/nr_hugepages
6. If host supports multiple NUMAs, follow the following steps:
a. Check how many NUMAs enabled on host.
$lscpu | grep NUMA
NUMA node(s): 2
NUMA node0 CPU(s): 0-15, 32-47
NUMA node1 CPU(s): 16-31, 48-63

In this example, we have 2 NUMAs. If only one NUMA is present, ignore this
step:
b. Check which device is provisioned:
$cat /sys/class/pci_bus/<Domain:Bus>\
/device/numa_node

c. Enable the Huge pages, which ever NUMA, device is located:

$echo 40> /sys/devices/system/node/node<nodenum>\
/hugepages/hugepages-1048576kB/nr_hugepages

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
63
 3. Design Example Quick Start Guide
683517 | 2024.11.04

d. Configure thread sequence based on which NUMA device is located, for

example:
#define THREAD_SEQ "0-15" in software/user/cli/\
perfq_app/perfq_app.h

3.5.2.3.2. Software Setup

Installing the Linux Kernel Driver

1. Install the UIO driver If we are proceeding with UIO support. If we are proceeding
with vfio, this step not required:
$ modprobe uio
2. Build the mcdma kernel driver and load:
a. $ cd software/kernel
b. $ make clean all -C driver/kmod/mcdma-custom-driver
c. $ insmod driver/kmod/mcdma-custom-driver/ifc_uio.ko
3. Verify whether driver is loaded or not:
$ lspci -d 1172:000 -v | grep ifc_uio
(Kernel driver in use: ifc_uio)

Currently, UIO is the default.

To use the vfio driver, modify UIO_SUPPORT in common/mk/common.mk as follows:

__cflags += -UUIO_SUPPORT

Currently, the build for the 64-bit OS is the default.

To change to building for a 32-bit OS, modify "software/user/common/mk/

common.mk" as follows:

Undefine IFC_PIO64 and define IFC_PIO32 and IFC_32BIT_SUPPORT

__cflags += -UIFC_PIO_64

__cflags += -DIFC_PIO_32

__cflags += -DIFC_32BIT_SUPPORT

For a 64-bit OS, if the 32-bit PIO is to be enabled instead of the 64-bit PIO, modify
"software/user/common/mk/common.mk" as follows:

Undefine IFC_PIO64 and define IFC_PIO32

__cflags += -UIFC_PIO_64

__cflags += -DIFC_PIO_32
1. Install vfio-pci module.
$ modprobe vfio-pci
2. Bind the device to vfio-pci

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
64
3. Design Example Quick Start Guide
683517 | 2024.11.04

a. If the device is bound to ifc_uio, unbind with the following command:

$ echo "<bdf>" > /sys/bus/pci/devices/<bdf>/driver/unbind
E.g: echo "0000:01:00.0" > /sys/bus/pci/devices/
0000\:01\:00.0/driver/unbind
b. Bind the device to vfio-pci
echo <PCI Vendor ID> <PCI Device ID> > /sys/bus/pci/
drivers/vfio-pci/new_id
Example: echo 1172 0000 > /sys/bus/pci/drivers/vfio-pci/
new_id

Note: For VFIO, in case of multi PF scenarios, you must check whether the BDFs are in same
IOMMU group or not using the command: readlink /sys/bus/pci/
devices/BDF/iommu_group

Example: readlink /sys/bus/pci/devices/0000:01:00.0/iommu_group

Note: If BDFs are in same IOMMU group, you need to apply the ACS patch, else its not
required.

Build and Install User Space Library

1. Build the library

$ cd software/user
For a 64-bit OS:
$ make clean all -C libmqdma/
For a 32-bit OS:
$ make clean 32BIT=1 all -C libmqdma/
2. Load the library
For a 64-bit system with CentOS:
$ rm -f /usr/lib64/libmqdmasoc.so
$ cp libmqdma/libmqdmasoc.so /usr/lib64/
For a 32-bit system with CentOS:
$ rm -f /usr/lib/libmqdmasoc.so
$ cp libmqdma/libmqdmasoc.so /usr/lib/
For hosts with Ubuntu OS:
$ cp libmqdma/libmqdmasoc.so /usr/local/lib

Make sure that ldconfig output contains libmqdma.

$ ldconfig -v | grep libmqdmasoc.so
Look for this in the output:
libmqdmasoc.so -> libmqdmasoc.so

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
65
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Enabling VFs and Create Guest VM by Using QEMU

Follow the steps below to create the guest environment and assign VF device to VM by
using QEMU:
1. Enable Virtual functions based on requirements:
$ echo 2 > /sys/bus/pci/devices/<bdf>/max_vfs

2. Install vfio-pci module

$modprobe vfio-pci
3. Bind the device to vfio-pci
a. If the device is bound to ifc_uio, unbind with the following command:
$ echo "<bdf>" > /sys/bus/pci/devices/<bdf>/driver/unbind
E.g: echo "0000:01:00.0" > /sys/bus/pci/devices/
0000\:01\:00.0/driver/unbind
b. Bind the device to vfio-pci
echo <PCI Vendor ID> <PCI Device ID> > /sys/bus/pci/
drivers/vfio-pci/new_id
Example: echo 1172 0000 > /sys/bus/pci/drivers/vfio-pci/
new_id
4. Use following parameters to start the Guest VM (on Intel machines, use QEMU
version 3.0.0-rc0). Here, a minimum of 8 GB of memory needs to be allocated to
VM.
$ qemu-3.0.0-rc0/x86_64-softmmu/qemu-system-x86_64 -smp 2 -m
8192M -boot c -machine q35,kernel-irqchip=split -cpu host -
enable-kvm -nographic -L /root/qemu-3.0.0-rc0/pc-bios -name
offload1 -hda <path_to_qcow2_file> -device vfio-pci,host=<bdf>
-netdev
type=tap,id=net6551,script=no,downscript=no,vhost=on,ifname=ne
t6551 -device virtio-net-pci,netdev=net6551 -device intel-
iommu,intremap=on,caching-mode=on -serial
telnet::5551,server,nowait -object memory-backend-
file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa
node,memdev=mem -mem-prealloc -monitor
telnet::3331,server,nowait&

Note: 1. On AMD machines, the parameter “-device intel-

iommu,intremap=on,caching-mode=on” is not required.
2. If multiple devices are in the same group, check for ACS enablement in the root
port and bridge.
3. Below are the Host and Guest VM configurations used for verification.

Table 42. Host System Configuration

Host System Configuration Details

Operating System CentOS Linux release 7.8

continued...

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
66
3. Design Example Quick Start Guide
683517 | 2024.11.04

Host System Configuration Details

Ubuntu 22.04 LTS

CentOS: 3.10.0-1127.10.1.el7.28.x86_64
Linux Kernel
Ubuntu: 5.15.0-52-generic x86_64

CPU Cores 96

RAM 128 GB (64 GB on single NUMA)

Hypervisor KVM

QEMU Version QEMU version 3.0.0-rc0

Table 43. Guest System Configuration

Host System Configuration Details

Operating System CentOS Linux release 7.8

Linux Kernel 3.10.0-1127.10.1.el7.28.x86_64

CPU Cores 2

RAM 8 GB

Establish Communication Between Host and QEMU

1. Login to Guest VM.

$telnet localhost 5551
2. Bring up interface.
$ifconfig eth0 up
3. Assign the IP address to eth0.
$ifconfig eth0 <1.1.1.11>

Execute the following command from the Host:

1. Bring up net6551. "net6551" is the tap interface that you give in the QEMU
command.
$ifconfig net6551 up
$ifconfig net6551 <1.1.1.12>
2. Copy the code by using scp.
Eg: $scp -r <directory> root@<GuestIP>:/
<path_to_the_directory>/
3. Refer to section 3.5.2.3.1 for updating the grub file in VM.
4. Enable NO_IOMMU mode inside the Guest VM.
Currently, QEMU does not support IOMMU. When running software on the Guest
VM, if you are using vfio, you need to enable the NO_IOMMU_MODE mode.
Edit software/dpdk/dpdk/drivers/net/mcdma/ rte_pmd_mcdma.h to
define the following macro.
Example: #define NO_IOMMU_MODE

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
67
 3. Design Example Quick Start Guide
683517 | 2024.11.04

By default, this macro is disabled.

5. Refer to Software Setup on page 64 for the procedure to build and install MCDMA.

3.5.2.3.3. Run the Reference Example Application

1. Build the reference application.

$ cd software/user/cli/perfq_app/
$ make clean && make all
$ ./perfq_app -h

2. This command displays the available options in the application, as shown in the
image below:

Refer to the README file located in the software/user/cli/perfq_app

directory for more information.
3. Perform a PIO test to check if the setup is correct. If successful, the application
displays a Pass status as shown in the image below:

Here the -b option should be provided with the correct BDF in the system.
4. Perform IP reset.
This step will perform an IP Reset. You can perform this step before every run.
Build devmem utility:
$ cd software/user/cli/devmem
$ make clean all
$ ./devmem 0000:01:00.0 0 0x00200120 0x1

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
68
3. Design Example Quick Start Guide
683517 | 2024.11.04

5. For Channel ID VF PF verification: Channel ID , VF and PF information is inserted

in AVST/AXIST data pattern. To test dma in proper channel and VF, it can be
tested as described below:
For AVST/AXIST LB : In the file software/user/common/mk/common.mk,
enable CID _PAT.
For Example: __
cflags += -DCID_PAT

Data validation mode supported. Performance mode is disabled.

__cflags += -DPERFQ_LOAD_DATA \
__cflags += -UPERFQ_PERF

Supported direction: Applicable Bi-directional (-i) only.

For AVST/AXIST PKT GEN: The following configuration should be made in
software/user/common/mk/common.mk
Enable both CID_PAT and IFC_PROG_DATA_EN

__cflags += -DCID_PAT
__cflags += -DIFC_PROG_DATA_EN
DIFC_PROG_DATA_EN

In validation mode, enable DPERFQ_LOAD_DATA flag and disable PERFQ_PERF

flag.

__cflags += -DPERFQ_LOAD_DATA
__cflags += -UPERFQ_PERF

Supported direction: Bi-directional (-Z), Tx (-t) and Rx (-r) as well.

6. Example of testing a loopback design with the following configuration:
Command:
$ ./perfq_app -b 0000:01:00.0 -p 32768 -l 5 -i -c 2 -d 2 -a 4

Configuration:
a. bdf (-b 0000:01:00.0)
b. 2 channels (-c 2)
c. Loopback (-i)
d. Payload length of 32,768 bytes in each descriptor (-p 32768)
e. Time Limit (-l 5)
f. Debug log enabled (-d 2)
g. One thread per queue (-a 4)

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
69
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16
configuration.

Note: This hardware test was run with the Agilex 7 P-Tile PCIe Gen4x16
configuration.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
70
3. Design Example Quick Start Guide
683517 | 2024.11.04

Figure 31. Custom AVST DMA Gen4 x16 : P-Tile Hardware Test Result

Note: For Rx/Tx test:

a. AVMM should use -u instead of -i
b. -i should be used for AVST/AXIST Loopback case
c. -z should be used in AVST/AXIST PKT case
7. Example of verifying on AVMM DMA:
Modify the below micro in the file: user/common/include/ifc_libmqdma.h
#define PCIe_SLOT 0 /* 0 - x16, 1 - x8 */

Command:
$ ./perfq_app -b 0000:01:00.0 -p 32768 -l 5 -u -c 2 -d 2 -a 4

Configuration:
a. bdf (-b 0000:01:00.0)
b. 2 channels (-c 2)
c. bi-directional DMA transfer (-u)
d. Payload length of 32768 bytes in each descriptor (-p 32768)
e. Time Limit set to 5 (-l 5)
f. Debug log enabled (-d 2)
g. One thread per queue (-a 4)

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
71
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: To test the data validity, you need to perform H2D then D2H operations.

Figure 32. Custom AVMM DMA Gen4 x16 : P-Tile Hardware Test Result

Note: Execute the following command for MSIX:

$ulimit -n 8192

To enable data validation using -v option, set the software flags in user/
common/mk/common.mk as follows:
cflags += -UPERFQ_PERF
cflags += -DPERFQ_LOAD_DATA

Meta Data Test

Meta data is the 8-byte private data that the host sends to the device in the H2D
direction and the device sends to the host in the D2H direction.
1. If we have meta data enabled in the bitstream, modify IFC_QDMA_META_DATA
like below in common/mk/common.mk
__cflags += -DIFC_QDMA_META_DATA
__cflags += -DPERFQ_LOAD_DATA
2. Continue with user library build and installation steps.
3. Command:
./perfq_app -p 128 -s 128 -d 1 -i -a 2 -b 0000:01:00.0 -v -c 1

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
72
3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: To verify the meta data functionality, every descriptor should have eof set to 1. Use
the commands in such a way that every descriptor contains a single file. Use
perfq_app with the -s parameter.

Custom PIO Read Write Test

You can read and write from PIO address range in bar 2 from any valid custom
memory.

Parameters for Write operation

-b <bdf>
-o
--pio_w_addr=<address>
--pio_w_val=<value to write>
--bar=<bar number>

Example:
# ./software/user/cli/perfq_app# ./perfq_app -b 0000:01:00.0 -o --
pio_w_addr=0x1010 --pio_w_val=0x30 --bar=2

WRITE: PIO Address = 0x1010 Value = 0x30, bar = 2

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
73
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Parameters for Read operation

-b <bdf>
-o
--pio_r_addr=<address>
--bar=<bar number>

Example:
# ./software/user/cli/perfq_app# ./perfq_app -b 0000:01:00.0 -o --
pio_r_addr=0x1010 --bar=2

READ: PIO Address = 0x1010 Value = 0x30, bar = 2

BAM Test

If the BAM support is enabled on hardware, enable the following flag in common/mk/
common.mk:

__cflags += -DIFC_PIO_256 → 256b read/write operations on PIO BAR

__cflags += -DIFC_PIO_128 → 128b read/write operations on PIO BAR

For PIO using Bypass with BAM/BAS user mode, you are required to change the define
parameter to undef: #undef IFC_QDMA_INTF_ST(software/user/common/
include/mcdma_ip_params.h)

To enable 256 bit read/write operations, this is the difference in common.mk:

Command: ./cli/perfq_app/perfq_app -b 0000:86:00.0 -o

To enable 128 bit read/write operations:

Command:

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
74
3. Design Example Quick Start Guide
683517 | 2024.11.04

To test the PIO/BAM performance, please use the following command:

Command:
[root@bapvemb005t perfq_app]# ./perfq_app -b 0000:01:00.0 --bam_perf -o
PIO 64 Write and Read Perf Test ...
Total Bandwidth: 0.14GBPS
Pass
[root@bapvemb005t perfq_app]#

• By default BAM/BAS, BAR is 2. If DMA Hardware supports both BAM/BAS and BAR
numbers are different, then pass BAR number parameter as below:
--bar=2 for BAM
--bar=0 for BAS
• For example:
./perfq_app -b 0000:01:00.0 --bam_perf -o --bar=2

• Performance mode of BAM tries to send the data for 10 seconds and calculates the
bandwidth
• PIO 256b test may display fail because of the reason that 2k memory only enabled
in device and PIO test trying to access the memory > 2k

BAS Test

Note: For Traffic Generator/Checker example design, you must disable MSI-X parameter,
IFC_QDMA_MSIX_ENABLE, in Custom Driver's software/kernel/common/
include/mcdma_ip_params.h if MSI-X is not enabled in the IP Parameter Editor
GUI. By default, the Custom Driver software parameter is enabled and MSI-X is
disabled in the IP. This mismatch prevents ifc_uio kernel module from being loaded.

For BAS x4:

1. Set the PCIe_SLOT “2” in ifc_libmqdma.h(software/user/ common/
include/ifc_libmqdma.h)
2. BAS x4 supports burst length 32 by default. In the file perfq_app.h (software/
user/cli/perfq_app/perfq_app.h)
#define IFC_MCDMA_BAS_X4_BURST_LENGTH 32

BAS Programming Sequence

The MCDMA BAS programming sequence consists of the following steps defined in the
following sections:

Using Traffic Generator (Write in Host Memory)

The following is the programming sequence:

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
75
 3. Design Example Quick Start Guide
683517 | 2024.11.04

1. Allocate DMA-able memory in the host system.

2. Program the base address with the write_map_table with the physical address of
the table.
3. Set the write address register with the offset in the block where the Traffic
generator needs to write the data.
4. Set how many number of bursts BAS should write in the host memory in the
WRITE_COUNT register.
5. Set enable bit to start traffic generation.

Using Traffic Checker (Reads from Host Memory)

The following belongs to Traffic checker:

1. Allocate DMA-able memory in the host system.
2. Program the base address with the read_map_table with the physical address of
the host memory.
3. Set the read address register with the offset in the block where the Traffic
generator needs to read the data.
4. Set how many number of bursts BAS should read in the host memory in the
READ_COUNT register.
5. Set enable bit to start traffic generation.
BAS Verification

BAS support is enabled on the hardware. Enable the following flag in user/common/
include/ifc_libmqdma.h
#define PCIe_SLOT 0 /* 0 – x16, 1 – x8, 2 - x4 */

To enable BAS in common.mk (software/user/common/mk/common.mk), use the

following command:
__cflags += -DIFC_MCDMA_BAS_EN

Commands:

To verify the write operation:

./cli/perfq_app/perfq_app -b 0000:01:00.0 -s 512 -e -t --bar=0

Figure 33. BAS Write Operation

To verify the read operation:

./cli/perfq_app/perfq_app -b 0000:01:00.0 -s 512 -e -r --bar=0

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
76
3. Design Example Quick Start Guide
683517 | 2024.11.04

Figure 34. BAS Read Operation

To verify the write and read operation:

./cli/perfq_app/perfq_app -b 0000:01:00.0 -s 512 -e -z --bar=0

Figure 35. BAS Write and Read Operation

Performance test:

The below log is collected on Gen3x16 P-Tile:

./cli/perfq_app/perfq_app -b 0000:01:00.0 -s 16384 –-bas_perf -z --bar=0

Figure 36. BAS Write and Read Performance Test

Note: You may not able to proceed with -z option. Please add flag #define
IFC_QDMA_INTF_ST in user/common/include/mcdma_ip_params.h as a
workaround to make it work.

Note: In case of VFIO, to run BAM+BAS+MCDMA, you need to create at least 3 VFs and run
on each VFs respectively. If you try to use one VF to run BAM+BAS+MCDMA
simultaneously in case of VFIO, it prompts with a resource busy.

Packet Gen Test

1. Configuration in user/cli/perfq_app/perfq_app.h
In the case of static channel mapping, modify the following parameters:
/* Number of PFs */
#define IFC_QDMA_PFS <number of PFs>

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
77
 3. Design Example Quick Start Guide
683517 | 2024.11.04

/* Channels available per PF */

#define IFC_QDMA_PER_PF_CHNLS <number of channels per PF>

/* Channels available per VF */

#define IFC_QDMA_PER_VF_CHNLS <number of channels per VF>

/* Number of VFs per PF */

#define IFC_QDMA_PER_PF_VFS <number of VFs per PF>

2. Continue the procedure in the steps in Software Setup on page 64 for building and
installing MCDMA.
3. Run the perfq_app application command:
$ ./perfq_app -b 0000:01:00.0 -p 32768 -d 2 -c 1\
-a 2 -l 5 -z -n –pf=<pf number> --vf=<vfnumber>

Note: If you run DMA on PF only, then "--vf " might not be required. However, if
you run DMA on VF, then "--pf" and "--vf" both might be needed as you
need to know from which PF the VF was spawned.
Note: The hardware test was run with Gen3 x16. Test with Gen4 x16 may be
added in a future release.
Note: • PF number and VF number start with 1 in command line parameters.
• Please run the performance on PF0 with -n parameter.

3.5.2.3.4. Testing Bitstream Configuration beyond 256 Channels (for MCDMA Custom
Driver)

Note: In case, if the *.sof file generated with the number of channels > 256, please follow
this procedure.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
78
3. Design Example Quick Start Guide
683517 | 2024.11.04

1. Enable following macros in user/common/mk/common.mk

__cflags += -DIFC_MCDMA_DIDF
__cflags += -UIFC_MCDMA_SINGLE_FUNC

Note: In case of AVMM, BDF is provided as an argument and you must define
IFC_MCDMA_SINGLE_FUNC. __cflags += -DIFC_MCDMA_SINGLE_FUNC
2. Go to this location:

cd user/cli/perfq_app/

3. Perform the following command:

make clean && make all

4. Run the test.

Configuration:
• 1 channel (-c 1)
• Packet gen bidirectional (-z)
• Payload length of 64 bytes in each descriptor (-p 64)
• Transfer the data 5 seconds (-l 2)
• Number of threads that needs to be used (-a 4)
• Dump the progress logs every second (-d 1)
• File size (-f 64)
• Batch size (-x 64)
Note: To configure –f and –x manually, IFC_MCDMA_FUNC_VER should be defined.
Command for pktgen:
./perfq_app -p 64 -l 2 -z -d 1 -c 2048 -a 4 -f 508 -x 508 -n

Command for loopback:

./perfq_app -p 64 -l 2 -i -d 1 -c 2048\
-a 4 -f 1 - x64

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
79
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Note:
a. Currently, in DIDF mode, single page is supported.
b. Simultaneous process, currently cannot be supported in DIDF mode. You can
run one process with 2k channels.

3.5.2.3.5. External Descriptor Mode Verification

Use the following steps to verify the external fetch descriptors:

1. Define the following parameters in software/user/common/include/
mcdma_iparams.h .
#define IFC_MCDMA_EXTERNL_DESC

2. Build the devmem utility.

cd ./software/user/cli/devmem/

make clean all

3. Disable the queues and FIFOs.

devmem 0000:09:00.0 0 0x8 0x1

4. Enable the queues and FIFOs

devmem 0000:09:00.0 0 0x8 0x0

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
80
3. Design Example Quick Start Guide
683517 | 2024.11.04

5. Command:

./ perfq_app -b 0000:09:00.0 -d 1\
-c 16 -a 8 -p 1024 -s 102400 -u

3.5.2.4. MCDMA DPDK Poll Mode Driver

3.5.2.4.1. Prerequisites

OS version and packages to be installed

CentOS: yum install numactl-devel

Ubuntu: apt-get install numactl libnuma-dev qemu qemu-kvm virt-

manager meson python3-pyelftools netperf iperf iperf3 ethtool
dwarves git

Set the Boot Parameters

Follow the step below to modify the default hugepages setting in grub files:

Edit /etc/default/grub file

Append the highlighted parameters to the GRUB_CMDLINE_LINUX line in the /etc/

default/grub file:

GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap

rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 iommu=pt
panic=1”

CentOS: GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root
rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G
hugepages=40 iommu=pt panic=1”

Ubuntu: GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G

hugepages=20 intel_iommu=on iommu=pt panic=1 quiet splash
vt.handoff=7"

File contents after the edit for CentOS is shown below:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb
default_hugepagesz=1G hugepagesz=1G hugepages=40 iommu=pt panic=1”
GRUB_DISABLE_RECOVERY="true"

In the case of memory allocation failure at the time of Virtual Function creation, add
the following boot parameters:
"pci=hpbussize=10,hpmemsize=2M,nocrs,realloc=on"

To bind the device to vfio-pci and use IOMMU, enable the following parameter:
intel_iommu=on

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
81
 3. Design Example Quick Start Guide
683517 | 2024.11.04

To use UIO and not enable the IOMMU lookup, add the following parameter:
iommu=pt

To use the AMD platform and the UIO driver, add the following parameter at boot

time: iommu=soft

An example /etc/default/grub file on ubuntu after the edits can be seen

below:
root@bapvecise042:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update

# /boot/grub/grub.cfg.

# For full documentation of the options in this file, see:

# info -f grub -n 'Simple configuration'

GRUB_DEFAULT="1>2"

GRUB_TIMEOUT_STYLE=hidden

GRUB_TIMEOUT=0

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20

intel_iommu=on iommu=pt panic=1"

# Uncomment to enable BadRAM filtering, modify to suit your needs

# This works with Linux (no patch required) and with any kernel that obtains

# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)

#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)

#GRUB_TERMINAL=console

# The resolution used on graphical terminal

# note that you can use only modes which your graphic card supports via VBE

# you can see them in real GRUB with the command `vbeinfo'

#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux

#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries

#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start

#GRUB_INIT_TUNE="480 440 1"

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
82
3. Design Example Quick Start Guide
683517 | 2024.11.04

Generate GRUB configuration files

To check whether the boot system is legacy or EFI-based, check the existence of the
following file:
$ls -al /sys/firmware/efi
If this file is present, the boot system is EFI-based. Otherwise, it is a legacy system.

• In case of a legacy system, execute the following command:

$ grub2-mkconfig -o /boot/grub2/grub.cfg

• In case of an EFI-based system, execute the following command:

$ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

• In case of Ubuntu, execute the following command:

grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
OR
grub2-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
OR
sudo grub update

Configuration Changes from BIOS

1. Reboot the system.
2. Verify the changes above.
$ cat /proc/cmdline
3. Set the huge pages
$ echo 40 > /proc/sys/vm/nr_hugepages

Install and Build Testpmd

Note: test-pmd is supported only for CentOS and not for Ubuntu.

If testing the MCDMA by using test-pmd, use the following steps; otherwise, if using
perfq_app, skip to Install PMD and Test Application (for CentOS) on page 86:
1. Download dpdk and apply the build patches.
Execute the following commands with root as user.
$ cd software/dpdk/dpdk/patches/v20.05-rc1
$ sh apply-patch.sh
2. Enable IGB_UIO module in build configuration.
Update the following macro in ./config/common_base to “y”. By default, igb_uio is
disabled.
CONFIG_RTE_EAL_IGB_UIO=y

3. Enable the below macro for channels more than 16:

CONFIG_RTE_ETHDEV_QUEUE_STAT_CNTRS=128

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
83
 3. Design Example Quick Start Guide
683517 | 2024.11.04

4. Undefine IFC_BUF_REUSE macro for Testpmd in drivers/net/mcdma/

rte_pmd_mcdma.h, by default it is enabled.
#undef IFC_BUF_REUSE

5. In case to avoid Tx drop count, enable the following macro in the file:
drivers/net/mcdma/rte_pmd_mcdma.h.
This is applicable for pkt-gen only. Skip this step for loopback.
#define AVOID_TX_DROP_COUNT

6. Build DPDK and install.

Execute the following steps:
a. $ export DPDK_DIR= <cloned dir>/software/dpdk/patches/
v20.05-rc1/dpdk
b. $ export RTE_SDK=${DPDK_DIR}
c. $ export DPDK_TARGET=x86_64-native-linuxapp-gcc
d. $ export RTE_TARGET=x86_64-native-linuxapp-gcc
e. $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
f. $ make config T=x86_64-native-linuxapp-gcc
g. $ rm -rf x86_64-native-linuxapp-gcc
h. $ make -j32 install T=$DPDK_TARGET DESTDIR=install
7. Install the UIO base module.
$ modprobe uio
8. Install the igb_uio module.
$ insmod x86_64-native-linuxapp-gcc/build/kernel/linux/
igb_uio/igb_uio.ko
9. Bind the device to the igb_uio driver.
$ ./usertools/dpdk-devbind.py -b igb_uio <BDF>
For example:
$ ./usertools/dpdk-devbind.py -b igb_uio 01:00.0
10. Run Testpmd cli.

No. of Queue Test PMD Command Line (Loop Mode)

1 ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -w 0000:01:00.0 -- --tx-first\

--nb-cores=1 --rxq=1 --txq=1 --rxd=512 --txd=512--max-pkt-len=<payload len> --no-flush-rx\
--stats-period 1 --burst=127 --txpkts=<payload len>

2 ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -w 0000:01:00.0 -- --tx-first\

--nb-cores=2 --rxq=2 --txq=2 --rxd=512 --txd=512 --max-pkt-len=<payload len> --no-flush-rx\
--stats-period 1 –burst=254 --txpkts=<payload len>

3 ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -w 0000:01:00.0 -- --tx-first\

--nb-cores=3 --rxq=3 --txq=3 --rxd=512 --txd=512 --max-pkt-len=64 --no-flush-rx\
--stats-period 1 --burst=508 --txpkts=<payload len>

continued...

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
84
3. Design Example Quick Start Guide
683517 | 2024.11.04

No. of Queue Test PMD Command Line (Loop Mode)

4 ./x86_64-native-linuxapp-gcc/app/testpmd -c ff -n 4 -w 0000:01:00.0 -- --tx-first\

--nb-cores=4 --rxq=4 --txq=4 --rxd=512 --txd=512 --max-pkt-len=64 --no-flush-rx\
--stats-period 1 --burst=127 --txpkts=<payload len>

8 ./x86_64-native-linuxapp-gcc/app/testpmd -c fff -n 4 -w 0000:01:00.0 -- --tx-first\

--nb-cores=8 --rxq=8 --txq=8 --rxd=512 --txd=512 --max-pkt-len=<payload len> --no-flush-rx\
--stats-period 1 --burst=127 --txpkts=<payload len>

Forwarding Test PMD Command Line

Mode

Only Rx Mode ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -w 0000:01:00.0 --\

--forward-mode=rxonly --nb-cores=1 --rxq=1 --txq=1 --rxd=512 --txd=512 --max-pkt-len=<payload len>
\
--no-flush-rx --stats-period 1 --burst=127 --txpkts=<payload len>

Only Tx Mode ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -w 0000:01:00.0 --\

--forward-mode=txonly --nb-cores=1 --rxq=1 --txq=1 --rxd=512 --txd=512 --max-pkt-len=<payload len>
\
--no-flush-rx --stats-period 1 --burst=254 --txpkts=<payload len>

Note: --burst=0 : If set to 0, driver default is used (i.e. 16 burst-size). Otherwise, the Test
PMD default burst size (i.e. 32) is used. The default Testpmd pkt-len, in the case of
Tx, is 64.

Use the following command to run test-pmd:

$./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 -w
0000:01:00.0 -- --tx-first --nb-cores=1 --rxq=1 --txq=1 --rxd=512
--txd=512 --max-pkt-len=64 --no-flush-rx --stats-period 1 --
burst=127 --txpkts=64

Parameters used:
1. BDF of device. (-w 0000:01:00.0)
2. Forwarding mode (--tx-first)
3. Number of cores (--nb-cores=1)
4. Number of RX and TX queues per port (--rxq=1 --txq=1)
5. Number of descriptors in the RX and TX rings (--rxd=512 --txd=512)
6. Max packet length (max-pkt-len=64)
7. Display statistics every PERIOD seconds (--stats-period 1)
8. Number of packets per burst (--burst=127)

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
85
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16 configuration.

Note: Hardware test with P-Tile Gen4 x16 may be added in a future release.

3.5.2.4.2. Install PMD and Test Application (for CentOS)

1. Download dpdk and apply the build patches.
Execute the following commands with root as user.
$ cd software/dpdk/patches/v20.05-rc1
$ sh apply-patch.sh
2. Enable the following parameters in the build configuration.
Update the following macro in ./config/common_base to “y”. By default, igb_uio is
disabled.
CONFIG_RTE_EAL_IGB_UIO=y
Enabling VFIO: If you want to use VFIO instead of UIO:
a. In the file ./dpdk/drivers/net/mcdma/rte_pmd_mcdma.h
undefine UIO_SUPPORT #undef UIO_SUPPORT

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
86
3. Design Example Quick Start Guide
683517 | 2024.11.04

b. Update the following macro in ./config/common_base to “y”

CONFIG_RTE_EAL_VFIO=y

3. Build DPDK and install.

Execute the following steps:
• $ export DPDK_DIR= <cloned dir>/software/dpdk/patches/
v20.05-rc1/dpdk
• $ export RTE_SDK=${DPDK_DIR}
• $ export DPDK_TARGET=x86_64-native-linuxapp-gcc
• $ export RTE_TARGET=x86_64-native-linuxapp-gcc
• $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
• $ make config T=x86_64-native-linuxapp-gcc
• $ rm -rf x86_64-native-linuxapp-gcc
• $ make -j32 install T=$DPDK_TARGET DESTDIR=install
4. Execute the following steps if proceeding with UIO support.
Install UIO base module.
$ modprobe uio
Install igb_uio module.
$ insmod x86_64-native-linuxapp-gcc/build/kernel/linux/
igb_uio/igb_uio.ko
Bind the device to the igb_uio driver.
$ ./usertools/dpdk-devbind.py -b igb_uio <BDF>
To install VFIO module (If VFIO is being used instead of UIO)
$modprobe vfio-pci

Bind the device to the vfio-pci driver.

$DPDK_DIR/usertools/dpdk-devbind.py -b vfio-pci <BDF>

5. Build the reference application.

$ cd examples/mcdma-test/perfq/
$ make clean all
$ make

3.5.2.4.3. Install PMD and Test Application (for Ubuntu)

Note: You must ensure that meson utility and elf utility are installed on the system before
building DPDK v21.11.2. If not installed, use commands "sudo apt install
meson" and "sudo apt install python3-pyelftools".

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
87
 3. Design Example Quick Start Guide
683517 | 2024.11.04

1. Execute patch script from v21.11.2 directory

#cd software/dpdk/dpdk/patches/v21.11.2/

#sh ./apply-patch.sh

2. Configure and build dpdk

#cd software/dpdk/dpdk/patches/v21.11.2/dpdk-stable

#meson build #DESTDIR=install ninja -C build install

#cd build

3. Build igb_uio kernel driver and mcdma-test/perfq app

#meson configure -Dexamples=mcdma-test/perfq -Denable_kmods=true
#ninja

4. Install uio_igb or vfio:

insmod dpdk-stable/build/kernel/linux/igb_uio/igb_uio

5. To bind the driver:

dpdk-stable/usertools/dpdk-devbind.py -b igb_uio <BDF>
OR
dpdk-stable/usertools/dpdk-devbind.py -b vfio-pci <BDF>
Example:
dpdk-stable/usertools/dpdk-devbind.py -b vfio-pci 01:00.0
To unbind the driver:
echo <BDF> > sys/bus/pci/devices/<BDF>/driver/unbind
Example
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind

6. cd examples/mcdma-test/perfq

3.5.2.4.4. Create VM Environment

Enable VFs
1. Refer to the Install PMD and Test Application (for CentOS) on page 86 section for
building and installing igb_uio kernel driver.
2. Enable Virtual functions based on requirements:

$ echo 2 > /sys/bus/pci/devices/<bdf>/max_vfs

Create Guest VM by using QEMU

Follow the steps needed to create a guest environment and assign VF device to VM by
using QEMU.
1. Unbind the device from UIO driver:

$./dpdk-devbind.py -u <bdf>

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
88
3. Design Example Quick Start Guide
683517 | 2024.11.04

2. Install vfio-pci module.

$modprobe vfio-pci

3. Bind the device to vfio-pci.

a. In case the device is binded to ifc_uio, unbind with the following command:

$ echo "<bdf>" > /sys/bus/pci/devices/\

<bdf>/driver/unbind

For example:
echo "0000:01:00.0" > /sys/bus/pci/devices/\
0000\:01\:00.0/driver/unbind

b. Bind the device to vfio-pci.

$ echo <PCI Vendor ID> <PCI Device ID> > /sys/bus/pci/drivers/vfio-pci/

new_id

For example:
echo 1172 0000 > /sys/bus/pci/drivers/vfio-pci/new_id

4. For CentOS VMs: Use QEMU version 3.0.0 rc0 on Intel machines. Creating a VM
with 8 GB RAM is advisable.
./qemu-3.0.0-rc0/x86_64-softmmu/qemu-system-x86_64 -smp 8 -m
10240M -boot c -machine q35,kernel-irqchip=split -cpu host -
enable-kvm -nographic -L /home/dev/QEMU/qemu-3.0.0-rc0/pc-bios
-name offload1 -hda /home/dev/QEMU/vm1.qcow2 -device vfio-
pci,host=01:00.4 -netdev
type=tap,id=net6551,script=no,downscript=no,vhost=on,ifname=ne
t6551 -device virtio-net-pci,netdev=net6551 -device intel-
iommu,intremap=on,caching-mode=on -serial
telnet::5551,server,nowait -object memory-backend-
file,id=mem,size=10240M,mem-path=/dev/hugepages,share=on -numa
node,memdev=mem -mem-prealloc -monitor
telnet::3331,server,nowait&
On another terminal (vm)
telnet localhost 5551
ifconfig eth0 up
user:root
pass:root
Bring up interface.
$ifconfig eth0 up
Assign the IP address to eth0.
$ifconfig eth0 <1.1.1.11>

On Host
$ifconfig net6551 up
$ifconfig net6551 <1.1.1.12>

Copy the code by using scp.

Example: $scp -r <directory> root@<GuestIP>:/
<path_to_the_directory>/
5. For Ubuntu VMs:

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
89
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Get the qemu 7.0.0 version and compile using below steps
wget https://download.qemu.org/qemu-7.0.0.tar.xz
tar xvJf qemu-7.0.0.tar.xz
cd qemu-7.0.0
./configure
make -j$nproc
cd ..

Get Ubuntu VM image

wget https://cloud-images.ubuntu.com/releases/jammy/release/ubuntu-22.04-
server-cloudimg-amd64.img

mv ubuntu-22.04-server-cloudimg-amd64.img ubuntu-22.04.qcow2

sudo qemu-img create -f qcow2 -F qcow2 -o backing_file=./ubuntu-22.04.qcow2

mcdma_VM-1.qcow2 50G

qemu-img info mcdma_VM-1.qcow2

sudo genisoimage -output VM-1-cidata.iso -volid cidata -joliet -rock user-

data meta-data

Command to launch the VM

sudo qemu-7.0.0/build/x86_64-softmmu/qemu-system-x86_64 -smp 8
-machine accel=kvm,type=q35,kernel-irqchip=split -cpu host -
enable-kvm -nographic -boot c -L qemu-7.0.0/build/pc-bios -cpu
host -m 20G -hda mcdma_VM-1.qcow2 -cdrom VM-1-cidata.iso -
device vfio-pci,host=$BDF -serial telnet::5551,server,nowait -
monitor telnet::3331,server,nowait&

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
90
3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: • On AMD machines, the parameter -device intel-

iommu,intremap=on,caching-mode=on is not required.
• If multiple devices are in the same group, check for ACS enablement in
root port and bridge.
• Below are the Host and Guest VM configurations used for verification:

Table 44. Host System Configuration

Host System Configuration Details

Operating System CentOS Linux release 7.8

Ubuntu 22.04 LTS

Linux Kernel CentOS: 3.10.0-1127.10.1.el7.28.x86_64

Ubuntu: 5.15.0-52-generic x86_64

CPU Cores 96

RAM 128 GB (64 GB on single NUMA)

Hypervisor KVM

Qemu Version QEMU version 3.0.0-rc0

Table 45. Guest System Configuration

Guest System Configuration Details

Operating System CentOS Linux release 7.8

Linux Kernel 3.10.0-1127.10.1.el7.28.x86_64

CPU Cores 2

RAM 8 GB

Establish Communication Between Host and VM

1. Login to Guest VM.

$telnet localhost 5551

2. Bring up the interface.

$ifconfig eth0 up

3. Assign the IP address to eth0

$ifconfig eth0 <1.1.1.11>

Execute the following commands from Host:

a. Bring up net6551. “net6551” is the tap interface that you give in the QEMU
command.

$ifconfig net6551 up
$ifconfig net6551 <1.1.1.12>

b. Copy the code by using scp.

For example:
$scp -r <directory> root@<GuestIP>:\
/<path_to_the_directory>/

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
91
 3. Design Example Quick Start Guide
683517 | 2024.11.04

c. Refer to Software Setup for updating the grub file in VM.

d. Enable NO_IOMMU mode inside the Guest VM.
Currently, QEMU does not support IOMMU. When you are running software on
Guest VM, if you are using vfio, you need to enable NO_IOMMU_MODE mode.
Edit common/mk/common.mk to define the following macro, for example:
__cflags += -DNO_IOMMU_MODE

By default, this flag is disabled.

e. Refer to Install PMD and Test Application (for CentOS) on page 86 for build
and install MCDMA.

3.5.2.4.5. Run the Reference Example Application

1. After the reference application is built, run it:

$ ./build/mcdma-test -- -h
2. This command will display the available options in the application as shown in the
image below:

3. Do a PIO test to check if the setup is correct. If successful, the application will
show a Pass status.
[root@BAPVENC011T perfq]# ./build/mcdma-test -- -b
0000:01:00.0 -o

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
92
3. Design Example Quick Start Guide
683517 | 2024.11.04

Here -b should be provided with the correct BDF in the system.

4. The DPDK driver can be used with the AVST/AXIST Packet Generate/Check design
example to test the packet generator and checker design.
The following diagram shows the testing strategy.

DPDK Testapp
MCDMA PMD
lgb_uio/vfio-pci
Host

H2D D2H

Pkt Gen Checker Pkt Generator

FPGA

Configuration in examples/mcdma-test/perfq/perfq_app.h
In the case of static channel mapping, modify the following parameters:
• /* PF count starts from 1 */
#define IFC_QDMA_CUR_PF <pf number>
• /* VF count starts from 1. Zero implies PF was used instead of VF */
#define IFC_QDMA_CUR_VF <vf number>
• /* Number of PFs */
#define IFC_QDMA_PFS <number of PFs>
/* Channels available per PF */
#define IFC_QDMA_PER_PF_CHNLS <number of channels per PF>
• /* Channels available per VF */
#define IFC_QDMA_PER_VF_CHNLS <number of channels per VF>
• /* Number of VFs per PF */
#define IFC_QDMA_PER_PF_VFS <number of VFs per PF>

perfq_app command line parameters:

Command:
$ ./build/mcdma-test -m 8192 --file-prefix=pf0 --\
-b 0000:01:00.0 -p 32768 -d 2 -c 1 -a 2 -l 5 -z -n

Configuration:
a. 1 channel (-c 1)
b. Packet generator bidirectional (-z)
c. Payload length of 32,768 bytes in each descriptor (-p 32768)
d. Transfer the data every 5 seconds (-l 5)
e. Dump the progress log every second (-d 2)
f. Configure the number of channels in ED (-n)
g. Number of threads to be used for DMA purpose. (-a 2)

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
93
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16 configuration.

Figure 37. DPDK Avalon-ST Packet Generate/Check Design Example Gen4 x16 : P-Tile
Hardware Test Results

5. The DPDK driver can also be used with the AVST/AXIST Device-side Packet
Loopback design example to test loopback.
The following diagram shows the testing strategy.

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
94
3. Design Example Quick Start Guide
683517 | 2024.11.04

DPDK Testapp
MCDMA PMD
lgb_uio/vfio-pci
Host

H2D D2H

HW Loopback
FPGA

Command: $ ./build/mcdma-test -m 8192 --file-prefix=pf0 -l

0-15 -- -b 0000:01:00.0 -p 32768 -d 2 -c 1 -a 2 -l 5 -i
Configuration:
a. 1 channel (-c 1)
b. Packet generator bidirectional (-i)
c. Payload length of 64 bytes in each descriptor (-p 32768)
d. Transfer the data every 5 seconds (-l 5)
e. Dump the progress log every second (-d 2)

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
95
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Note: This hardware test was run with the Stratix 10 GX H-tile PCIe Gen3 x16 configuration.

Figure 38. DPDK Avalon-ST Device-side Packet Loopback Design Example Gen4 x16 : P-
Tile Hardware Test Result

Channel ID VF PF Verification

For Channel ID VF PF verification: Channel ID , VF and PF information is inserted in

the AVST/AXIST data pattern. To test dma in a proper channel and VF, it can be tested
as described below:

For AVST/AXIST LB

1. Find the following configuration file

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
96
3. Design Example Quick Start Guide
683517 | 2024.11.04

a. software/dpdk/dpdk/examples/mcdma-test/perfq/meson.build (For
Ubuntu)
b. software/dpdk/dpdk/examples/mcdma-test/perfq/Makefile (For
CentOS).
2. Define the flags: CID_PAT and IFC_PROG_DATA_EN in the configuration files
mentioned in the above step.
3. Define the flag: DCID_PFVF_PAT.
__cflags += - DCID_PFVF_PAT

4. Only data validation mode is supported. Performance mode should be disabled.

__cflags += DPERFQ_LOAD_DATA
__cflags += - UPERFQ_PERF

5. Supported direction: Bi-directional (-i) only.

For AVST PKT GEN

1. Find the following configuration file

a. software/dpdk/dpdk/examples/mcdma-test/perfq/meson.build (For
Ubuntu)
b. software/dpdk/dpdk/examples/mcdma-test/perfq/Makefile (For
CentOS).
2. Define the flags: CID_PAT and IFC_PROG_DATA_EN in the configuration files
mentioned in the above step.
3. __cflags += - DCID_PAT
__cflags += DIFC_PROG_DATA_EN

4. Only data validation mode is supported. Performance mode should be disabled.

__cflags += DPERFQ_LOAD_DATA

5. Supported directions: Rx, Tx and Bi-directional (-z).

Example of Verifying on an AVMM Design

Modify the below macro in the following file: dpdk/dpdk/drivers/net/mcdma/

rte_pmd_mcdma.h
#define PCIe_SLOT 0 /* 0 – x16, 1 – x8 */

Use this command:

Command: $ ./build/mcdma-test -- -b 0000:01:00.0 -p\
32768 -l 5 -i -c 2 -d 2 -a 4

Configuration:
• bdf (-b 0000:01:00.0)
• 1 channel (-c 2)
• Loopback (-i)
• Payload length of 32768 bytes in each descriptor (-p 32768)

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
97
 3. Design Example Quick Start Guide
683517 | 2024.11.04

• Time limit set to 5 (-l 5)

• debug log enabled (-d 2)
• One thread per queue (-a 4)

Note: Reset the IP before starting DMA by using the following command: ./build/mcdma-
test -- -b <bdf> -e

Testing Bitstream Configuration beyond 256 Channels (for DPDK Poll Mode Driver)

Note: In case, if the *.sof file generated with the number of channels > 256, please follow
this procedure.

1. Define following macros in dpdk/drivers/net/mcdma/rte_pmd_mcdma.h

#define IFC_MCDMA_DIDF

2. Configure the mem zone in dpdk-stable/config/rte_config.h

#define RTE_MAX_MEMZONE 20480

In case of AVMM, BDF is provided as an argument and you must define examples/
mcdma-test/perfq/meson.build
-DIFC_MCDMA_SINGLE_FUNC

Figure 39. Example Command Output For AVMM

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
98
3. Design Example Quick Start Guide
683517 | 2024.11.04

Configuration for AVST/AXIST LB and AVST/AXIST pktgen Undefined single function

mode in examples/mcdma-test/perfq/meson.build
-UIFC_MCDMA_SINGLE_FUNC

Note: bdf is not provided to run the test

Figure 40. Example Command Output for AVST/AXIST

Command for pktgen: :

./build/mcdma-test –m 16384 –l 0-8 - -b -p 64 -l 1 -z -d 1 -c 2048 -a 4

Command for loopback:

./build/mcdma-test –m 16384 –l 0-8 - -b -p 64 -l 2 -i -d 1 -c 2048 -a

Note: In the current release, single page is supported in DIDF mode.

Note: In the current release, simultaneous process is not supported in DIDF mode. You can
run one process with 2K channels.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
99
 3. Design Example Quick Start Guide
683517 | 2024.11.04

BAM Test
1. If the BAM support is enabled on hardware, enable the following flags in: dpdk/
dpdk/drivers/net/mcdma/rte_pmd_mcdma.h
#define IFC_PIO_256 ➤ 256b
read/write operations on PIO BAR and undef other size
or
#define IFC_PIO_128 ➤ 128b
read/write operations on PIO BAR and undef other size

2. Enable the below flag for 256b read or write operations in: dpdk/dpdk/
drivers/net/mcdma/rte_pmd_mcdma.h
#define IFC_PIO_256
#undef IFC_PIO_128

Command: ./build/mcdma-test -- -b 0000:01:00.0 -o

3. Enable the below flag for 128b read or write operations in: dpdk/dpdk/
drivers/net/mcdma/rte_pmd_mcdma.h
#undef IFC_PIO_256
#define IFC_PIO_128

Command: ./build/mcdma-test -- -b 0000:01:00.0 -o

Figure 41. PIO 128b Write and Read Test

Note: For BAM_BAS bitstream, undefine IFC_QDMA_INTF_ST

#undef IFC_QDMA_INTF_ST in dpdk/ dpdk/drivers/net/mcdma/rte_pmd_mcdma.h

BAS Test

MCDMA BAS programming sequence consists of the following steps:

Using Traffic Generator (Write in Host Memory)

The MCDMA BAS programming sequence consists of the following steps:

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
100
3. Design Example Quick Start Guide
683517 | 2024.11.04

1. Allocate DMA-able memory in the host system.

2. Program the base address with the write_map_table with the physical address
of the table.
3. Set the write address register with the offset in block where Traffic generator
needs to write the data.
4. Set how many number of bursts BAS should write in host memory in the
WRITE_COUNT register.
5. Set the enable bit to start traffic generation.
Using Traffic Generator (Reads from Host Memory)
1. Allocate DMA-able memory in the host system.
2. Program the base address with the read_map_table with the physical address of
the host memory.
3. Set the read address register with the offset in block where Traffic checker needs
to read and verify the data.
4. Set how many number of bursts BAS should read from host memory in the
READ_COUNT register.
5. Set the enable bit to start traffic checker.
BAS Test

For x4 BAS:
1. Set the PCIe_SLOT “2” in rte_pmd_mcdma.h(dpdk/drivers/net/mcdma/
rte_pmd_mcdma.h)
2. X4 BAS supports burst length 32 by default. In the file perfq_app.h (dpdk/
examples/mcdma-test/perfq/perfq_app.h)
#define IFC_MCDMA_BAS_X4_BURST_LENGTH 32

If the BAS support is enabled on hardware, enable the following flag in: dpdk/dpdk/
drivers/net/mcdma/rte_pmd_mcdma.h
#define PCIe_SLOT 0 /* 0 – x16, 1 – x8, 2 - x4 */

To enable the config IFC_QDMA_INTF_ST in the file "software/dpdk/dpdk/

patches/v20.05-rc1/dpdk/drivers/net/mcdma/base/mcdma_ip_params.h",
use the following command:
#undef IFC_QDMA_INTF_ST

Commands:
1. To verify the write operation:

./build/mcdma-test -- -b 0000:01:00.0 --bar=0 --bas -s 512 -t

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
101
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Figure 42. BAS Write Operation

2. To verify the read operation:

./build/mcdma-test -- -b 0000:01:00.0 --bar=0 --bas -s 512 -r

Figure 43. BAS Read Operation

3. To verify the write and read operation:

./build/mcdma-test -- -b 0000:01:00.0 --bar=0 --bas -s 512 -z

Figure 44. BAS Write and Read Operation

Performance test:
The below log is collected on Gen3x16 H-tile
./build/mcdma-test -- -b 0000:01:00.0 --bar=0 --bas_perf -s 16384 -z

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
102
3. Design Example Quick Start Guide
683517 | 2024.11.04

Figure 45. Performance Test

Note: For DPDK: VF/PF cannot run BAM+BAS+MCDMA simultaneously within one
VM or hypervisor, You need to run one instance of VF / PF in independent
VM.

3.5.2.5. MCDMA Kernel Mode Network Device Driver

Currently MCDMA AVST 1-port Device-side Packet Loopback Design Example is

supported by the MCDMA Kernel Mode Network Device driver. This design example
demonstrates packet transfer from one PF to another PF.

3.5.2.5.1. Build and Install Netdev Driver

1. Build the mqdma kernel driver and load.

$ systemctl stop NetworkManager.service

$ cd software/kernel

$ make -C driver/kmod/mcdma-netdev-driver

$ insmod driver/kmod/mcdma-netdev-driver/ifc_mcdma_netdev.ko

2. Verify whether driver is loaded or not.

$ lspci -d 1172:000 -v | grep ifc_mcdma_netdev

Kernel driver in use: ifc_mcdma_netdev

$ ifconfig | grep ifc_mcdma
ifc_mcdma0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

3.5.2.5.2. Enable VFs if SRIOV is Supported

Enable VF devices, if you are required to use:

$echo <num_of_vfs> > /sys/bus/pci/devices/\
<bdf>/sriov_numvfs

For example: Creating 4 VFs on 0000:01:00.0

echo 4 > /sys/bus/pci/devices/0000\:01\:00.0/sriov_numvfs

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
103
 3. Design Example Quick Start Guide
683517 | 2024.11.04

3.5.2.5.3. Configure the Number of Channels Supported on the Device

Use the following command to set the number of channels:

$ ethtool -L <device> rx <chnls> tx <chnls>

For example, the following command sets 8 channels on ifc_mcdma0:

ethtool -L ifc_mcdma0 rx 8 tx 8

3.5.2.5.4. Configure the MTU Value

Select the MTU value so that, the sum of the MTU value and Ethernet header length is
aligned to 64.

Use the following command to set the MTU value:

$ ifconfig ifc_mcdma0 mtu <mtu value>

For example, the following command sets the MTU value as 1522. In this case, the
sum of the MTU and the Ethernet header is 1536, which is aligned to 64.
$ifconfig ifc_mcdma0 mtu 1522

3.5.2.5.5. Configure the Device Communication

Configuration for PF-PF communication by using debugfs interface

Loopback ED has the provision to switch the packets from one channel to another
channel.

cd /sys/kernel/debug/ifc_mcdma_config/

Configure from PF0 to PF1

echo src_chnl_no > src_chnl
echo dst_chnl_no > dst_chnl
echo 1 > update

Configure from PF1 to PF0

echo src_chnl_no > src_chnl
echo dst_chnl_no > dst_chnl
echo 1 > update

For example:
• Configure the communication from channel number 64 to channel number 0.
echo 64 > src_chnl
echo 0 > dst_chnl
echo 1 > update

• Configure the communication from channel number 0 to channel number 64.

echo 0 > src_chnl
echo 64 > dst_chnl
echo 1 > update

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
104
3. Design Example Quick Start Guide
683517 | 2024.11.04

Display the configuration

cat show

3.5.2.5.6. Configure Transmit Queue Selection Mechanism

The MCDMA Network Device Driver supports multiple queues. The following
mechanisms are provided to configure the transmit (H2D/Tx) queue usages:
• MCDMA Queue Selection
• Transmit Packet Steering (XPS)
• Default

MCDMA Queue Selection

When using the MCDMA Queue Selection algorithm, you can map multiple transmit
queues to a core. Any application running on that particular core uses one of these
mapped queues to transfer the data.

Multiple application instances can use multiple queues mapped to one core or different
cores enabling parallel transmission streams of packets.

Configuration

• Enable the IFC_SELECT_QUEUE_ALGO flag in driver/kmod/mcdma-netdev-

driver/Makefile.
• Map transmit queues to cores:
cd /sys/kernel/debug/ifc_mcdma0/
echo start_chnl_no > start_chnl
echo end_chnl_no > end_chnl
echo core_no > core
echo 1 > map_update
echo 1 > map_show
Example:
Map chnl 0 to chnl 7 to core5
echo 0 > start_chnl
echo 7 > end_chnl
echo 5 > core
echo 1 > map_update

• If a core is not mapped to any queue, use the last queue.

• By default, all transmit queues are mapped to core 0.

Transmit Packet Steering (XPS)

Transmit Packet Steering is a mechanism for selecting which transmit queue to use
when transmitting a packet on a multi-queue device.

When using XPS, you can map multiple cores to a queue.

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
105
 3. Design Example Quick Start Guide
683517 | 2024.11.04

Configuration

• Disable IFC_SELECT_QUEUE_ALGO flag in driver/kmod/mcdma-netdev-

driver/Makefile.
• Map Tx queues to cores:
echo cpu_mask > /sys/class/net/ifc_mcdma0/queues/tx-i
cat /sys/class/net/ifc_mcdma0/queues/tx-i
Example
Map core 0, 1, and 2 to transmit queue 5.
echo 7 > /sys/class/net/ifc_mcdma0/queues/tx-5

Default

In this case, the Linux default queue selection strategy is used.

Configuration

Disable IFC_SELECT_QUEUE_ALGO flag in driver/kmod/mcdma-netdev-driver/

Makefile.

3.5.2.5.7. Test Procedure by Using Name Space Environment

Use the following script to create name spaces and execute the commands.
1. Stop the network manager by using following command:

systemctl stop NetworkManager.service

2. Create the name spaces:

ip netns add <VM name>

For example:
ip netns add vm0
ip netns add vm1
ip netns add vm2
ip netns add vm3

3. Assign the interface to name space:

Command:
ip link set <interface> netns <namespace>
ip link set ifc_mcdma0 netns vm0
ip link set ifc_mcdma1 netns
vm1
ip link set ifc_mcdma2 netns vm2
ip link set ifc_mcdma3 netns vm3

4. Bring up the interface:

Command: ip netns exec vm0 ifconfig <interface> up
Example:
ip netns exec vm0 ifconfig ifc_mcdma0 up
ip netns exec vm1 ifconfig ifc_mcdma1 up
ip netns exec vm2 ifconfig ifc_mcdma2 up
ip netns exec vm3 ifconfig ifc_mcdma3 up

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
106
3. Design Example Quick Start Guide
683517 | 2024.11.04

5. Assigning the IP address:

Command: ip netns exec <namespace> ip addr add <ipaddr> dev
<interfacename>
Example:
ip netns exec vm1 ip addr add 1.1.1.11/24 dev ifc_mcdma1
ip netns exec vm0 ip addr add 1.1.1.10/24 dev ifc_mcdma0
ip netns exec vm2 ip addr add 1.1.1.12/24 dev ifc_mcdma2
ip netns exec vm3 ip addr add 1.1.1.13/24 dev ifc_mcdma3

6. Ping from one interface to another interface:

Command: ip netns exec <namespace> ping <ip address>
Example:
ip netns exec vm0 ping 1.1.1.11

Note: Observing the pkt-loss currently with TCP and UDP traffic overtime after
starting the traffic. This leads to hanging and closing of the iperf or
netperf connections.

3.5.2.5.8. PIO Test

1. Build and Install Netdev Driver. For more information, refer to Build and Install
Netdev Driver on page 103.
2. Build Netdev User Space application:

$cd software/user/cli/netdev_app
$make clean all

3. Initiate PIO Test

$./netdev_app --pio --ifname ifc_mcdma0

Related Information
Build and Install Netdev Driver on page 103

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
107
683517 | 2024.11.04

Send Feedback

4. Multi Channel DMA Intel FPGA IP for PCI Express

Design Example User Guide Archives
For the latest and previous versions of this document, refer to the Multi Channel DMA
Intel FPGA IP for PCI Express Design Example User Guide. If an IP or software version
is not listed, the user guide for the previous IP or software version applies.

© Altera Corporation. Altera, the Altera logo, the ‘a’ logo, and other Altera marks are trademarks of Altera
Corporation. Altera and Intel warrant performance of its FPGA and semiconductor products to current
specifications in accordance with Altera’s or Intel's standard warranty as applicable, but reserves the right to ISO
make changes to any products and services at any time without notice. Altera and Intel assume no 9001:2015
responsibility or liability arising out of the application or use of any information, product, or service described Registered
herein except as expressly agreed to inwriting by Altera or Intel. Altera and Intel customers are advised to
obtain the latest version of device specifications before relying on any published information and before placing
orders for products or services.
*Other names and brands may be claimed as the property of others.
 683517 | 2024.11.04

Send Feedback

5. Revision History for the Multi Channel DMA Intel FPGA

IP for PCI Express Design Example User Guide
Date Quartus Prime IP Version Changes
Version

2024.11.04 24.3 [H-Tile: 24.2.0] Added a Note about PIPE mode simulations using VCS*
[P-Tile: 8.2.0] in Steps to Run the Simulation: VCS/VCS MX.
[F-Tile: 9.2.0]
[R-Tile: 5.2.0]

2024.07.30 24.2 [H-Tile: 24.1.0] • Design Example Directory Structure: Removed the
[P-Tile: 8.1.0] sample, testapp, and simple_app rows from the
[F-Tile: 9.1.0] Directory Structure table.
[R-Tile: 5.1.0] • Replaced the config IFC_QDMA_INTF_AVST with
IFC_QDMA_INTF_ST in various sections.

2024.01.19 23.4 [H-Tile: 23.1.0] • R-Tile MCDMA IP - Design Examples for Endpoint:
[P-Tile: 7.1.0] Note added about Data Mover mode support for R-
Tile MCDMA IP.
[F-Tile: 8.0.0]
• Hardware Test Results: Information added about
[R-Tile: 4.1.0]
setting flags to enable data validation using -v
option.
• Supported Simulators: Note added about Data Mover
mode support for R-Tile MCDMA IP.
• Steps to Run the Simulation: Simulation command
for F-Tile MCDMA IP added.
• BAM Test: Commands updated in all steps.

2023.10.06 23.3 [H-Tile: 23.0.0] • MCDMA IP Modes: R-Tile and F-Tile information
[P-Tile: 7.0.0] updated
[F-Tile: 7.0.0] • P-Tile MCDMA IP - Design Examples for Endpoint:
New note added about simulation support
[R-Tile: 4.0.0]
• F-Tile MCDMA IP - Design Examples for Endpoint:
New note added about simulation support
• R-Tile MCDMA IP - Design Examples for Endpoint:
New notes added about simulation support
• Avalon-MM PIO using MCDMA Bypass Mode: New
notes added at the end of the section
• Single-Port Avalon-ST Packet Generate/Check: Note
added about Metadata support
• Avalon-ST Device-side Packet Loopback: Note added
about Metadata support
• Avalon-MM DMA: Note added about User FLR
interface support
• BAM_BAS Traffic Generator and Checker: Note
added MSI interface support
• Design Example Directory Structure: Note added at
the end of the section
• Procedure: Note added in Step 10 (c)
• Procedure: Available design examples information
updated for all modes in Step 10 (d)
• Supported Simulators: Information about x4 Hard IP
mode design example simulation support updated
continued...

© Altera Corporation. Altera, the Altera logo, the ‘a’ logo, and other Altera marks are trademarks of Altera
Corporation. Altera and Intel warrant performance of its FPGA and semiconductor products to current
specifications in accordance with Altera’s or Intel's standard warranty as applicable, but reserves the right to ISO
make changes to any products and services at any time without notice. Altera and Intel assume no 9001:2015
responsibility or liability arising out of the application or use of any information, product, or service described Registered
herein except as expressly agreed to inwriting by Altera or Intel. Altera and Intel customers are advised to
obtain the latest version of device specifications before relying on any published information and before placing
orders for products or services.
*Other names and brands may be claimed as the property of others.
 5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example
User Guide
683517 | 2024.11.04

Date Quartus Prime IP Version Changes

Version

• Steps to Run the Simulation: Steps (1) and (2)

updated
• Driver Support: Note added about software folders
• Run the Reference Example Application: Step (5)
added
• Custom PIO Read Write Test: New section added
• Channel ID VF PF Verification: New section added

2023.07.07 23.2 [H-Tile: 22.3.0] • MCDMA IP Modes: R-Tile information added

[P-Tile: 6.0.0] • MCDMA R-Tile Design Examples for Endpoint:
[F-Tile: 6.0.0] Information updated
[R-Tile: 3.0.0] • Simulation Results: Simulation log and Simulation
waveforms added
• Testbench Overview: R-Tile information added
• Supported Simulators: Aldec Riviera Pro simulator
information added
• Supported Simulators: R-Tile information added
• Set the Boot Parameters: Commands updated
• Installing the Linux Kernel Driver: Notes added
• Establish Communication Between Host and QEMU:
Macro command updated
• Run the Reference Example Application: Note added
about data validation
• Set the Boot Parameters: Commands updated
• Install PMD and Test Application (for Ubuntu): New
command added in Step (5)
• Create Guest VM by using QEMU: Information added
about creating the centOS VMs and Ubuntu VMs

2023.04.17 23.1 [H-Tile: 22.2.0] • Updated product family name to "Intel Agilex® 7".
[P-Tile: 5.1.0] • MCDMA R-Tile Design Examples for Endpoint: DPDK
[F-Tile: 5.1.0] Driver Support information added to the table
[R-Tile: 2.0.0] • MCDMA R-Tile Design Examples for Endpoint:
External Descriptor Controller information added to
the table
• Hardware and Software Requirements: Operating
system information added
• Testbench Overview: MCDMA R-Tile Testbench
information added
• Supported Simulators: New table added Supported
Simulators for MCDMA IP R-Tile
• Run the Simulation Script: Table Steps to Run the
Simulation removed
• Software Test Setup: Operating System information
updated
• Set the Boot Parameters: CentOS and Ubuntu
information added
• Enabling VFs and Create Guest VM by Using QEMU:
Host System Configuration table updated with
operating system information
• Run the Reference Example Application:
— Note added in Step (5)
— Custom AVMM DMA Gen4 x16 : P-Tile Hardware
Test Result: New screenshot added
• BAS Test: BAS x4 information added
• Testing Bitstream Configuration beyond 256
Channels (for MCDMA Custom Driver): Note added in
Step (4)
continued...

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
110
5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example
User Guide
683517 | 2024.11.04

Date Quartus Prime IP Version Changes

Version

• Prerequisites: OS version and packages to be

installed added
• Set the Boot Parameters: CentOS and Ubuntu
information added
• Install and Build Testpmd: Note added about Ubuntu
support
• Install PMD and Test Application (for Ubuntu): New
section added
• Testing Bitstream Configuration beyond 256
Channels (for DPDK Poll Mode Driver): New section
added
• BAM Test: Note added about BAM_BAS bitstream
• BAS Test: Note added about DPDK

2023.02.14 22.4 [H-Tile: 22.1.0] • Kernel mode char driver is no longer supported.
[P-Tile: 5.0.0] MCDMA Kernel Mode Character Device Driver section
removed. All other Chardev driver information also
[F-Tile: 5.0.0]
removed.
[R-Tile: 1.0.0]
• MCDMA R-Tile information added in the following
sections:
— MCDMA IP Modes
— Design Example Overview
— Hardware and Software Requirements
• MCDMA R-Tile Design Examples for Endpoint: New
section added
• BAM+BAS+MCDMA User Mode support information
added in following sections:
— MCDMA H-Tile Design Examples for Endpoint
— MCDMA P-Tile Design Examples for Endpoint
— MCDMA F-Tile Design Examples for Endpoint
— Driver Support
— Supported Simulators
• Hardware and Software Requirements: Development
Kit information updated
• Procedure: Step (7) updated
• Running the Design Example Application on a
Hardware Setup: Development Kit information
updated
• Program the FPGA: Note added
• Software Test Setup: Information added to run
custom driver with Ubuntu 22.04
• External Packages: Commands added for CentOS
and Ubuntu. Note added in Step (1)
• BAS Verification: Note added about running BAM
+BAS+MCDMA
• Testing Bitstream Configuration beyond 256
Channels: Note added in Step (1)

2022.10.28 22.3 H-Tile IP • MCDMA IP Modes: Table MCDMA IP Modes and FPGA
version: 22.0.0 Development Kit for Design Examples updated for P-
P-Tile IP Tile and F-Tile rows
version: 4.0.0 • Kernel Mode Driver Support removed for Device-side
F-Tile IP Packet Loopback Design Example in
version: 4.0.0 — MCDMA H-Tile Design Examples for Endpoint
— MCDMA P-Tile Design Examples for Endpoint
• MCDMA F-Tile Design Examples for Endpoint: Table
updated MCDMA F-Tile Design Examples for Endpoint
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
111
 5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example
User Guide
683517 | 2024.11.04

Date Quartus Prime IP Version Changes

Version

• Hardware and Software Requirements: Quartus

Prime version and Development Kit support
information updated
• Hardware Test Results: Figure Performance Test: P-
Tile updated
• Hardware Test Results: Command updated in Data
Validation Test
• Hardware Test Results: Command added to enable
external flag IFC_MCDMA_EXTERNL_DESC
• Supported Simulators: QuestaSim and Questa Intel
FPGA Edition are not supported in Quartus Prime
22.3 version for MCDMA P-Tile IP. Table Supported
Simulators for MCDMA IP P-Tile updated to reflect
this change.
• Set the Boot Parameters: Command in Step 6(b)
updated
• Installing the Linux Kernel Driver: Command in Step
2(b) updated
• Enabling VFs and Create Guest VM by Using QEMU:
Command in step 3(b) updated
• Meta Data Test: Command added in Step (1) for
__cflags
• Meta Data Test: Command updated in Step (3)
• BAS Verification: Command added to enable BAS in
common.mk
• Packet Gen Test: Command updated to run the
perfq_app application in Step (3)
• External Descriptor Mode Verification: Command in
Step (1) updated
• Install PMD and Test Application: Description to
enable VFIO added in Step (2)
• Install PMD and Test Application: Description to
install VFIO module and bind vfio-pci driver added in
Step (4)
• Create Guest VM by using QEMU: Commands
updated in Step 3(b)
• Testing Bitstream Configuration beyond 256
Channels: Command for pktgen updated
• BAS Test: Command added to enable the config
IFC_QDMA_INTF_ST

2022.08.24 22.2 H-Tile IP • Device-side Packet Loopback and Packet Generate/

version: 21.5.0 Check User Mode updated to BAM+MCDMA for the
P-Tile IP following topics
version: 3.1.0 — MCDMA H-Tile Design Examples for Endpoint
F-Tile IP — MCDMA F-Tile Design Examples for Endpoint
version: 3.0.0 • Device-side Packet Loopback and Packet Generate/
Check User Mode updated to BAM+BAS for MCDMA
P-Tile Design Examples for Endpoint
• Hardware Test Results: Hardware test result added
for Avalon-MM PIO using MCDMA Bypass Mode
• Single-Port Avalon-ST PIO Using MCDMA Bypass
Mode: Description updated
• Simulation Waveforms: Added for Single-Port
Avalon-ST Packet Generate/Check
• Simulation Log: Added for Single-Port Avalon-ST
Packet Generate/Check
• Hardware Test Result: Added for Single-Port Avalon-
ST Packet Generate/Check
continued...

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
112
5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example
User Guide
683517 | 2024.11.04

Date Quartus Prime IP Version Changes

Version

• Avalon-ST Device-side Packet Loopback: Description

updated
• Simulation Results: Updated for Avalon-ST Device-
side Packet Loopback
• Hardware Test Results: Updated for Avalon-ST
Device-side Packet Loopback
• Hardware Test Results: Intel Agilex F-Series P-Tile
PCIe Gen4 x16 added for Avalon-MM DMA
• Device-side Packet Loopback and Packet Generate/
Check User Mode updated to BAM+MCDMA for all
tables in Supported Simulators
• BAS Test: Note added
• BAS Verification: Commands updated for all
operations
• Example of Verifying on an AVMM Design:
Commands updated
• BAM Test: Commands updated in all steps
• BAS Test: Commands and Result screenshots
updated for all operations
4 Port AVST Mode has been deprecated from this
release. All related 4 Port Mode information has been
removed from the following sections:
• Design Example Overview
• MCDMA H-Tile Design Examples for Endpoint
• MCDMA P-Tile Design Examples for Endpoint
• MCDMA F-Tile Design Examples for Endpoint
• Avalon-ST Device-side Packet Loopback
• Example Testbench Flow for DMA Test with Packet
Generate/Check Design Example
• Driver Support

2022.04.29 22.1 H-Tile IP Sections Updated:

version: 21.4.0 • MCDMA IP Modes [Note added]
P-Tile IP • MCDMA H-Tile Design Examples for Endpoint [New
version: 3.0.0 section added]
F-Tile IP • MCDMA P-Tile Design Examples for Endpoint [New
version: 2.0.0 section added]
• MCDMA F-Tile Design Examples for Endpoint [New
section added]
• Hardware and Software Requirements [Version
updated]
• Single-Port Avalon-ST PIO Using MCDMA Bypass
Mode [Figure Title updated]
• Avalon-MM PIO Using MCDMA Bypass mode [Figure
Title updated]
• Hardware Test Results [Note added]
• BAM_BAS Traffic Generator and Checker [Section
title updated]
• External Descriptor Controller [Description updated
and High Level Block Diagram added]
• Hardware Test Results [Example Design Test Result
image updated]
• Design Example Directory Structure [kmod folder
structure updated]
• Supported Simulators [New Tables added]
• Run the Simulation Script [Table updated]
• Running the Design Example Application on a
Hardware Setup [Development kit support and Note
added]
continued...

Send Feedback Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User
Guide
113
 5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example
User Guide
683517 | 2024.11.04

Date Quartus Prime IP Version Changes

Version

• Driver Support [Driver Support for MCDMA Design

Examples Table updated]
• BAS Programming Sequence [Steps updated]
• Run the Reference Example Application [Test Result
image added]
• Run the Reference Example Application [Test Result
image added]
• Examples [Note added]

2022.02.06 21.4 H-Tile IP • Added new design example: External Descriptor

version: 21.3.0 Controller
P-Tile IP • Added information for the Traffic Generator/Checker
version: 2.2.0 example design in Supported Simulators
F-Tile IP
version: 1.1.0

2021.12.01 21.3 H-Tile IP Rev H-Tile 21.2.0—2K channel support for D2H
version: 21.2.0 Rev P-Tile 2.1.0—CS address width reduced from 29 to
P-Tile IP 14 bits
version: 2.1.0 Rev F-Tile 1.0.0:
F-Tile IP • F-Tile support added
version: 1.0.0
• BAS EP design example added
Added new design example: Traffic Generator/Tracker

2021.09.15 21.2 H-Tile IP • Added SRIOV support for DPDK PMD

version: 21.1.0 • Added support for kernel mode driver
P-Tile IP • Added the Multi Channel DMA for FPGA IP Design
version: 2.0.0 Example User Guide Archives section

2021.05.24 21.1 H-Tile IP • Added the single-port Avalon-ST design example

version: 2.0.0 • Added support for new BAM, BAS, BAM+BAS, and
P-Tile IP BAM+MCDMA user modes
version: 1.0.0 • Added support for the DPDK PMD driver
• Added support for the Xcelium simulator

2020.08.05 20.2 H-Tile IP Initial Release

version: 20.0.0

Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Send Feedback
Guide
114