DPDK on Ten64 Setup Guide (DRAFT)¶

This guide has been tested on Debian 12 (bookworm) with DPDK 23.03.

Preparation (update/change board configuration)¶

Change the DPL (Data Plane Layout)¶

The DPAA2 DPL needs to be changed to only assign GE0 (or another other single interface) as the 'management' interface for Linux. The other ports will be assigned to DPDK

(It is possible to 'unplug' and 'replug' Ethernet interfaces at runtime from Linux to DPDK, but it's very difficult to do correctly)

Boot the system into recovery firmware and download a copy of eth-dpl-eth0-only.dtb from the latest firmware release at archive.traverse.com.au.

(Click the "Browse" link next the firmware release and then "components" to download individual binaries)

Write the DPL like so:

mtd erase dpl && mtd write eth-dpl-eth0-only.dtb dpl
Unlocking dpl ...
Erasing dpl ...
Unlocking dpl ...

Writing from eth-dpl-eth0-only.dtb to dpl ...

NOTE: From this point, only eth0/GE0 will appear in Linux.

You can use the ls-addni tool to manually add interfaces that are not part of another DPRC container.

Change to legacy Ethernet mode (if not using NXP kernel)¶

The mainline Linux kernel with phylink (managed Ethernet interfaces) does not support working with "detached" DPMAC devices yet. If you are not using the NXP kernel, you will need to change the Ethernet devices to legacy mode.

(See here for required patch)

In recovery you can do:

fw_setenv gbemode legacy
fw_setenv sfpmode legacy

Or U-Boot:

setenv gbemode legacy
setenv sfpmode legacy
saveenv
reset

Update MC Firmware (optional, but recommended)¶

The current production Ten64 firmware (v0.8.x) ships with MC Firmware 10.20.

Recommend updating to the latest available from NXP at qoriq-mc-binary.

In recovery firmware:

mtd erase mcfirwmare && mtd write mc_ls1088a_10.37.0.itb mcfirmware

Installing DPDK¶

DPDK system preparation and install¶

For Debian and Ubuntu, install required dependencies for building kernel, restool and DPDK:

sudo apt-get install flex bison bc libssl-dev build-essential \
    meson python3-full python-is-python3 libnuma-dev libbsd-dev \
    pkg-config python3-pyelftools pandoc

Update / Install / Configure Kernel¶

Install NXP Kernel¶

At the time of writing (2023-07), the mainline kernel is missing some VFIO changes necessary for DPDK to function

Clone the latest available NXP kernel and compile it:

git clone --branch lf-6.1.y https://github.com/nxp-qoriq/linux linux-qoriq
cd linux-qoriq
make defconfig
# APT distribution users (Debian/Ubuntu) may also consider
# make deb-pkg -j$(nproc) to output a kernel .deb
make EXTRAVERSION="-nxp-lf-6-1" -j$(nproc)
make install

Update kernel command line¶

Add default_hugepagesz=1024m hugepagesz=1024m hugepages=8 isolcpus=1-7 to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub

$ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="earlycon default_hugepagesz=1024m hugepagesz=1024m hugepages=8 isolcpus=1-7 iommu.passthrough=1 arm-smmu.disable_bypass=0 net.ifnames=0"
GRUB_CMDLINE_LINUX=""

$ sudo update-grub

Ensure vfio-fsl-mc driver is loaded at boot¶

echo "vfio-fsl-mc" >> /etc/modules-load.d/vfio-fsl-mc.conf

Compile and install restool¶

$ git clone --branch lf-6.1.22-2.0.0 https://github.com/nxp-qoriq/restool

$ make -C restool make
$ sudo make -C restool install

$ sudo restool -m
MC firmware version: 10.37.0

Compile and install DPDK¶

$ git clone --branch v23.03 https://github.com/DPDK/dpdk
$ cd dpdk
$ meson arm64-build
$ ninja -C arm64-build install

Once all of the kernel, restool and DPDK are installed, reboot so the new kernel configuration is made effective.

Obtain dynamic_dpl script / dpdk-extras¶

git clone https://github.com/nxp-qoriq/dpdk-extras.git

Running dpdk-testpmd¶

First, use dynamic_dpl.sh to put a pair of MACs into a child container for DPDK. (Similar to creating a PCI virtual function)

cd dpdk-extras/dpaa2
sudo bash ./dynamic_dpl.sh dpmac.3 dpmac4 # Pass GE4 and GE5
Using board type as 1088
mkdir: cannot create directory ‘/mnt/hugepages’: File exists
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.

##################### Container  dprc.2  is created ####################

Container dprc.2 have following resources :=>

 * 16 DPBP
 * 8 DPCON
 * 8 DPSECI
 * 2 DPNI
 * 10 DPIO
 * 2 DPCI


######################### Configured Interfaces #########################

Interface Name        Endpoint              Mac Address
==============        ========              ==================
dpni.1                dpmac.3               00:00:00:00:0:3
dpni.2                dpmac.4               00:00:00:00:0:4

(Ignore incorrect MAC addresses, DPDK will fix them)

Then, start dpdk-testpmd

$ sudo -i
# export DPRC=dprc.2
# dpdk-testpmd -c 0xe
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
fslmc: Skipping invalid device (power)
fslmc: Skipping invalid device (vfio-dev)
fslmc: Skipping invalid device (msi_irqs)
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
PMD: dpni.1: netdev created, connected to dpmac.3
PMD: dpni.2: netdev created, connected to dpmac.4
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: dpaa2
Configuring Port 0 (socket 0)
dpaa2_net: Error: dpni_set_irq_enable():-19 (No such device)
Port 0: 00:0A:FA:24:30:91
Configuring Port 1 (socket 0)
dpaa2_net: Error: dpni_set_irq_enable():-19 (No such device)
Port 1: 00:0A:FA:24:30:92
Checking link statuses...
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 2 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=2
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=512 - RX free threshold=1
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=512 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
  port 1: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=512 - RX free threshold=1
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=512 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
Press enter to exit

Hit the enter key to stop when you are done.

Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 14586174       RX-dropped: 0             RX-total: 14586174
  TX-packets: 748610         TX-dropped: 0             TX-total: 748610
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 748610         RX-dropped: 0             RX-total: 748610
  TX-packets: 14586174       TX-dropped: 0             TX-total: 14586174
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 748610         RX-dropped: 0             RX-total: 748610
  TX-packets: 14586174       TX-dropped: 0             TX-total: 14586174
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 15334784       RX-dropped: 0             RX-total: 15334784
  TX-packets: 15334784       TX-dropped: 0             TX-total: 15334784
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye...

Other references¶

DPDK Manual: DPAA2 Poll Mode Driver
meta-freescale (Yocto) DPDK recipe. This is a useful reference if you are trying to (cross-)compile DPDK. At the time of writing, meta-freescale is using DPDK 20.11.

See NXP's Layerscape Linux Distribution PoC guide for Layerscape specific notes:

Kernel patches required for DPDK¶

At a minimum, on Linux 6.1: