Skip to content

Ten64 Network Performance Benchmark

This is a brief sampling of the Ten64's performance under ideal conditions.

We are being very modest with our performance numbers in advertising as there are some caveats today:

  • The Linux network stack does limit the performance a bit, we see the ultimate solution as XDP in the short term and hardware offload (AIOP) in the long term.

    For example, 3 Gbit/s single flow performance is easily achieved, but futher traffic flows may be inconsistent as it depends how the flows are passed to each core (via DPIO portals) based on flow hashing (such as the port numbers on each connection).

  • In the default configuration we ship the units with (all Ethernet ports enabled), performance is limited as the LS1088 doesn't have enough resources (buffers, flow tables etc.) to give optimal resources to all ports - see the network configuration page for more information.

For these tests, we have booted the system with the "eth0 only" DPL and added the two 10G ports dynamically via the ls-addni command - this ensures they have the most optimal settings in terms of queues, flow tables and inbound packet hashing.

iperf3 testing

These tests are done between a Linux client and Windows 10 workstation.

The Linux client has an Intel X520-T2 (Dual SFP+) card, while the Windows machine has an X540-T2 (Dual 10GBase-T). A Mikrotik S+RJ10 10GBase-T SFP module is used to provide a 10GBase-T connection on the Ten64.

Bridge mode

Command: iperf3 -P (num threads) -R -c (server) -t 60

Number of threads (-P) Thread 0 Thread 1 Thread 2 Thread 3 Thread 4 Total Gbit/s
1 3.08 3.08
2 3 2.93 5.93
3 3.04 2.88 2.56 8.48
4 1.87 2.69 1.75 2.73 9.04
5 1.91 2.03 1.84 1.68 1.84 9.3

As discussed above, the per-flow performance does vary more as flows are increased - this can be due to different packet flows being processed on the same core, while the fastest flow gets a core by itself.

Routed mode

Note: this test is done with static routing rules (no NAT)

Number of threads (-P) Thread 0 Thread 1 Thread 2 Thread 3 Thread 4 Total Gbit/s
1 2.91 2.91
2 2.81 2.6 5.41
3 2.64 2.52 2.15 7.3
4 2.33 2.36 1.97 2.39 8.3
5 2.17 2.07 2.07 1.46 1.37 9.3

DPDK

DPDK is one way to extract higher performance from the LS1088, by bypassing the kernel and doing all packet operations in userspace, using a poll-mode driver (PMD) mechanism rather than the traditional interrupt-driven mechanism in the kernel.

It should be noted these tests are only being done with a single core only.

To do this test, we just launch DPDK's testpmd which establishes a Layer2 bridge across a pair of ports.

Number of threads (-P) Thread 0 Thread 1 Thread 2 Thread 3 Thread 4 Total Improvement (relative to Linux bridge)
1 5.02 5.02 1.63
2 3.99 3.93 7.92 1.34
3 3.09 2.96 3.16 9.21 1.09
4 2.56 2.26 2.44 2.17 9.43 1.04
5 2.62 1.73 1.69 1.76 1.69 9.49 1.02

As we can see, even with just one core doing the work, DPDK can improve the single thread performance to 5Gbit/s, a 1.6x improvement, and DPDK is still able to hold an advantage as more flows are added.

The disadvantage of DPDK is that you need to port your entire network stack to DPDK, which runs in userspace. Some options exist, such as DANOS, TNSR etc, as well as OpenVSwitch-DPDK - but deployment and validation of these solutions requires a lot more time and effort over traditional kernel network stacks.

Practical Performance Turning

Disable IOMMU Passthrough

Linux can operate the IOMMU in two modes:

  • DMA Translation

    In this mode, the IOMMU hardware will 'translate' (remap) direct memory access (DMA) requests between the hardware and device drivers. It is more secure, but incurs a performance overhead

  • Pass-through mode

    In this mode, IOMMU hardware will pass DMA addresses directly through to device drivers without translation, which could pose issues in a virtual machine environment, as a guest VM could break out.

    Without the overhead of doing DMA translation, performance is improved.

We note that recent enterprise Linux distributions (RHEL 8, SLES 15) operate in passthrough mode by default1.

NXP also advises to use passthrough mode.

If your distribution does not use passthrough mode by default, you can specify iommu.passthrough=1. To disable passthrough mode you can use iommu.passthrough=0.

Only provisioning needed Ethernet ports in the DPL

As mentioned in the introduction, the default firmware configuration enables all 10 Ethernet ports (8 GBe + 2 SFP+).

This configuration stretches the resources of the LS1088's DPAA2 slightly, which means it can only push packet flows through one 'portal'.

By only provisioning the Ethernet ports you require, the DPAA2 can spread packet flows through eight portals - one per CPU core. (The LS1088 can support up to 9 ports without resource stretching).

If you have a WAN connection faster than >1Gbit/s we recommend doing this.

You can see the distribution of the packet flows in /proc/interrupts:

Default 10 Ports configuration

cat /proc/interrupts  | grep -E '(CPU\d|dpio)'
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
124:     119703          0          0          0          0          0          0          0  ITS-fMSI 230000 Edge      dpio.8
125:          0          0          0          0          0          0          0          0  ITS-fMSI 230001 Edge      dpio.7
126:          0          0          0          0          0          0          0          0  ITS-fMSI 230002 Edge      dpio.6
127:          0          0          0          0          0          0          0          0  ITS-fMSI 230003 Edge      dpio.5
128:          0          0          0          0          0          0          0          0  ITS-fMSI 230004 Edge      dpio.4
129:          0          0          0          0          0          0          0          0  ITS-fMSI 230005 Edge      dpio.3
130:          0          0          0          0          0          0          0          0  ITS-fMSI 230006 Edge      dpio.2
131:          0          0          0          0          0          0          0          0  ITS-fMSI 230007 Edge      dpio.1

You can see all network interrupts are processed through one portal only - dpio.8.

Reduced port configuration

cat /proc/interrupts  | grep -E '(CPU\d|dpio)'
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
124:     463934          0          0          0          0          0          0          0  ITS-fMSI 230000 Edge      dpio.8
125:          0     961301          0          0          0          0          0          0  ITS-fMSI 230001 Edge      dpio.7
126:          0          0     348047          0          0          0          0          0  ITS-fMSI 230002 Edge      dpio.6
127:          0          0          0     426312          0          0          0          0  ITS-fMSI 230003 Edge      dpio.5
128:          0          0          0          0     464717          0          0          0  ITS-fMSI 230004 Edge      dpio.4
129:          0          0          0          0          0    1344950          0          0  ITS-fMSI 230005 Edge      dpio.3
130:          0          0          0          0          0          0     337744          0  ITS-fMSI 230006 Edge      dpio.2
131:          0          0          0          0          0          0          0     377875  ITS-fMSI 230007 Edge      dpio.1

  1. See "Default IOMMU modes on Linux OSes" in An Introduction to IOMMU Infrastructure in the Linux Kernel