Skip to content

SFP+

The Ten64 has two SFP+ slots:

  • Lower slot (XG0) is connected to DPMAC2 (mapped to eth8 by default)
  • Upper slot (XG1) is connected to DPMAC1 (mapped to eth9 by default)

Both slots are in 10G mode (XFI<->SFP+) by default. To use a 1G SFP in one or both of these slots currently requires changing the BL2/RCW in flash (see below for details).

The SFP's are connected through a retimer (TI DS110DF410) which implements the clock recovery needed for SFP+. This is not a PHY device - it merely conditions the bitstream.

SFP+ managed by Linux

This is called "managed" mode for SFPs, see PHY management modes for an overview.

When the SFP is managed by Linux, you will be able to:

  • Use ethtool to view the SFP model information and current status (including laser diagnostics where applicable)
  • Have the SFP transmit control linked to the Linux network interface status
  • View thermal sensor readings via the hwmon subsystem (if sensors present on the SFP)

Viewing SFP data in Linux

You can use ethtool -m eth8 or ethtool -m eth9 to view the SFP status. For example:

ethtool -m eth8
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 10G Ethernet: 10G Base-SR
        Encoding                                  : 0x06 (64B/66B)
        BR, Nominal                               : 10300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 300m
        Length (62.5um)                           : 150m
        Length (Copper)                           : 0m
        Length (OM3)                              : 0m
        Laser wavelength                          : 850nm
        Vendor name                               : FS
        Vendor OUI                                : 00:90:65
        Vendor PN                                 : SFP-10GSR-85
        Vendor rev                                : A
        Option values                             : 0x00 0x1a
        Option                                    : RX_LOS implemented
        Option                                    : TX_FAULT implemented
        Option                                    : TX_DISABLE implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : F1811040433
        Date code                                 : 181116
        Optical diagnostics support               : Yes
        Laser bias current                        : 0.000 mA
        Laser output power                        : 0.0001 mW / -40.00 dBm
        Receiver signal average optical power     : 0.4645 mW / -3.33 dBm
        Module temperature                        : 23.00 degrees C / 73.40 degrees F
        Module voltage                            : 3.2520 V

You will see messages when the SFP is inserted and removed:

[ 2573.192064] sfp dpmac2_sfp: module FS               SFP-10GSR-85     rev A    sn G1909002828      dc 190925
[ 2616.676006] sfp dpmac2_sfp: module removed

and on link up:

[ 2715.146737] fsl_dpaa2_eth dpni.1 eth8: configuring for inband/10gbase-r link mode
[ 2715.215635] fsl_dpaa2_eth dpni.1 eth8: Link is Up - 10Gbps/Full - flow control off

Kernel patches required for full SFP support

As of 2021-05, not all features required are in the mainline kernel. You will need at a minimum:

For using 1G SFPs, you will also need (for kernels earlier than 5.13):

Additionally, the following provides a way to view SFP state via debugfs (part of 5.13 and later):

Copper SFPs also require drivers for their inbuilt PHYs, the Marvell 88E1111 is what is inside all the 1GBase-T SFPs we've seen so far:

CONFIG_MARVELL_PHY=m
CONFIG_MARVELL_10G_PHY=m

Using 1G SFPs

There are three possible SFP configurations on the Ten64, which currently require a bootloader reflash to take effect:

  • Both SFPs in 10G mode
  • One SFP (top/XG1) in 1G mode, Bottom SFP (XG0) in 10G mode
  • Both SFPs in 1G mode
Slot Default
bl2_qspi.pbl
One 1G
bl2_qspi_xg1_1g.pbl
Both 1G
bl2_qspi_both_sfp_1g.pbl
XG1 10G 1G 1G
XG0 10G 10G 1G

You can obtain the .pbl files from the firmware package and flash them in recovery like so:

mtd erase bl2 && mtd write bl2_qspi_xg1_1g.pbl bl2

U-Boot will detect when an SFP slot has been configured to 1G mode and print out a message like:

fsl-mc: ten64: set DPMAC1/XG1 to 1G mode

Manual/Advanced SFP control (for older kernels or non-Linux managed only)

The section below applies to older kernels only (which can't work with the managed mode), or instances where the SFP cannot be managed by Linux, such as using DPDK based applications.

GPIOs needed for fiber SFP+

On the Ten64, SFP's are directly driven by the SoC, there is no 'PHY' device that handles their control signals. Hence when a fiber SFP is used, the control signals do need to be driven by the host.

SFP+ control GPIOS (partial extract from GPIO)

GPIO Controller Pin Linux GPIO number (with default device tree) Default Direction Purpose Meaning of ‘0’ Meaning of ‘1 Notes
TCA9539 (U39) 0 368 Input Lower SFP+ TXFAULT No transmitter fault Transmitter Fault Has pull up
TCA9539 (U39) 1 369 Output Lower SFP+ TXDISABLE SFP+ transmitter disabled SFP+ transmitter enabled Has pull up – if the pin is not driven as output it goes to 1, this pin may not be effective with direct connect (DAC) cables
TCA9539 (U39) 2 370 Input Lower SFP+ PRESENT Lower SFP present Lower SFP not present
TCA9539 (U39) 3 371 Input Lower SFP+ LOS Lower SFP lost signal Lower SFP has signal
TCA9539 (U39) 4 372 Input Upper SFP+ TXFAULT No transmitter fault Transmitter Fault Has pull up
TCA9539 (U39) 5 373 Output Upper SFP+ TXDISABLE SFP+ transmitter disabled SFP+ transmitter enabled Has pull up – if the pin is not driven as output it goes to 1, this pin may not be effective with direct connect (DAC) cables
TCA9539 (U39) 6 374 Input Upper SFP+ PRESENT Upper SFP present Upper SFP not present
TCA9539 (U39) 7 375 Input Upper SFP+ LOS Upper SFP lost signal Upper SFP has signal
LS1088 GPIO4 11 395 Output SFP+ Left Down (usually bound to lower port) LED off LED on
LS1088 GPIO4 12 396 Output SFP+ Left Up (usually bound to lower port) LED off LED on
LS1088 GPIO2 29 477 Output SFP+ Right Down (usually bound to upper port) LED off LED on
LS1088 GPIO2 30 478 Output SFP+ Right Up (usually bound to upper port) LED off LED on

Link and activity LEDs are set up with the ledtrig subsystem.

Example SFP setup - OpenWrt

OpenWrt already has the SFP+ GPIOs defined by default, these are set in /etc/config/system:

$ cat /etc/config/system
config gpio_switch 'lower_sfp_txidsable'
        option name 'Lower SFP+ TX Disable'
        option gpio_pin '369'
        option value '0'

config gpio_switch 'upper_sfp_txdisable'
        option name 'Upper SFP+ TX Disable'
        option gpio_pin '373'
        option value '0'

config led 'led_sfp1down'
        option name 'SFP 1 RX'
        option sysfs 'ten64:red:sfp1:down'
        option trigger 'netdev'
        option mode 'link'
        option dev 'eth7'
...

Example SFP setup - standard Linux

To enable the lower SFP:

# Export lower SFP+ TXDISABLE to userspace:
echo 369 > /sys/class/gpio/export
# Make it an output
echo out > /sys/class/gpio/gpio369/direction
# Set value to 0 (enable TX)
echo 0 > /sys/class/gpio/gpio369/value

SFP I2C

The I2C bus for SFP's are connected through a PCA9540BD I2C multiplexer, the bottom SFP is channel 2 and top SFP is channel 1. Under Linux, this will be on /dev/i2c-4 and /dev/i2c-3 respectively, with the parent bus (LS1088 I2C4) on /dev/i2c-2.

Our device tree has the required setup for this.

For example, when an SFP is installed in the bottom cage:

# i2cdetect 4
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-4.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --

Providing there are no kernel consumers for the SFP I2C (e.g PHYLINK), you can dump the SFP EEPROM:

# i2cdump 4 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-4, address 0x50, mode byte
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 22 00 00 00 00 00 08 00 00 06 67 00 00 00    ??".....?..?g...
10: 00 00 01 00 4d 69 6b 72 6f 54 69 6b 20 20 20 20    ..?.MikroTik
20: 20 20 20 20 00 00 40 20 53 2b 52 4a 31 30 20 20        ..@ S+RJ10
30: 20 20 20 20 20 20 20 20 32 2e 30 37 05 00 00 b0            2.07?..?
40: 02 1a 00 00 39 43 35 43 30 33 42 34 45 46 44 30    ??..9C5C03B4EFD0
50: 20 20 20 20 31 38 30 39 32 36 20 20 68 00 06 50        180926  h.?P
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

The same SFP when installed in the top SFP cage:

# i2cdetect 3
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-3.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --

Note that some SFPs will disable their EEPROM when SFP TXDISABLE is active - you should use the SFP PRESENT to determine the status of the SFP cage rather than scanning the I2C bus.