SFP+
The Ten64 has two SFP+ slots:
- Lower slot (XG0) is connected to DPMAC2 (mapped to eth8 by default)
- Upper slot (XG1) is connected to DPMAC1 (mapped to eth9 by default)
Both slots are in 10G mode (XFI<->SFP+) by default. To use a 1G SFP in one or both of these slots currently requires changing the BL2/RCW in flash (see below for details).
The SFP's are connected through a retimer (TI DS110DF410) which implements the clock recovery needed for SFP+. This is not a PHY device - it merely conditions the bitstream.
SFP+ managed by Linux¶
This is called "managed" mode for SFPs, see PHY management modes for an overview.
When the SFP is managed by Linux, you will be able to:
- Use
ethtoolto view the SFP model information and current status (including laser diagnostics where applicable) - Have the SFP transmit control linked to the Linux network interface status
- View thermal sensor readings via the
hwmonsubsystem (if sensors present on the SFP)
Viewing SFP data in Linux¶
You can use ethtool -m eth8 or ethtool -m eth9 to view the SFP status.
For example:
ethtool -m eth8
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 300m
Length (62.5um) : 150m
Length (Copper) : 0m
Length (OM3) : 0m
Laser wavelength : 850nm
Vendor name : FS
Vendor OUI : 00:90:65
Vendor PN : SFP-10GSR-85
Vendor rev : A
Option values : 0x00 0x1a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : F1811040433
Date code : 181116
Optical diagnostics support : Yes
Laser bias current : 0.000 mA
Laser output power : 0.0001 mW / -40.00 dBm
Receiver signal average optical power : 0.4645 mW / -3.33 dBm
Module temperature : 23.00 degrees C / 73.40 degrees F
Module voltage : 3.2520 V
You will see messages when the SFP is inserted and removed:
[ 2573.192064] sfp dpmac2_sfp: module FS SFP-10GSR-85 rev A sn G1909002828 dc 190925
[ 2616.676006] sfp dpmac2_sfp: module removed
and on link up:
[ 2715.146737] fsl_dpaa2_eth dpni.1 eth8: configuring for inband/10gbase-r link mode
[ 2715.215635] fsl_dpaa2_eth dpni.1 eth8: Link is Up - 10Gbps/Full - flow control off
Kernel patches required for full SFP support¶
Full operation of the SFP has only been possible since kernel 5.16. If you are on an earlier kernel you will need:
You will need at a minimum (for kernels before 5.16):
- net: dpaa2-mac: add support for more 10G modes (note: the mainline solution is net: ethernet: use phylink_set_10g_modes() which depends on additional changes)
For using 1G SFPs, you will also need (for kernels earlier than 5.13):
Additionally, the following provides a way to view SFP state via debugfs (part of 5.13 and later):
Copper SFPs also require drivers for their inbuilt PHYs, the Marvell 88E1111 is what is inside all the 1GBase-T SFPs we've seen so far:
CONFIG_MARVELL_PHY=m
CONFIG_MARVELL_10G_PHY=m
Using 1G SFPs¶
There are three possible SFP configurations on the Ten64, which currently require a bootloader reflash to take effect:
- Both SFPs in 10G mode
- One SFP (top/XG1) in 1G mode, Bottom SFP (XG0) in 10G mode
- Both SFPs in 1G mode
| Slot | Default bl2_qspi.pbl |
One 1G bl2_qspi_xg1_1g.pbl |
Both 1G bl2_qspi_both_sfp_1g.pbl |
|---|---|---|---|
| XG1 | 10G | 1G | 1G |
| XG0 | 10G | 10G | 1G |
You can obtain the .pbl files from the firmware package and flash them in recovery like so:
mtd erase bl2 && mtd write bl2_qspi_xg1_1g.pbl bl2
U-Boot will detect when an SFP slot has been configured to 1G mode and print out a message like:
fsl-mc: ten64: set DPMAC1/XG1 to 1G mode
Manual/Advanced SFP control (for older kernels or non-Linux managed only)¶
The section below applies to older kernels only (which can't work with the managed mode), or instances where the SFP cannot be managed by Linux, such as using DPDK based applications.
GPIOs needed for fiber SFP+¶
On the Ten64, SFP's are directly driven by the SoC, there is no 'PHY' device that handles their control signals. Hence when a fiber SFP is used, the control signals do need to be driven by the host.
SFP+ control GPIOS (partial extract from GPIO)¶
| GPIO Controller | Pin | Linux GPIO number (with default device tree) | Default Direction | Purpose | Meaning of ‘0’ | Meaning of ‘1 | Notes |
|---|---|---|---|---|---|---|---|
| TCA9539 (U39) | 0 | 368 | Input | Lower SFP+ TXFAULT | No transmitter fault | Transmitter Fault | Has pull up |
| TCA9539 (U39) | 1 | 369 | Output | Lower SFP+ TXDISABLE | SFP+ transmitter disabled | SFP+ transmitter enabled | Has pull up – if the pin is not driven as output it goes to 1, this pin may not be effective with direct connect (DAC) cables |
| TCA9539 (U39) | 2 | 370 | Input | Lower SFP+ PRESENT | Lower SFP present | Lower SFP not present | |
| TCA9539 (U39) | 3 | 371 | Input | Lower SFP+ LOS | Lower SFP lost signal | Lower SFP has signal | |
| TCA9539 (U39) | 4 | 372 | Input | Upper SFP+ TXFAULT | No transmitter fault | Transmitter Fault | Has pull up |
| TCA9539 (U39) | 5 | 373 | Output | Upper SFP+ TXDISABLE | SFP+ transmitter disabled | SFP+ transmitter enabled | Has pull up – if the pin is not driven as output it goes to 1, this pin may not be effective with direct connect (DAC) cables |
| TCA9539 (U39) | 6 | 374 | Input | Upper SFP+ PRESENT | Upper SFP present | Upper SFP not present | |
| TCA9539 (U39) | 7 | 375 | Input | Upper SFP+ LOS | Upper SFP lost signal | Upper SFP has signal | |
| LS1088 GPIO4 | 11 | 395 | Output | SFP+ Left Down (usually bound to lower port) | LED off | LED on | |
| LS1088 GPIO4 | 12 | 396 | Output | SFP+ Left Up (usually bound to lower port) | LED off | LED on | |
| LS1088 GPIO2 | 29 | 477 | Output | SFP+ Right Down (usually bound to upper port) | LED off | LED on | |
| LS1088 GPIO2 | 30 | 478 | Output | SFP+ Right Up (usually bound to upper port) | LED off | LED on |
Link and activity LEDs are set up with the ledtrig subsystem.
Example SFP setup - OpenWrt¶
OpenWrt already has the SFP+ GPIOs defined by default, these are set in /etc/config/system:
$ cat /etc/config/system
config gpio_switch 'lower_sfp_txidsable'
option name 'Lower SFP+ TX Disable'
option gpio_pin '369'
option value '0'
config gpio_switch 'upper_sfp_txdisable'
option name 'Upper SFP+ TX Disable'
option gpio_pin '373'
option value '0'
config led 'led_sfp1down'
option name 'SFP 1 RX'
option sysfs 'ten64:red:sfp1:down'
option trigger 'netdev'
option mode 'link'
option dev 'eth7'
...
Example SFP setup - standard Linux¶
To enable the lower SFP:
# Export lower SFP+ TXDISABLE to userspace:
echo 369 > /sys/class/gpio/export
# Make it an output
echo out > /sys/class/gpio/gpio369/direction
# Set value to 0 (enable TX)
echo 0 > /sys/class/gpio/gpio369/value
SFP I2C¶
The I2C bus for SFP's are connected through a PCA9540BD I2C multiplexer, the bottom SFP is channel 2 and top SFP is channel 1. Under Linux, this will
be on /dev/i2c-4 and /dev/i2c-3 respectively, with the parent bus (LS1088 I2C4) on /dev/i2c-2.
Our device tree has the required setup for this.
For example, when an SFP is installed in the bottom cage:
# i2cdetect 4
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-4.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --
Providing there are no kernel consumers for the SFP I2C (e.g PHYLINK), you can dump the SFP EEPROM:
# i2cdump 4 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-4, address 0x50, mode byte
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 22 00 00 00 00 00 08 00 00 06 67 00 00 00 ??".....?..?g...
10: 00 00 01 00 4d 69 6b 72 6f 54 69 6b 20 20 20 20 ..?.MikroTik
20: 20 20 20 20 00 00 40 20 53 2b 52 4a 31 30 20 20 ..@ S+RJ10
30: 20 20 20 20 20 20 20 20 32 2e 30 37 05 00 00 b0 2.07?..?
40: 02 1a 00 00 39 43 35 43 30 33 42 34 45 46 44 30 ??..9C5C03B4EFD0
50: 20 20 20 20 31 38 30 39 32 36 20 20 68 00 06 50 180926 h.?P
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
The same SFP when installed in the top SFP cage:
# i2cdetect 3
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-3.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --
Note that some SFPs will disable their EEPROM when SFP TXDISABLE is active - you should use the SFP PRESENT to determine the status of the SFP cage rather than scanning the I2C bus.