sfpshow - fault-finding on Brocade Fibre Channel Switches
So you've hit a situation where a Fibre Channel link light has gone out and you don't know what is wrong. Alternately, perhaps you have extremely high latency on just one ESX hosts with regular storage disconnections (even though the link light looks fine)
At a high level, it is likely to be one of the following, but you don't want to go through and replace/re-seat/test each of these components:
- FC HBA (PCI Host Bus Adapter in the server or storage)
- SFP (Small Form-Factor Pluggable optical/copper transciever)
- Fibre Channel patch cable
- Structured Cabling
If you're using Brocade Fibre channel switches, there is some simple troubleshooting you can perform using statistics collected in the switch
sfpshow
This command will print stats pertaining to a particular port on the switch.
Ideally if you can capture this output regularly (e.g. monthly via
"supportshow"), it will give you an excellent baseline to compare
against when a fault occurs.
Below is an example of a relatively healthy port
switch01> sfpshow 3
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 200,400,800_MB/s M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: BROCADE
Vendor OUI: 00:05:1e
Vendor PN: 57-1000012-01
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF109320000071
Date Code: 090802
Temperature: 42 Centigrade
Current: 7.656 mAmps
Voltage: 3281.3 mVolts
RX Power: -4.9 dBm (320.6 uWatts)
TX Power: -3.3 dBm (464.9 uWatts)
Some things to note:
| Field | Meaning |
|---|---|
| Transciever | This shows the capabilities of the SFP. Supported speeds are in Megabytes per second, so this needs to be multiplied by 10 to get the megabits (or divided by 100 for gigabits) - 800 = 8Gbps |
| Length {microns} | This shows the maximum supported transmission distance for the specified type of cable. 9u is Single-Mode (the value is zero because this SFP doesn't support Single Mode), 50u is OM2/OM3/OM4 cable, 62.5u is OM1 cable. The cable specifications are printed on the cable. This doesn't take patching/connector attenuation into account - if you have lots of patch points etc, this transmission distance will be smaller. |
| Wavelength | 850nm will be Short Wave & Multi Mode. Worth double checking when making a new connection, in case the SFPs at each end don't match or the cable isn't supported |
| Current | Very cool - the ports come with a multimeter! This parameter is important - if the current is zero ("0.000") mAmps, the SFP in the FC switch is faulty and needs to be replaced. For Short-Wave SFPs, normal current is typically around 7.000 mAmps For long wave transcievers, this value will be higher (e.g. 26mAmps for 40Km transciever) |
| Voltage | Standard operating voltage for an SFP is 3.3V (3,300 Millivolts). If a single SFP shows a low vaultage, there is an electrical fault in the SFP and it should be replaced. I don't really come across this scenario too often though - often voltage will be OK when current is zero. |
| RX Power | This is the measurement of incoming light signal, converted to
electrical power by the SFP (and with a calculated signal). Not sure if
this is peak or average signal (please provide feedback if you know
more), however more importantly, the normal operating specifcations are
well documented by the vendors. "x.xdBm" = Signal power in decibel milliwatts. A signal power of 0 dBm is one milliwatt, a signal power of 3 dBm is 2milliwatts, 6 dBm is 4 milliwatts, etc Typically any signal value below -12db (e.g. -13db, -20db, etc) would indicate a fault outside of the FC Switch SFP. Some remedial actions to try include: - Clean the cable connectors - with the correct cleaning apparatus - Clean the source and destination SFP optical interface - with the correct cleaning apparatus - Replace the patch cables - Replace the structured cabling - Replace the SFP (in the HBA) in the server/storage - Replace the HBA in the server/storage |
| TX Power | Likewise, this is a measurement of the electrical power which is converted into optical transmission out of the SFP. Typically, TX Signal Power less than -12dBm would indicate that the SFP is unable to generate the light signal required for transmission. Or more simply, the SFP in the FC switch is faulty and needs to be replace. |
Normal SFP Specifications
Each vendor will publish the normal operating specifications for SFPs. These are also very useful when troubleshooting faults, as operations outside of these parameters will narrow down the search.
Brocade specs:
http://www.brocade.com/products/all/transceivers/product-details/transceiver-modules/specifications.page
Further Reading
Bill White has put together a really good article on Link Budget Loss
which explains the calculations behind DB Loss for connections, cables,
lengths, etc. Definitely worth a read (it is a short doc, easy to
digest).
http://www.redbooks.ibm.com/abstracts/tips0087.html?Open
Examples of faults
1) This SFP shows zero voltage and an output power of -17dBm. In this situation, the SFP itself is faulty
switch01> sfpshow 8
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c404000000000 200,400,800_MB/s M5,M6 sw Short_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:2 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: BROCADE
Vendor OUI: 00:05:1e
Vendor PN: 57-1000012-01
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 003a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAF1093100007HE
Date Code: 090728
Temperature: 38 Centigrade
Current: 0.000 mAmps
Voltage: 3303.9 mVolts
RX Power: -2.1 dBm (623.5 uWatts)
TX Power: -17.2 dBm (19.2 uWatts)
2) This SFP shows a low incoming power - indicating that it can't detect a light signal from the source. Perhaps there is nothing plugged in, or there is a remote fault in the server HBA or interconnecting optical cables. Either way, there is no light reaching the SFP. The SFP is otherwise healthy
switch03> sfpshow 37
Identifier: 3 SFP
Connector: 7 LC
Transceiver: 540c402000000000 200,400,800_MB/s M5,M6 sw Inter_dist
Encoding: 1 8B10B
Baud Rate: 85 (units 100 megabaud)
Length 9u: 0 (units km)
Length 9u: 0 (units 100 meters)
Length 50u: 5 (units 10 meters)
Length 62.5u:3 (units 10 meters)
Length Cu: 0 (units 1 meter)
Vendor Name: BROCADE
Vendor OUI: 00:05:1e
Vendor PN: 57-1000012-01
Vendor Rev: A
Wavelength: 850 (units nm)
Options: 001a Loss_of_Sig,Tx_Fault,Tx_Disable
BR Max: 0
BR Min: 0
Serial No: UAA209331552692
Date Code: 090811
Temperature: 36 Centigrade
Current: 7.100 mAmps
Voltage: 3314.2 mVolts
RX Power: -24.8 dBm (3.3 uWatts)
TX Power: -2.6 dBm (552.8 uWatts)