An Ethernet fabric configured so that congestion causes delay rather than dropped packets — a prerequisite for RoCE.
A Discipline, Not a Feature
There is no command that makes Ethernet lossless. What exists is a set of independent mechanisms — a traffic class, priority flow control on that class, ECN thresholds beneath the pause threshold, and enough reserved buffer — which together produce a class where congestion causes delay rather than discard.
Ethernet was designed the other way round, and deliberately so. Dropping under pressure and letting the endpoint recover is what let it scale cheaply for decades. Losslessness is an exception carved out of that behaviour for one class of traffic, and it is carved out by hand.
Why Headroom Is a Cabling Question
When a switch decides to pause, frames are already on the wire and more will be sent before the sender reacts. The receiver must have buffer reserved to absorb all of them, or it drops exactly the packets PFC existed to save. That reserve is headroom.
Its size depends on the link rate and on how long the wire is — a longer fibre run holds more in-flight data at the moment the pause is issued, and light covers roughly 200 metres per microsecond in glass. Switch defaults assume a modest reach. Move the same configuration onto long cross-hall runs without revisiting headroom and the fabric quietly stops being lossless at exactly the links that matter most.
The Invariant Problem
Losslessness is a property of a path, not of a device, which makes it fragile in a way that ordinary configuration is not. A single switch replaced during a maintenance window with default queueing, or one firmware upgrade that resets a threshold, removes the property for everything traversing it.
Nothing alarms. The symptom is a training job that takes longer than it used to, discovered days later. This is why mature operators treat threshold configuration as versioned infrastructure rather than device settings, and why Ultra Ethernet attacks the problem from the other end — making occasional loss cheap to recover from instead of making the fabric perfect. Ultra Ethernet vs RoCE sets out the contrast.
What Goes Wrong
Three faults account for most incidents. Priority mapping that differs between two vendors’ defaults, so the class is lossless on one side of a link and not the other. PFC enabled on more classes than necessary, which multiplies the blast radius of any pause. And a marginal optical link generating correctable errors, which throttles a lane and creates persistent congestion that no amount of tuning will fix.
That last one is worth emphasising because it is a physical-layer fault presenting as a congestion problem. Before retuning thresholds, read the DDM page and the FEC counters on the suspect ports.
Where You Meet It
Any GPU cluster running RoCE, and any converged fabric carrying NVMe over Fabrics. The RoCE PFC and ECN tuning guide covers the counters; the power and thermal calculator covers what running those links hot actually costs.
Lossless Ethernet reference
| Component standards | IEEE 802.1Qbb PFC, 802.1Qaz ETS, plus ECN in the IP header |
|---|---|
| Classes typically made lossless | one — occasionally two with storage |
| Threshold ordering | ECN low mark, ECN high mark, then PFC pause |
| Headroom | covers a link’s worth of in-flight frames plus reaction time |
| Headroom driver | link speed and cable length, not traffic volume |
| Health signal | ECN marks rising, PFC pause counters flat |
| Deadlock guard | PFC watchdog on every switch |
| Alternative approach | UEC transport, which tolerates occasional loss |