Tuning a RoCE Fabric: PFC, ECN and the Counters That Tell You the Truth
RoCE works at scale. It also has a reputation for being fragile, and that reputation is earned — not because the technology is bad, but because "lossless Ethernet" is an end-to-end invariant that a single mis-set threshold quietly breaks.
This is what the mechanisms actually do, and which numbers tell you whether yours is working.
Why Lossless Is Hard, In One Paragraph
RDMA lets a network adapter write directly into a remote machine's memory with no CPU involvement. It was designed for InfiniBand, where credit-based flow control means a sender never transmits without somewhere to put the data, so loss cannot occur. Ethernet has no such guarantee. Bolting RDMA onto Ethernet therefore requires configuring the network never to drop — and doing that consistently across every switch in every path is the whole difficulty.
ECN Does the Work
Explicit Congestion Notification is the mechanism that should be doing your congestion control. When a switch queue crosses a configured depth, the switch marks two bits in the IP header instead of dropping the packet. The receiver notices, tells the sender, and DCQCN reduces the sending rate.
The critical property is that this happens before anything has to be paused. Nothing is lost, nothing is blocked, and the sender simply slows down. A well-tuned fabric under heavy load looks like ECN marks rising and falling while PFC counters stay flat.
Tuning is a real exercise and it is workload-specific. Mark too late and the queue reaches the pause threshold. Mark too early and you throttle senders that did not need throttling, wasting fabric capacity. AI collectives make this harder than general traffic because they are synchronised — every accelerator wants to send at the same instant, so queues build in bursts rather than gradually.
PFC Is the Emergency Brake
Priority Flow Control pauses one traffic class on a link rather than dropping it. It is genuinely necessary as a backstop, and it has three side effects you should treat as costs rather than surprises.
Head-of-line blocking. Pausing a class on an upstream port stops all traffic in that class on that port, including flows headed somewhere entirely uncongested.
Congestion spreading. Pause propagates backwards hop by hop. One congested link can throttle a large part of the fabric, and the machines being throttled have nothing to do with the congestion.
Deadlock. In a topology with cycles, PFC can deadlock outright — each switch waiting on the next, indefinitely. This is rare, catastrophic, and requires deliberate design to avoid.
None of this means PFC should be disabled. It means PFC firing regularly is a symptom to investigate, not a mechanism that is working.
The Four Counters Worth Graphing
Before the first training run, get these four graphed per switch, per priority class, with history. Without them you are guessing.
PFC pause frames sent and received. The primary health signal. Should be near zero in steady state. A slow climb over days usually means a link or a queue is progressively degrading.
ECN marked packet count. Should be non-zero under load — that is ECN doing its job. Zero marks together with non-zero PFC pauses means your ECN thresholds are set too high.
Pre-FEC bit error rate, per port. Not a congestion metric, but it belongs on the same dashboard, because a marginal optical link produces symptoms that look exactly like fabric congestion. See our notes on module quality and the pre-FEC BER entry for how to read it.
Buffer occupancy per queue. Where available. This is what ECN and PFC are both reacting to, and seeing it directly removes most of the guesswork from threshold tuning.
Four Mistakes That Cost Real Throughput
Inconsistent configuration after a hardware swap. The most common failure by a wide margin. A replacement switch with default ECN and PFC settings breaks losslessness for every path through it, and nothing alarms. Put the fabric configuration under change control and audit it, do not trust it.
Treating PFC as the congestion control. If pause frames are how your fabric handles load, you have head-of-line blocking as a design feature. Fix ECN until PFC goes quiet.
Blaming the fabric for a physical-layer fault. A dirty connector on one spine uplink raises errors, which raises retransmissions, which looks like congestion. Check DDM receive power and pre-FEC BER before touching a threshold.
Ignoring the in-rack links. Server-to-leaf links are where most of the traffic starts. They are also where passive DAC is the right answer — cheaper, cooler and with nothing to contaminate. Every optical link you do not install is a link that cannot develop a physical-layer fault.
FAQ
Can I run RoCE without PFC? Newer NIC and congestion-control combinations tolerate loss far better than classic RoCEv2, and running without PFC is increasingly discussed. Validate it at your own scale with your own traffic before committing — the failure mode is a slow job, not an outage, so it is easy to miss.
What ECN thresholds should I use? There is no universal answer; they depend on link speed, buffer size and traffic pattern. Start from your switch vendor's AI reference configuration, then tune against the counters above under a real workload rather than a synthetic one.
Does this all go away with Ultra Ethernet? Much of it should. UEC's loss-tolerant transport removes the need for the fabric to be perfectly lossless. Until it ships and is proven, this is the work.
Does the optics choice affect RoCE? Only through reliability. A fabric with marginal modules generates errors that masquerade as congestion. That is an argument for traffic-tested optics with a known failure rate, which is exactly what we ship.
