Marking packets to signal congestion instead of dropping them — the early-warning half of lossless Ethernet.
What ECN Actually Marks
Explicit congestion notification uses the two least-significant bits of the IPv4 type-of-service byte — the six above them being the DSCP that selects the priority class. A sender that supports ECN sets those bits to an ECT codepoint; a congested switch flips them to CE and forwards the packet unchanged in every other respect.
Nothing is lost and nothing is retransmitted. The information travels forward with the data, and it is the receiver’s job to turn it into something the sender can act on. In RoCEv2 that is a congestion notification packet sent back to the source.
The Marking Curve
Switches do not mark on a single hard threshold. Below a low watermark nothing is marked; above a high watermark everything is; between them the marking probability rises with queue occupancy. That gradient is what lets DCQCN converge smoothly instead of oscillating between full rate and a crawl.
The absolute numbers depend on port speed and on how much buffer the silicon actually gives that queue. The arithmetic is worth doing once: an 800G port drains roughly 100 bytes every nanosecond, so a megabyte of queued data is about ten microseconds of transmission. Set the low mark in megabytes on a shallow-buffer switch and you have set it above the buffer.
Why the Loop Has Latency
ECN cannot act faster than a round trip. The mark has to reach the receiver, the CNP has to come back, and the NIC has to apply the new rate. Inside one hall the fibre contributes little — light travels about 5 µs per kilometre of glass, so a 100 m link is half a microsecond each way — but queueing and NIC processing are not negligible.
That delay is exactly why the thresholds are ordered the way they are. PFC reacts in one link time and needs no cooperation from the endpoint, so it will always win a race. ECN has to be given a head start.
What Goes Wrong
Two failure modes dominate. Thresholds set too high mean the queue reaches the pause line before marking has done anything, and the fabric runs on PFC backpressure with all the head-of-line blocking that implies. Thresholds set too low mean senders are throttled while links sit half idle, which reads as a slow cluster rather than a congested one.
The diagnostic pair is always the same: read ECN marking counters and PFC pause counters together. Marks rising with pause counters flat is a healthy fabric. Pause counters climbing at all is a tuning problem, not a capacity problem. The RoCE PFC and ECN tuning guide works through the specific counters.
What It Means for the Plant
ECN costs nothing optically — it is two bits in a header. What it buys is the ability to run leaf-to-spine links closer to line rate without a pause storm, which is the same thing as needing fewer of them. A fabric over-provisioned to stay off the pause brake is buying transceivers to compensate for configuration, and the TCO calculator will price that over three years.
ECN reference
| Field | the two low bits of the IPv4 TOS / IPv6 traffic class byte |
|---|---|
| Codepoints | 00 Non-ECT, 01 and 10 ECT, 11 CE (congestion experienced) |
| Marking policy | WRED-style, probability rising between a low and a high threshold |
| RoCEv2 feedback | CNP — congestion notification packet, receiver to sender |
| Sender algorithm | DCQCN |
| Reaction time | one network round trip, plus NIC processing |
| Ordering rule | both ECN thresholds must sit below the PFC pause threshold |
| Queue arithmetic | at 800G a port drains ~100 bytes/ns, so 1 MB of queue ≈ 10 µs |
Related Terms
PFC, lossless Ethernet, RoCE, Ultra Ethernet, InfiniBand vs RoCE vs Ethernet.