Remote direct memory access carried over ordinary Ethernet — one machine writes into another's memory without involving either CPU.
What RoCE Is
Remote direct memory access lets one machine’s adapter write into another machine’s memory with no system call, no kernel copy and no CPU cycles on either side. For distributed training, where the same gradients cross hundreds of accelerators thousands of times a second, that removed latency and removed CPU load is the difference between a cluster that scales and one that does not.
RoCE carries that over Ethernet rather than over InfiniBand. RoCEv2 wraps the InfiniBand transport header and payload in a UDP datagram on destination port 4791, which makes it routable across an ordinary IP fabric and lets it cross subnet boundaries. RoCEv1 was a bare Ethertype and could not.
Why the UDP Wrapper Matters More Than It Looks
Two consequences follow from that encapsulation and both shape the fabric. First, the ECN bits are in the IP header, so congestion marking works on RoCE traffic exactly as it works on anything else. Second, ECMP hashing sees a five-tuple, and the only field with any freedom in it is the UDP source port.
That second point is the origin of most RoCE fabric frustration. A collective produces a handful of enormous synchronised flows, each pinned by its hash to one path. Two of them land on the same spine link while another sits idle, and the fabric is congested and underused in the same instant.
The Lossless Requirement
RDMA assumed a fabric that does not drop. Classic RoCE recovery is go-back-N: a single lost packet discards everything sent after it, which turns a 0.1 per cent loss rate into a collapse rather than a 0.1 per cent slowdown. Newer implementations do better with selective retransmission, but the design assumption is baked in.
So the fabric is engineered not to drop — PFC to pause instead of discarding, ECN with DCQCN to slow senders before pausing is needed, and enough buffer to absorb what is already in flight. Getting that right on every switch in every path is the real work of running a RoCE cluster, and it is described under lossless Ethernet.
Where You Meet It
Every Ethernet GPU cluster, most distributed storage fabrics, and any HPC deployment that chose Ethernet over InfiniBand. In a specification, ‘lossless Ethernet’ almost always means ‘we are running RoCE’.
Optically it demands nothing special — the same 800G SR8 and 2×DR4 modules serve it as serve anything else. What it does demand is a clean plant, because a link burning FEC budget produces congestion that no threshold tuning will resolve. Compare the alternatives in InfiniBand vs RoCE vs Ethernet.
RoCE reference
| Current version | RoCEv2 |
|---|---|
| Encapsulation | UDP over IP over Ethernet |
| Destination UDP port | 4791 |
| RoCEv1 vs v2 | v1 was an Ethertype, not routable; v2 is routable |
| Entropy for ECMP | the UDP source port |
| Congestion control | ECN marking plus a sender-side rate algorithm, usually DCQCN |
| Loss behaviour | historically go-back-N; selective retransmit is newer |
| Fabric requirement | a lossless class, configured on every hop |
| CPU involvement | none on the data path once the queue pair is set up |