The short answer
Distributed GPU training exchanges the same gradients between hundreds of accelerators thousands of times a second. The fabric carrying that traffic is not a commodity: a fabric that drops packets under congestion turns a synchronised collective operation into a stall, and the whole cluster waits.
Three answers exist. InfiniBand is lossless by architecture. RoCE makes Ethernet lossless by configuration. Standard Ethernet does not try. The differences between them are less about raw bandwidth than about what happens when the fabric is busy.
What Is InfiniBand?
A purpose-built cluster interconnect with native RDMA, hardware-offloaded collective operations, and credit-based flow control. A sender is told exactly how much buffer the receiver has and never transmits more than that, so congestion cannot cause a drop — there are no pause thresholds to tune and no deadlock to design around.
It also runs collectives such as all-reduce in the switch fabric rather than in the GPUs. Generations track Ethernet speeds closely: HDR at 200G, NDR at 400G, XDR at 800G, on OSFP optics and DAC. See InfiniBand.
What Is RoCE (on lossless Ethernet)?
RDMA carried over ordinary Ethernet, routable over UDP/IP in its RoCEv2 form. The hardware is standard Ethernet switching, which means a far broader supply base and a skill set your team already has.
The catch is that RDMA expects a fabric that does not drop, so the network has to be configured not to: PFC to pause rather than discard, and ECN with DCQCN to slow senders before pausing is needed. Getting that tuning right, and keeping it right across a fabric of hundreds of switches, is the hard part. See RoCE.
What Is Standard Ethernet?
Ethernet as it comes: TCP, drop-on-congestion, no special configuration. Cheapest, simplest and by far the best understood.
For storage traffic, management, inference serving and anything that is not a tightly synchronised collective, it is entirely adequate and there is no reason to complicate it. It is only distributed training and similar tightly coupled workloads where drop-and-retransmit costs enough to matter.
InfiniBand vs RoCE vs Standard Ethernet
The rows that actually decide procurement are supply base, operational burden and what happens under congestion. Bandwidth is a tie.
| InfiniBand | RoCE (on lossless Ethernet) | Standard Ethernet | |
|---|---|---|---|
| Loss behaviour | Lossless by architecture | Lossless by configuration | Drops under congestion |
| Flow control | Credit-based — nothing to tune | PFC + ECN/DCQCN — must be tuned | None beyond TCP |
| RDMA | Native | RoCEv2 over UDP/IP | Not used |
| Collective offload | Yes, in the switch fabric | NIC-side, varies by vendor | No |
| Supply base | Effectively one dominant vendor | Broad — many switch vendors | Broadest |
| Operational burden | Low once built; separate skill set | High — fabric-wide tuning discipline | Lowest |
| Typical optics | OSFP, NDR/XDR | OSFP or QSFP-DD, 400G/800G | Anything |
| Cost per port | Highest | Middle | Lowest |
| Best fit | Large training clusters | Ethernet-standardised AI estates | Inference, storage, general |
The operational burden row deserves more than a phrase. A RoCE fabric is lossless only if every switch in every path has consistent priority mapping, ECN thresholds and PFC configuration. One switch replaced during a maintenance window with a default configuration silently breaks the property for everything flowing through it, and the symptom is a training job that got slower, not an alarm. That is the real cost of RoCE, and it is a staffing cost rather than a hardware one.
None of this changes the physical layer. All three run on the same optics and the same fibre — 800G modules, DAC and AOC in the rack, and MPO trunk between rows. The fabric decision is a switching and NIC decision; the cabling is the same either way.
Which One Should You Choose?
Large-scale training where performance dominates: InfiniBand
If the cluster's purpose is training large models and the budget follows the performance, InfiniBand remains the lowest-risk answer. It is lossless without anyone having to keep it lossless, and the in-network collective offload is a genuine architectural advantage rather than a marketing one. Accept that you are buying into one vendor's ecosystem and that your team needs InfiniBand skills.
An Ethernet-standardised organisation building AI capacity: RoCE
If everything else you run is Ethernet, staying Ethernet has real value: one supply base, one skill set, one monitoring stack. RoCE is entirely viable at scale — but budget for the tuning work explicitly, instrument PFC pause and ECN marking counters from day one, and make fabric configuration consistency a managed thing rather than a hope.
Inference, storage and general workloads: standard Ethernet
Do not make everything lossless because part of the estate needs to be. Inference serving, model storage, checkpointing and management traffic run perfectly well on ordinary Ethernet, and keeping them there means the lossless domain stays small enough to reason about.
Specifying now with an eye on the future: ask about UEC
Ultra Ethernet aims to give Ethernet a transport that tolerates loss and reordering efficiently, which would remove most of RoCE's tuning burden and make packet spraying practical. Ask switch and NIC vendors directly whether the hardware you are buying now will support it later — the answer materially affects the life of the purchase.
Frequently Asked Questions
Is InfiniBand actually faster than Ethernet?
Per link, no — 400G is 400G. The difference is behaviour under congestion and latency consistency. InfiniBand's credit-based flow control means a busy fabric degrades predictably; a badly tuned RoCE fabric can degrade sharply and unpredictably. For collectives, the tail latency matters more than the average.
Can I run RoCE without PFC?
Some newer NIC and congestion-control combinations tolerate occasional loss far better than classic RoCEv2 did, and running without PFC is increasingly discussed. It is not the default assumption, and if you intend to try it, validate it at scale with your own traffic pattern rather than trusting a vendor slide.
Do InfiniBand and Ethernet use different optics?
The optics are largely the same hardware — 400G and 800G modules in OSFP, on the same fibre. What differs is the coding and the qualification: an InfiniBand switch expects modules identified for its platform. Tell us the switch model on the RFQ and we code and test accordingly.
What is the actual cost difference?
InfiniBand switches and adapters carry a premium, and the single-vendor supply base means less negotiating room. Ethernet's advantage is competition. Where the gap narrows is total cost: a RoCE fabric that needs specialist attention to stay tuned has an operational cost that does not appear on the purchase order.
Which do the largest clusters use?
Both, in quantity. The industry has not converged, and the reasonable reading is that it will not until Ultra Ethernet is proven at scale. Design so that the cabling plant — the part that lives longest — does not depend on which way that goes.
The Parts We Build
Whichever fabric you choose, the physical layer is the same — and it is the part we build.
