Letting a network adapter read and write GPU memory directly, without staging through host memory.
What GPUDirect RDMA Does
An accelerator’s working set lives in its own high-bandwidth memory, and a network adapter normally addresses host DRAM. Bridging those two without help means the GPU copies into host memory, the adapter reads host memory, and the far end reverses both steps — two extra copies, four PCIe traversals and a CPU that has to be scheduled to orchestrate them.
GPUDirect RDMA lets the adapter address GPU memory as a peer over PCIe. The payload goes from HBM to the adapter to the fibre and back into HBM at the far end, and no host buffer is ever allocated for it.
Why PCIe Topology Decides the Result
Peer-to-peer PCIe transfers are fast when both devices sit beneath the same switch and considerably slower when the transaction has to climb to the CPU root complex and back down another branch. On some platforms that path is not merely slower but effectively unusable for peer traffic.
This is the reason GPU servers pair adapters with accelerators rather than concentrating all the networking in one place, and it is the physical root of rail-optimised cabling: if each accelerator has its own adapter, each accelerator has its own cable, and those cables go to eight different switches rather than one.
Where the Boundary Sits
GPUDirect governs traffic that leaves the node. Inside the node, peers reach each other over NVLink at bandwidths an order of magnitude higher, and no adapter is involved at all. The transition between those two regimes is exactly where a transceiver or a cable appears.
That makes the boundary of the scale-up domain the single most useful line to draw on a cluster diagram before you cost the cabling: everything crossing it is a port, an optic and a fibre run.
What Goes Wrong
The characteristic failure is silent. The stack falls back to the staged copy path, everything still works, and the cluster simply delivers a fraction of the collective bandwidth it should. Causes are usually a driver or kernel-module mismatch after an upgrade, an adapter placed on the wrong PCIe root, or a virtualisation layer that blocks peer-to-peer addressing.
Diagnose it with a collective bandwidth benchmark rather than a configuration flag, and check the physical layer at the same time — a rail running with correctable-error counters climbing looks much like a software fallback from the application’s point of view. The link-down diagnostic guide covers separating the two.
What It Means for the Plant
One adapter per accelerator sets the port count, and the port count sets the cable bill. An eight-way node presents eight fabric ports, commonly 400G or 800G, and in a rail design those eight leave the rack in eight directions. Size that with the MPO cabling planner and the breakout configurator together, because whether a switch port serves one adapter or two changes the answer completely.
GPUDirect RDMA reference
| Copies removed | two — one host bounce in each direction |
|---|---|
| PCIe traversals | two instead of four |
| Best topology | NIC and GPU under the same PCIe switch |
| Worst topology | peer traffic forced across the CPU root complex |
| Transports | InfiniBand verbs, RoCEv2, and UEC transport |
| Related variant | GPUDirect Storage — NVMe straight into GPU memory |
| Usual pairing | one NIC per GPU in rail-optimised nodes |
| Verification | collective benchmark bandwidth, not a configuration flag |
Related Terms
RoCE, InfiniBand, NVLink, rail-optimised topology, AI cluster cabling guide.