The three tiers of AI infrastructure: inside the node, across the cluster, and between data centres.
Three Tiers, Not Two
Scale-up means adding capability inside one tightly coupled domain: more accelerators sharing one NVLink fabric, addressing each other’s memory at terabytes per second. Scale-out means adding more such domains and joining them with a cluster network — InfiniBand or lossless Ethernet — at 400G or 800G per accelerator.
Scale-across is the third tier, and it is recent. It means connecting whole clusters in different buildings or different cities, because no single site has the power, the cooling or the floor space for the model that needs training.
The Order-of-Magnitude Rule
A fifth-generation scale-up link gives an accelerator 1.8 TB/s. An 800G fabric port gives it 100 GB/s. That first step down is roughly an order of magnitude per accelerator. Scale-across drops further again, and in different units: a metro coherent link gives a few hundred gigabits to a whole cluster, not to one GPU. Latency worsens in the same direction at every step.
The entire discipline of AI infrastructure design follows from that. Tensor parallelism, which exchanges activations at every layer, must stay inside scale-up or the model simply will not train at a useful rate. Pipeline parallelism tolerates scale-out. Data parallelism, which synchronises occasionally and in bulk, is the only pattern that survives scale-across.
What Each Tier Is Cabled With
The tier decides the medium before anything else does. Scale-up is electrical and mostly invisible to a cabling team — in-package, on-board, or a passive cartridge inside one rack. Scale-out is where pluggables live: DAC and AOC for in-rack and in-row hops, parallel single-mode over MPO trunks for anything crossing rows.
Scale-across is coherent territory — 400ZR and ZR+ pluggables over dark fibre or a DWDM line system, which is a different engineering conversation involving dispersion, OSNR and amplification. The standard lookup maps a distance and a rate to an actual part number.
How to Use the Model
Draw the tier boundaries on the cluster diagram before costing anything. Every link that crosses a boundary is a port, a module and a fibre run; every link that stays inside one is somebody else’s problem. That one exercise catches most of the errors in early AI cluster budgets, because it forces the question of how many accelerators are actually inside the scale-up domain — a number that has grown from eight to seventy-two and is still growing.
As it grows, the scale-out tier gets fewer but fatter links, and the copper share of them falls. Model the cost of that shift with the TCO calculator rather than assuming last generation’s ratio holds.
Tier reference
| Scale-up scope | one node, or one rack-scale memory domain |
|---|---|
| Scale-up bandwidth | 1.8 TB/s per GPU at the fifth NVLink generation |
| Scale-up media | in-package and in-rack electrical |
| Scale-out scope | node to node within a hall |
| Scale-out bandwidth | 400G or 800G per accelerator — 50 to 100 GB/s |
| Scale-out media | DAC, AOC and short-reach parallel optics |
| Scale-across scope | building to building, or metro |
| Scale-across media | coherent pluggables, 400ZR and ZR+ |
| Parallelism fit | tensor inside scale-up, pipeline and data outside |
Related Terms
NVLink, rail-optimised topology, leaf-spine, 400ZR and ZR+, coherent optics.