The interconnect

You can buy the same processors anyone else can buy. The interconnect is where a supercomputer stops being a pile of servers, and it is the component that most often decides whether a machine is usable at scale.

It is also the least intuitive part, because the figure of merit is not the one people expect. Data centre networking is sold on bandwidth. Supercomputer networking is bought on latency and, more than anything, on tail behaviour: how bad the worst message gets when the network is busy. A fabric with excellent average performance and a long tail will destroy a tightly coupled application, because at every synchronisation point the entire machine waits for the slowest participant.

Why latency dominates

Consider a stencil code: a PDE solver on a grid, the workhorse of computational science. Each timestep, every node computes on its own block, exchanges boundary values with its neighbours, and then all nodes synchronise before the next step.

The timestep takes as long as the slowest node’s compute plus the slowest message. Run a million timesteps, and every microsecond of per-step latency becomes a second of wall clock. Run for six hours and a five-microsecond regression in message latency costs you a measurable fraction of the allocation.

Now add that most large runs synchronise globally many times per second. A single straggler (one node on a congested path, one link with a marginal cable) propagates to the entire job. This is the tail at scale problem, and it is why HPC fabrics are engineered for consistency in a way that enterprise networks are not.

Concretely, the fabrics in this dataset target:

• Latency: 1–2 µs node to node, including software overhead
• Injection bandwidth: 25–100 GB/s per node
• Message rate: hundreds of millions of small messages per second
• Non-blocking or near-non-blocking bisection: any half of the machine can talk to the other half at full rate

That last property is the expensive one, and it is what oversubscribed enterprise networks deliberately do not provide.

Topologies

The topology is the wiring pattern: which node connects to which switch, and how switches connect to each other. It sets the cost, the worst-case hop count, and the bisection bandwidth.

Torus

Arrange nodes in a 3D (or higher) grid and connect each to its immediate neighbours, wrapping around at the edges. IBM’s BlueGene machines used this; Sequoia used a five-dimensional torus.

Advantages: no separate switches at all: the routing logic is on the compute chip, which makes it cheap and extremely power-efficient at scale. Nearest-neighbour communication, which is what stencil codes do, is perfect: one hop.

Disadvantages: the diameter grows with the cube root (or fifth root) of the node count, so distant nodes are many hops apart. All-to-all patterns are poor. And the mapping of your problem onto the physical topology suddenly matters a great deal: a job scheduler that hands you a fragmented, non-contiguous allocation will halve your performance.

Fujitsu’s Tofu is the surviving example: a six-dimensional mesh/torus, integrated onto the A64FX package, with no separate network card in the node at all.

Fat tree

A multi-level tree in which the links get fatter toward the root, so that bandwidth is preserved as you go up. This is what most InfiniBand installations use.

Advantages: uniform. Any node can reach any node in a small, fixed number of hops. Job placement barely matters, which makes life dramatically easier for the scheduler and for users.

Disadvantages: expensive. A full fat tree needs a great many switches and an enormous amount of cable, and cost grows faster than node count. Large installations usually taper (2:1 or 3:1 oversubscription at the top), which reintroduces placement sensitivity through the back door.

Dragonfly

Groups of nodes connected all-to-all within a group by cheap short links, and the groups connected all-to-all to each other by fewer, longer, more expensive links. Cray’s Aries used it; Slingshot uses a variant.

Advantages: very low diameter (typically three hops between any two nodes at any scale) using far less long-distance cable than a fat tree. This is the dominant topology in current leadership systems for a straightforward economic reason: optical cable is a large fraction of the network cost, and dragonfly minimises how much of it you need.

Disadvantages: the global links between groups are a shared, scarce resource. Two jobs that both need heavy inter-group traffic will contend for the same small set of links. Dragonfly networks live or die on their adaptive routing.

Full crossbar

Every node connected to every other through a single switching stage. The Earth Simulator did this across 640 nodes in 2002, using something on the order of 2,700 km of cable.

It is architecturally perfect and economically impossible past a few hundred nodes. Nobody has attempted it at scale since, and it is worth knowing about mainly as the boundary case that explains why everyone else accepts a compromise.

Congestion and adaptive routing

A network that routes each message along a fixed path is easy to reason about and fails badly under load. Two jobs that happen to share a link interfere, and neither user can see why their run got slower.

Adaptive routing lets a switch pick among several viable paths based on observed congestion. This is now standard, and it is the main thing distinguishing a modern HPC fabric from Ethernet-as-shipped. Slingshot’s specific claim is congestion control rather than merely adaptive routing: it identifies which flow is causing congestion and throttles that one, so a badly behaved job degrades itself rather than its neighbours.

That matters because these machines are shared. On a system running dozens of concurrent jobs, performance variability from network interference is one of the most common and most frustrating user complaints, and it is invisible to any benchmark run on an empty machine.

Collective operations

Applications rarely send only point-to-point messages. They perform collectives: operations involving every process in a group:

• Broadcast: one to all
• Reduce: combine a value from every process into one (sum, max)
• All-reduce: reduce, then broadcast the result back to everyone
• All-to-all: every process sends distinct data to every other

All-reduce is the important one. It is the synchronisation point in essentially every iterative solver, and it is the dominant communication pattern in distributed neural network training, where the gradients from every worker must be summed after every batch.

A naive all-reduce over N processes takes N steps. A tree implementation takes log N. A ring implementation moves optimal total bytes at the cost of more steps. Which is fastest depends on message size, topology and process count, and good MPI libraries switch algorithms at tuned thresholds.

Modern fabrics go further and perform reductions in the network itself: the switches do the arithmetic as the data passes through, so the result is computed on the way to its destination rather than being gathered and redistributed. For large-scale training this is the difference between the network being a bottleneck and not.

Who supplies it, and what happened to that market

The fabric layer used to be more diverse than it is now, and the consolidation happened fast.

Through the 2010s the field held Cray (SeaStar, Gemini, Aries), Mellanox (InfiniBand), Intel (Omni-Path), Fujitsu (Tofu), IBM (BlueGene’s integrated networks), and NUDT (TH Express) as genuinely independent suppliers. Between September 2019 and April 2020, HPE acquired Cray and NVIDIA acquired Mellanox. Intel discontinued Omni-Path in 2019, and it was taken over by Cornelis Networks in 2020.

The result is that the two companies supplying most accelerator FLOPS also supply most fabric FLOPS: accelerator share and interconnect share stopped being independent variables. That is charted, with the entity resolution that makes it visible, on the interconnect share analysis.

There is also a live architectural argument. InfiniBand has historically owned HPC; Ethernet has historically owned everything else. The Ultra Ethernet effort and vendor-specific enhancements (RoCE, Spectrum-X) are an attempt to bring Ethernet’s economics to AI fabrics. Whether that succeeds is one of the more consequential open questions in the industry, because it determines whether the fabric layer stays a specialist market or becomes a commodity one.

The part that is easy to underestimate

Cabling. A large system has hundreds of thousands of individual links. Optical transceivers fail at a low but nonzero rate, and at that population a low rate is several failures a week.

This is not a footnote. A marginal cable that passes link training but drops occasional packets produces exactly the pathology described at the top of this page: one slow node, at every synchronisation, for the whole job. Finding it is genuinely hard, and a meaningful part of what a large facility’s operations staff does is hunt for links that are technically working and quietly ruining someone’s allocation.