Compute·Atlas

How supercomputers work/Chapter 1

Chapter 01 · 12 min read

What actually makes a supercomputer different

A supercomputer is not a fast computer. It is a very large number of ordinary computers wired together tightly enough to work on one problem at once, and almost every design decision in the machine follows from that one fact.

Ask what separates a supercomputer from a good workstation and the intuitive answer (it is faster) is wrong in a way that matters. The processors inside Frontier run at about 2 GHz. A laptop bought the same year ran at 4 GHz or better. Per core, on a single thread, the laptop wins.

Frontier is roughly four million times faster than that laptop anyway, because it has 9,066,176 cores and the laptop has eight, and because the machine around those cores is built so they can work on one problem simultaneously rather than on nine million separate ones.

That is the whole thing. Everything else in this section (the memory hierarchy, the interconnect topology, the file system, the cooling plant, the programming model) is a consequence of trying to make an enormous number of ordinary processors behave, for a few hours, like one very large one.

Why not just build a faster processor

Because nobody knows how, and they have not known since about 2005.

For roughly thirty years, processors got faster in a way that required no effort from software. Dennard scaling described the happy arrangement: as transistors shrank, their power density stayed constant, so each generation could run more transistors at a higher clock in the same thermal envelope. Code written in 1990 ran faster in 1995 because the clock was faster. Nobody had to do anything.

That ended in the mid-2000s. Leakage current stopped scaling with feature size, power density began climbing, and clock speeds flattened out between 3 and 5 GHz, where they have essentially remained for twenty years. The transistors kept shrinking (Moore’s law had a decade or more left in it), but the free speed did not come with them.

What the extra transistors bought instead was more processors: multiple cores per chip, then many cores, then wide vector units, then accelerators with thousands of simple cores. Every one of those is a parallelism story. Since 2005, essentially all performance growth in computing has come from doing more things at once rather than doing each thing faster, and the entire supercomputing field is that trend taken to its logical end.

The ceiling: Amdahl’s law

Parallelism has a hard limit, and it is unforgiving.

Gene Amdahl’s observation in 1967 was arithmetic, not engineering. If a fraction p of a program can be parallelised and the remaining (1 − p) is inherently serial, then with N processors the best possible speedup is:

speedup = 1 / ((1 − p) + p/N)

Take the limit as N goes to infinity and the p/N term vanishes, leaving 1 / (1 − p). If 1% of your program is serial, the maximum speedup you can ever achieve (with infinite processors, a perfect interconnect and no communication cost) is 100×. Not 100× per processor. One hundred times, total, forever.

At the scale of the machines in this dataset, that number is brutal. Frontier has nine million cores. To use even 1% of them effectively, a program must be more than 99.99% parallel. A single serial bottleneck (one global synchronisation, one rank-0 gather, one file opened by a single process) will flatten a machine that cost half a billion dollars.

This is why “we bought a bigger cluster” so rarely translates into proportionate speedup, and why the hard part of high-performance computing has never been buying the hardware.

The escape hatch: Gustafson’s law

Amdahl’s law assumes a fixed problem. In 1988 John Gustafson pointed out that this is not how anybody actually uses a supercomputer.

Nobody buys an exascale machine to run yesterday’s simulation faster. They buy it to run a bigger simulation: finer mesh, more atoms, higher resolution, longer simulated time. And in most scientific codes, the serial fraction does not grow with problem size while the parallel fraction does. Doubling the grid resolution of a climate model doubles the work in the parallel part and leaves the setup and I/O roughly where they were.

So the practical measure is not “how much faster does the same problem run” (strong scaling, which Amdahl governs and which fails quickly), but “how much bigger a problem can I run in the same wall-clock time”: weak scaling, which is far more forgiving.

The distinction is not academic. It determines what a machine is for. A system that scales strongly is useful for turnaround time on a fixed question; one that scales weakly is useful for asking a bigger question. Most leadership-class systems are procured on weak-scaling arguments, and most disappointment with them comes from users who expected strong scaling.

What these machines actually run

The workloads that justify this architecture share a shape: they are large, regular, and decomposable into pieces that mostly talk to their neighbours.

Partial differential equations on a grid. Climate, weather, computational fluid dynamics, combustion, astrophysics. Divide the physical domain into blocks, give each block to a processor, and at every timestep exchange the boundary values with the adjacent blocks. The computation is local; the communication is local; the whole thing scales as long as the boundaries stay small relative to the interiors.

Dense linear algebra. Matrix factorisation, eigenvalue problems. This is what the benchmark on which these machines are ranked actually measures, and it is unusually well-suited to them, which is precisely the criticism of that benchmark, covered in how performance is measured.

Molecular dynamics. Millions of atoms, forces computed pairwise with cutoffs, integrated forward in femtosecond steps. Nearest-neighbour communication, enormous timestep counts, and a hard serial dependency between steps: you cannot compute step n+1 before step n.

Monte Carlo and ensemble methods. Embarrassingly parallel: thousands of independent runs with different seeds or parameters. These scale essentially perfectly and are the least interesting case architecturally, because they barely need the interconnect at all.

Neural network training, which is a newer arrival and a genuinely different shape. See AI clusters versus HPC systems.

Why not just use the cloud

Increasingly, for some workloads, you can, and Eagle exists in this dataset precisely because Microsoft submitted a slice of Azure for measurement. But the gap is real and it is mostly about the network.

A general-purpose cloud region is optimised for millions of independent tenants doing unrelated things. Its network is built to be fair, oversubscribed and resilient, not to deliver 200 Gb/s of non-blocking bandwidth between every pair of nodes with sub-microsecond latency. A tightly coupled PDE solver run across a hundred ordinary cloud instances will spend most of its time waiting.

What has changed is that cloud providers now build dedicated HPC and AI fabrics: InfiniBand or equivalent, non-oversubscribed, with placement groups that guarantee physical locality. Those are supercomputers that happen to be rented by the hour. The architecture converged; the business model did not.

The remaining differences are less about capability than about economics and access: a national laboratory system is procured once and amortised over five to seven years of near-continuous use at high utilisation, which is a very different cost structure from renting the same capability at retail.

The shape of the rest of this section

Given that a supercomputer is a parallelism machine, the engineering problems follow in a predictable order: