Inside a node

A supercomputer is bought by the node. Frontier has 9,408 of them, Fugaku has 158,976. Understanding one node is most of understanding the machine.

The organising fact is this: arithmetic is cheap and moving data is expensive. A double-precision multiply-add costs a few picojoules. Fetching the operands from DRAM costs a hundred times that. Fetching them from another node costs a thousand times that. Modern node design is almost entirely a set of strategies for not moving data.

The usual way to express this is the ridge point of the roofline model: on a current accelerator, you must perform on the order of 50 to 100 floating-point operations for every byte you read from memory in order to keep the arithmetic units busy. Most real scientific code does nothing like that. It is memory-bound, and its performance is set by bandwidth rather than by FLOPS.

The two-processor node

Almost every large system built since 2012 has the same basic node: one or more general-purpose CPUs, and one or more accelerators that do the actual arithmetic.

The CPU runs the operating system, the MPI library, the parts of the code that branch unpredictably, and the orchestration. It has a small number of complex cores (24 to 96), each with deep out-of-order execution, large caches and sophisticated branch prediction. It is optimised for latency: getting one instruction stream through as fast as possible in the presence of unpredictable control flow.

The accelerator is optimised for throughput: thousands of simple cores, shallow pipelines, no speculation worth speaking of, and enormous register files so that when one group of threads stalls on memory another can run immediately. It cannot handle a branchy workload at all. Given regular, dense, parallel arithmetic it will outperform the CPU by an order of magnitude at the same power.

The division of labour follows: the CPU decides what to do, the accelerator does it. In the seed dataset here, that pattern begins with Roadrunner in 2008 (Opterons paired with Cell processors), and becomes the default with Titan in 2012.

The memory hierarchy, and why HBM changed things

Between the arithmetic units and the data sits a hierarchy, and every level of it is a tradeoff of capacity against latency and bandwidth.

Level	Typical capacity	Typical latency	Rough bandwidth
Registers	kilobytes	~1 cycle	enormous
L1 cache	32–128 KB per core	~4 cycles	very high
L2 cache	0.5–4 MB per core	~15 cycles	high
L3 / last level	32–256 MB per socket	~50 cycles	moderate
HBM	64–192 GB per package	~350 ns	3–8 TB/s
DDR	256 GB–2 TB per socket	~90 ns	0.2–0.6 TB/s
Node-local NVMe	1–30 TB	~10 µs	3–14 GB/s
Parallel file system	10–1000 PB	~1 ms	1–75 TB/s aggregate

Two things in that table are worth dwelling on.

First, the numbers span nine orders of magnitude of latency, from a nanosecond in a register to a millisecond on the file system. A single badly placed access can cost more than a million arithmetic operations.

Second, HBM has higher latency than DDR but far more bandwidth, and for this workload class bandwidth is what matters. High Bandwidth Memory is DRAM stacked vertically and connected to the processor through a very wide interface (thousands of bits wide, rather than the 64-bit channels of a DIMM) usually on the same package. You trade capacity and cost for bandwidth, and for memory-bound scientific code that is almost always the right trade.

HBM is also the industry’s tightest supply constraint. It is difficult to manufacture, the stacking yield is the binding step, and it is produced by three companies. The parts taxonomy carries the generations; the practical consequence is that accelerator output volume is often set by HBM availability rather than by logic-die capacity.

Fugaku took this to its conclusion: the A64FX processor has 32 GB of HBM2 on package and no DDR at all. There is no slower, larger tier. You get high bandwidth and a hard capacity ceiling, and the machine is designed around living within it.

NUMA: memory that is nearer and further

A node with two CPU sockets does not have one pool of memory. It has two, each attached to one socket, joined by a link between them. A core reading memory attached to its own socket is fast; reading memory attached to the other socket goes across that link and is measurably slower.

This is Non-Uniform Memory Access, and it is invisible in the programming model until it destroys your performance. The operating system will happily allocate a page on the wrong socket. Standard practice is first-touch allocation (the page is physically placed near whichever core first writes to it), which means initialisation loops must be parallelised the same way the compute loops are, or every core spends the run reaching across the socket link for data it initialised serially at startup.

Modern chiplet-based CPUs have made this worse rather than better: a single socket may itself contain several NUMA domains, one per chiplet group. Getting placement right is now routine tuning work rather than an exotic optimisation.

The PCIe problem, and how it was solved

For a decade the accelerator sat on the far side of a PCI Express bus. That bus was, relative to everything around it, catastrophically slow: tens of gigabytes per second, against memory bandwidths measured in terabytes.

The consequence shaped a decade of programming. You did not move data to the accelerator to do a little work; you moved it once, did as much as possible, and moved the result back. Kernels were written to be large. Data structures were duplicated. A significant fraction of “porting to GPU” effort went into managing two separate memory spaces and the copies between them.

Three things fixed it, in order:

Coherent CPU-to-accelerator links. Summit used NVLink between POWER9 and V100 rather than PCIe, giving the GPU coherent access to host memory. Frontier uses AMD’s Infinity Fabric between its EPYC and its MI250X modules for the same reason.

Unified memory. One address space, with the runtime migrating pages on demand. Correct by default, and slow if you ignore it entirely, but it made large legacy codes portable in months rather than years.

Physical integration. MI300A puts 24 Zen 4 CPU cores and the CDNA 3 GPU chiplets on the same package with 128 GB of shared HBM3. GH200 does the equivalent with a Grace Arm CPU and a Hopper GPU over NVLink-C2C. There is no host-to-device copy because there is no separate host memory.

That convergence has a consequence for how this site records data, incidentally: on El Capitan the CPU edge and the accelerator edge point at parts made of the same silicon by the same supplier, which is why the methodology explains how the accelerator-share calculation avoids double-counting them.

The node that has no accelerator

Not every machine took this road, and the exceptions are instructive rather than backward.

Fugaku’s A64FX is a 48-core Arm processor with 512-bit vector units and on-package HBM. There is no accelerator, no separate memory space, no host-device copy, and no porting effort beyond vectorisation. It reached number one in the world.

Hawk, and ARCHER2 made the same choice for a different reason: their users run engineering codes with irregular control flow and complex data dependencies that vectorise poorly and port to GPUs badly. Buying an accelerated machine for those users would have bought FLOPS they could not reach.

This is why the accelerator share analysis keeps a CPU-only band rather than dropping those systems. Their absence from the accelerator market is a real fact about what compute is for, not a gap in the data.

What a node costs to feed

Some grounding numbers for a current accelerated node:

• Peak double-precision arithmetic: 60–130 TFLOP/s per accelerator
• HBM bandwidth: 3–8 TB/s per accelerator
• Host memory bandwidth: 0.4–0.8 TB/s
• Interconnect injection bandwidth: 25–100 GB/s per node
• Power: 700–1,500 W per accelerator, 2.5–6 kW per node

Note the shape. Between the accelerator’s own memory and the network there is a factor of roughly fifty. Any algorithm that needs to move data off-node at a rate proportional to its arithmetic will be limited by the network by a wide margin, which is why the next chapter is about the interconnect, and why it is the part of these machines that vendors compete on hardest.