How supercomputer performance is measured

Every performance number on this site carries a flops_basis field saying what kind of number it is, because “FLOPS” on its own is close to meaningless. This chapter is why.

Rpeak: the number that is arithmetic, not measurement

Rpeak is theoretical peak performance. It is not measured; it is multiplied out:

Rpeak = cores × vector width × FLOPs per cycle × clock frequency

It is the rate at which the machine would compute if every arithmetic unit issued a useful operation every cycle, forever, with data appearing free of charge. No real code approaches it. It is useful as an upper bound and as a specification of what was purchased, and it is misleading in every other context.

Rmax: what HPL actually measured

Rmax is a measured result from the High Performance Linpack benchmark. HPL solves a dense system of linear equations Ax = b by LU factorisation with partial pivoting, on the largest matrix that fits in memory.

The reason HPL became the standard is historical and practical: it is easy to run, it has one tunable knob that matters (problem size), it produces one number, and it has been run consistently since 1993, which makes a thirty-year time series possible at all. Very little else in computing has that property.

Its properties as a workload are unusual:

• Arithmetic intensity is very high. The factorisation is O(n³) operations on O(n²) data, so the bigger the matrix the more arithmetic per byte. This is exactly the regime accelerators are best at.
• Communication is modest and structured, mostly broadcasts along rows and columns of a process grid.
• It is almost perfectly load-balanced, with no irregular control flow, no sparse structure, no adaptive refinement.

In other words, HPL was designed (accidentally) to make this hardware look as good as it can look.

The efficiency ratio, and what it tells you

Rmax divided by Rpeak is the fraction of theoretical peak the machine achieved on its most favourable workload. Every system page here shows it. Reading across the dataset:

• CPU-only vector and scalar machines historically reached 85–93%. Fugaku reaches about 82%; the K computer reached about 93%.
• Accelerated systems typically reach 65–80%. Frontier is around 66%; Aurora around 51%.

The pattern is consistent and it is not a defect. Accelerators buy enormous peak arithmetic that is harder to keep fed; CPUs buy less peak that is easier to use. A machine with lower efficiency on HPL may still be by far the faster machine: Frontier at 66% of a very large peak beats the K computer at 93% of a small one by three orders of magnitude.

What the ratio does tell you is how much headroom the architecture leaves on the table, and how sensitive it is likely to be to memory behaviour on codes less friendly than HPL.

HPCG: the corrective

The High Performance Conjugate Gradient benchmark was introduced in 2013 by some of the same people responsible for HPL, explicitly because HPL had stopped being representative.

HPCG solves a sparse system iteratively. Its properties are the inverse of HPL’s: low arithmetic intensity, irregular memory access, frequent global synchronisation through dot products, and a multigrid preconditioner with poor data locality. It looks like real scientific code because it was designed to.

Machines score 0.5% to 3% of Rpeak on HPCG. The same hardware that reaches 70% on HPL reaches 2% here. That factor of thirty-five is the honest gap between what these machines can do on ideal work and what they do on ordinary work.

HPCG rankings are published alongside HPL rankings and are read by roughly nobody outside the field, which is a reasonable summary of the incentive structure around benchmarking.

Which FLOPS, exactly

A FLOPS figure without a precision attached is not a figure. The same accelerator delivers wildly different rates depending on the format:

Format	Bits	Typical relative rate	Used for
FP64 (double)	64	1×	Scientific computing, HPL
FP32 (single)	32	2×	Graphics, some simulation
TF32 / BF16	19 / 16	8–16×	Neural network training
FP8	8	16–32×	Inference, some training
INT8	8	16–32×	Quantised inference

A vendor claiming “20 petaflops” from a single rack is almost certainly quoting FP8 or INT8 with sparsity assumed. The same hardware in FP64 might deliver a fortieth of that. Both numbers are true; only one of them is comparable with the figures on this site, which are FP64 throughout unless the page says otherwise.

This divergence is recent and structural. Scientific computing needs FP64 because error accumulates over millions of timesteps. Neural network training tolerates (and now prefers) low precision, because stochastic gradient descent is robust to noise and lower precision means more arithmetic per watt and per byte of memory bandwidth. Accelerator vendors have followed the larger market, and the FP64 to FP16 ratio on modern parts has widened accordingly. A machine bought for AI and a machine bought for simulation are increasingly not the same machine even when they contain the same chip.

MLPerf, and what it does differently

For AI workloads the relevant benchmark suite is MLPerf, and its design choices are instructive by contrast with HPL.

It measures time to train a specified model to a specified accuracy, not raw throughput. That is a much harder thing to game: you cannot win by running faster in a way that harms convergence. It covers several tasks rather than one. And it separates “closed” submissions, which must use a prescribed model and optimiser, from “open” ones, which may not.

The weakness is that the reference models age, and optimising for a benchmark model is not the same as optimising for the model you actually want to train.

What none of these measure

Real application performance, which is what a facility is actually procured on. Large systems are bought against a suite of the operator’s own codes at agreed problem sizes, with acceptance criteria written into the contract. Those results are usually not public, and they are the numbers the buyer cares about.

Utilisation. A machine at 60% utilisation delivers less science than a slower machine at 95%. Scheduling efficiency, queue wait times and job failure rates matter enormously and appear on no ranking.

Time to solution including everything else. If the run takes two hours and writing the output takes four, the FLOPS figure is not describing your day. See storage and I/O.

Energy per result. The Green500 reranks the same submissions by performance per watt, which is a genuine corrective, though it inherits every one of HPL’s representativeness problems.

How this site treats the numbers

Every performance figure in this dataset carries an explicit basis:

• hpl_measured: a measured HPL result
• vendor_peak: a theoretical peak from the vendor or operator
• operator_stated: a figure the operator published without stating a benchmark
• our_estimate: derived by us, which requires naming the published method that produced it

The build refuses to accept a performance figure with no basis, and no our_estimate figures are published at all yet. The reasoning is on the methodology page: a number that gets quoted and turns out wrong costs more than an omission.

Where a system appeared on a public ranking, we record the rank and edition as a pointer and link to the canonical entry rather than reproducing the table. Blue Waters is the case that makes the argument for us: NCSA declined to submit an HPL result at all, on the grounds that it did not represent their science workload. It was among the largest systems in the world and appears on no ranking. A directory built only from ranked results would not know it existed.