AI clusters versus HPC systems

Photograph an AI training cluster and a modern HPC system and you will struggle to tell them apart. Same accelerators, same liquid cooling, same InfiniBand. The differences are real and mostly invisible.

Different arithmetic

Scientific simulation needs FP64. A climate model integrates millions of timesteps and rounding error compounds; a structural analysis whose answer depends on precision is not an answer. Double precision is not a preference, it is a correctness requirement.

Neural network training does not need it and does not want it. Stochastic gradient descent is robust to noise (arguably benefits from it), and training runs happily in BF16 or FP8 with a few numerically sensitive operations kept in higher precision. Lower precision means more arithmetic per watt, more per unit of memory bandwidth, and more parameters in the same HBM.

The consequence, as measuring performance sets out, is that the same accelerator delivers rates differing by a factor of thirty depending on the format, and vendors have followed the larger market. The FP64 to low-precision ratio on current parts has widened enough that a machine bought for AI and a machine bought for simulation are drifting apart even when they contain the same chip.

Different communication

HPC is dominated by nearest-neighbour exchange: each node talks to its immediate neighbours in a decomposed domain, frequently, in small messages, with global synchronisation between steps. Latency dominates.

Large-scale training is dominated by all-reduce: after every batch, gradients from every worker are summed and redistributed. Fewer, much larger messages, and bandwidth matters more than latency. Modern training splits several ways at once (data parallel across replicas, tensor parallel within a layer, pipeline parallel across layers), and each has a different communication signature, with the tensor-parallel traffic being the most intense and therefore confined to the fastest domain available.

This is why rack-scale NVLink domains exist. Keeping tensor-parallel traffic inside a coherent 72-GPU domain and using the slower fabric only for the less intense parallelism is the entire architectural argument for that class of product.

Different failure tolerance

HPC checkpoints because a lost run is lost work and the job cannot proceed past a failed rank.

Training checkpoints too, but with more slack. A step can sometimes be retried, a straggler dropped, a replica replaced. Frameworks include elastic training that reconfigures around lost workers. The workload is statistically forgiving in a way a PDE solver is not.

The scales involved make this necessary rather than optional: a frontier training run occupies tens of thousands of accelerators for weeks, and at that population and duration hardware failure during the run is a certainty rather than a risk.

Different scheduling

An HPC facility runs a queue of thousands of jobs from hundreds of users, and optimises for throughput and fairness.

A frontier training cluster frequently runs one job. It was built for that job, it is occupied by that job for weeks, and the scheduling problem is closer to capacity planning than to queueing. Utilisation is measured differently and the failure of a single node has a different meaning entirely.

The disclosure gap, which is the important one

Here is the part that matters for anyone trying to measure this industry.

Most of the largest compute built in the last three years appears on no public ranking. The reasons are straightforward:

• There is no incentive. A national laboratory submits an HPL result because public accountability for public money is part of the deal. A commercial operator gains nothing and discloses a competitive asset.
• HPL measures FP64, which is not what the machine was bought for and not where it looks good.
• Running HPL at scale costs days of a cluster whose time is extremely valuable.
• Cluster composition changes continuously as capacity is added, so a single measurement has a short shelf life.

The result is a systematic and growing hole in the public record. The ranked world contains national laboratory systems and a small number of voluntary commercial submissions. It does not contain the bulk of what has actually been built.

This dataset holds exactly three calibration points against that gap: Eagle, a slice of Azure that Microsoft chose to submit; and Selene, and Eos, NVIDIA benchmarking its own reference architectures. Three measured samples of the architecture that the entire unranked buildout is made of.

Note what is absent even from those. Microsoft disclosed Eagle’s performance and core count. It did not disclose the power draw or the physical site. This site records both as blank, because blank means we found no source, and treating a missing disclosure as a zero is how aggregate estimates go wrong.

Why we publish no estimate of the gap

The obvious next move is to estimate the shadow: take the measured calibration points, derive performance per accelerator, apply that to publicly disclosed accelerator counts, and publish a figure for total unranked compute.

We have not, and the reasoning is on the methodology page. Every input to that calculation is uncertain in a way that compounds: disclosed accelerator counts are frequently aspirational or cumulative rather than installed, utilisation is unknown, the mix of parts is unknown, and the efficiency of a commercial cluster on any given workload is unknown. A number built from four uncertain inputs, published with a confidence tier that readers will strip off the moment they quote it, is a liability rather than an asset.

That analysis (the shadow list) is one of the two remaining pages on the analysis roadmap, and it ships when the estimator ships, with the method fully exposed. An unsourced number that gets quoted and turns out wrong costs more credibility than an omission.

What the two worlds still share

Everything physical. The power problem is identical and the AI buildout has made it worse. The cooling problem is identical. The interconnect engineering is largely shared, which is why the same vendors supply both. HBM supply constrains both, from the same three manufacturers. And essentially all the leading-edge silicon in both worlds comes out of the same foundry.

That last point is the one the domestic substitution index has to caveat explicitly, and it applies with equal force here: a supply chain that looks diverse at the level of brand names is remarkably concentrated one layer down.