Operating a supercomputer

A leadership-class system is not switched on and used. It is operated, by a staff of dozens, for five to seven years, at utilisation rates that would be considered abusive in any other computing context.

Getting time on one

Nobody buys an account. Time on a national system is allocated through a competitive process: peer-reviewed proposals, awarded in node-hours, typically annually.

The units are worth understanding because they shape behaviour. An allocation of 500,000 node-hours on a machine with 9,000 nodes is about 55 hours of the whole machine, or more realistically a few hundred jobs of a few hundred nodes each. Allocations are use-it-or-lose-it, which produces a distinct end-of-quarter rush and a queue that behaves very differently in the last two weeks of an allocation period.

The largest programmes (INCITE in the US, and its equivalents elsewhere) deliberately favour jobs that need most or all of the machine. A facility that bought an exascale system does not want it running thousands of small jobs, because those could have run anywhere. Capability computing, not capacity computing, is the justification for the capital.

The scheduler

Slurm runs most of the world’s large systems. Its job is to pack a stream of heterogeneous requests onto a fixed resource while satisfying policy, and it is a harder problem than it appears.

A user submits a script declaring node count, wall-clock limit and a queue. The scheduler maintains priority (a function of fair-share history, job size, queue time and any policy multipliers the site applies), and dispatches accordingly.

The mechanism that makes it work is backfill. Large jobs must wait for enough nodes to free simultaneously, which drains the machine as the scheduler holds nodes idle waiting for the last few. Backfill fills those holes with smaller jobs that can finish before the reservation starts. This is why an accurate wall-clock estimate matters: a job that requests 24 hours and needs 2 will not fit into backfill windows, will wait longer, and will hold nodes it is not using. Users who systematically over-request are the single largest source of avoidable idle capacity at most sites.

Typical utilisation targets are 90–95%. Reaching them requires a healthy mix of job sizes, which is why sites care about their queue composition as much as their hardware.

Something is always broken

With nine million cores, hundreds of thousands of DIMMs, tens of thousands of accelerators and hundreds of thousands of optical links, component failure is not an event. It is a rate.

Large systems run with mean time between failures measured in hours, not months. Realistic figures for a leadership-class machine are a hardware fault somewhere every few hours, and a fault serious enough to kill a running job perhaps daily.

The failure modes, roughly in order of frequency:

• Correctable memory errors, constantly, absorbed by ECC and logged
• Uncorrectable memory errors, which kill the job on that node
• Optical transceiver and cable failures, at a low per-link rate that becomes several a week at population scale
• Accelerator faults: falling off the bus, thermal events, silent data corruption
• Power supply and voltage regulator failures
• File system events, which affect everyone at once

The consequence for users is checkpoint-restart, and it is why a meaningful percentage of a large machine’s total delivered work is spent writing insurance rather than producing results.

Silent data corruption deserves separate mention because it is the one that frightens people. A bit flips, no error is raised, and the computation produces a wrong answer that looks entirely plausible. At this scale it happens. Defences are algorithmic (checksums on data structures, re-computation of suspicious results, algorithms with built-in residual checks), and no facility claims the problem is solved.

Acceptance: the months before anyone gets to use it

A system is not accepted on delivery. There is a contractual acceptance process, typically several months, and it is more adversarial than it sounds because a substantial payment usually depends on it.

Typical criteria: sustained performance on the operator’s own application suite at agreed problem sizes; a stability period (often 30 consecutive days) at a required availability level; demonstrated file system bandwidth; power draw within envelope; and full-system runs to shake out the failures that only appear at scale.

That last one is where the time goes. Problems that never appear on 500 nodes appear reliably on 9,000, and the shakedown period on a new leadership system routinely surfaces firmware bugs, thermal problems and fabric pathologies that took the vendor months to fix. The gap between “installed” and “in production” is commonly six months to a year, and this site’s announcement-to-first-light analysis records first_operational_date rather than delivery date for exactly this reason.

The lifecycle

Years 0–1: installation, acceptance, early science programme with a small set of prepared codes.

Years 1–4: production. Utilisation above 90%, the allocation programme in full swing, the machine doing what it was bought for.

Years 4–6: still productive, increasingly expensive per unit of work as newer hardware arrives elsewhere, and increasingly hard to keep in spares.

Years 6–7: decommissioning. Sometimes in place, as Jaguar was rebuilt into Titan reusing the site and much of the infrastructure. Sometimes outright, as Summit was retired in 2024 after six years.

Decommissioning is a real project. There is data to migrate, users to move to a successor, hardware to dispose of under environmental and sometimes security rules, and (for classified systems) media destruction requirements that are non-trivial at petabyte scale.

Historical systems keep their pages on this site after decommissioning, which is a deliberate choice: an installed base that vanishes from the record when the machine is switched off makes every historical analysis impossible. It is also why the schema carries status_as_of: an operational system with no retirement date means we found no source saying it was retired, and the generation decay analysis must not read that absence of evidence as evidence of absence.

The staff

A large facility runs on people, and the roles are more specialised than outsiders expect: systems engineers for the hardware and OS, storage engineers, network engineers, facility engineers for power and cooling, a user support team, and (distinctively) a group of computational scientists whose job is to help researchers port and tune their codes.

That last group is the one people underestimate. A machine whose users cannot reach 5% of peak is not delivering the science it was funded for, and the difference between a well-supported facility and a poorly-supported one is frequently larger than the difference between two generations of hardware.