Storage and I/O

A leadership-class system computes at exaflop rates and writes to a file system that delivers a few tens of terabytes per second. That is a ratio of roughly 100,000 to one between arithmetic and output, and it has been getting worse for twenty years: compute has grown faster than storage bandwidth in every generation.

The consequence is that I/O is where large runs actually get stuck, and it is the part of the machine users are least prepared for.

Why an ordinary file system will not do

The file system on a laptop assumes one machine, a handful of concurrent writers, and a POSIX consistency model that is cheap to honour because there is only one node to keep consistent.

Now make it nine million cores, tens of thousands of processes opening files at the same instant, and a requirement that a single file be written cooperatively by all of them. POSIX semantics (in particular, the guarantee that a write is immediately visible to every other process in the correct order) become extraordinarily expensive to maintain, because they demand coordination across the whole machine.

A parallel file system solves this by separating three concerns:

• Object storage servers hold the actual data, striped across many of them
• Metadata servers hold the directory tree, file names, permissions and the map of which stripe lives where
• Clients on every compute node talk to both, with a distributed lock manager arbitrating

Files are striped: a single logical file is cut into chunks distributed across dozens or hundreds of servers, so many processes can write different regions simultaneously at aggregate bandwidth far beyond any one device.

The three that matter

Lustre is the most widely deployed in HPC. Open source, extremely fast on large sequential I/O, and historically weak on metadata: a single metadata server was a bottleneck for years, and while distributed metadata now exists, workloads that create millions of small files remain a good way to make a Lustre file system unhappy. Frontier’s Orion is a Lustre file system on ClusterStor E1000, around 700 PB with a peak read rate in the region of 75 TB/s.

IBM Storage Scale, still universally called GPFS, takes a different design decision: metadata is distributed from the outset and every node can act as a metadata server. Historically stronger on mixed and small-file workloads, historically more expensive. Summit’s Alpine file system was Spectrum Scale at around 250 PB.

DAOS is the interesting outlier. Rather than layering a POSIX file system over block devices, it is an object store designed from scratch for byte-addressable persistent memory and NVMe, bypassing the kernel block layer entirely. It offers a key-value and array interface natively and POSIX as a compatibility layer rather than the other way round. Aurora uses it at around 230 PB, and it is the only leadership-class system in this dataset where the integrator did not also supply the storage.

Checkpointing, and why it dominates

Here is the thing nobody expects: most of the I/O on a large machine is not results. It is insurance.

A system with nine million cores, hundreds of thousands of memory modules and hundreds of thousands of optical links has a mean time between failures measured in hours. A run that needs a week of wall clock will be interrupted many times. The only defence is checkpoint-restart: periodically write the entire application state to disk, and on failure restart from the last checkpoint rather than the beginning.

The arithmetic of this is unforgiving. Suppose the application state is 5 PB (plausible for a large run), and the file system sustains 10 TB/s. One checkpoint takes 500 seconds. If the MTBF is six hours, you must checkpoint often enough that expected lost work is small, which might mean every hour. That is 500 seconds of every 3,600 spent writing, or roughly 14% of the machine’s time spent on insurance.

Get the ratio wrong in either direction and you lose: checkpoint too rarely and you lose more work per failure than you saved; too often and you spend the allocation writing. There is a classic optimum (checkpoint interval proportional to the square root of (2 × checkpoint cost × MTBF)), and it is one of the more genuinely useful pieces of theory in the field.

This is what burst buffers are for: a fast intermediate tier, usually node-local or rack-local NVMe, that absorbs the checkpoint at very high speed and drains it to the parallel file system in the background while compute resumes. The application sees the fast tier; the durable copy lands later.

The metadata problem

The failure mode that surprises people is not bandwidth. It is metadata.

A file system can be delivering terabytes per second of streaming bandwidth and fall over completely because a job has 100,000 processes each opening its own output file in the same directory. Every one of those is a metadata operation requiring a lock, and directories with hundreds of thousands of entries behave badly on every parallel file system ever built.

The canonical mistake is file-per-process I/O: every rank writes its own file. It is trivially easy to program and it is the single most reliable way to make an entire facility’s storage unusable for everyone else. Sites police it. Some refuse to schedule jobs that do it at scale.

The alternatives are collective I/O through MPI-IO, where the library aggregates writes from many ranks into a smaller number of large, well-aligned operations, and higher-level libraries (HDF5, NetCDF, ADIOS), which handle aggregation, alignment and portable self-describing formats on the application’s behalf. Using one of these is close to mandatory at scale.

The tiers

A large facility has several storage tiers with very different economics:

Tier	Media	Capacity	Bandwidth	Retention
Node-local scratch	NVMe	TB per node	GB/s per node	Job lifetime
Burst buffer	NVMe	1–100 PB	10s of TB/s	Hours to days
Parallel file system	HDD + flash	100 PB–1 EB	1–75 TB/s	Weeks, with purge
Campaign / project	HDD	10s–100s PB	GB/s	Months to years
Archive	Tape	EB	Slow, high latency	Decades

Tape is not a historical curiosity. It remains the cheapest durable medium per byte by a wide margin, it draws no power at rest, and every large facility still runs a robotic tape library. Data that must survive for thirty years lives on tape.

The parallel file system almost always carries a purge policy: files untouched for some number of weeks are deleted. This exists because there is no capacity discipline otherwise, and it is responsible for a substantial fraction of all human distress at supercomputing facilities.

Why this site records storage sparsely

Of the systems in this dataset, only a handful carry a storage edge, and that is a data gap rather than a modelling choice. Storage is frequently procured separately from the compute system, commissioned on a different schedule, shared between several machines, and described in far less detail in public announcements than the compute is.

It is one of the places where the ranking-centred view of these machines is thinnest (a spec string names the processor and the fabric and stops), and it is one of the coverage gaps openly listed on the analysis roadmap.