Programming a supercomputer

A supercomputer exposes parallelism at three levels, and a program that does not address all three will leave most of the machine idle.

1. Between nodes: thousands of separate machines with separate memory, which must exchange data explicitly over the network.
2. Within a node: dozens of cores sharing memory, plus one or more accelerators with their own.
3. Within a core or accelerator: vector units, or thousands of accelerator threads that must be given regular, coalesced work.

Different tools address each, they compose awkwardly, and getting all three right in one code base is the actual job.

Between nodes: MPI

The Message Passing Interface has been the standard since the mid-1990s and shows no sign of being displaced. It is a library, not a language: processes have distinct ranks, separate address spaces, and communicate by explicitly sending and receiving.

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

/* exchange boundary data with neighbours */
MPI_Sendrecv(send_buf, n, MPI_DOUBLE, right, tag,
             recv_buf, n, MPI_DOUBLE, left,  tag,
             MPI_COMM_WORLD, &status);

/* everyone needs the global residual */
MPI_Allreduce(&local, &global, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);

MPI’s durability is often mistaken for inertia. It survives because its model (explicit, distributed, no shared state) is an honest description of what the hardware is, and because thirty years of scientific software is written in it. Its collectives are also where an enormous amount of vendor optimisation lives, including the in-network reductions described in the interconnect.

The criticism is fair too: it is verbose, easy to deadlock, and offers no help with load balancing. Alternatives exist (PGAS languages such as UPC and Coarray Fortran, task-based runtimes such as Legion and Charm++), and each has real adherents and a small fraction of the installed code base.

Within a node: OpenMP

Threads across the cores of a node, expressed as compiler directives so that the serial code remains readable and compilable without them:

#pragma omp parallel for reduction(+:sum)
for (int i = 0; i < n; i++) {
    sum += a[i] * b[i];
}

Almost every large HPC application is MPI + OpenMP: one MPI rank per socket or per NUMA domain, OpenMP threads within it. This is the standard structure, and it exists because pure MPI wastes memory duplicating data per rank while pure OpenMP cannot cross a node boundary.

Since version 4.0 OpenMP has also offered target directives for offloading to accelerators, which is the most portable of the offload options and generally not the fastest.

On the accelerator: three dialects of the same idea

This is where portability breaks, and where the cost lives.

CUDA is NVIDIA’s, dates from 2007, and is the reason NVIDIA has the position it has. It is not primarily a hardware story: it is fifteen years of libraries (cuBLAS, cuDNN, cuFFT, NCCL) plus profilers, debuggers and an enormous body of documentation and trained people. Competitors match the silicon far more easily than they match that.

HIP is AMD’s, and is deliberately near-identical to CUDA. AMD ships a translation tool that mechanically converts most CUDA source to HIP, which is how Frontier, and LUMI were made usable by codes originally written for NVIDIA hardware. The mechanical part works; the tuning does not transfer, and the tuning is most of the performance.

SYCL is the open standard, implemented by Intel as oneAPI and used on Aurora. Single-source C++, standards-based, portable in principle across vendors.

The three express the same computation and are not interchangeable in practice. Occupancy tuning, memory access patterns, shared-memory tiling, warp or wavefront sizes (the details that separate a working port from a fast one) are all vendor-specific.

What porting actually costs

A large scientific application is 100,000 to several million lines, often with Fortran dating to the 1980s at its core, maintained by a small team who are domain scientists rather than performance engineers.

Moving it to a new accelerator, in the order it actually happens:

1. Get it to compile and run correctly on one node. Weeks to months.
2. Find the hot loops. Usually a small fraction of the code accounts for most of the time, which is the good news.
3. Restructure data layout. Array-of-structures becomes structure-of-arrays; loops are reordered for coalesced access. This changes code everywhere, not just in the hot loops, and it is where most of the effort goes.
4. Offload and tune. Kernel by kernel.
5. Eliminate the copies. The first working version moves data host-to-device constantly and is often slower than the CPU original. Fixing this is a second, separate pass.
6. Scale it. Everything that worked on one node meets the network.
7. Re-validate the science. Different arithmetic order gives different rounding, so results differ in the last digits. Establishing that the new answers are still correct is a scientific question, not a software one, and it is frequently the longest step.

For a major code this is a multi-year effort at the scale of several full-time staff. The US Exascale Computing Project ran for seven years and cost on the order of $1.8 billion, and a large fraction of that was software rather than hardware.

Which is the real answer to why an incumbent accelerator vendor is hard to displace. The switching cost is not the hardware; it is the years of work already spent tuning to the hardware you have. It is also why Fugaku’s architecture is more interesting than it first appears: an Arm CPU with wide vectors and on-package memory needs vectorisation, not a rewrite into a new offload dialect.

The software the machine actually runs

Beneath the application:

• Node OS: a Linux derivative, often trimmed. HPE Cray OS is SUSE-derived; most European and US systems run RHEL or SLES; commercial AI clusters overwhelmingly run Ubuntu. BlueGene ran a minimal compute node kernel with a Linux I/O node layer above it.
• Scheduler: Slurm nearly everywhere, PBS Pro at some sites.
• MPI implementation: usually a vendor build derived from MPICH or Open MPI, tuned to the fabric.
• Compilers: vendor compilers, plus GCC and LLVM. Compiler quality is a genuine differentiator and a genuine risk.
• Module system: Lmod or environment modules, because a site supports dozens of incompatible software stacks simultaneously.
• Containers: Apptainer or Podman, increasingly, since reproducing a software environment across facilities is otherwise miserable.

The operating system is also, quietly, a sovereignty signal. Chinese national systems run Kylin, which is a policy fact rather than a technical one, and it is one of the four roles scored on the domestic substitution index.