2026-10-02 · Compute Atlas

Aurora: eight years from contract to first list

Aurora took eight years from a 2015 award to its first TOP500 entry, after Knights Hill was cancelled. It now holds 1.012 exaflops, and about 96% of recorded Intel Max GPUs in this dataset.

Aurora at Argonne National Laboratory is a 10,624-node HPE Cray EX system with 63,744 Intel GPUs and 21,248 Intel Xeon Max CPUs. Its TOP500 result is 1.012 exaflop/s on High-Performance Linpack (FP64) against a 1.98 exaflop/s theoretical peak, at 38.7 MW. That makes it the second system to cross one exaflop, after Frontier.

The most useful thing to know about it is how long it took. The contract was announced in April 2015 for a 180-petaflop machine due in 2018. The processor it was designed around was cancelled, the target was rewritten to one exaflop, and the machine that eventually arrived in 2023 shares almost nothing with the one that was ordered. In this dataset, no other system has a longer gap between announcement and first light.

How it came to exist

On 9 April 2015 the Department of Energy announced a $200 million award to Intel, with Cray as subcontractor, for a system called Aurora and a smaller bridge system called Theta (The Register, 2015). Aurora was to be 180 petaflops, at least 50,000 nodes built on Intel’s Knights Hill Xeon Phi, with 7 PB of memory and a 13 MW power requirement, in 2018. It was part of the CORAL procurement, a joint buy with Oak Ridge and Livermore (FedScoop). Theta arrived first and is in the dataset as Theta, at 5.88 petaflops Rmax.

The design did not survive. By November 2017 the contract had been rewritten to target more than one exaflop with delivery in 2021, and Intel cancelled Knights Hill, saying it had “re-targeted investment in this program to focus on achieving exascale class computing” (TOP500 News). In March 2018 Argonne described the original machine as cancelled and replaced by a new “A21” design, with installation planned for late 2020 and acceptance in early 2022 (The Next Platform, 2018).

By March 2019 the cost was reported as $500 million for an exascale machine due in 2021, using Intel GPUs, Xeon Scalable CPUs, CXL and Optane persistent memory (The Register, 2019). Intel unveiled the GPU, Ponte Vecchio, at SC19 in November 2019, then planned on Intel’s 7 nm process (The Register, 2019). The shipped compute nodes use HBM-equipped Sapphire Rapids and Slingshot instead, and the Optane modules ended up in the storage servers.

The schedule slipped again. Wikipedia, citing the Department of Energy, records an October 2020 announcement of a further six-month delay that ended Aurora’s status as the first US exascale system (Wikipedia). By April 2023 The Register was attributing the remaining delay to the Sapphire Rapids CPU schedule (The Register, 2023). Blade installation was completed on 22 June 2023 (Intel).

The results then came in stages. The November 2023 TOP500 list had Aurora second at 585.34 petaflops, measured on about half the machine (TOP500, Nov 2023). On 14 May 2024 Argonne announced 1.012 exaflops using 87% of the nodes (TechXplore, Argonne release). Production and general availability began on 27 January 2025 (Aurora architecture paper, arXiv). Counting from the April 2015 award, that is 8 years and 7 months to the first list entry and almost 10 years to production.

What is inside

Each node has two Xeon CPU Max (Sapphire Rapids, 52 cores, 64 GB of on-package HBM2e each) and six Intel Data Center GPU Max 1550 (Ponte Vecchio) accelerators, plus eight HPE Cassini network interfaces at 200 Gb/s (arXiv). Across 10,624 nodes that is 21,248 CPUs and 63,744 GPUs in 166 racks (Intel). The paper gives 10.62 PB of DDR5 (which works out to about 1 TB per node by our division), 9.52 PB of HBM2e and 147 PB/s of aggregate HBM bandwidth.

The network is HPE Slingshot-11 in a one-dimensional dragonfly: 175 groups, of which 166 are compute, 8 storage and 1 service, built from 64-port Rosetta switches. Argonne counts 84,992 network endpoints (Argonne release, Jan 2025), which is exactly 10,624 nodes times eight NICs. Global bisection bandwidth is 0.69 PB/s.

Storage is unusual. The main tier is Intel DAOS, an object store with at least 31 TB/s peak bandwidth, on 1,024 dual-socket servers each holding 16 NVMe drives and 16 Optane persistent-memory modules. There is also a 100 PB Lustre system called Flare at about 650 GB/s (arXiv). The software stack is oneAPI, with SYCL, OpenMP, Kokkos and a HIP translation layer, on SUSE Linux Enterprise Server 15 (TOP500 system page; arXiv).

The paper reports 4 kW sustained per compute blade. The system occupies about 10,000 square feet, weighs 600 tons and uses 300 miles of network cable (Argonne release). The TOP500 record shows 38,698 kW for the Linpack run.

How it performs, and what it does

All figures below are from TOP500 or the Aurora paper unless noted.

HPL (FP64): 1,012.00 petaflops Rmax against 1,980.01 petaflops Rpeak, so about 51% of peak (our division). The run used 9,234 nodes (arXiv).
Efficiency: 1,012 petaflops over 38.7 MW is about 26 gigaflops per watt (our division). The November 2023 half-machine run was 585 petaflops at 24.7 MW, about 24 gigaflops per watt.
HPL-MxP (mixed precision, FP16 and FP32 with FP64 refinement): 10.6 exaflops at ISC 2024 per Argonne, 11.64 exaflops on 9,500 nodes at SC24 per the paper, ranked first on that benchmark both times.
HPCG: 5.61 petaflops on 4,096 nodes, a conjugate-gradient benchmark limited by memory bandwidth, not arithmetic.
IO500: first place at SC23 with 300 servers.

Aurora’s rank has drifted down as new machines arrived: second in November 2023 and June 2024, third in November 2024 and November 2025, and fourth in June 2026 according to its TOP500 page. We did not fetch the June 2026 list and do not name who passed it.

On workloads, the Early Science Program had 19 projects and the paper lists 34 production INCITE projects in the first year (arXiv). Argonne’s January 2025 release names AI protein design, battery materials, fusion, drug design, digital twins, and data analysis from the Advanced Photon Source and the Large Hadron Collider, and cites training large language models for science as a target (Argonne release). The paper also reports figure-of-merit improvement ratios of 89x for EXAALT, 84x for ExaSMR and 277x for ExaSky, three Exascale Computing Project applications. We did not retrieve the baselines those ratios are measured against.

The test system, Sunspot, has two racks and 128 nodes with the same per-node configuration. It has been on site since December 2022, and by April 2023 had served more than 180 researchers from over 20 teams. Argonne’s project director described it as “basically a miniature version of Aurora” (Argonne release, via EurekAlert). It is not on the TOP500.

Who supplies what

Intel was prime contractor, designed both processors, and supplies the DAOS software. HPE, which absorbed Cray in 2019, builds the Cray EX cabinets and Slingshot. The operator is Argonne. The accelerator vendor is Intel and the integrator is HPE, so the two roles sit with different companies. See the supply concentration analysis.

Intel’s GPU business and Aurora

Aurora ended up carrying nearly the whole Ponte Vecchio programme. In May 2024, Intel told partners Ponte Vecchio was “starting its sunsetting process”, would fill existing clusters and not pursue new ones, and pointed to Falcon Shores for 2025 (ServeTheHome). In January 2025, co-CEO Michelle Johnston Holthaus said Intel would “leverage Falcon Shores as an internal test chip” rather than sell it (TechCrunch). So the 63,744 GPUs in Aurora are the largest deployment of a line that has no external successor.

What does not add up, or is not public

Cost. Reported figures are $200 million (2015) and $500 million (2019). The Next Platform reports that Intel Federal took a $300 million write-off in 2021 and speculates that Argonne may effectively pay about $200 million net, while saying it is unsure. Neither Intel nor DOE has published a final price in anything we read.
Peak. The Next Platform estimated 2.01 exaflops and Intel said over 2; TOP500 lists 1.98. Wikipedia notes that an earlier claim of more than 2 exaflops peak “never was realized”, which is its wording, not a primary source.
HPL-MxP. 10.6 exaflops (ISC24 announcement) and 11.64 exaflops (SC24, paper) are different lists, not conflicting measurements.
Storage. The Next Platform and the dataset say 230 PB for DAOS. The paper gives 260 PB raw and about 220 PB usable after erasure coding. Intel’s June 2023 release states the capacity as “220 terabytes”, which looks like a unit error.
Cooling and facility. We found nothing on cooling-loop temperature, facility power capacity or an annual cost. The dataset records direct liquid cooling from the HPE cabinet but we could not retrieve the ALCF page that it cites.
Nodes unused for HPL. Why 1,390 nodes were left out of the 1.012 exaflop run is not stated in what we read.
ALCF pages. The main ALCF and Argonne news pages returned 403 to us, so several figures rest on mirrors or the paper.

What this dataset says

This section describes the dataset as published on 2 October 2026. The dataset changes as rows are added and corrected.

The announcement-to-first-light comparison covers 209 systems with both dates, at year resolution. Aurora is the top row at 8 years, ahead of Fugaku and Blue Waters at 6. Its systems row gives announced 2015, first operational 2023-12, Rmax 1,012 petaflops, Rpeak 1,980.01, 9,264,128 cores and 38,698 kW, with confidence “reported”. Its components rows give 63,744 GPUs and 21,248 CPUs, matching TOP500 and Intel. In the component rows, Aurora’s 63,744 Max 1550 GPUs make up 96.5% of the 66,024 recorded across four systems with a known GPU count: Dawn (1,024), SuperMUC-NG Phase 2 (960) and Clementina XXI (296), plus Stampede3, which has no count. That sum is our arithmetic, and it means the accelerator share and interconnect share pages depend heavily on this one machine for Intel and Slingshot-11 figures. The only list appearances recorded for Aurora are the November 2023 TOP500 (rank 2) and Green500 (rank 37), so the dataset does not yet record the full-system run, and the verification table marks the ALCF source unreachable (see data trust). Argonne’s fleet is also changing: Solstice and Equinox are planned Blackwell systems financed through a different model.

Sources

deep-diveexascalehpcacceleratorshistory