Eight of the 72 GPUs in each NVLink domain are deliberate hot spares; Meta did not say how much of the remaining idle capacity is dead hardware versus reserve. Without the spares, a single failed GPU would take out an entire rack and interrupt training. A failure still causes a shorter interruption, but a spare can slot in without waiting for a physical swap.

The spare pool is one of several measures behind a sharp reliability gain. When the first phase of Prometheus began, Meta saw roughly 1,000 interruptions per day for every 8,000 GPUs. An interruption does not necessarily mean a GPU died; it means something crashed or timed out, forcing the run to roll back to its last good checkpoint. Meta says it has since cut that rate by 99 percent, to about 11 per day per 8,000 GPUs.

Other tactics include isolating faults to a single data-parallel model replica so the rest of the cluster keeps training, and distributed checkpointing that asynchronously saves model state, gradients and optimizer data to multi-tier storage. Meta also had to tame the NVLink fabric itself: with over 600,000 connections and 72 GPUs per domain, up from eight on earlier H100 and H200 systems, even marginal signal degradation triggers frequent recovery cycles, so the company built continuous telemetry and ran L11 diagnostic burns to weed out weak links before production.

The results show up in goodput, a measure of useful throughput: Meta reports more than 85 percent for pretraining and over 90 percent for reinforcement learning on its latest model, Muse Wonder. Power is another live issue, as ramping tens of thousands of GPUs up and down creates spikes called transients that can overwhelm power supply equipment. The target now is just two interruptions per day per 8,000 GPUs by the time the cluster reaches a million accelerators.