A Firmware Rollout and Its Lessons
A firmware rollout I was running had been progressing smoothly, zone by zone, for six days until it abruptly halted. The page signals a change with a subtle vibration, but at 1:14 AM, it feels like a fire alarm. In one particular zone, the number of hosts that experienced an unclean reboot during the update exceeded the safety threshold, prompting the automation to halt the rollout and flag the issue for manual intervention.
Initially, the assumption was that we had introduced a bug. However, upon investigation, we realized that the hosts encountering issues had been stuck for hours before the update reached them, due to a known hardware hang. The reboot during the update simply brought this problem to the forefront. We adjusted our check to exclude hosts that were already unhealthy before the update, resumed the rollout, and successfully completed it nine days later.
This incident taught me a valuable lesson: When evaluating health signals, it's crucial to consider the host's state prior to the change. Otherwise, the release process might be blamed for issues that were already present.
This scenario highlights a layer of AI infrastructure that often goes unnoticed by users: the release process responsible for updating firmware, drivers, and run-times across thousands of machines while ensuring uninterrupted job execution.
As AI clusters scale, this release process becomes a critical determinant of their overall performance. Hardware failures are inevitable at this scale, and most fixes and mitigations are distributed via firmware and software updates. A fix for a known hang, for instance, only helps once it's deployed to all affected hosts, and the release process must ensure this deployment without causing further disruptions.
Hardware Failures and Their Impact
Meta's research on training Llama 3 offers insightful data on this matter. Over a 54-day pre-training period with approximately 16,000 H100 GPUs, they recorded 466 job interruptions, with 419 of them unexpected. About 78% of these interruptions were linked to confirmed or suspected hardware issues. Moreover, synchronized training caused significant fluctuations in power consumption, reaching tens of megawatts.
When considering a small failure rate for individual components, scaled across tens of thousands of accelerators, high-speed links, and racks drawing over 100 kilowatts each, the cumulative effect becomes a constant stream of incidents. This isn't a failure of engineering; it's a matter of basic arithmetic.
Measuring Success: Goodput
The cost of these failures is measured by goodput, the percentage of paid-for compute that effectively advances a model. Google Cloud uses this term to describe a similar concept. In a job that fails every three hours and checkpoints every 30 minutes, each failure discards approximately 15 minutes of progress on average. Adding the time for fault detection, node swapping, and restart, the job loses roughly a sixth of its time due to these interruptions.
Releases in Modern AI Infrastructure
In my role as a release engineer, I've witnessed a significant evolution in releases. Initially, releases involved pushing new code to servers with mostly static underlying infrastructure. Today, a release in our AI cluster involves managing multiple independently versioned components:
- Accelerator firmware
- Management controller firmware
- Network card software
- Drivers
- Run-times and diagnostics
Each of these components has its own release cycle, and they all need to work in harmony on every machine within a job. A typical release on my team updates most of these components simultaneously.
Deploying to a custom AI accelerator fleet is a transformative process, requiring a fundamental rethink of data center architecture. The cluster operates as a single disaggregated supercomputer, with chips connected via dense point-to-point fabrics, racks drawing substantial power, and ASICs rapidly shifting from idle to peak performance.