GPU failures are inevitable at scale, especially in distributed training where one bad node halts the entire job. This post details how to build self-healing infrastructure that detects hardware degradation early and automates remediation to maximize compute efficiency and model availability.
Authors: Nick Rutigliano, Argha C, Samartha Chandrashekar
How we detect failing GPUs and network fabric, remediate machines automatically, and let long-running jobs survive hardware loss.
Open Netflix, and the home screen is assembled for you: which rows appear, the order they appear in, the titles within them, and the artwork for each title are chosen by machine learning models trained on how hundreds of millions of members watch. Personalization is one of the most widely used things we build, and members feel it every time they open the app. Like a growing share of Netflix, it runs on GPUs, which train those models and increasingly serve them live.
Those GPUs have to stay healthy for that experience to hold up, and keeping them healthy is harder than it sounds. A single large training job can run for weeks across hundreds of GPUs spread over many machines. It runs as a gang: the ranks move in lockstep, exchanging data every step, so the job advances only as fast as its slowest participant and stops entirely if one participant dies. Every GPU, every high-speed network adapter, and every interconnect must keep working throughout the run. At the scale at which modern machine learning runs, individual GPUs and network links can fail during a run, so the underlying platform must be prepared for this and handle it without a person in the loop.
This post describes how we keep our GPU infrastructure healthy for training and inference. We cover how we detect failing hardware before and during jobs, how we automatically remediate a bad machine, and how we let a long job survive the loss of a machine. These training workloads run on our modern Kubernetes-native Compute platform.
Netflix aims to entertain the world, and machine learning has been part of how we do that for years, from the recommendations that help members find something to watch to the systems that help produce and promote titles. As generative AI advances, we are investing in it on both sides of the business: for members, through better recommendations and higher-quality promotional assets; and for storytelling, through creative tools for our production partners.
All of this runs on GPUs, for both training and inference, and GPU capacity is among the fastest-growing and most expensive parts of our infrastructure. That makes reliability a business concern on two fronts. — -
Inference is a growing share of that GPU footprint as more of these AI features reach members, so the reliability and efficiency that matter for multi-week training runs increasingly matter for always-on serving too.
We start from the baseline provided by the cloud provider. Cloud providers run instance status checks on virtual machines: a system status check for the underlying host’s power, network, and hardware reachability; an instance status check to verify that the operating system is up and reachable; and a check on attached storage. It emits scheduled maintenance events when it plans to retire or reboot a host, and it can automatically recover an instance that becomes impaired. We use all of these, and the maintenance events feed one of our own checks. They catch a machine that is down, unreachable, or slated for maintenance, and they stop at the instance level. They confirm the box is up and reachable. They do not verify that each GPU, its on-chip memory, its interconnect, and its network fabric are healthy enough to support distributed training.
The next step most teams take is a GPU-level check to confirm that the driver responds and that the GPUs are visible. That catches a dead or missing GPU, but misses the cases that take the most time.
Because a job runs as a gang, any one of these on any one machine can degrade or stop the whole job, and none of them shows up in an instance-level check or a driver-responds check. The health system has to assume the hardware can lie and the tools that inspect it can break.
We treat reliability as four layers, each catching what the layer below cannot:
1/ Node and hardware health: find bad GPUs, network adapters, interconnects, memory errors, and thermal issues, then taint or recycle the machine.
2/ Scheduler and platform: retry at the task & job level, validate a machine before it takes work, and drain & recycle machines safely.
3/ Training loop: checkpoint and use elastic workers so a job survives losing a machine.
4/ Observability: metrics, service-level objectives, and failure attribution across the other three.
The first two layers keep hardware problems away from jobs. The third accepts that some loss is unavoidable and makes it cheap. The fourth tells us whether the first three are working.
Checks run in two tiers. Device-level checks look at each GPU. A node-level agent, built on the open-source Kubernetes node-problem-detector, runs on every GPU machine every 30 seconds and turns check results into conditions on the Kubernetes node, the same mechanism Kubernetes uses to mark a node NotReady. Controllers watch those conditions and decide whether the machine can take on work.
GPU device checks are built on NVIDIA’s DCGM for driver-reported error codes, uncorrectable memory errors, interconnect integrity, and thermal state, and networking adapter checks inspect the host state for the count and link state of the Elastic Fabric Adapter interfaces that carry cross-machine traffic. A few of these checks deserve detail.
When a check fails, the machine’s node condition changes, and controllers take over. We distinguish two levels. A problem condition blocks new work from landing while jobs are running, because pulling a job off a machine is disruptive, and many problems clear on their own. If the condition persists beyond a tolerance window, it escalates to a failure condition, which evacuates the machine, drains its tasks (overriding the normal disruption limits to move quickly), terminates it, and replaces it.
Karpenter node auto-repair is used on our compute stack. We have customized it with a policy and a controller that maps each node condition to the duration we tolerate it before acting, so a fast-fatal error is actioned within minutes. At the same time, a flaky signal gets a longer window, and a cap limits how much of a cluster we will disrupt at once, so a bad rollout of the checks themselves cannot cascade into tearing down a whole cluster. We plan to upstream this feature to all Karpenter users.
Not every fault requires a full replacement, and replacement is the slowest way to restore capacity. We match the action to the fault. Killing the offending job clears a software wedge. Resetting a single GPU clears many transient device faults. Rebooting the machine clears the firmware state that has drifted over a long uptime. Terminating and replacing is reserved for hardware that fails a test we trust. For that last case, we record the GPU’s immutable hardware identifier, which stays with the physical device across reboots and relaunches even when the instance changes, report it to the cloud provider, and keep our own list, so a device that fails repeatedly does not quietly rejoin the pool under a new instance.
One subtlety sits between detection and scheduling. A freshly checked machine was handed to the scheduler just as its checks were still settling, so a job could land on it a moment before a slow check flagged it. We now hold capacity until the checks have run and settled, which closed that window.
A machine that just booted is not trusted until it proves itself. Before we release its capacity, it passes system checks, confirms the GPUs and their topology are present, and runs a round of GPU diagnostics. We then wait for the health checks to run twice, because the longer checks, such as counting network adapters, take more than one cycle to fire, and releasing after a single pass would let work land before those checks are reported. Only then do we remove the startup taint. On teardown, we run deeper diagnostics and record the GPU identifiers, which feed the same bad-hardware list, so hardware that failed once does not silently cycle back into service.
Even with all of the above, some machines are lost mid-job, so we make that cheap. Jobs checkpoint their state, and on a lost machine, they resume from the last healthy checkpoint rather than restarting from the beginning of a multi-week run. Workers are elastic and retryable, and the scheduler automatically retries platform-caused failures.
Attribution is what makes automatic retries safe. If we retried every failure, we would mask genuine bugs in a job and burn GPU hours looping on an error the platform cannot fix, so we classify each failure as platform or application before retrying. One example of why this matters: a class of timeouts in NCCL, the collective communication library GPUs use to exchange data, appeared to be a network fault and was traced to data-loading stalls at high worker counts. The GPUs and the network were fine, and the fix was in the data pipeline. Treated as hardware failures, those timeouts would have cycled healthy machines without fixing the cause.
We track GPU reliability along three service-level objectives, each split by cause, so a miss points at the layer responsible:
Each objective is split into hardware, driver, platform, and user causes. Hence, an availability dip from bad hardware reads differently from one caused by a scheduler change or a user misconfiguration. We optimize for goodput, the useful work the fleet completes, because a job held back by one slow GPU can look healthy on an uptime chart while getting almost nothing done. Specific targets for each objective are provided here for illustrative purposes.
These pieces form a loop. Hardware is validated at allocation, before a job starts, with a short collective communication run and pairwise network tests. Hence, any machine that fails within the first minute fails before the job is admitted. During the job, we watch passively for driver errors, memory errors, thermal events, and more. We view each job as a distributed system, which lets us identify a single slow rank dragging the rest, rather than only whole-machine failures. When something is wrong, remediation is automatic and graded. The loop is the same whether a job runs through our batch scheduler, Kueue, or Ray.
Not every workload needs a whole GPU, and leaving small inference jobs on dedicated GPUs wastes capacity. We partition GPUs with MIG for memory-level isolation between tenants, so a fault or a memory hog in one partition does not affect another. We also reconfigure those partitions without restarting the machine, so that a pool can shift between many small slices and a few large ones as demand changes. GPU and network adapter allocation is device-plugin-based today, to move to Kubernetes Dynamic Resource Allocation once it is supported in Karpenter. This matters because a GPU talking to a network adapter or working with a CPU on the wrong PCIE root of the machine incurs a bandwidth penalty.
Device-level allocation also changes how we handle a single bad GPU. With DRA support, instead of losing the whole machine, we could isolate the one faulty device and keep the others serving, which matters most for inference, where many small jobs share a machine. And because allocation is expressive, a job can request the capabilities it needs, such as memory or tensor-core support, and the platform matches a suitable GPU type, so users are not pinned to a model they had to choose in advance.
Moving from basic checks and manual draining to layered checks and automatic remediation changed two things users feel. More of the GPU fleet is available for scheduling at any time because bad machines are caught and cycled quickly, rather than sitting in the pool or waiting on an on-call engineer. And fewer jobs fail due to platform issues because most faults are caught before a job lands, and the rest are automatically remediated and retried. Graded remediation also returns capacity in minutes for faults that only require a job kill, a GPU reset, or a reboot, rather than the longer wait for a replacement machine.
Since the above improvements have landed, we’ve seen:
We are extending active checks to cover additional failure modes under load, widening burn-in so more classes of failures can be cleared before a user encounters them, and making remediation more granular so more error types are cleared in place rather than requiring a full node replacement loop for certain Xid errors. Additionally, we want to do more periodic diagnostics between jobs.
We are working to make placement more topology-aware using Kubernetes Dynamic Resource Allocation (DRA) once it lands in Karpenter, for example, by aligning on Compute/IMEX Domains, PCIe roots, and NUMA nodes.
More on the performance side for multi-GPU training, we plan to run a custom Linux scheduler built on sched_ext. Firstly, we can prioritize processes during collective coordination, such as when GPUs exchange results before the next round of compute, thereby reducing overall GPU idle time by establishing a fleet-wide priority for these tasks. The effect is small on one machine but grows across hundreds of machines, where a few percent of throughput amounts to a large amount of capacity. Secondly, we can prioritize data-loading processes to minimize the bottleneck that sometimes occurs when data is not loaded onto GPUs quickly enough to maintain steady progress.
While we already monitor throttling events, GPUs do not have to report throttling for it to hurt a job. One running ten to fifteen degrees hotter than its peers, even within healthy limits, sets the pace for the gang, so we can treat a large temperature gap as a fault, stop the job, and move it to cooler GPUs.
Additionally, we plan to expand our NCCL RAS (reliability, availability, serviceability) monitoring to identify NCCL issues sooner, such as mismatched collectives or hangs, to more quickly resume work.
Hardware fails, and at this scale, it fails often enough that the platform has to treat failure as normal. Detecting problems before and during jobs, automatically remediating machines, and allowing long jobs to resume after a loss lets teams run large-scale training and inference without babysitting the hardware. If you like these kinds of problems, we are hiring.
Reliable GPU compute for large training & inference jobs was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Continue reading on the original blog to support the author
Read full articleThis article provides a blueprint for scaling LLM inference in production by integrating open-source engines like vLLM into existing microservice architectures. It highlights the trade-offs between raw performance and operational flexibility when managing high-scale GPU workloads.
This article demonstrates how to build a scalable ML platform that decouples model innovation from client applications. It provides a blueprint for managing complex routing, A/B testing, and high-throughput inference (1M+ RPS) in a distributed microservices environment.
Automating failure diagnosis at scale reduces engineering toil and compute waste. By wrapping LLMs in deterministic constraints like line-number extraction and strict timeouts, Netflix achieves 100% error coverage while maintaining the reliability needed for critical data pipelines.
Standard container isolation fails for high-throughput workloads due to hardware-level contention. This approach shows how to offer VM-level performance guarantees without losing the developer velocity and managed experience of a container platform.
