This migration shows how to replace complex homegrown infrastructure with standardized Kubernetes-native tools like Kueue. It enables advanced batch features like preemption and fair sharing at scale while maintaining a transparent experience for developers.
By Alvin Bao, Alex Petrov, Jennifer Lai, Aidan Sherr, and Samartha Chandrashekar
As a part of the journey to transition Netflix’s compute infrastructure to be more Kubernetes-native, we have leaned into incorporating components from the Kubernetes ecosystem into our container platform Titus. One example of this is our use of Kueue, a cloud-native job queueing system for batch workloads, which has largely replaced the custom queuing and scheduling logic in our homegrown managed batch solution Compute Managed Batch (CMB). In this post, we’ll give an overview of what motivated the migration, how we migrated millions of batch jobs to use Kueue, and what Kueue allows us to offer as a Compute platform.
CMB is a managed batch solution that allows users and applications to execute and manage workloads that run to completion. Using a tenant hierarchy, workloads are managed and queued with ordered execution through priorities, and capacity is managed on a per-tenant basis. Workloads that are submitted to CMB are then run on Titus. The features of Titus relevant to CMB are workload federation across multiple cells (Kubernetes clusters) and federated capacity reservations. This means CMB can talk to a single Titus endpoint to get/submit workloads and update capacity reservations without having to worry about the underlying cell/cluster topology.
Tenants provide a grouping mechanism for jobs submitted on behalf of certain organizations, platforms, or applications. Users can create and organize tenants however best suits their organization or use case. For example, an organization may use a single tenant across several applications or a complex hierarchical structure that matches its team and application ownership structure.
Tenants are associated with a capacity configuration. The capacity configuration defines the amount of compute capacity available to the tenant and provides certain guarantees around isolation from other tenants. The capacity configuration contains weight (used for fair sharing) and resource dimensions.
There are two types of tenants in CMB:
With regards to capacity configuration, tenants can use 2 types of capacity:
Reserved Capacity
For internal tenants, if a user specifies reserved capacity, it is fair-shared across the subtree and usable by the leaf tenants under that internal tenant.
For leaf tenants, if a user specifies reserved capacity, it partitions capacity within the hierarchy so that other tenants cannot reserve the same resources. Those reserved resources are not shared with any other tenant, ensuring throughput for a given leaf tenant.
Shared Capacity
The Compute team maintains a global pool of shared capacity that any tenant can burst into, in addition to its reserved capacity. Reservations are not required to use CMB, so a tenant can run out of shared capacity entirely. The pool is fair-shared across tenants, but in CMB, this applied only at admission: CMB had no preemption, so once a job was admitted, it ran to completion regardless of shifts in fair-share demand.
Kueue changes the semantics for both types of capacity, which the fair sharing and preemption section covers.
Here is an example of what a tenant hierarchy looks like:
CMB was created in 2018, before or alongside many of the open-source batch compute offerings available today. Over the years, as the Kubernetes ecosystem has evolved, many of the features that CMB offered or strived to offer have been included in these open source projects e.g., fair sharing, hierarchical tenants, capacity management, priority queuing. In addition, it became increasingly cumbersome to develop new features such as preemption when CMB was so far removed from the underlying Kubernetes cluster.
The team took a look at what it would take to modernize our batch abstraction and settled on Kueue for the following reasons:
This initiative of migrating CMB workloads to Kueue became known as Netflix Batch. The key tenets of our migration were the following:
The key difference between the old and new flows is that we defer queuing and scheduling to Kueue, which is enabled in each Kueue-enabled Titus cell. Titus federation routes the job to Kueue cells using our custom Kueue router.
For us as operators, the migration was as simple as clicking a button on a tenant in our UI (as shown in the example above). This also allows us to easily rollback changes if there were issues.
Under the hood, this enrollment converts internal tenants to Cohorts and leaf tenants to a ClusterQueue + LocalQueue. The capacity configuration on a given tenant is converted into resource flavors and nominal quotas. The architecture for this looks as follows:
Kueue is fully rolled out in production, with it managing millions of batch workloads. In the future, we’re looking at options to enroll more of Titus batch workloads into this more managed experience. We have also productionized more fair sharing and preemptions to address better utilization of reserved capacity. In addition, our learnings are being leveraged by other internal teams, including those building Kubernetes-native training infrastructure, to inform their job scheduling and queuing configurations.
With Kueue, Preemption-based Fair Sharing allows Netflix Batch to maintain reservation semantics while lending resources to other tenants when those reservations are not in use. In addition, preemption allows Netflix Batch to preempt lower-priority workloads for higher-priority workloads. For our customers, this means that tenants can use more idle capacity from reservations, submit more jobs without the risk of starvation, and have quicker turnaround times for business-critical workloads.
An example preemption configuration on a ClusterQueue that we would be using is as follows:
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: "team-a-cq"
spec:
preemption:
reclaimWithinCohort: Any
withinClusterQueue: LowerPriority
With these features deployed, Compute has seen a significant increase in average resource utilization.
This work would not have been possible without the great work of the entire Compute team at Netflix.
How Netflix Simplified Batch Compute with Kueue was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Continue reading on the original blog to support the author
Read full articleThis article provides a blueprint for scaling LLM inference in production by integrating open-source engines like vLLM into existing microservice architectures. It highlights the trade-offs between raw performance and operational flexibility when managing high-scale GPU workloads.
Understanding service dependencies is critical for incident response. This architecture demonstrates how to process massive telemetry data in real-time using backpressure and local aggregation to provide actionable observability without overwhelming storage systems.
Data corruption can bypass traditional code-centric CI/CD pipelines. This approach treats data as code, using production traffic and chaos engineering to validate high-velocity metadata, ensuring streaming reliability by detecting corrupted states before they impact the global user base.
In complex microservices architectures, understanding dependencies is crucial for incident response. Netflix's real-time map reduces MTTR by replacing manual mental models with accurate, multi-layered insights into service relationships and blast radius.
