Engineers can optimize resource utilization by turning idle, preemptible capacity into value-generating work. Meta's approach shows how to build elastic ML systems that scale dynamically with available hardware while protecting baseline latency and reliability through isolation and load shedding.
Inference fleets face a capacity paradox: They can be globally constrained while meaningful pockets of capacity remain locally idle. At Meta, we built a system that aggregates these pockets into a shared opportunistic capacity pool and turns them into better ads recommendations. By putting otherwise idle capacity to work, the system helps people see more relevant ads and improves advertiser outcomes without requiring additional capacity orders.
Using this opportunistic capacity safely is difficult. It’s distributed across team-owned capacity allocations, regions, and accelerator types, and may be reclaimed with little notice. The ads inference system must therefore adapt workload as capacity supply changes without overloading the dedicated capacity that serves baseline workload.
We address this challenge with two complementary elasticity levers. Rank More Ads (RMA) adjusts the number of candidates evaluated by the ranking stages. Serve Better Models (SBM) adjusts the share of requests routed to a more powerful model.
Physical and logical isolation protects the capacity reserved for baseline serving. Separate feedback loops control RMA candidate volume and SBM model coverage, while value-aware load shedding at both servers and clients allows the system to degrade gracefully when capacity is suddenly reclaimed. Together, these mechanisms allow optional work to expand and contract with opportunistic capacity while protecting latency and reliability.
Inference fleets must maintain enough capacity to meet demand under strict latency and reliability targets. However, demand and capacity rarely align perfectly across time, regions, resource combinations, and accelerator types. Capacity can therefore remain idle in parts of a fleet even when the fleet is constrained overall. Common causes include:
These mismatches create an industry-wide inference capacity paradox: A fleet can be globally constrained while meaningful pockets of capacity remain locally idle. At Meta’s scale, these temporary pockets can collectively represent substantial capacity and create better ads recommendations when used safely.
Across a heterogeneous fleet at Meta’s scale, instantaneous idle inference capacity is inevitable — even with proactive planning designed to minimize structural mismatches. However, because this capacity is inherently tied to fluctuating baseline demands, it comes with distinct operational constraints — it’s preemptible (can be reclaimed at any time), highly volatile in its availability, and carries no long-term guarantees.
To successfully consume this volatile resource without compromising baseline performance, workloads must exhibit certain structural properties:
Based on these structural properties, we identified two complementary levers that meet these criteria to consume this idle capacity and generate large value upside.
For each user request, the ads recommendation system must select a small number of ads from millions of candidates in less than a second. The system uses a sequence of ranking stages in which increasingly powerful machine learning models evaluate progressively smaller candidate sets. The later stages only evaluate the top candidates from the previous stage.
RMA increases the number of candidates evaluated by the ranking stages when opportunistic capacity is available. Evaluating more candidates increases the chance of finding ads that are relevant to the user. Because the additional candidates pass through CPU-heavy upstream (e.g., retrieval, feature processing) as well as accelerator-heavy inference, RMA generally requires both CPU and accelerator surplus.
The ads evaluation can use models with different levels of complexity. More powerful models can improve prediction quality, but it’s not efficient to use them for every request with dedicated capacity.
SBM changes the share of requests evaluated by the more powerful models as accelerator capacity changes. It keeps the number of ads entering the ranking stages constant, so upstream CPU demand remains comparatively stable. This allows SBM to use accelerator-only surplus that RMA cannot consume.
The SBM model-router component uses request-value signals to prioritize the requests expected to benefit most from a more powerful model.
The two levers consume different combinations of resources. RMA requires both CPU and accelerator capacity, while SBM can use accelerator capacity without substantially increasing upstream CPU work. Together, they can use a broader range of opportunistic capacity than either lever alone.
The incremental impact of each lever also changes with traffic, current coverage, and the available resource mix. At a given moment, the next unit of opportunistic capacity may surface more relevant ads by ranking additional candidates to serve better ads or by routing more requests to a powerful model for improved prediction accuracy. We periodically adjust the allocation between RMA and SBM based on their measured performance and resource requirements.
Opportunistic capacity can be reclaimed faster than the control loops that regulate work across a large serving fleet. We therefore design the system around one requirement – losing opportunistic capacity may reduce incremental ranking work, but it must not overload the capacity that serves the baseline workload. We support this requirement through isolated routing targets, two workload-specific feedback loops, and fast local load shedding.
Optional work must not consume the capacity reserved for baseline serving. SBM and RMA enforce this separation differently.
For SBM, the isolation is physical. The baseline and more powerful models use separate routing targets. Dedicated replicas are sized to serve requests with the baseline model, while replicas on opportunistic capacity serve requests selected for the more powerful model. When opportunistic capacity contracts, the system reduces the share of requests routed to the more powerful model. Those requests use the baseline model instead, whose capacity is already provisioned for them.
For RMA, the isolation is logical. The system calculates how much additional candidate-ranking work can be supported by the model replicas deployed on opportunistic capacity and uses total capacity to set a global budget for RMA. Each model replica may receive a mixture of baseline and additional work; individual requests do not need to be physically separated. Baseline traffic is protected by limiting the total incremental work to the amount that these replicas can support.
When opportunistic capacity contracts, RMA reduces its global candidate-volume target and SBM reduces its more-powerful-model coverage. In both cases, the system removes optional work rather than allowing it to spill onto the capacity reserved for baseline serving. This keeps baseline traffic within its latency and reliability targets.
RMA uses a closed feedback loop composed of a central proportional-integral-derivative (PID) controller and client-side enforcement. Looking through a simplified lens, the controller uses observed tail latency to calculate a target for the number of ads sent to downstream ranking, and clients apply that target to new requests. When latency is below the target, the controller increases the number of ads ranked. When latency rises, it reduces the number. This allows RMA to use the latency signal, which is influenced by the capacity situation, to converge on a sustainable candidate volume as conditions change.
SBM uses a corresponding feedback loop with central control and client-side routing. The central controller sets the target coverage for the more powerful model, and the client-side model router decides which eligible requests use that model and which continue through the baseline model.
The router uses a request-value signal to group requests into segments based on predicted likelihood of user engagement with the ad. An explore-and-exploit process periodically measures the incremental outcome of using the more powerful model in each segment and updates the allocation between the two models. This is necessary because the incremental benefit of the more powerful model does not always align with the absolute value of a request. The process also preserves exploratory traffic across segments so the system continues to collect the observations needed to adapt, rather than reinforcing an outdated routing decision.
A calibration loop keeps the realized traffic share close to its target as the traffic mix changes across regions and over time. When capacity contracts, the central controller reduces the target coverage and the client routes fewer requests to the more powerful model. Requests removed from the opportunistic path continue through the baseline model.
The RMA and SBM feedback loops continuously align opportunistic work with available capacity, but their central components operate on the order of minutes. A sudden preemption can therefore remove capacity before clients receive new targets. The system adds value-aware protection for both workloads that can react immediately:
Server-side shedding: If the opportunistic pool loses capacity before the clients’ view has caught up, servers immediately remove lower-value opportunistic work. For RMA, they reduce the number of candidates processed within individual requests; SBM applies an equivalent value-aware shedding policy to requests routed to the more powerful model. This absorbs rapid changes close to the serving bottleneck.
Client-side shedding: The RMA and SBM clients use the latest controller targets as the guideline but also observe the server status directly. When it’s observed that the server is overloaded, clients shed lower-value opportunistic work directly, i.e., RMA requests fewer additional ads, while SBM routes fewer requests to the more powerful model.
Both forms of load shedding react within a few milliseconds, bridging the interval before the controller-client loops converge. Both server-side shedding and client-side shedding use value signals when selecting work to remove, which minimizes the effect during the transition.
We aim to use more opportunistic capacity productively by extending the operating range of RMA and SBM and broadening the set of elasticity levers. For RMA, this means supporting higher candidate volumes across ranking stages; for SBM, using more powerful models that turn additional accelerator capacity into better predictions. We are also developing new levers such as precomputation elasticity, which performs work ahead of request time and is therefore less constrained by the location and timing of live traffic.
Opportunistic capacity is fragmented across regions and accelerator types, and it’s useful only when compatible models and supporting resources are available in the same place. We are expanding model portability and cross-region inference capabilities so that more of the available capacity can be used efficiently.
The incremental impact of each lever changes with traffic, coverage, and the available resource mix. At a given moment, the next unit of opportunistic capacity may surface more relevant ads by ranking additional candidates or by routing more requests to a powerful model. We are building an optimization-based system to automatically adjust the allocation of AI inference capacity to its possible elasticity levers based on the benefits to users and advertisers. The goal is to maximize the return of the capacity subject to the constraints that each use case has in terms of regional constraints, hardware compatibility, and durability requirements for the idle capacity, among others.
This work reflects critical contributions from a cross functional v-team across Meta:
David Amsallem, Mohit Bharti, Maddie Digne, Stephen Dimmick, Maomao Ding, Amit Dutta, Sebin George, Aashik Gowda, Atul Goyal, Mohak Goyal, Nelson Guo, Tiger Hao, Yaning Huang, Dianshi Li, Yang Li, Bai Liu, Bofeng Liu, Wei Liu, Yisi Liu, Daniel Molina Hurtado, Maciej Naruszewicz, Arka Pal, Annan Pan, Brian Park, Brian Peschka, Michael Shao, Bikash Sharma, Alexander Sun, Weiman Sun, Kaiyue Tao, Patrick Taylor, Kevin Tong, Ellie Wen, Chao Xie, Lu Yang, Sihan You, Hansen Zhang, and Anthony Zhu
We also thank the following leaders for their sponsorship and support:
Neeraj Agrawal, Justin Charlap, Brian Ellis, Prabhakar Goyal, Hong Jiang, Shri Karandikar, Santanu Kolay, Uladzimir Pashkevich, Rob Schwartzberg, Eric Shan, Matt Steiner, Nathan Yan, Tak Yan, Caijie Zhang, and Alex Zhong
The post Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations appeared first on Engineering at Meta.
Continue reading on the original blog to support the author
Read full articleScaling recommendation systems to LLM-scale is often cost-prohibitive. Meta's approach demonstrates how co-designing hardware and software with intelligent request routing can break the inference trilemma, delivering high-performance AI at global scale with industry-leading efficiency.
Meta is shifting AI infrastructure from proprietary silos to open, Ethernet-based standards. These innovations in liquid cooling, petabit-scale fabrics, and standardized switch layers enable the massive scale required for next-gen AI models while fostering an open hardware ecosystem.
Prometheus showcases the hardware and software co-design needed to scale AI infrastructure to a million GPUs. It provides a blueprint for achieving high training 'goodput' and reliability at the gigawatt scale, essential for the next generation of foundational models.
Meta demonstrates how to apply LLM-scale architectures and scaling laws to recommendation systems. By unifying disparate models into a single foundation model and optimizing GPU utilization with 5D parallelism, they achieve massive performance gains while maintaining serving efficiency.
