Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations

Summary

Meta built a system that aggregates fragmented idle inference capacity from a globally constrained fleet into a shared opportunistic pool to enhance ads recommendations. Idle capacity arises from four mismatches: peak provisioning with demand variation, model development timing with capacity delivery, regional resource ratios, and accelerator types. Two elasticity levers exploit this preemptible, volatile capacity: Rank More Ads increases candidates evaluated by ranking stages, using both CPU and accelerator surplus, and Serve Better Models routes more requests to a more powerful model, using accelerator-only surplus. The system protects baseline serving through isolated routing, minute-scale feedback loops that adjust candidate volume and model coverage, and millisecond-scale value-aware load shedding that removes lower-value work during sudden preemption. This allows optional work to expand and contract with opportunistic capacity, driving better ad relevance and advertiser outcomes without requiring additional capacity orders.

Takeaways

1. 1

   Idle inference capacity arises from four mismatches: peak provisioning vs demand variation, model development vs capacity delivery timing, regional resource-ratio mismatch, and accelerator-type mismatch.

2. 2

   Rank More Ads (RMA) increases candidate volume using both CPU and accelerator surplus, while Serve Better Models (SBM) increases model coverage using accelerator-only surplus, enabling use of a broader range of opportunistic capacity.

3. 3

   The system protects baseline serving with isolated routing targets, minute-scale feedback loops for work regulation, and millisecond-scale value-aware load shedding at servers and clients.

FAQ