Hugging FaceKyle WiggersProblem & solution
Ai2 replaced its priority-based GPU cluster scheduler with a system combining GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract to prioritize high-impact research while maintaining full occupancy. The previous priority scheduler induced GPU squatting, priority inflation, and constant negotiation over non-preemptable workloads, exemplifying a tragedy of the commons. In the new model, managers allocate proportional GPU time budgets to projects, making resource allocation a transparent administrative process rather than an operational debate. The scheduler tracks occupancy over a time window with weights set by budgets, and time-slicing enforces minimum runtime protection before workloads become preemptible. However, interactive sessions suffered from the 8-hour cap, prompting investment in a CPU-only cluster and restorable sessions, while potential capacity fragmentation for large workloads remains under investigation.
Ai2’s AI Infrastructure team manages thousands of NVIDIA H100, B200, and B300 GPUs for about 150 internal researchers running LLM and VLM training, robotics RL simulation, and post-training for scientific agentic use cases. Demand far exceeds supply, with outstanding requests for 2-3x more GPUs than available.
Replaced a priority-based scheduler with a system that uses GPU time budgets allocated by managers, hierarchical fair-share scheduling that tracks occupancy over a time window with weights set by budgets, and a time-slicing contract that enforces minimum runtime protection before workloads become preemptible.
GPU time budgets allow managers to allocate proportional shares to projects, making resource allocation a transparent administrative process rather than a case-by-case operational debate.
The hierarchical fair-share scheduler tracks occupancy over a time window using weights set by budgets, preventing GPU squatting by drawing from the user’s allocation.
Interactive sessions faced challenges with the 8-hour protected runtime cap, leading to plans for a CPU-only cluster and restorable sessions to preserve researcher state.
By requiring all GPU time requests to be funded by a budget; any workload, including a squatting no-op, consumes from the user's allocation, making it more expensive than honestly engaging in the budgeting debate.
Researchers could hold interactive sessions for up to a week in the old system, but with time-slicing they were subject to an 8-hour cap on protected runtime, after which sessions become preemptible, causing state loss and waiting time.
They conducted live explanatory sessions using real examples and introduced new visualizations showing allocation usage and the metric used for sorting the workload queue, which helped researchers understand prioritization decisions.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Buy Now, Pay Later is a deferred-settlement trade-off: merchants gain conversion while underwriting, fraud, and collections risk move elsewhere. This diagram maps the benefit, the cost, and the condition that decides.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
