Celery Beat: Where Should the Schedule Live?
Compare Celery Beat scheduler backends—file, SQL, and Redis/RedBeat—by where the active schedule lives, how updates propagate, and what failure modes each introduces.
NVIDIAElizabeth GoodmanAnnouncement
NVIDIA Mission Control 2.5 extends its autonomous recovery engine, previously limited to Slurm-based HPC, to Kubernetes workloads managed via NVIDIA Run:ai for AI factories, coordinating three separate services: the scheduler for workload placement, the job recovery service for failure-context decisions, and the hardware recovery service for node isolation and diagnostics. When a hardware fault occurs, job recovery restarts the workload on healthy nodes while hardware recovery concurrently isolates the suspect GPU. For transient software errors, only job restart occurs without hardware quarantine. Early degradation signals cause proactive node removal before job scheduling. This consistent recovery model across Kubernetes and Slurm reduces MTTR and improves GPU utilization by shifting from reactive manual intervention to proactive self-healing, benefiting both platform teams and AI practitioners.
Large-scale AI training on thousands of GPUs over days or weeks demands proactive monitoring and automation. Traditional reactive approaches struggle with hardware anomalies, software errors, and resource contention at that scale. Previously, NVIDIA Mission Control’s autonomous recovery engine only supported Slurm-based HPC. Mission Control 2.5 extends this capability to Kubernetes workloads orchestrated by NVIDIA Run:ai, providing a self-healing framework for AI factories.
The engine integrates three coordinated components: the Kubernetes scheduler (NVIDIA Run:ai) for workload placement, the autonomous job recovery service for context-aware restart decisions, and the autonomous hardware recovery service for node isolation and diagnostics. It handles three failure paths: hardware faults trigger parallel job restart and hardware isolation; transient software errors retry without hardware action; early degradation proactively removes nodes before job placement.
Mission Control 2.5 extends autonomous recovery to Kubernetes via NVIDIA Run:ai, supporting both Kubernetes and Slurm with a consistent recovery model.
Separating job recovery from hardware recovery allows workloads to restart without waiting for root-cause analysis, reducing downtime.
The system distinguishes hardware faults, transient software errors, and early degradation, applying appropriate recovery to minimize unnecessary hardware quarantine.
It supports Slurm-based HPC and Kubernetes workloads via NVIDIA Run:ai. The engine works with both schedulers using the same three-layer control loop, providing a consistent recovery model across environments.
Job recovery restarts the workload on healthy nodes while hardware recovery concurrently isolates and diagnoses the suspect GPU. This parallel approach reduces downtime compared to waiting for root-cause analysis.
Only job restart occurs without hardware quarantine. The hardware recovery service stays in background monitoring unless new evidence points to a real hardware issue, preserving scheduling pool size.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Compare Celery Beat scheduler backends—file, SQL, and Redis/RedBeat—by where the active schedule lives, how updates propagate, and what failure modes each introduces.
Autonomous Recovery is a closed loop that senses degradation, classifies a fault as transient or hardware, recovers workloads and quarantines hardware in parallel, then verifies against an RTO.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.