The production platform for open-weight AI inference

Summary

The update presents a production inference platform for open-weight, licensed closed-weight, and fine-tuned models, aimed at giving teams control over performance, cost, quality, and functionality without building an inference stack from scratch. It supports uploading weights or adapters from Hugging Face, S3, or local machines, uses tested deployment profiles for hardware, quantization, tensor parallelism, speculative decoding, and serving engine choices, and lets operators tune placement and autoscaling signals. Endpoints can keep a stable interface while multiple deployments receive traffic through canary, blue-green, rolling, A/B, or shadow testing, with threshold-based automatic rollback. The rebuilt caching and distribution layer proactively warms shared weights; internal testing reports roughly 4× faster warm starts across several frontier models, while a closed beta adds full-weight and LoRA reinforcement learning and supervised fine-tuning with deployable checkpoints.

Takeaways

1. 1

   Deployment profiles package tested choices for hardware, quantization, tensor parallelism, speculative decoding, and serving engines, while allowing more manual control when workloads require it.

2. 2

   Autoscaling can use inflight requests, GPU utilization, time to first token, latency, decoding speed, or throughput instead of relying on one generic metric.

3. 3

   Internal testing found roughly 4× faster warm starts across several frontier models after rebuilding the model caching and distribution layer.

FAQ