---
title: "Model Serving"
description: "40 posts about Model Serving, summarised, each linking to the original."
---

# Model Serving
> 40 posts about Model Serving, summarised, each linking to the original.

## Articles

### [Databricks and NVIDIA: Building for the Agentic Era](https://yomu.fyi/post/databricks-and-nvidia-building-for-the-agentic-era.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Hanlin Tang, Tejas Sundaresan
- Published: Jun 17, 2026
- Type: Announcement

Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.


### [What’s new in Genie Code at Data + AI Summit 2026](https://yomu.fyi/post/what-s-new-in-genie-code-at-data-ai-summit-2026.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Julia Powell, Gal Oshri, Weston Hutchins
- Published: Jun 17, 2026
- Type: Announcement

At Data + AI Summit 2026, Databricks announced expansions to Genie Code for complex, agentic data and ML work. The changes include a full-page command center for managing concurrent threads and assets, upgrades across production ML engineering, and scheduled tasks that run prompts while users are away. For ML workflows, Genie Code uses Databricks production expertise and Genie Ontology, integrates with MLflow and Model Serving, and can move GPU jobs to AI Runtime while using workspace environment features. It can write features, coordinate edits, run and debug code, compare candidates, inspect endpoint health, and diagnose issues, with users deciding what to keep. Scheduled tasks are described as coming soon, creating reviewable threads from prompts and optional Databricks assets.


### [What’s New in the AI Platform: Agents for ML Engineering, Our Deep Learning Platform, and New Capabilities for Real-Time ML](https://yomu.fyi/post/what-s-new-in-the-ai-platform-agents-for-ml-engineering-our-deep-learn.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Tejas Sundaresan, Mike Del Balso
- Published: Jun 17, 2026
- Type: Announcement

The announcement presents three additions to the Databricks AI Platform: Genie Code support for ML engineering, AI Runtime’s serverless GPU environment, and expanded real-time ML capabilities. Genie Code integrates with Unity Catalog, Feature Store, training, serving, monitoring, and MLflow, assisting with feature engineering, model training, deployment, evaluation, and production operations. AI Runtime provides on-demand serverless NVIDIA A10 and H100 GPUs, supports high-performance multinode training with RDMA and high-performance data loading, and adds Lakeflow Jobs, DABs, MLflow, and Unity Catalog integration. For real-time ML, the platform adds declarative feature engineering, streaming features, online feature serving on Lakebase, and enhanced Model Serving targeting 300K+ QPS with under 10ms p99 latency overhead. Reported customer examples include faster workflows, lower infrastructure costs, and production scaling beyond 100K QPS.


### [Forward Deployed Engineering: Delivering Business Outcomes with AI](https://yomu.fyi/post/forward-deployed-engineering-delivering-business-outcomes-with-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jason Martin
- Published: Jun 11, 2026
- Type: Announcement

Databricks is formalizing its Forward Deployed Engineering (FDE) organization to address customers’ shift from migration and data-pipeline requests toward business outcomes with AI. FDE brings Professional Services together around an engineering-led model that embeds engineers with customers, supports modernization and production AI, and works with partners and Databricks R&D. In the cited examples, teams migrated five-plus petabytes of JPMC Consumer and Community Banking Risk data and more than 500 notebooks in four months, while Fox used Lakebase, AI Search, Databricks Apps, and Model Serving to redesign fan experiences. The organization says its engagements use shared OKRs, rapid prototype-to-production delivery, embedded engineering, outcome-aligned commercial options, and global partner coverage, with Fox reporting that Sports AI users spend approximately twice as long in the app.


### [How ERGO Hestia reduced time-to-market with Databricks Lakebase and Model Serving](https://yomu.fyi/post/how-ergo-hestia-reduced-time-to-market-with-databricks-lakebase-and-mo.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Klaudia Ratkowska, Maciej Majewski, Oliver Börner, Alexander Migunov
- Published: Jun 11, 2026
- Type: Problem & solution

ERGO Hestia redesigned its real-time pricing platform to reduce deployment friction across more than 100 models and 1,000 variables while preparing B2C capabilities. Previously, processed data moved from Databricks through extraction jobs, external Azure PostgreSQL, and a custom caching adapter, creating governance overhead, deployment coordination, and latency spikes during large refreshes. The new architecture uses Lakebase Sync Tables as an online serving layer and Databricks Model Serving Endpoints, keeping data, request logic, and model serving within the lakehouse; Unity Catalog supplies lineage, version tracking, access controls, and audit trails. An incremental migration started with a low-criticality endpoint, measuring 20ms latency and less than 5% CPU utilization at 40 requests per second, before expanding toward larger workloads and the planned decommissioning of PostgreSQL.


### [AI Serving Platform That Adapts to Your Model](https://yomu.fyi/post/ai-serving-platform-that-adapts-to-your-model.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Anshul Gupta
- Published: Jun 10, 2026
- Type: Problem & solution

Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability.


### [DiffusionGemma: The Developer Guide](https://yomu.fyi/post/diffusiongemma-the-developer-guide.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Ian Ballantyne, Omar Sanseviero
- Published: Jun 10, 2026
- Type: Tutorial

DiffusionGemma is an experimental text-generation model and developer guide covering serving, customization, and non-autoregressive generation. Built on the Gemma 4 backbone, its 26B Mixture of Experts architecture activates 3.8B parameters during inference, refines 256-token canvases in parallel, and shifts the bottleneck from memory bandwidth to compute. Block autoregressive diffusion commits denoised 256-token blocks to the KV cache, combining bidirectional denoising with causal prefill for longer sequences; vLLM integration provides an OpenAI-compatible local server. A Sudoku fine-tuning example using Hackable Diffusion reports improvement from roughly 0% base-model success to 80%, while SFT reduces the demonstrated solve from 48 to 12 steps through adaptive stopping. The guide also identifies bidirectional attention and re-noising as mechanisms for global constraint handling and self-correction.


### [3x Faster Search: Parallel Test-Time Scaling with Instructed-Retriever-1](https://yomu.fyi/post/3x-faster-search-parallel-test-time-scaling-with-instructed-retriever.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: Jun 4, 2026
- Type: Announcement

Instructed-Retriever-1 is introduced as the retrieval-specialized model behind an update to Agent Bricks Knowledge Assistant, targeting search quality and latency. It parallelizes query and filter generation for broader candidate recall, then applies multi-pivot groupwise reranking to improve precision, using one model for both retrieval stages. The model was trained on synthetic enterprise-style environments and evaluated on realistic Knowledge Assistant workloads, where it scored 81.0 nDCG@10 for reranking and matched Claude Sonnet 4.5 retrieval quality on KARLBench. Serving uses a Mixture-of-Experts architecture, FP8 quantization, speculative decoding, and infrastructure tuning; the update reports more than 3x faster search, 2x faster answer generation, and TTFT around two seconds. End-to-end latency was consistently below 10 seconds in the offline evaluation setup.


### [Bringing Gemma 4 12B to your Laptop: Unlocking Local, Agentic Workflows with Google AI Edge](https://yomu.fyi/post/bringing-gemma-4-12b-to-your-laptop-unlocking-local-agentic-workflows.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Google AI Edge Team
- Published: Jun 3, 2026
- Type: Announcement

Google DeepMind’s Gemma 4 12B is presented as an open model for running agentic, multimodal capabilities locally on everyday laptops through the Google AI Edge stack. The release covers Google AI Edge Gallery for local data analysis and coding, including Python generation, execution, chart rendering, and a reported 3D rendering workflow that can generate dependencies, code, and self-corrections in one turn. Google AI Edge Eloquent adds fully on-device dictation, transcription, and Voice Edit commands for rewriting or translating selected text, while LiteRT-LM’s new serve command exposes the model through a local, OpenAI-compatible endpoint for tools and frameworks. The post says Gemma 4 12B delivers a 60%+ overall quality improvement over prior models for Eloquent’s use case and keeps data on-device, with performance and memory benchmarks provided in the model card.


### [Gemma 4 12B: The Developer Guide](https://yomu.fyi/post/gemma-4-12b-the-developer-guide.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: André Susano Pinto, Andreas Steiner, Karolis Misiunas, Karsten Roth, Michael Tschannen, Omar Sanseviero
- Published: Jun 3, 2026
- Type: Announcement

Gemma 4 12B is a dense multimodal model with a unified, encoder-free architecture, native audio input, and a size intended for local execution on dedicated GPU laptops with 16GB VRAM or unified memory. Instead of separate vision and audio encoders, it uses a 35M-parameter vision embedder that projects raw 48x48 patches and a linear projection for raw 16 kHz audio sliced into 40ms frames; both feed a decoder-only transformer with shared weights. That design reduces multimodal latency and fragmented memory footprints, while allowing downstream adapters such as LoRA or full tuning to update the multimodal token loop in one pass. Capabilities include automatic speech recognition, agentic reasoning, diarization, video understanding, and coding, with examples covering a locally built image-processing app and five minutes of video processed at 1 FPS with audio; LiteRT-LM adds offline macOS apps, sandboxed Python execution, and an OpenAI-compatible local API server with stateless prefix caching.


### [Reliable LLM Inference at Scale](https://yomu.fyi/post/reliable-llm-inference-at-scale.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ying Chen, Wendy Hu, Ankit Mathur, Mike Eastham, Pei-Lun Liao, Wai Wu, Arjun DCunha
- Published: May 27, 2026
- Type: Explainer

Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP\_NUM\_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.


### [How Superhuman and Databricks built a 200K QPS inference platform together](https://yomu.fyi/post/how-superhuman-and-databricks-built-a-200k-qps-inference-platform-toge.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Myke Troianovskyi, Christoph Stüber, Wai Wu, Arjun DCunha, Amine El Helou, Tian Ouyang, Jarek Odzga, Alex Coleman
- Published: May 8, 2026
- Type: Problem & solution

Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.


### [Re-imagining ML Serving Infra: My Winter Internship at Ramp](https://yomu.fyi/post/re-imagining-ml-serving-infra-my-winter-internship-at-ramp.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Michael Jiang
- Published: Apr 10, 2026
- Type: Problem & solution

Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing.


### [Improving Composer through real-time RL](https://yomu.fyi/post/improving-composer-through-real-time-rl.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Jacob Jackson, Ben Trapani, Nathan Wang & Wanqi Zhu
- Published: Mar 26, 2026
- Type: Explainer

Composer's real-time RL program uses production inference tokens and user responses to improve a coding model amid 10–100x increases in inference volume. It serves checkpoints in production, turns interactions into reward signals, updates model weights, evaluates suites including CursorBench, and can deploy a new version behind Auto about every five hours. Using real users and environments removes simulated-user modeling uncertainty; keeping data fully or almost fully on-policy supports frequent updates, while Composer 1.5's A/B test showed 2.28% more edits persisting in the codebase, 3.13% fewer dissatisfied follow-ups, and 10.3% lower latency. Production feedback also exposed reward-hacking behaviors, including deliberately broken tool calls and excessive clarification, prompting negative examples and reward-function changes. The team is adapting the loop for longer, less frequent but higher-fidelity interactions and exploring specialization for organizations with distinct coding patterns.


### [Modernising Grab’s model serving platform with NVIDIA Triton Inference Server](https://yomu.fyi/post/modernising-grab-s-model-serving-platform-with-nvidia-triton-inference.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Daniel Tai
- Published: Oct 21, 2025
- Type: Problem & solution

Catwalk, Grab’s machine learning model serving platform, accumulated technical debt by maintaining separate inference engines for ONNX, PyTorch, TensorFlow, and vLLM, contributing to higher latency, lower throughput, and rising costs. The team evaluated NVIDIA Triton Inference Server and began consolidating those engines around it, initially adopting ONNX. The Triton manager combines a server manager, which retrieves and verifies models, prepares configurations, launches Triton, and monitors health, with a proxy that translates legacy API requests without requiring client code changes. Benchmarks showed at least five times the traffic capacity for a transformer model; rollout reached over 50% of online deployments in 10 days, while some larger models saw p90 latency fall from 120 ms to 20 ms.


### [Heroku AI Expands Model Offering with OpenAI’s gpt-oss-120b](https://yomu.fyi/post/heroku-ai-expands-model-offering-with-openai-s-gpt-oss-120b.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Anush DSouza
- Published: Aug 20, 2025
- Type: Announcement

Heroku has made OpenAI’s open-weight gpt-oss-120b available through Heroku Managed Inference and Agents, giving developers an OpenAI-compatible chat completions API for compatible SDKs and frameworks. The model uses a Mixture-of-Experts architecture with 117 billion total parameters, activating 5.1 billion per token and enabling operation on a single 80GB GPU; its Apache 2.0 license permits inspection and fine-tuning. It is designed for agentic workflows involving instruction following, function calling, web searches, and Python execution, while reported benchmarks place it near or above several open-weight models and sometimes OpenAI o4-mini. The add-on provides deployment, secure application access, agent actions, monitoring, and scaling without infrastructure management, with pricing of $0.15 per million input tokens and $0.60 per million output tokens.


### [From failure to success: The birth of GrabGPT, Grab’s internal ChatGPT](https://yomu.fyi/post/from-failure-to-success-the-birth-of-grabgpt-grab-s-internal-chatgpt.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenbo Wei
- Published: May 19, 2025
- Type: Problem & solution

In March 2023, an effort to reduce repetitive questions overwhelming Grab’s ML Platform support channels began as a chatbot for platform documentation and became GrabGPT, an internal ChatGPT-like tool. The initial chatbot used chatbot-ui and more than 20,000 words of Q&A documentation, but GPT-3.5-turbo’s 8,000-token limit forced summaries below 800 words; embedding search also performed poorly, so the approach was abandoned. The pivot extended existing frameworks, added Google login, and deployed a tool through Grab’s catwalk model-serving platform, later named GrabGPT. It reached 300 registered users on day one, more than 3,000 users and 600 daily active users by month three, and nearly all Grabbers according to the post, with private routing, global accessibility, multi-provider model support, and auditable interactions cited as reasons for adoption.


### [Editing Files at 1000 Tokens per Second](https://yomu.fyi/post/editing-files-at-1000-tokens-per-second.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Aman Sanger
- Published: May 14, 2024
- Type: Explainer

Large code edits expose coding agents to laziness, inaccuracy, and latency, especially when frontier models must rewrite hundreds of lines. The post presents fast apply, a specialized full-file code-edit model that separates planning from applying: Cursor's frontier model plans the change, while the apply model rewrites the current file. It evaluates roughly 450 edits on files under 400 lines, using Claude-3 Opus as grader, and finds llama-3-70b-ft nearly matches claude-3-opus-diff while outperforming GPT-4-Turbo and GPT-4o. Rather than generate conventional diffs, it uses full-file rewrites and search/replace hunks, which avoid line-number errors and give models more output tokens for reasoning. A custom speculative decoding method, deployed with Fireworks, reaches approximately 1,000 tokens per second on a 70b model, about 13x faster than vanilla Llama-3-70b inference and 9x faster than the previous GPT-4 deployment.


### [Inference characteristics of Llama](https://yomu.fyi/post/inference-characteristics-of-llama.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Aman Sanger
- Published: Jul 20, 2023
- Type: Benchmark

Llama-2-70B is evaluated against GPT-3.5-turbo on serving cost and latency, with the comparison deliberately excluding GPT-4 and noting fine-tuning as a separate reason to choose Llama. The evaluation serves Llama on two 80-GB A100 GPUs in 16-bit precision, then uses transformer FLOP, memory-bandwidth, and KV-cache analysis to explain the difference between prompt processing and token generation. The measurements indicate that Llama-2 has worse completion-token pricing than GPT-3.5 on two A100s, while its prompt-token pricing is more than three times cheaper; eight A100s might improve pricing but introduce unacceptable latency. The resulting guidance favors GPT-3.5 for most workloads and open-source Llama models for prompt-heavy tasks such as classification or reranking, with batch processing and workloads lacking prompt tokens identified as additional cases where Llama may make sense.


### [The Evolution of Kit: Automating Marketing Using Machine Learning](https://yomu.fyi/post/the-evolution-of-kit-automating-marketing-using-machine-learning.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 4, 2020
- Type: Explainer

Kit began as a heuristic, rules-based virtual employee that simplified Facebook advertising for Shopify merchants by selecting products and budgets while defaulting other settings to best practices. The post describes its evolution into an artificially intelligent assistant that uses merchant spending behavior and store signals such as traffic and sales to recommend marketing actions. Kit's machine-learning design combines regression to predict likely spending with classification to estimate whether a budget and store state are likely to generate sales; training uses feature engineering, Apache Spark, Google Cloud ML Engine, and monitored model metrics. For real-time predictions, Bigtable stores engineered features, while a Tornado web container delegates model computation to TensorFlow Serving. The resulting system powered one third of Kit's marketing campaigns.


[Newer posts](https://yomu.fyi/topic/model-serving.md)