Loading…
Topic
11 posts about MLOps. Every summary links to the original.
See the idea
AML systems detect financial crime by unifying fragmented data sources, applying machine-learning models, and routing suspicious activity through analyst review—turning raw transactions into actionable cases.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Databricks’ AI Research team uses AI Runtime (AIR) to manage hundreds of parallel research workstreams for products like Genie, Smart Routing, and AI Functions, training models from a few billion to nearly a trillion parameters. AIR performs preflight checks on GPU nodes to ensure health and communication before training, and monitors node health during execution to detect issues like NCCL failures or thermal throttling. The AIR CLI and custom Docker containers enable fast experimentation by launching jobs in seconds without rebuilding environments, supporting both supervised fine-tuning and reinforcement learning. MLflow integration automatically tracks experiments, while Unity Catalog FUSE mounting gives governed access to research data across workspaces without changing training code. The team has scaled to agent-led training, where AI agents propose changes, launch trials, and evaluate results, increasing training job volume by 8x across AI Functions research workloads.
Model versioning alone is insufficient for production AI because changes in inputs, preprocessing, prompts, retrieval, tool contracts, and serving settings can alter behavior independently. The post proposes a versioned release manifest that captures all interdependent components, including model revision, prompt revision, retrieval index revision, embedding revision, pipeline revision, and runtime revision covering token limits, batching, and timeouts. This manifest enables atomic release testing and rollback, with an evaluation gate that tests the full retrieval-generation path using deterministic checks and semantic rubrics on a versioned dataset. Practical steps include canary deployments that compare candidate and control signals, with explicit stop conditions and rollback procedures that restore compatible dependencies rather than just older model weights. The post concludes that the smallest useful implementation is a manifest, evaluation job, load test, release-aware traces, and a rehearsed rollback, and asks whether the on-call engineer can identify and restore a compatible known-good version.
Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests.
Tevogen Bio faced drug development costing upwards of $3 billion and requiring 10–12 years; selecting targets for its single-HLA-restricted SARS-COV2 product took 18–24 months of wet-lab work. Tevogen.AI partnered with Microsoft and Databricks to build and curate a multi-terabyte protein-sequence library and parallelize processing for machine-learning training. Databricks Data + AI Platform, Medallion Architecture, Unity Catalog, and distributed computing cut processing from 50 days to 24 hours under governed, fine-grained access controls. The dataset reached 24 million proteins, 16 billion datapoints, ~700 million unique peptides, and ~37 million cross-matched expert articles. An MLOps framework automated training, inference, monitoring, and retention; the alpha PredicTcell model used XGBoost and ESM models and reported 93–97% recall and 38–43% accuracy, while beta PredicTcell and alpha AdapTcell work began.
The guide explains how MLOps frameworks close the gap between notebook experimentation and reliable, scalable machine-learning production by applying automation, version control, and continuous delivery across the lifecycle. It describes experiment tracking, model registries, workflow orchestration, feature stores, deployment and serving, and monitoring as the capabilities needed to make training reproducible, prevent training-serving skew, automate pipelines, and detect degradation after release. MLflow is presented as a modular open-source option with Tracking, Model Registry, Models, and Projects, supporting run history, lifecycle stages, standardized packaging, and reproducible execution across environments. Kubeflow suits Kubernetes-standardized organizations and large distributed GPU workloads, while Metaflow, cloud-native services, and unified platforms such as Databricks are positioned as alternatives according to developer experience, cloud alignment, or operational preferences.
Machine learning (ML) is a subset of artificial intelligence that learns statistical patterns from data to make predictions or decisions without hard-coded rules. The guide distinguishes ML from generative AI, which creates new content, and from deep learning, a specialized ML approach using multilayered neural networks that learns features automatically and typically requires more data and computing power. It describes a workflow of preparing and splitting data, training models by comparing predictions with actual outcomes, and adjusting parameters to reduce error, illustrated with image classification and customer-churn scoring. It also covers supervised, unsupervised, and reinforcement learning, applications including computer vision and natural language processing, newer architectures such as transformers and Mamba, and MLOps practices for deployment, monitoring, and retraining.
The Big Book of AgentOps presents AgentOps as an operating discipline for building, deploying and improving production AI agents, combining architecture, evaluation, observability, governance, security and cost management. It addresses the operational risks of tool calls, enterprise-data retrieval, API actions, retries and multi-step execution, where quality, permissions and spend can become difficult to control. The six-chapter book covers four agent architectures, four deployment patterns, a seven-phase lifecycle, DevOps practices, operationalization planning and stakeholder alignment, with Unity Catalog, Unity Gateway and MLflow central to its Databricks-based deployment guidance. Its practices include building evaluation datasets from real traces, calibrating automated judges with subject-matter-expert feedback, designing tracing and access controls into systems, and using RACI matrices to clarify ownership. The announcement cites customer examples including FactSet’s 44% accuracy improvement, DXC Technology’s 30% total-cost-of-ownership reduction, and ICE’s 77% syntactic accuracy and 96% execution matches.
Swiggy’s pLTV effort addresses early-stage lifetime-value prediction for Food Delivery and Instamart, where sparse signals, zero-inflated outcomes, long tails, and trial-driven acquisition make conversion models unsuitable. It encodes 350+ pre-order signals across 12 groups in a simple multilayer perceptron with three shared hidden layers, task-specific heads, a custom MAE loss, and Adam optimization. A four-head variant added order-count predictions for each vertical, producing marginally better performance with 135K rather than 363K parameters. Evaluation emphasized ranking through Spearman correlation and decile diagonal coverage, reaching above 0.75 Spearman and 70–80% diagonal coverage. In production bidding experiments, the in-house model delivered better retention and higher gross order value per acquired user at no incremental platform cost, while a third-party platform acquired more users per dollar.
This guide surveys practical machine learning applications across industries, from medical imaging and fraud detection to demand forecasting, document processing, and customer-service automation. It defines supervised, unsupervised, semi-supervised, and reinforcement learning, then relates technique selection to the business question, data type, and availability of labels. Technical coverage includes convolutional neural networks for image analysis, transformers and large language models for generative AI, and time-series workflows using cleaning, lag features, and forecasting. The operational guidance addresses feature stores, experiment tracking, CI/CD, drift monitoring, retraining, cost optimization, fairness, privacy, explainability, and model risk management. Templates and checklists structure projects around business problems, data sources, metrics, architecture, measured outcomes, validation, governance, and escalation, while cited Databricks case studies and tools provide implementation examples.
The guide compares DevOps and MLOps, framing both as practices for reliable production systems but distinguishing traditional application code from machine-learning systems involving data and models. DevOps uses source control, automated testing, staging, and CI/CD, while MLOps governs the “holy trinity” of code, data, and models through separate versioning, quality checks, lineage, and access controls. Its central operational difference is Continuous Training: MLOps pipelines can validate model quality, monitor drift, and trigger retraining when data distributions shift, alongside testing model-serving infrastructure and promoting models through a registry. The guide also assigns responsibilities across data scientists, ML engineers, data engineers, and IT operations, and recommends dedicated MLOps for business-critical, frequently retrained models, DevOps for applications without ML, and split pipelines for hybrid products.
The guide presents machine learning delivery as a lifecycle problem, arguing that failures arise more often from planning, scoping, communication, and expectations than from algorithmic limitations. It recommends assessing data readiness before development by inventorying sources, checking quality and coverage, planning labeling, and engineering features. Once a measurable business outcome is defined, teams should match the learning paradigm and model complexity to the problem, establish cross-validation and holdout or temporal validation, and estimate compute and storage early. Baselines, reproducible experiments, deployment monitoring, model versioning, CI/CD, and retraining workflows support the transition from prototype to production. The guide also calls for representative proof of concepts and A/B testing or control groups to attribute business impact before scaling.
The best engineering reads, once a week.
