Loading…
Topic
MLflow
38 posts about MLflow. Every summary links to the original.
- 38 posts
- 3 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How FactSet cut model training and deployment from days to hours with Databricks AI Runtime
FactSet’s Text-to-Formula (TTF) system converts natural-language investment queries from professionals into correct FactSet Query Language (FQL) formulas. Initially built on retrieval-augmented generation (RAG), the approach hit a quality ceiling as queries became more specialized and formulas more complex. FactSet adopted fine-tuned open models within a Compound AI architecture, reducing end-to-end latency by about 60 percent and improving quality for complex formulas. Fine-tuning runs on Databricks AI Runtime with serverless GPU infrastructure, where engineers choose compute based on prompt length and output needs, and the pipeline integrates with MLflow, the Lakehouse, and Lakeflow Jobs for a unified workflow. This setup cut model setup, training, and deployment from days to hours, enabling the team to retrain every two to three weeks as new data and models become available.
From post-training to production: How TinyFish runs its web-native model on Databricks AI Runtime
TinyFish builds infrastructure for enterprises that need agents to work across the live web and is developing its own specialized web-native model. The company uses Databricks AI Runtime to manage GPU compute for post-training with supervised fine-tuning and moving toward Direct Preference Optimization, MLflow for evaluation, and Model Serving for production deployment. TinyFish evaluates its model by measuring full agent task completion rather than individual responses, achieving a 77.6% success rate on its internal go/no-go benchmark where 1,691 of 2,178 tasks were completed successfully. Over a 15-day period, the web-native model handled 294,781 agent runs across 6.4 million tool steps at an estimated serving cost of $0.088 per run or $0.0040 per tool step. By consolidating training, evaluation, serving, and operational data on Databricks, TinyFish reduces infrastructure overhead and maintains a continuous development cycle from agent behavior to deployment.
RADAR: Catch gray failures with anomaly detection
Gray failures are partial customer-facing problems that can leave health checks green while a subset of users quietly fails. RADAR (Reliability Anomaly Detection, Alerting, and Root-cause analysis) addresses this gap by collecting error counts and distinct affected users, segmented by error code and region, then applying the unsupervised streaming SPOT model. SPOT learns normal behavior from the previous 14 days using a single risk parameter, while downstream stages enrich, filter, deduplicate, and route alerts before providing anomaly details and AI/BI Genie-backed dashboards for investigation. Running RADAR internally reduced incident-discovery time by 95% with over 90% precision and no human needed to spot the pattern. The post also describes a Databricks scaffold that maps metrics, detection, alerting, and root-cause analysis to platform components and can be assembled by an AI agent from a metric, scaffold, and prompt.
What did AI cost you this quarter?
AI infrastructure costs are often billed to a platform team as one lump sum, leaving Finance without a departmental breakdown and teams without feedback on their usage. The post presents a working demo on Red Hat OpenShift AI that attributes requests and token consumption through department-specific MaaS API keys, shared model servers, Prometheus metrics, Perses dashboards, and MLflow traces. In its example, three departments generate 79,420 inference requests and consume 634 million tokens against a $14,200 quarterly bill, while Marketing’s 32% GPU utilization reveals roughly $34,000 in annual waste. The system combines aggregate usage filters with request-level details such as prompts, token counts, latency, and calculated cost, and the post argues that showback helps identify inefficient allocations and model mismatches rather than police teams.
Building a knowledge assistant over code
Building a knowledge assistant over code requires more careful chunking than indexing ordinary business documents, because code has nested structure, cross-file dependencies, and multiple file formats. The comparison built three Databricks Knowledge Assistants over the Casper’s Kitchens repository: fixed-size character chunks, LangChain language-aware splitting, and AST-based splitting with structure and metadata headers. The AST approach used ASTChunk to preserve units such as functions, classes, or statement blocks and prepend file paths plus class/function hierarchies to chunks. MLflow GenAI evaluation ran 46 questions with retrieval sufficiency, groundedness, and custom answer-correctness judges; AST chunking achieved 89% retrieval sufficiency and 70% fully correct answers, versus 85% and 59% for the naive baseline. The advantage was strongest for disambiguating structurally similar code, while value lookups and complete definition retrieval were comparable across methods.
Building an A/B testing analysis framework for mobile gaming on Databricks
HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.
coSTAR: How we ship AI agents at Databricks fast, without breaking things
Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution.
MLOps Frameworks: A Complete Guide to Tools and Platforms for Production ML
The guide explains how MLOps frameworks close the gap between notebook experimentation and reliable, scalable machine-learning production by applying automation, version control, and continuous delivery across the lifecycle. It describes experiment tracking, model registries, workflow orchestration, feature stores, deployment and serving, and monitoring as the capabilities needed to make training reproducible, prevent training-serving skew, automate pipelines, and detect degradation after release. MLflow is presented as a modular open-source option with Tracking, Model Registry, Models, and Projects, supporting run history, lifecycle stages, standardized packaging, and reproducible execution across environments. Kubeflow suits Kubernetes-standardized organizations and large distributed GPU workloads, while Metaflow, cloud-native services, and unified platforms such as Databricks are positioned as alternatives according to developer experience, cloud alignment, or operational preferences.
How Addepar scales investment workflows with Databricks AI agents
Addepar, a global technology and data platform, manages nearly $9 trillion in assets and moved from older systems to a unified data intelligence platform on Databricks running on AWS. It ingests hundreds of disparate data feeds, standardizes and enriches them, and delivers results through products, APIs and data sharing. Since adoption, pipeline costs fell 60% versus legacy infrastructure, saving more than $2 million, while new pipelines and integrations became five times faster. Addison, a native AI experience embedded in the platform, combines governed portfolio Q&A, proactive insights and action-oriented workflows, using live data and Addepar’s core calculations engine. Addepar is adopting Databricks Agent Bricks, including Supervisor Agent flows with Genie-powered analytics, while Foundation Model Serving and managed MLflow support governed model serving, traceability, evaluation and iteration for human-supervised agentic workflows.
AiChemy: Next
AiChemy is a multi-agent assistant for cross-disciplinary research that combines external MCP servers including OpenTargets, PubChem, and PubMed with Databricks-managed chemical libraries. It can identify disease targets and drug candidates, retrieve chemical and pharmacokinetic properties, assess safety and toxicity, and attach findings to verifiable data sources. Its guided workflow covers disease-to-target-to-drug discovery and literature validation, while a vector search over a 250,000-molecule ZINC15 index uses 1024-bit ECFP embeddings to find compounds similar to Elacestrant. To build a custom supervisor, users connect public MCP servers, transform structured drug data into a Genie space, and index unstructured chemical data for similarity search. They can assemble it with no-code Agent Bricks or deploy a Langgraph supervisor with Lakebase, MLflow AgentServer, and a React interface; invocations are logged and traced through MLflow using OpenTelemetry standards.
A practical approach to end-to-end Solvency II reporting in Databricks
Solvency II reporting connects data, actuarial reserving, capital calculation, quantitative reporting templates (QRTs), the ORSA, approvals, and disclosure, but fragmented systems make readiness and cross-process questions difficult to manage. The described Databricks implementation provides a governed workflow and control tower around existing actuarial and capital suites, automating ingestion checks, data-quality review, reconciliation, orchestration, reporting, and audit trails. It uses MLflow-managed reserving models alongside engines such as Prophet, RAFM, and Igloo, while narrow-scope AI agents can investigate QRT mismatches, suggest remediation, route questions, and support human review. An LLM can draft an ORSA from current numbers, while scenario analysis assesses doubling a cyber book over 12 months; reviewers retain decision authority. The workflow is presented as easier to monitor and investigate.
Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow
Zepto describes its customer-support system, processing more than 100,000 tickets daily, and the assurance problems that emerged as volume, categories, and customer behavior changed. Its response is an evaluation-first architecture on Databricks and MLflow, using development and production loops connected by a quality gate. Agent traces capture prompts, completions, retrieved documents, tool calls, latencies, and decision paths through MLflow, OpenTelemetry spans, and Unity Catalog-backed Delta tables; stakeholders define evaluation pillars and gates around a living golden dataset. The dataset grew from 500 examples with an eight-point development–production accuracy gap to 5,247 examples with a 0.4-point gap over six months, while prompt optimization and multiple scorers support iteration. Reported outcomes include 80% plus of tickets fully managed with human oversight, a 65% reduction in support cost or tickets, 20% higher CSAT, 8% better accuracy, and faster development and resolution.
Evaluate AI agents with IBM CLEAR & EvalHub on OpenShift AI
Reliable AI agents can fail in production through incorrect tool use, off-topic reasoning, or plausible but incorrect answers that unit tests, static benchmarks, and logs may not expose. This tutorial builds an eval-driven trace-evaluation pipeline on Red Hat OpenShift AI 3.4 or later, using MLflow to store structured agent traces and evaluation artifacts, IBM CLEAR to score reasoning quality, relevance, and tool usage with an LLM judge, and EvalHub to orchestrate OpenShift jobs. It uploads eight prerecorded multistep research-agent traces, each containing at least 30 spans, then verifies them in MLflow before deploying EvalHub, registering the ibm-clear provider, and configuring judge-model credentials. Standard mode evaluates steps holistically, while SPARC mode gives tool calls separate scoring rows; the resulting workflow produces critiques and ranked failure patterns with frequency, severity, and step or agent attribution.
How Discovery Bank delivers hyper-personalized banking at scale: behavioral AI, governed data, and real-time decisioning
Discovery Bank’s hyper-personalized banking model addresses how to make interactions relevant to individual behavior while meeting financial-services demands for scale, speed, security, and governance. The bank combines demographic, transactional, digital-engagement, savings, borrowing, credit-risk, rewards, and lifestyle-related data on the Databricks Data and AI Platform, producing governed reusable features, indicators, scores, forecasts, and recommendations. Behavioral models and a reusable next-best-action decisioning layer support personalization, fraud detection, servicing, and banker assistance; TRUST alerts assess transactions against client-specific norms and can escalate from explanation to account locking. A four-layer architecture adds control services and specialized generative AI and agents to trusted data and analytical foundations, with governance remaining in the loop. Reported results include a 40% uplift in client engagement impact, 20x faster pipeline development and data processing, 5x faster data-product creation, more than 300 models built per day, and return on investment above 500%.
How Databricks is turning video into searchable, actionable intelligence
Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.
End-to-End RAG Workflow: How Retrieval Augmented Generation Works
Retrieval Augmented Generation (RAG) connects a large language model to external knowledge at inference time, addressing outdated training data and access to proprietary or real-time information. The workflow ingests and normalizes sources, splits documents into chunks, embeds them in a vector store, retrieves context, assembles a prompt, and generates an answer. Semantic search can be combined with BM25 keyword search through reciprocal rank fusion, while reranking can improve precision; the same embedding model must be used during ingestion and querying. The guide presents evaluation and deployment considerations, including separate measurement of retrieval precision and generation faithfulness, versioning, monitoring, and containerized components. It identifies poor retrieval as the most common failure mode and explains that RAG reduces, but does not eliminate, hallucinations.
Data scientists: Powering the future of AI and analytics
Data scientists connect statistics, programming, and domain knowledge to turn raw data into models, recommendations, and decisions with business consequences. The role now extends beyond classical modeling to large language models, generative AI applications, agentic systems, and production workflows involving deployment, monitoring, and retraining. Modern practice combines Python, SQL, and libraries such as scikit-learn, PyTorch, Spark, and MLflow with data engineering basics, communication, and domain expertise. Data scientists contribute across problem framing, data access, preparation, feature engineering, experimentation, deployment, and lifecycle monitoring, with governed access and lineage helping align training data with production data. The article concludes that AI assistants and agents can automate routine work, but human judgment remains necessary for trustworthy evaluation, business framing, and actionable recommendations.
What’s new in Genie Code at Data + AI Summit 2026
At Data + AI Summit 2026, Databricks announced expansions to Genie Code for complex, agentic data and ML work. The changes include a full-page command center for managing concurrent threads and assets, upgrades across production ML engineering, and scheduled tasks that run prompts while users are away. For ML workflows, Genie Code uses Databricks production expertise and Genie Ontology, integrates with MLflow and Model Serving, and can move GPU jobs to AI Runtime while using workspace environment features. It can write features, coordinate edits, run and debug code, compare candidates, inspect endpoint health, and diagnose issues, with users deciding what to keep. Scheduled tasks are described as coming soon, creating reviewable threads from prompts and optional Databricks assets.
Introducing Genie ZeroOps: Put your data and AI operations on autopilot
Genie ZeroOps is an autonomous background agent for monitoring and operating data and AI assets, including jobs, pipelines, tables, and ML models. It continuously detects visible and silent failures, uses Unity Catalog lineage and platform observability to assess root causes, generates remediation through development workflows, and verifies fixes in isolated sandboxes. These environments use shallow, zero-copy table clones, scoped permissions, and network isolation, so proposed changes run against real data without touching production or applying anything before approval. For ML workloads, the agent can diagnose degraded predictions, train a candidate on corrected features, evaluate it against the production model’s existing eval suite and criteria, and support live-traffic ramping when it is measurably better. Genie ZeroOps is entering private preview in the coming weeks, initially supporting jobs, pipelines, tables, and ML workloads; Apps and Lakebase databases are on the roadmap.
Empower your healthcare agents with ready-to-use MCP on Databricks Marketplace
Databricks announces ready-to-use Model Context Protocol (MCP) servers for healthcare and life sciences through Databricks Marketplace, addressing the need to combine curated biomedical knowledge, timely data, specialized tools, and private records. Listings include services for drug and target intelligence, literature, clinical trials, FDA information, Medicare coverage, ontologies, real-world evidence, clinical semantics, and interoperability, while Climb connects live public sources with private Gold-layer data under Unity Catalog governance. Marketplace and custom MCP servers are centralized in the MCP Catalog and governed by Unity AI Gateway, with Genie Spaces, AI Search, Unity Catalog functions, and SQL Warehouses available as managed MCP servers. Users can assemble agents in AI Playground, Agent Bricks, or notebooks, then deploy endpoints or apps with MLflow tracing, evaluation, human feedback, and AI guardrails; examples span molecular-property lookup, clinical questions, and drug research.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.