Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Dynamically Scaled Activation Steering
Activation steering has emerged as a way to guide generative models toward desired behaviors, but uniform interventions can reduce performance when steering is unnecessary. DSAS, or Dynamically Scaled Activation Steering, addresses this by separating when to steer from how to steer and adapting intervention strength across layers and inputs. At generation time, it computes context-dependent scaling factors that modify existing steering transformations, and it can also be jointly optimized end-to-end with the steering function. Combined with existing methods, DSAS improves the reported Pareto front between toxicity mitigation and utility preservation, while adding minimal computational overhead; experiments also apply it to a text-to-image diffusion model for modulating specific concepts and show token-level interpretability through the strength and location of steering.
coSTAR: How we ship AI agents at Databricks fast, without breaking things
Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution.
Business Analytics Tools: A Complete Guide for Data
Business analytics tools have evolved beyond static reporting as organizations seek explanations, forecasts, and immediate actions. This guide categorizes visualization and dashboard platforms, self-service analytics, statistical tools, spreadsheets, and SQL-based query tools, then examines how AI adds natural-language querying, anomaly detection, summaries, and forecasting. It argues that inconsistent, duplicated, or differently governed data can undermine even sophisticated dashboards. Lakehouse architecture combines data-lake scalability with warehouse governance, performance, and SQL compatibility, enabling tools such as Tableau, Power BI, and Looker to use broader, fresher datasets alongside analytics and AI workloads. Reported examples include 94% faster BI queries at Anker Innovations, 98% faster query performance at AnyClip, and consolidation across 120+ global analysts at JLL.
MLOps Frameworks: A Complete Guide to Tools and Platforms for Production ML
The guide explains how MLOps frameworks close the gap between notebook experimentation and reliable, scalable machine-learning production by applying automation, version control, and continuous delivery across the lifecycle. It describes experiment tracking, model registries, workflow orchestration, feature stores, deployment and serving, and monitoring as the capabilities needed to make training reproducible, prevent training-serving skew, automate pipelines, and detect degradation after release. MLflow is presented as a modular open-source option with Tracking, Model Registry, Models, and Projects, supporting run history, lifecycle stages, standardized packaging, and reproducible execution across environments. Kubeflow suits Kubernetes-standardized organizations and large distributed GPU workloads, while Metaflow, cloud-native services, and unified platforms such as Databricks are positioned as alternatives according to developer experience, cloud alignment, or operational preferences.
What is Machine Learning? How It Works, Types and Use Cases
Machine learning (ML) is a subset of artificial intelligence that learns statistical patterns from data to make predictions or decisions without hard-coded rules. The guide distinguishes ML from generative AI, which creates new content, and from deep learning, a specialized ML approach using multilayered neural networks that learns features automatically and typically requires more data and computing power. It describes a workflow of preparing and splitting data, training models by comparing predictions with actual outcomes, and adjusting parameters to reduce error, illustrated with image classification and customer-churn scoring. It also covers supervised, unsupervised, and reinforcement learning, applications including computer vision and natural language processing, newer architectures such as transformers and Mamba, and MLOps practices for deployment, monitoring, and retraining.
Building the future of security with NAB with Lakewatch
National Australia Bank (NAB) is working with Databricks as one of five design partners for Lakewatch, an Open Security Lakehouse intended for modern security operations. NAB, part of Australia’s critical infrastructure, ingests more than 30TB of security data daily while confronting AI-enabled threats, expanding digital services, and growing third-party integrations. The partnership brings NAB’s threat-informed cyber-defense experience to a platform designed to unify signals across environments, apply advanced analytics and AI-driven workflows, generate higher-confidence insights, and automate response actions. Lakewatch is presented as an open, composable, AI-native foundation built natively on the Lakehouse for organizations facing constantly evolving threats, supporting enterprise-wide correlation, proactive threat hunting, detection, monitoring, and response at scale.
Databricks Announces Lakewatch: New, Agentic SIEM
Databricks announces Lakewatch, an agentic SIEM intended to help organizations defend against increasingly sophisticated, AI-driven attackers. Launching in Private Preview, it unifies security, IT, and business data in a governed lakehouse built on Unity Catalog, while supporting open OCSF formats and storage in customer-controlled cloud object stores. Lakeflow Connect ingests and normalizes major security sources, and Detection-as-Code lets teams define YAML rules with SQL queries or Python notebooks, backtest them against historical data, and deploy through CI/CD. Genie Code and Genie Spaces provide AI-assisted ingestion, detection authoring, rule refinement, investigation, and natural-language threat hunting, while custom models can support anomaly detection, behavioral analytics, and entity risk scoring. The announcement also introduces an Open Security Lakehouse Ecosystem and acquisitions of Antimatter and SiftD.ai; Lakewatch is available in Private Preview, with Adobe and Dropbox among the named customers.
How agentic software development will change databases
AI agents are changing software development from a slow, linear process into rapid evolutionary experimentation, creating many small, ephemeral applications and databases. The post identifies three resulting database requirements: near-zero marginal cost with elastic growth, native branching for rapid variants, and open ecosystems that agents can use reliably. Lakebase addresses branching through O(1) metadata copy-on-write at the storage layer, so agents can branch production or test state without physical data copying and pay only for experiment compute. Its decoupled, serverless compute can scale in sub-second intervals and down to zero, while standard open Postgres page formats in cloud object storage keep data accessible to external engines and tools. Lakebase telemetry reports about 10 branches per project on average, some nested beyond 500 iterations, and roughly four times as many databases created by agents as by humans.
Announcing the 2026 Built
Databricks has announced its third annual Built-On Databricks Startup Challenge, inviting early-stage startups that use Databricks as a core part of customer-facing, B2B applications to compete for more than $1M in prizes. Eligible companies must have been founded within the last five years, be incorporated in one of the listed countries, and be no later than Series B if funded. The competition offers a potential Databricks Ventures investment of up to $600,000, up to $500,000 in Databricks product credits across finalists, and access to Databricks resources, workshops, mentorship, and feedback. Judging considers Databricks usage, market potential, and founding team strength and experience; applications close May 22, 2026, with judging in late May and winner announcements during the week of June 15, 2026, at the Databricks Data + AI Summit in San Francisco.
Announcing General Availability and Open Sourcing of Unity Catalog Business Semantics
Databricks announces the general availability of Unity Catalog Business Semantics, a unified semantic foundation for consistent business definitions across BI dashboards, notebooks, SQL applications, and AI agents. Built into Unity Catalog, it applies existing governance, security, lineage, and access controls to centrally defined Metric Views, whose metadata includes display names, formats, and synonyms. Metric Views are authored declaratively in SQL, compiled and executed deterministically at query time, and can use materializations with automatic pre-aggregation, incremental refresh, intelligent query rewriting, and transparent routing; materialization remains in Preview. The core Metric View implementation is being open sourced in Apache Spark OSS for an upcoming release, with Unity Catalog OSS v0.5 support coming soon, while a point-and-click authoring UI is in Public Preview.
How Addepar scales investment workflows with Databricks AI agents
Addepar, a global technology and data platform, manages nearly $9 trillion in assets and moved from older systems to a unified data intelligence platform on Databricks running on AWS. It ingests hundreds of disparate data feeds, standardizes and enriches them, and delivers results through products, APIs and data sharing. Since adoption, pipeline costs fell 60% versus legacy infrastructure, saving more than $2 million, while new pipelines and integrations became five times faster. Addison, a native AI experience embedded in the platform, combines governed portfolio Q&A, proactive insights and action-oriented workflows, using live data and Addepar’s core calculations engine. Addepar is adopting Databricks Agent Bricks, including Supervisor Agent flows with Genie-powered analytics, while Foundation Model Serving and managed MLflow support governed model serving, traceability, evaluation and iteration for human-supervised agentic workflows.
Why agentic analytics starts with a well
Agentic analytics increases the need for trustworthy semantics and governance because AI systems must interpret business definitions, trace outputs, and support follow-up questions beyond static dashboards. Nick Eayrs argues that fragmented metrics across BI tools, proprietary semantic layers, and duplicated pipelines erode trust, slow decisions, and increase engineering cost. The proposed foundation is an open, interoperable semantic layer where certified metrics such as revenue, churn, and customer lifetime value are explicitly defined, reusable, and accessible through SQL to BI, AI interfaces, notebooks, and agents. Organizations should establish governed data foundations and a catalog first, certify metrics next, then add AI and evaluate its outputs. Examples include NTT Docomo reducing manual log-analysis time from 66 hours to 6 hours and Net One Systems cutting support response time by 75 percent.
Accelerate business insights with Lakeflow Connect, now with a Free Tier
Databricks is introducing a Lakeflow Connect Free Tier to help teams ingest SaaS application and operational database data into the Databricks Data + AI Platform, addressing silos that can leave AI agents without complete enterprise context. Each workspace automatically receives 100 free DBUs per day for managed SaaS and database connectors only, with capacity advertised as up to 100 million records per workspace per day across eligible supported sources. Lakeflow Connect offers 30+ connectors, including SaaS services and databases, through a point-and-click UI and API, with Unity Catalog providing unified governance and end-to-end lineage. The offer uses standard pricing without volume-based premiums or row-based pricing, but actual capacity varies with workload characteristics, record sizes, and source types.
AiChemy: Next
AiChemy is a multi-agent assistant for cross-disciplinary research that combines external MCP servers including OpenTargets, PubChem, and PubMed with Databricks-managed chemical libraries. It can identify disease targets and drug candidates, retrieve chemical and pharmacokinetic properties, assess safety and toxicity, and attach findings to verifiable data sources. Its guided workflow covers disease-to-target-to-drug discovery and literature validation, while a vector search over a 250,000-molecule ZINC15 index uses 1024-bit ECFP embeddings to find compounds similar to Elacestrant. To build a custom supervisor, users connect public MCP servers, transform structured drug data into a Genie space, and index unstructured chemical data for similarity search. They can assemble it with no-code Agent Bricks or deploy a Langgraph supervisor with Lakebase, MLflow AgentServer, and a React interface; invocations are logged and traced through MLflow using OpenTelemetry standards.
How MakeMyTrip achieved millisecond personalization at scale with Databricks
MakeMyTrip needed sub-second personalization for “last-searched” hotels across its consumer and corporate travel businesses, serving millions of daily users. Spark Structured Streaming’s micro-batch mode consistently delivered one to two seconds, while adopting Flink would have added a second engine, duplicated logic and increased operational overhead. The team instead adopted Databricks Real-Time Mode (RTM), using continuous data flow, concurrent pipeline scheduling and streaming shuffle to process events with millisecond-level latency on Spark. Its unified pipeline merges B2C and B2B Kafka streams, enriches events with each user’s “Last N” hotels from Aerospike, and writes results to Redis for serving in under 50ms. After tuning partitions, backpressure and rate limits, the production deployment reduced P50 latency from approximately 1.23 seconds to 44ms, P99 latency from over one minute to about 500ms, and increased click-through rate by 7%.
How FSIs eliminate silos between clients, operations, and finance
Financial institutions often split portfolio, operations, actuarial, and finance work across separate systems, even when teams depend on the same portfolios, positions, liabilities, and transactions. Databricks presents a unified, governed platform combining Databricks SQL, Unity Catalog Metric Views, Lakebase, Genie, Databricks Apps, and Lakeflow Designer to connect analysis with operational action. A worked workflow follows an actuary's duration analysis, a portfolio manager's scenario modeling and mandate changes, operations' IBOR/ABOR reconciliation, and finance's approval of adjustments and ledger entries. Governance features including row-level security, column masking, RBAC, ABAC, and shared business semantics keep access and terminology consistent, while Lakebase records corrections in auditable, timestamped tables. The result described is a closed-loop process in which strategy, risk, and accounting use the same governed data rather than reconciling separate systems.
How Zalando built a unified data foundation for AI and analytics on Databricks
Zalando describes how it is building a unified data foundation for analytics and AI across a microservices-driven ecosystem serving more than 50 million active customers and 7,000 brands and partners. Its architecture combines private domain-owned catalogs for autonomous development with a centrally governed shared catalog that exposes cross-team data through Dynamic Views, enabling identity-based policies, GDPR checks, column classification, and auditability. Databricks Metric Views provide a governed semantic layer where business logic and KPI definitions are centralized and reusable across SQL, dashboards, notebooks, and AI agents. Genie uses those metrics plus enriched metadata to answer natural-language questions more reliably, although poorly curated or unaggregated data can increase costs; Zalando plans broader enterprise adoption through curated spaces and additional orchestration.
Powering clinical research innovation: How TriNetX uses Databricks to accelerate drug development
TriNetX operates a federated real-world health data network spanning more than 230 healthcare organizations and needed infrastructure to make growing volumes of RWD easier to use for clinical research. Databricks now provides the centralized lakehouse architecture, consolidating electronic health record data while supporting custom datasets, consulting, ML models, proprietary algorithms, APIs and self-service access. TriNetX is extending this foundation with a beta Query Assistant for natural-language analysis and a Support Assistant prototype built with Databricks’ Agent Bricks, requiring no programming expertise for the former. In 2025, pharmaceutical clients reduced protocol amendments by up to 50%, while an AI-enhanced site-identification effort achieved a 63% site acceptance rate and nine-day average response time. Other models suggest enrollment conversion could rise from 33% to 85%, with expanded APIs and Genie exploration planned for 2026.
What is AIOps?
AIOps applies AI and machine learning to IT operations to analyze logs, metrics, traces, events, alerts, and related system data. It detects anomalies, correlates events, uses topology and dependencies to identify likely root causes, and supports guided or automated responses. The guide positions AIOps between observability and action, reducing noise and adding context without replacing observability, DevOps, or human judgment. Its workflow includes data ingestion, normalization, anomaly detection, event correlation, root-cause analysis, automation and orchestration, and collaboration workflows, with human-in-the-loop review recommended for higher-risk production changes. It contrasts domain-centric and domain-agnostic approaches and recommends evaluating coverage, depth, integrations, setup effort, and organizational fit before expanding automation.
The official ClickHouse provider for Apache Airflow is now available
Apache Airflow users can now use an upstream ClickHouse provider, offering an officially maintained alternative to community plugins or custom integration code. The provider, apache-airflow-providers-clickhousedb, uses ClickHouse Connect over HTTP(S), integrates with Airflow’s common SQL framework, and supplies ClickHouseHook for bulk inserts, streaming, and client-specific operations. The post documents installation, connection setup, standard SQLExecuteQueryOperator DAGs, bulk insertion, and local or managed deployment through Astronomer, including TLS settings for ClickHouse Cloud. Production examples include Astronomer’s Astro Observe, Chartmetric’s 15-million-row playlist pipeline every five minutes, and ClickHouse’s warehouse, which schedules 76 DAGs across more than 40 sources and moves about 6 billion rows daily. The provider is available now, with future capabilities guided by community requests.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.