Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
How Ecolab rebuilt retail intelligence on Databricks and Anthropic Claude
Ecolab needed to combine audits, health inspections, pest telemetry, and other data from nine systems so retail teams could answer location-specific compliance questions. Its Retail Intelligence application is a native Databricks App using Lakebase Postgres, Lakeflow, and Spark Declarative Pipelines to move governed data into a Unity Catalog lakehouse, while Foundation Model APIs serve Claude Sonnet, Claude Haiku, and Gemini. A Coordinator Agent delegates requests to specialized agents that use Vector Search, SQL, Unity Catalog Functions, and an external MCP server; a Response Agent returns cited answers, with short- and long-term memory stored through Lakebase. The system also applies five Judge LLMs, MLflow tracing, and ai_query() batch inference. Report preparation fell from two weeks to under two minutes, while the assistant supports approximately twelve languages at about 98% accuracy.
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.
Simplifying Data and Product Integrations with a Data Abstraction Layer
Uber built a domain-agnostic Data Abstraction Layer (DAL) to decouple data consumers from changing physical datasets and simplify advertiser reporting. The DAL is an RPC service that accepts a logical table, schema, time range, and filters; logical tables define the interface, while configuration describes locations, relationships, availability windows, and rollup transforms. It generates database-specific queries, runs them in parallel across systems such as an internal OLAP database, Docstore, and Apache Hive, and assembles responses with aggregation and rollup behavior determined by the request. Removing a dimension can produce an account-level response with metrics automatically rolled up, while mixed realtime and daily tables can cover different portions of a requested time range. For advertiser reporting, the approach reduced new report delivery from several weeks or months to under two days and enabled dynamic filters, report builders, and third-party API integrations.
How to make the case for giving your AI Agent system access
AI Agents can answer support questions without backend access, but they cannot complete changes such as payment-plan updates, claims, or account edits. Connecting an Agent to CRM, billing, order-management, or other systems moves support from explanation to end-to-end resolution, and the source frames that shift as the basis for an engineering request. Data from a 2026 Customer Service Transformation Report shows improved metrics among 87% of teams with mature AI deployment, versus 62% overall, while only 10% report reaching that stage; an internal comparison found Procedures with system access produced the largest gains in workflows requiring judgment, branching, live data, or handoffs. The recommended case starts with one high-volume, repeatable workflow, maps its data and actions, and defines a narrow pilot. Teams can then progress from no integration to read-only access and finally write actions, using measured results to justify further work.
Making FlashAttention-4 faster for inference
FlashAttention-4, originally designed for pre-training, required modifications to efficiently handle LLM inference workloads dominated by the memory bandwidth-limited decode phase. Modal’s contributions include adjusting parallelism strategies from query parallelism to key/value parallelism, implementing split KV to parallelize across KV tiles, and adding a heuristic for optimal splits. They added support for FP8 inputs, arbitrary KV page sizes via cp.async loads, and reduced wasted work for small query sizes by using a single query tile. The kernel also gained extended GQA packing for irregular head ratios, using cp.async for query tiles to bypass TMA alignment restrictions. These changes, measured in throughput improvements up to 3x for single-token decode and up to 2.4x for small page sizes, demonstrate the leverage of flexible parallelism and tile-based programming for inference optimization.
Governing agent autonomy with Auto-review
Auto-review governs local agent autonomy as a continuum, allowing routine work to proceed while reviewing actions whose consequences depend on context, user intent, and security stakes. A small classifier agent runs before tool calls in the same RPC stream as the parent agent, and can inspect workspace contents with ReadFile, Grep, Glob, and ListDir when a command alone is insufficient. Its blocks return explanations rather than automatically prompting users, enabling the parent agent to narrow or replace a risky action. Evaluation used 6,122 deduplicated labeled rows from internal developer sessions plus synthetic high-consequence cases, and tested tool use, overrides, parse failures, and behavioral flapping. Early results show the classifier blocks around 4% of reviewed actions, while about 7% of Auto-review chats experience at least one interruption.
Catch visual regressions with Snapshots, now in beta
Sentry has released the beta version of Sentry Snapshots to catch visual regressions on pull requests across any platform with a frontend. The tool captures screenshots of an application across various viewports, themes, and languages, compares them against a baseline, and blocks pull requests when diffs are detected. Teams can attach custom context metadata, including test file paths, to help developers and AI agents locate test sources and generate additional test cases. By leveraging the sentry-mcp integration, REST API, and sentry-cli, workflows can retrieve base images, diff masks, and run local diff verification before pushing commits. Originally created at Emerge Tools for mobile applications, this iteration expands automated visual testing capabilities to broader frontend codebases.
Stop building data products. Start building data services.
Howden’s rapid acquisition pace exposed limits in an enterprise data model built around one product per use case, downstream quality checks, and dashboard-driven consumption. Group Chief Data Officer Barry Panayi describes shifting to open, governed data services, moving mastering and quality checks closer to ingestion, and codifying reconciliation in the Accord data model. On Databricks, the company consolidated more than 100 sources of record, standardized pipelines and shared code, and built reusable assets for cross-domain analytics, while continuing to productionize models as consistent services. The account argues that AI agents require a composable services layer, and that insight lag—the time between data existing and being usable—matters more than freshness; conversational analytics through Genie also reduced dashboard-building work.
Introducing Boost+
Boost+ is an updated promotion for delivery drivers that adds pay to eligible trips during periods and locations of high demand. The promotion applies automatically when a trip’s pickup is in a designated blue zone during listed hours, and the extra amount appears in the fare shown on the trip request. Its calculation uses estimated active time—from accepting the trip through delivery—rather than time spent online waiting; for example, a 30-minute trip at $10 per active hour receives $5 in Boost+ pay. Drivers can find upcoming opportunities in the Discover tab and on the map, view total Boost+ earnings in the trip receipt, and combine the promotion with Quests and other promotions. Availability varies by city, and time beyond the estimated active time does not qualify for Boost+ pay.
Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP
Part 2 of the series examines how PyTorch profiler traces change from a single bias-enabled nn.Linear to a three-layer MLP with an intervening activation, using scripts run on an NVIDIA A100-SXM4-80GB GPU. It explains that nn.Linear represents y = x @ w.T + b, dispatches aten::addmm, and folds bias addition into the cuBLAS GEMM epilogue, while aten::t only rewrites weight strides and launches no GPU kernel. For the MLP, eager execution uses three GEMMs plus separate activation and multiplication kernels, whereas torch.compile fuses the latter operations and keeps the intermediate out of HBM, while leaving the GEMMs unchanged. A hand-written Liger Triton kernel provides the same fusion without Dynamo guards or compile latency, trading Inductor’s shape-specialized speed for generic execution across changing shapes.
Building ClickCannon - a tool for benchmarking ClickHouse
ClickCannon was created to answer how much CPU, memory, and disk are needed for ClickStack observability workloads, while accounting for schema, ingestion rate, hardware, and concurrent queries. It replays about 2 TiB of real OpenTelemetry trace data, converted to ClickHouse Native format with ZSTD compression, rather than relying only on synthetic datasets. A custom multithreaded Go reader/inserter provides control over uncompressed throughput and exposes stream metrics such as compressed and uncompressed bytes, row counts, and block counts, while the framework simulates users and query workloads during ingestion. The project evolved into an open-source, generic ClickHouse benchmarking tool for custom insert and query workloads, with fine-grained control over throughput, concurrency, and user behavior; the post reports that it can push substantial throughput from a single machine.
Scaling AI Through Data Fluency
Aer Lingus is redirecting a significant share of its IT and change spending from traditional maintenance toward a Databricks-powered data foundation, addressing legacy systems that trap information in departmental silos. Dave O’Donovan says the airline spent the past 18 months prioritizing platform development, governance, data quality and data literacy rather than chasing each new AI product. Databricks was selected for a unified lakehouse architecture, with data warehousing, Genie’s plain-English querying and real-time operational data intended to broaden access beyond specialist teams. At Aer Lingus’s Operations Control Center, combining sensor and operational inputs gives teams a fuller real-time view for disruption decisions, while commercial teams use live insights to adjust pricing. The transformation also includes a Data Literacy Academy, a 75/25 capacity split between foundational work and innovation, a 20-person Continuous Improvement team, and experiments with agents for business-case development and CFO review.
AWS and Databricks at Data + AI Summit 2026: Accelerating real-world AI innovation
AWS and Databricks describe their expanded collaboration at Data + AI Summit 2026, where AWS returns as a Legend Sponsor with sessions, demos, customer stories, and industry forums. The partnership centers on generative AI adoption, unified governance, and open data architectures, including an agentic stack that combines Amazon Bedrock, Bedrock AgentCore, Kiro, and the Databricks Data + AI Platform. A featured integration uses a governed MCP connection through Databricks Apps so AgentCore can query Unity Catalog-governed data, ask AI/BI Genie questions, and read low-latency state from Lakebase while honoring existing permissions. AWS will demonstrate these workflows at Booth #100 and present a session on federating Unity Catalog to AWS Glue, alongside customer examples including Mastercard, Talkdesk, nCino, Addepar, and Workday. Attendees can also join technical conversations, receptions, and a 14-day Databricks on AWS Marketplace trial with $400 in usage credits.
Announcing the Public Preview of Custom URLs
Databricks has announced the public preview of Custom URLs, giving each account a single branded domain that serves its workspaces. Previously, workspace-specific URLs made navigation, sharing, bookmarking, and account-wide features more cumbersome, while users had to log in repeatedly when switching workspaces. Custom URLs use a shared account session to verify access and create workspace sessions seamlessly, while preserving authorization boundaries and keeping existing per-workspace URLs functional. The feature provides a unified Genie entry point, cross-workspace Unity Catalog lineage, and an account-level URL that remains stable during disaster-recovery failover, including for downstream tools using existing connection strings. Activation requires Unified Login; Frontend Private Link workspaces fall back to per-workspace URLs, and account admins can claim a URL, enable it, and optionally turn on automatic redirects.
How Rivian drives trusted, AI-powered decisions at the speed of thought with Databricks
Rivian is building electric vehicles and services that require fast, trusted decisions across manufacturing, supply chain, finance, service and operational planning, while business users need reliable metrics and insights. Using Databricks AI/BI, Genie, Unity Catalog metric views, Databricks Apps and AI-assisted engineering, the company is consolidating dashboards, semantic definitions, permissions, sensitive data and AI-powered workflows on one governed foundation. Rivian migrated a massive multi-domain dashboard base in less than six months, is standardizing more than 50 metrics, and worked with Databricks as a design partner on roughly 58 product features. The resulting self-service analytics and operational applications cut supply-chain monitoring time by 60 to 70%, reduce inventory investigations from over 30 minutes to under two, predict stock-out risk more than four days ahead, and reduce some ingestion setup time by more than 60%, supporting AI-powered decisions without competing versions of the truth.
Piloting a new offline heatmap with delivery zones
The Driver app is piloting a new offline heatmap on its home screen in select US cities. It shows where couriers can go online instantly and indicates demand levels intended to help them find trips more quickly. Each outlined delivery zone is color-coded red for high demand, orange for medium demand, yellow for low demand, or grey for areas open only to Platinum and Diamond couriers. The map refreshes every 10 minutes; couriers can tap a zone to see its name or navigate there, then see nearby busy restaurants and stores after going online. The announcement lists participating locations across the United States.
Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions
As Lyft scaled, different teams developed conflicting definitions for key business metrics due to the lack of centralized version control and shared standards. To resolve this, Lyft implemented an internal Metric Semantic Layer as a Python package that stores authoritative metric definitions in YAML files with Jinja SQL templates. The system restricts onboarding to Golden Metrics used across multiple applications, requiring team-based approval from both Business and Operational Owners for any definition changes. Standardized definitions are exposed through Python APIs, integrated into the Amundsen data catalog and self-service user interfaces, and surfaced to AI tools via a Model Context Protocol.
Datadog and ClickHouse partner to bring full-fidelity data to modern observability
ClickHouse and Datadog announce a partnership aimed at reducing the tradeoff between retaining complete telemetry and keeping it accessible through familiar observability workflows. New Preview capabilities let organizations route logs directly to ClickHouse through Datadog Observability Pipelines and search those logs from the Datadog Log Explorer. Pipelines can parse, enrich, filter, transform, and redact data before storage, while ClickHouse Cloud’s separation of storage and compute supports long-term retention, independently scaled query resources, and workload isolation. Federated log search queries data directly in ClickHouse, avoiding duplicate copies or re-ingestion while allowing high-volume logs to remain searchable and historical investigations to use full-fidelity telemetry. The partnership combines Datadog’s collection and investigation experience with ClickHouse’s storage and analytics foundation; the native integration and federated search are available in Preview.
Jumpstart your Data Modeling with Databricks Industry Data Models
Databricks is publishing a public library of 40 Lakehouse industry data models designed to provide Silver-layer foundations for analytics and machine learning. Each industry offers a Minimum Viable Model and Expanded Coverage Model derived from the same model.json, with breadth rather than attribute depth distinguishing the scopes. A rules-driven AI agent applies more than 200 structural checks across 14-plus modeling domains, enforcing hierarchy, primary and foreign keys, normalization, division balance, data types, governance tags, and acyclic relationships. The models deploy to Unity Catalog in three physical cataloging styles and include DDL, schemas, metric views, classification tags, ontology, diagrams, and synthetic data with valid references. The airline ECM example contains 19 domains, 420 products, 17,278 attributes, 420 primary keys, and 2,877 foreign keys, while the models remain customizable starting points requiring domain expertise and organizational review.
AI Serving Platform That Adapts to Your Model
Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.