Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Introducing Omnigent: A Meta-Harness to Combine, Control and Share Your Agents
Omnigent is an open-source meta-harness designed to combine, control, and share agents across different harnesses, models, and interfaces. Databricks says users currently juggle multiple agents and copy context between them, while builders struggle to combine or replace harnesses with incompatible interfaces. Its runner wraps terminal-based agents and SDKs in sandboxed sessions with a uniform API for messages, files, streamed text, and tool calls; a server adds policies, sharing, and access through terminal, web, mobile, native Mac OS, and APIs. Features include live collaboration, hosted sandbox execution, contextual security and cost policies, OS isolation, and multi-harness authoring. Released in alpha under Apache 2.0, Omnigent is intended to provide a durable layer above changing agents and harnesses.
Statement on the US government directive to suspend access to Fable 5 and Mythos 5
Anthropic says the US government issued an export-control directive requiring it to suspend Fable 5 and Mythos 5 access for every foreign national, including foreign-national Anthropic employees, inside or outside the United States. The directive arrived at 5:21 p.m. ET and did not specify the national-security concern; Anthropic says it understands the basis to be a potential narrow, non-universal jailbreak involving a request to inspect a codebase and fix software flaws. The company says it reviewed the underlying report, found only previously known minor vulnerabilities, and judged the demonstrated capability widely available from other models, including OpenAI’s GPT-5.5. Anthropic is complying by disabling both models for all customers and says it is working to restore access.
From Wall Street to Data Platforms
Kim Hatton, Databricks’ Global Financial Services Marketing Leader, describes how two decades in regulated financial-services marketing led her toward technology and data-centered strategy. She says financial institutions still need to unlock value while navigating compliance, but their divisions and systems make a unified customer view difficult. Her account points to Unity Catalog’s unified governance and single source of truth for breaking down silos and supporting requirements including GDPR, customer identity, and sensitive workloads. It also describes Lakebase’s separation of compute and storage for faster ML/AI agent experimentation, alongside Genie’s plain-language data analysis, which can reduce work that otherwise takes months and expensive third parties. Hatton connects these tools with faster, more confident marketing and accurate decision-making in regulated workflows, while also describing Databricks’ inclusive, high-energy culture.
What is enterprise intelligence?
Enterprise intelligence (EI) is presented as an organization-wide capability combining business intelligence, knowledge management, enterprise search and AI to turn structured and unstructured data into decisions and actions. Unlike traditional BI, which centers on dashboards, reports and structured data, EI connects these capabilities through a shared architecture and governed business context. The described stack includes a lakehouse-based data foundation, batch and streaming pipelines, governance, semantics, analytics, search, machine learning, generative AI and a decision layer. Shared definitions such as “active customer” and “monthly revenue” are intended to keep dashboards, queries and AI agents aligned, while maintained context addresses knowledge that becomes stale as the business changes. The result described is a common trusted source from which people, applications and agents can produce insights and initiate actions.
ClickHouse achieves AWS Retail Competency
ClickHouse has achieved the AWS Retail Competency in the Advanced Data Insights category, a designation recognizing validated technical expertise and demonstrated customer success in retail real-time analytics. Retail workloads described here combine high-volume, high-cardinality, high-concurrency data with hard freshness requirements, including inventory, pricing, personalization, and peak seasonal demand. The post attributes the platform's fit to compressed columnar storage, parallel queries, continuous Kafka and Amazon S3 ingestion through ClickPipes, compute-compute separation, and AWS support for private networking, multi-region deployment, and Marketplace billing. Examples include Mercado Libre's 90-TB-per-day observability ingestion and trace lookups reduced from 15–20 minutes to about two seconds, while other customers report cost, speed, and analyst-efficiency gains.
How Dropbox uses MCP and Dash to close the design-to-code security gap
Dropbox developed a system using the Model Context Protocol (MCP) and Dash's semantic search to bridge the gap between security threat models and code implementation. By retrieving original security documents during pull requests, an LLM agent automatically evaluates whether the proposed code adheres to previously agreed-upon security requirements. This approach surfaces design regressions and missing controls that traditional static analysis tools miss.
Enabling Evolutionary Database Development: Database branching with Lakebase, the conclusion
The post concludes a series on how copy-on-write database branching in Databricks Lakebase changes team-scale evolutionary database development without changing its underlying methodology. For a team of fifty developers, long-running tier branches and ephemeral feature branches form a parent-linked promotion hierarchy, replacing separately provisioned environment instances and enabling promotion by merge, rollback by repoint, and computable schema divergence. Governance is declared once and inherited per branch, with policies intended to prevent transitions that contradict the parent chain; Unity Catalog captures metadata for attribution and audit. The DBA's role becomes platform engineering, while agents operate inside an executable SCM state machine with documented inputs, outputs, schema validation, and enforced gates. An optional TDD layer adds dedicated roles, acceptance-criterion scenarios, RED-GREEN-REFACTOR cycles, and artifact contracts, and the conclusion presents the resulting workflow as operational for human and agent practitioners.
Results from first Anthropic Public Record
Anthropic’s first Public Record survey measures how Americans viewed AI in November and December 2025, using a nationally representative online sample of 51,993 people aged 16 and over. Respondents most often hoped AI would help cure diseases (48%), while job loss was the leading concern (64%), followed by cognitive dependency (56%) and misinformation (52%). More than 70% supported government involvement in AI, especially around privacy, child safety, and liability, but only 15% trusted AI companies to decide how AI should be developed and used. The results describe broad agreement across partisan, geographic, and educational groups, with differences usually reflecting intensity rather than direction; Anthropic says the survey will be repeated to track changing attitudes as capabilities and adoption develop.
Talk to all your data, wherever it lives
Lakehouse Federation addresses the challenge of reasoning across enterprise data spread among AWS Glue, Snowflake, Oracle, BigQuery, Postgres, and legacy formats without first migrating it. By connecting external sources in place and syncing their metadata into Unity Catalog, Databricks applies shared permissions, lineage, and access controls while preserving source data. Federated comments and descriptions give Genie schema context, while Unity Catalog Semantics lets teams define governed metrics such as ROI once for consistent use across Genie, dashboards, and notebooks. The example connects an AWS Glue marketing database, carries its metadata, defines an ROI metric view, and asks Genie which campaigns led ROI last quarter. The post reports an immediate, accurate answer from live Glue data, and describes planned richer semantics, broader federation, and possible performance gains from managed tables.
Better, faster, less wrong: Enhancing issue grouping
Sentry upgraded its AI-driven issue grouping system to better prevent duplicate issues without merging distinct application errors. The team trained lightonai/modernbert-embed-large using Matryoshka Representation Learning on hundreds of thousands of stacktrace pairs labeled by Claude Sonnet 4.5. To optimize the high-throughput ingestion pipeline, embeddings were truncated from 768 to 64 dimensions, combined with bfloat16 precision, PyTorch SDPA, and CUDA graph compilation. A phased live rollout used threshold-gated index backfilling and automatic fallback to prevent issue spikes during migration. In production, the v2 model cuts the overgrouping rate from 8% to 4%, increases prevented duplicate issues to 70%, and delivers 6x faster inference.
Unlocking semantics for AI: How Mercedes-Benz Korea built trusted “Talk to Data” at scale
Mercedes-Benz Korea piloted a “Talk to Data” architecture that extends its Databricks analytics foundation with a governed semantic layer for enterprise AI, rather than treating the effort as a chatbot project. The design moves Power BI DAX KPI logic into Unity Catalog Business Semantics and Metric Views, keeping sources, joins, measures, dimensions, comments, and synonyms alongside governed Lakehouse data. Genie spaces use curated metric views for domain questions, while Agent Bricks composes persona-based agents, with Unity Catalog enforcing row- and column-level access. An automated DAX-to-Metric-View transpiler parses semantic models, maps tables, generates draft definitions, flags non-automatable measures, and reports conversion gaps. The documented playbook combines gold-layer curation, KPI validation, regression testing, Genie optimization, persona agents, and Databricks Apps; the pilot reports AI answers aligned with established KPI definitions and BI reporting logic.
Forward Deployed Engineering: Delivering Business Outcomes with AI
Databricks is formalizing its Forward Deployed Engineering (FDE) organization to address customers’ shift from migration and data-pipeline requests toward business outcomes with AI. FDE brings Professional Services together around an engineering-led model that embeds engineers with customers, supports modernization and production AI, and works with partners and Databricks R&D. In the cited examples, teams migrated five-plus petabytes of JPMC Consumer and Community Banking Risk data and more than 500 notebooks in four months, while Fox used Lakebase, AI Search, Databricks Apps, and Model Serving to redesign fan experiences. The organization says its engagements use shared OKRs, rapid prototype-to-production delivery, embedded engineering, outcome-aligned commercial options, and global partner coverage, with Fox reporting that Sports AI users spend approximately twice as long in the app.
Production-Ready Agents Need A Production-Ready Data Platform
AI development teams face constant shifts in model providers and agent frameworks, demanding data platforms that provide scalable, real-time context management. Agentic workloads require blending unstructured enterprise data, short-term session state, and persistent long-term memory. MongoDB addresses these requirements through its native JSON document model and integrated retrieval capabilities, combining full-text search, vector search, and hybrid search directly over operational data. Customer implementations such as DevRev, ElevenLabs, and Adobe rely on Atlas to achieve sub-100 millisecond hybrid retrieval and handle billions of requests. Additionally, MongoDB is collaborating with LangChain and ecosystem partners to establish open reference architectures and shared interfaces for portable agent memory across frameworks.
Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest
The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.
How ERGO Hestia reduced time-to-market with Databricks Lakebase and Model Serving
ERGO Hestia redesigned its real-time pricing platform to reduce deployment friction across more than 100 models and 1,000 variables while preparing B2C capabilities. Previously, processed data moved from Databricks through extraction jobs, external Azure PostgreSQL, and a custom caching adapter, creating governance overhead, deployment coordination, and latency spikes during large refreshes. The new architecture uses Lakebase Sync Tables as an online serving layer and Databricks Model Serving Endpoints, keeping data, request logic, and model serving within the lakehouse; Unity Catalog supplies lineage, version tracking, access controls, and audit trails. An incremental migration started with a low-criticality endpoint, measuring 20ms latency and less than 5% CPU utilization at 40 requests per second, before expanding toward larger workloads and the planned decommissioning of PostgreSQL.
Welcoming the first cohort of Databricks student fellows
Databricks announces its inaugural Student Fellows cohort, selected from more than 5,000 applications submitted by students at hundreds of universities and from dozens of countries. The program targets students who contribute on campus, apply data and AI in practice, and intend to pursue careers in the field, with fellows acting as bridges between academic theory and the real-world scale of the Databricks platform. Five students are profiled, with experience spanning large-scale AI systems, ETL pipelines, computer vision, machine learning platforms, robotics, data modeling, and local retrieval-augmented generation architectures. During the coming academic year, the cohort will receive training from Databricks experts and hands-on experience solving complex data challenges, while building a launchpad for future internship opportunities. The program invites applications for its next cohort in Fall 2026 and points readers to Databricks Free Edition.
Geospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3
Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST_* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions.
Azure Databricks at Data + AI Summit 2026 featuring Industry Leaders and Partners
Data + AI Summit 2026 brings Databricks and Microsoft leaders, partners, and customers together June 15–18, with in-person and virtual programming focused on Azure Databricks. The collaboration presents Azure Databricks as a first-party Microsoft offering for unifying data, analytics, and AI on a secure, scalable foundation, with sessions covering ecosystem integration, federated analytics, governance, modernization, and AI applications. One technical example introduces zero-copy federation between Azure Data Manager for Energy and Databricks compute, preserving ADME as the source of truth while avoiding large-scale data copies. Another shows Unity Catalog External Locations extending governed access to Microsoft OneLake without ETL pipelines, while customer sessions describe Apache Iceberg and Apache Spark integration, fragmented data consolidation, and production-grade finance workflows using Azure Document Intelligence.
Introducing Postgres to Postgres ClickPipes in ClickHouse Cloud
Postgres-to-Postgres migrations between providers can require manual schema handling, data movement, CDC, and cutover coordination, with risks around database objects and long-running loads. Postgres to Postgres ClickPipes, now in public beta, provides a fully managed migration service in ClickHouse Cloud designed to reduce operational overhead and support minimal downtime. It automates schema dump and restore, including secure connections, role conflicts, version differences, transactional application, retries, and UI or notification-based error reporting, while PeerDB supplies parallel initial loads and CDC capabilities. Independent pull and push processes use retries and alerting for transient errors, and the roadmap includes features to assist with cutover into Postgres by ClickHouse.
Empower your healthcare agents with ready-to-use MCP on Databricks Marketplace
Databricks announces ready-to-use Model Context Protocol (MCP) servers for healthcare and life sciences through Databricks Marketplace, addressing the need to combine curated biomedical knowledge, timely data, specialized tools, and private records. Listings include services for drug and target intelligence, literature, clinical trials, FDA information, Medicare coverage, ontologies, real-world evidence, clinical semantics, and interoperability, while Climb connects live public sources with private Gold-layer data under Unity Catalog governance. Marketplace and custom MCP servers are centralized in the MCP Catalog and governed by Unity AI Gateway, with Genie Spaces, AI Search, Unity Catalog functions, and SQL Warehouses available as managed MCP servers. Users can assemble agents in AI Playground, Agent Bricks, or notebooks, then deploy endpoints or apps with MLflow tracing, evaluation, human feedback, and AI guardrails; examples span molecular-property lookup, clinical questions, and drug research.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.