Loading…
Topic
Lakebase
53 posts about Lakebase. Every summary links to the original.
- 53 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
The Rise of Sports Intelligence: How the Lakehouse Turns Tracking Data into Competitive Advantage
Professional basketball’s Hawk-Eye SkeleTRACK feed produces roughly 22,620 positional updates per second—about 65 million records per 48-minute game—yet teams often cannot turn that volume into timely, trusted decisions. The integration gap comes from separate vendors for tracking, wearables, video, scouting, and medical data, alongside calibration differences, weak provenance, and compute limits. The Databricks Data + AI Platform is presented as a governed lakehouse that ingests feeds with Lakeflow, refines them through medallion layers, and uses Unity Catalog for lineage, access control, and auditing. Models for shot probability, injury risk, and fatigue can run alongside serving and custom applications, with Lakebase supporting sub-second interactive queries. Applications include proactive load management, real-time coaching intelligence, and enriched broadcast or fan experiences across tracking-rich sports.
What if the answer was already in your data?
Kythera Labs is building an AI-native healthcare strategy platform on Databricks to give health systems access to strategic intelligence that historically required specialized analysts or consulting firms. Its foundation converts 339 billion medical and prescription claims covering more than 300 million patients into governed, event-based data, resolving providers, harmonizing codes across 130 vocabularies, and reconstructing patient journeys. Healthcare Strategy Agent, built with Agent Bricks, lets executives ask questions such as where oncology referrals are going and receive analyses of leakage, competing providers, physicians, and reimbursement opportunity in minutes. A Louisiana health system went live within ten days and reported 150% greater visibility into encounters, 12% more keepage, 22% less leakage, and $3.8 million in estimated annualized retained-encounter value. Unity Catalog, Lakebase, Delta Lake, Delta Sharing, and serverless infrastructure provide shared governance, lineage, access controls, and operational integration.
What’s coming next to Free Edition
Databricks is expanding Free Edition, launched twelve months ago to help people learn modern data and AI, after providing more than $10M in credits to over 500,000 learners. The five additions are Genie Code, serverless GPUs subject to availability, Lakebase, Agent Bricks, and Lakeflow Designer. Genie Code can generate, run, interpret, and refine code autonomously for tasks such as dataset analysis, pipeline cleaning, and visualization, while GPUs support deep-learning workloads including training, fine-tuning, inference, and large-dataset processing. Lakebase adds a fully managed Postgres-compatible database for data apps and AI agents; Agent Bricks supplies composable tools, memory, orchestration, and evaluation; and Lakeflow Designer enables visual pipeline construction. Together, these additions are presented as a complete free toolkit covering data engineering, analytics, machine learning, app development, and AI agents, allowing users to build end-to-end data and AI projects in Free Edition.
What’s new in Databricks Data + AI Platform security and compliance at Data + AI Summit 2026
At Data + AI Summit 2026, Databricks announced security and compliance capabilities for scaling Genie, Lakebase, serverless workloads, and AI-powered applications without relying solely on manual provisioning, static network controls, or siloed compliance programs. Automatic Identity Management (AIM) for Microsoft Entra ID is generally available on AWS and Google Cloud, AIM for Okta is in Public Preview, and Context-Based Ingress is in Public Preview across all three clouds for policies based on network source, identity, and access scope. Private Network Gateway, in Private Preview on Azure Databricks, provides one secure connection from serverless workloads to private networks, while expanded Private Link support extends to Lakebase and other services. Compliance additions include Azure Serverless coverage, HITRUST across AWS, Azure, and Google Cloud, ISMAP on Azure and AWS, expanded AWS GovCloud availability, and planned FedRAMP High support on Azure Commercial.
What’s New in the AI Platform: Agents for ML Engineering, Our Deep Learning Platform, and New Capabilities for Real-Time ML
The announcement presents three additions to the Databricks AI Platform: Genie Code support for ML engineering, AI Runtime’s serverless GPU environment, and expanded real-time ML capabilities. Genie Code integrates with Unity Catalog, Feature Store, training, serving, monitoring, and MLflow, assisting with feature engineering, model training, deployment, evaluation, and production operations. AI Runtime provides on-demand serverless NVIDIA A10 and H100 GPUs, supports high-performance multinode training with RDMA and high-performance data loading, and adds Lakeflow Jobs, DABs, MLflow, and Unity Catalog integration. For real-time ML, the platform adds declarative feature engineering, streaming features, online feature serving on Lakebase, and enhanced Model Serving targeting 300K+ QPS with under 10ms p99 latency overhead. Reported customer examples include faster workflows, lower infrastructure costs, and production scaling beyond 100K QPS.
Sciene AI Companion: building an autonomous Customer Success platform on Databricks
Sciene built AI Companion for Quartile’s Customer Success organization, where CSMs support more than 1,000 brands and previously spent substantial time preparing decks, reconstructing context, and investigating account changes. The platform addresses personalization at scale, high-volume content generation, and root-cause diagnosis by combining account data, CSM communication styles, company principles, and cross-domain business data. Its Email Hub cuts reply time from 15–30 minutes to about three minutes, Meeting Hub reduces preparation for 80+ slide decks from over two hours to around 10 minutes, and Account Flagging reduces diagnosis of flagged accounts from 30+ minutes to about five. Databricks provides the shared governed foundation: Delta Sharing supplies data without copies, Lakebase stores operational state, and SQL Warehouses serve analytical, AI, and operational workloads from the same tables. The design keeps CSMs responsible for judgment while giving them current context for customer interactions.
Announcing Lakebase Search: agent-native retrieval built into Lakebase Postgres
Lakebase Search is a beta offering on AWS and Azure that adds hybrid vector and full-text retrieval to Lakebase Postgres. It uses the lakebase_vector and lakebase_text extensions to keep retrieval, memory, operational data, and hybrid search in one backend. lakebase_vector retains pgvector types and operators, applies RaBitQ clustering and compression for 32x smaller indexes, and targets more than 1B vectors, while lakebase_text replaces GIN with object-storage-optimized BM25 ranking. A tiered cache keeps hot data on NVMe and places colder data in object storage; the source reports lower memory needs, faster index builds, and cold-cache startup than standard pgvector HNSW in its LAION-100M benchmark. The extensions also combine vector similarity and keyword relevance with reciprocal rank fusion in a single SQL query, enabling joins and tenant filtering alongside transactional workflows.
From Wall Street to Data Platforms
Kim Hatton, Databricks’ Global Financial Services Marketing Leader, describes how two decades in regulated financial-services marketing led her toward technology and data-centered strategy. She says financial institutions still need to unlock value while navigating compliance, but their divisions and systems make a unified customer view difficult. Her account points to Unity Catalog’s unified governance and single source of truth for breaking down silos and supporting requirements including GDPR, customer identity, and sensitive workloads. It also describes Lakebase’s separation of compute and storage for faster ML/AI agent experimentation, alongside Genie’s plain-language data analysis, which can reduce work that otherwise takes months and expensive third parties. Hatton connects these tools with faster, more confident marketing and accurate decision-making in regulated workflows, while also describing Databricks’ inclusive, high-energy culture.
Enabling Evolutionary Database Development: Database branching with Lakebase, the conclusion
The post concludes a series on how copy-on-write database branching in Databricks Lakebase changes team-scale evolutionary database development without changing its underlying methodology. For a team of fifty developers, long-running tier branches and ephemeral feature branches form a parent-linked promotion hierarchy, replacing separately provisioned environment instances and enabling promotion by merge, rollback by repoint, and computable schema divergence. Governance is declared once and inherited per branch, with policies intended to prevent transitions that contradict the parent chain; Unity Catalog captures metadata for attribution and audit. The DBA's role becomes platform engineering, while agents operate inside an executable SCM state machine with documented inputs, outputs, schema validation, and enforced gates. An optional TDD layer adds dedicated roles, acceptance-criterion scenarios, RED-GREEN-REFACTOR cycles, and artifact contracts, and the conclusion presents the resulting workflow as operational for human and agent practitioners.
How ERGO Hestia reduced time-to-market with Databricks Lakebase and Model Serving
ERGO Hestia redesigned its real-time pricing platform to reduce deployment friction across more than 100 models and 1,000 variables while preparing B2C capabilities. Previously, processed data moved from Databricks through extraction jobs, external Azure PostgreSQL, and a custom caching adapter, creating governance overhead, deployment coordination, and latency spikes during large refreshes. The new architecture uses Lakebase Sync Tables as an online serving layer and Databricks Model Serving Endpoints, keeping data, request logic, and model serving within the lakehouse; Unity Catalog supplies lineage, version tracking, access controls, and audit trails. An incremental migration started with a low-criticality endpoint, measuring 20ms latency and less than 5% CPU utilization at 40 requests per second, before expanding toward larger workloads and the planned decommissioning of PostgreSQL.
How Ecolab rebuilt retail intelligence on Databricks and Anthropic Claude
Ecolab needed to combine audits, health inspections, pest telemetry, and other data from nine systems so retail teams could answer location-specific compliance questions. Its Retail Intelligence application is a native Databricks App using Lakebase Postgres, Lakeflow, and Spark Declarative Pipelines to move governed data into a Unity Catalog lakehouse, while Foundation Model APIs serve Claude Sonnet, Claude Haiku, and Gemini. A Coordinator Agent delegates requests to specialized agents that use Vector Search, SQL, Unity Catalog Functions, and an external MCP server; a Response Agent returns cited answers, with short- and long-term memory stored through Lakebase. The system also applies five Judge LLMs, MLflow tracing, and ai_query() batch inference. Report preparation fell from two weeks to under two minutes, while the assistant supports approximately twelve languages at about 98% accuracy.
Enabling Evolutionary Database Development: database branching with Lakebase, continued
This installment revisits Evolutionary Database Design and argues that Databricks Lakebase removes the infrastructure constraints that kept several of its database-change practices aspirational. Lakebase is a managed Postgres database with compute separated from shared durable storage, while copy-on-write branches create a new pointer and divergence marker in roughly one second without copying parent data. That enables per-developer, per-PR, and per-experiment databases, real Postgres test branches instead of mocks or in-memory substitutes, and CI isolation at pull-request granularity. The updated playbook adds idempotent migrations, destructive testing, and database-level A/B prototyping, with Unity Catalog governance inherited by branches and agents receiving branches rather than production access. Jen’s example corrupts production-shaped data to test an inventory-code migration and compares column and lookup-table designs, rejecting the latter because its common read path requires a join.
Data + AI Summit 2026: Insider’s Guide for Financial Services Leaders
Data + AI Summit 2026 features a dedicated financial services program for leaders evaluating AI transformation across banking, payments, insurance, and professional services. It lists sessions on proprietary data for underwriting, responsible AI in banking and payments, and AI delivery in professional services featuring First American, American Modern Insurance Group, Vantage Bank Texas, Santander, FIS, Acxiom, EXL, Bain, and EY. The Financial Services Forum includes executive firesides with leaders from Morgan Stanley, JPMorganChase, Mastercard, and RBC Capital Markets. A financial services lounge at the Moscone Expo offers demos, Databricks experts, and Agentic Banker and Virtual CFO use cases. Training courses on AI Agents, Lakebase, and apps plus hands-on labs and certification sessions are presented as a route from strategy to execution with executives and technical teams splitting focus.
Enabling Evolutionary Database Development: database branching with Lakebase
Evolutionary database development treats migrations as first-class CI/CD artifacts, but shared development databases still limit per-developer isolation and fast, realistic feedback. The post follows Jen as she adds location, batch, and serial fields to inventory, coordinating application changes, production-data preservation, schema migration, and tests for storing, reading, and displaying the new values. It contrasts shared databases and simplified local substitutes with Databricks Lakebase copy-on-write branches, created in one second with zero storage at creation, including for a terabyte-scale production database. Jen pairs code and database branches, while CI creates a temporary branch, validates an idempotent and reversible migration, runs application and database tests, and posts a schema diff, enabling combined review and shifting DBA attention toward design and maintainability.
Introducing Always-On pricing: automatic savings for Databricks Lakebase
Databricks introduces Always-On pricing for Lakebase, offering a lower rate for baseline compute while retaining elastic autoscaling. On any Lakebase Postgres Autoscaling project, operators turn off scale-to-zero and set an autoscaling range; the minimum capacity becomes the baseline, which receives the lower rate after 24 hours of continuous use. Capacity above the minimum continues to autoscale up to the configured maximum and is billed at standard Autoscaling rates, while users can re-enable scale-to-zero later. The pricing targets established workloads with a consistent activity floor; intermittent or new workloads can retain scale-to-zero to avoid paying for idle hours and uncertain baseline estimates. The announcement says the baseline price is 25% lower, and an additional 50% promotional discount runs through January 31, 2027.
Announcing Lakebase Change Data Feed (CDF)
Lakebase Change Data Feed (CDF) is available in Public Preview to reduce the manual effort of moving data from operational databases into downstream systems. The feed is enabled once for all tables in a project, stored and governed in Unity Catalog Managed Tables, and readable by engines, models, and agents without separate extraction pipelines. From one shared feed, teams can build streaming pipelines with SDP, create materialized views with DBSQL, or compute and store embeddings with Agent Bricks, while consumers remain isolated from the primary operational workload. The announcement positions Lakebase as the native Bronze layer in a medallion architecture, complementing Synced Tables and providing governance and lineage across the data lifecycle.
Building a FHIR-native health data platform on Databricks Lakebase
Healthcare organizations often keep a FHIR server, analytics warehouse, and ETL pipelines separate, replicating clinical data and fragmenting governance. The proposed architecture standardizes incoming HL7v2, C-CDA, X12, and proprietary data into FHIR using Health Samurai converters, terminology normalization, MDM/MPI deduplication, and Implementation Guide validation. Aidbox, Health Samurai's FHIR server and database, runs on Databricks Lakebase, while Moonlink synchronizes operational and analytical formats without ETL. This exposes one governed dataset through Spark, SQL, ML, AI/BI, FHIR API, SMART on FHIR, and SQL on FHIR ViewDefinitions. The architecture supports use cases including EHR optimization, value-based care, member engagement, and compliance capabilities, with insights connected to clinician and billing workflows through SMART on FHIR and CDS Hooks.
Unlock seamless and cost-effective marketing campaigns with Lakebase
Retail marketing teams often store customer segments in OLTP databases, creating underused capacity between campaigns and synchronization work whenever new segments are requested. Lakebase, Databricks’ implementation of a lakebase architecture, separates storage from compute and uses elastic, serverless Postgres that can scale to zero when idle and up for bursty campaign traffic. The integration with the Lakehouse uses managed Synced Tables for customer segments and Lakehouse Sync for continuous CDC-based replication of operational data into Unity Catalog Delta tables. The SAP Engagement Cloud example configures a Lakebase Autoscaling project, native Postgres credentials, an ISRG Root X1 certificate, and snapshot synchronization for datasets with more than 10% updated. The post concludes that this design lowers idle and sizing costs, reduces pipeline maintenance, and supports low-latency point lookups, while warning that Lakebase is not optimized for large scans or classic OLAP.
How to Build Real-Time Fraud Detection using Spark Real-Time Mode and Lakebase
Card fraud operates in seconds, while organizations need to score and block suspicious transactions before settlement without adding a separate streaming stack. This Solution Accelerator presents an open-source, end-to-end implementation on Databricks, covering transaction ingestion, Spark Real-Time Mode processing, fraud scoring, Lakebase feature serving, and a Databricks Apps monitoring dashboard. RTM processes streaming workloads in milliseconds and targets sub-300ms latency, while Lakebase supplies a managed, serverless Postgres serving layer with sub-millisecond reads. The design keeps batch ETL, training, streaming, and governance within the same platform. Deployment progresses from a synthetic rate-source quick start to Kafka-based parsing, stateful per-card velocity tracking with TTL, enrichment, weighted scoring, routing, and optional MLflow-trained RandomForest scoring, followed by a Streamlit app for live decisions.
Databricks context engineer associate: the industry’s first certification for reliable AI agent systems
Databricks announces the Databricks Certified Context Engineer Associate, described as the industry’s first certification purpose-built for context-aware AI systems. It assesses whether practitioners can design, assemble, and govern the information AI agents receive at inference time, including system prompts, retrieval, memory, external tools, context-window management, and governance. Candidates work with Databricks technologies such as AI Search, Lakebase, MLflow, MCP, and Unity Catalog, while advanced coverage includes multi-agent systems, long-horizon workflows, and evaluation of context decisions. The beta exam will be available free to Data + AI Summit attendees, who may take it once; beta results are expected in 6–8 weeks.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.