Loading…
Topic
Lakebase
52 posts about Lakebase. Every summary links to the original.
- 52 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Clinical operations intelligence belongs on the Lakehouse
The Site Feasibility Workbench addresses fragmented clinical operations data by placing a site-selection decision-support application, its models, and data on Databricks. Released as a fully open-source Databricks App, it combines a FastAPI backend, React frontend, Unity Catalog-governed data, Lakebase operational state, and AI/BI Genie within the workspace. TA-segmented LightGBM models use the organization’s CTMS, EDC, and IRT history alongside real-world evidence, patient access, qualification, Open Payments, and protocol-execution signals to rank sites. Each prediction stores SHAP attributions in a governed Unity Catalog Delta table, while MLflow tracks training runs and model versions; saved shortlists persist to Lakebase and Genie answers governed-data questions. The six-step workflow supports protocol selection through final shortlisting, including diversity considerations; the release is a decision-support layer rather than a source of record, and deployment reportedly takes about 30 minutes before security review and validation.
MCP Marketplace brings real-time intelligence to agentic applications
The MCP Marketplace connects agentic applications with real-time external intelligence alongside enterprise data. The problem appears in use cases such as loan approval, where historical internal records omit market conditions, updated credit signals, property changes, and competitor activity, making manual research a bottleneck. Databricks Marketplace provides governed access to MCP servers from You.com, Moody’s, and Cotality, while Unity Catalog authenticates connections and tracks access and lineage; Lakebase stores state, decisions, and audit trails across multi-step workflows. Examples show agents combining internal data with web research, credit ratings and sector outlooks, or property-resolution and mortgage signals before surfacing decisions for human review, including a commercial-loan flow with recorded sources, timestamps, and approver.
How nOps rebuilt their cloud optimization platform on Databricks Lakebase, and why other ISVs should too
nOps migrated its production cloud optimization platform to Databricks Lakebase, a fully managed PostgreSQL database integrated with the Databricks Lakehouse. Its previous setup separated customer-facing operational data from analytics, requiring scheduled jobs, cron-based change-detection logic, and synchronization work that introduced lag and operational overhead, especially as workloads expanded to GCP and Azure. In the new architecture, Lakebase is the central Postgres database and single source of truth for the front end and AI infrastructure, while the Lakehouse continuously consumes its data for analytics and metric computation. Databricks Metric Views are automatically surfaced in the application, and data flows one way without direct write-back. The reported results are faster data pipelines, lower operational overhead, and a better customer experience, with auto-scaling, auto-stop, restore, OAuth, and compatibility features supporting adoption.
Backstage with Lakebase
Backstage with Lakebase describes a proof of concept that moves Backstage’s stateful internal developer portal from standard Postgres to Databricks Lakebase, a serverless Postgres surface in the Databricks Data + AI Platform. Connection required an app-config.yaml change, PgSearchEngine instead of the default in-memory search, and short-lived OAuth JWT credentials generated with databricks postgres generate-database-credential; Knex migrations then ran cleanly. Using copy-on-write branches, the team cloned a roughly 63 MB catalog in 1.09 seconds and recovered 32 deleted entities in 3.78 seconds, while observing that point-in-time recovery snaps backward to the nearest WAL record. The post argues that cheap, isolated database state can replace much mocking and staging iteration with real-data development, branch-based testing, schema validation in CI, and automated cleanup, while noting that VS Code/Cursor synchronization tooling remains underway.
Agents are ready, but your architecture probably isn't
Enterprise AI initiatives often produce activity rather than value because organizations start with technology instead of a defined outcome and overlook data architecture, governance, and semantic context. Agentic systems add risk when they can send messages, update records, place orders, or delete records, making permissions and situational controls essential. The discussion argues that dashboards and batch pipelines are poorly matched to low-latency, high-scale agent workloads, which require transactional infrastructure alongside existing analytics. Lakebase is presented as that transactional foundation, while AgentBricks, Databricks Apps, and Genie provide agent development and monitoring, application delivery, and conversational data access. The recommended path is to define success first, isolate a focused pilot, learn what works, and redesign underlying processes rather than merely add AI to them.
Operationalizing AI for public sector fraud prevention
Public-sector agencies face fraud methods including synthetic identities, deepfake-enhanced documentation, and personalized social engineering, while legacy controls remain fragmented and difficult to scale. The fictional Services Bureau demonstrates an operating model that combines Databricks Apps, Lakebase, Unity Catalog, Delta Sharing, Agent Bricks, and AI/BI Genie in a single fraud-operations environment. Governed data lands in Delta tables; Unity Catalog applies attribute-based access control, masks PII by role, and provides lineage, while agents connect live lakehouse queries, agency policies, and external fraud signals through MCP. Analysts review evidence and recommendations, then approve, override, or escalate cases, keeping human judgment central. The described workflow turns weeks of manual investigation into a day, supports dashboards and conversational SQL, and is presented as making fraud decisions faster, more secure, transparent, and defensible.
How leading tech companies are killing the builder's tax with Lakebase
The post argues that AI-native applications face a “builder’s tax” because operational databases and lakehouse analytics are separated by CDC, ETL/ELT, and reverse ETL pipelines. This architecture can leave AI systems with stale data while duplicating governance and shifting engineering effort toward orchestration and failure management. It presents Lakebase, a fully managed serverless Postgres engine integrated with the Databricks Data + AI Platform, as a shared operational foundation where applications, agents, analytics, and governance use the same data. The proposed pattern combines a lakehouse intelligence layer, a low-latency Lakebase execution layer, and a continuous learning loop, with pgvector supporting agent memory and semantic search. Reported customer examples include faster feature onboarding, reduced on-call disruption, higher agent throughput, and consolidated data estates, though the results vary by organization.
Inside one of the first production deployments of Lakebase: LangGuard's agentic workflow governance engine
LangGuard is presented as a runtime enforcement layer for enterprise agentic workflows, monitoring actions, decisions, tools, credentials, and intent across connected systems. Its GRAIL data fabric records multidimensional trace data, builds a live knowledge graph, and evaluates allow/deny/modify decisions against policy before tools, datasets, or models are accessed. The deployment uses Databricks Lakebase as the operational system of record, relying on PostgreSQL, serverless autoscaling, scale-to-zero, compute-storage disaggregation, hot-data caching, and copy-on-write branching for policy testing. LangGuard chose this architecture to handle bursty trace writes and low-latency enforcement reads without provisioning for peak demand, while keeping operational data available to Databricks analytics and AI capabilities without additional ETL. The stated next step is predictive governance: training behavioral models on historical traces to flag anomalous agent behavior before a policy violation.
How conversational analytics removes the BI bottleneck
The interview presents conversational analytics as a way to move beyond fixed dashboards and turn governed data into decisions and workflows. Databricks’ Genie lets nontechnical users ask open-ended questions in plain language across Lakebase, Lakehouse, and federated sources, while Unity Catalog applies shared definitions and access controls. Lakebase provides transactional storage, instant provisioning, environment forking without copying underlying data, and scale described as billions or trillions of records; examples include real-time matchmaking, routing, and purchase decisions. The discussion stresses that human oversight, refreshed data, business context, and controls against hallucination remain necessary, citing Fox Sports’ chatbot deployment as an example. It concludes that organizations combining trusted semantics with operational data systems can make faster decisions and automate work, while the competitive gap may compound for slower adopters.
AI App Development: Guide To Building AI-Powered Apps
Production-grade AI app development combines model orchestration, transactional state, governance, security, and live-data integration rather than only interface construction. The guide recommends defining users, outcomes, success metrics, timelines, and AI-relevant journeys, then shipping must-have features before enhancements. It evaluates builders by end-to-end support, technical audience, platform integration, code export, CI/CD, data connections, pricing, and deployment limits. For Databricks teams, it describes Databricks Apps, Declarative Automation Bundles, and Lakebase: serverless app compute, version-controlled deployment, and managed PostgreSQL with synced Unity Catalog Delta tables. It also recommends monitoring outputs, collecting correction feedback, refining prompts with automated evaluations, and governing models through MLflow, concluding that integrated infrastructure can leave teams focused on production AI workflows.
Take Control: Customer-Managed Keys for Lakebase Postgres
Lakebase Customer Managed Keys (CMK) gives regulated enterprises control of the root of trust through customer-owned keys from AWS KMS, Azure Key Vault, or Google Cloud KMS. Because Lakebase separates persistent storage from elastic Postgres compute that can scale to zero, CMK protects both layers and their caches rather than storage alone. Its hierarchical envelope-encryption model uses the CMK as the root, transient Key Encryption Keys (KEKs) to wrap Data Encryption Keys (DEKs), and unique DEKs for each data segment, while plaintext CMKs remain in the customer’s KMS. Account Admins bind a configured key to a workspace, after which new projects inherit the CMK; rotation needs no re-encryption or downtime, revocation terminates compute instances, and audit services record cryptographic operations.
Building real-time product search on Databricks
Real-time product search for an online marketplace must retrieve, filter, rank, and respond in milliseconds while balancing latency, relevance, revenue, and click-through rate. The Databricks architecture divides the pipeline into ingestion, retrieval, and refinement, using Lakeflow pipelines and AI Functions for source processing, AI Search for embeddings and hybrid retrieval, and Lakebase for sub-10ms operational signals. It recommends validating p95/p99 latency, throughput, retrieval quality, and engagement metrics, with MLflow supporting systematic evaluation and caching helping meet strict latency requirements. As an example, FOX Sports used AI Search for thousands of QPS and achieved a 2x improvement in query success rate, combining streaming ingestion, two-phase retrieval, model serving, caching, and trending searches.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.