DatabricksDatabricks StaffExplainer
The 40-year-old database rule agents just broke: How LTAP unifies OLTP and OLAP workloads
Summary
LTAP (Lake Transactional/Analytical Processing) addresses the long-standing separation between operational row-oriented databases and analytical column-oriented systems, which traditionally required pipelines and copied data. It uses Lakebase’s stateless, ephemeral compute decoupled from durable lake storage to expose live operational data for analytical queries without moving it or loading the transaction-serving system. The storage layer preserves Postgres data types and physical representation while writing data into Parquet, with a hot row-format tier for operational access and a cooler columnar tier for analytical reads; Apache Spark and SQL can read that representation directly. By separating serverless operational and analytical compute, LTAP allows each workload to use specialized engines and independent capacity, while a unified catalog applies governance across both. The stated motivation is agentic workloads such as fraud detection, where stale copies miss fast events and uncontrolled analytical queries can degrade transactions.
Context
Operational and analytical workloads use different storage layouts and query patterns, so they have traditionally run in separate systems connected by pipelines and copied data. AI agents need fast decisions from both current operational and historical analytical data, while heavy analytical queries can interfere with transaction processing and stale copies can miss time-sensitive events.
Approach / What changed
LTAP unifies transactional and analytical data in one logical storage layer. Lakebase separates stateless, ephemeral compute from durable lake storage, preserves Postgres data representations in Parquet, and maintains hot row-format and cooler columnar tiers. Separate serverless compute serves each workload, while a unified catalog governs access across the data.
Takeaways
LTAP runs analytical queries against live operational data without moving it or placing analytical load on the transaction-serving system.
The storage layer preserves the exact physical representation of Postgres data while exposing it through Parquet, with row and columnar tiers for different access patterns.
Separate serverless compute for operational and analytical workloads avoids the cost and operational constraints of one engine handling both jobs, while unified catalog policies govern the shared data.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.