---
title: "Simplify AI agent orchestration with Lakebase Postgres"
description: "CLA and Databricks built a production document-processing application for auditing that reduces extraction time from hours to minutes without compromising quality, using Databricks-native services including Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes. Lakebase serves as the orchestration layer’s single source of truth for tasks and execution attempts, coordinating long-running work, retries, leases, priorities, rate limits, costs, and status visibility. The queue uses Postgres patterns including FOR UPDATE SKIP LOCKED, priority and FIFO ordering, expiring leases for crash recovery, and database-backed concurrency controls. Databricks Jobs process PDFs through intelligent document processing and vision/LLM calls, while MLflow Tracing records execution and cost details and dashboard updates combine fast Postgres data with slower billing queries. In production, this architecture avoids external brokers and schedulers while providing durable task management, real-time visibility, and per-task cost attribution."
---

# Simplify AI agent orchestration with Lakebase Postgres

[Databricks](https://yomu.fyi/company/databricks) · Li Yu, Michelle JanneyCoyle, Jon Cormack, Yarri Bryn, Alec Sorensen, Darshana Nair · Jul 22, 2026

**Type:** Problem & solution

## Summary

CLA and Databricks built a production document-processing application for auditing that reduces extraction time from hours to minutes without compromising quality, using Databricks-native services including Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes. Lakebase serves as the orchestration layer’s single source of truth for tasks and execution attempts, coordinating long-running work, retries, leases, priorities, rate limits, costs, and status visibility. The queue uses Postgres patterns including FOR UPDATE SKIP LOCKED, priority and FIFO ordering, expiring leases for crash recovery, and database-backed concurrency controls. Databricks Jobs process PDFs through intelligent document processing and vision/LLM calls, while MLflow Tracing records execution and cost details and dashboard updates combine fast Postgres data with slower billing queries. In production, this architecture avoids external brokers and schedulers while providing durable task management, real-time visibility, and per-task cost attribution.

## Context

Document parsing for auditing creates distributed-systems challenges including unpredictable task latency, model request and token limits, workload prioritization, per-task cost attribution, crash recovery, and real-time progress visibility. Combining separate queueing, orchestration, and observability systems adds infrastructure, authentication, monitoring, and integration overhead.

## Approach / What changed

The architecture uses Lakebase Postgres as the orchestration state store, with tasks and task\_attempts tables supporting concurrent priority-aware dequeuing, lease-based recovery, retries, throttling, and cost tracking. Databricks Apps provide the interface and worker dashboard, Lakeflow Jobs execute document parsing, Unity Catalog Volumes store PDFs, and MLflow Tracing records execution details.

## Takeaways

- FOR UPDATE SKIP LOCKED lets multiple workers dequeue tasks concurrently without selecting the same row, while priority ordering with created\_at preserves FIFO behavior within each priority level.
- Expiring leases and a periodic sweeper automatically re-enqueue tasks abandoned by terminated workers, supporting crash recovery without an external coordination service.
- Filtering billing data by submitted Databricks Job run IDs enables per-application and per-task cost attribution, including operator filters for dates, statuses, and agents.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Databricks](https://yomu.fyi/topic/databricks), [Lakebase](https://yomu.fyi/topic/lakebase), [Observability](https://yomu.fyi/topic/observability), [Postgres](https://yomu.fyi/topic/postgres)

- Source: [Databricks](https://www.databricks.com/blog/simplify-ai-agent-orchestration-lakebase-postgres)
- Source URL: https://www.databricks.com/blog/simplify-ai-agent-orchestration-lakebase-postgres
- Ingested by Yomu: 2026-08-30T16:53:46.222Z

[Read original post](https://www.databricks.com/blog/simplify-ai-agent-orchestration-lakebase-postgres)