Circuit Breakers: The Recovery Loop That Spends Capacity
A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity.
DatabricksBruno Tafur, Tushar MadanTutorial
As a Databricks Apps application scales from a few users to thousands across an organization, it must handle growing demand without latency spikes or outages. The guide proposes a four-stage approach: set targets during design, build with async code and recoverable state, test with load simulations, and operate with monitoring. Keep app compute focused on serving the UI; send heavy work to Lakebase, Databricks Jobs, or Sandbox, and choose a framework like AppKit that offers built-in caching and telemetry. Separate session state into soft state (rebuildable from hard state) and hard state stored durably in Lakebase or Unity Catalog so any instance can reconstruct a session after restarts or scaling. Use load tests to validate concurrency and bursts, then consult system metrics on CPU, memory, timeouts, and connections to decide whether to scale up to a larger instance or scale out with more instances.
Developers building Databricks Apps often face performance and cost challenges when usage grows from a few users to thousands. This guide provides a structured approach to scaling, covering architecture decisions, framework choices, state management, and testing across four stages.
A four-stage methodology: design (set latency and peak-user targets, choose architecture and framework), build (write async code, separate soft and hard state), test and deploy (run load tests with concurrency and bursts), and operate (monitor system metrics, use Insights for bottlenecks, scale up or out based on data). Key recommendations include using AppKit, offloading heavy work to dedicated services, and connecting to Lakebase with a budgeted connection pool.
Before writing code, define performance targets (p95 latency) and scalability targets (peak concurrent users) to shape architecture decisions and validate during testing.
Keep app compute focused on serving the UI; move heavy computations, long-running tasks, and untrusted code to dedicated services like Lakebase, Databricks Jobs, or Sandbox.
Split session state into soft state (temporary, rebuildable from hard state) and hard state (durable in Lakebase or Unity Catalog) so any instance can recover sessions after restarts or scaling events.
Scale it in four stages: design, build, test and deploy, and operate. Set latency and peak-load targets first, keep app compute for serving the UI, and write async code with recoverable state. Then let load tests and telemetry decide whether to scale up or out.
It depends on the limit hit first. Scale out by adding instances when requests arrive in bursts or a single-process app is bound to one core. Scale up when the app needs more memory or can use extra cores with multiple workers.
Split session state into soft state (rebuildable) and hard state (durable in Lakebase or Unity Catalog). Databricks Apps keeps session affinity on a best-effort basis, but any instance must be able to rebuild sessions from hard state.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity.
Actionable banking wires real-time event triggers, unified customer context, and agent memory into one pipeline that ends at a human decision, not an automatic transaction.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.