From Ringing Line to Resolved Contact: How a Call Center Works
Follow one contact through a call center: arrival, routing, handling by a human or AI agent, resolution, and the cost-per-contact math that decides which path to use.
A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity.
A circuit breaker wraps calls to a dependency and watches errors over a time window. When the error rate crosses a threshold, it opens and returns a fallback or fast failure instead of letting slow calls pile up. After a cooldown it moves to half-open, allowing limited probes; enough successes close it, while a failed probe reopens it. The practical trap is that half-open probes still occupy worker capacity, so timeout and cooldown settings must be tuned together rather than treated as independent knobs.
When a breaker opens, half-open probes can still be allowed to run until they time out. On a small worker pool those timed-out probes occupy capacity, so the dependency outage becomes a local capacity outage; tuning probe timeout and cooldown together limits that waste.
A fixed error count does not scale cleanly as traffic or instance count grows. Heroku’s CB2 uses percentage-based thresholds so the breaker trips on error rate rather than an absolute number; either way, align the threshold with the error window you actually observe.
Start in logging-only mode so parameters and unexpected internal-API behavior can be reviewed before traffic is affected. Then monitor state changes, guard against duplicate notifications from race conditions, and set explicit low network-level timeouts so slow dependencies fail fast.
The posts behind it
Fly.io responded to CVE-2026-99999, a Linux kernel bug in IPv4 checksum-offload metadata validation, by patching and rebooting its entire fleet over 10 hours on October 16, 2026. The bug allowed a guest to rewrite two bytes of an IPv4 header after the host applied routing and firewall rules, enabling firewall bypass and an out-of-bounds read of up to 32 bytes of host memory. The team split each region into waves so no Managed Postgres cluster had both nodes rebooted simultaneously, with the largest region requiring 17 waves. Some hosts failed to return cleanly, including a chassis in gru that lost its power supply, delaying recovery for those machines until October 20. The post-mortem led to a new Incident Communications Plan that shifts communication decisions from incident responders to the exec team.
As a Databricks Apps application scales from a few users to thousands across an organization, it must handle growing demand without latency spikes or outages. The guide proposes a four-stage approach: set targets during design, build with async code and recoverable state, test with load simulations, and operate with monitoring. Keep app compute focused on serving the UI; send heavy work to Lakebase, Databricks Jobs, or Sandbox, and choose a framework like AppKit that offers built-in caching and telemetry. Separate session state into soft state (rebuildable from hard state) and hard state stored durably in Lakebase or Unity Catalog so any instance can reconstruct a session after restarts or scaling. Use load tests to validate concurrency and bursts, then consult system metrics on CPU, memory, timeouts, and connections to decide whether to scale up to a larger instance or scale out with more instances.
Misconfigured circuit breakers can leave an application effectively unavailable during a dependency outage by allowing timed-out requests to consume worker capacity. Using Shopify’s Semian implementation, the post explains how name, error_threshold, error_timeout, half_open_resource_timeout, and success_threshold govern circuit opening, recovery, and wasted utilization across failing service instances. It models timeout spikes for a single worker, notes that separate names isolate services and instances, and presents an equation for steady-state additional utilization, while noting that context-switch costs are not included. A live test closely matched the equation’s prediction. In a Rails worker with two threads and 42 Redis instances, changing half_open_resource_timeout from 0.25 seconds to 50ms and error_timeout from 2 to 30 seconds reduced modeled extra utilization from 263% to 4%, with slower recovery; an author’s edit says success_threshold does not affect steady-state utilization.
Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.
More like this
Follow one contact through a call center: arrival, routing, handling by a human or AI agent, resolution, and the cost-per-contact math that decides which path to use.
Cilium uses eBPF programs in the Linux kernel to enforce identity-based network policies and capture L7 observability for Kubernetes workloads. This diagram traces the path from packet arrival to policy verdict and Hubble flow.
Build vs. buy is a per-layer decision, not a single platform choice. This flow splits a system into layers, tests each for differentiation, then sets the buy criteria and the exit path.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.