Loading…
Topic
5 posts about Load Balancing. Every summary links to the original.
See the idea
A five-checkpoint workflow for assignment problems: write the spec, name the objective, test whether choices interact, pick a solver class by scale, then validate against live state.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
ZGateway is a stateless proxy tier introduced to unify traffic from ZippyDB’s large, diverse client fleet and address the reliability limits of direct access. By terminating TLS, authorizing requests, applying admission control and shaping, and routing through regional tiers, it replaces a many-to-many connection mesh with bounded client-to-gateway and gateway-to-server hops. Its embedded thick C++ client batches and coalesces shard requests, while a shared pipeline supports pure-proxy and read-through-cache tiers; per-use-case metrics, traces, and quota usage provide operational visibility. The gateway handles more than 1 billion operations per second, carries about 40% of ZippyDB traffic, and adds about 6% computational overhead to an average use case, with traffic projected to exceed 60%; production work also consolidated transaction state handling across 100% of transaction traffic without a reliability regression.
The 2026-07-28 Model Context Protocol specification release candidate addresses production bottlenecks caused by the earlier session-oriented HTTP model, including session pinning, sticky routing, lost state, and Redis-backed infrastructure. It removes the initialize/initialized handshake and Mcp-Session-Id, carrying protocol version, client information, and capabilities in _meta on each request while exposing method and name through HTTP headers. The update also introduces ttlMs/cacheScope caching, Multi Round-Trip Requests for elicitations and task progress, issuer and resource-indicator verification, and full JSON Schema 2020-12 support. The post says the stateless core enables ordinary load balancing, serverless deployment, and transparent failover, while SDK beta releases for TypeScript, Python, Go, and C# support migration.
Real-time AI agents require infrastructure that manages continuous, bidirectional conversations rather than isolated requests, including interruptions that halt speech generation and update context without dropping the connection. The post argues that QPS and instantaneous CPU utilization are insufficient because they miss long-lived committed work and silent sessions that can become active simultaneously. It proposes application-level active-session tracking around the stream lifecycle, with cleanup in finally blocks and consistent snapshots, then combines session pressure with CPU and memory utilization in a hybrid routing model. The model normalizes session counts into a rate and uses target utilization, smoothed utilization, cost per session, and a safety scaler to estimate effective capacity and regulate new assignments. Validation should use long-lived, varied workloads and metrics such as session distribution, overloaded assignments, startup latency, dropped sessions, and counter behavior; high-concurrency trackers may require sharded or LongAdder-style aggregation.
Heroku’s migration of Rendezvous, the bidirectional proxy that connects Heroku Run sessions to one-off dynos, from EC2 Classic to AWS VPCs caused one customer’s app to hit H12 timeouts. The move replaced a single Elastic IP ingress and process-level TLS termination with a Network Load Balancer, whose larger TLS frames exposed a blocking flaw in Rendezvous’s single-threaded Ruby channel loop. That loop used IO.select on the TCP socket and dyno terminal, but OpenSSL::SSL::SSLSocket could retain unread buffered data after a partial read; IO.select saw no TCP bytes and blocked beyond the Heroku Router’s 30-second limit. Engineers reproduced the behavior across two languages, instrumented the instance agent, then fixed the channel with separate threads and corrected socket-reading and IO.select handling. After staging tests, the customer moved back to VPC Rendezvous and confirmed the issue was resolved.
Fly’s 1 May 2020 changelog records enhancements to flyctl, a new Open Source page, and a revised load-balancing allocation algorithm. Version 0.0.117 added flyctl auth docker, allowing deployments of images directly to Fly by adding registry.fly.io to Docker’s authenticated registries, and introduced --image-label for selecting a specific image label at deployment time. The release also increased the timeout at the end of deployment monitoring to prevent erroneous “No Deployment” messages and changed deploy version negotiation to resolve API compatibility errors. For global applications with large numbers of instances, the new algorithm directs each incoming connection to an instance from the closest least-loaded service among three randomly selected healthy services in the app’s three nearest regions.
The best engineering reads, once a week.
