Loading…
Topic
378 posts about Scalability. Every summary links to the original.
See the idea
Consolidating agent memory, retrieval, and governance onto MongoDB Atlas removes sync and vendor layers, but ties scaling and control to one platform. This trade-off view shows the payoff, the cost, and the condition that decides it.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
At Anthropic, Claude’s ability to iterate on code at high speed has caused git-based source control to become the bottleneck in their development workflow. To handle the growth from 2,500 to over 30,000 daily code merges, the team built Scion, an in-house version control system, fully migrating their monorepo from git in five months. Scion diverges from git by assuming clients are always online, shifting computation to the server, and using horizontally scalable data structures without any fully materialized copy of the repository. The system stores changes as entries in a write-ahead log, supports sparse and virtualized checkouts, and enables cheap, lock-free commits that scale with change size rather than repository size. Anthropic shares Scion’s design principles to help the broader ecosystem prepare for AI-driven scaling challenges, noting that similar pressure points will emerge for other organizations as AI coding tools become more common.
Margaret Gould Stewart, Facebook’s Director of Product Design, discusses the challenge of designing products for over a billion users, citing Google Search, YouTube, and Facebook as examples of universal human needs. She emphasizes balancing audacity with humility, taking oneself out of the design process to focus on user needs. User research remains critical, using both quantitative data and ethnographic studies across different cultures and regions, with products tested extensively before release. For business products, usability must come before beauty, but delight on top of a usable foundation is powerful, as seen in consumer expectations. Stewart also highlights the importance of diverse teams and inclusive hiring programs like Facebook University to better represent the global user base.
Cyber Week compresses a year’s worth of system failures into 72 hours, making it the ultimate stress test for communication infrastructure. Developers must optimize APIs with rate limiting, caching, and traffic shaping to absorb sudden spikes without dropping connections. Email delivery requires early IP warmup, SPF/DKIM/DMARC configuration, and limiting promotional sends to recently engaged customers to maintain inbox placement. Voice pipelines need proactive capacity planning for Calls Per Second quotas, fallback URLs, and STIR/SHAKEN attestation to prevent fraud and dropped calls. Monitoring should use a unified dashboard with per-channel metrics, correlation IDs, and pre-written runbooks rehearsed in game days to turn inevitable failures into routine events.
Intercom published its third book, Intercom on Customer Support, sharing principles used while scaling to over 8,000 customers across 85 countries. The book examines customer support from three angles: starting and scaling a team, planning the quality of support, and ensuring support has positive relationships with other teams. It argues that quality customer support is a competitive advantage, not a cost, aligning with Forrester’s “age of the customer” concept from 2011. Two key macro trends—all businesses becoming internet businesses and the shift to recurring revenue—place the future lifetime revenue of the customer at the center of every support discussion. As boundaries between sales and support dissolve, the book addresses challenges like supporting free users, scaling the team, and controlling costs, while remaining tool- and tactic-agnostic.
In this 2021 year-end compilation, Intercom reflects on insights from its podcast series Inside Intercom, featuring conversations with business leaders and pioneers like CEO Karen Peacock, Hubert Palan from Productboard, and Amanda Renteria from Code for America. They discuss leadership, product strategy, customer experience, and the challenges of running a business during the pandemic, including building world-class customer experiences and addressing diversity gaps. Speakers like Farhan Virji from Later share how B2C support strategies can inform B2B teams, while Will Larson of Calm explores technical leadership careers beyond management. The episode also marks Intercom’s 10th anniversary, with founders Des Traynor and Paul Adams reflecting on maintaining speed as the company scales and avoiding unnecessary processes that slow progress. Zanade Mann and Janeen Uzzell offer insights on allyship, diversity, and ensuring knowledge equity in government services, as well as the importance of advocating for underrepresented groups.
Companies scaling their customer service should first ask whether their tools are scalable, as legacy channels like phone and email are labor-intensive and difficult to scale. A next-generation messenger-based support solution can improve efficiency by collecting information upfront, enabling reps to handle multiple chats, and using bots for simple queries. Second, teams must maintain high standards as conversation volumes rise by using automation to resolve repetitive questions, freeing agents for complex issues. Third, businesses should adapt to evolving customer needs by incorporating self-serve and proactive support layers, reducing inbox volume and allowing focus on high-value VIP customers. Laying this infrastructure early can result in significant time and cost savings while improving customer satisfaction.
As agentic applications move from answering questions to taking actions, enterprises risk AI sprawl from duplicated integrations, inconsistent policies, and fragmented context. Databricks asserts scaling requires three pillars: choice across models and tools, context grounded in governed enterprise data and business semantics, and control through consistent permissions, policies, observability, and cost management. Its platform provides Agent Bricks for unified agent fleet building and governance, Omnigent as a common layer above agent harnesses, and Unity Gateway as a centralized control plane for models, agents, and tools. Context is shared via Unity Catalog for data and AI asset governance and Genie Ontology for common business concepts, letting teams reuse governed context across applications. Control uses Unity Gateway for access policies, budgets, and traces and Databricks Sandbox for isolated execution, with MLflow enabling evaluation and workflow inspection to maintain reliability as agents scale.
AI-driven demand is constant, unpredictable, and extreme, unlike earlier internet and cloud eras where spikes were more predictable. Enterprises need a data layer that can absorb sudden bursts of traffic without re-architecture. MongoDB announced Atlas Infinite, a new deployment option within MongoDB Atlas that independently scales compute and storage within minutes. Atlas Infinite supports replica sets with 10x the data capacity of Atlas Core and delivers 189% more throughput per dollar in internal tests. MongoDB 9.0 was also released, offering up to 35% faster find-one queries and new security features like queryable encryption with prefix, suffix, and substring searches.
Red Hat conducted a performance and scale test of the argocd-agent pull architecture for GitOps, using 42 physical machines, 500 spoke clusters, and software including OpenShift 4.21 and OpenShift GitOps 1.20. The pull model decentralizes reconciliation to lightweight agents on each managed cluster, which pull configurations from a central hub, improving security by eliminating inbound administrative ports and reducing cross-cluster network traffic by up to 80%. In the bootstrap scenario, generating and syncing 31,200 applications across 480 clusters completed in 13 minutes 3 seconds, while the sync-to-commit scenario, which includes a polling delay, took 13 minutes. The test revealed bottlenecks including Kubernetes API limits on the hub, informer processing slowdowns causing watch restarts, event writer retry issues under load, and ApplicationSet controller RAM consumption. The team also deleted all 31,200 applications in 4 minutes 53 seconds using cascading deletion, and the insights are informing future OpenShift GitOps releases.
Meta has open-sourced Rebalancer, a generic library for solving assignment problems that has been used internally for over nine years. Rebalancer separates problem specification from solution, using a three-step approach with modeling constructs like dimensions, partitions, scopes, and utilization, and a high-level spec API for common objectives and constraints. Problems are translated into an expression graph, and Rebalancer offers two solving techniques: an optimal solver using mixed integer programming (MIP) with commercial or open-source solvers, and a local search solver that evaluates millions of moves per second for large-scale problems. The local search solver handles problems with over a million objects and 5k bins, with an average solve time of 171 seconds. Rebalancer is used for roughly 40 million assignment problems daily across Meta’s infrastructure, and the open-source release includes a debugging UI called Rebalancer Explorer. The accompanying paper was published at OSDI’24.
A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.
Spotify examines how AI-accelerated development and broader industry compute demand challenged quality across content processing, fleet updates, regional failover, and the mobile app. At its scale—777 million monthly active users, about 100 million concurrent clients, 11–12 million backend requests per second, and nearly 3,000 production services—small control gaps can become visible to listeners and creators. A June 24 processing incident combined competing batch work, higher per-episode compute, and a scheduler bug that cut throughput about 10%, leading to hour-long publication delays; Spotify responded with end-to-end monitoring, scheduler fixes, prioritization, and more capacity. Incident reviews found no material direct contribution from AI-authored code, but change volume outpaced some verification controls, prompting stronger review, testing, rollout, observability, rollback, quality signals, and failover practices. Spotify reports no corresponding rise in rework rate, while continuing to watch increasing PR size and complexity.
Uber’s failover redesign exposed a limitation in its long-standing single-controller model for scaling stateless workloads. To reuse idle capacity during regional failover, the platform needed multiple orchestrators to influence replica counts without overloading UDC, which remained on the normal deployment lifecycle path. The solution introduced the ServiceScale CRD and Service Scale Controller, allowing separate actors to record steady-state and temporary failover intent in Kubernetes and letting SSC reconcile the combined decision. Production work then exposed risks from informer caches that lagged reality and near-simultaneous multi-writer updates, including ReplicaSet metadata-spec drift that disrupted proportional scaling and could leave workloads stuck. Uber added fleet-wide detection, an automated UDC healer, a long-term scaling-path fix, and extensive kind integration tests, staging, and canaries; the year-long rollout avoided customer-impacting outages while establishing a more extensible control plane.
Cursor’s self-hosted machines keep tool execution inside an organization’s own network, leaving the codebase, build outputs, and secrets on internal infrastructure while the agent handles tool calls locally. The feature supports My Machines for connecting one laptop or VM and team pools that provide named queues of workers, with capacity expanding as requests arrive and contracting when workers disconnect. Pools are repository-independent, can hibernate idle machines, and can restore them within a reconnect window, avoiding the need to keep costly capacity warm. Cloud agents can run on infrastructure including AWS Lambda, Coder, Cloudflare, Daytona, Modal, Namespace, Vercel, and E2B, while Linux and Mac workers add computer use for clicking, typing, screenshots, and browser control.
Salesforce built DREAM (DDoS Response and Mitigation) to defend its distributed cloud against hyperscale, machine-speed DDoS attacks, with a stated goal of responding in seconds while protecting more than 4.5 million domains and 3 million customer orgs at 99.99% availability. The platform targets rapidly evolving application-layer attacks in a shared, multi-tenant environment by combining AI-driven inference with distributed orchestration for telemetry, policy deployment, notifications, retries, and fault-tolerant execution across regions. The team selected Temporal rather than rebuilding distributed-systems primitives, and used AI coding assistants to migrate about 100,000 lines at roughly ten times prior velocity; more than 90% of the rewrite was AI-assisted, with human reviews and test coverage above 95%. Production lessons moved bulk data outside orchestration and bounded workflow history, while DREAM reduced time to first mitigation fivefold.
The article examines whether ClickHouse is “winning” observability by separating the storage-and-query question from the broader work of collection, schema design, correlation, visualization, alerting, and investigation. It argues that ClickHouse fits telemetry because its columnar layout stores fields separately, enabling compression and allowing analytical queries to read only selected columns across millions or billions of events; ordering keys, sparse primary indexes, and data-skipping indexes further reduce work. The discussion covers both rich, wide events and conventional logs, traces, and metrics, and presents ClickStack as a combination of ClickHouse, an OpenTelemetry Collector, and a UI that preserves familiar workflows. The conclusion is qualified: ClickHouse has become the de facto storage and query engine for many teams and a strong default for large telemetry workloads, but it has not won observability as a whole, while full-text indexes, PromQL support, and broader product layers remain important.
Cloudflare's Big Pineapple platform, which powers 1.1.1.1 and other DNS services, stores over 250 billion cache entries simultaneously. To decrease fleet-wide memory consumption, engineers implemented five changes to in-memory cache data structures. Key modifications included swapping growable types like Vec and String for Box slices to eliminate capacity overhead, combining distinct record sections into a single list indexed by offsets, omitting redundant owner domain names, and serializing record bytes into contiguous buffers. In benchmarks, these optimizations decreased per-entry memory footprints by 56% while improving cache lookup latency by 19% and insert throughput by 43%. Across production instances, the rollout reduced working-set memory by approximately 100 terabytes.
The post presents partner-built, industry-specific solutions on Databricks Lakebase for financial services, manufacturing and energy, retail and consumer sectors, travel and hospitality, healthcare and life sciences, communications and media, and the public sector. Lakebase is described as a serverless Postgres transactional database on the Databricks Data and AI Platform, combining operational and analytical workloads under unified governance and providing operational serving, agent memory, audit trails, and transactional runtime capabilities. Examples include regulation-change monitoring, AI-native claims operations, KYC and personalization accelerators, near-real-time fraud detection, telecom customer-lifecycle decisioning, and public-sector emergency Situation Reports, using components such as Unity Catalog, Lakeflow, Genie, Mosaic AI agents, and Agent Bricks. The stated outcome is production-ready acceleration from insight to action, including faster compliance response, claims adjudication in minutes, low-latency decisions, and reduced manual effort, with partners positioned to help scope pilots.
The post presents Lakebase Postgres as a design for agent workloads that need isolated copies, restores, replicas, historical queries, and many short-lived databases without repeatedly moving large datasets. It contrasts conventional data-centric OLTP with a transaction-centric model in which PostgreSQL’s write-ahead log (WAL) is the source of truth, with LSNs making database history addressable. Lakebase splits compute, which runs standard Postgres and uses RAM and local NVMe caches, from storage, where safekeepers replicate WAL through quorum acknowledgements, a pageserver materializes pages, and object storage retains immutable history. Commits wait for replicated WAL acknowledgement, not page uploads, while reads request pages at an LSN. The design supports point-in-time queries, cheap branch or read-only compute, scale-to-zero suspension, and shared transactional and analytical access, while still relying on RAM and NVMe for performance.
Shared GPU model pools allow platform teams to maximize utilization, but mixed workloads often contend for serving capacity with differing service requirements. Generally available in Red Hat AI Inference 3.5, llm-d flow control introduces explicit admission control prior to backend routing. An Endpoint Picker sits in the request path and evaluates active priority bands from highest to lowest integer values whenever pool saturation is detected. Saturation metrics combine per-replica queue depth and key-value cache pressure to determine when dispatch gates should engage. Once a priority band is admitted, tenant fairness policies distribute dispatch turns, and ordering rules select the next request to forward to a vLLM replica via Envoy.
The best engineering reads, once a week.
