# Scalability > 84 posts about Scalability, summarised, each linking to the original. ## Articles ### [MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet](https://yomu.fyi/post/metaroce-a-new-rdma-transport-built-for-ai-scale-ethernet.md) - Company: [Meta](https://yomu.fyi/company/meta.md) - Author: By Arvind Srinivasan, Neil Spring, Omar Baldonado, Rajiv Krishnamurthy - Published: Aug 24, 2026 Scaling frontier AI training and inference across massive GPU clusters places severe demands on network fabrics, yet conventional RoCE relies on Priority Flow Control to enforce in-order packet delivery. Meta introduced MetaRoCE, a clean-sheet RDMA transport protocol engineered specifically for lossy, multiplane commodity Ethernet without requiring PFC. By shifting intelligence to endpoint NICs, the protocol sprays packets across fine-grained logical paths, writing out-of-order data directly into destination memory without reorder buffers or head-of-line blocking. Congestion is managed via per-path ECN windows paired with receiver-driven rate hints returned in acknowledgments. In validation on a 64-node AMD GPU cluster running collective operations, MetaRoCE outperformed RoCEv2, sustaining roughly 86% throughput under 1% packet loss and achieving linear scaling across multiplane fabrics. ### [MTIA 300: Meta’s First Training Chip with Built-in NICs and Communication-Offloading Engines](https://yomu.fyi/post/mtia-300-meta-s-first-training-chip-with-built-in-nics-and-communicati.md) - Company: [Meta](https://yomu.fyi/company/meta.md) - Author: By Rajiv Krishnamurthy, Wes Bland - Published: Aug 24, 2026 Training recommendation models poses distinct network bottlenecks because massive embedding tables trigger frequent collective operations that compete with computation on standard GPUs. Meta designed the MTIA 300 accelerator to overcome this by embedding two network chiplets containing twelve 800 Gbps RDMA NICs directly inside the chip package. The architecture offloads communication from its compute grid to sixteen dedicated message engines equipped with RISC-V cores and near-memory compute reduction blocks. Working alongside the co-designed HCCL library, the chip autonomously compiles and executes collective communication subgraphs without host CPU intervention. On a 150-billion-parameter production recommendation model, MTIA 300 delivered communication speeds 3.9 times faster than an equivalent GPU cluster while keeping concurrent compute degradation under 0.5 percent. ### [A Tale of Two Flink Autoscalers](https://yomu.fyi/post/a-tale-of-two-flink-autoscalers.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Aug 21, 2026 Netflix operates over 30,000 Apache Flink jobs across multiple AWS regions, ranging from managed pipelines to complex stateful DAGs. Its initial in-house autoscaler used external telemetry from Mantis and Atlas to scale TaskManagers as a single unit, cutting resource usage by 25–45% but failing on multi-operator stateful topologies. To address these limitations, Netflix adopted the Apache Flink Autoscaler library, which calculates True Processing Rate and per-vertex parallelism using internal metrics. The architecture uses Temporal workflows orchestrated within a Spring Boot application to evaluate jobs individually and prevent cross-job blast radiuses. Implementing the open-source autoscaler reduced annualized compute expenditures by 58% for client telemetry and logging, saving approximately $1.1 million. ### [Improving infrastructure efficiency for growing demand in the age of AI](https://yomu.fyi/post/improving-infrastructure-efficiency-for-growing-demand-in-the-age-of-a.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev - Published: Aug 18, 2026 Dropbox manages growing infrastructure demands from AI and standard workloads by maximizing the efficiency of its existing data centers rather than just building more capacity. Through a system-level approach spanning software, hardware, and physical facility design, the company dynamically manages power and balances workloads. These optimizations, combined with higher storage density and data-driven hardware lifecycle management, have significantly reduced energy use per unit of storage. ### [How Standardizing Product Telemetry Reduced Time to Insight by 97%](https://yomu.fyi/post/how-standardizing-product-telemetry-reduced-time-to-insight-by-97.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 11, 2026 Salesforce faced scaling bottlenecks when individual engineering teams maintained fragmented, custom telemetry pipelines requiring manual data transformations and dashboard creation every release. To resolve these silos, the engineering organization established the Product Data Platform (PDP) using a standardized custom schema built atop Monitoring Cloud infrastructure. This framework mandates core telemetry fields for consistent metric generation while offering governed optional attributes for product-specific flexibility. An AI-driven Model Context Protocol tool assists developers by recommending compliant instrumentation patterns directly within their workflows. Today, the unified platform processes 45 billion rows of data daily across 19,000 distinct events and 2,000 product features, reducing dashboard refresh latency by 97 percent from one month to daily updates. ### [How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…](https://yomu.fyi/post/how-and-why-netflix-built-a-real-time-distributed-graph-part-3-queryin.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Aug 7, 2026 Netflix designed a query execution serving layer for its Real-Time Distributed Graph to power sub-100ms responses across diverse graph traversal workloads spanning billions of nodes and edges. To prevent compounding network delays in multi-hop queries, the engine uses a breadth-first traversal model that batches entity lookups across entire frontier levels rather than tracing paths depth-first. The entire serving architecture relies on asynchronous composition across small thread pools of 16 to 24 threads, ensuring no thread blocks while waiting on remote storage or enrichment I/O. Selective caching via EVCache targets stable properties with volatility-matched TTLs, producing 70 to 80 percent cache hit rates and reducing backend storage calls by three to four times. A layered filtering hierarchy pushes depth and edge limits directly to the storage tier, eliminating bespoke code changes while isolating system resources against excessive fan-out. ### [From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking](https://yomu.fyi/post/from-user-sequences-to-scaling-laws-a-multi-stage-architecture-for-met.md) - Company: [Meta](https://yomu.fyi/company/meta.md) - Author: By Steven De Gryze, Parshva Doshi, Sean O'Byrne, Arnold Overwijk, Dinesh Ramasamy, Lee Xiong - Published: Aug 5, 2026 Scaling sequence models for real-time ad recommendations faces latency bottlenecks and scaling ceilings when hybrid configurations process both long user histories and sparse feature interactions online. Meta addresses these trade-offs with a multi-stage architecture that decouples heavy, asynchronous offline user modeling from lightweight, real-time online ranking. Deep upstream transformer models process sequences of thousands of actions to generate cached user embeddings, which downstream online models combine with candidate ad features using dense tokenization and target-aware multi-head attention. This framework produces predictable, LLM-style log-linear scaling curves between compute and normalized entropy without disproportionate increases in online serving costs. Implemented as part of Meta's Generative Ads Recommendation Model, the design achieved a cumulative 6% lift in Instagram conversions and a 3% lift in Facebook conversions. ### [How Salesforce Eliminated Single-Region Risk and Reduced Downtime Blast Radius at 4B Metrics/Min](https://yomu.fyi/post/how-salesforce-eliminated-single-region-risk-and-reduced-downtime-blas.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 5, 2026 Salesforce redesigned its internal observability platform, Argus, to eliminate single-region failure risks while ingesting approximately 4 billion metrics per minute. Operating out of a single AWS region previously created a global blast radius and generated high cross-region data transfer costs. To mitigate these risks without the prohibitive expense of full multi-region replication, the engineering team implemented a geo-local architecture that processes and stores telemetry closer to its origin across production geographies. A newly introduced federation query layer relies on Elasticsearch mappings to route queries selectively to relevant regional clusters, avoiding broad fan-out. The platform also adopted metadata caching for wildcard queries and implemented HTTP 206 partial-response handling to maintain visibility when individual regions experience downtime. ### [Removing the Security Barrier to Agentforce Adoption](https://yomu.fyi/post/removing-the-security-barrier-to-agentforce-adoption.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 3, 2026 Enterprise AI adoption across regulated industries stalls when internal governance mandates private, dedicated network paths for sensitive data. To eliminate the need for manual VPN tunnels and complex proxy layers, Salesforce re-engineered Private Connect into a multi-cloud networking platform connecting customer environments to Hyperforce services. The Private Connect v2.0 architecture utilizes private links, transit gateways, and direct endpoint routing alongside an automated control plane for rapid Hyperforce regional rollouts. In addition to supporting cross-substrate interconnects for Azure and decoupling endpoint management, the platform incorporates a generic connector framework supporting tools like Snowflake, Databricks, and Kafka. Today, the system provisions secure connectivity in under 30 minutes while processing roughly 120 TB of data and 683 million requests monthly across 15 AWS regions. ### [GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model](https://yomu.fyi/post/gem-training-how-meta-doubled-the-efficiency-of-its-llm-scale-ads-foun.md) - Company: [Meta](https://yomu.fyi/company/meta.md) - Published: Aug 3, 2026 Meta's Generative Ads Recommendation Model (GEM) trains across thousands of GPUs using a hybrid architecture featuring trillions of sparse embedding parameters and billions of dense parameters. Standard training infrastructure built for large language models struggles with recommendation-specific workloads characterized by jagged user activity histories, asymmetric attention patterns, and numerical sensitivity. To overcome these bottlenecks, engineering teams introduced custom kernels like Jagged Flash Attention and Generalized Dot-Product Attention alongside MXFP8 mixed-precision execution. Pairing these kernel optimizations with topology-aware 5D parallelism doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization while scaling total training FLOPs fourfold over twelve months. ### [GenRec: Towards LLM-Native Recommendation at Netflix](https://yomu.fyi/post/genrec-towards-llm-native-recommendation-at-netflix.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Jul 30, 2026 Netflix developed GenRec, an LLM-backed recommendation ranker built by adapting an internal foundation model for large-scale personalization. Traditional recommendation stacks rely on thousands of hand-crafted features and specialized architectures that are costly to maintain across diverse content types. To replace manual feature pipelines, GenRec verbalizes user histories, metadata, and contexts into natural-language prompts and trains with multi-objective losses, including catalog-aware ranking and reward-weighted alignment. At inference time, the model executes in prefill-only mode on vLLM without decoding text. In large-scale online A/B testing against a mature production ranker, GenRec achieved statistically significant improvements in short-term and long-term metrics while using fewer labeled examples. ### [Building Reliable Production AI with Durable Workflows](https://yomu.fyi/post/building-reliable-production-ai-with-durable-workflows.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Jul 27, 2026 AI prototypes rely on simple prompt-response interactions, but production AI systems handling thousands of records must manage distributed execution issues like worker crashes, rate limits, and transient failures. Rather than treating long-running operations as single large jobs or rebuilding state from logs, systems require durable workflows that preserve execution history across failures. In Agentforce Grid, execution is organized into a hierarchy where column runs divide into recoverable batches and rows composed of discrete activities. Each batch checkpoints after every successful row, retrying up to ten times with exponential backoff and resuming strictly at incomplete work to prevent duplicated model calls. Exposing progress across these hierarchical boundaries ensures that both users and engineers can distinguish completed, failing, and retryable units. ### [How AI Rebuilt Salesforce’s Decades-Old Localization Pipeline](https://yomu.fyi/post/how-ai-rebuilt-salesforce-s-decades-old-localization-pipeline.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Jul 23, 2026 Salesforce needed to deliver major releases across 34 languages as product localization volume surged by more than 35% under fixed budgets and unchanged delivery windows. To address these scaling constraints, the engineering team replaced their legacy translation workflows with an LLM-powered orchestration pipeline integrating prompt engineering, context engineering, and multi-stage validation. The architecture transforms existing style guides, glossaries, and terminology databases into structured context, routing interface strings through specialized prompt stages—up to 85 stages for some languages—to enforce product context, grammar, and branding rules. Combined with an automated validation loop spanning AI editing, AI validation, and human review, this system reduced localization costs by 50 to 90% while accelerating turnaround times across enterprise releases. ### [How our universal content processing platform Riviera evolved for AI and beyond](https://yomu.fyi/post/how-our-universal-content-processing-platform-riviera-evolved-for-ai-a.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash - Published: Jul 20, 2026 Dropbox developed Riviera, a centralized content processing platform that evolved from an internal file preview generator into a shared infrastructure supporting over 300 file formats. By decomposing complex conversion tasks into modular, reusable transformations, the system now powers search indexing, media transcoding, and AI context extraction across Dropbox products. ### [In-House LLM Serving at Netflix](https://yomu.fyi/post/in-house-llm-serving-at-netflix.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Jul 17, 2026 Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas. ### [Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler](https://yomu.fyi/post/modernizing-the-meta-ads-service-with-an-open-source-kernel-scheduler.md) - Company: [Meta](https://yomu.fyi/company/meta.md) - Published: Jul 13, 2026 Meta's ad serving fleet encountered latency regressions and technical debt after Linux kernel upgrades introduced the default EEVDF scheduler. To resolve these performance drops across billions of daily requests, the engineering team implemented sched\_ext, an open-source, BPF-based scheduling framework integrated into Linux kernel v6.12. The custom scheduling policy uses domain knowledge to soft-partition CPUs dynamically into separate pools for latency-critical request threads and background work, enhancing L3 cache locality. Packaged as a user-space binary, the scheduler allows policy iterations and rollouts via process restarts without kernel rebuilds. The deployment achieved a 28% reduction in p99 ads retrieval latency, a 1.1% increase in ranked ads, and 3.28 megawatts in fleet-wide power savings. ### [Migrating Counter Service storage: Design choices and learnings](https://yomu.fyi/post/migrating-counter-service-storage-design-choices-and-learnings.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jia Long Loh - Published: Jul 3, 2026 Grab migrated its Counter Service storage backend from a legacy wide-column database to Aerospike to support real-time anti-fraud windowed aggregations across tens of thousands of queries per second. To decouple storage from the Rust reader service, engineers introduced a storage facade using enum dispatch, avoiding the per-query heap allocations of boxed trait objects. The reader used configuration-driven operating modes to support shadow reads and deterministic traffic splitting without requiring code updates. On the write path, the schema was redesigned to collapse multiple bucket records into a single sorted map per counter, using atomic server-side operations to increment counters and prune expired entries. This data model redesign reduced total record counts and in-memory primary index usage by over an order of magnitude while enabling a zero-downtime transition. ### [Meta’s AI Storage Blueprint at Scale](https://yomu.fyi/post/meta-s-ai-storage-blueprint-at-scale.md) - Company: [Meta](https://yomu.fyi/company/meta.md) - Author: By Sidharth Bajaj, Venkatraghavan Srinivasan - Published: Jul 1, 2026 Meta redesigned its BLOB-storage architecture on top of the foundational Tectonic block layer to eliminate GPU stalls and accelerate AI training workflows. The legacy storage system suffered from multi-layered metadata lookups, cross-region latency, and dataplane proxy bottlenecks that conflicted with the millisecond access requirements of flash-based AI clusters. To resolve these issues, the team collapsed disparate metadata into a single flat schema backed by ZippyDB for O(1) path lookups and introduced a fat client SDK capable of streaming data directly from storage servers. The updated stack also leverages spare GPU host memory as a distributed data cache alongside distributed read-plan caches to handle severe traffic spikes during checkpointing and model loading. These protocol and architectural changes deliver predictable latencies, reduce power consumption, and maintain high throughput during distributed training. ### [How Netflix Simplified Batch Compute with Kueue](https://yomu.fyi/post/how-netflix-simplified-batch-compute-with-kueue.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Jun 22, 2026 Netflix transitioned its managed batch compute infrastructure from a homegrown solution called Compute Managed Batch to Kueue on its Titus container platform. CMB previously relied on custom scheduling and admission-only fair sharing without preemption, making feature development cumbersome as the Kubernetes ecosystem evolved. To modernize the platform transparently, Netflix mapped internal tenants to Cohorts and leaf tenants to ClusterQueues and LocalQueues while routing jobs through a custom Kueue router. Kueue operates alongside existing Titus scheduling profiles rather than replacing the kube-scheduler, preserving cluster placement efficiency. The migration was completed in four weeks across millions of batch workloads, significantly increasing average resource utilization through preemption-based fair sharing. ### [Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world](https://yomu.fyi/post/scaling-beyond-one-how-airbnb-evolved-its-data-architecture-for-a-mult.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Patrick Lam - Published: Jun 9, 2026 To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging. [Older posts](https://yomu.fyi/topic/scalability/page/2.md)