Loading…
Topic
377 posts about Scalability. Every summary links to the original.
See the idea
Consolidating agent memory, retrieval, and governance onto MongoDB Atlas removes sync and vendor layers, but ties scaling and control to one platform. This trade-off view shows the payoff, the cost, and the condition that decides it.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Pinterest home feed candidate generation relies on large-scale retrieval, but standard two-tower models produce single embeddings that struggle to capture diverse concurrent user intentions. To solve this, Pinterest expanded Conditional Learned Retrieval (CLR) to generate condition-aware embeddings conditioned on explicit interest, Pin, and Board contexts. The team evolved model representations by appending condition tokens to user sequences in transformers, integrating the PinFM foundation model, and introducing hierarchical semantic IDs to improve cold-start coverage. On the serving side, engineers consolidated duplicate feature requests and shifted inference to GPU instances using NVEmbed under a one-query-to-many-documents paradigm. These infrastructure and modeling changes unlocked new condition types, achieved seven-figure cost savings, and reduced p90 model retrieval latency from 80ms down to 12ms.
Red Hat OpenShift supports compute scaling using Cluster Autoscaler integrated with the OpenShift machine API. The mechanism relies on paired ClusterAutoscaler and MachineAutoscaler custom resources to manage cluster-wide limits and per-MachineSet replica boundaries. Scaling decisions evaluate pending pods against node allocatable capacity and hypothetical node capacity annotations while respecting expander strategies like Random, LeastWaste, or Priority. Cluster Autoscaler checks for unnecessary nodes every 10 seconds, scaling down when utilization falls below configured thresholds and workloads can safely migrate. To mitigate provisioning delays, clusters can adopt overprovisioning via low-priority pause containers or use APIs like ProvisioningRequest and CapacityBuffer for predictive scaling.
Authentication and identity requirements vary substantially as software organizations scale from solo builders to multi-brand enterprises. Building and maintaining authentication in-house frequently consumes valuable engineering cycles and incurs ongoing costs associated with security patching and edge cases. To address these distinct development stages, Auth0 structures its platform across Free, Essentials, Professional, and Enterprise tiers with differentiated B2C and B2B pricing. Early-stage and growing teams gain access to enterprise connections, SCIM user synchronization, and multi-tenant routing through Auth0 Organizations. Larger customer deployments utilize features like multi-app Single Sign-On and passwordless logins, which have driven reported outcomes such as a 20% conversion lift and a 66% reduction in identity management time.
Selecting between relational and non-relational databases requires matching the data store to requirements for structure, consistency, scale, and query complexity. Relational systems organize data in predefined tables, enforce schemas and keys, and provide ACID transactions, making them suited to structured, transactional workloads and complex joins. Non-relational systems support documents, key-value pairs, graphs, or wide-column structures, often using flexible schemas and schema-on-read while scaling horizontally and accepting eventual consistency for throughput and availability. The trade-offs include vertical versus horizontal scaling, SQL versus database-specific query languages, and database-enforced versus application-level integrity. The guidance recommends documenting workload needs, prototyping with realistic volumes and failure scenarios, and considering polyglot persistence when different workloads require specialized systems.
Generative AI training across hundreds of thousands of GPUs makes the scale-out network a first-order performance bottleneck, because synchronized collectives expose weaknesses in conventional Ethernet. Static ECMP, packet loss controls such as PFC, and slowly tuned congestion control can create stragglers, head-of-line blocking, latency spikes, and multi-tenant interference. Spectrum-X Ethernet addresses these conditions with hardware-accelerated adaptive per-packet routing, targeted ECN- and RTT-based congestion control, and a SuperNIC Plane Load Balancer that distributes traffic across independent network planes. Spectrum-X Multiplane uses shallow two-tier fat trees and host-edge optical devices to provide path diversity while scaling a two-tier design to over 128,000 endpoints. In the cited tests, Spectrum-X sustained 98% of theoretical line rate across GPU pairs, held P99 tail latency to 8–9 microseconds under 75% load, recovered from a link flap in 2.68 milliseconds, and limited bandwidth loss under 10% fabric failure to 11%.
Agentic AI inference has shifted from single-turn requests to long, stateful workflows that reason, invoke tools, coordinate subagents, and reuse growing context, increasing token and power demands. The post presents SemiAnalysis AgentX, an InferenceX benchmark that replays prerecorded Claude Code sessions turn by turn, preserving context, input/output lengths, reasoning time, tool-call latency, and KV-cache pressure while varying concurrency. It evaluates tokens per megawatt alongside E2E normalized interactivity, standard interactivity, E2E latency, and TTFT; NVIDIA-measured Vera Rubin NVL72 results, pending SemiAnalysis review, show up to 30x higher throughput per megawatt than GB300 NVL72 at 160 tokens per second per user. GB300 also delivers up to 15x H200 throughput per megawatt, up to 10x lower cost per million tokens, and roughly 80x higher throughput on Kimi K3 2.8T, supported by MoE runtimes, communication overlap, Dynamo, and NVLink.
Power-constrained AI factories measure efficiency by AI output per watt, because facility overhead, rack losses, and operational inefficiency reduce the power available to compute. In an illustrative 100 MW budget, 60% reaches AI load, while static rack provisioning can strand headroom that another rack could use. NVIDIA DSX MaxLPS addresses this with dynamic power allocation, workload-profile power optimization, and 45° C thermal and site design, including warm-water liquid cooling. Its Dynamic Power Software, currently in Developer Preview, models topology from utility to GPUs and reallocates unused capacity within policy-defined budgets, with optional DSX Exchange integration. For Vera Rubin NVL72, NVIDIA projects up to 40% more Rubin GPU capacity within the same power budget, while validation compares unmanaged and managed runs using throughput, latency, errors, power, utilization, and compliance.
Agentic AI factories connect diverse users, agents, applications, data sources, and storage systems to massively accelerated compute, making dedicated DPU processing essential for line-rate networking, storage, and security. Scale-In evolves north-south access into a coordinated infrastructure domain built with BlueField-4, NVIDIA DOCA, and Spectrum-X Ethernet. BlueField-4 combines programmable control-plane processing with inline data-path engines, a 64-core Grace CPU, LPDDR5X memory, PCIe Gen6 host connectivity, and an 800 Gb/s network interface. The architecture offloads security, tenant isolation, storage protocols, provisioning, and telemetry from host CPUs; the source states that Spectrum-X storage paths can provide up to 1.45x the throughput of off-the-shelf Ethernet, while independent control and observability support more consistent operations as AI infrastructure scales.
Agentic AI fleets must handle unpredictable trajectories in which long sequential reasoning chains are interrupted by short bursts of parallel tool calls, retrieval, sandbox, or sub-agent work. Telemetry from 163,594 sessions found over 97% had unique trajectory profiles, making fleet sizing around several specialized CPU designs impractical; the sequential path is latency-bound while fan-out requires concurrency. It presents NVIDIA Vera CPU as a single balanced design point, combining strong loaded per-core performance, concurrency, memory bandwidth, and a low-latency monolithic architecture, with Olympus cores using a wide front end, branch prediction, and deep out-of-order execution. Using estimated SPEC CPU® 2026 results across four agentic workloads, it reports up to 1.5x the agentic performance of AMD Venice, while noting that Venice figures were estimated from SPECrate® 2026 data and that results may vary.
Serving agentic AI at more than 3,000 output tokens per second while retaining a 100K-token KV cache creates a small-batch tensor-parallelism challenge: fixed communication and coordination costs can offset distributed-compute gains. NVIDIA Groq 3 LPX addresses this with compiler-scheduled execution across 256 LP30 local processing units, 128 GB of collective SRAM, and 96 112-Gbps C2C links per chip, planning transfers before execution and overlapping communication with computation at 320-byte-vector granularity. Artificial Analysis measured 3,431 output tokens per second at 100K context on Gemma 4 31B, and 3,382 tokens per second at 10K context. On SPEED-Bench coding tasks, the same system reached a median 4,767 tokens per second and a P80 of 5,520. The post describes pairing LPX with Vera Rubin NVL72 through prefill-decode, attention-FFN, or external-drafter speculative-decoding configurations for projected large-model agentic workloads.
Indian organizations in regulated industries are modernizing legacy data infrastructures with MongoDB to achieve scale and compliance. Physics Wallah, serving over 7 million students, built a compliant multi-tenant payment platform on MongoDB Atlas using ACID transactions, Change Streams for audit trails, and native search. Flipkart uses MongoDB Atlas for high-throughput writes and real-time payment aggregation during mega sales like The Big Billion Days, with seamless horizontal sharding. IBS Software scaled three mission-critical platforms, processing billions of records and millions of searches daily with sub-100ms API responses and 99.99% uptime, while iTravel Cruise achieves 100,000 TPS at 5ms latency. Companies like Kore.ai and PineLabs use MongoDB Enterprise Advanced for on-premises deployments requiring high availability, strict security, and low-latency transaction processing.
LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility.
Quantitative strategies need instrument groupings that capture both ordinary co-movement and dependence during market stress, while remaining stable and inexpensive to recompute. The workflow constructs rolling absolute Pearson-correlation and tail pairwise dependence matrices (TPDM), factorizes them with nonnegative SymNMF using AdaptGrow, and derives soft loadings, hard argmax labels, and structural-break diagnostics. A trace-based formulation reduces estimated peak storage from approximately 20n2 to 4n2 bytes, while row-sharded distributed execution communicates O(nk) data; the system uses eigenspectrum-based selection between full-batch and block-stochastic AdaGrad. Across 250 synthetic windows, the setup tests planted membership changes and a joint tail-stress episode: ARI is intended to identify reclassification, whereas TPDM reveals co-crash dependence that ordinary correlation can miss. Reported factorization timings include 13.0 and 12.4 seconds for 100,000 instruments across four GB200 GPUs, and approximately two and four minutes for one million instruments across 64 GPUs on 16 nodes, not end-to-end timings.
Netflix operates over 30,000 Apache Flink jobs across multiple AWS regions, ranging from managed pipelines to complex stateful DAGs. Its initial in-house autoscaler used external telemetry from Mantis and Atlas to scale TaskManagers as a single unit, cutting resource usage by 25–45% but failing on multi-operator stateful topologies. To address these limitations, Netflix adopted the Apache Flink Autoscaler library, which calculates True Processing Rate and per-vertex parallelism using internal metrics. The architecture uses Temporal workflows orchestrated within a Spring Boot application to evaluate jobs individually and prevent cross-job blast radiuses. Implementing the open-source autoscaler reduced annualized compute expenditures by 58% for client telemetry and logging, saving approximately $1.1 million.
AgentFlo built autonomous AI sales agents to address cart abandonment, unassisted messaging channels, and severe traffic spikes across eCommerce platforms. The system uses the Strands Agents SDK alongside Amazon Bedrock AgentCore to manage intent, recommend products, and execute transactions. Incoming customer communications pass through AWS Fargate before entering AgentCore runtime, which executes model-driven agents inside isolated microVMs with stateful sessions lasting up to eight hours. Tool execution is brokered separately through AgentCore Gateway to ensure operations like cart modifications scale independently from model reasoning. This deployment model enables merchants to launch specialized conversational agents in minutes without pre-provisioning capacity or maintaining custom container configurations.
Shopify rebuilt its observability infrastructure after metrics, logs, and traces became fragmented across vendors, with unpredictable costs and limited cross-signal analysis. It created Observe, a unified platform that runs metrics, logs, traces, profiles, and exceptions on self-hosted ClickHouse, treating them as structured, time-ordered, high-dimensional events and serving roughly 50 million events per second normally and 100 million at BFCM peak, or about 110 GB/s. Kafka buffers telemetry for large synchronous durable writes, while materialized views support flexible schemas, autocomplete, typed hot keys, and identifier-based correlation; the latter reportedly makes targeted searches 10x faster. Shopify reports a 16x query improvement out of the box, exceeding 30x at peak performance, with data queryable in under a minute, and says it would likely choose ClickHouse Cloud if starting again because managed operations could reduce the self-hosting burden.
Workplace operations platform SafetyCulture previously relied on a multi-team, manual onboarding process for enterprise single sign-on setups across diverse identity providers, which created operational overhead and accounted for approximately 30 percent of engineering support tickets. To resolve these scaling bottlenecks, the engineering team integrated Auth0's Self-Service Enterprise Configuration, implementing a step-by-step wizard for OpenID Connect and custom SAML setups. The resulting workflow allows customer administrators to configure identity settings, test logins before committing, and manually toggle production activation. Moving from proof-of-concept to general availability took two months, achieving a 50 percent unassisted completion rate. Four months post-launch, SafetyCulture recorded a 60 percent reduction in SSO setup-related customer support tickets.
Federating vision-language models (VLMs) across institutions requires keeping images, text, and prompts local while coordinating training and combining updates from sites with different task and modality mixes. NVIDIA FLARE addresses this workflow through server-scheduled rounds, client-local execution, and an explicit update contract, while supporting large-object externalization, pull-based tensor streaming, and disk-backed aggregation for large payloads. The post presents FedUMM as an adapter-based design in which clients train LoRA adapters over a frozen BLIP3o backbone and send only adapter updates, while full-model training remains possible when required. In reported eight-client experiments, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round, improved VQA v2 by 0.7 points, and retained about 97% of centralized performance on VQA v2 and GenEval; the evaluation used simulated sites and does not establish clinical performance or formal privacy guarantees.
Long system prompts improve agent behavior but increase inference time, cost, and the GPU capacity needed to serve a given traffic level. Shopify compresses the Sidekick GraphQL agent’s roughly 6,000-token system prompt into about 1,500 gist tokens, using special vocabulary tokens whose embeddings are learned by knowledge distillation while the model weights remain frozen. Training compares teacher logits from the full prompt with student logits from the gist-token prompt using KL divergence; at inference, the prompt is replaced with the gist-token string, requiring no custom serving path. In load tests at 350 requests per minute, compression reduced median time to first token from 438ms to 354ms and end-to-end latency from 6.8s to 4.2s, while raising throughput from 20.2 to 23.4 QPS and enabling 14% fewer GPUs in production without losing prediction quality.
Dropbox manages growing infrastructure demands from AI and standard workloads by maximizing the efficiency of its existing data centers rather than just building more capacity. Through a system-level approach spanning software, hardware, and physical facility design, the company dynamically manages power and balances workloads. These optimizations, combined with higher storage density and data-driven hardware lifecycle management, have significantly reduced energy use per unit of storage.
The best engineering reads, once a week.
