Loading…
Architecture
173 posts about Architecture. Every summary links to the original.
Meta ·
Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization
Meta developed Hierarchical Interest Representation as an upstream representation layer to optimize deep funnel ad ranking across its platforms. The architecture organizes interactions among users, advertisers, and products into a heterogeneous, time-decayed graph and projects sparse raw edges into multi-level super-graphs of latent interest primitives. To encode these structures, a transformer-based hierarchical encoder incorporates multimodal product semantics processed via language models alongside topology-aware attention biases. Computation scales across billions of entities by pairing deep hash ID embeddings with FlexAttention to evaluate graph biases on the fly without materializing full matrices. Self-supervised cross-view distillation trains student representations against broader teacher graph views to yield universal embeddings and interest tokens.
By Yuhui Ouyang, Di Wang, Sreedal Menon, Jie TianSlack ·
Shipyard: How We Built Slack’s Next-Generation EC2 Platform
Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows.
Archie GunasekaraNetflix ·
Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned
Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.
Netflix Technology BlogMeta ·
Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler
Meta's ad serving fleet encountered latency regressions and technical debt after Linux kernel upgrades introduced the default EEVDF scheduler. To resolve these performance drops across billions of daily requests, the engineering team implemented sched_ext, an open-source, BPF-based scheduling framework integrated into Linux kernel v6.12. The custom scheduling policy uses domain knowledge to soft-partition CPUs dynamically into separate pools for latency-critical request threads and background work, enhancing L3 cache locality. Packaged as a user-space binary, the scheduler allows policy iterations and rollouts via process restarts without kernel rebuilds. The deployment achieved a 28% reduction in p99 ads retrieval latency, a 1.1% increase in ranked ads, and 3.28 megawatts in fleet-wide power savings.
Grab ·
Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption
Grab transitioned its petabyte-scale data lake from Hive Parquet to Apache Iceberg to resolve metadata bottlenecks, small file fragmentation, and lack of native ACID support. Implementing Iceberg with Z-ordering reduced navigation query runtimes from 70 seconds to 6 seconds and cut daily S3 API costs by up to 95% on key operational datasets. To seamlessly manage concurrent table formats without forcing engineers to specify format prefixes, Grab developed and open-sourced UnifiedSparkCatalog. This custom catalog transparently detects table types via the Hive Metastore, routes queries to format-specific handlers, and retains backwards compatibility for legacy Hive operations.
Rahul PentiGrab ·
Migrating Counter Service storage: Design choices and learnings
Grab migrated its Counter Service storage backend from a legacy wide-column database to Aerospike to support real-time anti-fraud windowed aggregations across tens of thousands of queries per second. To decouple storage from the Rust reader service, engineers introduced a storage facade using enum dispatch, avoiding the per-query heap allocations of boxed trait objects. The reader used configuration-driven operating modes to support shadow reads and deterministic traffic splitting without requiring code updates. On the write path, the schema was redesigned to collapse multiple bucket records into a single sorted map per counter, using atomic server-side operations to increment counters and prune expired entries. This data model redesign reduced total record counts and in-memory primary index usage by over an order of magnitude while enabling a zero-downtime transition.
Jia Long LohMeta ·
Meta’s AI Storage Blueprint at Scale
Meta redesigned its BLOB-storage architecture on top of the foundational Tectonic block layer to eliminate GPU stalls and accelerate AI training workflows. The legacy storage system suffered from multi-layered metadata lookups, cross-region latency, and dataplane proxy bottlenecks that conflicted with the millisecond access requirements of flash-based AI clusters. To resolve these issues, the team collapsed disparate metadata into a single flat schema backed by ZippyDB for O(1) path lookups and introduced a fat client SDK capable of streaming data directly from storage servers. The updated stack also leverages spare GPU host memory as a distributed data cache alongside distributed read-plan caches to handle severe traffic spikes during checkpointing and model loading. These protocol and architectural changes deliver predictable latencies, reduce power consumption, and maintain high throughput during distributed training.
By Sidharth Bajaj, Venkatraghavan SrinivasanNetflix ·
GenPage: Towards End-to-End Generative Homepage Construction at Netflix
Netflix traditionally constructs its structured two-dimensional homepage through a complex, multi-stage recommender pipeline that separates candidate generation and ranking across rows and entities. To simplify this architecture and optimize directly for whole-page user satisfaction, Netflix developed GenPage, an end-to-end generative transformer model that autoregressively builds the entire homepage from raw tokenized context. The system relies on a domain-specific tokenizer to compress engagement history, context injection for cold start, hybrid row decoding to minimize decoding steps, and reinforcement learning post-training. In online A/B testing against the production baseline, GenPage delivered statistically significant gains in core user engagement metrics while cutting end-to-end serving latency by 20 percent. Offline evaluations further showed that enriching context prompts improved recommendation quality more effectively than increasing model capacity in the current operating regime.
Netflix Technology BlogNetflix ·
How Netflix Simplified Batch Compute with Kueue
Netflix transitioned its managed batch compute infrastructure from a homegrown solution called Compute Managed Batch to Kueue on its Titus container platform. CMB previously relied on custom scheduling and admission-only fair sharing without preemption, making feature development cumbersome as the Kubernetes ecosystem evolved. To modernize the platform transparently, Netflix mapped internal tenants to Cohorts and leaf tenants to ClusterQueues and LocalQueues while routing jobs through a custom Kueue router. Kueue operates alongside existing Titus scheduling profiles rather than replacing the kube-scheduler, preserving cluster placement efficiency. The migration was completed in four weeks across millions of batch workloads, significantly increasing average resource utilization through preemption-based fair sharing.
Netflix Technology BlogGrab ·
Palana (Part 2): Architecting isolation, identity, and auditability for AI agents
Grab's Palana platform provisions isolated, Kubernetes-native runtime environments for autonomous AI agents using dedicated per-agent namespaces and role-based access controls. The architecture separates network enforcement across layers, applying Layer 3 and Layer 4 containment with Cilium and NetworkPolicy alongside Layer 7 application filtering evaluated by Open Policy Agent. Agent interactions with large language models route through a LiteLLM proxy wrapper that retrieves credentials from HashiCorp Vault based on Kubernetes pod context rather than client headers. Secrets management is divided between directly readable agent paths and proxy-only placeholder paths that prevent raw tokens from residing in runtime filesystems. An automated reaper monitors multi-source activity signals to shut down idle compute resources while preserving persistent storage and configuration state.
Kevin LittlejohnGrab ·
Palana (Part 1): Why Grab built a secure platform for autonomous AI Agents
Autonomous AI agents introduce significant operational and security risks when granted network access, persistent state, and credentials. To address these concerns without impeding developer productivity, Grab created Palana, an in-house Kubernetes-native execution substrate. The platform isolates each agent workload within its own namespace, pairing it with dedicated storage, network policies, and role-based access control. Network egress is funneled through an Envoy and Open Policy Agent proxy layer that audits requests and injects credentials from HashiCorp Vault using placeholder tokens, keeping raw secrets outside the agent runtime. This design allows Grab to securely host hundreds of long-running workflows, remote coding environments, and automation bots.
Kevin LittlejohnAirbnb ·
Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world
To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.
Patrick LamAirbnb ·
Sitar-agent: Building a reliable dynamic configuration sidecar at scale
Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.
Bo TengAirbnb ·
When history fails you, borrow from geography
When historical travel data became uninformative during the asynchronous post-COVID recovery, Airbnb redesigned its demand forecasting to borrow signals across geographies rather than waiting for local data. Using a hierarchical Bayesian framework, the team propagated updated posterior estimates from early-recovering corridors as informative priors for structurally similar, later-recovering corridors. This allowed accurate corridor-level demand forecasting in near real time during periods of high disruption and uneven market reopening.
Harrison KatzGrab ·
From decentralized Docs-as-Code to a centralized repository: Evolving Grab's documentation strategy
Grab originally adopted a decentralized Docs-as-Code workflow to manage Markdown documentation alongside service code, but the model strained as engineering scaled. Dispersed repositories led to fragmented terminology, uncoordinated repository moves, uneven linting standards, and noisy search results. To resolve these issues, the organization transitioned to a centralized documentation repository with standardized templates, automated CI quality gates, and a unified search overlay combining fuzzy title search with Glean. Consolidating Markdown into a single repository also established a clean, structured knowledge base tailored for internal AI agents. The centralized setup fostered continuous maintenance, averaging 27 daily commits and an update merged roughly every 50 minutes.
Karen KueSlack ·
Slack AI: The Path to Multi-Cloud
Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.
Shaurya KethireddyAirbnb ·
Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure
Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput.
Lucen ZhaoGithub ·
From latency to instant: Modernizing GitHub Issues navigation performance
GitHub Issues addressed navigation latency by shifting workloads to the client using a local-first, stale-while-revalidate architecture. To evaluate perceived delays, the team tracked user transitions through Highest Priority Content thresholds, aiming for instant rendering in under 200 milliseconds. The core implementation added a persistent client-side cache using IndexedDB, paired with a synchronous in-memory tier to serve hot issue payloads without asynchronous overhead. A selective preheating mechanism resolves missing cache entries for high-intent links without overburdening backend capacity with redundant requests. Following broad rollout, the proportion of instant React soft navigations increased from 4% to approximately 22%, yielding an overall cache-hit ratio of around 33%.
Natalie GuevaraSlack ·
From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.
Mahendran VasagamGrab ·
Data Mesh at Grab (Part II): The foundational tools behind certification
Grab operationalizes its Signals Marketplace data mesh through integrated platforms designed for continuous data certification and observability. The central metadata management platform, Hubble, extends open-source DataHub to model metadata as an event-driven graph and expose search, lineage, ownership, and data contracts. An automated certification engine built on the DataHub Actions framework continuously evaluates metadata changes, classifying assets into Uncertified, Certified, CertifiedPlus, or Revoked states. Genchi serves as the data quality observability layer, using Temporal and Kafka to run checks for freshness, volume completeness, schema stability, and semantic rules. To eliminate false-positive alerts caused by decoupled cron schedules, Genchi integrates with the Lighthouse monitoring service to trigger quality tests immediately upon pipeline completion.
Aezo Teo