Loading…
Migrations
24 posts about Migrations. Every summary links to the original.
The Cloudflare Blog – Brought to you by EmDash
Cloudflare migrated its official blog to EmDash, an Astro-focused content management system, acting as internal Customer Zero to evaluate scaling and usability. To prepare for traffic spikes reaching thousands of requests per second, engineers used k6 to execute ramp, breakpoint, and burst load tests. The resulting production architecture runs EmDash on Cloudflare Workers, pairing it with Workers Cache, an object cache backed by Workers KV, and Hyperdrive connected to PlanetScale. This layered caching design serves 99.5 percent of static files and 70 percent of requests from cache, flattening p95 latency under real-world loads reaching 850 requests per second. The migration also introduced a frontend redesign using the Kumo design system and added Model Context Protocol servers for automated agent workflows.
Kody JacksonGrab ·
Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption
Grab transitioned its petabyte-scale data lake from Hive Parquet to Apache Iceberg to resolve metadata bottlenecks, small file fragmentation, and lack of native ACID support. Implementing Iceberg with Z-ordering reduced navigation query runtimes from 70 seconds to 6 seconds and cut daily S3 API costs by up to 95% on key operational datasets. To seamlessly manage concurrent table formats without forcing engineers to specify format prefixes, Grab developed and open-sourced UnifiedSparkCatalog. This custom catalog transparently detects table types via the Hive Metastore, routes queries to format-specific handlers, and retains backwards compatibility for legacy Hive operations.
Rahul PentiGrab ·
Migrating Counter Service storage: Design choices and learnings
Grab migrated its Counter Service storage backend from a legacy wide-column database to Aerospike to support real-time anti-fraud windowed aggregations across tens of thousands of queries per second. To decouple storage from the Rust reader service, engineers introduced a storage facade using enum dispatch, avoiding the per-query heap allocations of boxed trait objects. The reader used configuration-driven operating modes to support shadow reads and deterministic traffic splitting without requiring code updates. On the write path, the schema was redesigned to collapse multiple bucket records into a single sorted map per counter, using atomic server-side operations to increment counters and prune expired entries. This data model redesign reduced total record counts and in-memory primary index usage by over an order of magnitude while enabling a zero-downtime transition.
Jia Long LohNetflix ·
How Netflix Simplified Batch Compute with Kueue
Netflix transitioned its managed batch compute infrastructure from a homegrown solution called Compute Managed Batch to Kueue on its Titus container platform. CMB previously relied on custom scheduling and admission-only fair sharing without preemption, making feature development cumbersome as the Kubernetes ecosystem evolved. To modernize the platform transparently, Netflix mapped internal tenants to Cohorts and leaf tenants to ClusterQueues and LocalQueues while routing jobs through a custom Kueue router. Kueue operates alongside existing Titus scheduling profiles rather than replacing the kube-scheduler, preserving cluster placement efficiency. The migration was completed in four weeks across millions of batch workloads, significantly increasing average resource utilization through preemption-based fair sharing.
Netflix Technology BlogGrab ·
Scaling out Distroless adoption With AI
Grab is transitioning its microservices to Distroless base images to eliminate unnecessary binaries and reduce vulnerability risks, but the migration risks runtime failures from missing shared objects and system utilities. To safely validate container execution in continuous integration without staging dependencies, the team relied on medium tests that run containerized services alongside internal dependencies managed by Testcontainers. Because hundreds of services lacked this test harness, Grab implemented an agentic workflow using Claude Code and Model Context Protocol integrations to inspect repositories, generate test boilerplate, and resolve configuration errors. Once test baselines are established, an automated patch-test-compare pipeline updates Dockerfiles, constructs multi-stage builds for necessary dynamic libraries, and creates draft merge requests for human approval.
Jia Yee ChongAirbnb ·
Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world
To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.
Patrick LamAirbnb ·
Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure
Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput.
Lucen ZhaoSlack ·
From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.
Mahendran VasagamSlack ·
How Slack Rebuilt Notifications 📣
Slack redesigned its legacy notification architecture to address user confusion, conflicting desktop and mobile preference systems, and tight coupling between notification content and push delivery. The engineering team unified four separate preference models into a single framework with three channel options: All new posts, Mentions, and Mute. To ensure safe rollback and backwards compatibility without direct database mutations, Slack implemented a read-time translation strategy alongside a new desktop_push_enabled preference and auto-saving cross-platform React UI components. This decoupling of activity from push delivery resulted in a fivefold sustained increase in settings engagement, reduced reliance on per-channel overrides, and eliminated synchronization mismatches across mobile and desktop clients.
Frances CoronelGrab ·
Grab's Mac Cloud Exit supercharges macOS CI/CD
Scaling iOS CI/CD workloads across multiple large mobile applications led to substantial expenses and queue delays under rented cloud macOS infrastructure. To mitigate network latency to Git servers and cut compute overhead, Grab relocated its build fleet from a US cloud provider to a colocation data center in Malaysia. The infrastructure deployment relies on over 200 bare-metal Mac minis across 42RU racks, purposefully avoiding virtualization solutions that introduced performance and stability penalties. Phased migration and guardrail monitoring verified cluster stability while achieving 20% to 40% reductions in pipeline and app build times. Operating the regional bare-metal cluster over an estimated three-year hardware replacement lifecycle is projected to yield 2.4 million USD in total cost of ownership savings.
Madushan GamageGrab ·
Modernising Grab’s model serving platform with NVIDIA Triton Inference Server
Grab's machine learning serving platform, Catwalk, experienced mounting technical debt, elevated latency, and rising costs from maintaining disparate inference engines for various frameworks. To address these limitations, the team adopted NVIDIA Triton Inference Server to establish a unified engine, starting with an ONNX migration. They built a custom Triton manager component featuring a proxy layer to translate legacy API requests alongside a server manager that handles model downloads, verification, configuration, and health checks. Within ten days, over half of online deployments migrated seamlessly without requiring client code modifications. The transition reduced p90 latency from 120ms to 20ms on large transformer models and cut average infrastructure spend by approximately 20 percent across evaluated services.
Daniel TaiGrab ·
Taming the monorepo beast: Our journey to a leaner, faster GitLab repo
Grab's decade-old Go monorepo grew to 12.7 million commits and 250GB of Git data, causing Gitaly replication delays of up to four minutes that routed all read traffic exclusively to the primary node and slowed developer operations. After staging tests proved that shallow history reduced replication lag from hundreds of seconds to under three seconds, standard rewriting tools like git filter-repo and git rebase failed due to complex merge histories and repository scale. To overcome runner memory limits and lengthy git garbage collection cycles, the engineering team implemented a custom two-phase migration script. The script selectively migrated 2,000+ critical dependency tags and one month of recent history, flattening merge commits, embedding legacy hashes for traceability, and reducing total commit volume by 99.9%.
Nagendra GangwarGrab ·
Grab's service mesh evolution: From Consul to Istio
Grab operated over 1,000 microservices across hybrid infrastructure using Consul alongside a fallback mechanism called Catcher. Single-point-of-failure vulnerabilities in Consul servers and limited support for multi-cluster operations prompted an evaluation of alternative mesh technologies, ultimately leading to the selection of Istio. Grab avoided the standard single-control-plane-per-cluster pattern by deploying multiple external control planes in dedicated Kubernetes clusters arranged in active-active pairs. Migration began in Q4 2024, shifting traffic across AWS and GCP while handling both HTTP and gRPC protocols with gradual traffic-shifting and rollback mechanisms.
Hilman KurniawanGrab ·
Counter Service: How we rewrote it in Rust
The Integrity Data Platform team rewrote Counter Service, a high-throughput Golang microservice serving event counts for fraud rules and machine learning models, to evaluate the operational return on investment of Rust. Rather than performing a line-by-line translation, engineers approached the service as a black box, reimplementing core application logic from scratch to satisfy established gRPC contracts across Scylla and Redis. The team resolved internal Go tooling dependencies by building custom configuration template parsers using the nom parser combinator and selected targeted open-source crates such as fred.rs and Cadence. Adapting to Rust required navigating cooperative, stackless async execution compared to Go's preemptive concurrency model, alongside managing borrow checker constraints. Ultimately, the rewrite achieved a 70% reduction in infrastructure costs while maintaining comparable service performance.
Jia Long LohGrab ·
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab redesigned its Spark observability platform, Iris, to overcome limitations associated with its previous Telegraf, InfluxDB, and Grafana stack. InfluxDB presented operational challenges due to limited SQL compatibility, poor handling of string metadata, and query degradation on high-cardinality identifiers. The team replaced InfluxDB with StarRocks to serve as a unified analytical engine for both real-time cluster metrics and historical analysis. StarRocks ingests metrics directly from Kafka via routine load tasks, storing worker and Spark event data in partitioned duplicate-key OLAP tables linked by worker and application identifiers. This architecture eliminated intermediate ingestion agents, simplified S3 data lake backups, and enabled a custom web application alongside Superset for consistent querying.
Huong VuongGrab ·
Turbocharging GrabUnlimited with Temporal
GrabUnlimited experienced scaling bottlenecks, corrupted membership states, and elevated production incidents after its subscriber base grew by over 1000%. The original architecture relied on Amazon SQS state machines, 5-minute Redis locks, and daily batch cron jobs that overwhelmed the database and lacked granular idempotency during upstream retries. To eliminate these failure modes, the engineering team migrated the core membership lifecycle to Temporal's workflow orchestration engine. Replacing batch cron jobs with Temporal Timers distributed renewal operations throughout the day, while matching workflow IDs prevented race conditions between renewals and cancellations. This architectural transition resolved database bottlenecks and yielded an 80% reduction in open production incidents.
Michel ParrenoGrab ·
How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication
Grab split a backend service's read and write functionalities into separate services to allow independent scaling. Migrating the write path required transferring processing from 16 source Kafka streams—averaging 20,000 reads per second into DynamoDB tables and output streams—with zero data loss or duplication. Standard feature flags were ruled out because rollout propagation delays could introduce minutes of duplicate or missing data during flag toggling. Instead, engineers extracted processing logic into a shared monorepo commons package that used coordinated timestamps to trigger simultaneous cutovers across both services. Temporary validation sinks verified processing accuracy in production prior to the cutover, completing the stream-by-stream migration across three weeks without downtime.
Md RiyadhGrab ·
No version left behind: Our epic journey of GitLab upgrades
Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.
Saurabh VajpayeeGrab ·
Migrating from Role to Attribute-based Access Control
Grab's streaming data platform team migrated the Kafka Control Plane from Role-Based Access Control to Attribute-Based Access Control to eliminate operational bottlenecks and manual permission management. The previous model required defining hundreds of roles, permissions, and group mappings in an internal IAM service, leading to approval delays and stale memberships. Under the new architecture, user attributes sync from the HRMS and token payloads, while resource attributes are tagged upon creation or backfilled to reflect department and team ownership. Open Policy Agent evaluates access requests defined in Rego via middleware by comparing user attributes with resource metadata. This transition eliminated over 200 roles, 200 permissions, and roughly 3,000 unused IAM resources while automating access provisioning for new joiners.
Minh Khoi NguyenGrab ·
Go Modules- A Guide for monorepos (Part 1)
Grab transitioned its large Go monorepo dependency management from Glide to Go modules while retaining an existing vendor directory structure. The team generated root go.mod configurations from glide.yaml and used go mod vendor without directly enabling module-mode builds. Incompatible nested sub-vendor paths were excluded by placing empty go.mod files, relying on the rule that modules cannot contain other modules. Post-migration maintenance revealed challenges with dependency inheritance and implicit go.mod updates during builds, which engineers investigated using go mod graph and digraph to trace dependency paths.
Michael Cartmell