Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
Grab ·
Grab's service mesh evolution: From Consul to Istio
Grab operated over 1,000 microservices across hybrid infrastructure using Consul alongside a fallback mechanism called Catcher. Single-point-of-failure vulnerabilities in Consul servers and limited support for multi-cluster operations prompted an evaluation of alternative mesh technologies, ultimately leading to the selection of Istio. Grab avoided the standard single-control-plane-per-cluster pattern by deploying multiple external control planes in dedicated Kubernetes clusters arranged in active-active pairs. Migration began in Q4 2024, shifting traffic across AWS and GCP while handling both HTTP and gRPC protocols with gradual traffic-shifting and rollback mechanisms.
Hilman KurniawanGrab ·
DispatchGym: Grab’s reinforcement learning research framework
Applying reinforcement learning to dispatch systems is often hindered when the chosen control levers exert weak influence over reward functions. To streamline research, Grab built DispatchGym, a framework that connects reinforcement learning algorithms to a dispatch process simulation via the Gymnasium API. The simulation emphasizes directional accuracy over absolute precision, allowing researchers to evaluate relative metric shifts across supply and demand scenarios. Built in modular Python and accelerated with Numba, the system allows data scientists to test code locally and launch distributed Spark executions with a single command-line call. The framework has been used to evaluate various contextual bandit models and action sampling strategies for tuning dispatch hyperparameters.
Tan Sien YiGrab ·
Counter Service: How we rewrote it in Rust
The Integrity Data Platform team rewrote Counter Service, a high-throughput Golang microservice serving event counts for fraud rules and machine learning models, to evaluate the operational return on investment of Rust. Rather than performing a line-by-line translation, engineers approached the service as a black box, reimplementing core application logic from scratch to satisfy established gRPC contracts across Scylla and Redis. The team resolved internal Go tooling dependencies by building custom configuration template parsers using the nom parser combinator and selected targeted open-source crates such as fred.rs and Cadence. Adapting to Rust required navigating cooperative, stackless async execution compared to Go's preemptive concurrency model, alongside managing borrow checker constraints. Ultimately, the rewrite achieved a 70% reduction in infrastructure costs while maintaining comparable service performance.
Jia Long LohCanva ·
Measuring Commercial Impact at Scale at Canva
Canva needed a scalable, standardized way to calculate the commercial impact of thousands of annual experiments on key business metrics like Monthly Active Users and Annual Recurring Revenue. Previously, disparate teams spent over six hours per experiment performing manual, error-prone calculations across fragmented spreadsheets and inconsistent data models. To resolve this, Canva built the IMPACT app using Snowflake, Streamlit, Snowpark, and Cortex to provide a self-serve platform tied directly to its central finance model. The application scales local uplift by actual audience exposure and supports pre-experiment scenario modeling alongside post-experiment tracking. A custom deployment workflow generates pull-request-isolated Streamlit environments in Snowflake stages, reducing time-to-insight to under ten minutes while enabling multiple developers to safely build and demo features in parallel.
Jun YeGrab ·
The complete stream processing journey on FlinkSQL
Grab previously relied on Apache Zeppelin notebooks for interactive stream processing exploration, but faced lagging Flink version upgrades, five-minute cluster cold starts, and poor integration with internal platforms. To address these limitations, the team migrated to a shared FlinkSQL gateway architecture structured into compute, integration, and query layers. The new setup uses a Hive Metastore catalog to expose Kafka topics as relational tables, while a custom control plane handles authentication and headless REST APIs over Flink's native interface. For production workflows, a configuration-based portal accepts SQL logic and automatically provisions and deploys Flink pipelines within ten minutes. This transition reduced ad-hoc query response times to under one minute and eliminated the need to maintain version adapter shims.
Calvin TranCanva ·
Yes, You Can Use AI in Our Interviews. In fact, we insist
Canva has updated its technical hiring process to require backend, machine learning, and frontend engineering candidates to utilize artificial intelligence tools such as Copilot, Cursor, and Claude during interviews. The transition addresses the limitations of traditional computer science fundamentals tests, which focused on writing algorithmic code from scratch even though AI assistants can generate complete solutions in seconds. To better evaluate on-the-job engineering capabilities, the company replaced its legacy screening with an AI-assisted coding competency that features complex, ambiguous product challenges such as designing an airport control system. Interviewers assess how candidates clarify requirements, guide tools on subtasks, debug flawed output, and verify that AI-generated code meets production quality standards. Candidates receive advance notice of these expectations, helping ensure prospective hires demonstrate strong engineering judgment when collaborating with assistive coding technologies.
Simon NewtonOdysseus to AI: Matt Greenwood on the Dev Interrupted Podcast
Matt Greenwood, Chief Innovation Officer at Two Sigma, outlines strategies for managing technological change, integrating artificial intelligence into systematic investment, and building supportive engineering organizations. Greenwood describes sustained innovation through the S-curve using an epsilon and omega approach, combining small iterative steps with a broad long-term vision. To direct resources amid rapid advances in machine learning and large language models, he presents a functional framework categorizing AI roles into advisory insights, oracle outcome validation, operational task automation, and agentic coordination. This categorization aims to automate routine workflows while keeping human creativity, control, and higher-level thinking at the center of the investment process. Additionally, the organization fosters employee engagement through initiatives such as an internal hacker lab where cross-disciplinary teams build projects ranging from robots to racing simulators.
Emily MajewskiTinder ·
How We Decomposed Tinder’s Monolith
Tinder faced significant agility and build performance challenges caused by an iOS codebase monolith containing over 1,000 files and 150,000 lines of code. Manual extraction efforts risked creating massive, unmanageable pull requests that would require constant rebasing against the main branch. To systematically decompose the target into Swift sub-modules, the team mapped declarations and references via the Swift compiler into a directed graph. They iteratively extracted leaf nodes with an in-degree of zero across sequential phases and automated common code adjustments, including module dependencies, imports, access control levels, and dependency injection. The automated decomposition completed in under six months with zero P0 incidents, reducing monolith build times by 78% and disallowing future additions to the monolith target.
TinderGrab ·
Effortless enterprise authentication at Grab: Dex in action
Grab needed a centralised system to simplify identity management, satisfy audit requirements, and standardise authentication across internal and external tools like Databricks and Datadog. The engineering team selected OpenID Connect as their standard protocol and adopted Dex, an open-source CNCF identity aggregator. Dex acts as an intermediary between applications and multiple identity providers to issue standardised OIDC tokens. To secure service-to-service communication, Grab implemented token exchange with trusted peer relationships rather than relying on privileged service accounts. Dex also provides a kill-switch mechanism that can route authentication traffic to an alternate provider during identity provider outages.
Kah Wei LeeGrab ·
From failure to success: The birth of GrabGPT, Grab’s internal ChatGPT
Grab's machine learning platform team initially faced overwhelming volumes of repetitive user inquiries across their internal support channels. An initial attempt to automate answers using the open-source chatbot-ui framework and GPT-3.5-turbo failed to scale because the 8,000-token context limit could not accommodate extensive documentation, and embedding search proved inadequate. The project then pivoted to create an internal conversational AI platform called GrabGPT by wiring chatbot-ui with Google authentication and Grab's catwalk model-serving infrastructure. The resulting internal service rapidly expanded across the organization, providing auditable interactions, multi-model support across OpenAI, Claude, and Gemini, and private network routing to safeguard corporate data.
Wenbo WeiGrab ·
Streamlining RiskOps with the SOP agent framework
Manual Account Takeover (ATO) investigations in Risk Operations traditionally demand intensive cross-referencing across systems, manual SQL execution, and high-pressure decision-making prone to human error. To resolve these bottlenecks, an SOP-driven LLM agent framework models investigative workflows as natural-language tree structures with explicit function notations like @function_name. Execution is coordinated between an SOP planner, which traverses the tree using a Depth-First Search strategy, and a Worker Agent that parses JSON-formatted steps to invoke database queries and APIs. Once all steps evaluate their decision criteria, the framework synthesizes the collected data into an actionable summary report. Implementing this architecture automated 87% of ATO cases and dropped average ticket handling time from 22 minutes to 3 minutes.
Fujiao LiuTinder ·
How On-Device AI Models Find Your Best Tinder Profile Photos
Tinder developed an on-device AI Photo Selector to recommend compelling profile photos directly from a user's camera roll without exposing private assets. To establish a reference face, the system extracts facial landmarks via Apple's Vision framework and normalizes the crop using affine transformations. To prevent funnel drop-offs from mandatory selfies, the architecture includes a fallback mechanism that detects a single recognizable face in existing profile photos. Processing workflows execute concurrently across thousands of local assets while downloading and initializing TensorFlow Lite models in parallel using Combine. A dedicated analytics queue synchronizes operation counts with DispatchGroup to ensure accurate metric capture before the global rollout.
TinderGrab ·
Introducing the SOP-driven LLM agent frameworks
Standard operating procedure (SOP) driven Large Language Model agent frameworks address common generative AI challenges such as hallucinations, non-standard output formats, and branching navigation errors. Workflows are represented as hierarchical trees where nodes encapsulate actions or decision points that can be created using a visual editor and annotated with explicit external function calls. Execution relies on a tripartite architecture consisting of a Depth-First Search planner module with backtracking, an adaptive worker agent that limits API exposure and compresses context, and a multilingual user agent. Supporting tools include a Graph Retrieval-Augmented Generation pipeline, a plugin system integrating Python and SQL, and a state stack for pausing workflows during human intervention. In production deployments for fraud and account takeover investigations, the framework automated up to 87% of cases while cutting handling times substantially.
Fujiao LiuGrab ·
Evaluating performance impact of removing Redis-cache from a Scylla-backed service
Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs.
Md RiyadhGrab ·
Facilitating Docs-as-Code implementation for users unfamiliar with Markdown
Adopting Docs-as-Code practices at Grab posed challenges for non-engineering team members who lacked familiarity with GitLab and Markdown syntax. To bridge this gap, Grab built a rich-text WYSIWYG TechDocs editor directly into their Backstage developer portal while preserving GitLab as the central storage backend. The team selected the Lexical framework over alternatives to enable deep customization and support non-standard elements like Kroki diagrams, Draw.io, and Excalidraw. The editor integrates OAuth 2.0 authentication to generate single-commit merge requests automatically and uses a split-screen live preview alongside concurrent session tracking to minimize Git conflicts. This approach lowered the technical barrier for non-engineers to contribute to documentation without disrupting standard repository workflows.
David KhuTinder ·
Tinder’s migration to Elasticsearch 8
Tinder modernized its recommendation system by migrating its core Elasticsearch infrastructure from an end-of-life Elasticsearch 6 deployment to Elasticsearch 8 on Kubernetes. The legacy setup relied on manual EC2 provisioning and custom scripts, preventing the engineering team from utilizing modern vector search capabilities while incurring substantial operational overhead. Tinder implemented Elastic Cloud on Kubernetes integrated with its in-house Infrastructure as Code framework, Scaffold, while refactoring custom Java ranking plugins to handle Elasticsearch 8 breaking changes like prohibited negative scoring. The migration was validated through offline benchmarking, online verification, and a three-wave A/B testing rollout. This transition completed with zero outages and under 0.2% data discrepancy, reducing p99 search latency by 12% to 56% and cutting annual infrastructure costs by over one million dollars.
TinderGrab ·
Improving Hugo stability and addressing oncall challenges through automation
Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem.
Shuguang XiangNextdoor ·
Scaling Nextdoor’s Datastores: Part 5
Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.
Slava MarkeyevNextdoor ·
Scaling Nextdoor’s Datastores: Part 4
Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set_if_version and del_if_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.
Ronak ShahNextdoor ·
Scaling Nextdoor’s Datastores: Part 3
Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.
Ronak Shah