---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Grab's service mesh evolution: From Consul to Istio](https://yomu.fyi/post/grab-s-service-mesh-evolution-from-consul-to-istio.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hilman Kurniawan
- Published: Jul 16, 2025

Grab operated over 1,000 microservices across hybrid infrastructure using Consul alongside a fallback mechanism called Catcher. Single-point-of-failure vulnerabilities in Consul servers and limited support for multi-cluster operations prompted an evaluation of alternative mesh technologies, ultimately leading to the selection of Istio. Grab avoided the standard single-control-plane-per-cluster pattern by deploying multiple external control planes in dedicated Kubernetes clusters arranged in active-active pairs. Migration began in Q4 2024, shifting traffic across AWS and GCP while handling both HTTP and gRPC protocols with gradual traffic-shifting and rollback mechanisms.


### [DispatchGym: Grab’s reinforcement learning research framework](https://yomu.fyi/post/dispatchgym-grab-s-reinforcement-learning-research-framework.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tan Sien Yi
- Published: Jul 7, 2025

Applying reinforcement learning to dispatch systems is often hindered when the chosen control levers exert weak influence over reward functions. To streamline research, Grab built DispatchGym, a framework that connects reinforcement learning algorithms to a dispatch process simulation via the Gymnasium API. The simulation emphasizes directional accuracy over absolute precision, allowing researchers to evaluate relative metric shifts across supply and demand scenarios. Built in modular Python and accelerated with Numba, the system allows data scientists to test code locally and launch distributed Spark executions with a single command-line call. The framework has been used to evaluate various contextual bandit models and action sampling strategies for tuning dispatch hyperparameters.


### [Counter Service: How we rewrote it in Rust](https://yomu.fyi/post/counter-service-how-we-rewrote-it-in-rust.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jia Long Loh
- Published: Jun 20, 2025

The Integrity Data Platform team rewrote Counter Service, a high-throughput Golang microservice serving event counts for fraud rules and machine learning models, to evaluate the operational return on investment of Rust. Rather than performing a line-by-line translation, engineers approached the service as a black box, reimplementing core application logic from scratch to satisfy established gRPC contracts across Scylla and Redis. The team resolved internal Go tooling dependencies by building custom configuration template parsers using the nom parser combinator and selected targeted open-source crates such as fred.rs and Cadence. Adapting to Rust required navigating cooperative, stackless async execution compared to Go's preemptive concurrency model, alongside managing borrow checker constraints. Ultimately, the rewrite achieved a 70% reduction in infrastructure costs while maintaining comparable service performance.


### [Measuring Commercial Impact at Scale at Canva](https://yomu.fyi/post/measuring-commercial-impact-at-scale-at-canva.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Jun Ye
- Published: Jun 20, 2025

Canva needed a scalable, standardized way to calculate the commercial impact of thousands of annual experiments on key business metrics like Monthly Active Users and Annual Recurring Revenue. Previously, disparate teams spent over six hours per experiment performing manual, error-prone calculations across fragmented spreadsheets and inconsistent data models. To resolve this, Canva built the IMPACT app using Snowflake, Streamlit, Snowpark, and Cortex to provide a self-serve platform tied directly to its central finance model. The application scales local uplift by actual audience exposure and supports pre-experiment scenario modeling alongside post-experiment tracking. A custom deployment workflow generates pull-request-isolated Streamlit environments in Snowflake stages, reducing time-to-insight to under ten minutes while enabling multiple developers to safely build and demo features in parallel.


### [The complete stream processing journey on FlinkSQL](https://yomu.fyi/post/the-complete-stream-processing-journey-on-flinksql.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Calvin Tran
- Published: Jun 12, 2025

Grab previously relied on Apache Zeppelin notebooks for interactive stream processing exploration, but faced lagging Flink version upgrades, five-minute cluster cold starts, and poor integration with internal platforms. To address these limitations, the team migrated to a shared FlinkSQL gateway architecture structured into compute, integration, and query layers. The new setup uses a Hive Metastore catalog to expose Kafka topics as relational tables, while a custom control plane handles authentication and headless REST APIs over Flink's native interface. For production workflows, a configuration-based portal accepts SQL logic and automatically provisions and deploys Flink pipelines within ten minutes. This transition reduced ad-hoc query response times to under one minute and eliminated the need to maintain version adapter shims.


### [Yes, You Can Use AI in Our Interviews. In fact, we insist](https://yomu.fyi/post/yes-you-can-use-ai-in-our-interviews-in-fact-we-insist.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Simon Newton
- Published: Jun 11, 2025

Canva has updated its technical hiring process to require backend, machine learning, and frontend engineering candidates to utilize artificial intelligence tools such as Copilot, Cursor, and Claude during interviews. The transition addresses the limitations of traditional computer science fundamentals tests, which focused on writing algorithmic code from scratch even though AI assistants can generate complete solutions in seconds. To better evaluate on-the-job engineering capabilities, the company replaced its legacy screening with an AI-assisted coding competency that features complex, ambiguous product challenges such as designing an airport control system. Interviewers assess how candidates clarify requirements, guide tools on subtasks, debug flawed output, and verify that AI-generated code meets production quality standards. Candidates receive advance notice of these expectations, helping ensure prospective hires demonstrate strong engineering judgment when collaborating with assistive coding technologies.


### [Odysseus to AI: Matt Greenwood on the Dev Interrupted Podcast](https://yomu.fyi/post/odysseus-to-ai-matt-greenwood-on-the-dev-interrupted-podcast.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: May 27, 2025

Matt Greenwood, Chief Innovation Officer at Two Sigma, outlines strategies for managing technological change, integrating artificial intelligence into systematic investment, and building supportive engineering organizations. Greenwood describes sustained innovation through the S-curve using an epsilon and omega approach, combining small iterative steps with a broad long-term vision. To direct resources amid rapid advances in machine learning and large language models, he presents a functional framework categorizing AI roles into advisory insights, oracle outcome validation, operational task automation, and agentic coordination. This categorization aims to automate routine workflows while keeping human creativity, control, and higher-level thinking at the center of the investment process. Additionally, the organization fosters employee engagement through initiatives such as an internal hacker lab where cross-disciplinary teams build projects ranging from robots to racing simulators.


### [How We Decomposed Tinder’s Monolith](https://yomu.fyi/post/how-we-decomposed-tinder-s-monolith.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: May 23, 2025

Tinder faced significant agility and build performance challenges caused by an iOS codebase monolith containing over 1,000 files and 150,000 lines of code. Manual extraction efforts risked creating massive, unmanageable pull requests that would require constant rebasing against the main branch. To systematically decompose the target into Swift sub-modules, the team mapped declarations and references via the Swift compiler into a directed graph. They iteratively extracted leaf nodes with an in-degree of zero across sequential phases and automated common code adjustments, including module dependencies, imports, access control levels, and dependency injection. The automated decomposition completed in under six months with zero P0 incidents, reducing monolith build times by 78% and disallowing future additions to the monolith target.


### [Effortless enterprise authentication at Grab: Dex in action](https://yomu.fyi/post/effortless-enterprise-authentication-at-grab-dex-in-action.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kah Wei Lee
- Published: May 23, 2025

Grab needed a centralised system to simplify identity management, satisfy audit requirements, and standardise authentication across internal and external tools like Databricks and Datadog. The engineering team selected OpenID Connect as their standard protocol and adopted Dex, an open-source CNCF identity aggregator. Dex acts as an intermediary between applications and multiple identity providers to issue standardised OIDC tokens. To secure service-to-service communication, Grab implemented token exchange with trusted peer relationships rather than relying on privileged service accounts. Dex also provides a kill-switch mechanism that can route authentication traffic to an alternate provider during identity provider outages.


### [From failure to success: The birth of GrabGPT, Grab’s internal ChatGPT](https://yomu.fyi/post/from-failure-to-success-the-birth-of-grabgpt-grab-s-internal-chatgpt.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenbo Wei
- Published: May 19, 2025

Grab's machine learning platform team initially faced overwhelming volumes of repetitive user inquiries across their internal support channels. An initial attempt to automate answers using the open-source chatbot-ui framework and GPT-3.5-turbo failed to scale because the 8,000-token context limit could not accommodate extensive documentation, and embedding search proved inadequate. The project then pivoted to create an internal conversational AI platform called GrabGPT by wiring chatbot-ui with Google authentication and Grab's catwalk model-serving infrastructure. The resulting internal service rapidly expanded across the organization, providing auditable interactions, multi-model support across OpenAI, Claude, and Gemini, and private network routing to safeguard corporate data.


### [Streamlining RiskOps with the SOP agent framework](https://yomu.fyi/post/streamlining-riskops-with-the-sop-agent-framework.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: May 8, 2025

Manual Account Takeover (ATO) investigations in Risk Operations traditionally demand intensive cross-referencing across systems, manual SQL execution, and high-pressure decision-making prone to human error. To resolve these bottlenecks, an SOP-driven LLM agent framework models investigative workflows as natural-language tree structures with explicit function notations like @function\_name. Execution is coordinated between an SOP planner, which traverses the tree using a Depth-First Search strategy, and a Worker Agent that parses JSON-formatted steps to invoke database queries and APIs. Once all steps evaluate their decision criteria, the framework synthesizes the collected data into an actionable summary report. Implementing this architecture automated 87% of ATO cases and dropped average ticket handling time from 22 minutes to 3 minutes.


### [How On-Device AI Models Find Your Best Tinder Profile Photos](https://yomu.fyi/post/how-on-device-ai-models-find-your-best-tinder-profile-photos.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Apr 28, 2025

Tinder developed an on-device AI Photo Selector to recommend compelling profile photos directly from a user's camera roll without exposing private assets. To establish a reference face, the system extracts facial landmarks via Apple's Vision framework and normalizes the crop using affine transformations. To prevent funnel drop-offs from mandatory selfies, the architecture includes a fallback mechanism that detects a single recognizable face in existing profile photos. Processing workflows execute concurrently across thousands of local assets while downloading and initializing TensorFlow Lite models in parallel using Combine. A dedicated analytics queue synchronizes operation counts with DispatchGroup to ensure accurate metric capture before the global rollout.


### [Introducing the SOP-driven LLM agent frameworks](https://yomu.fyi/post/introducing-the-sop-driven-llm-agent-frameworks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: Apr 25, 2025

Standard operating procedure (SOP) driven Large Language Model agent frameworks address common generative AI challenges such as hallucinations, non-standard output formats, and branching navigation errors. Workflows are represented as hierarchical trees where nodes encapsulate actions or decision points that can be created using a visual editor and annotated with explicit external function calls. Execution relies on a tripartite architecture consisting of a Depth-First Search planner module with backtracking, an adaptive worker agent that limits API exposure and compresses context, and a multilingual user agent. Supporting tools include a Graph Retrieval-Augmented Generation pipeline, a plugin system integrating Python and SQL, and a state stack for pausing workflows during human intervention. In production deployments for fraud and account takeover investigations, the framework automated up to 87% of cases while cutting handling times substantially.


### [Evaluating performance impact of removing Redis-cache from a Scylla-backed service](https://yomu.fyi/post/evaluating-performance-impact-of-removing-redis-cache-from-a-scylla-ba.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Md Riyadh
- Published: Apr 11, 2025

Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs.


### [Facilitating Docs-as-Code implementation for users unfamiliar with Markdown](https://yomu.fyi/post/facilitating-docs-as-code-implementation-for-users-unfamiliar-with-mar.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: David Khu
- Published: Apr 4, 2025

Adopting Docs-as-Code practices at Grab posed challenges for non-engineering team members who lacked familiarity with GitLab and Markdown syntax. To bridge this gap, Grab built a rich-text WYSIWYG TechDocs editor directly into their Backstage developer portal while preserving GitLab as the central storage backend. The team selected the Lexical framework over alternatives to enable deep customization and support non-standard elements like Kroki diagrams, Draw.io, and Excalidraw. The editor integrates OAuth 2.0 authentication to generate single-commit merge requests automatically and uses a split-screen live preview alongside concurrent session tracking to minimize Git conflicts. This approach lowered the technical barrier for non-engineers to contribute to documentation without disrupting standard repository workflows.


### [Tinder’s migration to Elasticsearch 8](https://yomu.fyi/post/tinder-s-migration-to-elasticsearch-8.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Apr 1, 2025

Tinder modernized its recommendation system by migrating its core Elasticsearch infrastructure from an end-of-life Elasticsearch 6 deployment to Elasticsearch 8 on Kubernetes. The legacy setup relied on manual EC2 provisioning and custom scripts, preventing the engineering team from utilizing modern vector search capabilities while incurring substantial operational overhead. Tinder implemented Elastic Cloud on Kubernetes integrated with its in-house Infrastructure as Code framework, Scaffold, while refactoring custom Java ranking plugins to handle Elasticsearch 8 breaking changes like prohibited negative scoring. The migration was validated through offline benchmarking, online verification, and a three-wave A/B testing rollout. This transition completed with zero outages and under 0.2% data discrepancy, reducing p99 search latency by 12% to 56% and cutting annual infrastructure costs by over one million dollars.


### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 20, 2025

Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem.


### [Scaling Nextdoor’s Datastores: Part 5](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-5.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Slava Markeyev
- Published: Mar 19, 2025

Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.


### [Scaling Nextdoor’s Datastores: Part 4](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-4.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025

Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db\_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set\_if\_version and del\_if\_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.


### [Scaling Nextdoor’s Datastores: Part 3](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-3.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025

Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.


[Newer posts](https://yomu.fyi/page/30.md) · [Older posts](https://yomu.fyi/page/32.md)