---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [From failure to success: The birth of GrabGPT, Grab’s internal ChatGPT](https://yomu.fyi/post/from-failure-to-success-the-birth-of-grabgpt-grab-s-internal-chatgpt.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenbo Wei
- Published: May 19, 2025

Grab's machine learning platform team initially faced overwhelming volumes of repetitive user inquiries across their internal support channels. An initial attempt to automate answers using the open-source chatbot-ui framework and GPT-3.5-turbo failed to scale because the 8,000-token context limit could not accommodate extensive documentation, and embedding search proved inadequate. The project then pivoted to create an internal conversational AI platform called GrabGPT by wiring chatbot-ui with Google authentication and Grab's catwalk model-serving infrastructure. The resulting internal service rapidly expanded across the organization, providing auditable interactions, multi-model support across OpenAI, Claude, and Gemini, and private network routing to safeguard corporate data.


### [Streamlining RiskOps with the SOP agent framework](https://yomu.fyi/post/streamlining-riskops-with-the-sop-agent-framework.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: May 8, 2025

Manual Account Takeover (ATO) investigations in Risk Operations traditionally demand intensive cross-referencing across systems, manual SQL execution, and high-pressure decision-making prone to human error. To resolve these bottlenecks, an SOP-driven LLM agent framework models investigative workflows as natural-language tree structures with explicit function notations like @function\_name. Execution is coordinated between an SOP planner, which traverses the tree using a Depth-First Search strategy, and a Worker Agent that parses JSON-formatted steps to invoke database queries and APIs. Once all steps evaluate their decision criteria, the framework synthesizes the collected data into an actionable summary report. Implementing this architecture automated 87% of ATO cases and dropped average ticket handling time from 22 minutes to 3 minutes.


### [How to easily charge customers across accounts in an organization](https://yomu.fyi/post/how-to-easily-charge-customers-across-accounts-in-an-organization.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Daniel Wood
- Published: May 7, 2025

Stripe’s Organizations customer and payment method sharing addresses the difficulty of giving customers a unified checkout when a business uses separate accounts for business lines or countries. After access is enabled, all new and existing customers and their saved card payment methods can be shared between selected accounts, using the same IDs, so a saved card can be charged in another account without recollection. Only selected customer fields—such as name, email, phone, tax IDs, and preferred locales—are shared; metadata, shipping, and other fields remain account-specific, while non-card payment methods cannot be charged cross-account. The guide demonstrates testing in an organization sandbox, creating Checkout Sessions with the shared customer ID, viewing organization-wide transactions, and using a single organization webhook; sharing is in preview and requires customer consent.


### [Introducing Stripe Workflows: Tailoring Payments to Your Business Needs](https://yomu.fyi/post/introducing-stripe-workflows-tailoring-payments-to-your-business-needs.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ben Smith
- Published: May 7, 2025

Stripe introduces Stripe Workflows, a Dashboard-based system for orchestrating Stripe products and APIs through responsive, end-to-end automation. It addresses the complexity of coordinating multiple Stripe products by replacing custom event-handling and dependency-management code with visual workflows that support real-time triggers, branching logic, dynamic fields, and more than 600 event triggers. Workflows includes execution monitoring with event-payload and step-level tracing, 30-day default log retention, exponential-backoff retries, built-in idempotency, and recursion protection capped at a depth of five. The post describes applications including reducing chargeback risk, routing high-value transactions for compliance review, and creating custom-priced subscriptions from finalized quotes without additional infrastructure.


### [Introducing Stripe's new public preview release channel](https://yomu.fyi/post/introducing-stripe-s-new-public-preview-release-channel.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Paul Anderson
- Published: May 7, 2025

Stripe announces its new public preview release channel, which provides early access to API features before their General Availability launches. Preview features are described as operationally stable and production-ready, but they may receive breaking changes more often than GA features and may have geographic, onboarding, or approval restrictions. Each preview uses a distinct API version ending in preview, such as 2025-04-30.preview, selected through the Stripe-Version header; preview versions also appear in the API reference and changelog. Beta SDKs will use these preview versions and include preview features, with releases differentiated from GA counterparts by suffixes such as 5.2.0-beta.1. The example shows that a feature released only in preview is rejected by a GA version, requiring an explicit version change before use; this separation lets teams integrate earlier and provide feedback while avoiding unintended adoption of features with known restrictions.


### [Sessions 2025 Developer Track resources](https://yomu.fyi/post/sessions-2025-developer-track-resources.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ashley Ansari
- Published: May 7, 2025

Stripe’s Sessions 2025 developer track recap presents new products, APIs, and enhancements for building and managing financial infrastructure. Stripe’s Workflows provides visual automation for real-time events, with more than 600 triggers, branching logic, dynamic fields, and built-in error handling, while the MCP server and agent toolkit connect AI tools and agents to Stripe documentation, account data, APIs, and financial capabilities. Accounts v2 unifies connected-account management, and fully localized Connect Mobile Embedded Components bring onboarding to iOS and Android; the Payment Element also gains support for the Checkout Sessions API. New Stripe Apps UI components, generally available Sandboxes, Organization-wide sharing of customers and payment methods, and conversational documentation extend customization, testing, and collaboration.


### [How On-Device AI Models Find Your Best Tinder Profile Photos](https://yomu.fyi/post/how-on-device-ai-models-find-your-best-tinder-profile-photos.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Apr 28, 2025

Tinder developed an on-device AI Photo Selector to recommend compelling profile photos directly from a user's camera roll without exposing private assets. To establish a reference face, the system extracts facial landmarks via Apple's Vision framework and normalizes the crop using affine transformations. To prevent funnel drop-offs from mandatory selfies, the architecture includes a fallback mechanism that detects a single recognizable face in existing profile photos. Processing workflows execute concurrently across thousands of local assets while downloading and initializing TensorFlow Lite models in parallel using Combine. A dedicated analytics queue synchronizes operation counts with DispatchGroup to ensure accurate metric capture before the global rollout.


### [Introducing the SOP-driven LLM agent frameworks](https://yomu.fyi/post/introducing-the-sop-driven-llm-agent-frameworks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: Apr 25, 2025

Standard operating procedure (SOP) driven Large Language Model agent frameworks address common generative AI challenges such as hallucinations, non-standard output formats, and branching navigation errors. Workflows are represented as hierarchical trees where nodes encapsulate actions or decision points that can be created using a visual editor and annotated with explicit external function calls. Execution relies on a tripartite architecture consisting of a Depth-First Search planner module with backtracking, an adaptive worker agent that limits API exposure and compresses context, and a multilingual user agent. Supporting tools include a Graph Retrieval-Augmented Generation pipeline, a plugin system integrating Python and SQL, and a state stack for pausing workflows during human intervention. In production deployments for fraud and account takeover investigations, the framework automated up to 87% of cases while cutting handling times substantially.


### [Optimizing Stripe API performance in Lambda with caching strategies](https://yomu.fyi/post/optimizing-stripe-api-performance-in-lambda-with-caching-strategies.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Apr 21, 2025

The post addresses performance, cost, latency, and rate-limiting challenges that arise when applications make high-volume requests to Stripe's API from AWS Lambda. It presents a two-level cache using ElastiCache for Redis as the first layer and DynamoDB as the durable second layer, with Lambda coordinating cache reads, Stripe retrieval, and writes. On a miss, the function checks Redis, then DynamoDB, fetches the customer from Stripe if necessary, and stores the response in both layers with one-hour Redis and 24-hour DynamoDB TTLs. The DynamoDB design uses composite keys, a GSI for cache-type and timestamp queries, TTL cleanup, and environment-specific capacity settings; the post also discusses Redis connection pooling, Lambda sizing, capacity planning, cost control, monitoring, and webhook-based invalidation.


### [Evaluating performance impact of removing Redis-cache from a Scylla-backed service](https://yomu.fyi/post/evaluating-performance-impact-of-removing-redis-cache-from-a-scylla-ba.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Md Riyadh
- Published: Apr 11, 2025

Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs.


### [Because nobody likes being charged twice](https://yomu.fyi/post/because-nobody-likes-being-charged-twice.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ben Smith
- Published: Apr 10, 2025

Payment workflows can produce duplicate charges when a customer retries after a connection drop while the backend is still retrying the original request. The post frames network timeouts, crashes, database locks, downstream errors, and user interruptions as normal distributed-systems failures that can leave payment state uncertain. It recommends generating a unique idempotency key for one specific action, sending it with the Stripe request, and reusing it on every retry; Stripe recognizes the key for 24 hours and returns the original response. It also recommends placing payment jobs in a queue such as Amazon SQS, where workers retry failures and route repeatedly failing messages to a dead-letter queue. Together, queues preserve work during temporary outages and idempotency makes retries safe, while careful key scope and timing avoid stale or cross-operation behavior.


### [Using an AWS microservice architecture for subscription management](https://yomu.fyi/post/using-an-aws-microservice-architecture-for-subscription-management.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Apr 8, 2025

Enterprise subscription management must handle mid-cycle plan changes, usage-based billing adjustments, synchronized service updates, atomicity, failed operations, data consistency, and auditability beyond basic recurring billing. This sample architecture uses an event-driven AWS design in which subscription changes are published to EventBridge and routed to Step Functions workflows. EventBridge provides at-least-once delivery, content-based filtering, dead-letter queue support, and integration with Step Functions, while Lambda functions validate changes, calculate Stripe proration, update external subscriptions, and trigger service-specific actions. Step Functions supplies retries with exponential backoff for StripeTemporaryError, DynamoDB stores current and historical subscription data using transactional updates, and Stripe calls use an event identifier as an idempotency key. The pattern is presented as a flexible foundation for reliability, maintainability, compensation for partial changes, and auditability in systems handling financial transactions and service provisioning.


### [Facilitating Docs-as-Code implementation for users unfamiliar with Markdown](https://yomu.fyi/post/facilitating-docs-as-code-implementation-for-users-unfamiliar-with-mar.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: David Khu
- Published: Apr 4, 2025

Adopting Docs-as-Code practices at Grab posed challenges for non-engineering team members who lacked familiarity with GitLab and Markdown syntax. To bridge this gap, Grab built a rich-text WYSIWYG TechDocs editor directly into their Backstage developer portal while preserving GitLab as the central storage backend. The team selected the Lexical framework over alternatives to enable deep customization and support non-standard elements like Kroki diagrams, Draw.io, and Excalidraw. The editor integrates OAuth 2.0 authentication to generate single-commit merge requests automatically and uses a split-screen live preview alongside concurrent session tracking to minimize Git conflicts. This approach lowered the technical barrier for non-engineers to contribute to documentation without disrupting standard repository workflows.


### [Tinder’s migration to Elasticsearch 8](https://yomu.fyi/post/tinder-s-migration-to-elasticsearch-8.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Apr 1, 2025

Tinder modernized its recommendation system by migrating its core Elasticsearch infrastructure from an end-of-life Elasticsearch 6 deployment to Elasticsearch 8 on Kubernetes. The legacy setup relied on manual EC2 provisioning and custom scripts, preventing the engineering team from utilizing modern vector search capabilities while incurring substantial operational overhead. Tinder implemented Elastic Cloud on Kubernetes integrated with its in-house Infrastructure as Code framework, Scaffold, while refactoring custom Java ranking plugins to handle Elasticsearch 8 breaking changes like prohibited negative scoring. The migration was validated through offline benchmarking, online verification, and a three-wave A/B testing rollout. This transition completed with zero outages and under 0.2% data discrepancy, reducing p99 search latency by 12% to 56% and cutting annual infrastructure costs by over one million dollars.


### [Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery](https://yomu.fyi/post/unlocking-powerful-analytics-with-stripe-data-pipeline-and-google-bigq.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Sushant Jain
- Published: Mar 21, 2025

As businesses scale, the post addresses the need to move Stripe revenue data into BigQuery for recurring reporting and analysis. It describes configuring Stripe Data Pipeline to export transactions, invoices, subscriptions, and payments to Google Cloud Storage, then using a Python-based Google Cloud Run Job to discover the latest snapshot through data\_load\_times.json and enumerate each table’s Parquet files. The loader sends those files to BigQuery with the Google Cloud clients, creates missing tables, uses WRITE\_TRUNCATE to retain the latest snapshot, and parallelizes table loads with ThreadPoolExecutor while collecting errors. After deployment and a test execution, the workflow is verified in BigQuery and scheduled to run every six hours, providing an automated path to refreshed Stripe data for analytics.


### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 20, 2025

Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem.


### [Scaling Nextdoor’s Datastores: Part 5](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-5.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Slava Markeyev
- Published: Mar 19, 2025

Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.


### [Scaling Nextdoor’s Datastores: Part 4](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-4.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025

Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db\_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set\_if\_version and del\_if\_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.


### [Scaling Nextdoor’s Datastores: Part 3](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-3.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025

Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.


### [Scaling Nextdoor’s Datastores: Part 2](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-2.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Tushar Singla
- Published: Mar 19, 2025

Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.


[Newer posts](https://yomu.fyi/page/32.md) · [Older posts](https://yomu.fyi/page/34.md)