Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Grab's service mesh evolution: From Consul to Istio
Grab describes migrating its service mesh from Consul and the Catcher fallback across a microservices ecosystem of more than 1,000 services spanning VMs, EKS clusters, AWS, and GCP. The existing setup exposed fleet-wide risk from a single Consul server issue, added fallback complexity, and lacked advanced traffic management, security controls, and robust multi-cluster support. After evaluating Application Load Balancer, Istio, AWS Lattice, and Linkerd for stability, performance, integration, and operational simplicity, the team selected Istio for its Kubernetes integration, flexible deployment models, and feature set. The design uses multiple control planes on dedicated Kubernetes clusters, active-active pairs, organization-based mesh boundaries, custom resource definitions, and cross-cluster endpoint discovery to support isolation, availability, routing, security, and observability. Migration began in Q4 2024 with GCP-to-AWS work alongside gRPC and HTTP traffic initiatives, using gradual shifting, rollback mechanisms, and performance monitoring while teams manage legacy systems and cross-mesh connectivity.
Remixing Shopify's Admin: How We Made It 30% Faster and AI-Ready
Shopify rebuilt its Admin architecture to address inconsistent loading, duplicate requests, and fragmented route knowledge across a TypeScript codebase serving 67 million daily page views. With 101 teams contributing about 350 pull requests daily and the codebase containing 3,184,248 lines of TypeScript, the work adopted Remix with Vite and SPA mode. Typed route manifests, globbed by Vite, now define metadata and a single source of truth for 1,017 routes, while Remix loaders start data fetching before large component bundles finish loading. Idle-time prefetching warms the Apollo cache, and persistent IndexDB storage helps frequently visited pages load between sessions. The changes delivered a 30% improvement in perceived load times, standardized transitions, and created an architecture in which Remix actions and zod schemas let Sidekick discover forms, validate inputs, and submit tasks without loading unnecessary JavaScript.
Implementing scalable metered billing with Stripe: How Edgee handles billions of events
Edgee needed usage-based billing for a business combining flat-rate subscriptions with charges for proxy requests and delivered data events, while processing billions of monthly requests across more than 100 edge locations. Its architecture captures web data at the edge through a proxy and records raw metering data centrally in Google BigQuery. To control several terabytes of data, Edgee aggregates usage by tenant and sends hourly Stripe meter events through AWS Lambda, with graduated pricing and idempotency handled by Stripe. A Go configuration defines meters, prices, flat-rate plans, and pricing tiers, while retry logic and idempotent updates support resilience during disruptions. The resulting billing system remains stateless and low-latency while providing accurate monthly invoicing and a simple hosted customer interface.
Counter Service: How we rewrote it in Rust
The Integrity Data Platform team rewrote Counter Service, a high-QPS Golang microservice that consumes streams, writes counts to Scylla, and exposes gRPC queries backed by Scylla and Redis. They selected it because BatchRead reached tens of thousands of QPS while remaining constrained in functionality, making infrastructure savings potentially worth the rewrite effort and Rust learning curve. Rather than translating code line by line, the team treated the existing service as a black box, preserved its interfaces and contracts, and rebuilt the application logic from scratch; it also assessed Rust ecosystem support for dependencies including tonic, Actix Web, Fred, and the Scylla driver. The rewrite delivered similar performance with 70% infrastructure savings, according to the abstract. The conclusion reports the service as 80% cheaper, endorses Rust for new services, and cautions that async Rust is challenging.
Measuring Commercial Impact at Scale at Canva
Canva needed a scalable, standardized way to calculate the commercial impact of thousands of annual experiments on key business metrics like Monthly Active Users and Annual Recurring Revenue. Previously, disparate teams spent over six hours per experiment performing manual, error-prone calculations across fragmented spreadsheets and inconsistent data models. To resolve this, Canva built the IMPACT app using Snowflake, Streamlit, Snowpark, and Cortex to provide a self-serve platform tied directly to its central finance model. The application scales local uplift by actual audience exposure and supports pre-experiment scenario modeling alongside post-experiment tracking. A custom deployment workflow generates pull-request-isolated Streamlit environments in Snowflake stages, reducing time-to-insight to under ten minutes while enabling multiple developers to safely build and demo features in parallel.
How Ramp Fixes Merchant Matches with AI
Ramp describes the difficulty of mapping payment-processor card acceptor data to useful merchant records when names, MCCs, and locations are vague, misleading, or affected by rebranding. Users can submit a replacement merchant name, website, and category, but deciding whether the request is legitimate and whether to update an existing record or create one requires additional context from related receipts, extracted line items, and transaction memos. Ramp addresses this with an AI agent built around an LLM, embeddings, rapid online analytical processing queries, multimodal retrieval-augmented generation, and guardrails that constrain permitted changes. The agent handles requests in under 10 seconds, and monitoring indicates it improves nearly 99% of transaction classifications, while fewer than 10% receive a second correction request. The same flow has also been extended to batch remapping and credit-card statement matching.
The complete stream processing journey on FlinkSQL
The post describes replacing per-user Zeppelin notebooks with a shared FlinkSQL gateway to improve interactive stream processing over Kafka data and support production pipelines. Zeppelin notebooks were limited by support for Flink 1.17, roughly five-minute cold starts, and isolation from internal platforms, while current Flink was 1.20. The replacement uses compute, integration, and query layers: a shared FlinkSQL gateway submits SQL through its REST API to session-cluster jobs, while a custom control plane handles authentication, authorization, session management, orchestration, and paginated results for UI and headless API clients. An interactive UI uses a Hive Metastore catalog to expose Kafka topics as tables, and the platform returns results in one minute rather than five; a configuration-based deployment tool provisions resources and deploys SQL-defined Flink pipelines within ten minutes. After release, the post reports an uptick in created pipelines and interactive queries, but it does not quantify that increase.
Navigating The AI Revolution: The Importance Of Adaptation
A TDWI survey conducted with MongoDB and Capgemini found that only 11% of companies have AI applications in production, despite widespread recognition of AI’s value. The report identifies data readiness challenges including managing diverse data types, ensuring accessibility, and providing sufficient compute power, with less than half of companies managing newer data formats and only 41% feeling they have enough compute. The post argues that AI applications are dynamic, continuously adapting and learning from user behavior, requiring a flexible data foundation unlike traditional static software. MongoDB positions its modern database solutions as essential, citing examples like Lombard Odier’s modernization with generative AI tooling and Victoria’s Secret’s AI-powered visual search using MongoDB Atlas Vector Search. The post concludes that organizations must modernize their tech stacks and execution to keep pace with AI-driven change, or risk being left behind.
Resilient Import Maps - Better Theme Development and Beyond
Shopify found that JavaScript modules and import maps failed in themes and App Blocks when modules loaded before a map, or when multiple maps collided. These restrictions reflected the requirement that module resolution remain stable, but real pages assembled by themes, apps, Shopify, and extensions could not reliably guarantee ordering or a single map. Working with web-platform contributors, Shopify proposed a reconciliation mechanism with one global import map, a resolved module set, and merging rules that preserve existing resolutions and drop conflicting or unsafe additions. The change reached the HTML specification and shipped in Chromium and WebKit; Chrome supports it from version 133 and Safari from 18.4. Shopify also uses es-modules-shim support for browsers without native support, while Firefox support is hoped for.
How We Decomposed Tinder’s Monolith
Tinder faced significant agility and build performance challenges caused by an iOS codebase monolith containing over 1,000 files and 150,000 lines of code. Manual extraction efforts risked creating massive, unmanageable pull requests that would require constant rebasing against the main branch. To systematically decompose the target into Swift sub-modules, the team mapped declarations and references via the Swift compiler into a directed graph. They iteratively extracted leaf nodes with an in-degree of zero across sequential phases and automated common code adjustments, including module dependencies, imports, access control levels, and dependency injection. The automated decomposition completed in under six months with zero P0 incidents, reducing monolith build times by 78% and disallowing future additions to the monolith target.
Effortless enterprise authentication at Grab: Dex in action
Grab built Concedo to centralise access management, authentication, and authorisation across internal applications and external platforms. Its customised Google OAuth2.0-based mechanism created integration challenges with systems such as Databricks and Datadog, producing fragmented sign-on experiences and administrative overhead. To standardise identity, the team adopted OpenID Connect (OIDC) and used Dex, a federated OpenID Connect provider that aggregates identity providers and issues standardised OIDC tokens. Dex supports single sign-on, multiple identity providers, token delegation between trusted peer services, and a kill switch that can redirect traffic during an identity provider outage. The implementation separated application authentication concerns and established a single identity source, while a future phase will address fragmented authorisation through a unified model.
From failure to success: The birth of GrabGPT, Grab’s internal ChatGPT
In March 2023, an effort to reduce repetitive questions overwhelming Grab’s ML Platform support channels began as a chatbot for platform documentation and became GrabGPT, an internal ChatGPT-like tool. The initial chatbot used chatbot-ui and more than 20,000 words of Q&A documentation, but GPT-3.5-turbo’s 8,000-token limit forced summaries below 800 words; embedding search also performed poorly, so the approach was abandoned. The pivot extended existing frameworks, added Google login, and deployed a tool through Grab’s catwalk model-serving platform, later named GrabGPT. It reached 300 registered users on day one, more than 3,000 users and 600 daily active users by month three, and nearly all Grabbers according to the post, with private routing, global accessibility, multi-provider model support, and auditable interactions cited as reasons for adoption.
Streamlining RiskOps with the SOP agent framework
Risk Operations teams investigating Account Takeover (ATO) cases must gather data across systems, run SQL queries, and make decisions under pressure, creating delays and error risk. The SOP-driven LLM agent framework represents the workflow as a natural-language, indentation-based tree whose steps specify queries, external functions, and decision criteria. During execution, the SOP planner navigates the tree and produces JSON-formatted instructions, while a Worker Agent invokes APIs or SQL queries and stores results in a feedback loop. The process ends by synthesising investigation data into a report and final decision. In reported results, handling time fell from 22 to 3 minutes per ticket, 87% of ATO cases were automated since launch, and the error rate was zero.
How On-Device AI Models Find Your Best Tinder Profile Photos
Tinder developed an on-device AI Photo Selector to recommend compelling profile photos directly from a user's camera roll without exposing private assets. To establish a reference face, the system extracts facial landmarks via Apple's Vision framework and normalizes the crop using affine transformations. To prevent funnel drop-offs from mandatory selfies, the architecture includes a fallback mechanism that detects a single recognizable face in existing profile photos. Processing workflows execute concurrently across thousands of local assets while downloading and initializing TensorFlow Lite models in parallel using Combine. A dedicated analytics queue synchronizes operation counts with DispatchGroup to ensure accurate metric capture before the global rollout.
Mobile Bridge: Making WebViews Feel Native
Shopify’s mobile app has about 600 screens, but building every non-critical screen natively or in React Native was costly and slowed development, while conventional WebViews felt slow, disconnected, and out of place. Mobile Bridge addresses this hybrid-app problem with native iOS and Android modules that preload, authenticate, cache, and reuse WebViews, reducing P75 load time from 6 seconds to 1.4 seconds, including network latency and rendering. Built on @remote-ui/rpc, the framework uses JavaScript, CSS, Polaris overrides, title and action APIs, TransportableView, snapshots, and native modal transitions to preserve state and align web screens with native navigation and appearance. WebViews can also invoke native features such as date pickers, barcode scanning, and Add to Wallet; Shopify says the standalone library is being adopted across other products while critical features remain native or React Native.
Optimizing Stripe API performance in Lambda with caching strategies
The post addresses performance, cost, latency, and rate-limiting challenges that arise when applications make high-volume requests to Stripe's API from AWS Lambda. It presents a two-level cache using ElastiCache for Redis as the first layer and DynamoDB as the durable second layer, with Lambda coordinating cache reads, Stripe retrieval, and writes. On a miss, the function checks Redis, then DynamoDB, fetches the customer from Stripe if necessary, and stores the response in both layers with one-hour Redis and 24-hour DynamoDB TTLs. The DynamoDB design uses composite keys, a GSI for cache-type and timestamp queries, TTL cleanup, and environment-specific capacity settings; the post also discusses Redis connection pooling, Lambda sizing, capacity planning, cost control, monitoring, and webhook-based invalidation.
Evaluating performance impact of removing Redis-cache from a Scylla-backed service
A Rust service at Grab serves tens of thousands of peak QPS for batched count requests, using Scylla, Redis, and MySQL counter configurations. Its 15-minute cache-key truncation and five-minute TTL caused cache-hit declines and recurring Scylla QPS spikes at the end of each quarter-hour, when recent-data requests generated new keys. The experiment progressively disabled Redis, while a daily major compaction job was added to Scylla nodes; the service continued combining minutely, hourly, and daily tables for range calculations. With Redis fully decommissioned, Scylla cache hits rose from about 30% to 70% of reads, cache misses stayed about the same despite doubled traffic, peak CPU moved from roughly 50% to 52%, and overall service performance was maintained while eliminating a component responsible for about 25% of service cost.
Because nobody likes being charged twice
Payment workflows can produce duplicate charges when a customer retries after a connection drop while the backend is still retrying the original request. The post frames network timeouts, crashes, database locks, downstream errors, and user interruptions as normal distributed-systems failures that can leave payment state uncertain. It recommends generating a unique idempotency key for one specific action, sending it with the Stripe request, and reusing it on every retry; Stripe recognizes the key for 24 hours and returns the original response. It also recommends placing payment jobs in a queue such as Amazon SQS, where workers retry failures and route repeatedly failing messages to a dead-letter queue. Together, queues preserve work during temporary outages and idempotency makes retries safe, while careful key scope and timing avoid stale or cross-operation behavior.
Facilitating Docs-as-Code implementation for users unfamiliar with Markdown
Grab introduced a rich-text, WYSIWYG editor in its TechDocs portal to help non-engineering contributors work within a Docs-as-Code process despite limited familiarity with Markdown and GitLab. The editor keeps GitLab as the central storage system, uses Lexical to import and export Markdown, and supports page reordering, renaming, deletion, formatting, diagrams, and switching between normal and Markdown views. Three user journeys cover editing existing TechDocs and creating stand-alone or entity-linked documentation, with GitLab OAuth2, merge-request creation, repository access checks, and scheduler-based entity-catalog registration after merges. A phased rollout added drafts, image uploads, diagram integrations, concurrent-session visibility, and live preview, while pending-MR displays and conflict-resolution tools address collaboration risks.
Tinder’s migration to Elasticsearch 8
Tinder modernized its recommendation system by migrating its core Elasticsearch infrastructure from an end-of-life Elasticsearch 6 deployment to Elasticsearch 8 on Kubernetes. The legacy setup relied on manual EC2 provisioning and custom scripts, preventing the engineering team from utilizing modern vector search capabilities while incurring substantial operational overhead. Tinder implemented Elastic Cloud on Kubernetes integrated with its in-house Infrastructure as Code framework, Scaffold, while refactoring custom Java ranking plugins to handle Elasticsearch 8 breaking changes like prohibited negative scoring. The migration was validated through offline benchmarking, online verification, and a three-wave A/B testing rollout. This transition completed with zero outages and under 0.2% data discrepancy, reducing p99 search latency by 12% to 56% and cutting annual infrastructure costs by over one million dollars.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.