Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Road localisation in GrabMaps
Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation.
Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling
Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%.
Finding the right balance of speed and security through just-in-time access to cloud resources
Ramp’s rapid growth exposed problems in its AWS access model: three broad roles created persistent production access for some senior engineers while junior engineers depended on seniors for queries. The team first reorganized permissions into more than 20 team-specific AWS Identity Center groups, each backed by permission sets and a Terraform module that presents IAM configuration through human-readable resource lists. It then introduced just-in-time access through ConductorOne by pairing each non-production role with an elevated production role, removing standing production access and provisioning entitlements through AWS SSO group membership. Team-owned requests were automatically approved, while cross-team requests went to the relevant tech lead, with temporary access and backup approval handling described in the design. The resulting model reduced exposure, preserved fast access to needed resources, and recorded production activity for accountability and audit purposes.
Scaling marketing for merchants with targeted and intelligent promos
Grab previously relied on globally assigned, heuristic promotional campaigns where all users could redeem offers until limits were reached. This lack of targeting and customisation failed to optimize promo spending or meet specific merchant business objectives. To solve this, the engineering team built Bullseye, an automated AI-driven promotional assignment system that customises and assigns offers to consumers. The architecture integrates an operations user interface, backend services, Amazon S3 storage, and Spark jobs executing a suite of predictive modules including eater segmentation, campaign impact simulation, and customer response modeling. Since deploying Bullseye in 2021, the system has increased food campaign sales while decreasing promo spend across food and Mart campaigns.
Stepping up marketing for advertisers: Scalable lookalike audience
Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled.
How Ramp Accelerated Machine Learning Development to Simplify Finance
Ramp describes how its machine-learning organization used Metaflow to shorten the path from prototype to production for models spanning credit risk, fraud, growth, product, and Ramp Intelligence. Its initial riskiness model exposed long vendor-managed jobs, weak logging, limited Docker support, and substantial platform friction; the model took months to build. Metaflow, with developer-experience improvements, lets teams define flows in Python, run them locally or on AWS Batch, visualize results with cards, and connect production execution to Step Functions and Airflow. Ramp chose AWS-managed infrastructure, initially using Fargate before encountering startup and resource constraints, and later built a MetaflowOperator to simplify Airflow triggering and log access. After adoption, Ramp shipped eight additional models in ten months, recorded more than 6,000 Flow runs, and reports that data scientists can largely self-service while platform engineers spend less time debugging infrastructure.
From Pre-trained to Fine-tuned: Nextdoor’s Path to Effective Embedding Applications
Nextdoor transitioned its ranking and recommendation pipelines from traditional continuous and discrete interaction features to transformer-based representation learning. The engineering team deployed pre-trained Sentence-BERT models to generate multilingual post and comment representations, which were aggregated daily by interaction type to form user embeddings. To improve search recall, the team fine-tuned sentence transformers on unlabeled query session logs using contrastive learning and integrated HNSWlib for approximate nearest neighbor retrieval. Subsequent iterations incorporated labeled feedback, BERTopic for coarse personalization, and experiments with CLIP image embeddings. Infrastructure scaling challenges were addressed by performing embedding transformations directly within FeatureStore and optimizing feature payload formats to minimize microservice network bandwidth.
The So-fine Real-time ML Paradigm
During Etsy’s CodeMosaic hackathon, a team investigated stateful training and online learning to reduce the roughly 40-hour delay between user actions and model-weight updates. They split the work across real-time training data, an incremental-learning service, and evaluation, using Etsy’s Beacon Main Kafka stream, Kafka SQL, Rivulet, and TensorFlow-based Ads models. The data team joined multiple sources despite Avro schema difficulties, while the service team loaded an older model and nearly implemented incremental weight updates; the planned head-to-head evaluation against daily batch training was not completed. The project estimated that replacing daily cold-start training for four ads models could save about $212,000 annually in Google Cloud costs, with possible metric gains from earlier events. Production would still require infrastructure and safeguards against sudden performance degradation.
Building hyperlocal GrabMaps
Southeast Asia presents unique mapping challenges, including fast-changing road restrictions, new roads appearing rapidly, and severe traffic congestion. To address these conditions, GrabMaps integrates hyperlocal data derived from local team knowledge, rides and deliveries GPS pings, and KartaView imagery with OpenStreetMap data. The pipeline downloads OpenStreetMap .pbf files for Asia and converts them into Parquet format using a Parquetizer. GrabMaps processes spatial data using Apache Spark and bi-directional graphs divided by geohash to detect complex features such as intersection links based on road parallelism, naming, direction, and shape geometry. Tagging these intersection links enables downstream navigation services like GrabNav to calculate shorter routing alternatives and provide more accurate pricing.
Tracking Application-Level Consistency with LiteFS
LiteFS must keep an application’s view consistent when a write reaches a primary and a subsequent read is served by a lagging replica. The project first considered making applications track each database’s replication position—a TXID paired with a post-apply checksum—and wait for replicas to catch up, but rejected that complexity and abandoned language-specific libraries. It instead introduces a thin HTTP proxy in front of the application: method-based request inspection routes writes to the primary, attaches the committed TXID in a cookie, and delays reads until the local replica reaches that position or times out. The proxy also exposes lag through health checks, allowing Fly.io to remove unhealthy replicas from rotation and restore them after reconnection. This design provides read-your-writes guarantees with little application change, but depends on cookies and conventional HTTP request/response traffic, limiting use with some applications and websocket-heavy frameworks.
Streamlining Grab's Segmentation Platform with faster creation and lower latency
Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies.
Unsupervised graph anomaly detection - Catching new fraudulent behaviours
Fraud detection systems relying on historical training data often struggle to detect new fraudulent modus operandi because emerging patterns lack labeled supervision. Grab developed GraphBEAN, an unsupervised autoencoder model designed to detect anomalous patterns across bipartite interaction graphs representing consumers and merchants. The architecture uses graph convolution layers to encode node and edge attributes, then reconstructs features and edge existence through separate feature and structure decoders. High reconstruction errors flag rare, anomalous behaviors at both the node and edge levels. An automated pipeline applies heuristic fraud tags to these scores and routes them to human analysts and automated mitigation systems.
Multiple Logs for Resiliency
An unexpected network outage can make an application misbehave while also removing crucial log data from the normal Fly.io pipeline. The post traces that pipeline from application STDOUT through Fly.io, NATS, the Log Shipper, Vector, and a third-party provider, then presents two ways to add redundancy. One strategy broadcasts framework logs to a separate file on each machine, preferably on a persistent volume with rotation; a Rails example uses ActiveSupport::Logger.broadcast. The other runs a separate app that subscribes directly to NATS, parses messages, and writes selected fields to disk, with a Fly authentication token and provisions for reconnection and error handling. File copies are more failsafe but raw and harder to fetch, while a custom shipper can produce more usable data but is less bulletproof.
Moving Fast by Moving Slow: How We Built Payments at Ramp
Ramp describes why it delayed the launch of BillPay by about two months to replace its existing, ad hoc bank-transfer infrastructure with a new payments service. The earlier system encoded product-specific concepts such as employee bank accounts and reimbursement transfers, forcing each new use case to add bespoke data models, jobs, and scheduling logic. This friction became especially costly as the team anticipated additional payment products and vendor payments scheduled for future dates. The redesign used generic primitives—ACHDetails and TransferCanonical—to separate business logic from money-movement logic; it also treats failed transfers and retries as concerns requiring an auditable identity, history, and consistent provider mapping. The post argues that stress-testing assumptions, modeling relevant degrees of freedom, limiting conditional logic, and building platforms after recurring patterns can make future products faster to launch.
Tokenized Tokens
Fly.io describes a security problem in its Rails-based platform: as the application accumulated secrets, Rails became the part of the attack surface least trusted, yet still handled many operations involving bearer tokens. It compares model encryption, Vault or KMS, and narrowly scoped secret-handling services, noting that each leaves gaps or creates operational overhead when secrets must be read for arbitrary third-party APIs. The proposed Tokenizer is a stateless HTTP proxy holding a Curve25519 private key; encrypted secrets travel in Proxy-Tokenizer headers, and the proxy injects plaintext or computes HMACs while host allowlists constrain misuse. A companion SSOkenizer performs OAuth2 flows outside Rails, tokenizes resulting bearer tokens, and returns credentials usable only through Tokenizer. The tools are open source, and the design provides a path for third-party integrations and SSO without adding raw secrets to Rails.
Zero traffic cost for Kafka consumers
Grab's real-time data streaming platform team reduced infrastructure expenses by enabling Kafka consumers to fetch from the closest partition replica instead of partition leaders. The platform previously operated across three AWS Availability Zones with three-way replication, resulting in cross-AZ network fees that constituted half of the Kafka platform's total cost. To address this, the team performed a zero-downtime upgrade to Kafka 3.1, configured a rack-aware replica selector using AWS Availability Zone IDs, and updated consumer SDKs to populate the client rack parameter. The rollout produced a 25% drop in cross-AZ traffic within three months under steady data volumes. However, the change increased end-to-end latency by up to 500ms, created potential CPU load skews across zones, and removed graceful isolation during broker rotation maintenance.
Go module proxy at Grab
Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster.
PII masking for privacy-grade machine learning
Data engineers at Grab require real-world streaming data to tune machine learning models, but user personal data cannot be exposed in non-production environments. To solve this, the data streaming team implemented an automated masking pipeline between production and staging Kafka clusters. Developers explicitly tag Personally Identifiable Information (PII) types in Protocol Buffers schemas, which a CI script validates against a keyword list before packaging the schemas into Scala JARs. An in-house Apache Flink application running in production consumes the unmasked streams, applies dynamic and consistent transformations like keyed HMACs or seeded random values, and produces sanitised records to staging. Untagged new fields in schema updates are automatically dropped by the deserialiser until the masking application is redeployed, preventing accidental data leaks.
Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation
Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.
How we improved our iOS CI infrastructure with observability tools
Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.