Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Improving Hugo stability and addressing oncall challenges through automation
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
Scaling Nextdoor’s Datastores: Part 5
Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.
Scaling Nextdoor’s Datastores: Part 4
Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set_if_version and del_if_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.
Scaling Nextdoor’s Datastores: Part 3
Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.
Scaling Nextdoor’s Datastores: Part 2
Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker_uuid and Spark event metrics through app_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.
TechDocs at Grab: Cultivating a culture of quality documentation
Grab’s TechDocs working group describes a two- to three-year effort to make high-quality documentation part of everyday engineering culture. Built on the Helix engineering platform and a Docs-as-Code workflow, TechDocs gives teams a central place to create, manage, and access technical content, while user research informs tool and process changes. The programme combines inventory-taking, policy, team enablement, and recurring measurement, using broad surveys and 1:1 observation to identify fragmented tools, weak ownership, search problems, and maintenance gaps. Research found that only 2% of engineers reported resolving 80–100% of on-call questions through documentation; later initiatives included an editor, writing training, practical workshops, and recognition tied partly to documentation quality and on-call-bot response accuracy. Planned work includes better navigation, training, feedback loops, and editing workflows.
How the Tinder iOS App reduced the size of our localizations by 95% using Emerge
Supporting over 50 languages across numerous statically linked targets led to a substantial build size footprint in Tinder's iOS application. Because Apple's code signing mandates a minimum 4KB per file, shipping dozens of localized files per target inflated the final application package. Tinder addressed this by stripping comments and whitespace, merging localized strings into a single file per language via custom Bazel rules and Aspects, and compressing the strings using Emerge's SmallStrings tool into LZFSE files. The runtime decompresses these files dynamically through existing code-generated string accessors. Consequently, Tinder reduced download size by 10.7MB and install size by 51.3MB with no impact on developer workflows.
Behind the scenes of Canva's DesignDNA campaign
Canva launched DesignDNA in December 2024 as a personalized year-in-review campaign to highlight user achievements and showcase generative AI capabilities. Because internal privacy rules strictly prohibited inspecting personal designs, the engineering and creative teams inferred user preferences from the style and theme metadata tags on public templates. An initial keyword-matching algorithm paired 95% of users with one of seven emerging design trends, while generative AI keyword expansion increased coverage to 99%. The team also synthesized over one million localized poems and distinct design personalities using tools like Magic Write and Dream Lab. Finally, dynamic Canva template elements populated through URL parameters assembled 95 million distinct multi-page stories across nine locales.
Turbo-Charging ML Development
Ramp describes Turbo, a YAML-based configuration system created to reduce the time and engineering effort required to deploy machine-learning models while preserving reproducibility and explainability. Its abstraction represents end-to-end feature and model pipelines, datasets, and shared train and predict jobs; Python entities and aggregates built with Pydantic are materialized through JobSpec and executed locally or remotely before models are stored. A developer can run a training workflow with a YAML file and a few lines of code, then adapt the same schema for prediction by changing a few configuration lines. The first Credit Risk model required nearly four months and more than 3,000 lines of deployment code; with Turbo, new models deploy in under five minutes using about 40 YAML lines on average. The system also standardizes practices such as holiday alignment in time-series models and supports nearly 20 deployed models built by nine developers.
Building resilient webhook handlers in AWS: Implementing DLQs for Stripe events
Reliable Stripe webhook processing must account for lost deliveries from network or service outages, out-of-order events, and duplicates caused by Stripe retries. The proposed AWS architecture uses API Gateway for signature validation and throttling, an SQS FIFO queue for ordered delivery and content-based deduplication, Lambda for processing, DynamoDB for event-ID idempotency, and an SQS DLQ for failed messages. In the CloudFormation example, the main queue has a 300-second visibility timeout and sends messages to the FIFO DLQ after three receives; DynamoDB records expire through a seven-day TTL. Lambda retries failures with exponential backoff, while CloudWatch monitors queue depth, latency, and errors. The design is presented as a scalable foundation, with multi-region failover available at added cost and complexity, though single-region deployment may suffice for many applications.
Adopting Jetpack Compose for Etsy’s Android App
Etsy adopted Jetpack Compose as its preferred way to build Android app UIs, after a staged program that culminated in rewriting the Shop screen. Because UI toolkits are central to development, engineers created a curriculum, rebuilt internal design-system components, and used Compose’s interoperability with XML Views to migrate gradually. They then tested a full internal screen with navigation, system UI, coroutines for API data fetching, and Kotlin Flows supporting unidirectional data flow alongside Macramé. After Compose bottom sheets reached real buyers and sellers, the Shop rewrite took about three months and entered an A/B experiment against the old implementation. The new screen improved initial rendering time by 5%, made interactions more fluid, and improved several user actions, while engineers reported fewer lines of code, easier testing, greater business-logic coverage, and higher satisfaction.
Image replacement in Canva designs using reverse image search
Canva needed an automated way to replace media in design templates, such as when third-party licensing partnerships expire across more than 150 million images. Existing recommendation engines, perceptual hashing, and text metadata searches failed to capture visual similarity hierarchies or ensure replacement relevance. To build a reverse image search system, engineers evaluated embedding models including CLIP, ViTMAE, DreamSim, CaiT, and DINOv2 alongside an external vector database supporting metadata filtering. Evaluation on sample datasets identified DINOv2 as the best model for preserving subjects, background context, and color tones in photos. Integrated into the Template Assistant as a human-in-the-loop tool, the automated suggestions increased image replacement speeds by 4.5 times during initial pilot testing.
Scaling Large Language Models for e-Commerce: The Development of a Llama-Based Customized LLM
eBay developed e-Llama, customized 8B and 70B language models, by adapting Meta’s Llama 3.1 models to e-commerce rather than training entirely from scratch. The approach uses continued pretraining on filtered public listings, product reviews, classifier-selected open-source examples, replay data, and 10% non-English general-domain data to add domain knowledge while limiting catastrophic forgetting. Training ran on 480 NVIDIA H100 GPUs using Megatron-LM, 3D parallelism, flash-attention-2, a 1:1 general-to-e-commerce sampling ratio, a learning-rate maximum set to 10% of the original, and one trillion tokens. The 70B model required about one month and 340,000 GPU-hours. E-commerce benchmarks improved approximately 25% in English and 30% in non-English, while general-domain NLU degraded 1% for e-Llama 70B; subsequent instruction tuning added safety and instruction-following guardrails.
From RAG to Richness: How Ramp Revamped Industry Classification
Ramp's former industry classification system combined third-party data, sales-entered information, and customer self-reporting, creating inconsistent, overly broad, and unauditable categories. Precise classification was needed for compliance, portfolio monitoring, sales targeting, and product analytics, despite fuzzy boundaries, sparse data, and no ground truth. Ramp migrated to six-digit NAICS codes and built an in-house Retrieval-Augmented Generation model that embeds business data, retrieves candidate codes, and uses an LLM to select a final prediction. The system evaluates retrieval with accuracy-at-k and final predictions with a hierarchy-aware fuzzy-accuracy metric, while ClickHouse stores knowledge-base embeddings and Kafka logs intermediate results. Deployed guardrails validate output codes, and the model has improved data quality, consistency, auditability, and control over tuning and costs.
Character Prefix Conditioning
The post poses a problem in code completion: generating a language-model completion that begins with the characters already typed by the user. Because modern models process token sequences rather than characters, naively tokenizing the input can produce the wrong result when the cursor falls inside a token. It introduces character prefix conditioning, defined as sampling tokens from an autoregressive distribution while constraining their decoded character string to start with the given prefix. The challenge is to construct an efficient autoregressive algorithm for this conditional distribution while minimizing calls to the original language model; the post requests either an algorithm description or an implementation.
The foundations of Canva’s continuous data platform with Snowpipe Streaming
As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.
Embracing passwordless authentication with Grab’s Passkey
Grab’s Passkey adds passwordless authentication to its app to address phishing, keystroke-logging, brute-force, and other security threats associated with passwords plus the friction of OTP-based login. Built on the FIDO standard and implemented through WebAuthn, the flow uses a backend, frontend, and authenticator: the authenticator creates and stores a passkey, while Grab stores the public key and metadata. Registration uses a server-generated challenge and parameters including rp.id, user information, resident-key requirements, and supported public-key algorithms; sign-in returns a credential whose signature is verified against the stored public key. Google Password Manager and iCloud Keychain can sync passkeys across supported devices, and users authenticate with their screen lock rather than entering a password or waiting for an OTP. The implementation is described as faster, smoother, phishing-resistant login with potential savings on third-party authentication and messaging costs, while Grab plans to evaluate web login expansion through adoption, feedback, and usage data.
Turbocharging GrabUnlimited with Temporal
Grab’s GrabUnlimited flagship membership program struggled as subscribers grew by over 1000% from January 2022 to June 2023. Its original design used a daily cron job for renewals, Amazon’s SQS to distribute state handlers, and five-minute Redis locks, but outages, corrupted states, concurrent cancellation and renewal, missed benefits, and non-idempotent retries created operational problems. The team migrated the service to a Temporal workflow-based system, enhancing its ability to handle millions of subscribers with greater efficiency and resilience. The source reports an 80% reduction in open production incidents and says the change improved the membership experience for users. This first part focuses on designing a robust and scalable membership system, while migration challenges and lessons are reserved for a subsequent installment.
The science of routing print orders
Canva's global print network requires selecting optimal suppliers to balance delivery times, packaging counts, and environmental emissions. To resolve these challenges before user checkout, the engineering team designed a modular routing architecture that decouples graph construction, decision logic, and path traversal. During graph traversal, the system generates action objects capturing forward paths and decision query results, which are compiled into timestamped routing logs in blob storage for asynchronous auditing. Utilizing preprocessed graph queries alongside ElastiCache, Redis, and database read replicas, the infrastructure sustains high-throughput evaluation without coupling cost logic to traversal code. As a result, print routing completes within an average of 50 milliseconds at the 99th percentile during peak usage while maintaining 99.999% data availability.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.