Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Graph for fraud detection
Rapid expansion across multiple business verticals introduced diverse, evolving fraud patterns at Grab, making traditional rule engines and decision trees inadequate due to their heavy reliance on extensive feature engineering and labeled data. To address cold starts and detect unknown fraud, the engineering team implemented a semi-supervised Relational Graph Convolutional Network (RGCN) across millions of connected ecosystem entities. The model propagates information along neighborhood nodes through graph convolutional layers to produce node embeddings and output fraud probabilities. Trained on graphs where only a small percentage of nodes are labeled, the RGCN achieved an AUROC close to 1 and distinctly separated fraudulent embeddings from genuine ones. The approach also offers high explainability by surfacing dense clusters of shared physical devices and addresses ongoing challenges in real-time prediction and noisy connections.
Real-Time Collaboration with Replicache and Fly-Replay
Distributed real-time collaboration becomes difficult when a single server coordinating document changes is overloaded and documents must be split across machines without exposing routing complexity to clients. This demo combines Replicache, a custom WebSocket endpoint, Fly Machines, and the Fly-Replay header to route each document’s connection through a shared router to the machine responsible for that document. The server keeps connections by spaceID, sends initial and incremental state, and tracks messages already delivered; the client overrides Replicache’s push and pull handlers to exchange data through the persistent socket. The resulting application has React frontend, Express router, and Replicache/WebSocket backend layers, with each todo list assigned to one backend machine. The author cautions that the approach is demonstrated only in a basic demo and should receive serious production testing and tuning.
Query expansion based on user behaviour
Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.
Caching Without Marshal Part 2: The Path to MessagePack
Rails caching relied heavily on Marshal, but cached Ruby-specific objects could break after code changes when old payloads were loaded by code without the former classes. MessagePack provided a generic binary format whose core types exclude Ruby-specific Object and instance-variable encodings, while extension types allowed production-specific serializers such as the compact Date representation. During a roughly six-month migration, the team ran both formats, prefixed MessagePack payloads with a version byte, and used encoding failures to identify unsupported cache values. To scale beyond 128 extension types, they added an Object catchall based on as_pack and from_pack, plus Struct and T::Struct modules whose attribute digests detect refactors and turn stale data into cache misses. After logs showed that Marshal was no longer used, the core monolith moved exclusively to MessagePack, with the Paquito gem extracting much of the migration work.
How to Categorize and Prevent Risks of Sensitive Links in URLScan
Security integrations with URLScan can accidentally index sensitive links when API calls are executed with public visibility. Tinder Security Labs constructed targeted queries to uncover exposed links across file-sharing services, enterprise ticketing platforms, and single sign-on portals. Investigations revealed that exposed links included document magic links containing contracts and bank details, as well as active new-hire registration links that allowed attackers to complete registration workflows and configure two-factor authentication. In one SSO query alone, researchers discovered 1,300 indexed invitation links, with 20 to 50 new links appearing daily. The researchers collaborated with URLScan to purge the exposed indexes from public searches and advised organizations to configure security scanning tools to use private or unlisted visibility settings.
Reducing BigQuery Costs: How We Fixed A $1 Million Query
During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.
Using mobile sensor data to encourage safer driving
Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.
How to Structure Your Data Team for Maximum Influence
Embedded data science leaders may support multiple product and business areas, creating competing work streams, specializations, and stakeholder relationships. The post compares swim lanes and stochastic process assignments against principles of efficiency, influence, stakeholder clarity, stability, growth, and flexibility. It recommends a hybrid “diamond defense” in which scientists are loosely assigned to zones for specialization while a bullpen, including the manager and an additional scientist, supplies reserve capacity. Resources can be redirected to overloaded areas, and team members can rotate into vacancies or cover leave. The framework is presented as combining clear ownership and subject-matter depth with the flexibility to handle change, though the post warns that excessive rotation can create volatility and turnover.
Leveraging Go Worker Pools to Scale Server-side Data Sharing
Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.
Automatic rule backtesting with large quantities of data
Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.
pg_jsonschema: JSON Schema support for Postgres
pg_jsonschema is a Postgres extension released on the Supabase platform that adds JSON Schema validation for json and jsonb data types. The post addresses a trade-off in storing complex webhook payloads as documents: JSON simplifies the model and accommodates changing payloads, but an unconstrained column cannot enforce required structure or types. It introduces JSON Schema as a formal, human-readable way to describe those constraints, including an object with a required string foo attribute and no additional properties. The extension provides json_matches_schema and jsonb_matches_schema, which can be used in a Postgres CHECK constraint to validate inserted documents. In the example, the constraint handled all eight test cases correctly, and invalid rows produced a constraint-violation error.
How we store and process millions of orders daily
The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.
Rate limiting with Redis
Rate limiting at Ramp must handle third-party API quotas, globally paced Celery tasks, and inbound API traffic limited by authenticated application identities rather than only mutable IP addresses. Celery’s per-worker controls were insufficient for global limits, while queue prefetching could let rate-limited tasks block workers and stall queues. The team created a general framework for comparing algorithms and evaluated fixed-window, sliding-window, leaky/token-bucket, and Generic Cell Rate Algorithm (GCRA) approaches. They selected GCRA for implementation simplicity and compute and memory efficiency, storing a theoretical arrival time in Redis and using a Redis lock to prevent races. The implementation also uses floating-point timestamps, Redis server time, key expiration, unit tests, and manual testing; it has been introduced gradually across the three use cases, with monitoring, while additional algorithms remain planned.
Spin Cycle: Shopify’s SFN Team Overcomes a Cloud-Development Spiral
Shopify’s Fulfillment Network (SFN) worked to run its monolithic Rails application in Spin, a cloud-based development tool that provides on-demand Kubernetes containers instead of local service environments. Because SFN glues together Shopify and many external services, the team had to decide whether to run, mock, or share each dependency, while frequent changes to SFN, Spin, and those dependencies created regressions. The three-engineer team identified a human feedback dynamic: painful Spin experiences discouraged adoption, which allowed more Spin-specific breakage, while regular use could expose regressions before merge. To break that spiral, the team prioritized basic functionality for common validation flows, fixed disruptions there, and asked developers to use Spin daily and add missing capabilities; the intended outcome was a self-reinforcing “Loop of Happy Equilibrium,” though the transition is described as a hypothesis.
Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model
Shopify Inbox needed to classify millions of monthly customer messages so merchants could prioritize responses and guide product decisions. The team adopted a data-centric approach, improving the dataset rather than primarily changing model architectures. An initial prototype used Universal Sentence Encoder embeddings, UMAP and HDBScan clustering, semi-supervised labeling, and a neural classifier, but reached about 70% accuracy with 35% high-confidence coverage. For version 2.0, the team created a more explicit taxonomy, used human-labeled data and fine-tuned DistilBERT, while addressing privacy and class imbalance through data handling and targeted annotation. In production, version 2.0 achieved about 90% accuracy and 80% high-confidence coverage, although the taxonomy and training data require ongoing updates as conversation trends change.
How we automated FAQ responses at Grab
Internal engineering on-call engineers at Grab spent substantial working hours handling repetitive questions in Slack channels, such as how-to inquiries and access permission requests. To resolve this without building an in-house tool, the team conducted an anonymized vendor comparison and selected OneBar through an employee voting process and a phased proof-of-concept. Initial rollouts were restricted by contract to 20 channels, leading the team to prioritize deployment based on Slack message volume and member counts. Populating the knowledge base required roughly a quarter of consistent updates alongside tech talks, while a targeted crowdsourcing campaign among new onboarders expanded the glossary and grew usage to approximately 3,000 users.
Typeahead Search at Nextdoor
Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.
Graph Networks - 10X investigation with Graph Visualisations
Fraud detection traditionally required investigators to manually combine large datasets from disparate anti-fraud systems using statistical methods, which proved slow and inefficient. Grab built an interactive Graph Visualisation platform to transform raw records into connected visual maps without requiring manual queries or switching tools. The platform manages over three billion nodes and edges, allowing investigators to selectively expand data points and replay chronological events using temporal filters. Visual relationship mapping helps teams verify account appeals, uncover device-sharing rings, and spot anti-money laundering behavior through transaction density patterns.
Making Open Source Safer for Everyone with Shopify’s Bug Bounty Program
Shopify describes how a bug bounty report led to application fixes and Rails contributions aimed at making open-source software safer. The 2021 Shopify Experiments report showed that a leaked development-environment secret could be used to generate trusted-looking data and reach remote code execution in a local development environment. The root cause was ActiveSupport’s MessageEncryptor using Ruby’s unsafe Marshal serializer by default in Rails versions up to 7.0.X, allowing untrusted data to trigger deserialization risks. Shopify first configured JSON in its application, then helped change the default serializers for MessageEncryptor and MessageVerifier to JSON in Rails 7.1.X, with an upgrade path for existing applications. It also developed a patch that monitors Marshal load and dump calls, raises on unexpected gem-originated calls, and runs in local development and continuous integration to prevent new usage.
The Journey to Cloud Development: How Shopify Went All-in on Spin
Shopify’s development environment, the Ruby-based dev tool and xhyve-backed railgun VM, no longer handled the growing number of repositories, services, integrations, and resource demands around its monoliths. Developers needed more services, faced sluggish code-build-test loops, and encountered costly state changes when tophatting collaborators’ work. The Environments team concluded that development environments needed cloud-like elasticity, then tested an automatically configured local Kubernetes cluster and easy creation of Google Compute Engine VMs; the source also describes Isospin and Spin as the resulting direction. Local Kubernetes did not gain broad organic traction, while the current Isospin implementation is described as the team’s best iteration for developers working on majestic monoliths. The team plans to evolve it into an application-development library built on dev, add dependency inference, and address the tension between persistent environments and Kubernetes pod relocation.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.