Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
2.3x faster using the Go plugin to replace Lua virtual machine
Talaria, an open-source distributed time-series database developed at Grab, previously allowed users to run custom data transformation scripts during ingestion using a Lua virtual machine. Launching and executing Lua scripts caused significant performance overhead when processing large volumes of events. To resolve this bottleneck, the team replaced the Lua VM with Go plugins compiled as Linux shared libraries (.so files). Benchmarks revealed that calling Go plugins achieves performance on par with native Go functions, executing roughly 2.3 times faster and consuming 2.3 times less memory than cached Lua VMs. Both execution methods conform to a unified Handler interface to load and run custom transformations.
Safer deployment of streaming applications
Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.
Message Center - Redesigning the messaging experience on the Grab superapp
Grab redesigned its messaging infrastructure from GrabChat to Message Center to overcome two-party chat limitations and support complex superapp requirements like group conversations and varied user roles. The architecture separates core processing logic from message delivery by splitting the system into a backend processor and an independently scalable postman service. Communication relies on an in-house TCP gateway named Hermes that proxies client payloads via gRPC, alongside Apache Kafka streams and Amazon SQS delay queues. Custom client-server acknowledgements and a DynamoDB event store ensure reliable message delivery even across dropped mobile TCP connections and offline reconnects.
Adventures in Garbage Collection: Improving GC Performance in our Massive Monolith
Shopify investigated high Ruby garbage-collection latency in its monolith, where minor marks took about 3 ms but major marks could exceed four seconds and some requests spent multiple seconds in GC. Logging showed that roughly 75% of major marks were triggered by oldmalloc and nearly 25% by shady, while nofree became dominant after oldmalloc was addressed. The team used production experiments, improved metrics, changed GC environment settings, increased initial heap slots, and added out-of-band collection during request processing. A 128 GB oldmalloc threshold reduced major marking by about 20%; later out-of-band collection cut the P99.99 and P99.9 GC-time tail by almost tenfold and largely eliminated median-request pauses. The resulting configuration, with collection frequency adapting from every 128 to every 512 requests, was deployed across the fleet after about three weeks.
Evolution of quality at Grab
As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%.
How OVO determined the right technology stack for their web-based projects
OVO faced maintainability issues caused by fragmented web technology stacks, including PHP, Vue, React, Nuxt, and Go, alongside outdated documentation. This fragmentation resulted in severe context-switching costs during code reviews and prolonged onboarding times for new engineers. To establish a single primary front-end stack, the team compared React and Vue against maintainability principles encompassing operability, simplicity, and evolvability. Vue was selected because its standardized framework structure and documentation minimized architectural variations across projects. Implementation entailed creating a standardized Vue boilerplate, updating the internal UI library, and scheduling periodic upgrades for legacy React services rather than migrating them entirely.
Migrating from Role to Attribute-based Access Control
Grab's streaming data platform team migrated the Kafka Control Plane from Role-Based Access Control to Attribute-Based Access Control to eliminate operational bottlenecks and manual permission management. The previous model required defining hundreds of roles, permissions, and group mappings in an internal IAM service, leading to approval delays and stale memberships. Under the new architecture, user attributes sync from the HRMS and token payloads, while resource attributes are tagged upon creation or backfilled to reflect department and team ownership. Open Policy Agent evaluates access requests defined in Rego via middleware by comparing user attributes with resource metadata. This transition eliminated over 200 roles, 200 permissions, and roughly 3,000 unused IAM resources while automating access provisioning for new joiners.
Securing GitOps pipelines
Grab's real-time data platform team transitioned from an Atlantis-driven Terraform workflow to an in-house GitOps platform called Khone to manage streaming infrastructure resources like Kafka topics and Flink pipelines. The earlier setup suffered from coarse-grained access controls, required manual merge request comments, and lacked flexible validation capabilities within native configuration files. Khone derives environment parameters directly from standardized directory paths and uses Python with the python-hcl2 library to inspect and validate resource definitions before executing Terraform stages in parallel. To prevent configuration tampering in merge requests, CI/CD pipeline definitions and execution scripts are isolated in a separate administrative repository and fetched during job runs using shallow Git clones.
New zoom freezing feature for Geohash plugin
Grab introduced a zoom freeze capability to its Geohash plugin for the Java OpenStreetMap Editor (JOSM). Previously, zooming in or out forced the plugin to calculate and render new geohashes automatically, causing visual clutter when users needed to focus on a specific region. The updated interface introduces a toggle label allowing editors to switch between enabling and disabling zoom freezing. While frozen, the current geohash boundaries remain locked during map scaling, and options to show smaller or larger geohashes are disabled. Disabling the feature instantly updates the display with the appropriate geohash subdivisions for the current zoom level.
HappyTeams unlocks better performance and reduces cost with Supabase
HappyTeams, a bootstrapped startup, needed its database to support thousands of users and continued feature development without consuming scarce technical resources. Its Heroku Postgres deployment had developed scaling problems, including sluggish or uneven performance and limits on concurrent connections, prompting a database change during an application rebuild in early 2022. The company migrated to Supabase in less than an hour using pg_dump and pg_restore, and adopted PgBouncer for connection pooling; it selected the service for performance, connection flexibility, support and documentation, onboarding, and lower costs. The source reports that Supabase removed the performance issue, made pooling effortless, and helped HappyTeams release features faster, including a comments feature completed in two days, while the company considered expanding into Edge Functions and the API client.
Catching Anomalies Early in Mobile App Releases
Nextdoor deploys weekly mobile updates across iOS and Android using phased rollouts starting at 1% adoption to minimize blast radius. Standard aggregate observability cannot reliably detect early regressions because early adopters skew significantly more active than average users, obscuring silent drops in overall metric noise. To overcome this selection bias, Nextdoor built App Release Anomaly Detection using difference-in-differences causal inference. The methodology verifies pre-adoption trends with standard deviation bounds and fits a linear regression model to estimate the effect against an unobserved counterfactual. During a rollout of iOS version v1.234.5, this model identified a statistically significant app session decline at 1% adoption, isolating the regression 10 days earlier than week-over-week metrics.
From Ruby to Node: Overhauling Shopify’s CLI for a Better Developer Experience
Shopify rewrote its unified CLI from Ruby to TypeScript running on Node after growing dependencies on Node tooling, Hydrogen’s preference for npm-managed dependencies, and a broader fit with web technologies. The rewrite aims to reduce system requirements, improve terminal consistency, and make contribution easier, while using a modular architecture of domain NPM packages built on @shopify/cli-kit. The team established shared code, UI, convention, and principle foundations, addressed cross-OS support, used TypeScript for stronger contracts, and applied Cucumber for end-to-end testing. The Node CLI unified and streamlined app development and added extensibility such as functions, but Theme commands still use Ruby and removing that dependency remains future work.
Building Obsidian, Tinder’s Design System
As Tinder scaled over a decade, maintaining visual consistency across its client applications became unsustainable and increased engineering overhead. To address this, the UI Platform team developed Obsidian, a unified design system combining design tokens, reusable components, and documentation. Designers define base and context tokens in Figma using the Tokens Studio plugin, which are then transformed by Style Dictionary into native artifacts for iOS, Android, and Web. Web developers integrate these tokens using Atomizer to generate static stylesheets, while a Zeroheight documentation portal and a custom token lookup tool assist legacy migrations. The ongoing migration has streamlined collaboration between designers and engineers, resolved styling ambiguities, and improved color contrast accessibility across platforms.
Graph service platform
Grab's GrabDefence team required a dedicated graph infrastructure to proactively identify mobile fraud patterns, such as multiple accounts operating on shared physical devices and suspicious financial loops. To address this, the team built a four-layer Platform as a Service that encapsulates graph database operations behind uniform RESTful APIs for OLTP search and OLAP analysis. The architecture utilizes Amazon S3 for raw data files, Amazon Neptune for graph storage, DynamoDB for schema and metadata configurations, and Kafka for streaming ingestion. When users trigger data loading tasks, the service validates entity attributes against schemas stored in DynamoDB before importing records into Neptune. This infrastructure allows investigators to traverse adjacent account IDs and visualize complex entity relationships without managing underlying database runtimes.
Migrating our Largest Mobile App to React Native
Shopify’s largest mobile app, Shopify Mobile, had 300 screens per platform and more than six years of development, making a full rewrite impractical because it could take at least three years and halt feature work. The team first adopted React Native incrementally, but migration was projected to take four to five years while requiring three design-system implementations, three architectures, and costly native interoperability. It then switched to “Iterative Porting”: engineers were trained through the five-milestone RN Accelerator, surfaces were inventoried and prioritized with a RICE-based score, and eligible features were migrated behind flags, tested, monitored for two weeks, then completed by removing the native version. The effort also introduced Native Routes, modal deeplink handling on iOS, and component-level Safe Area fixes; with root screens ported and infrastructure in place, the team reports that ports are accelerating, although substantial work remains.
Automatically Rotating GitHub Tokens (So You Don’t Have To)
Shopify needed automated GitHub workflows that could clone from private repositories, update organization-level projects, and trigger downstream workflows but could not use GITHUB_TOKEN. Personal access tokens supplied those capabilities, yet introduced developer churn, user-attributed actions, and no organization-enforced rotation or expiry. The proposed system uses GitHub Actions and a GitHub App to generate repository secrets that rotate automatically while preserving PAT-like references in existing workflows. Its serverless design delegates scheduling, managed infrastructure, approvals, secret management, auditing, and break-glass operation to GitHub, while a custom TypeScript action encrypts and writes generated secrets. Prototyping exposed that parallel one-second jobs could accrue multiple billable minutes, prompting sequential execution in a consolidated workflow; the stated goal is improved resilience to developer churn and reduced leaked-secret blast radius.
Zero trust with Kafka
Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.
How KartaCam powers GrabMaps
Grab's Geo team required an efficient, low-cost way to collect fresh street-level imagery across Southeast Asia, where professional mapping equipment is prohibitively expensive and smartphone crowdsourcing yields inconsistent quality. To address this, Grab built KartaCam, a custom mapping device equipped with a 12MP sensor, dual-band GNSS, 4G LTE, and onboard edge AI. Edge machine learning models evaluate scene suitability, check image quality, filter for map-relevant objects, and blur personal data directly on the device prior to upload. Deploying a four-camera KartaCam 360 array delivers panoramic coverage and point-of-interest data comparable to commercial mapping rigs at roughly one-twentieth the hardware cost.
Using Server Sent Events to Simplify Real-time Streaming at Scale
Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.
Cost Efficient Snowflake CI
Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.