Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Fly Behind The Scenes: Fresh Logging
Fly replaced its centralized Graylog logging platform after seven servers could not process peak volumes of 20,000–30,000 logs per second, causing drops. Vector, written in Rust, now runs on every server, parses and transforms logs from journald and other applications, and sends them directly to Elasticsearch, removing Graylog as a middleman. The pipeline adopts the Elasticsearch Common Schema (ECS), with Fly-specific fields for app names, allocation IDs, and regions, while Vector configurations are distributed through the existing configuration management system. The migration took about two weeks alongside other work; logs are now available quickly enough to support failed-allocation queries that previously often raced processing, and future sinks may include Papertrail, Honeycomb, or Kafka.
Optimally Scaling Kafka Consumer Applications
Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.
Images as Code: Representing Localized and Evolving Products on Marketing Pages
Shopify needed product visuals on more than 35 localized domains to stay accurate as features, currencies, languages, and product UI changed. An SVG experiment made translated text difficult to manage because SVG lacks word wrapping, while export settings, effects such as Gaussian blur, and repeated translation hooks created maintenance and performance problems. The replacement represented interfaces in HTML and JavaScript through ScaleContentAsImage, which wraps fixed-width content, computes available space, applies a CSS transform scale, preserves layout with a spacer, absolutely positions the transformed element, and updates calculations on resize. The team concluded HTML/JavaScript was more maintainable and flexible for design, documentation, animations, and localization, while noting remaining work around lazy-loaded images, reusable-visual documentation, and future CSS Houdini possibilities.
Our Journey to Continuous Delivery at Grab (Part 1)
Around the end of 2018, Grab's backend architecture consisted of roughly 270 services managed through fragmented, manual deployment workflows. Engineers copied release parameters between build logs, wiki pages, Slack bots, and multiple Jenkins jobs, leading to high operational friction and an average of 10 business days between production updates for a service. To streamline delivery, Grab built Conveyor, an internal automation platform built on top of open-source Spinnaker. Conveyor introduced a custom user interface focused on pipeline visibility and a pipeline-as-code DSL called Artificer using Jsonnet files in the monorepository. The platform automatically registers build artifacts with commit metadata to eliminate manual parameter entry and automatically provisions integration, staging, and production pipelines.
Enforcing Modularity in Rails Apps with Packwerk
Large Rails monoliths can develop high coupling, low cohesion, cross-component calls, shared Active Record models, and classes that know too much because Ruby and Rails provide limited boundary enforcement. Shopify created Packwerk, an open-source static analysis tool that groups Ruby files into packages and enforces dependency and privacy boundaries, including controlled public APIs for constants. It reports the violation type and location with actionable next steps, integrates with CI, and can run locally; installation begins by adding the gem and running packwerk init. Because Ruby static analysis is complex, Packwerk ignores constants that are not autoloaded, reducing false positives at the cost of false negatives. The excerpt says it runs in six Shopify Rails applications, with 48 packages and 30 boundary enforcements in the core codebase, while adoption also prompted work on dependency inversion.
Securing and Managing Multi-cloud Presto Clusters with Grab’s DataGateway
Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment.
How Shopify Reduced Storefront Response Times with a Rewrite
Shopify rewrote the server-side Storefront Renderer, which loads Liquid themes and storefront data before returning HTML, because the legacy Rails-monolith implementation had developed stricter performance demands and rising time-to-first-byte as traffic grew. The new single-purpose application separates storefront traffic from checkout, admin, and API traffic, uses active-active replication with dedicated read replicas, and adds mechanisms for high-load resilience. During migration, a Ruby verifier compares status codes, headers, and bodies from both implementations, while a custom Lua module on OpenResty samples production traffic and routes requests based on verification results. The rollout had reached more than 90% feature parity, and the new implementation averaged 4x faster server response times, with ongoing work aimed at full parity and retiring the legacy system.
Go Modules- A Guide for monorepos (Part 2)
Managing dependencies in a multi-module monorepo created developer friction at Grab due to unexpected changes from previous vendoring attempts and accidental imports. Because Go modules were not yet enabled directly for builds, the team implemented a continuous integration check that executes go mod vendor and rejects merge requests if any diffs exist in go.mod or the vendor directory. Adopting this CI check required configuring SSH deploy keys for private repositories, adding retry logic for network-related false positives, and standardizing on a single Go version to prevent checksum discrepancies. To streamline ongoing maintenance across hundreds of dependencies, the team developed an automated tool named AutoVend Bot. The bot runs go list -m -u all to detect updates and opens a scheduled batch of merge requests each day for human review.
Using DNS Traffic Management to Add Resiliency to Shopify’s Services
Shopify consolidated several team-specific DNS traffic-management approaches that relied on manual changes, pull requests, or provider APIs and had uneven failover, redundancy, and active/active capabilities. The replacement was a fully automated, self-served system for more than 40 domains owned by more than 12 teams and handling more than 100M requests per day. It stores domain configuration in Git, deploys it to two DNS providers, and exposes spy commands for fast updates and failovers. Its traffic-management design uses regionalized matching, endpoint-status filtering, weighted sharing, and fallback layers, while globally matching endpoints ensure an existing domain receives an answer. A later provider migration demonstrated the value of centralized ownership: the team updated Terraform and tooling, moved nameservers incrementally, and kept the change transparent to application owners despite concurrent incidents.
How We’re Solving Data Discovery Challenges at Shopify
Shopify describes data discovery as a response to growing numbers of data assets and the difficulty of finding, governing, and using them across the data lifecycle. At Shopify, pre-Artifact discovery relied on people, Slack, GitHub, and job logs, while teams lacked granular ownership, change-impact visibility, and documented reporting context. They built Artifact, a search-and-browse tool backed by a centralized metadata model, ingestion jobs, Elasticsearch indexes, GraphQL APIs, and a graph database for filterable upstream and downstream lineage. Its landing page supports browsing and plain-English search, while details pages expose documentation, schema, descriptions, usage, ownership, and lineage; initial scope prioritized frequently used, high-impact, queryable assets and downstream reports and dashboards. Since launch in early 2020, weekly Data team usage exceeded 30% and monthly retention exceeded 50%, above expectations, with future work planned for additional assets, integrations, stewardship, change management, notifications, and metadata APIs.
Making Time to Save You Time: How We Sped Up Time-Related Syscalls on Dynos
A customer support ticket exposed poor performance in clock_gettime(3) and gettimeofday(2) for an application making many timing calls through transaction tracing. Heroku's Runtime Infrastructure team found that Linux was selecting the Xen clocksource, whose Xen version did not support virtual dynamic shared object (vDSO) acceleration, forcing costly userspace-to-kernel context switches. The team evaluated available clocksources and selected tsc, which uses the Time Stamp Counter and supports vDSO; they assessed concerns about backwards clock drift, including older CPU inconsistencies and Xen VM freezing, against their newer Intel CPUs and EC2 environment. After vdsotest validation, the configuration was deployed across the Common Runtime and Private Spaces dyno fleet, and the initiating customer plus other customers observed performance improvements.
The Journey of Deploying Apache Airflow at Grab
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
How We Built Size.link
Size.link is a free Shopify AR tool that lets shoppers view a product’s dimensions in their surroundings through a smartphone camera. It was created during April Hack Days as a quick alternative to commissioning 3D models, with requirements including browser-based operation, speed, accuracy, and iOS and Android support. Because iOS lacks JavaScript WebAR, the implementation uses AR Quick Look with USDZ files and Android’s Scene Viewer with GLB files. Rather than generate files dynamically with slow USDZ tooling or pre-generate billions of dimension combinations, the team modifies binary template data, replacing scale or vertex float values to resize the cube while preserving its outline. The resulting Ruby server generates USDZ files in well under one millisecond, and the same strategy produces GLB files; future work could add textures for standard-sized products.
Media at Scale: Callbacks vs pipelines
Shopify’s Rails monolith needed to add native product videos and 3D models while continuing to support an image infrastructure containing more than 7 billion images. The design question was whether media creation should rely on Active Record callbacks or an explicit pipeline, with transactions protecting interdependent database writes. Callbacks were quick for simple cases, but adding media-specific behavior such as video thumbnails spread conditionals across models and made lifecycle ordering difficult to follow and debug. The pipeline design routes requests through a single Product Create Media Service and media handlers, each organized into before_transaction, during_transaction, and after_transaction steps, while confining logic to one media type. This structure separates concerns, controls creation order, limits model access, and makes implementation details easier to understand and maintain as the feature grows.
How We Built Our In-house Chat Platform for the Web
Grab extended its in-house chat platform to the web to support its internal Customer Support portal. Because the existing TCP gateway only supports unicast connections with one active connection per user, opening multiple browser tabs would repeatedly disconnect previous tabs. Rather than undertaking a complex migration to multicast connections on the server, the team adopted a hybrid client-side strategy using SharedWorker and BroadcastChannel APIs. The implementation uses a SharedWorker to maintain a single WebSocket connection per domain while a BroadcastChannel syncs events across all open tabs. A custom wrapper over the worker manages version transitions during deployments to avoid race conditions across tabs.
Spark Joy by Running Fewer Tests
Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests.
Go Modules- A Guide for monorepos (Part 1)
Grab transitioned its large Go monorepo dependency management from Glide to Go modules while retaining an existing vendor directory structure. The team generated root go.mod configurations from glide.yaml and used go mod vendor without directly enabling module-mode builds. Incompatible nested sub-vendor paths were excluded by placing empty go.mod files, relying on the rule that modules cannot contain other modules. Post-migration maintenance revealed challenges with dependency inheritance and implicit go.mod updates during builds, which engineers investigated using go mod graph and digraph to trace dependency paths.
Building and Scaling a Global Chatbot using Heroku + Terraform
Live Agent, Salesforce’s embeddable chatbot, must handle customer interactions and requests at global scale while remaining highly available across regions. Its eight-person team chose Heroku to avoid managing bare metal, cloud virtual machines, databases, and other third-party services, while relying on managed Java support and regional infrastructure. The platform runs ten apps with Heroku Postgres, Redis, and Apache Kafka dependencies, and deploys them inside Heroku Private Spaces to isolate network boundaries and data. Terraform defines and replicates apps, add-ons, custom domains, logging, and profiling configuration across regions, allowing changes to propagate through a pull request. The result described is a simpler operational model in which Heroku handles maintenance and the team can scale the same architecture into new geographic locations.
Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing
Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in `-only-testing` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.
Refactoring Legacy Code with the Strangler Fig Pattern
Shopify’s Ruby on Rails Shop model had grown beyond 3,000 lines, accumulated numerous responsibilities, and developed weak semantic boundaries, making store settings a target for extraction. The team used Flog’s ABC-based score—assignments, branches, and calls—to identify a particularly disordered area, then applied Martin Fowler’s Strangler Fig Pattern to move the locked_settings behavior into Capital::SettingsToLock and a dedicated record/data source. The seven-step process defined and tested a new interface, redirected callers, introduced and backfilled the new data source, switched reads, and finally stopped legacy writes and removed the old column and code. Keeping the old system available during the transition supported incremental changes, monitoring, reversibility, and the stated no-downtime requirement; the completed example left the new system in place with legacy remnants removed.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.