Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,538 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Cache Invalidation Complexity: Rails 5.2 and Dalli Cache Store
Rails 5.2+ applications can fail to invalidate ActiveRecord-backed cache entries when cache versioning is enabled alongside a third-party store such as dalli_store 2.7.8 or earlier. Rails’ recyclable cache keys keep the cache_key stable and store an ActiveRecord cache_version in the entry, preventing unrelated entries from being evicted, but Dalli only understands the key and therefore repeatedly returns stale data. The post explains how to check the production cache_store and active_record.cache_versioning settings, then compares switching to Rails’ :mem_cache_store, disabling versioning, and upgrading Dalli to 2.7.9+. Its recommendation is to use :mem_cache_store and flush the cache, while noting that Dalli 2.7.9 fixes invalidation by putting the version into the key rather than implementing recyclable keys.
Lost in Translations: Bringing the World to Shopify
Shopify describes the engineering challenge of internationalizing a platform whose twelve years of development left thousands of hardcoded English strings across millions of lines of code. The team first used cops, linters, and tests to stop new violations, then automated extraction of existing content into module-specific dictionaries using keys derived from file paths and strings. A Rails I18n-based pseudolocalization backend replaced characters and doubled vowels, making untranslated content visible while exposing UI problems caused by longer words; UTF-8 migrations and application fixes addressed character-set and internationalization bugs. Finally, an internal Translation Platform connected GitHub webhooks and a translation management system, automatically submitting changed language files and returning approved translations as commits or pull requests. After six months, Shopify was available in a variety of languages, while the team continued addressing localization issues involving formatting, geography, payments, shipping, iconography, and regional requirements.
Mohammed Ridwanul Islam: How Mentorship, the T Model and a Pen Are the Keys to His Success
Mohammed Ridwanul Islam's career story connects his upbringing in Noakhali, Bangladesh, varied work and music experience, and interest in engineering with his current role on Shopify's Eventscale team. He describes how trying to build a game led him to write C# and later study Electrical Engineering at the University of Waterloo, where an internship became a full-time position. Eventscale develops tools, libraries, and infrastructure for collecting event-oriented streaming data across web, backend, and mobile platforms, supporting analytics and operational needs. Mentorship during a difficult team project taught him project management concepts, while habits, VIM, Java, and writing designs on paper shape his working practices. His advice is to build broad competence alongside deep expertise, and to prioritize problem-solving, values, social impact, and continued growth.
Dev Degree
Dev Degree is Shopify’s work-integrated learning program, combining an accredited university degree with double the experience of a traditional co-op. It emerged from concerns that graduates often knew theory but lacked practical software-development skills, and it required a long-term partnership with a university willing to change curriculum and assessments. At Carleton, students take three courses each semester while working 25 hours weekly at Shopify; students also complete a first-year Developer Skills Training program, submit practicum reports every four months, and receive technical and Life@Shopify mentors. The program allocates at least 20% of degree credits to practicums and uses learning outcomes influenced by ACM and IEEE Computer Society standards. Shopify reports 95% retention, women representing over 50% of students, and students becoming productive after nine months; it has expanded to York University, while the company says broader adoption is needed.
Introducing the Deprecation Toolkit
Shopify’s Deprecation Toolkit is an open-source Ruby gem for tracking deprecated code in an application and preventing new deprecations from entering a codebase. It addresses the limits of warning messages and the impracticality of immediately raising on every existing deprecation, especially in a large monolith with complex dependency upgrades. The toolkit records deprecations triggered by test suites in YAML whitelist files, then compares later test runs with the accepted set; mismatches trigger configurable behavior, raising an error by default. Existing deprecations can therefore remain acceptable while teams fix them gradually, whereas introduced or removed deprecations are surfaced in CI. Shopify reports that using the toolkit reduced deprecations, supported dependency upgrades, blocked pull requests introducing new ones, and enabled team-based tracking of remediation progress.
Mockers - Overcoming Testing Challenges at Grab
Grab operates over 250 microservices communicating over HTTP and gRPC, making shared staging environments costly, ambiguous in ownership, and fragile due to inconsistent data and uncoordinated deployments. To address these testing bottlenecks, Grab created Mockers, a Go SDK and CLI tool backed by a central monorepo of mock servers for local-box and CI testing. Mockers automatically generates HTTP and gRPC mock servers from Swagger specifications and protobuf files, returning configured network responses without internal business logic. By incorporating Grab's in-house chaos SDK middleware, Mockers also enables repeatable resiliency and contract testing locally without relying on code-level mocks. While Grab still mandates integration testing on distributed staging environments with live data, Mockers enables developers to detect complex defects and contract mismatches earlier.
Journey of a Tourist via Grab
Grab analyzed platform ride data from millions of tourist passengers representing over 150 countries visiting Singapore. Over 60% of these tourist riders originated from Southeast Asia, while non-regional visitors mainly came from China, the United States, and India. Seasonal demand revealed a trimodal distribution for tropical travelers aligning with holiday periods, contrasting with a September-to-January peak for visitors escaping winter in four-season climates. Airport trips showed that nearly 90% of tourist passengers headed directly to hotels, concentrated heavily in central areas like Orchard, Bugis, Downtown Core, and Kallang. Additional key destinations included major shopping districts, iconic dining locations like Newton Food Centre and Chijmes, and medical centers, which saw tourist ride volume grow over 500% between 2015 and 2017.
Mobile Tophatting at Shopify
Shopify requires developers to test features because it has no QA team, and its manual “tophatting” means running changes locally before merge. Mobile tophatting was slow because switching branches could invalidate Xcode/Android Studio caches, with builds taking up to 15 minutes. The new process skips compilation by exporting CI-built iOS and Android binaries into disposable PR environments, exposing them through the CI API and GitHub commit statuses, then extending the dev command to authenticate, find and filter artifacts, select an app and simulator or emulator, install it, and launch it. Developers can target pull requests, repositories, branches, or build URLs, while iOS apps are zipped .app folders and Android apps retain APK names. The team reports a positive response and lists local artifact caching, real-device support, and simpler commit or branch inputs as improvements.
Shaping the Future of Payments in the Browser
Shopify tested whether the Payment Request API was ready for its merchants and buyers on supported browsers. The team defined checkout completion rate, time to purchase, drop-off, and qualitative checkout experience as success measures, then ran an A/B experiment with selected merchants and eligible buyers. The isolated Ruby app injected storefront JavaScript, checked eligibility and credit-card support, and routed buyers to either the regular checkout or a browser-provided payment sheet. The experiment recorded a 7% checkout conversion rate drop and a 60% drop at the payment sheet, although time to completion medians were promising. Shopify concluded Payment Request was not yet a good fit, while expecting greater familiarity, payment-sheet improvements, and third-party payment methods through Payment Handler API to help adoption.
Iterating Towards a More Scalable Ingress
Shopify’s infrastructure uses Kubernetes to serve millions of requests per minute, and its growth exposed ingress performance problems during frequent application deployments. After replacing Google Cloud Load Balancer Controller (glbc) with endpoint-aware ingress-nginx, the Edgescale team focused on reducing the NGINX reloads that caused latency increases. They added dynamic configuration: the controller POSTs endpoint lists to an NGINX HTTP endpoint, where lua-nginx-module updates an in-memory shared dictionary instead of regenerating the configuration file and reloading NGINX. Reload avoidance preserves keepalive connections and load-balancing state while reducing the resource effects of replacing worker processes. Tests found ingress-nginx outperformed glbc by multiple orders of magnitude at the 99.99th latency percentile or higher, while dynamic configuration produced its largest gains during deploys, especially beyond the 80th percentile.
Applying Seccomp Filters at Runtime for Go Binaries
Seccomp filters let a Linux process restrict its available system calls, reducing the process attack surface beyond the Dyno’s existing restrictions. This tutorial builds a Go binary that creates /tmp/moo, uses strace -c to identify its runtime syscalls, and configures a whitelist with libseccomp-golang. The filter starts with libseccomp.ActErrno returning EPERM for unlisted calls, then adds ActAllow rules for named calls before loading the filter at runtime; the example also checks kernel support and installs libseccomp-dev. With the listed calls enabled, the program creates the folder successfully, while a later syscall.Exec attempt to run ls -l cannot perform unlisted ioctl, getdents, and statfs operations and reports “Operation not permitted.” The post then describes deploying the dependency with Govendor or Godeps on Heroku.
Fabric for Open Source Android
Open-source Android applications must remain buildable for contributors without exposing the credentials used by third-party services. Kickstarter’s Android application added Crashlytics through Fabric for crash reporting and real-time analytics, but integrating it required protecting the Fabric API key and secret from public source control. The post presents two normal SDK configuration paths—manifest metadata or fabric.properties—then uses a generated properties file, an example fallback, and Gradle logic to supply values from environment variables when available. Contributors without access to the secret repository receive an example apiKey value of 0, allowing the Fabric SDK to compile without the invalid null-key error, while the project’s real credentials remain outside the open-source codebase.
Beyond Web and Worker: Evolution of the Modern Web App on Heroku
Over a decade, the standard Heroku web application has expanded from a web process, database, worker, and Redis queue into a broader architecture for web, mobile, data, and operational demands. The post connects this shift to snappier single-page experiences, native smartphone apps, APIs as integration surfaces, growing streaming data, increasing system complexity, and scale. Its proposed components include a content delivery network, API Gateway, Apache Kafka, object storage, data warehouses, and query or visualization tools, alongside service-oriented architecture and scalable-first design. The architecture separates functionality into smaller services for adaptability, while acknowledging that more components create more scaling and operational levers. It presents Heroku’s PaaS and managed services as a way to reduce that burden, then leaves deployment, communication, CI/CD, monitoring, and security as topics for follow-up posts.
How We Designed the Quotas Microservice to Prevent Resource Abuse
As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.
Behind The Code: Jennice Colaco, Backend Developer
Jennice Colaco is a backend developer who has worked on Shopify’s Marketing Technology team for 10 months. She holds a Bachelor of Applied Science in Computer Engineering from the University of Toronto, and describes her team’s work as building tools that help merchants market products to customers. She treats programming languages as problem-solving tools, is improving her Ruby skills, and uses byebug to step through code line by line when debugging. Her workday combines early planning, coding, pull-request reviews, pairing sessions, and Slack collaboration, while difficult problems are approached through reading, research, whiteboarding, and asking peers for support. She is taking an in-house Ruby course and describes Shopify’s culture as supportive of individual growth, with ongoing conversations about healthy work environments.
A Guide to Mindful Communication in Code Reviews
Code reviews are presented as both technical work and communication, teaching, and learning. The guide defines mindful communication as listening and speaking with compassion, kindness, and self-awareness, then applies that perspective to reviewers and authors. It recommends self-checking before commenting, asking open questions instead of assuming, omitting linter-detectable nits, signaling what can wait, giving example code and documentation, and ending substantial feedback with “What do you think?” Reviewers are encouraged to approve unless major problems exist, while large changes and architectural decisions should receive broader discussion. Authors should link relevant documentation, assume good intent unless feedback is hostile, keep diffs focused, and seek feedback early; these practices take time but can become a learnable habit that varies by team.
Grab Senior Data Scientist Liuqin Yang Wins Beale-Orchard-Hays Prize
Grab Senior Data Scientist Dr. Liuqin Yang, Professor Defeng Sun, and Professor Kim-Chuan Toh received the 2018 Beale-Orchard-Hays Prize for their research paper introducing SDPNAL+. The software employs a majorised semismooth Newton-CG augmented Lagrangian method to solve large-scale semidefinite programming problems with nonnegative constraints. While traditional methods struggled beyond matrix dimensions of 2,000 and 5,000 constraints, SDPNAL+ successfully scales to matrix dimensions of 9,261 and over 12 million constraints. In benchmark testing, the software solved a problem on a desktop PC in 1.5 hours that required 122 hours on a 56-core CPU and 128-GPU cluster using a traditional solver. Grab implements these optimisation techniques to accelerate its passenger-driver allocation algorithms by hundreds of times.
Building Grab’s Experimentation Platform
Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.
Scaling iOS CI with Anka
Shopify needed a unified CI/CD system that could give iOS developers the same reliable, disposable build environments already used for Linux and Android projects. An earlier VMware ESXi setup used a shared storage area network, but its maintenance burden and write throughput limit created stability and scalability problems. The replacement uses Anka, a Docker-like macOS virtualization layer built on Apple’s Hypervisor.framework: one Core i7 Mac Mini runs one VM with local storage, while layered images are pushed to registries and pulled across two MacStadium data centers. Buildkite agents clone temporary snapshots, boot suspended VMs in under a second, run steps, upload artifacts, and delete the VMs afterward. Shopify reports about 350 iOS build jobs per hour for roughly 60 iOS developers, with easier Xcode and macOS upgrades, no shared-storage single point of failure, and horizontal scaling, although capacity remains manually adjusted and spiky demand can increase queue times.
Rolling the Heroku Redis Fleet
Heroku proactively rolled its Redis fleet to a version not vulnerable to CVE-2018-11218 during an embargo, aiming to protect customers while limiting disruption and downtime. The operation relied on immutable infrastructure: engineers built, tested, reviewed, and released a new server image, then replaced High Availability standbys before scheduling customer maintenance. During each update, the team verified replication, broke the link so the standby could accept writes, redirected application config vars to the new primary, de-provisioned the old primary, and created a replacement standby. Automation controlled concurrency and limited replacement work to the on-call operator’s business hours, while customers received advance and completion emails. The fleet roll covered all Heroku Redis customers within five weeks of notification and finished one week before the public announcement; the process also applies to Heroku Postgres and Apache Kafka on Heroku.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI455
- Developer Experience446
Weekly digest
The best engineering reads, once a week.