Loading…
Browse by type
Explainer reads
Explainer posts on Yomu, summarised newest first.
- 1,382 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Explainer posts
Handling Addresses from All Around the World
International address handling is difficult because countries and locales differ in field order, required fields, terminology, translations, and display conventions. Shopify models each address with fields such as names, address lines, zones, postal code, city, country code, and phone, then stores country-specific metadata in YAML-backed country data. A reusable format describes form and display order, allowing clients to reorder fields, prepopulate country and zone values, and add locale-specific characters such as Japan’s 〒 and 様. Label keys and translations handle terms such as province, prefecture, postcode, and pincode, while an API endpoint keeps applications synchronized and accepts locale information. The team also provides @shopify/address and higher-level components, while identifying caching as a remaining issue to reduce repeated network calls.
Running Apache Kafka on Kubernetes at Shopify
Shopify adopted Apache Kafka in 2014 as a data bus for reliable messaging, initially collecting events and aggregating logs, then expanded to multi-tenant regional and aggregate clusters delivering billions of messages daily. In 2017, it migrated Kafka infrastructure from data centers to the cloud with zero downtime by mirroring regional data to both aggregate clusters before moving producers and consumers to cloud clusters. Shopify chose Kubernetes over GCP virtual machines because StatefulSets provide ordered, graceful deployment and scaling, while Custom Resources and Controllers support cluster-specific automation. The implementation uses self-built Kafka images in an own registry, persistent volumes, node and inter-pod affinity controls, taints, rack-awareness across zones, readiness probes, and resource-monitoring sidecars to limit restart and availability risks. The migration completed successfully, and Shopify now runs multiple regional clusters plus an aggregate cluster that mirrors data for the data warehouse.
Creating Locale
Shopify’s move to support six additional languages exposed a challenge in its admin graphs: condensing numbers without distorting locale-specific conventions. The team needed locale-aware abbreviations for sales, visits, and customer data, because unshortened values could consume axis space. Existing JavaScript libraries either exceeded the requirement or conflicted with Shopify’s stack, while the Intl API lacked implemented number-condensing support. During Hack Days, the developers used abbreviation data from Unicode’s Common Locale Data Repository (CLDR) and currency information from Intl.js to build functions for locale-specific numbers and currencies. Their package handles examples such as Portuguese “3 mil” and non-Western systems including India’s lakhs and crores, and was released as condense-number on npm in July 2018.
Mohammed Ridwanul Islam: How Mentorship, the T Model and a Pen Are the Keys to His Success
Mohammed Ridwanul Islam's career story connects his upbringing in Noakhali, Bangladesh, varied work and music experience, and interest in engineering with his current role on Shopify's Eventscale team. He describes how trying to build a game led him to write C# and later study Electrical Engineering at the University of Waterloo, where an internship became a full-time position. Eventscale develops tools, libraries, and infrastructure for collecting event-oriented streaming data across web, backend, and mobile platforms, supporting analytics and operational needs. Mentorship during a difficult team project taught him project management concepts, while habits, VIM, Java, and writing designs on paper shape his working practices. His advice is to build broad competence alongside deep expertise, and to prioritize problem-solving, values, social impact, and continued growth.
Dev Degree
Dev Degree is Shopify’s work-integrated learning program, combining an accredited university degree with double the experience of a traditional co-op. It emerged from concerns that graduates often knew theory but lacked practical software-development skills, and it required a long-term partnership with a university willing to change curriculum and assessments. At Carleton, students take three courses each semester while working 25 hours weekly at Shopify; students also complete a first-year Developer Skills Training program, submit practicum reports every four months, and receive technical and Life@Shopify mentors. The program allocates at least 20% of degree credits to practicums and uses learning outcomes influenced by ACM and IEEE Computer Society standards. Shopify reports 95% retention, women representing over 50% of students, and students becoming productive after nine months; it has expanded to York University, while the company says broader adoption is needed.
Journey of a Tourist via Grab
Grab analyzed platform ride data from millions of tourist passengers representing over 150 countries visiting Singapore. Over 60% of these tourist riders originated from Southeast Asia, while non-regional visitors mainly came from China, the United States, and India. Seasonal demand revealed a trimodal distribution for tropical travelers aligning with holiday periods, contrasting with a September-to-January peak for visitors escaping winter in four-season climates. Airport trips showed that nearly 90% of tourist passengers headed directly to hotels, concentrated heavily in central areas like Orchard, Bugis, Downtown Core, and Kallang. Additional key destinations included major shopping districts, iconic dining locations like Newton Food Centre and Chijmes, and medical centers, which saw tourist ride volume grow over 500% between 2015 and 2017.
Beyond Web and Worker: Evolution of the Modern Web App on Heroku
Over a decade, the standard Heroku web application has expanded from a web process, database, worker, and Redis queue into a broader architecture for web, mobile, data, and operational demands. The post connects this shift to snappier single-page experiences, native smartphone apps, APIs as integration surfaces, growing streaming data, increasing system complexity, and scale. Its proposed components include a content delivery network, API Gateway, Apache Kafka, object storage, data warehouses, and query or visualization tools, alongside service-oriented architecture and scalable-first design. The architecture separates functionality into smaller services for adaptability, while acknowledging that more components create more scaling and operational levers. It presents Heroku’s PaaS and managed services as a way to reduce that burden, then leaves deployment, communication, CI/CD, monitoring, and security as topics for follow-up posts.
Behind The Code: Jennice Colaco, Backend Developer
Jennice Colaco is a backend developer who has worked on Shopify’s Marketing Technology team for 10 months. She holds a Bachelor of Applied Science in Computer Engineering from the University of Toronto, and describes her team’s work as building tools that help merchants market products to customers. She treats programming languages as problem-solving tools, is improving her Ruby skills, and uses byebug to step through code line by line when debugging. Her workday combines early planning, coding, pull-request reviews, pairing sessions, and Slack collaboration, while difficult problems are approached through reading, research, whiteboarding, and asking peers for support. She is taking an in-house Ruby course and describes Shopify’s culture as supportive of individual growth, with ongoing conversations about healthy work environments.
Rolling the Heroku Redis Fleet
Heroku proactively rolled its Redis fleet to a version not vulnerable to CVE-2018-11218 during an embargo, aiming to protect customers while limiting disruption and downtime. The operation relied on immutable infrastructure: engineers built, tested, reviewed, and released a new server image, then replaced High Availability standbys before scheduling customer maintenance. During each update, the team verified replication, broke the link so the standby could accept writes, redirected application config vars to the new primary, de-provisioned the old primary, and created a replacement standby. Automation controlled concurrency and limited replacement work to the on-call operator’s business hours, while customers received advance and completion emails. The fleet roll covered all Heroku Redis customers within five weeks of notification and finished one week before the public announcement; the process also applies to Heroku Postgres and Apache Kafka on Heroku.
Heroku CLI: Completing Autocomplete
Heroku CLI Autocomplete had to support a dynamic, plugin-based command set while completing values that often reside behind the Heroku API. Instead of static shell case statements, its cache builder iterates through installed commands and generates setters that supply command, flag, and value completions when the user presses Tab. A shim sourced from the user’s .bashrc or .zshrc limits profile changes, fails silently when unavailable, and permits updates without repeated edits. API-backed values are cached on disk, while oclif custom hooks invalidate or rebuild relevant caches after commands change data. The completion function also parses the partially entered command line to provide contextual results, such as add-on aliases for a specified app; the work is being incorporated into an oclif plugin.
Shopify Interns Share Their Tips for Success
Shopify interns and a former intern describe how they approached interviews, daily work, and growth across backend, development, solutions engineering, iOS, and application security roles. Their routines include stand-ups, pull-request review, coding, meetings, pair programming, production monitoring, and regular breaks, with some teams organizing work around three-week release cycles. Interview advice centers on completing the application, explaining non-traditional backgrounds, demonstrating passion and adaptability, communicating technical solutions to different audiences, and being honest about knowledge gaps. Inside Shopify, contributors say success depends on asking questions early, seeking feedback and mentorship, taking initiative, talking beyond one’s team, and holding strong opinions weakly. The post also announces Fall 2018 intern hiring, with applications due May 11, 2018, at 9:00 AM EST.
Event Sourcing made Simple
Event Sourcing stores application changes as immutable events, providing a history that can explain how current data came to be. The post introduces four components: Events record changes, Calculators update Aggregates from those events, and Reactors trigger side effects or create further events. Kickstarter implemented a minimal framework for d.rip using regular Active Record aggregates, a regular SQL database, synchronous atomic updates, and roughly 200 lines of home-made code. The approach supports replaying history to reconstruct past state, repair calculator bugs, backfill new aggregate fields, optimize read models, and forward events to analytics platforms, while presenting naming and mutation-to-event decomposition as important challenges.
MJIT: A Method Based Just-in-time Compiler for Ruby
Ruby programs are compiled into YARV instructions and executed by a stack-oriented virtual machine, while register-oriented representations can expose more opportunities for compiler optimization. The post explains how YARV represents a simple expression, contrasts stack and register architectures, and introduces Vladimir Makarov’s Ruby RTL, an intermediate representation distinct from GCC’s RTL. MJIT uses RTL generated by MRI to make speculative, operand-specific optimizations, then compiles methods to machine code rather than relying only on interpretation. The interview discusses deoptimization: when assumptions change, MJIT generates a new machine-code version and can switch between JITed execution and RTL interpretation. The participants say this overhead should be tiny and should not degrade performance overall, while acknowledging edge cases where MJIT is not faster and that benchmarking JITs is difficult.
Deconstructing Monolithic Apps with SHIFT Commerce
SHIFT Commerce describes its move from a monolithic application toward microservices, while arguing that architectural change should address concrete problems rather than follow fashion. The company identifies communication overhead, governance and high-risk deployments, and language or infrastructure limitations as reasons to extract functionality, including its Order Management Suite, admin panel, and integration system. SHIFT uses managed Apache Kafka on Heroku as an asynchronous event stream, with chronologically ordered, durable messages organized into topics so services can maintain local datasets. Its eight-step migration progressively adds producers, builds a consumer-backed database, tests and replicates logic, enables feature-flagged traffic, exchanges events, and removes deprecated monolith code. SHIFT reports more regular releases, lower latency on I/O-restricted API calls, and prevention of user-facing outages, while saying the migration remains ongoing.
A Pods Architecture To Allow Shopify To Scale
Shopify outgrew its ability to scale by buying larger database servers, so it introduced database sharding in 2015. Sharding improved performance and horizontal scalability but made actions dependent on every shard, increasing the risk of platform-wide unavailability. In 2016, Shopify reorganized its runtime around pods: each pod groups shops on isolated datastores, while shared workers, application servers, and load balancers are restricted to one pod at a time. Sorting Hat assigns each request to a pod and location, while every web request and delayed job is kept within one pod; paired data centers and Pod Mover support recovery without dropping requests or jobs. The design makes a pod the unit for scalability, resiliency, and disaster recovery, and simplifies evacuating an entire data center.
Dissecting Kubernetes Deployments
Kubernetes Deployments provide declarative updates for Pods and ReplicaSets, enabling application versions to roll out gradually rather than restarting every Pod at once. The Deployment controller uses informers and triggers for Deployment, ReplicaSet, and Pod events, then synchronizes state by creating ReplicaSets and scaling old and new sets according to replica counts, max surge, and available Pods. During a rolling update, it waits for new Pods to respond before terminating old ones; progress deadlines help detect stalled rollouts, including crash loops. After completion, old ReplicaSets are cleaned up but a configured number is retained for rollback, which requires changing image tags rather than continually using :latest. The post also shows a Go informer that watches Deployment creation and update events and explains that the trigger-based design supports custom controllers.
Implementing ChatOps into our Incident Management Procedure
Shopify’s incident management process separates coordination from technical remediation: the incident manager on call (IMOC) leads response, while component experts fix production issues and a Support Response Manager handles public communication. The process follows a simplified Incident Command System hierarchy and moves from detection and confirmation through communication, mitigation, verification, documentation, investigation, and root cause analysis. Its ChatOps chatbot, Spy, runs in Slack through Lita and integrates with PagerDuty, StatusPage, and GitHub via webhooks, providing commands for paging, incident startup, status communication, mitigation, and stopping incidents. Spy binds incidents to a shared #war-room, records notes and event times, generates service disruption documents, sends status-page reminders, and can summon the IMOC squad when an incident lasts beyond a specified period. The result is a more consistent response that reduces context switching and post-incident toil while supporting detailed post-mortems.
Deep Dive into Database Timeouts in Rails
Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout_timeout, connect_timeout, and read_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.
Namespacing Actions for Redux
Kickstarter’s front-end team encountered a Redux action-namespacing problem when multiple instances of the same functional React component responded to actions intended for only one instance. They rejected local state because the application used functional components and redux-loop, which routes state updates and side effects through reducers, and they sought programmatic namespacing compatible with combineReducers and a low-surface-area component API. The team compared nested reducers, higher-order reducers, and a closure-based module pattern; the latter met the stated constraints with namespaced actions, wrapper components, reducers, and a namespace returned by a factory that defaults to a UUID. Namespaces must be preserved when reducers trigger further actions, including redux-loop results, and complex composition can lose the required binding. The approach has worked for most reusable and compound components, while possible performance costs from many reducers remain a future concern.
How Shopify Merchants can Measure Retention
Shopify addresses customer retention for merchants operating non-contractual businesses, where churn is not directly observable. Common rules that label customers inactive after N days create arbitrary binary states, so the post proposes probabilistic modeling with the BG/NBD (Beta Geometric / Negative Binomial Distribution) model. It represents each customer with a purchase rate and churn probability, estimates their population distributions, and uses age, recency, and repeat-purchase frequency as sufficient statistics. Shopify fits a modified implementation in lifetimes using Apache Spark DataFrames, subsampling up to 50k customers per shop for performance, and produces activity probabilities and expected future purchases for more than 500 million historical merchant customers. The model is interpretable and supports customer ranking and targeted campaigns, but it handles seasonality poorly and does not easily incorporate additional variables such as country or products purchased.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.