Loading…
Shopify
Global commerce platform powering millions of businesses with essential infrastructure and innovative engineering solutions.
328 postsWebsite
Latest articles
Shopify ·
Using Betas to Deploy New Features Safely
Continuous deployment makes changing a running system risky, especially for a platform serving more than a million merchants and changing code multiple times daily. The post presents beta flags as a control mechanism: per-Subject BetaFlags support explicit activation, while BetaRollouts use a percentage and a digest-based modulo calculation to assign a stable, growing subset as rollout increases. It then places a higher-level Feature abstraction above those primitives, allowing direct targeting, percentage rollout, per-Subject opt-outs, and a 100% kill switch that can halt a feature without deleting thousands of database records. The guidance also covers testing permutations, notifying owners through BetaIdentifier metadata, and removing stale flags after rollout. These patterns provide operational control and safer deployment, while requiring substantial implementation and ongoing cleanup.
2023-10-18Shopify ·
Technical Mentorship Reimagined: Time-bound and No Awkward Asks Necessary
Shopify’s Engineering Mentorship Program was created to give developers a structured way to pursue technical and leadership growth alongside the company’s existing just-in-time collaboration. Survey findings indicated that developers wanted to learn from colleagues but struggled to make dedicated time, motivating a recurring program designed around those constraints. Each six-week cycle uses voluntary sign-up, automated matching based on expertise, reporting relationships, and job level, weekly one-hour meetings, goal setting, Slack support, and an optional guidebook. The time-bound format lets participants try a mentorship relationship without an open-ended commitment, while mentors also practice teaching and leadership; by 2020, more than 300 developers had participated as mentees and more than 200 as mentors, 98% of survey respondents said they would recommend it, and demand exceeded available mentor capacity.
2023-10-18Shopify ·
How to Make Dashboards Using a Product Thinking Approach
The guide presents dashboards as automatically updated collections of visualisations or metrics for monitoring business questions and argues they should be built with a product-thinking approach. It first recommends deciding whether a dashboard is appropriate, then defining one clear goal, a specific audience, and metrics that reflect their needs before sketching the layout with stakeholders. Implementation guidance covers readable and efficient queries, version control, peer review, visual hierarchy, restrained content, business and data context, refresh frequency, technical reviews, and user sign-off. After launch, teams should market the dashboard, make it discoverable, monitor usage and outcomes, iterate when it falls short, and assign ownership for maintenance. The intended outcome is a dashboard that stays useful and delivers business impact by keeping users front and centre.
2023-10-18Shopify ·
Managing Google Cloud Platform Project-Wide SSH Keys
Google Cloud project-wide SSH keys can remain in project metadata indefinitely after a user connects to a VM, potentially allowing matching private-key holders to access Linux VMs with root access unless project-wide keys are blocked. Shopify developed SSH-Pruner because OS Login did not work with Google Kubernetes Engine, while deleting all keys also removed Google’s keys, disrupted scripts, and created slow metadata updates. The new Go application uses Google Cloud APIs to find organization projects with Compute Engine enabled, reads the “ssh-keys” metadata, parses entries, retains unexpired keys, writes pruned common instance metadata, and records readable logs; Google-managed “sshKeys” entries are skipped. SSH-Pruner was already running as a cron job on some projects, with a slow rollout planned to limit workflow disruption.
2023-10-18Shopify ·
Using GraphQL for High-Performing Mobile Applications
GraphQL is presented as a way to address mobile developers’ REST API pain points, especially repeated JSON-to-model mappings and failures when server fields or types change. It provides a strongly typed, self-documented schema contract, while introspection enables tooling to generate client code and surface deprecations at compile time. The post describes mobile practices including using reusable UI-oriented fragments, aliases for readable data models, and directives that conditionally include or skip fields; Shopify uses the open-source Syrup tool to generate strongly typed Swift and Kotlin code from application queries. This flexibility shifts responsibility toward query design: Shopify calculates field costs, limits calls through a replenishing per-user cost bucket, and requires clients to consider query complexity for responsiveness and server resources. GraphQL is positioned as language-independent and suited to bandwidth, latency, and UX-sensitive clients.
2023-10-18Shopify ·
Apache Beam for Search: Getting Started by Hacking Time
Search relevance can use clickstream signals such as clicks and purchases to promote popular results and demote results users do not favor, but batch processing can delay those adjustments by days. The Discovery team at Shopify uses Apache Beam, a unified batch and stream processing system, to support real-time and historical search-behavior workflows, including boosting, popularity aggregates, offline evaluation sets, and reinforcement learning tasks. The article’s central lesson is that Beam requires careful handling of event time, processing time, delayed events, out-of-order data, and watermarks, especially when combining streams. A Kafka source can use Kafka processing time, client create time with a five-minute out-of-order allowance, or a custom timestamp policy that reads the SearchQueryEvent.searchTimestamp field; Beam windows add further buffering and lateness controls. Accurate timestamping is presented as the first milestone for replaying clickstream data reliably before tackling more complex operations such as streaming joins.
2023-10-18Shopify ·
How Shopify Uses WebAssembly Outside of the Browser
Shopify describes using WebAssembly (Wasm) outside the browser to run untrusted Partner code on its own infrastructure, reducing partners’ need to operate scalable web services and avoiding network latency for time-sensitive customizations. Wasm provides a sandboxed, stack-based environment with explicit host imports, protected call stacks, and runtime type checking, while targeting near-native performance and multiple source languages. Shopify wraps Fastly’s open-source Lucet runtime and compiler in a Rust web service called the Wasm Engine, which manages module I/O and storage and applies results during runtime processes such as web requests. In a test executing 100,000 modules per minute for roughly five minutes, Lucet execution hovered around 100 μs and total p99 execution time was about 4 ms. Shopify selected AssemblyScript, with TypeScript-like syntax, for first-class developer support and is extending its tooling, SDKs, monitoring, and language support.
2023-10-18Shopify ·
Resiliency Planning for High-Traffic Events
Shopify’s preparation for Black Friday Cyber Monday (BFCM) centers on building and maintaining resiliency practices for a high-traffic platform and large development teams. Regular load and stress tests verify required capacity, probe limits, detect regressions, and compare behavior across differently configured shops. Teams document expected user experiences and dependency failure modes in a resiliency matrix, then use game days to test that model against reality, including alerts, paging, operator readiness, and response procedures. Incident analysis treats disruptions as learning opportunities rather than stopping at a single “root cause”; nearer to BFCM, Shopify slows feature and service changes while increasing work on performance, reliability, scale, runbooks, tools, and bottlenecks. The source says these practices support a platform that powered over $5.1 billion in sales.
2023-10-18Shopify ·
Simplify, Batch, and Cache: How We Optimized Server-side Storefront Rendering
The post explains how a new Ruby-based server-side Storefront Renderer reduced the time required to serve Shopify storefront requests. It combines MySQL multi-statement queries, handcrafted SQL, a thin data-mapping layer built from plain old Ruby objects, query book-keeping with eager- and lazy-loading, multiple LRU caching layers, and techniques for reducing memory allocations. For a product page, one database round trip can load the product, variants, images, shop, theme, and related resources; later requests can replay previously observed queries early, while less frequently used data remains lazy-loaded. The resulting renderer serves 75% of requests in under ~45ms, 90% in under ~230ms, and 99% in under ~900ms, with average response time nearly five times faster than the previous implementation.
2023-10-18Shopify ·
How to Reliably Scale Your Data Platform for High Volumes
Shopify’s data platform prepared for BFCM 2020, when merchant sales exceeded $5.1 billion and platform throughput rose by an average of 150 percent. The Data Platform Engineering team focused on throughput risk, which grows with volume and is greatest near ingestion and processing, while prioritizing work through four service tiers. For Tier 1 and Tier 2 systems, engineers defined each service’s primary objective, identified adjustable controls such as job frequency, overlap intervals, CPU and memory requests, BigTable node pools, and ProxySQL connection queues, then used controlled load tests to expose capacity and recovery risks. The resulting work included cluster scaling, load-balancing adjustments, query and job tuning, and improved indices, schemas, dashboards, playbooks, SLOs, incident handling, and alerts. The post reports that BFCM 2020 completed without merchant-impacting incidents; only a handful of minor, self-healing incidents occurred, while monitoring and occasional tuning preserved spare capacity.
2023-10-18Shopify ·
Organizing 2000 Developers for BFCM in a Remote World
Shopify organized a cross-company program to prepare its platform for Black Friday and Cyber Monday (BFCM), coordinating more than 2,500 people responsible for over 1,000 deploys across services. Engineering Program Managers absorbed administrative, communication, and reporting work so BFCM leads could focus on technical challenges, while departmental champions assigned preparation tasks. Preparation covered maintenance, game days, load tests, capacity increases, weekly tracking, and a Caution Tape bot requiring risk-to-value assessments for pull requests in critical areas. Deploys were slowed through smaller batches, longer canaries, and a planned weeklong lock while communication used Slack, GitHub, documentation, presentations, emails, and office hours. The effort coincided with $5.1+ billion in BFCM sales and load testing plus resource scaling paid off, but developer productivity declined amid inconsistent messaging; future plans called for a more balanced approach, automation, and reusable communication templates.
2023-10-18Shopify ·
The State of Ruby Static Typing at Shopify
Shopify’s Ruby monolith receives about 400 commits and 40 deployments daily, spanning 37,000 files, 622,000 methods, and over 2,000,000 calls, making correctness difficult despite rigorous review and 150,000 automated tests. Since 2018, Shopify’s Ruby Infrastructure team has pursued static typing and adopted Sorbet, discussed in a 2020 Shipit! event. The post describes Sorbet’s handling of simple collections such as T::Array[Integer], nested types using T.untyped, and dedicated classes for complex structures, while noting obstacles involving ActiveSupport::Concern, implicit inclusion requirements, metaprogramming, and dynamic GraphQL resolvers. For Rails projects, guidance is to begin with typed: false, generate RBI files using Tapioca or sorbet-rails, then move suitable files toward typed: true based on reuse, errors, collaboration, or churn, checking generated files into the repository.
2023-10-18Shopify ·
A World Rendered Beautifully: The Making of the BFCM 3D Data Visualization
The 2020 Black Friday Cyber Monday Globe and a related Live View visualization had to represent orders from one million merchants within two months while remaining realistic and performant. The team used a reusable layer architecture with three.js, physically based rendering, 32-bit EXR environment maps, and custom materials and shaders for the Earth and carbon-offset bubble effects. To display thousands of orders, it replaced arc-heavy rendering with particle-based visuals built from Points, BufferGeometry attributes, and ShaderMaterial. An automatic optimizer monitors frame rate and, when it stays below 55 fps for more than two seconds, lowers canvas pixel ratio to 1x and reduces environment-map resolution, helping low-power devices return to 60 fps. The resulting implementation became an internal library for future globes.
2023-10-18Shopify ·
Capacity Planning at Scale
Shopify’s BFCM preparation combines traffic forecasting, resource planning, and platform-wide scalability testing to reduce the risk of outages during its busiest sales period. The engineering team worked with data scientists to model expected traffic, reviewed historical numbers, added safety margins, and created a master Google Cloud resource plan covering CPUs, storage, machine types, locations, and project-level quotas. Because 2020 brought unusual uncertainty from COVID-19-driven changes in online commerce, Shopify modeled multiple demand scenarios and chose the more aggressive growth cases, while retaining buffers for changes and regional failover. It also ran the Oktoberfest scale-up on Google Kubernetes Engine, using internal load-testing tooling powered by go-lua and embedded Lua to simulate storefront browsing, cart additions, checkout, and mock payment processing. The exercise exposed bottlenecks, which teams fixed before repeating the test; the successful rerun increased confidence for BFCM and reinforced a longer-term aim of more automated, dynamic, and resilient systems.
2023-10-18Shopify ·
Pummelling the Platform–Performance Testing Shopify
Shopify prepares for high-traffic sales events such as BFCM by repeatedly simulating customer traffic on representative test shops, aiming to expose bottlenecks before merchants experience them. Its Platform Conditioning team supports service-owning teams with approachable performance-testing tools and processes, distinguishing load tests for known traffic levels from stress tests that push services toward failure. A Go load generator runs Lua flows through lightweight VMs, models realistic traffic shapes and costly endpoints, and can issue tens of millions of requests per minute or more. Tests are initiated through Slack using the internal ChatOps tool Spy, which returns links to dashboards, logs, and results. Cronograma provides formal experiments with hypotheses, observations, conclusions, alert detection, Slack timelines, and comparable metrics such as response times, 5xx errors, and requests per minute; the tools have led to performance improvements and give Shopify confidence in future sales events.
2023-10-18Shopify ·
Vouching for Docker Images
Kubernetes does not enable Docker image signature verification by default, leaving organizations to decide whether images are trusted before they run in a cluster. The post explains code signing through hashes and public/private keys, then describes Binary Authorization as a combination of metadata storage, an enforcer, and a signing service. Using Grafeas and Kritis, or GKE’s Binary Authorization, cluster policies require specified attestors to sign images; invalid or unsigned images are blocked before their containers start. Voucher supplies the missing signing component as a REST service running in Cloud Run or Kubernetes: build pipelines submit image paths, Voucher performs security checks, signs qualifying images, and records signatures in the metadata service. Its extensible checks can verify internal ownership, GitHub commits, approvals, builder identity, and vulnerability counts, while distinct cluster policies balance compliance requirements with faster experimentation; Voucher later moved into the Grafeas organization.
2023-10-18Shopify ·
Static Typing for Ruby
Shopify’s Ruby monolith spans 37,000 files, 622,000 methods, and more than 2,000,000 calls, making fast feedback and stability difficult despite rigorous reviews and 150,000 automated tests. Sorbet was selected after the team evaluated requirements for gradual typing, speed, and support for Ruby and Rails features including metaprogramming, overloading, and class reopening. Its RBI files represent constructs it cannot infer, while per-file sigils allow adoption to progress without blocking development, and SorbetMetrics tracks sigils, typed calls, and method signatures. Shopify treats typing as a product, combining CI enforcement, developer support, surveys, and interviews to guide rollout and measure sentiment. The excerpt reports 80% of monolith files, including tests, at typed: true or higher, with almost half of calls and methods covered and type checking under 15 seconds on developer machines.
2023-10-18Shopify ·
Adopting Sorbet at Scale
Shopify describes its adoption of Sorbet static typing across a Ruby monolith containing 37,000 files, 622,000 methods, and more than 2,000,000 calls. At the time of writing, Sorbet ran on every pull request, 80% of files were typed: true or higher, almost half of calls were typed, and half of methods had signatures. To handle Ruby and Rails idioms, the team built RuboCop Sorbet for compatibility rules, Tapioca for gem and DSL RBI generation, and Spoom for programmatic tooling, metrics, and LSP access. A controlled experiment found fewer production NoMethodErrors in files typed: true after typing about 20% of the application, although the results were preliminary and signatures had not yet been added. Shopify planned to reach typed: true across all files, improve Rails support, and continue collaboration around Sorbet and RBS.
2023-10-18Shopify ·
How to Build a Production Grade Workflow with SQL Modelling
Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.
2023-10-18Shopify ·
How to Introduce Composite Primary Keys in Rails
Shopify addressed inefficient data access in its multi-tenant Rails application, where auto-incrementing primary keys interleaved records from many shops even though queries usually targeted one shop. InnoDB stores rows in B+ trees and loads pages into a buffer pool, so the team changed orders and other suitable tables to composite primary keys such as [shop_id, order_id] while keeping an auto-incrementing id secondary key and configuring Active Record to use id. Supporting Ghostferry, migration tooling, and data extraction systems also had to accommodate the schema change, with unique secondary constraints avoided in some cases because LHM migrations could deadlock. On the most queried table, common queries improved 5–6x, median buffer pool reads fell from 1.8 to 1.2, and elapsed database time dropped roughly one hour per day per shard, while inserts became about 10x slower.
2023-10-18