Loading…
Shopify
Global commerce platform powering millions of businesses with essential infrastructure and innovative engineering solutions.
328 postsWebsite
Latest articles
Shopify ·
Making Shopify’s Flagship App 20% Faster in 6 Weeks Using a Novel Caching Solution
Shop’s home feed, the app’s most used feature, aggregated orders from millions of Shopify and non-Shopify merchants plus tracking data from dozens of carriers, accounted for 30% of database load and affected application performance. Before each database update, the custom write-through cache marks a user’s cache invalid, applies the write, then refreshes and revalidates it. To handle concurrent updates, it uses a separate pending-writes key and a short-expiry key to distinguish active writes from interrupted processes, while Active Record Concerns preserve the existing ORM interface. After a staged validation rollout, the cache reduced database load by 15%, overall app latency by about 20%, and overall GraphQL latency by 20%, with a double-digit decrease in CPU usage.
2023-10-18Shopify ·
Using Rich Image and Text Data to Categorize Products at Scale
Shopify revisited product categorization as its platform expanded to millions of merchants selling billions of products across diverse industries. The model uses product titles, descriptions, types, vendors, collections, tags, and images, converting them into embeddings with Multilingual BERT and MobileNet-V2 before hierarchical classification across the seven-level Google Product Taxonomy. Each taxonomy level is treated as a separate multi-class task, with parent predictions fed into child predictions and weighted losses combined during training; confidence thresholds filter predictions at each level. The upgraded system increased leaf precision by 8% while almost doubling coverage, although thresholds require balancing accuracy, coverage, merchant experience, and performance in sensitive categories, and future work includes addressing data imbalance and adding merchant-level features.
2023-10-18Shopify ·
A Kotlin Style .copy Function for Swift Structs
Immutable state objects are used to keep rendering code from modifying shared values, supporting clearer value origins, fewer bugs, and more focused testing. Kotlin data classes provide a copy function that replaces selected fields while preserving the rest, allowing the new object to become the source of truth. Swift structs lack an equivalent convenience, and defining a complete initializer plus a parameter-based copy method becomes cumbersome, particularly with many or nested properties. Optional properties expose a further issue: nil must sometimes mean “set this field to nil,” not “use the existing value.” A functional builder closure solves that distinction by collecting overrides while retaining the original struct’s values for other fields, and a Sourcery stencil can generate the initializer and builder boilerplate.
2023-10-18Shopify ·
5 Steps for Building Machine Learning Models for Business
The article presents a five-step workflow for deciding when and how to build machine learning products in a business setting. It recommends delaying machine learning for new products, establishing data-collection foundations, and testing simple heuristics as baselines before investing in a model. For an initial model, teams should favor simple algorithms, limited feature sets, and off-the-shelf solutions where appropriate, then measure business-aligned impact before optimizing. The workflow also calls for planned model iteration and explicit comparison with existing versions, while accounting for prediction stability when changes could affect user trust. The authors say they used these practices at Shopify to help build and scale machine learning products and to productionize models intended to drive impact.
2023-10-18Shopify ·
Diggin’ and Fetchin’ with TruffleRuby
The investigation starts with production Ruby code that chains Hash#fetch calls and uses an empty-hash default to return an IdentityObject for missing nested data. It compares that workaround with Hash#dig, which is cleaner but does not provide fetch's same control over missing keys, defaults, and explicit nil values. To prototype a combined Hash#dig_fetch method, the author modifies TruffleRuby, using Primitive's hash_get_or_undefined and undefined? to distinguish absent keys from keys containing nil, then replaces recursive traversal with an iterative implementation in shared Diggable logic. For a hash with nine nested keys, iterative changes lift dig from about 2.5M to 16M iterations per second and lift dig_fetch from about 2.5M to 15.5M. The dig_fetch prototype enables the original refactor, but it is not ready for general Ruby adoption because Array and Struct interoperability and corresponding MRI work remain.
2023-10-18Shopify ·
Modelling Developer Infrastructure Teams
Developer Infrastructure at Shopify spans local and cloud development environments, CI systems, deploys, and technology-specific foundations for web and mobile applications. The post models these teams against a three-phase develop–validate–deploy workflow, distinguishing horizontally integrated teams that own one phase from vertically integrated teams whose systems span phases for a particular technology. It uses this model to identify dependencies, gaps, differing scopes, collaboration patterns, and three work avenues—concentration, expansion, and interfacing—with the mission of improving and scaling the cycle. The analysis informed organizational changes, including a Client Foundations team, closer collaboration, and plans for workflow-wide user research focused on the developer journey from writing a patch to production.
2023-10-18Shopify ·
Bridging the Gap Between Developers and End Users
Shopify describes how distance between developers and end users can weaken product context and empathy as companies grow. The company created Bridge the Gap in 2014, connecting employees with merchants through Support teams. The program uses workshops, an internal podcast built from support-call clips and discussions, and voluntary one-hour shadowing sessions in which participants observe Support Advisors helping merchants via chat, phone, or email. Direct exposure is presented as a way to understand unexpected product use, identify customer workarounds that may inform fixes or new products, broaden domain knowledge, and develop listening, communication, patience, empathy, and expectation-setting skills. The broader goal is to make merchants accessible to Shopify’s workforce as people rather than abstract accounts or personas.
2023-10-18Shopify ·
Understanding GraphQL for Beginners–Part Three
Part Three of a beginner GraphQL series introduces mutations for creating, updating, and deleting database objects, extending the earlier focus on queries that fetch data. Using the existing Rails repository with Food and Nutrition ActiveRecord models, it shows how to generate mutation classes with rails g graphql:mutation and register root fields in mutation_type.rb. The tutorial defines input arguments, uses snake_case fields that GraphQL converts to camelCase, and implements a resolve method that creates or locates records and returns typed responses. It demonstrates foodCreate, foodUpdate, and foodDelete in GraphiQL, while exercises ask readers to build corresponding Nutrition mutations. It concludes that mutations use the same endpoint as queries and that each schema field has a resolve method that returns its response.
2023-10-18Shopify ·
Connecting with Mob Programming
A six-person team drawn from three teams had six weeks to complete an ambitious Shipping project, while several members were new to the domain or company and had not worked together. Pairing was uncommon, and pull requests received little substantive feedback, creating uncertainty about whether the team was growing, connecting, and learning. After Sheldon Nunes introduced mob programming, the team used a shared screen and ten-minute driver rotations, with each driver pushing changes before the next person pulled them. This kept everyone engaged, accelerated reversible decisions, limited context dumps, and made most code review back-and-forth unnecessary. Across nearly daily sessions for three months, the team reported stronger relationships, easier feedback, restored remote learning, and better support for newcomers, while identifying drawing, public discoverable calls, simple environments, cross-disciplinary participation, and breaks as important practices; tests and documentation were considered poor fits for mobbing.
2023-10-18Shopify ·
A Guide to Running an Engineering Program
Shopify describes a playbook for running large engineering programs as platform complexity grows across products, tooling, and architecture. The framework begins with a Program Plan covering the problem statement, objectives, guiding principles, definition of done, risks, mitigation, staffing, scope, and a path to completion. Execution uses six-week cycles that revisit unfinished goals, regressions, emerging risks, and the planned route to completion; the described program spans 200 people, nine sub-organizations, and 92 projects. Supporting rituals include weekly updates, status checks, risk and escalation triage, cycle reviews, retrospectives, RFCs, and performance testing with Lua scripts orchestrated by Genghis. The post concludes that engineering program management must adapt to each organization and that a combination of scheduled and ad hoc rituals helped Shopify pursue its goals.
2023-10-18Shopify ·
Perspectives on React Native from Three Shopify Developers
Three Shopify developers—web developer AJ Robidas, Android developer Michelle Fernandez, and iOS developer Ash Furrow—describe how React Native changes mobile development for people from different backgrounds. They say its React-based model lets web developers build for iOS and Android, helps native developers create interfaces faster and reach a wider audience, and supports collaboration across web, Android, and iOS teams. The developers discuss surprises including React Native’s small set of building blocks, fast UI feedback, native bridge modules, existing SDK integration, and apps that can be difficult to distinguish from Swift or Objective-C software. They also identify challenges such as mobile-specific performance concerns across the RN Bridge, the Metro and Node ecosystem, and the large dependency stack. At Shopify, Visual Studio Code, command-line tooling, internal foundations, workshops, and cross-disciplinary teams support adoption.
2023-10-18Shopify ·
Shopify-Made Patterns in Our Rails Apps
Shopify describes patterns and infrastructure adaptations used to keep its large Rails monolith maintainable as its organization and platform scale. Its dev command-line tool standardizes project setup, dependencies, services, environment variables, integrations, and command aliases across hundreds of active projects. Shopify Core uses pods to partition shops across independent databases and extends the shard concept across provisioning, deployment, load balancers, caching, and servers, while splitting code into domains without separate databases. The post also details safeguards including short-term Active Record migrations consolidated periodically into structure.sql, automated tests, a pod-aware maintenance task system, and a translation platform that commits translated strings through Git. Finally, Monorail imposes versioned JSON event schemas between Kafka producers and consumers and supports review and automated scrubbing of PII fields.
2023-10-18Shopify ·
Shopify's Path to a Faster Trino Query Execution: Infrastructure
Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.
2023-10-18Shopify ·
High Availability by Offloading Work Into the Background
Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.
2023-10-18Shopify ·
Understanding GraphQL for Beginners–Part One
The article introduces GraphQL as an open-source data query and manipulation language for APIs, contrasting it with REST's fixed response shapes and multiple endpoint calls. GraphQL lets clients describe exactly which fields they need, returning that requested structure through a single endpoint rather than a cookie-cutter response. Using a Toronto Eagles example, the article explains how one GraphQL query can combine requests for a team's founding year, captain's name, and last championship, avoiding three separate RESTful trips whose latency can vary. It also uses burger and bank withdrawal analogies to explain over-fetching, under-fetching, flexible frontend iteration, and fewer endpoints, then points readers to a forthcoming Ruby on Rails implementation in part two.
2023-10-18Shopify ·
Understanding GraphQL for Beginners–Part Two
The second part of this beginner tutorial uses a food-focused Ruby on Rails application to show how GraphQL fields expose database data. It reviews the graphql directory, where mutations hold data-changing classes, types define returned structures, and food_app_schema.rb defines available queries. It distinguishes root fields, which are GraphQL entry points that define response structure, from object fields, which represent attributes, then demonstrates generating Food and Nutrition types and adding an all_food root field with a non-null list return type. Through GraphiQL at localhost:3000/graphiql, it runs selective queries, adds nutrition to food, and introduces an id argument for retrieving one record; requests pass through graphql_controller’s execute method and return JSON responses.
2023-10-18Shopify ·
Let’s Encrypt x Shopify: Securing the Web 4.5 Million Domains at a Time
Shopify describes how its SSL team moved merchants’ stores to HTTPS and ultimately secured more than 4.5 million domains with Let’s Encrypt. The effort began against certificate providers whose APIs were not designed for fully automated provisioning: errors often arrived as human-readable messages, and imposed throttling could have stretched initial issuance to about 100 days. Shopify tested the ACME protocol, valued its standardized specification and open-source implementation, and first added Let’s Encrypt as a backup certificate authority to improve emergency key or certificate-chain rotation. The team rolled out that integration in a few hours, compared with months for earlier providers, then made Let’s Encrypt its primary authority after finding it reliable. The account presents this change as evidence that ACME can make certificate automation more responsive and allow certificate authorities to be added without redesigning Shopify’s infrastructure.
2023-10-18Shopify ·
Rate Limiting GraphQL APIs by Calculating Query Complexity
Shopify’s GraphQL Admin API uses calculated query costs to address two limitations of request-based rate limiting: identical credits for responses of different sizes and equal treatment of reads and side-effecting writes. The server statically analyzes each query, assigning one point to objects and interfaces or unions, zero to scalars and enums, two plus the requested object count to connections, and 10 to mutations. Clients receive 50 points per second up to a 1,000-point limit, while responses expose requested and actual costs; when fewer connection records are returned, the difference is refunded. The post reports a linear correlation between calculated query complexity and execution time, giving Shopify more predictable load for infrastructure scaling and a way to identify performance outliers.
2023-10-18Shopify ·
10 Lessons Learned From Online Experiments
After completing ten online experiments over the past year, the author describes recurring statistical and implementation traps that can distort experiment results. The discussion stresses choosing independent randomization units, examining whether the analysis unit matches the randomization unit, and using simulations to test suspected false-positive effects before changing an experiment plan. It also shows how request-level traffic analysis and close collaboration with engineering can avoid unnecessary implementation work, while generous logging, A/A tests, and investigation of sample ratio mismatches can expose redirect failures or qualification bugs. The author reports that switching one experiment’s analysis unit from session to user changed its result from negative to positive, and argues for monitoring results without stopping early, investigating anomalies, and slicing averages across important user segments.
2023-10-18Shopify ·
Querying Strategies for GraphQL Clients
GraphQL clients can face slower, larger queries and difficult roll-outs as product screens gain features and data. The Orders & Fulfillments teams examined these issues while targeting a sub-one-second mobile page load on a reliable network. Starting with a product-list query that loads 100 products, the post recommends pagination with page-size/index controls and a hasNextPage field, plus performance tripwires to track loading time. It describes @include and @skip directives, runtime query construction, and chained queries for feature flags or parameters, while warning that chained requests add latency. For growing screens, it separates unrelated filters, permissions, and banners into parallel queries, enabling independent scaling, partial rendering, and less redundant pagination work, while cautioning that server load and partial-data state require monitoring and possible refactoring.
2023-10-18