Loading…
Shopify
Global commerce platform powering millions of businesses with essential infrastructure and innovative engineering solutions.
328 postsWebsite
Latest articles
Shopify ·
Shopify Embraces Rust for Systems Programming
Shopify has adopted Rust as its official systems programming language while retaining Ruby for server-side business logic, and has joined the Rust Foundation. The change addresses growing systems-programming needs including high-performance network servers, Ruby native extensions, and WebAssembly, where Shopify seeks speed, productivity, safety, and community-driven open source. Shopify cites Rust’s predictable native-code performance, fine-grained memory control, compile-time safety, support for concurrency, and interoperability with C and Ruby through tools and crates such as bindgen, rb-sys, and magnus. The company says Rust performed strongly in its safety evaluation and that its initial projects exposed more errors at compile time, while acknowledging that performance still depends on design and measurement and that internal education, tooling, and community participation remain unfinished work.
2023-10-18Shopify ·
3 (More) Tips for Optimizing Apache Flink Applications
Shopify presents three additional practices for optimizing large stateful Apache Flink applications, focusing on parallelism, sink capacity, and combining heterogeneous sources. It recommends starting with execution-environment parallelism, then matching task-manager capacity to the highest parallelism value; for example, parallelism 100 requires 25 task managers with four slots each. Sink bottlenecks can propagate backpressure upstream, so batching writes and correcting data skew—including with key bucketing when a low-cardinality or uneven key is unavoidable—can improve distribution, though bucketed results must later be combined. For archived Kafka topics, HybridSource exposes cloud-storage history and the live Kafka topic as one ordered logical DataStream, automatically switching after the archive is exhausted and using object-storage partitions to accelerate historical backfills.
2023-10-18Shopify ·
Three Essential Remote Work Practices for Engineering Teams
After Shopify became a fully remote-first company, its engineering organization refined how teams communicate, collaborate, and work across time zones. The recommended team structure concentrates engineers in related time-zone regions to maximize synchronous availability for discussion, pair programming, mentorship, and questions, while relying on asynchronous writing and documentation between teams. Teams are also asked to define communication norms that specify which medium to use, expected response times, and whether contact is appropriate after hours; documenting these rules reduces ambiguity and protects focus. Individuals should intentionally shape routines and work-life boundaries around their own energy, productivity, health, and responsibilities, then integrate those preferences with team norms. Together, these practices support flexible remote work by combining synchronous collaboration within teams, asynchronous coordination across teams, and greater control over the working day.
2023-10-18Shopify ·
Planning in Bets: Risk Mitigation at Scale
Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.
2023-10-18Shopify ·
Using Server Sent Events to Simplify Real-time Streaming at Scale
Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.
2023-10-18Shopify ·
How to Export Datadog Metrics for Exploration in Jupyter Notebooks
Datadog dashboards can constrain metric analysis because they offer a limited set of visualizations and lack tooling for complex work such as statistical modeling. They also aggregate data over wider time ranges: the examples contrast one-second metrics across 15 minutes with two-hour intervals across 30 days, potentially hiding interesting events. The guide uses Datadog’s REST API and Python in Jupyter notebooks, requiring an API key, an APP key, and a metric query such as CPU utilization over time. It defines a time range, splits it into buckets whose width is controlled by time_delta, requests each window in a loop, appends the results, and converts them into a dataframe. The resulting data can be examined with tools such as seaborn; a KDE plot is used to inspect the distribution of system CPU utilization, with the exported data offering greater granularity than the dashboard example.
2023-10-18Shopify ·
Our Solution for Measuring React Native Rendering Times
Shopify needed a reliable way to measure React Native rendering times and verify that mobile apps remained as fast as native while adopting React Native. The open-source @shopify/react-native-performance library measures app startup, navigation, and screen re-render times by timing from native startup, a user action, or a UI event until a screen is fully rendered and interactive. It uses an invisible marker view with a native counterpart to capture end-to-end timing, including React Native-to-native bridge communication, and models incremental rendering as interactive or non-interactive render passes in a state machine. Reports can be sent through onReportPrepared to analytics tools such as Amplitude, where percentile TTI dashboards compare screens over time, expose bottlenecks, and help catch regressions; Shopify also uses screen-level Apdex internally.
2023-10-18Shopify ·
Implementing Server-Driven UI Architecture on the Shop App
The Shop Store team implemented a server-driven UI architecture for the Shop App’s Store Screen to personalize sections and layouts for different merchants. Previously, the client rendered a fixed layout, limiting customization, experiment timing, and the speed of fixes after weekly app releases. The new design uses template processing, orchestration, data loading, and a shop-server GraphQL layer, with ProductsSection and CollectionsSection types and configurable GridLayout or ShelfLayout options. On the client, ServerDrivenStoreScreen and StoreSectionContainer render the sections returned by the server, while default layouts allow older app versions to handle unfamiliar section or layout types. The architecture enables backend-controlled experiments and section changes without requiring an app release, while giving the product team a system intended to grow with the store experience.
2023-10-18Shopify ·
What We Learned from Open-Sourcing FlashList
FlashList is Shopify’s React Native list library, created after Shopify’s migration from native technology exposed poor list performance in React Native. The post explains how a four-person team prototyped a library that combined FlatList’s familiar API with RecyclerListView’s performance, while addressing blank spacing and low frame rates on some devices. It also describes a launch planned six months ahead around a landing page, documentation, blog storytelling, social media, conference exposure, and feedback from external users. The post says FlashList was used in production on some Shopify app screens, and the launch achieved 1,000 GitHub stars in less than 24 hours. The broader conclusion is that successful open-source projects require product thinking, user empathy, community building, and a long-term authority-building strategy in addition to code.
2023-10-18Shopify ·
Caching Without Marshal Part 2: The Path to MessagePack
Rails caching relied heavily on Marshal, but cached Ruby-specific objects could break after code changes when old payloads were loaded by code without the former classes. MessagePack provided a generic binary format whose core types exclude Ruby-specific Object and instance-variable encodings, while extension types allowed production-specific serializers such as the compact Date representation. During a roughly six-month migration, the team ran both formats, prefixed MessagePack payloads with a version byte, and used encoding failures to identify unsupported cache values. To scale beyond 128 extension types, they added an Object catchall based on as_pack and from_pack, plus Struct and T::Struct modules whose attribute digests detect refactors and turn stale data into cache misses. After logs showed that Marshal was no longer used, the core monolith moved exclusively to MessagePack, with the Paquito gem extracting much of the migration work.
2023-10-18Shopify ·
Caching Without Marshal Part 1: Marshal from the Inside Out
Rails caching commonly relies on Ruby’s Marshal serialization, which can encode complex objects but also embeds their class names in cache entries. Shopify describes an incident in which a beta-flag refactor changed classes while old and new code overlapped during deployment; old code then failed on cached instances containing unfamiliar class names and methods, despite passing CI. The article examines Marshal through Ruby’s marshal.c implementation, covering atomic and composite types, instance variables, object references, circularity via TYPE_LINK, and core-type subclasses via TYPE_UCLASS. It presents MessagePack as a more compact, stricter, and controllable alternative intended to make caching safer by default, while deferring the cache migration and implementation details to the series’ next part.
2023-10-18Shopify ·
Apollo Cache is Your Friend, If You Get To Know It
Shopify’s migration from Apollo GraphQL client 2 to client 3 prompted a closer examination of the Apollo InMemoryCache after past bugs were linked to misunderstanding or misuse. The cache stores an in-memory representation of queried data for the current browser session, and its lifecycle covers fetching, normalization, updating and merging, then garbage collection and eviction. Fetch policies determine whether data comes from the cache, the network, or both; under the default cache-first policy, incomplete data triggers a network request. Normalization breaks responses into objects, assigns cache identifiers usually from __typename and id or configured key fields, and stores them in a flattened structure. Automatic UI updates depend on matching identifiers and suitable query or mutation responses, while objects made unreachable by changed identifiers can remain until garbage collection removes them.
2023-10-18Shopify ·
Reducing BigQuery Costs: How We Fixed A $1 Million Query
During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.
2023-10-18Shopify ·
Mixing It Up: Remix Joins Shopify to Push the Web Forward
Shopify announces that the open-source web framework Remix and its team are joining the company to help developers deliver lightning-fast, resilient web experiences. Remix combines multi-page and single-page strengths through a full-stack React framework with data loading and code splitting handled without requiring front-end developers to manage those backend tasks. It also emphasizes progressive enhancement, with links, buttons, and forms working before JavaScript loads, and supports edge, service-worker, and Node runtimes. Shopify will use Remix across projects including Hydrogen; Hydrogen is moving from server components to Remix's data-loading pattern for faster performance and a simpler developer experience, with ~90% of developers' code expected to remain unchanged. Remix will remain independent and open-source while Shopify supports its roadmap and community.
2023-10-18Shopify ·
The Management Poles of Developer Infrastructure Teams
Developer infrastructure managers must balance three poles: management support, system and domain expertise, and organization-wide road maps, with the tension intensified by teams that often lack dedicated product managers and support critical systems for hundreds or thousands of users with six to eight developers. Domain-focused teams build deep knowledge needed for maintenance, migrations, incident investigation, on-call coverage, and developer-experience improvements, but their local road maps can diverge from the department’s highest-impact opportunities. The proposed responses are temporary individual assignments, having a whole team contribute to another team’s goals, and forming a cross-functional tiger team; each shifts trade-offs among focus, maintenance, expertise, motivation, and management oversight. A month-long tiger team built the first Spin proof of concept, which led to a dedicated team, but no structure eliminates the tensions, so choices must be judged according to context.
2023-10-18Shopify ·
A Software Engineer's Guide to Working Across Time Zones
Working across time zones can make live collaboration difficult when teammates are nine to 12 hours apart, as described by a developer based in Singapore. The proposed alternative is to treat much of the work as asynchronous and deliberately document progress, blockers, decisions, and context. Examples include taking turns across working hours, annotating non-obvious pull request changes, posting end-of-day summaries with links or investigation notes, and recording meetings with agendas, key decisions, and action items. During Shopify's Hack Days prototype work on Linkpop, a Singapore-based frontend developer handed off updates before sleeping while North American teammates extended backend functionality and responded to blockers. The post concludes that these practices become habits that support handoffs, reduce back-and-forth, preserve information for absent teammates and future reference, and can provide longer uninterrupted focus periods.
2023-10-18Shopify ·
Hubble: Our Tool for Encapsulating and Extending Security Tools
Shopify’s Trust team faced rising complexity in securing a distributed device fleet, including many tools, provisioning and patching work, varied networks, and the loss of in-person remediation. Hubble was built as a unified layer over mobile device management and other security systems, ingesting and standardizing their data while sending commands back with granular access controls and centralized auditing. IT staff use it for inventory, device management, and security, while employees can view device health and compliance, receive remediation guidance, and manage test-device or beta participation. The platform also serves as a standardized source for automation, and the post presents encapsulation and cross-team investment as ways to reduce administrative overhead, support proactive security, and limit risk.
2023-10-18Shopify ·
How to Structure Your Data Team for Maximum Influence
Embedded data science leaders may support multiple product and business areas, creating competing work streams, specializations, and stakeholder relationships. The post compares swim lanes and stochastic process assignments against principles of efficiency, influence, stakeholder clarity, stability, growth, and flexibility. It recommends a hybrid “diamond defense” in which scientists are loosely assigned to zones for specialization while a bullpen, including the manager and an additional scientist, supplies reserve capacity. Resources can be redirected to overloaded areas, and team members can rotate into vacancies or cover leave. The framework is presented as combining clear ownership and subject-matter depth with the flexibility to handle change, though the post warns that excessive rotation can create volatility and turnover.
2023-10-18Shopify ·
Finding Relationships Between Ruby’s Top 100 Packages and Their Dependencies
RubyGems’ phased MFA rollout required owners of gems with at least 180 million downloads to use MFA, but raised a supply-chain question: could a popular gem depend on a less-downloaded gem that remained an account-takeover target? The investigation loaded rubygems.org data dumps to identify 112 gems above the threshold, queried the API for direct dependencies, and used Bundler with Gemfile.lock files to resolve transitive dependencies. It found 13 big gems with small direct dependencies and 24 with small dependencies overall, attributing the mismatch mainly to newer dependencies and gems shipped with Ruby, such as racc and rexml. Graph visualizations, breadth-first traversal, and a custom depth-first search then mapped dependency structure and paths; the post leaves open whether to do nothing or enforce MFA early for the 24 technically insecure gems.
2023-10-18Shopify ·
On the Importance of Pull Request Discipline
The article argues that pull request (PR) discipline matters beyond code correctness because implementation and maintenance both benefit from changes that are clearly structured and documented. It describes a PR as a proposal whose title, summary, commits, comments, reviewers, labels, projects, and linked issues should intentionally communicate scope, design reasoning, low-level details, ownership, and historical connections. Its practical guidance focuses on precise titles, useful summaries, logically ordered commits, deliberate remote-branch changes, defined PR boundaries, and moving asides into follow-up PRs. It also recommends smaller, well-ordered PRs, permalink-based code references, distinctive branch-name prefixes, and learning Git commands including reflog, cherry-pick, and rebase. The article treats discipline as judgment rather than rigid law, allowing emergency exceptions while encouraging post-resolution documentation.
2023-10-18