---
title: "Performance"
description: "817 posts about Performance, summarised, each linking to the original."
---

# Performance
> 817 posts about Performance, summarised, each linking to the original.

## Articles

### [Reducing BigQuery Costs: How We Fixed A $1 Million Query](https://yomu.fyi/post/reducing-bigquery-costs-how-we-fixed-a-1-million-query.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 3, 2022
- Type: Problem & solution

During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.


### [Mixing It Up: Remix Joins Shopify to Push the Web Forward](https://yomu.fyi/post/mixing-it-up-remix-joins-shopify-to-push-the-web-forward.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 31, 2022
- Type: Announcement

Shopify announces that the open-source web framework Remix and its team are joining the company to help developers deliver lightning-fast, resilient web experiences. Remix combines multi-page and single-page strengths through a full-stack React framework with data loading and code splitting handled without requiring front-end developers to manage those backend tasks. It also emphasizes progressive enhancement, with links, buttons, and forms working before JavaScript loads, and supports edge, service-worker, and Node runtimes. Shopify will use Remix across projects including Hydrogen; Hydrogen is moving from server components to Remix's data-loading pattern for faster performance and a simpler developer experience, with ~90% of developers' code expected to remain unchanged. Remix will remain independent and open-source while Shopify supports its roadmap and community.


### [What is PostgreSQL commitfest and how to contribute](https://yomu.fyi/post/what-is-postgresql-commitfest-and-how-to-contribute.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: pavel
- Published: Oct 27, 2022
- Type: Tutorial

PostgreSQL Commitfest is the community's month-long review process for patches intended for merging into core. Contributors submit patches to pgsql-hackers, create a Git branch from master, consolidate changes, generate a versioned patch, and associate its mailing-list thread with an open Commitfest entry. The process expects authors to explain rationale, respond to review, rebase or revise patches, and review a comparable number of other patches. Submitted changes are checked by CFBot across architectures and operating systems, while statuses such as "Needs review," "Ready for committer," and "Waiting on author" track progress. Commitfests occur five times yearly, with the March event tied to feature-freeze for the October major release.


### [Postgres Full Text Search vs the rest](https://yomu.fyi/post/postgres-full-text-search-vs-the-rest.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: victor
- Published: Oct 14, 2022
- Type: Benchmark

The post compares PostgreSQL Full Text Search with dedicated search engines including MeiliSearch, OpenSearch, SQLite FTS, and Typesense, asking whether a separate system is justified for common application search. It presents a PostgreSQL setup using a generated stored column built with to\_tsvector('english', ...) and a GIN index, queried with websearch\_to\_tsquery. The benchmark loads a movie dataset and focuses on query speed after data ingestion, using comparable scripts and default configurations with limited tuning. PostgreSQL was close in performance to several alternatives; in-memory SQLite was fastest, while Typesense and MeiliSearch handled misspellings correctly and OpenSearch ingested data quickly. The conclusion is that PostgreSQL can provide good-enough search quality and speed for basic cases without adding another system, although dedicated engines offer capabilities PostgreSQL does not.


### [Postgres WASM by Snaplet and Supabase](https://yomu.fyi/post/postgres-wasm-by-snaplet-and-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: burggraf
- Published: Oct 3, 2022
- Type: Explainer

Supabase and Snaplet open-sourced postgres-wasm, a PostgreSQL server that runs inside a browser and supports browser persistence, pg\_dump restoration, and logical replication with a remote database. Rather than compiling PostgreSQL directly to WebAssembly, the project packages a stripped-down Linux image with PostgreSQL in a Buildroot-based virtual machine, then runs it through v86 in the browser. PostgreSQL 14 startup failures were resolved by disabling v86 JIT compilation, while image optimization used a compressed 9P filesystem, asynchronous HTTPS loading, and page\_poison=on to produce an initial state of about 12MB. Websocket proxying provides outbound connectivity and a forked proxy with nginx tunnels incoming connections to PostgreSQL. The project remains experimental because the download is currently about 30MB, but the post identifies offline data, testing, development environments, snapshots, and support as potential uses.


### [Lessons From Building iOS Widgets](https://yomu.fyi/post/lessons-from-building-ios-widgets.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 22, 2022
- Type: Explainer

Shopify’s widget team describes a third-generation redesign for iOS and Android after the original iOS widgets offered only two metrics and consumed substantial space. The widgets target merchants who need fresh shop analytics, using native SwiftUI on iOS because WidgetKit extensions have memory constraints, serialize SwiftUI view hierarchies to disk, and lacked official React Native support. Independent widget-side fetching uses shared session-management code and tokens; repeated update callbacks caused duplicate network requests and token-refresh races, so the team introduced a short-lived, widget-specific cache. Replacing existing widgets also exposed migration limits: changing widget kind identifiers could blank or remove installed widgets, leading Shopify to deprecate old widgets while releasing new ones and guide merchants through the transition. The team planned further customization and integration with iOS 16 features, including watch and lock-screen complications and live activities.


### [Lessons From Building Android Widgets](https://yomu.fyi/post/lessons-from-building-android-widgets.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 22, 2022
- Type: Explainer

Shopify revisited its Android widgets alongside an iOS widget effort, aiming to give merchants faster access to frequently changing business metrics. The team chose native Android development because React Native lacked official RemoteViews support, while widget configuration data was persisted with shared preferences and used to build requests. Because Android could invoke onUpdate() multiple times per cycle, they added a one-minute, short-lived cache keyed by widgetID and mindful of configuration such as locale. This reduced redundant network calls, system load, and incorrect analytics, while the widget rollout also exposed migration limitations between old fixed-size widgets and new space-responsive ones, leading Shopify to deprecate the old widgets rather than transform them.


### [Leveraging Go Worker Pools to Scale Server-side Data Sharing](https://yomu.fyi/post/leveraging-go-worker-pools-to-scale-server-side-data-sharing.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 9, 2022
- Type: Problem & solution

Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.


### [Choosing a Postgres Primary Key](https://yomu.fyi/post/choosing-a-postgres-primary-key.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: victor
- Published: Sep 8, 2022
- Type: Explainer

Choosing a Postgres primary key involves balancing simplicity, storage, sortability, standards compliance, information leakage, and collision avoidance. The discussion starts with natural and surrogate keys, then compares integer and biginteger identity columns with serial sequences, noting that sequences can have gaps and exposed numeric IDs may reveal information. It then examines pseudorandom and secure random numeric IDs before turning to UUIDs, including UUIDv4 and time-based alternatives, whose generation methods and index behavior differ. Postgres integration, size queries, and the pg\_idkit library provide practical ways to generate and evaluate identifiers; the suggested rules of thumb favor integers for simplicity, UUIDv4 for collision avoidance, xid for time-based sort friendliness, and UUIDv6 or UUIDv7 for a more standards-oriented choice.


### [How the SQLite Virtual Machine Works](https://yomu.fyi/post/how-the-sqlite-virtual-machine-works.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Sep 7, 2022
- Type: Explainer

SQLite processes SQL through a parse-optimize-execute pipeline that turns query text into executable virtual-machine instructions. It tokenizes characters, parses them into an AST such as a Select struct, and uses table statistics to choose between a full table scan and index-based access. The source describes sqlite\_stat1's coarse index and duplicate-value counts alongside sqlite\_stat4's distributed samples, then traces bytecode that opens the persons table, compares favorite\_color with "blue", and returns matching columns. Registers hold intermediate values, cursors iterate through tables, and sqlite3\_step() resumes execution after each returned row; the VM has 186 opcodes as of writing, while bind parameters allow prepared statements to skip repeated parsing and optimization.


### [RailsConf 2022: 10 Shopify Tech Talks You Might Have Missed](https://yomu.fyi/post/railsconf-2022-10-shopify-tech-talks-you-might-have-missed.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 30, 2022
- Type: Announcement

Ruby Central has posted videos from RailsConf talks and workshops online, following the 2022 in-person conference where several Shopify Ruby on Rails team members attended and around a dozen gave talks. The collection features ten Shopify sessions on Rails’s state, Ruby’s YARV virtual machine, cache serialization, algorithmic complexity, historical Ruby gems, supply-chain attacks, team trust, GraphQL delivery, and regular-expression denial-of-service attacks. Several talks use concrete technical examples: MessagePack replaced Marshal in the core monolith’s cache, OpenTelemetry is used to diagnose performance issues, and one tutorial demonstrates GraphQL delivery through WebSockets, Webhooks, WebAssembly code, and Bulk Operations. The sessions also connect implementation details with broader engineering practices, including investment in open-source teams, playful problem solving, pair programming, Rubocop, compilers, and examining older code to inform current decisions.


### [Shopify Data’s Guide To Opportunity Sizing](https://yomu.fyi/post/shopify-data-s-guide-to-opportunity-sizing.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 25, 2022
- Type: Explainer

Opportunity sizing gives businesses a structured way to estimate an initiative’s potential impact and compare it with the cost of pursuing alternatives. The framework recommends expressing an estimate as an expected change in a metric, with a range, timeframe, and documented assumptions; an annualized view makes initiatives easier to compare. Directional t-shirt sizing uses rough benchmarks for early ideation and sanity checks, while bottom-up sizing uses observed performance from a comparable product or system, such as email click-through rates, to produce a more accurate estimate. Top-down sizing is intended for new initiatives without comparable systems and starts with broader market or audience information before narrowing the estimate. Shopify recommends connecting initiative-level changes to wider business goals, testing estimates against actual results after launch, and refining the process over time.


### [How We Built Oxygen: Hydrogen’s Counterpart for Hosting Custom Storefronts](https://yomu.fyi/post/how-we-built-oxygen-hydrogen-s-counterpart-for-hosting-custom-storefro.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 23, 2022
- Type: Explainer

Oxygen is Shopify’s hosting counterpart for Hydrogen custom storefronts, designed to provide globally available, fast storefronts with secure Shopify integration without additional third-party hosting costs. Its architecture combines GitHub and GitHub Actions with Cloudflare Workers and edge distribution, Shopify CDN, Admin API, Identity and Access Management, and Oxygen-scoped developer tooling and observability. Requests pass through a Gateway Worker, which validates authorization for the shop and storefront version before using Cloudflare’s Dynamic Dispatch API to route traffic to a Storefront Worker running Hydrogen code. Rather than build proprietary infrastructure, Shopify reused Cloudflare’s worker platform to reduce upfront complexity, scale, and security risks, while its observability stack uses Grafana, Cortex, Loki, and Tempo. Oxygen is used in production by Allbirds, Shopify Supply, Shopify Hardware, and Denim Tears, and is generally available to Plus merchants.


### [PostgREST v10: EXPLAIN and Improved Relationship Detection](https://yomu.fyi/post/postgrest-v10-explain-and-improved-relationship-detection.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: steve\_chavez
- Published: Aug 19, 2022
- Type: Announcement

PostgREST v10 adds HTTP and supabase-js access to PostgreSQL execution plans for reads, writes, and RPC requests, with the feature restricted to development-oriented use through db-plan-enabled. Plans can be returned as text or JSON, analyzed to show execution time and RLS filtering, and queried in verbose form for the pg\_stat\_statements query identifier and related settings. The release also improves relationship detection: unique or foreign-key primary keys produce one-to-one JSON objects, while computed relationships let SQL functions define efficient links for views and materialized views. Many-to-many detection now requires a join table’s primary key to include both foreign-key columns, a breaking change; the post says PostgREST 10 was not yet on the Supabase Platform and was planned for release there over the following month.


### [Realtime: Multiplayer Edition](https://yomu.fyi/post/realtime-multiplayer-edition.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: wenbo,stas
- Published: Aug 18, 2022
- Type: Announcement

Supabase announces the general availability of Realtime’s multiplayer features, Broadcast and Presence, alongside a new channel-based architecture and supabase-js support. Realtime originally used Postgres logical replication, rather than LISTEN/NOTIFY, to avoid the latter’s 8,000-byte payload limit and the latency of querying each change from the database. In the new design, Channels act as rooms that support ephemeral publisher/subscriber Broadcast messages, shared Presence state, and Postgres change events; Presence uses Phoenix Tracker, a delta-based CRDT, for eventually consistent synchronization and removes disconnected users automatically. Realtime now runs as a distributed Elixir and Phoenix cluster on Fly, with clients connecting to the nearest node and nodes communicating across a backbone network to improve connection speed, stability, and latency. Free-plan active projects and projects created from August 18 have access, while other projects are scheduled for migration over the following weeks.


### [How We Enable Two-Day Delivery in the Shopify Fulfillment Network](https://yomu.fyi/post/how-we-enable-two-day-delivery-in-the-shopify-fulfillment-network.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 18, 2022
- Type: Explainer

Shopify Fulfillment Network (SFN) enables merchants to offer two-day delivery across the United States by balancing inventory across Shopify-certified warehouses and choosing suitable carrier services. Shopify’s platform adds GraphQL APIs that let any third-party logistics provider upload pre-computed delivery dates, while storefront and checkout surfaces display them to buyers. SFN calculates schedules using inventory proximity, product weight, dimensions and hazard constraints, warehouse calendars and capacity, carrier pickups, closures, capacity limits, and time-in-transit data. It groups SKUs with shared inventory locations and characteristics into reusable schedules, maps each SKU to one, and supports direct delivery-date lookups in under 100 milliseconds; the selected date is returned to SFN for fulfillment routing.


### [Instant Performance Upgrade: From FlatList to FlashList](https://yomu.fyi/post/instant-performance-upgrade-from-flatlist-to-flashlist.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 17, 2022
- Type: Announcement

Shopify’s React Native Foundations team created FlashList as a faster, easier alternative to React Native’s FlatList, targeting frame drops, blank cells, and high memory use across a wide range of devices. FlatList creates new views during updates, while FlashList recycles already-allocated cells and moves some layout operations to native, using a 250-pixel default buffer to reduce memory footprint and load times. Its API is nearly the same as FlatList, so an existing list can be swapped by changing the component name and optionally adding estimatedItemSize, with features such as type-specific recycling, flexible column spans, and FPS and blank-cell metrics. Extensive Android testing found performance of 60 FPS or greater even on low-end devices, and Shopify Mobile uses FlashList as its default while Shop moved search to it.


### [When is JIT Faster Than A Compiler?](https://yomu.fyi/post/when-is-jit-faster-than-a-compiler.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 11, 2022
- Type: Explainer

JIT compilation can outperform ahead-of-time compilation or interpretation for highly dynamic languages such as Ruby by optimizing runtime behavior under assumptions that static approaches cannot safely make. Ruby permits redefining operators, equality, iteration, hashing, and other behavior at any time, even from a background thread, forcing an interpreter to check whether definitions changed and limiting AOT code to conservative behavior. YJIT addresses this by compiling frequently used methods with assumptions about current definitions, then invalidating and discarding compiled versions when those assumptions become false; later calls can run interpreted code until a new version is compiled. The post presents this speculative optimization and de-optimization model as the reason a JIT can be faster, while noting that YJIT had known bugs in Ruby 3.1 and was expected to be closer to production-ready for some uses in Ruby 3.2.


### [Mastering React’s Stable Values](https://yomu.fyi/post/mastering-react-s-stable-values.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 2, 2022
- Type: Explainer

React stable values are values, often returned by hooks, that retain the same value across renders, and the post explains why their identity matters in effects, callbacks, memoization, and component rendering. useState and useReducer update functions and useRef objects are stable, while array literals, object literals, and anonymous functions are recreated on each render; useCallback and useMemo can produce stable values under appropriate dependencies. Unstable dependencies prevent useEffect behavior from remaining limited to intended changes and stop useCallback and useMemo cached results from being reused, while an unstable callback passed to useFocusEffect causes repeated unsubscribe and resubscribe cycles. The post also contrasts invoking an inline helper with rendering it as a component, explains why the latter can rebuild a React Native subtree on every render, and recommends defining components outside other components and using React.memo or callbacks judiciously.


### [10 Tips for Building Resilient Payment Systems](https://yomu.fyi/post/10-tips-for-building-resilient-payment-systems.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 28, 2022
- Type: Explainer

The post distills lessons from five years working on Shopify’s payment infrastructure into guidance for building systems that tolerate failure and large-scale traffic. It frames the advice partly around onboarding developers who were new to payments or Shopify’s scale, and notes that load testing for BFCM 2021 began in July with globally distributed load generation. It recommends low timeouts across HTTP clients and data stores, circuit breakers such as Semian, and capacity reasoning with Little’s Law, rate limiting, and load shedding. Monitoring should cover latency, traffic, errors, and saturation, while structured, machine-readable logs make centralized records searchable; load tests use benchmark gateways that mimic production partner latency and capacity. For incidents, Shopify uses defined roles, Slack-based coordination, and retrospectives that turn incorrect assumptions into safeguards and training material.


[Newer posts](https://yomu.fyi/topic/performance/page/32.md) · [Older posts](https://yomu.fyi/topic/performance/page/34.md)