Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,535 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
On the Importance of Pull Request Discipline
The article argues that pull request (PR) discipline matters beyond code correctness because implementation and maintenance both benefit from changes that are clearly structured and documented. It describes a PR as a proposal whose title, summary, commits, comments, reviewers, labels, projects, and linked issues should intentionally communicate scope, design reasoning, low-level details, ownership, and historical connections. Its practical guidance focuses on precise titles, useful summaries, logically ordered commits, deliberate remote-branch changes, defined PR boundaries, and moving asides into follow-up PRs. It also recommends smaller, well-ordered PRs, permalink-based code references, distinctive branch-name prefixes, and learning Git commands including reflog, cherry-pick, and rebase. The article treats discipline as judgment rather than rigid law, allowing emergency exceptions while encouraging post-resolution documentation.
How to Write Code Without Having to Read It
The article argues that developers can minimize code reading when fixing bugs or changing features by focusing on information not already checked by tools. It recommends locating a user-visible entrypoint, then making a deliberately breaking edit and climbing the callstack through build errors rather than reading each caller. Strict types, tagged unions, exhaustive switches, linters, and unit tests expose incompatible inputs, missing cases, deprecated patterns, and affected consumers; pure functions separated from side-effect code keep tests simple and informative. After the tooling-driven passes, developers manually confirm behavior, revisiting the design when runtime issues remain and making future changes more "unread"-able.
Supabase Beta September 2022
Supabase’s September 2022 update centers on reducing technical debt and clearing its backlog after five Launch Weeks. The team ran three Kaizen weeks focused respectively on QA and testing, documentation, and issue backlog and automation, closing over 250 open issues and 50 pull requests while pausing feature development. Product releases included Auth UI, a pre-built React component for Supabase Auth with custom themes and extensible styles, plus an open-source Postgres WASM release with Snaplet for running Postgres in a browser through an embeddable Linux Virtual Machine. Security changes to Dashboard database interactions simplify permissions and reduce the security surface area; they were scheduled for automatic application in November or could be run earlier through the Dashboard. Other updates covered Edge Functions, documentation, migration tooling, and community projects.
The Engineering Story Behind Flex Comp
Shopify’s Flex Comp lets employees allocate total rewards among base salary, Restricted Stock Units (RSUs), and Stock Options, with allocation changes during windows and no one-year equity cliffs. Building the system required a prioritized backlog, cross-functional teams, early API contracts, and privacy controls around its code and data. For sensitive analytics events, the team encrypted entire Kafka payloads with RSA, using 4096-bit keys for ordinary events and 6144-bit keys for larger payloads, while later identifying hybrid encryption as a better design. On the frontend, typed hooks, mocked data, and artificial delays allowed business logic and interface work to proceed in parallel while contracts evolved. Version one shipped to Shopify’s employees, with documented technical and design debt and future decisions retained for subsequent iterations.
Safely Adding NOT NULL Columns to Your Database Tables
Shopify’s Database Migrations team investigated whether Large Hadron Migrator (LHM) can safely add NOT NULL columns to existing MySQL tables with minimal downtime. LHM creates a shadow table, installs triggers for INSERT, UPDATE, and DELETE operations, and copies records in batches before renaming the tables. The experiments varied whether the new column had a DEFAULT value or UNIQUE index and whether MySQL used strict or non-strict mode, evaluating trigger compatibility and record counts. A NOT NULL column with a DEFAULT was backward compatible, but adding a UNIQUE index could cause data loss; without a DEFAULT, strict mode broke compatibility while non-strict mode used implicit defaults, which may be undesirable.
Postgres WASM by Snaplet and Supabase
Supabase and Snaplet open-sourced postgres-wasm, a PostgreSQL server that runs inside a browser and supports browser persistence, pg_dump restoration, and logical replication with a remote database. Rather than compiling PostgreSQL directly to WebAssembly, the project packages a stripped-down Linux image with PostgreSQL in a Buildroot-based virtual machine, then runs it through v86 in the browser. PostgreSQL 14 startup failures were resolved by disabling v86 JIT compilation, while image optimization used a compressed 9P filesystem, asynchronous HTTPS loading, and page_poison=on to produce an initial state of about 12MB. Websocket proxying provides outbound connectivity and a forked proxy with nginx tunnels incoming connections to PostgreSQL. The project remains experimental because the download is currently about 30MB, but the post identifies offline data, testing, development environments, snapshots, and support as potential uses.
A Software Engineer’s Guide to Giving Feedback that Actually Matters
The post presents feedback as a professional tool that can either support growth and collaboration or fuel ego-driven competition. It focuses first on pull request reviews, where engineers should prioritize critical code and logic over stylistic nitpicks, use linters or pre-commit hooks for uniform formatting, and frame comments to preserve a two-way educational dynamic. Recommended practices include discussing team review processes, explaining feedback goals, drafting messages before sending, reinforcing strengths, asking questions, separating behavior from identity, and making comments specific and timely. For broader workplace issues, the post advises asking whether feedback must be said, by the speaker, and immediately, then using considerate timing and SMART framing to improve the chance that it will be heard and produce change.
A Flexible Framework for Effective Pair Programming
Pair programming is presented as a mentoring and onboarding tool for early-career developers in Shopify’s Dev Degree program, where interns use it to build technical and communication skills. The author created a flexible framework to address progress tracking, session goals, a safe space, and communication, then refined it through an interactive RenderATL workshop with more than 30 attendees. The framework lets pairs choose roles, structure, agenda, environment, and communication practices; one described style assigns a driver to type and explain thought processes while a navigator reviews, observes, and suggests edge cases or refactorings, with frequent role switches. It also warns against hostile, passive-aggressive, and overly absolute language, and the pilot received positive intern feedback for supporting learning goals in a flexible format while continuing to evolve.
Lessons From Building iOS Widgets
Shopify’s widget team describes a third-generation redesign for iOS and Android after the original iOS widgets offered only two metrics and consumed substantial space. The widgets target merchants who need fresh shop analytics, using native SwiftUI on iOS because WidgetKit extensions have memory constraints, serialize SwiftUI view hierarchies to disk, and lacked official React Native support. Independent widget-side fetching uses shared session-management code and tokens; repeated update callbacks caused duplicate network requests and token-refresh races, so the team introduced a short-lived, widget-specific cache. Replacing existing widgets also exposed migration limits: changing widget kind identifiers could blank or remove installed widgets, leading Shopify to deprecate old widgets while releasing new ones and guide merchants through the transition. The team planned further customization and integration with iOS 16 features, including watch and lock-screen complications and live activities.
Lessons From Building Android Widgets
Shopify revisited its Android widgets alongside an iOS widget effort, aiming to give merchants faster access to frequently changing business metrics. The team chose native Android development because React Native lacked official RemoteViews support, while widget configuration data was persisted with shared preferences and used to build requests. Because Android could invoke onUpdate() multiple times per cycle, they added a one-minute, short-lived cache keyed by widgetID and mindful of configuration such as locale. This reduced redundant network calls, system load, and incorrect analytics, while the widget rollout also exposed migration limitations between old fixed-size widgets and new space-responsive ones, leading Shopify to deprecate the old widgets rather than transform them.
Introducing LiteFS
LiteFS is an open-source distributed file system for SQLite that addresses the durability, replication, availability, and latency limitations of deploying SQLite on a single server. LiteFS interposes a FUSE pass-through filesystem between an application and its database, identifies transactions from SQLite’s journal files, packages page changes into LTX files, and ships them to replicas without requiring application-code changes. A Consul-backed, time-based lease elects a primary writer, while local replicas serve reads; checksum state embedded in each LTX file helps detect out-of-order application and preserve byte-for-byte consistency. Because replication is asynchronous, the design trades some durability guarantees for performance and operational simplicity, while defensive checksum checks address potential split-brain failure modes. The project is described as beta-ready, with WAL-mode support, write forwarding, S3 replication, and encryption listed as planned features.
What is a Full Stack Data Scientist?
Shopify defines full stack data science as an ownership-oriented, end-to-end approach to solving business problems across the data science lifecycle. Rather than specializing in one stage, practitioners participate in discovery and analysis, acquisition, data modeling, pipeline development, and production delivery. The role combines business acumen, communication, Python and SQL programming, exploratory data analysis, data engineering, and machine learning, while treating machine learning as a tool rather than a default and starting simple before adding complexity. Shopify describes the model as generalist with T-shaped development: broad capability is paired with deeper expertise in a specific area, enabling individual strengths within teams to address complex challenges.
Scale-to-Zero Minecraft server with Terraform and Fly Machines
This tutorial presents a scale-to-zero Minecraft Java server built with Terraform and Fly Machines. It addresses the cost and resource burden of keeping a multiplayer server running when nobody is playing, while allowing it to restart when a player connects. The configuration provisions a Fly.io app, a Firecracker-based machine using Geoff Bourne’s itzg/minecraft-server Docker image, a 15GB persistent Fly Volume, an IPv4 address, and exposed TCP port 25565. Environment variables enable Autostop with estimated and initial timeouts of 120 seconds, while the machine receives four shared vCPUs and 8GB of RAM and mounts world data at /data. After terraform apply, the machine shuts down after the timeout and automatically starts again when Minecraft reconnects, though any TCP traffic can wake it and incur costs.
Managing React Form State Using the React-Form Library
Shopify’s tutorial shows how to manage a React product form with the open-source react-form library from its Quilt repository. It replaces component-level useState and earlier input and submit handlers with the useForm and useField hooks, which provide form and field state, event handlers, submission, reset, and dirty tracking. The example connects each TextField to its corresponding fields entry, using its value and onChange properties so typed input is captured and displayed. It then wires the hook’s submit function, resets the form after data is handled, and disables Save until either field changes. Finally, the notEmpty validation hook requires a title and displays a required-field error when a submission contains only a description.
Leveraging Go Worker Pools to Scale Server-side Data Sharing
Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.
Choosing a Postgres Primary Key
Choosing a Postgres primary key involves balancing simplicity, storage, sortability, standards compliance, information leakage, and collision avoidance. The discussion starts with natural and surrogate keys, then compares integer and biginteger identity columns with serial sequences, noting that sequences can have gaps and exposed numeric IDs may reveal information. It then examines pseudorandom and secure random numeric IDs before turning to UUIDs, including UUIDv4 and time-based alternatives, whose generation methods and index behavior differ. Postgres integration, size queries, and the pg_idkit library provide practical ways to generate and evaluate identifiers; the suggested rules of thumb favor integers for simplicity, UUIDv4 for collision avoidance, xid for time-based sort friendliness, and UUIDv6 or UUIDv7 for a more standards-oriented choice.
Automatic rule backtesting with large quantities of data
Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.
Supabase Beta August 2022
Supabase’s August 2022 Beta Update recaps Launch Week 5 releases, the Supa Hackathon, platform changes, migration resources, and community contributions. The release set includes Supabase CLI v1 with TypeScript type generation, the Management API Beta for programmatic project and organization control, and the supabase-js v2 release candidate with type support, new authentication methods, an async Auth overhaul, and Edge Functions improvements. Supabase also announced SOC2 compliance and made Realtime Presence and Broadcast available to all projects, alongside Vault, Auth UI, dashboard permissions, JSON schema validation, pg_graphql v0.4.0, and MFA early access. From September 11, HTTP API requests will redirect to HTTPS, and clients no longer need to send anon or service keys in the apiKey header; client-library users require no change.
How the SQLite Virtual Machine Works
SQLite processes SQL through a parse-optimize-execute pipeline that turns query text into executable virtual-machine instructions. It tokenizes characters, parses them into an AST such as a Select struct, and uses table statistics to choose between a full table scan and index-based access. The source describes sqlite_stat1's coarse index and duplicate-value counts alongside sqlite_stat4's distributed samples, then traces bytecode that opens the persons table, compares favorite_color with "blue", and returns matching columns. Registers hold intermediate values, cursors iterate through tables, and sqlite3_step() resumes execution after each returned row; the VM has 186 opcodes as of writing, while bind parameters allow prepared statements to skip repeated parsing and optimization.
Four Approaches to Debugging Server-side WebAssembly
Server-side WebAssembly offers Shopify Functions low cold-start latency and robust security, but debugging remains difficult because step-debugger support is less refined. The post compares four approaches: compiling to a native architecture, using LLDB or GDB with Wasmtime, using a dedicated WebAssembly debugger, and using browser developer tools. Native debugging generally provides the best variable rendering and a familiar workflow, though it may require stubs for external calls and cannot reproduce WebAssembly-specific behavior. Wasmtime keeps execution closer to the target environment, while dedicated and browser-based tools offer WebAssembly-specific capabilities but have missing features or lower-fidelity variable views. The recommended choice depends on whether the issue reproduces natively, and browser-based tooling may become more accessible as it improves.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.