---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Media at Scale: Callbacks vs pipelines](https://yomu.fyi/post/media-at-scale-callbacks-vs-pipelines.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 9, 2020
- Type: Problem & solution

Shopify’s Rails monolith needed to add native product videos and 3D models while continuing to support an image infrastructure containing more than 7 billion images. The design question was whether media creation should rely on Active Record callbacks or an explicit pipeline, with transactions protecting interdependent database writes. Callbacks were quick for simple cases, but adding media-specific behavior such as video thumbnails spread conditionals across models and made lifecycle ordering difficult to follow and debug. The pipeline design routes requests through a single Product Create Media Service and media handlers, each organized into before\_transaction, during\_transaction, and after\_transaction steps, while confining logic to one media type. This structure separates concerns, controls creation order, limits model access, and makes implementation details easier to understand and maintain as the feature grows.


### [How We Added Client Rate Throttling to the Platform API Gem](https://yomu.fyi/post/how-we-added-client-rate-throttling-to-the-platform-api-gem.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Richard Schneeman
- Published: Jul 8, 2020
- Type: Explainer

The platform-api gem needed client-side rate throttling for Heroku’s API, which uses GCRA and allows a bucket of up to 4,500 request tokens, replenished at roughly 75 per minute. The goal was to avoid 429 responses without creating retry traffic, while operating within the client library and without coordination across machines. The work combined tests with a simulated server, processes and threads, charting, and metrics for average retry rate, maximum sleep, request-count variance, and workload-clearing time. The winning strategy adjusted sleep downward using the remaining request count relative to the 4,500-request maximum; in simulation it achieved a 3.07% retry rate, 17.32-second maximum sleep, 78.44 standard-deviation request count, and 84.23 seconds to clear 4,500 requests. The change shipped behind a null strategy before becoming the default in version 3.0.0+, with feedback reporting no significant breaks or severe application dysfunction.


### [Physical vs Logical Backups in PostgreSQL](https://yomu.fyi/post/physical-vs-logical-backups-in-postgresql.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Jul 7, 2020
- Type: Explainer

PostgreSQL backups fall into two distinct categories: logical backups, which convert data into SQL command files, and physical backups, which copy the underlying file system storage. Logical backups generated with pg\_dump or pg\_dumpall allow single-database targeting and provide the only practical route for migrating across major Postgres versions with differing internal storage formats. Conversely, physical backups suit larger databases where long-running logical exports can degrade concurrent query performance and risk failure. Physical backups also integrate with Write Ahead Log files through tools like WAL-G to support Point in Time Recovery and minimize Recovery Point Objectives. Selecting between these two strategies depends on whether administrative simplicity, version portability, cluster size, or precise disaster recovery takes precedence.


### [The Reformer - Pushing the limits of language modeling](https://yomu.fyi/post/the-reformer-pushing-the-limits-of-language-modeling.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Jul 3, 2020
- Type: Explainer

Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.


### [A First Look at Reanimated 2](https://yomu.fyi/post/a-first-look-at-reanimated-2.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 2, 2020
- Type: Explainer

Reanimated 2 is presented as a new approach to building gestures and animations in React Native, replacing the declarative domain-specific language of Reanimated 1 with animation worklets. Worklets are JavaScript functions declared with the “worklet” directive that execute independently on the UI thread, can receive parameters, access JavaScript-thread constants, call other worklets synchronously, and invoke JavaScript-thread functions asynchronously. The API introduces five hooks for creating shared and derived values, handling gesture events, and assigning animated styles or properties; a drag example combines shared values, gesture context, and velocity-based decay. The article also describes a liquid-swipe example using bezier interpolation, gesture handling, and physics-based animation, and reports that the new design lowers the entry barrier, supports cases previously requiring the bridge, and improves initialization performance.


### [Supabase Alpha June 2020](https://yomu.fyi/post/supabase-alpha-june-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Jul 1, 2020
- Type: Announcement

Four months into development, Supabase released its June 2020 alpha update focused on simplifying PostgreSQL management and eliminating regional performance bottlenecks. The release introduces a spreadsheet-like interface for drilling down into relational tables alongside automated detection and editing support for JSON and JSONB columns. To address latency issues previously caused by hosting all projects exclusively in Singapore, the platform now allows developers to select specific database regions. In line with its zero lock-in principle, Supabase exposed daily backups on the dashboard and enhanced the SQL editor with improved syntax highlighting. The team also expanded by hiring PostgREST maintainer Steve Chavez while continuing efforts toward future multi-region replication and WAL-G backup integration.


### [Run Apollo Server Close to Your Users](https://yomu.fyi/post/run-apollo-server-close-to-your-users.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 1, 2020
- Type: Tutorial

Run Apollo Server Close to Your Users presents an edge deployment pattern for GraphQL APIs. The post argues that, much like serving JavaScript and markup from a CDN, placing API servers and an application cache near users can improve application speed. Its approach combines Fly.io’s platform for running API servers close to users with a global Redis cache service, then uses Apollo Server’s cache capabilities and first-class Redis support. A demo GraphQL API built from the Open Library REST API accompanies the guide, with both a guide and source code available. The author frames the design through a newly adopted GraphQL perspective, emphasizing static typing and describing the resulting setup as CDN-like infrastructure for an API.


### [How We Built Our In-house Chat Platform for the Web](https://yomu.fyi/post/how-we-built-our-in-house-chat-platform-for-the-web.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Vasu Krishnamoorthy
- Published: Jun 29, 2020
- Type: Problem & solution

Grab extended its in-house chat platform to the web to support its internal Customer Support portal. Because the existing TCP gateway only supports unicast connections with one active connection per user, opening multiple browser tabs would repeatedly disconnect previous tabs. Rather than undertaking a complex migration to multicast connections on the server, the team adopted a hybrid client-side strategy using SharedWorker and BroadcastChannel APIs. The implementation uses a SharedWorker to maintain a single WebSocket connection per domain while a BroadcastChannel syncs events across all open tabs. A custom wrapper over the worker manages version transitions during deployments to avoid race conditions across tabs.


### [How CDNs Generate Certificates](https://yomu.fyi/post/how-cdns-generate-certificates.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 25, 2020
- Type: Explainer

Fly describes how its CDN automatically generates TLS certificates for customer hostnames with Let’s Encrypt and ACME, focusing on why the older tls-sni-01 validation method was unsafe on shared hosting. ACME authenticates an account with an ECDSA/EdDSA key, creates an order, and verifies domain control through challenges; Fly uses tls-alpn-01, which avoids requiring DNS access or port 80. For validation, Fly requests a challenge, creates a self-signed certificate containing Let’s Encrypt’s token, stores it in distributed certificate storage, and signals the CA to connect with a ClientHello selecting acme-tls/1 via ALPN. The Rust proxy recognizes that ALPN case, retrieves the challenge certificate, and serves it to Let’s Encrypt, which completes the challenge and permits certificate issuance. Fly assigns each application a unique routable IPv6 address and requires CNAME resolution to it for ALPN validation; wildcard certificates instead use DNS challenges.


### [Building a GraphQL API in JavaScript](https://yomu.fyi/post/building-a-graphql-api-in-javascript.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Owen Ou
- Published: Jun 24, 2020
- Type: Tutorial

GraphQL lets API clients specify the fields they need instead of accepting a server-defined response, reducing unnecessary payload and potentially combining data from multiple stores behind one endpoint. The tutorial builds a JavaScript/Node.js implementation by loading Schema Definition Language (SDL) with graphql-tools, defining Query, User, and Address types, and connecting fields to resolver functions. Resolvers receive field arguments and can use context for authorization, illustrated by restricting address access to the current user. It also recommends keeping business logic separate from resolvers and notes that GraphQL does not eliminate database performance, caching, logging, monitoring, or versioning concerns. The discussion presents GraphQL as useful for evolving, front-end-heavy applications, while warning that its single HTTP status code and schema versioning limitations can complicate error handling and breaking changes.


### [Writing Better, Type-safe Code with Sorbet](https://yomu.fyi/post/writing-better-type-safe-code-with-sorbet.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 24, 2020
- Type: Explainer

Ruby repositories can hide unsafe method calls and ambiguous data shapes when types are inferred from variable names alone. Sorbet static type checking addresses these risks with method signatures, typed structs, enums, and interfaces. It can flag a potentially nil return before a chained call, distinguish database output fields from input fields, and enforce contracts for synchronous and asynchronous indexers in a dependency-injected hexagonal structure. The article also describes Sorbet’s gradual typing, including five strictness levels and namespace-by-namespace adoption using the minimum typed level of true. The stated conclusion is that enforced type safety catches errors unit tests may miss and helps prevent unsafe code from reaching production.


### [Flyctl meets JSON](https://yomu.fyi/post/flyctl-meets-json.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 23, 2020
- Type: Announcement

Flyctl adds a --json, or -j, flag that presents command output as JSON to support automation around Fly applications. For GraphQL-backed commands it emits the API data, while logs become structured JSON and deployment progress arrives as a stream of Source/Status/Message objects. The post demonstrates piping flyctl list apps --json through jq to filter fields, convert hostnames and statuses to CSV, and select undeployed apps whose names feed xargs flyctl info --json -a; jq -s then gathers the resulting objects into an array. Because deployment output currently mixes JSON with non-JSON messages, the post recommends using --detach and polling flyctl status or running flyctl monitor instead of relying on the JSON stream for deployment monitoring.


### [From Project to Productionized with Python](https://yomu.fyi/post/from-project-to-productionized-with-python.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ed Morley
- Published: Jun 22, 2020
- Type: Tutorial

The post presents a hands-on tutorial for Python developers moving a local Django application into production on Heroku. It frames the work around 12-factor practices and covers repository hygiene, modular settings, continuous delivery stages, static-asset middleware, environment configuration, database provisioning, deployment, scaling, and troubleshooting. The configuration changes include excluding virtual environments, caches, SQLite data, and generated static files; moving settings.py to settings/base.py; updating DJANGO\_SETTINGS\_MODULE; adding Whitenoise; and supplying SECRET\_KEY and a Heroku Postgres add-on. The deployment uses Git, the Heroku CLI, git push heroku main, and heroku ps:scale web=1, with heroku logs --tail and heroku run bash for diagnosis. The walkthrough concludes with the app deployed and accessible through heroku open.


### [Shopify's Data Science & Engineering Foundations](https://yomu.fyi/post/shopify-s-data-science-engineering-foundations.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 18, 2020
- Type: Explainer

Shopify’s Data Science & Engineering team describes the foundations it uses to provide daily insights to internal teams, merchants, and partners across a rapidly changing commerce ecosystem. Its data warehouse follows a shared dimensional-modelling philosophy, with modelled data built on Spark in a single GitHub repository and made broadly queryable through Presto, while ETL jobs are unit tested and prevented from failing silently. Centralized dashboards, reproducible vetted data points, peer review, product-aligned sub-teams, accessible communication, and cross-team collaboration extend those foundations into analysis and decision-making. Together, these practices let team members reuse existing work, produce trusted and understandable metrics, address the specialist gap through collaboration, and derive new metrics with fewer than 50 lines of SQL.


### [Spark Joy by Running Fewer Tests](https://yomu.fyi/post/spark-joy-by-running-fewer-tests.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 11, 2020
- Type: Problem & solution

Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests.


### [Fly Changelog for 9th June](https://yomu.fyi/post/fly-changelog-for-9th-june.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 9, 2020
- Type: Announcement

The 9th June Fly Changelog describes updates to flyctl, Fly platform tooling, and related web sites, centered on installation, application management, monitoring, and Deno builds. New installation instructions support macOS and Linux through a shell script, while a PowerShell command provides native Windows installation and flyctl now installs under $HOME/.fly/bin or configured $FLYCTL\_INSTALL/bin. New commands let users restart applications, list accessible apps and organizations with text, organization, and status filters, or resume deployment monitoring with flyctl monitor. The open command accepts a URL path, secret updates automatically start monitoring for redeployment, and upgrade messages move to stderr for CI tooling. The Deno Buildpack now reads configuration from a required .config file, which also allows selecting the Deno version; an empty file is valid, while its absence produces an explanatory error.


### [Understanding Programs Using Graphs](https://yomu.fyi/post/understanding-programs-using-graphs.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 2, 2020
- Type: Explainer

TruffleRuby uses a sea-of-nodes graph as an intermediate representation after parsing, allowing its just-in-time compiler to optimize Ruby programs and translate them to machine code. The explanation contrasts this graph with an abstract syntax tree, then shows how control flow, data flow, side effects, pure computations, loops, and phi nodes are represented through boxes and arrows. A three-way conditional demonstrates global value numbering: a repeated multiplication becomes one movable computation that can float across branches without changing program behavior. A loop example shows backward control flow and repeated functional computation, while the discussion weighs graph-based optimization against poor compactness and readability at larger scales. Shopify is building graph-drawing and Ruby decompilation tools to inspect optimization at codebase scale.


### [Supabase Alpha May 2020](https://yomu.fyi/post/supabase-alpha-may-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Jun 1, 2020
- Type: Announcement

Supabase announced its May 2020 alpha development updates following acceptance into a fully remote Y Combinator cohort and unexpected attention from Hacker News. Operating under a ship early and often philosophy, the team continued deploying functional changes despite ongoing alpha bugs and platform instability. This milestone includes a revamped user interface configured to support incoming product capabilities alongside an initial, experimental Table View designed to rival tools like Airtable. Furthermore, Supabase rolled out realtime listeners that allow developers to subscribe to database changes comparable to Firebase functionality. The platform also added instant RESTful APIs and auto-updating documentation generated dynamically by introspecting modifications to the underlying Postgres schema.


### [Go Modules- A Guide for monorepos (Part 1)](https://yomu.fyi/post/go-modules-a-guide-for-monorepos-part-1.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: May 29, 2020
- Type: Problem & solution

Grab transitioned its large Go monorepo dependency management from Glide to Go modules while retaining an existing vendor directory structure. The team generated root go.mod configurations from glide.yaml and used go mod vendor without directly enabling module-mode builds. Incompatible nested sub-vendor paths were excluded by placing empty go.mod files, relying on the rule that modules cannot contain other modules. Post-migration maintenance revealed challenges with dependency inheritance and implicit go.mod updates during builds, which engineers investigated using go mod graph and digraph to trace dependency paths.


### [Fly Changelog for 29th May](https://yomu.fyi/post/fly-changelog-for-29th-may.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 29, 2020
- Type: Announcement

This Fly Changelog records significant changes to the Fly platform, tooling, and websites for the week of 29 May 2020. flyctl 0.0.124 adds apps pause and apps resume commands, which remove all running instances or start one running instance, respectively; regions set replaces the regions in a region pool list; and -v/--verbose provides more detailed command output. The release also improves flyctl apps list by displaying each application's status, while the Deno Buildpack supports unstable Deno builds when an .unstable directory is added to the application directory. Earlier in the month, flyctl 0.0.123 introduced open, which finds a deployed application's hostname and instructs a web browser to open it.


[Newer posts](https://yomu.fyi/posts/page/231.md) · [Older posts](https://yomu.fyi/posts/page/233.md)