---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Unpopular Opinions: Reddit vs Twitter Data](https://yomu.fyi/post/unpopular-opinions-reddit-vs-twitter-data.md)
- Company: [Brandwatch](https://yomu.fyi/company/brandwatch.md)
- Author: Gemma Joyce
- Published: Aug 12, 2020
- Type: Benchmark

An analysis of 1.6 million social-media mentions from January 1 to June 30, 2020 examines consumers’ “unpopular opinions” and compares Reddit with Twitter. Overall mentions rose 34% from March through June versus the preceding four months, while Reddit’s r/unpopularopinions posts increased 105% during that period. Across online sources, pop-culture characters generated 63,000 mentions, followed by TV shows at 55,000, fandoms at 45,000, and books at 44,000; Friends was criticised for aspects considered unacceptable today. Reddit contributed 958,000 mentions, including 19,000 about subreddits, while Twitter contributed 554,000, led by 15,000 movie and TV opinions. The platform breakdown also surfaces differences such as Reddit’s meta discussions and Twitter’s opinions about money, chocolate, and ice.


### [Go Modules- A Guide for monorepos (Part 2)](https://yomu.fyi/post/go-modules-a-guide-for-monorepos-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Aug 12, 2020
- Type: Problem & solution

Managing dependencies in a multi-module monorepo created developer friction at Grab due to unexpected changes from previous vendoring attempts and accidental imports. Because Go modules were not yet enabled directly for builds, the team implemented a continuous integration check that executes go mod vendor and rejects merge requests if any diffs exist in go.mod or the vendor directory. Adopting this CI check required configuring SSH deploy keys for private repositories, adding retry logic for network-related false positives, and standardizing on a single Go version to prevent checksum discrepancies. To streamline ongoing maintenance across hundreds of dependencies, the team developed an automated tool named AutoVend Bot. The bot runs go list -m -u all to detect updates and opens a scheduled batch of merge requests each day for human review.


### [Supabase Auth](https://yomu.fyi/post/supabase-auth.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Aug 5, 2020
- Type: Announcement

Following an early community launch on Hacker News, the open-source Firebase alternative Supabase faced heavy developer demand for a dedicated authentication and authorization solution. In response, the team evaluated existing open-source tools such as KeyCloak and Kratos before landing on an architecture combining a forked Netlify GoTrue server with PostgreSQL and PostgREST. The resulting Supabase Auth system provides client methods like signUp and signIn alongside authorization powered by PostgreSQL Row Level Security and policies. By leveraging SQL policies to enforce access rules directly in the database, developers avoid repetitive middleware filtering while maintaining granular access controls. Future roadmap plans include adding third-party OAuth providers, custom email templates, and simplified policy management interfaces for non-technical users.


### [An Introduction to DNS Traffic Management](https://yomu.fyi/post/an-introduction-to-dns-traffic-management.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 5, 2020
- Type: Explainer

DNS traffic management applies data-driven DNS responses to direct clients to different service endpoints, extending DNS’s basic role of mapping domain names to IP addresses. The post introduces DNS through its history from HOSTS.TXT, explains zones, records, nameservers, root servers, TTL, and negative TTL, then connects caching behavior to routing decisions. Low TTLs can make changes visible sooner, but repeated DNS resolution adds latency: with a one-second resolution time, a 15-second TTL could add 5,760 seconds per day, compared with 1,440 seconds at 60 seconds; resolver behavior can also make delays slightly longer. Use cases include active-passive failover, active-active traffic sharing, progressive green-blue deployments, and location-based routing, while Shopify’s system evolved into an automated, self-served platform serving more than 40 domains across 12 teams and over 100M requests per 24 hours.


### [Using DNS Traffic Management to Add Resiliency to Shopify’s Services](https://yomu.fyi/post/using-dns-traffic-management-to-add-resiliency-to-shopify-s-services.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 5, 2020
- Type: Problem & solution

Shopify consolidated several team-specific DNS traffic-management approaches that relied on manual changes, pull requests, or provider APIs and had uneven failover, redundancy, and active/active capabilities. The replacement was a fully automated, self-served system for more than 40 domains owned by more than 12 teams and handling more than 100M requests per day. It stores domain configuration in Git, deploys it to two DNS providers, and exposes spy commands for fast updates and failovers. Its traffic-management design uses regionalized matching, endpoint-status filtering, weighted sharing, and fallback layers, while globally matching endpoints ensure an existing domain receives an answer. A later provider migration demonstrated the value of centralized ownership: the team updated Terraform and tooling, moved nameservers incrementally, and kept the change transparent to application owners despite concurrent incidents.


### [Let’s Debug a Node.js Application](https://yomu.fyi/post/let-s-debug-a-node-js-application.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Julián Duque
- Published: Aug 3, 2020
- Type: Tutorial

Node.js applications can be difficult to debug because asynchronous workflows complicate stack traces, while crashes may require core-dump analysis through complex command-line tools. The article presents a layered toolkit: production-oriented logging with Pino or Winston, interactive inspection through Node Inspect and Chrome DevTools, IDE integrations, and the standalone ndb debugger. Running node --inspect-brk launches an application for inspection at chrome://inspect, where breakpoints, stepping, call stacks, heap snapshots, and memory profiling are available; ndb additionally supports edit-in-place changes. For catastrophic crashes, llnode maps JavaScript frames and objects to C/C++ representations from a core dump generated with process.abort, while debug filters namespaced logs and trace and clarify improve asynchronous traces and remove Node.js internals, respectively, for local debugging.


### [Continuous PostgreSQL Backups using WAL-G](https://yomu.fyi/post/continuous-postgresql-backups-using-wal-g.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Aug 2, 2020
- Type: Tutorial

Continuous PostgreSQL backups require coordinating physical base snapshots with ongoing write-ahead log archiving to support point-in-time recovery. The open-source WAL-G utility simplifies this workflow by managing backup pushes and fetches to cloud storage services such as Amazon S3. In a typical setup on Ubuntu with PostgreSQL 12, envdir supplies AWS credentials and storage prefixes to WAL-G commands embedded directly in the PostgreSQL archive configuration. Regular base backups can be automated via cron jobs, which minimizes the volume of WAL archives that must be replayed during a restore. When recovering to a new instance, administrators fetch the latest base backup, define a restore command with targeted recovery timestamps in the configuration, and initiate replay using a recovery signal file.


### [Supabase Alpha July 2020](https://yomu.fyi/post/supabase-alpha-july-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Aug 2, 2020
- Type: Announcement

Supabase announced the release of Supabase Auth following five months of development, providing resources and a video tutorial on implementing authentication with PostgreSQL Row Level Security. The Table Editor now allows users to drill multiple levels deep into table relationships as well as add, delete, and download rows directly. Newly created projects also include access to additional Postgres extensions, including pgsql-http, pgjwt, plpgsql\_check, and pljava. Further platform updates include an or filter in the client library, expanded capabilities for the postgres-meta REST management server, and persistence for open tabs in the Table View. Additionally, the auto-generated documentation now integrates authentication guides, and visual page transition glitches have been resolved.


### [Sandboxing and Workload Isolation](https://yomu.fyi/post/sandboxing-and-workload-isolation.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 29, 2020
- Type: Explainer

Workload isolation makes it harder for a vulnerability in one service to compromise the rest of a platform, and becomes especially important when hosting other people’s code. The post surveys isolation mechanisms from chroot and privilege separation through namespaced containers, mandatory access control, seccomp-bpf, capabilities, and language runtimes, before considering virtualized approaches. It emphasizes that network exposure is often the most consequential attack surface, while chroot alone leaves access to ptrace, procfs, device nodes, and networking. Firecracker is presented as the selected VMM: a small Rust implementation that limits itself with seccomp-bpf and runs under an external jailer providing chroot, namespaces, and dropped privileges. The closing menu includes nsjail, unprivileged Docker with tighter filtering, gVisor, and Firecracker or Kata; absent hostile code, the author would tune nsjail for return on investment.


### [Migrating Large TypeScript Codebases To Project References](https://yomu.fyi/post/migrating-large-typescript-codebases-to-project-references.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 29, 2020
- Type: Tutorial

Shopify’s merchant admin UI grew from a Ruby on Rails ERB-based front-end into a TypeScript, React, and GraphQL codebase, and its size eventually made VSCode’s TypeScript language tooling take roughly two to three minutes to initialize. The team investigated the bottleneck with TypeScript’s performance diagnostics, VSCode TSServer logs, and support from the TypeScript team, which recommended splitting the monolithic configuration with project references. They built the TypeTrack VSCode plugin to measure initialization time, then migrated dependency-graph leaf nodes such as packages and tests, creating project-level tsconfig.json files and references incrementally. The migration reduced initialization times for sampled files from about 155 seconds to 8–13 seconds. The post explains how to resolve missing module errors, respect project-reference restrictions, and continue upward through the dependency graph while distributing migration work across teams.


### [Serve small with Fly.io and GoStatic](https://yomu.fyi/post/serve-small-with-fly-io-and-gostatic.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 27, 2020
- Type: Tutorial

The post shows how to deploy a tiny static signpost site on Fly.io using GoStatic, with the aim of keeping the application’s footprint and running cost low. GoStatic is a self-contained web server that can run in a bare Docker image; the described image is 6MB, while the completed deployment reports 7.7MB. The setup uses fly init examplegostatic -p 8043, a two-line Dockerfile based on pierrezemb/gostatic, and an index.html copied to /srv/http/index.html; Fly routes ports 80 and 443 to GoStatic’s port 8043 and provides HTTPS. After deployment reaches one healthy instance, the post reduces the default micro-2x VM from 512MB RAM to micro-1x, which provides 128MB. This matches the small workload and is presented as a way to reduce running costs.


### [Ground Control to Major TOML: What Is a TOML File and Why Do Buildpacks Use TOML?](https://yomu.fyi/post/ground-control-to-major-toml-what-is-a-toml-file-and-why-do-buildpacks.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Joe Kutner
- Published: Jul 22, 2020
- Type: Explainer

The Cloud Native Buildpacks project uses TOML for developer-facing configuration after weighing its readability, predictability, and fit for lightweight settings against YAML, JSON, and XML. TOML is a plain-text format with a .toml extension, built around key-value pairs, headers, sections, arrays, and nested tables; it permits comments, avoids indentation-based structure, and supports explicit types. Buildpack users generally do not write these files from scratch, but when configuration is needed, the format provides clarity without complex schemas or large declarative configuration. Buildpacks favor convention over configuration and detect behavior from application contents, so TOML suits the simple human-defined properties around their tooling, while JSON remains for machine-readable formats such as OCI image metadata. The trade-off is that TOML tooling for parsing and querying is less readily available, and the format may still feel unfamiliar, but prior use in containerd, Cargo, and Poetry supported the choice.


### [Hugo's There - Flying with Hugo and Caddy](https://yomu.fyi/post/hugo-s-there-flying-with-hugo-and-caddy.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 21, 2020
- Type: Tutorial

A new community venture’s website is built with Hugo, served by Caddy, and deployed on Fly, with a custom makeroni.cc domain added after the initial deployment. The process initializes a Fly app on port 80, uses Hugo’s Quickstart and Ananke theme, and employs a multi-stage Docker build based on klakegg/hugo:0.74.0-onbuild, copying generated files into caddy:2.1.1. Caddy serves the static files with file\_server while auto\_https is disabled because Fly handles certificate management; separate rules serve makeroni.cc and redirect the old fly.dev hostname. DNS A and AAAA records connect the domain to Fly, and flyctl certs create requests a Let’s Encrypt certificate through DNS validation, leaving the site ready to serve securely after redeployment.


### [How We’re Solving Data Discovery Challenges at Shopify](https://yomu.fyi/post/how-we-re-solving-data-discovery-challenges-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 21, 2020
- Type: Problem & solution

Shopify describes data discovery as a response to growing numbers of data assets and the difficulty of finding, governing, and using them across the data lifecycle. At Shopify, pre-Artifact discovery relied on people, Slack, GitHub, and job logs, while teams lacked granular ownership, change-impact visibility, and documented reporting context. They built Artifact, a search-and-browse tool backed by a centralized metadata model, ingestion jobs, Elasticsearch indexes, GraphQL APIs, and a graph database for filterable upstream and downstream lineage. Its landing page supports browsing and plain-English search, while details pages expose documentation, schema, descriptions, usage, ownership, and lineage; initial scope prioritized frequently used, high-impact, queryable assets and downstream reports and dashboards. Since launch in early 2020, weekly Data team usage exceeded 30% and monthly retention exceeded 50%, above expectations, with future work planned for additional assets, integrations, stewardship, change management, notifications, and metadata APIs.


### [Making Time to Save You Time: How We Sped Up Time-Related Syscalls on Dynos](https://yomu.fyi/post/making-time-to-save-you-time-how-we-sped-up-time-related-syscalls-on-d.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Will Farrington
- Published: Jul 16, 2020
- Type: Problem & solution

A customer support ticket exposed poor performance in clock\_gettime(3) and gettimeofday(2) for an application making many timing calls through transaction tracing. Heroku's Runtime Infrastructure team found that Linux was selecting the Xen clocksource, whose Xen version did not support virtual dynamic shared object (vDSO) acceleration, forcing costly userspace-to-kernel context switches. The team evaluated available clocksources and selected tsc, which uses the Time Stamp Counter and supports vDSO; they assessed concerns about backwards clock drift, including older CPU inconsistencies and Xen VM freezing, against their newer Intel CPUs and EC2 environment. After vdsotest validation, the configuration was deployed across the Common Runtime and Private Spaces dyno fleet, and the initiating customer plus other customers observed performance improvements.


### [The Journey of Deploying Apache Airflow at Grab](https://yomu.fyi/post/the-journey-of-deploying-apache-airflow-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chandulal Kavar
- Published: Jul 14, 2020
- Type: Problem & solution

Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.


### [How We Built Size.link](https://yomu.fyi/post/how-we-built-size-link.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 13, 2020
- Type: Problem & solution

Size.link is a free Shopify AR tool that lets shoppers view a product’s dimensions in their surroundings through a smartphone camera. It was created during April Hack Days as a quick alternative to commissioning 3D models, with requirements including browser-based operation, speed, accuracy, and iOS and Android support. Because iOS lacks JavaScript WebAR, the implementation uses AR Quick Look with USDZ files and Android’s Scene Viewer with GLB files. Rather than generate files dynamically with slow USDZ tooling or pre-generate billions of dimension combinations, the team modifies binary template data, replacing scale or vertex float values to resize the cube while preserving its outline. The resulting Ruby server generates USDZ files in well under one millisecond, and the same strategy produces GLB files; future work could add textures for standard-sized products.


### [Alpha Launch Postmortem](https://yomu.fyi/post/alpha-launch-postmortem.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Jul 10, 2020
- Type: Incident / postmortem

An unexpected viral submission on Hacker News prompted sudden alpha traffic for Supabase, attracting 30,000 visitors and over 1,400 signups in one week. The surge strained initial infrastructure, which initially relied on a single server running Docker Compose for middleware and Digital Ocean for customer databases. System bottlenecks emerged rapidly, including hit limits on Digital Ocean droplet counts, frequent physical server faults, and a three-hour provisioning outage when Cloudflare reached its 1,000-subdomain limit. In response, the team migrated middleware to multiple globally load-balanced ECS clusters via AWS Global Accelerator. Supabase also transitioned 1,800 servers from Digital Ocean to AWS t3a instances, nearly halving database startup times from 90 seconds to 50 seconds.


### [Flyctl Evolved - Fly Changelog](https://yomu.fyi/post/flyctl-evolved-fly-changelog.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 10, 2020
- Type: Announcement

Flyctl v0.0.137 introduces an app-centric command structure while deprecating the apps subcommand, whose operations now have top-level equivalents such as init, destroy, move, restart, resume, suspend, and list apps. The release changes move, restart, resume, suspend, and status to accept an app name as a final argument, while retaining status -a; list also gains app-name fragments and status or organization filters, and list orgs shows accessible organizations. The init flow offers builders or Dockerfile use, can generate a simple hello-world deployment, and supports --dockerfile to skip the builder query, while info and version gain scripting-oriented output controls. Secret changes block operations on suspended deployments and can monitor deployments unless --detach is used. Platform fixes concatenate multiple HTTP/2 cookie headers during HTTP/1 downcasting, add Via headers to requests and responses, and improve deployment image optimization.


### [What are PostgreSQL Templates?](https://yomu.fyi/post/what-are-postgresql-templates.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Jul 9, 2020
- Type: Explainer

PostgreSQL bases every new database creation on an existing template database within the cluster, defaulting to template1. While administrators can directly modify template1 with tables, data, extensions, or procedural languages, altering it risks breaking future database creation commands if mistakes happen. Setting an existing database as a custom template using the ALTER DATABASE command allows users with the CREATEDB privilege to instantiate customized databases without polluting system templates. Any database creation from a template requires zero active connections on that template at execution time, making pg\_dump the preferred tool for replicating active production environments. The immutable template0 system database serves as a fallback to recreate corrupted templates, restore clean dumps, or initialize databases with alternative character encodings.


[Newer posts](https://yomu.fyi/posts/page/230.md) · [Older posts](https://yomu.fyi/posts/page/232.md)