---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Reducing BigQuery Costs: How We Fixed A $1 Million Query](https://yomu.fyi/post/reducing-bigquery-costs-how-we-fixed-a-1-million-query.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 3, 2022
- Type: Problem & solution

During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.


### [Supabase Beta October 2022](https://yomu.fyi/post/supabase-beta-october-2022.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: ant\_wilson
- Published: Nov 2, 2022
- Type: Announcement

Supabase’s October 2022 update reports a broad set of product, documentation, and community releases, while announcing that Launch Week 6 will run from December 12–16. The new supabase-js v2 and supabase-flutter v1 SDKs reached full release, alongside a revamped Next.js quickstart with pre-built Auth UI and Auth Helpers and a Next.js 13 example for fetching and caching data from the Serverless API. Documentation covers Supabase Auth’s server-side rendering flow, and a database-testing framework uses pgTAP and pg\_prove; Edge Functions also gained support for GET, PUT, PATCH, and DELETE requests. Additional changes include functions support in supabase-py, session refreshing, provider refresh tokens, request-location detection through X-Forwarded-For, expanded logging, and storage downloads via a query parameter, alongside three featured community apps and hiring announcements.


### [A love letter to React](https://yomu.fyi/post/a-love-letter-to-react.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 2, 2022
- Type: Explainer

React’s 2013 release had a lasting impact on frontend development and influenced Phoenix LiveView through its reactive component model, JSX colocation, and efficiency. Chris McCord explains how a component’s state and render function create a simple programming model, then shows LiveView adapting that model to a stateful server process with HTML-aware function components and slots. The discussion argues that tightly coupled markup and code should live together, while extensible components preserve HTML structure and support customization without bespoke templates. For rendering efficiency, React computes virtual DOM diffs, whereas LiveView tracks static and dynamic HEEx template parts, sends minimal diffs over the network, and applies client-side patches with morphdom. The conclusion credits React with shaping Phoenix’s direction and argues that backend frameworks should provide HTML-aware component systems.


### [Logbook: October 21 to 28, 2022](https://yomu.fyi/post/logbook-october-21-to-28-2022.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 31, 2022
- Type: Announcement

This weekly Logbook changes format from tracking every monthly change to an over-generalized weekly tl;dr of Fly.io updates intended to help with deploying or managing apps. It reports new Postgres documentation for performing failovers, regional failovers, setting up and attaching a database, and migrating one from Heroku, while warning that Fly Postgres is not fully managed and requires customer monitoring. The post also points to an example of integrating the Elastic Stack (ELK) with a Laravel app for user analytics and reporting, plus instructions for shutting down an idle Phoenix app on Fly Machines after a configurable period without requests. It explains that newly launched Fly apps do not yet deploy to Machines in the same way, while fly pg create does use Fly Machines, and directs readers to comparisons of the app versions. Finally, it notes colorblind-friendly dashboard improvements and previews more Postgres documentation work.


### [Mixing It Up: Remix Joins Shopify to Push the Web Forward](https://yomu.fyi/post/mixing-it-up-remix-joins-shopify-to-push-the-web-forward.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 31, 2022
- Type: Announcement

Shopify announces that the open-source web framework Remix and its team are joining the company to help developers deliver lightning-fast, resilient web experiences. Remix combines multi-page and single-page strengths through a full-stack React framework with data loading and code splitting handled without requiring front-end developers to manage those backend tasks. It also emphasizes progressive enhancement, with links, buttons, and forms working before JavaScript loads, and supports edge, service-worker, and Node runtimes. Shopify will use Remix across projects including Hydrogen; Hydrogen is moving from server components to Remix's data-loading pattern for faster performance and a simpler developer experience, with ~90% of developers' code expected to remain unchanged. Remix will remain independent and open-source while Shopify supports its roadmap and community.


### [The Management Poles of Developer Infrastructure Teams](https://yomu.fyi/post/the-management-poles-of-developer-infrastructure-teams.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 28, 2022
- Type: Explainer

Developer infrastructure managers must balance three poles: management support, system and domain expertise, and organization-wide road maps, with the tension intensified by teams that often lack dedicated product managers and support critical systems for hundreds or thousands of users with six to eight developers. Domain-focused teams build deep knowledge needed for maintenance, migrations, incident investigation, on-call coverage, and developer-experience improvements, but their local road maps can diverge from the department’s highest-impact opportunities. The proposed responses are temporary individual assignments, having a whole team contribute to another team’s goals, and forming a cross-functional tiger team; each shifts trade-offs among focus, maintenance, expertise, motivation, and management oversight. A month-long tiger team built the first Spin proof of concept, which led to a dedicated team, but no structure eliminates the tensions, so choices must be judged according to context.


### [What is PostgreSQL commitfest and how to contribute](https://yomu.fyi/post/what-is-postgresql-commitfest-and-how-to-contribute.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: pavel
- Published: Oct 27, 2022
- Type: Tutorial

PostgreSQL Commitfest is the community's month-long review process for patches intended for merging into core. Contributors submit patches to pgsql-hackers, create a Git branch from master, consolidate changes, generate a versioned patch, and associate its mailing-list thread with an open Commitfest entry. The process expects authors to explain rationale, respond to review, rebase or revise patches, and review a comparable number of other patches. Submitted changes are checked by CFBot across architectures and operating systems, while statuses such as "Needs review," "Ready for committer," and "Waiting on author" track progress. Commitfests occur five times yearly, with the March event tied to feature-freeze for the October major release.


### [A Software Engineer's Guide to Working Across Time Zones](https://yomu.fyi/post/a-software-engineer-s-guide-to-working-across-time-zones.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 27, 2022
- Type: Tutorial

Working across time zones can make live collaboration difficult when teammates are nine to 12 hours apart, as described by a developer based in Singapore. The proposed alternative is to treat much of the work as asynchronous and deliberately document progress, blockers, decisions, and context. Examples include taking turns across working hours, annotating non-obvious pull request changes, posting end-of-day summaries with links or investigation notes, and recording meetings with agendas, key decisions, and action items. During Shopify's Hack Days prototype work on Linkpop, a Singapore-based frontend developer handed off updates before sleeping while North American teammates extended backend functionality and responded to blockers. The post concludes that these practices become habits that support handoffs, reduce back-and-forth, preserve information for absent teammates and future reference, and can provide longer uninterrupted focus periods.


### [Using mobile sensor data to encourage safer driving](https://yomu.fyi/post/using-mobile-sensor-data-to-encourage-safer-driving.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Laiyi Lin
- Published: Oct 25, 2022
- Type: Problem & solution

Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.


### [supabase-flutter v1 Released](https://yomu.fyi/post/supabase-flutter-v1-released.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: tyler\_shukert
- Published: Oct 21, 2022
- Type: Announcement

Supabase has released the stable v1 of its supabase-flutter SDK after several months of developer preview feedback and improvement. The release targets developer experience by reducing boilerplate and introducing more intuitive APIs, including removal of the required .execute() call from select, insert, update, delete, and stream operations; query errors are now thrown rather than returned. Authentication methods use more descriptive names such as signInWithPassword and signInWithOAuth, while onAuthStateChange returns a Dart stream. The SDK also adds first-class support for Realtime Multiplayer's broadcast and presence methods, including low-latency data sharing and client-status tracking, and the announcement provides updated documentation, a quick start guide, and a v0-to-v1 migration guide.


### [Hubble: Our Tool for Encapsulating and Extending Security Tools](https://yomu.fyi/post/hubble-our-tool-for-encapsulating-and-extending-security-tools.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 21, 2022
- Type: Explainer

Shopify’s Trust team faced rising complexity in securing a distributed device fleet, including many tools, provisioning and patching work, varied networks, and the loss of in-person remediation. Hubble was built as a unified layer over mobile device management and other security systems, ingesting and standardizing their data while sending commands back with granular access controls and centralized auditing. IT staff use it for inventory, device management, and security, while employees can view device health and compliance, receive remediation guidance, and manage test-device or beta participation. The platform also serves as a standardized source for automation, and the post presents encapsulation and cross-team investment as ways to reduce administrative overhead, support proactive security, and limit risk.


### [supabase-js v2 Released](https://yomu.fyi/post/supabase-js-v2-released.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: thor\_schaeff
- Published: Oct 20, 2022
- Type: Announcement

Supabase-js v2 has been released after a release-candidate period during which the team incorporated user feedback and updated documentation, examples, quickstart guides, and a migration guide. The release’s major addition is built-in TypeScript support, including a CLI that generates types directly from a database. It also includes improvements aimed at addressing some of the largest pain points identified by users, with the announcement providing updates and differences from v1. Version 2.0 was produced through contributions from several Supatroopers, more than 100 contributors to the libraries, and more than 450 contributors to the documentation and websites; the listed package contributions include functions-js, gotrue-js, postgrest-js, realtime-js, storage-js, and supabase-js.


### [How to Structure Your Data Team for Maximum Influence](https://yomu.fyi/post/how-to-structure-your-data-team-for-maximum-influence.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 20, 2022
- Type: Problem & solution

Embedded data science leaders may support multiple product and business areas, creating competing work streams, specializations, and stakeholder relationships. The post compares swim lanes and stochastic process assignments against principles of efficiency, influence, stakeholder clarity, stability, growth, and flexibility. It recommends a hybrid “diamond defense” in which scientists are loosely assigned to zones for specialization while a bullpen, including the manager and an additional scientist, supplies reserve capacity. Resources can be redirected to overloaded areas, and team members can rotate into vacancies or cover leave. The framework is presented as combining clear ownership and subject-matter depth with the flexibility to handle change, though the post warns that excessive rotation can create volatility and turnover.


### [Finding Relationships Between Ruby’s Top 100 Packages and Their Dependencies](https://yomu.fyi/post/finding-relationships-between-ruby-s-top-100-packages-and-their-depend.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 19, 2022
- Type: Explainer

RubyGems’ phased MFA rollout required owners of gems with at least 180 million downloads to use MFA, but raised a supply-chain question: could a popular gem depend on a less-downloaded gem that remained an account-takeover target? The investigation loaded rubygems.org data dumps to identify 112 gems above the threshold, queried the API for direct dependencies, and used Bundler with Gemfile.lock files to resolve transitive dependencies. It found 13 big gems with small direct dependencies and 24 with small dependencies overall, attributing the mismatch mainly to newer dependencies and gems shipped with Ruby, such as racc and rexml. Graph visualizations, breadth-first traversal, and a custom depth-first search then mapped dependency structure and paths; the post leaves open whether to do nothing or enforce MFA early for the 24 technically insecure gems.


### [Postgres Full Text Search vs the rest](https://yomu.fyi/post/postgres-full-text-search-vs-the-rest.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: victor
- Published: Oct 14, 2022
- Type: Benchmark

The post compares PostgreSQL Full Text Search with dedicated search engines including MeiliSearch, OpenSearch, SQLite FTS, and Typesense, asking whether a separate system is justified for common application search. It presents a PostgreSQL setup using a generated stored column built with to\_tsvector('english', ...) and a GIN index, queried with websearch\_to\_tsquery. The benchmark loads a movie dataset and focuses on query speed after data ingestion, using comparable scripts and default configurations with limited tuning. PostgreSQL was close in performance to several alternatives; in-memory SQLite was fastest, while Typesense and MeiliSearch handled misspellings correctly and OpenSearch ingested data quickly. The conclusion is that PostgreSQL can provide good-enough search quality and speed for basic cases without adding another system, although dedicated engines offer capabilities PostgreSQL does not.


### [On the Importance of Pull Request Discipline](https://yomu.fyi/post/on-the-importance-of-pull-request-discipline.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 13, 2022
- Type: Tutorial

The article argues that pull request (PR) discipline matters beyond code correctness because implementation and maintenance both benefit from changes that are clearly structured and documented. It describes a PR as a proposal whose title, summary, commits, comments, reviewers, labels, projects, and linked issues should intentionally communicate scope, design reasoning, low-level details, ownership, and historical connections. Its practical guidance focuses on precise titles, useful summaries, logically ordered commits, deliberate remote-branch changes, defined PR boundaries, and moving asides into follow-up PRs. It also recommends smaller, well-ordered PRs, permalink-based code references, distinctive branch-name prefixes, and learning Git commands including reflog, cherry-pick, and rebase. The article treats discipline as judgment rather than rigid law, allowing emergency exceptions while encouraging post-resolution documentation.


### [How to Write Code Without Having to Read It](https://yomu.fyi/post/how-to-write-code-without-having-to-read-it.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 11, 2022
- Type: Tutorial

The article argues that developers can minimize code reading when fixing bugs or changing features by focusing on information not already checked by tools. It recommends locating a user-visible entrypoint, then making a deliberately breaking edit and climbing the callstack through build errors rather than reading each caller. Strict types, tagged unions, exhaustive switches, linters, and unit tests expose incompatible inputs, missing cases, deprecated patterns, and affected consumers; pure functions separated from side-effect code keep tests simple and informative. After the tooling-driven passes, developers manually confirm behavior, revisiting the design when runtime issues remain and making future changes more "unread"-able.


### [Supabase Beta September 2022](https://yomu.fyi/post/supabase-beta-september-2022.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: ant\_wilson
- Published: Oct 5, 2022
- Type: Announcement

Supabase’s September 2022 update centers on reducing technical debt and clearing its backlog after five Launch Weeks. The team ran three Kaizen weeks focused respectively on QA and testing, documentation, and issue backlog and automation, closing over 250 open issues and 50 pull requests while pausing feature development. Product releases included Auth UI, a pre-built React component for Supabase Auth with custom themes and extensible styles, plus an open-source Postgres WASM release with Snaplet for running Postgres in a browser through an embeddable Linux Virtual Machine. Security changes to Dashboard database interactions simplify permissions and reduce the security surface area; they were scheduled for automatic application in November or could be run earlier through the Dashboard. Other updates covered Edge Functions, documentation, migration tooling, and community projects.


### [The Engineering Story Behind Flex Comp](https://yomu.fyi/post/the-engineering-story-behind-flex-comp.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 5, 2022
- Type: Explainer

Shopify’s Flex Comp lets employees allocate total rewards among base salary, Restricted Stock Units (RSUs), and Stock Options, with allocation changes during windows and no one-year equity cliffs. Building the system required a prioritized backlog, cross-functional teams, early API contracts, and privacy controls around its code and data. For sensitive analytics events, the team encrypted entire Kafka payloads with RSA, using 4096-bit keys for ordinary events and 6144-bit keys for larger payloads, while later identifying hybrid encryption as a better design. On the frontend, typed hooks, mocked data, and artificial delays allowed business logic and interface work to proceed in parallel while contracts evolved. Version one shipped to Shopify’s employees, with documented technical and design debt and future decisions retained for subsequent iterations.


### [Safely Adding NOT NULL Columns to Your Database Tables](https://yomu.fyi/post/safely-adding-not-null-columns-to-your-database-tables.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 4, 2022
- Type: Explainer

Shopify’s Database Migrations team investigated whether Large Hadron Migrator (LHM) can safely add NOT NULL columns to existing MySQL tables with minimal downtime. LHM creates a shadow table, installs triggers for INSERT, UPDATE, and DELETE operations, and copies records in batches before renaming the tables. The experiments varied whether the new column had a DEFAULT value or UNIQUE index and whether MySQL used strict or non-strict mode, evaluating trigger compatibility and record counts. A NOT NULL column with a DEFAULT was backward compatible, but adding a UNIQUE index could cause data loss; without a DEFAULT, strict mode broke compatibility while non-strict mode used implicit defaults, which may be undesirable.


[Newer posts](https://yomu.fyi/posts/page/202.md) · [Older posts](https://yomu.fyi/posts/page/204.md)