---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Vouching for Docker Images](https://yomu.fyi/post/vouching-for-docker-images.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 1, 2020
- Type: Explainer

Kubernetes does not enable Docker image signature verification by default, leaving organizations to decide whether images are trusted before they run in a cluster. The post explains code signing through hashes and public/private keys, then describes Binary Authorization as a combination of metadata storage, an enforcer, and a signing service. Using Grafeas and Kritis, or GKE’s Binary Authorization, cluster policies require specified attestors to sign images; invalid or unsigned images are blocked before their containers start. Voucher supplies the missing signing component as a REST service running in Cloud Run or Kubernetes: build pipelines submit image paths, Voucher performs security checks, signs qualifying images, and records signatures in the metadata service. Its extensible checks can verify internal ownership, GitHub commits, approvals, builder identity, and vulnerability counts, while distinct cluster policies balance compliance requirements with faster experimentation; Voucher later moved into the Grafeas organization.


### [The November Fly Changelog](https://yomu.fyi/post/the-november-fly-changelog.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 30, 2020
- Type: Announcement

The November 2020 Fly changelog records new example applications, Fly CLI enhancements, and platform updates. Guides now cover Gogs, MQTT with Mosquitto, MinIO, Node-RED, and Redis with and without TLS, and demonstrate preview persistent Volumes for database persistence, data storage, preserved programming state, and queued messages. The CLI added bulk secret imports from key=value files, including multiline values enclosed in triple double quotes, while list commands gained filtering, sorting, and JSON output; the Deno builtin added a Version setting defaulting to 1.5.4. Platform changes included an upgrade to Firecracker v0.23, improved restart and connection handling, timeout fixes, and stronger application-context isolation. The changelog also notes new crash-log context, humanized app times, improved image resolution, and corrected internal port and metrics handling.


### [More Appkata Examples to Try on Fly](https://yomu.fyi/post/more-appkata-examples-to-try-on-fly.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 27, 2020
- Type: Announcement

Two new Appkata examples show how to run selected applications on Fly: Node-RED and Redis secured with TLS. Node-RED is a GUI flow-programming environment built on Node, with modules for connecting MQTT, Redis, Graphana, and other systems, plus creating web dashboards; the guide is intended to get it running quickly. The secured Redis example uses mkcert to simplify creation of TLS certificates, extending an earlier Redis example that presented Redis as a simple service. The post also points to Community.fly.io as a place for questions, hints, and tips, noting that Fly accounts can use single sign-on to make participation easier.


### [Democratising Fare Storage at Scale Using Event Sourcing](https://yomu.fyi/post/democratising-fare-storage-at-scale-using-event-sourcing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Nov 23, 2020
- Type: Problem & solution

Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.


### [Static Typing for Ruby](https://yomu.fyi/post/static-typing-for-ruby.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020
- Type: Explainer

Shopify’s Ruby monolith spans 37,000 files, 622,000 methods, and more than 2,000,000 calls, making fast feedback and stability difficult despite rigorous reviews and 150,000 automated tests. Sorbet was selected after the team evaluated requirements for gradual typing, speed, and support for Ruby and Rails features including metaprogramming, overloading, and class reopening. Its RBI files represent constructs it cannot infer, while per-file sigils allow adoption to progress without blocking development, and SorbetMetrics tracks sigils, typed calls, and method signatures. Shopify treats typing as a product, combining CI enforcement, developer support, surveys, and interviews to guide rollout and measure sentiment. The excerpt reports 80% of monolith files, including tests, at typed: true or higher, with almost half of calls and methods covered and type checking under 15 seconds on developer machines.


### [Adopting Sorbet at Scale](https://yomu.fyi/post/adopting-sorbet-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020
- Type: Explainer

Shopify describes its adoption of Sorbet static typing across a Ruby monolith containing 37,000 files, 622,000 methods, and more than 2,000,000 calls. At the time of writing, Sorbet ran on every pull request, 80% of files were typed: true or higher, almost half of calls were typed, and half of methods had signatures. To handle Ruby and Rails idioms, the team built RuboCop Sorbet for compatibility rules, Tapioca for gem and DSL RBI generation, and Spoom for programmatic tooling, metrics, and LSP access. A controlled experiment found fewer production NoMethodErrors in files typed: true after typing about 20% of the application, although the results were preliminary and signatures had not yet been added. Shopify planned to reach typed: true across all files, improve Rails support, and continue collaboration around Sorbet and RBS.


### [How to Build a Production Grade Workflow with SQL Modelling](https://yomu.fyi/post/how-to-build-a-production-grade-workflow-with-sql-modelling.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020
- Type: Problem & solution

Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.


### [Postgres Views](https://yomu.fyi/post/postgres-views.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Nov 18, 2020
- Type: Explainer

Postgres views serve as query shortcuts that execute underlying SQL statements upon retrieval without generating new tables or persisting duplicate data. By encapsulating complex multi-table joins, standard views provide query consistency across applications, simplify repetitive calls, improve logical schema organization, and enhance security by restricting sensitive columns. In contrast, materialized views physically store query results on disk, dramatically reducing read latency for heavy queries spanning millions of rows. Because materialized views introduce the trade-off of stale data, administrators must periodically run the refresh command based on workload tolerances for use cases like analytics and internal dashboards. Materialized views should not substitute query optimization, as underlying query efficiency remains essential.


### [Appkata - Example Apps on Fly](https://yomu.fyi/post/appkata-example-apps-on-fly.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 18, 2020
- Type: Tutorial

Appkata introduces four example backend apps for Fly, intended to show how to deploy common services while learning the platform’s deployment patterns. The examples cover Redis as a customizable, volume-backed key/value database; MinIO as a single-node, S3-compatible object store exposed through Fly’s TLS edge connections; Gogs as a private Git server using SQLite3 and Fly volumes; and Mosquitto as an MQTT broker with TLS, username/password authentication, and volume-backed message persistence. The examples emphasize adapting existing Docker images, attaching persistent storage, and exposing services securely. The post presents these as an initial set that will expand over the following weeks, alongside a Fly Community site for questions and guidance.


### [Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models](https://yomu.fyi/post/leveraging-pre-trained-language-model-checkpoints-for-encoder-decoder.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Nov 9, 2020
- Type: Tutorial

Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.


### [Porting fairseq wmt19 translation system to transformers](https://yomu.fyi/post/porting-fairseq-wmt19-translation-system-to-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Stas Bekman
- Published: Nov 3, 2020
- Type: Tutorial

The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.


### [Supabase Alpha October 2020](https://yomu.fyi/post/supabase-alpha-october-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Nov 2, 2020
- Type: Announcement

Eight months into building Supabase, the team released product updates focused on stability, reliability, and performance while preparing to transition from Alpha to Beta. The release introduces supabase-js version 1.0, incorporating extensive community feedback to deliver major developer experience improvements across client libraries. Platform interface updates include tabs and full-screen mode for the SQL Editor, alongside new keyboard shortcuts to help power users navigate and manipulate tables rapidly. For user authentication, Supabase added magic link capabilities, enabling passwordless login flows for both new and existing users either as a standalone method or alongside passwords. Under continuous improvement efforts, the team converted realtime-js to TypeScript, heavily optimized dashboard routing performance, and established a repository to benchmark all integrated open source tools.


### [Hyperparameter Search with Transformers and Ray Tune](https://yomu.fyi/post/hyperparameter-search-with-transformers-and-ray-tune.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Nov 2, 2020
- Type: Tutorial

Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter\_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.


### [Supabase.js 1.0](https://yomu.fyi/post/supabase-js-1-0.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Oct 30, 2020
- Type: Announcement

Supabase has launched Supabase.js 1.0, updating its client library ecosystem with breaking API changes, scalability enhancements, and new documentation tooling. The release introduces native TypeScript implementations across all underlying sub-libraries, including supabase-js, postgrest-js, gotrue-js, and realtime-js, which also enables auto-generated reference docs. In response to community feedback, database and authentication operations now return structured error objects alongside data payloads instead of throwing runtime exceptions. Realtime client scalability is improved by consolidating multiple active subscriptions into a single socket connection per Supabase client. Additionally, the release updates authentication workflows via the newly created gotrue-js library to support third-party logins and OAuth providers alongside revised method names.


### [Keeping 170 Libraries Up to Date on a Large Scale Android App](https://yomu.fyi/post/keeping-170-libraries-up-to-date-on-a-large-scale-android-app.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lucas Nelaupe
- Published: Oct 30, 2020
- Type: Problem & solution

Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.


### [Fly Answers Questions - Suspend, Resume, Restart and Redis](https://yomu.fyi/post/fly-answers-questions-suspend-resume-restart-and-redis.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 30, 2020
- Type: Explainer

Fly’s answers clarify how to stop an application temporarily, restart instances, configure minimum capacity, use Redis, and invoke the CLI. The fly suspend command reduces an app’s instance count to zero while preserving its status and settings; fly resume first restores one instance, checks it, and then redeploys from the previous configuration. By contrast, fly restart restarts each instance in place without triggering a deployment, while applications cannot set a minimum instance count of zero except by suspending. Fly’s Redis implementation does not support Pub/Sub, scripting, or Geo commands, and flyctl now creates a symlink to the shorter fly command as migration toward the latter continues with script compatibility in mind.


### [How to Introduce Composite Primary Keys in Rails](https://yomu.fyi/post/how-to-introduce-composite-primary-keys-in-rails.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 29, 2020
- Type: Tutorial

Shopify addressed inefficient data access in its multi-tenant Rails application, where auto-incrementing primary keys interleaved records from many shops even though queries usually targeted one shop. InnoDB stores rows in B+ trees and loads pages into a buffer pool, so the team changed orders and other suitable tables to composite primary keys such as \[shop\_id, order\_id\] while keeping an auto-incrementing id secondary key and configuring Active Record to use id. Supporting Ghostferry, migration tooling, and data extraction systems also had to accommodate the schema change, with unique secondary constraints avoided in some cases because LHM migrations could deadlock. On the most queried table, common queries improved 5–6x, median buffer pool reads fell from 1.8 to 1.2, and elapsed database time dropped roughly one hour per day per shard, while inserts became about 10x slower.


### [The October Fly Changelog: Preview Disks and DNS and Better Builtins](https://yomu.fyi/post/the-october-fly-changelog-preview-disks-and-dns-and-better-builtins.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 27, 2020
- Type: Announcement

The October Fly changelog records platform and CLI updates, centered on preview persistent disks, configurable builtins, environment-variable configuration, and status monitoring. The new volumes commands let Fly applications create persistent disks whose data survives restarts, deployments, and suspension; the feature remains in preview. Applications can now set environment variables in fly.toml instead of using the secrets feature, while builtin settings expose controls such as Deno permissions and HTTPS auto-upgrading and logging for the static web server. flyctl 0.0.145 also adds Python/Procfile builtin support, a --watch status mode that polls every five seconds, a --rate option, default-org selection in init, and fixes for token checks and wildcard certificate domains.


### [Building Mental Models of Ideas That Don’t Change](https://yomu.fyi/post/building-mental-models-of-ideas-that-don-t-change.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 22, 2020
- Type: Explainer

Amid a constant stream of frameworks, languages, and platforms, the post argues that learning enduring principles is more useful than trying to absorb every new tool. It presents mental models as a checklist for prioritizing concepts, recording what matters, and making more informed decisions across engineering and management. Engineering examples include alerting on silent failures, queuing background jobs for faster responses and retries, using caching and denormalization for read-heavy workloads, and sharding or deliberate design choices for write-heavy systems. The post also recommends horizontal scaling for long-term growth, then turns to practices such as aligning incentives, clarifying product purpose, building rapport before feedback, and operating a few months ahead. The stated aim is to use these models to make more informed decisions rather than chase an ever-expanding list of new technologies.


### [BPF, XDP, Packet Filters and UDP](https://yomu.fyi/post/bpf-xdp-packet-filters-and-udp.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 20, 2020
- Type: Explainer

Deploying unmodified UDP applications across an anycast edge and worker network is harder than proxying HTTP, because UDP carries no convenient metadata for forwarding the original socket addresses. The initial implementation used userland services, AF\_PACKET sockets, and custom proxying, but the post argues that kernel-level packet handling is a better fit than adding and monitoring another service. It explains BPF’s evolution into eBPF, then describes using XDP for fast ingress processing, XDP\_REDIRECT around WireGuard limitations, and TC BPF on the worker side; Firecracker tap devices make reply traffic appear as host ingress. The verifier requires bounds checks and debugging benefits from dropwatch and perf-based packet tracing. The result is a beta deployment path for UDP applications on Fly.io.


[Newer posts](https://yomu.fyi/posts/page/228.md) · [Older posts](https://yomu.fyi/posts/page/230.md)