---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Deep Learning over the Internet: Training Language Models Collaboratively](https://yomu.fyi/post/deep-learning-over-the-internet-training-language-models-collaborative.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Max Ryabinin, Lucile Saulnier
- Published: Jul 15, 2021
- Type: Problem & solution

Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.


### [App Modularisation at Scale](https://yomu.fyi/post/app-modularisation-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Amar Jain
- Published: Jul 13, 2021
- Type: Problem & solution

Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.


### [Shopify-Made Patterns in Our Rails Apps](https://yomu.fyi/post/shopify-made-patterns-in-our-rails-apps.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 13, 2021
- Type: Explainer

Shopify describes patterns and infrastructure adaptations used to keep its large Rails monolith maintainable as its organization and platform scale. Its dev command-line tool standardizes project setup, dependencies, services, environment variables, integrations, and command aliases across hundreds of active projects. Shopify Core uses pods to partition shops across independent databases and extends the shard concept across provisioning, deployment, load balancers, caching, and servers, while splitting code into domains without separate databases. The post also details safeguards including short-term Active Record migrations consolidated periodically into structure.sql, automated tests, a pod-aware maintenance task system, and a translation platform that commits translated strings through Git. Finally, Monorail imposes versioned JSON event schemas between Kafka producers and consumers and supports review and automated scrubbing of PII fields.


### [Welcome spaCy to the Hugging Face Hub](https://yomu.fyi/post/welcome-spacy-to-the-hugging-face-hub.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Omar Sanseviero, Ines Montani
- Published: Jul 13, 2021
- Type: Announcement

Hugging Face has integrated spaCy pipelines into the Hugging Face Hub to simplify the distribution, testing, and deployment of natural language processing models across the community. More than 60 canonical models from the spaCy 3.1 release are hosted in the spaCy organization, alongside discoverable community pipelines. Users can package and upload models directly through the command line using the spacy-huggingface-hub extension, which auto-generates required metadata and model cards. Hosted pipelines offer direct pip installation from live wheel URLs, interactive browser exploration, and production-ready HTTP endpoints through the Inference API. The platform provides immediate interactive widget support for named entity recognition components, with upcoming support planned for text classification and part-of-speech tagging pipelines.


### [Shopify's Path to a Faster Trino Query Execution: Infrastructure](https://yomu.fyi/post/shopify-s-path-to-a-faster-trino-query-execution-infrastructure.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 9, 2021
- Type: Problem & solution

Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.


### [High Availability by Offloading Work Into the Background](https://yomu.fyi/post/high-availability-by-offloading-work-into-the-background.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 8, 2021
- Type: Explainer

Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.


### [Deploy Hugging Face models easily with Amazon SageMaker](https://yomu.fyi/post/deploy-hugging-face-models-easily-with-amazon-sagemaker.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Jul 8, 2021
- Type: Announcement

Hugging Face and Amazon introduced an inference solution integrating Hugging Face Transformers with Amazon SageMaker. The SageMaker Hugging Face Inference Toolkit and specialized Deep Learning Containers enable deployment of trained models or publicly available Hub models to managed production endpoints. Users configure endpoints with minimal code using the SageMaker Python SDK, standard pipelines, or custom inference scripts that override default behaviors. The architecture supports models stored in Amazon S3 as well as direct references via HF\_MODEL\_ID and HF\_TASK environment variables. Deployments benefit from native AWS infrastructure features, including built-in monitoring, Identity and Access Management permission controls, and Virtual Private Cloud connectivity.


### [Reshaping Chat Support for Our Users](https://yomu.fyi/post/reshaping-chat-support-for-our-users.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Elisa Monacchi
- Published: Jul 7, 2021
- Type: Problem & solution

Grab transitioned from voice hotlines and third-party tools to an in-house native chat support system integrated into their CRM. The team validated the platform through an MVP and user shadowing to address session disconnections, agent context switching, and routing bottlenecks. To optimize support operations at scale, they introduced dynamic queue limits based on Little's law, machine learning autocomplete suggestions for agents, and duration timers with visual nudges. These enhancements reduced chat waiting times by 30%, unresponsive users by 7%, and overall chat handling duration by 22%.


### [Protecting reserved roles with PostgreSQL Hooks](https://yomu.fyi/post/protecting-reserved-roles-with-postgresql-hooks.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Steve Chavez
- Published: Jul 2, 2021
- Type: Problem & solution

Supabase required granting database customers the CREATEROLE privilege to manage custom roles while preventing them from dropping or altering backend service roles such as supabase\_storage\_admin. Because stock PostgreSQL lacks a native mechanism to define custom reserved roles, the team created the SupaUtils extension using PostgreSQL hooks. Loaded via shared\_preload\_libraries, SupaUtils intercepts utility statements by overriding the global ProcessUtility\_hook function pointer. The extension inspects incoming statements like ALTER ROLE and DROP ROLE against a configurable list defined in postgresql.conf using DefineCustomStringVariable. If a targeted role matches the configured reserved roles, the hook raises an error, preserving backend infrastructure roles while permitting broader role administration.


### [Understanding GraphQL for Beginners–Part One](https://yomu.fyi/post/understanding-graphql-for-beginners-part-one.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 2, 2021
- Type: Explainer

The article introduces GraphQL as an open-source data query and manipulation language for APIs, contrasting it with REST's fixed response shapes and multiple endpoint calls. GraphQL lets clients describe exactly which fields they need, returning that requested structure through a single endpoint rather than a cookie-cutter response. Using a Toronto Eagles example, the article explains how one GraphQL query can combine requests for a team's founding year, captain's name, and last championship, avoiding three separate RESTful trips whose latency can vary. It also uses burger and bank withdrawal analogies to explain over-fetching, under-fetching, flexible frontend iteration, and fewer endpoints, then points readers to a forthcoming Ruby on Rails implementation in part two.


### [Understanding GraphQL for Beginners–Part Two](https://yomu.fyi/post/understanding-graphql-for-beginners-part-two.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 2, 2021
- Type: Tutorial

The second part of this beginner tutorial uses a food-focused Ruby on Rails application to show how GraphQL fields expose database data. It reviews the graphql directory, where mutations hold data-changing classes, types define returned structures, and food\_app\_schema.rb defines available queries. It distinguishes root fields, which are GraphQL entry points that define response structure, from object fields, which represent attributes, then demonstrates generating Food and Nutrition types and adding an all\_food root field with a non-null list return type. Through GraphiQL at localhost:3000/graphiql, it runs selective queries, adds nutrition to food, and introduces an id argument for retrieving one record; requests pass through graphql\_controller’s execute method and return JSON responses.


### [FYI: Livebook 0.2 with Kino](https://yomu.fyi/post/fyi-livebook-0-2-with-kino.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 1, 2021
- Type: Announcement

Livebook 0.2 was released with Kino, a client-side-focused feature for animating data changes, alongside new input support and the data\_table feature. Elixir cells can read input values with IO.gets("input name: "), although those values arrive as strings and may require parsing or other processing. data\_table presents data in a table with pagination and sorting, while Livebook is positioned for teaching Elixir, documenting libraries, experimenting with Nerves and embedded hardware, and connecting to a live production system. The post also notes that ElixirConfEU and ElixirConf US 2021 were announced as hybrid or mixed-format events, with ElixirConfEU scheduled for September 9–10 and ElixirConf US for October 12–15 across physical and online sessions.


### [Debugging High Latency Due to Context Leaks](https://yomu.fyi/post/debugging-high-latency-due-to-context-leaks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Jun 30, 2021
- Type: Incident / postmortem

Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.


### [Globally Distributed Postgres](https://yomu.fyi/post/globally-distributed-postgres.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 30, 2021
- Type: Problem & solution

Globally distributed Postgres describes a way to serve a conventional CRUD application across regions while keeping Postgres writes directed to one primary and reads near users. Fly’s proxy sends each request to the local application instance, where most requests succeed; when a write reaches a read-only replica, Rails catches PG::ReadOnlySqlTransaction and returns a fly-replay header naming PRIMARY\_REGION. The proxy then retries the complete HTTP request inside its network, avoiding per-query cross-region database routing and allowing the application to retain its ordinary connection behavior. The approach can also support session-based read-your-own-writes and replica-freshness checks, but it is not suitable for every workload; the text contrasts it with geographically distributed databases such as CockroachDB.


### [Monitoring Elixir Apps on Fly.io With Prometheus and PromEx](https://yomu.fyi/post/monitoring-elixir-apps-on-fly-io-with-prometheus-and-promex.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 28, 2021
- Type: Tutorial

Monitoring Elixir applications on Fly.io, the post presents a way to add production visibility without administering a Prometheus instance. Fly.io’s managed service scrapes an application’s metrics endpoint at a configured interval, with fly.toml specifying the port and path; Prometheus stores the resulting time-series snapshots and Grafana queries them with PromQL. PromEx supplies Elixir instrumentation through plugins for components such as the application, BEAM, Phoenix, Ecto, and Phoenix LiveView, while optionally uploading dashboards and annotating application lifecycle events through Grafana’s HTTP API. The example enables Fly.io metrics on port 4000 at /metrics and configures PromEx dashboards for a Todo List application. The stated result is an instrumented application with dashboards and annotations in minutes, while monitoring can also support rollback decisions, stress-test analysis, prioritization, and capacity planning.


### [Sentence Transformers in the Hugging Face Hub](https://yomu.fyi/post/sentence-transformers-in-the-hugging-face-hub.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Omar Sanseviero, Nils Reimers
- Published: Jun 28, 2021
- Type: Announcement

Sentence Transformers has collaborated with the Hugging Face Hub as part of its v2 release to make sentence, paragraph, and image embedding models readily accessible. The Hub now hosts over 90 pretrained models supporting more than 100 languages for use cases like semantic search and multilingual zero-shot classification. Alongside direct model loading in minimal code, the platform provides interactive Hub widgets for feature extraction and sentence similarity computation. Developers can execute inference programmatically through Inference API endpoints. Additionally, the library supports direct model uploading to the Hub via save\_to\_hub, which automatically generates a model card detailing layer architecture and dual usage with Sentence Transformers and Transformers.


### [Let’s Encrypt x Shopify: Securing the Web 4.5 Million Domains at a Time](https://yomu.fyi/post/let-s-encrypt-x-shopify-securing-the-web-4-5-million-domains-at-a-time.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 24, 2021
- Type: Explainer

Shopify describes how its SSL team moved merchants’ stores to HTTPS and ultimately secured more than 4.5 million domains with Let’s Encrypt. The effort began against certificate providers whose APIs were not designed for fully automated provisioning: errors often arrived as human-readable messages, and imposed throttling could have stretched initial issuance to about 100 days. Shopify tested the ACME protocol, valued its standardized specification and open-source implementation, and first added Let’s Encrypt as a backup certificate authority to improve emergency key or certificate-chain rotation. The team rolled out that integration in a few hours, compared with months for earlier providers, then made Let’s Encrypt its primary authority after finding it reliable. The account presents this change as evidence that ACME can make certificate automation more responsive and allow certificate authorities to be added without redesigning Shopify’s infrastructure.


### [Rate Limiting GraphQL APIs by Calculating Query Complexity](https://yomu.fyi/post/rate-limiting-graphql-apis-by-calculating-query-complexity.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 22, 2021
- Type: Explainer

Shopify’s GraphQL Admin API uses calculated query costs to address two limitations of request-based rate limiting: identical credits for responses of different sizes and equal treatment of reads and side-effecting writes. The server statically analyzes each query, assigning one point to objects and interfaces or unions, zero to scalars and enums, two plus the requested object count to connections, and 10 to mutations. Clients receive 50 points per second up to a 1,000-point limit, while responses expose requested and actual costs; when fewer connection records are returned, the difference is refunded. The post reports a linear correlation between calculated query complexity and execution time, giving Shopify more predictable load for infrastructure scaling and a way to identify performance outliers.


### [10 Lessons Learned From Online Experiments](https://yomu.fyi/post/10-lessons-learned-from-online-experiments.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 17, 2021
- Type: Explainer

After completing ten online experiments over the past year, the author describes recurring statistical and implementation traps that can distort experiment results. The discussion stresses choosing independent randomization units, examining whether the analysis unit matches the randomization unit, and using simulations to test suspected false-positive effects before changing an experiment plan. It also shows how request-level traffic analysis and close collaboration with engineering can avoid unnecessary implementation work, while generous logging, A/A tests, and investigation of sample ratio mismatches can expose redirect failures or qualification bugs. The author reports that switching one experiment’s analysis unit from session to user changed its result from negative to positive, and argues for monitoring results without stopping early, investigating anomalies, and slicing averages across important user segments.


### [Querying Strategies for GraphQL Clients](https://yomu.fyi/post/querying-strategies-for-graphql-clients.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 15, 2021
- Type: Tutorial

GraphQL clients can face slower, larger queries and difficult roll-outs as product screens gain features and data. The Orders & Fulfillments teams examined these issues while targeting a sub-one-second mobile page load on a reliable network. Starting with a product-list query that loads 100 products, the post recommends pagination with page-size/index controls and a hasNextPage field, plus performance tripwires to track loading time. It describes @include and @skip directives, runtime query construction, and chained queries for feature flags or parameters, while warning that chained requests add latency. For growing screens, it separates unrelated filters, permissions, and banners into parallel queries, enabling independent scaling, partial rendering, and less redundant pagination work, while cautioning that server load and partial-data state require monitoring and possible refactoring.


[Newer posts](https://yomu.fyi/posts/page/220.md) · [Older posts](https://yomu.fyi/posts/page/222.md)