---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Convert Transformers to ONNX with Hugging Face Optimum](https://yomu.fyi/post/convert-transformers-to-onnx-with-hugging-face-optimum.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Jun 22, 2022
- Type: Tutorial

Exporting Hugging Face Transformers models to the ONNX format can be accomplished through three different abstraction levels. The low-level approach utilizes torch.onnx.export, requiring manual specification of dummy inputs, input names, output names, opset versions, and dynamic axes configurations. At the intermediate level, the transformers.onnx package simplifies the conversion process by relying on FeaturesManager and prebuilt configuration objects to handle dynamic axis definitions automatically. The high-level method uses Hugging Face Optimum classes such as ORTModelForSequenceClassification by setting the from\_transformers flag to True inside from\_pretrained. This Optimum export leverages transformers.onnx internally and produces a model ready for immediate inference execution or integration into pipelines.


### [Python data loading with Supabase](https://yomu.fyi/post/python-data-loading-with-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Jun 17, 2022
- Type: Tutorial

Supabase offers a PostgreSQL-based backend-as-a-service platform alongside an open-source Python SDK for automating CRUD operations and data-intensive tasks. After spinning up a new project in the Supabase dashboard, developers can configure relational schemas in the Table Editor by establishing foreign key constraints between tables like Vendor and Product. Authentication for the SDK relies on passing project URL parameters and API keys, which can be securely stored and loaded through environment variables. Programmatic data population utilizes the supabase client library combined with the Faker package to generate realistic synthetic entries and push them directly to database tables using insert and execute calls. Populated records can subsequently be examined in real time through the dashboard Table Editor interface to verify schema integrity and data persistence.


### [How to make Rust leak memory (also: how to make it stop)](https://yomu.fyi/post/how-to-make-rust-leak-memory-also-how-to-make-it-stop.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 15, 2022
- Type: Incident / postmortem

fly-proxy, a Rust-based proxy, leaked memory because its obsolete custom HoneycombLayer retained child spans under a long-lived parent span. The post follows a request from browser input through DNS, TCP, TLS, HTTP/2, and asynchronous Rust components including an aya-built eBPF program, tokio, hyper, and tracing, before explaining how the leak surfaced. In the custom layer, every closed child span was added to its parent’s collection, so each request consumed a small amount of memory while a high-level async task remained alive. Removing that legacy code fixed the primary problem: memory usage barely exceeded 50MB during a load test, while remaining growth came from queues and latency data retained for processing rather than the identified leak.


### [Making Open Source Safer for Everyone with Shopify’s Bug Bounty Program](https://yomu.fyi/post/making-open-source-safer-for-everyone-with-shopify-s-bug-bounty-progra.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 15, 2022
- Type: Problem & solution

Shopify describes how a bug bounty report led to application fixes and Rails contributions aimed at making open-source software safer. The 2021 Shopify Experiments report showed that a leaked development-environment secret could be used to generate trusted-looking data and reach remote code execution in a local development environment. The root cause was ActiveSupport’s MessageEncryptor using Ruby’s unsafe Marshal serializer by default in Rails versions up to 7.0.X, allowing untrusted data to trigger deserialization risks. Shopify first configured JSON in its application, then helped change the default serializers for MessageEncryptor and MessageVerifier to JSON in Rails 7.1.X, with an upgrade path for existing applications. It also developed a patch that monitors Marshal load and dump calls, raises on unexpected gem-originated calls, and runs in local development and continuous integration to prevent new usage.


### [Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration](https://yomu.fyi/post/intel-and-hugging-face-partner-to-democratize-machine-learning-hardwar.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Julien Simon
- Published: Jun 15, 2022
- Type: Announcement

Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.


### [How We Built Shopify Party](https://yomu.fyi/post/how-we-built-shopify-party.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 14, 2022
- Type: Explainer

Shopify Party is a browser-based internal tool designed to make virtual hangouts more social and reduce video fatigue after Shopify moved to remote work. Its 3D spaces favor non-prescriptive play, letting participants explore and manipulate objects, join or leave at any point, and use a small control set rather than follow prescribed rules or win conditions. The initial version was built in three months with Unity 2020 and the Universal Render Pipeline, exported to WebGL, while Normcore synchronizes game state, provides VoIP and networked physics, and uses room time to keep autonomous objects aligned across clients. Performance work included third-party assets, GPU instancing for repeated models, simpler materials, and adjustable render resolution. The launch exceeded expectations, with hundreds of games played each week, and the tool is positioned as an option alongside video conferencing rather than a replacement.


### [Director of Machine Learning Insights \[Part 3: Finance Edition\]](https://yomu.fyi/post/director-of-machine-learning-insights-part-3-finance-edition.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Britney Muller
- Published: Jun 14, 2022
- Type: Explainer

Machine learning leaders in financial institutions face significant operational hurdles when integrating automated models into production environments. Practitioners must navigate complex legacy architectures, strict regulatory oversight, and privacy mandates tied to personally identifiable financial records. Integrating models often fails when organizations lack clear communication buffers, deploy models using improper prediction windows, or treat systems as opaque black boxes without verifying underlying mechanics. Despite these challenges, financial applications increasingly rely on machine learning for anti-money laundering compliance, automated fraud screening, and loan underwriting. Success in this regulated sector demands comprehensive evaluation across representative input spaces, robust ongoing model monitoring, and explainable decision outputs.


### [Logbook - 2022-06-10](https://yomu.fyi/post/logbook-2022-06-10.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 10, 2022
- Type: Announcement

Fly.io’s June 10, 2022 logbook lists platform, dashboard, deployment, and account-management changes for its globally distributed application hosting service. WireGuard peer synchronization now adds only peers whose state changed, while removing a global lock from proxy tracing instrumentation improves TLS handshaking and HTTP parsing under load; the post says these changes should speed userspace features, avoid API timeouts, and reduce response times. Deployments gain build-time secrets through fly deploy’s --build-secret flag, larger rolling-deploy batches that are expected to make multi-region releases 2–5x faster, and restored Turboku webhook redeployment from Heroku. Dashboard updates include intelligent longpolling fallback, visible certificate deletion controls, monitoring links, safer app deletion confirmation, and corrected metrics and coupon visibility, while the Jupyter launcher now requires a credit card after abuse.


### [8 Data Conferences Shopify Data Thinks You Should Attend](https://yomu.fyi/post/8-data-conferences-shopify-data-thinks-you-should-attend.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2022
- Type: Explainer

Shopify’s data scientists and engineers curated eight upcoming data conferences for 2022, covering virtual, hybrid, and in-person formats. The selection supports learning and development by helping practitioners hear from peers about applications, techniques, use cases, and opportunities in the wider data community. Hybrid options include Data + AI Summit, Transform, RecSys, and ODSC West, with topics spanning analytics, machine learning, open source technologies, recommender systems, MLOps, natural language processing, and big data. In-person choices—PyData London, KDD, Re-Work Deep Learning Summit, and Crunch—feature tutorials, research, business applications, AI challenges, data-team practices, and machine learning at scale, with locations including London, Washington D.C., Toronto, and Budapest.


### [How facial recognition technology keeps you safe](https://yomu.fyi/post/how-facial-recognition-technology-keeps-you-safe.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kai Feng Tee
- Published: Jun 9, 2022
- Type: Explainer

Grab utilizes facial recognition technology across its platform for driver authentication, passenger verification, and digital electronic Know Your Customer (e-KYC) processes. The core pipeline consists of image preprocessing through face detection and alignment, anti-spoofing checks, feature extraction into high-dimensional vector embeddings, and downstream verification or search. To counter spoof attacks like screen replays, synthetic moiré patterns are generated and cropped face patches are used during training and inference to focus on local structures rather than global semantic noise. Face verification challenges involving shallow ID datasets and masked faces are resolved using semi-Siamese training architectures and masked data augmentations.


### [The Journey to Cloud Development: How Shopify Went All-in on Spin](https://yomu.fyi/post/the-journey-to-cloud-development-how-shopify-went-all-in-on-spin.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 9, 2022
- Type: Problem & solution

Shopify’s development environment, the Ruby-based dev tool and xhyve-backed railgun VM, no longer handled the growing number of repositories, services, integrations, and resource demands around its monoliths. Developers needed more services, faced sluggish code-build-test loops, and encountered costly state changes when tophatting collaborators’ work. The Environments team concluded that development environments needed cloud-like elasticity, then tested an automatically configured local Kubernetes cluster and easy creation of Google Compute Engine VMs; the source also describes Isospin and Spin as the resulting direction. Local Kubernetes did not gain broad organic traction, while the current Isospin implementation is described as the team’s best iteration for developers working on majestic monoliths. The team plans to evolve it into an application-development library built on dev, add dependency inference, and address the tension between persistent environments and Kubernetes pod relocation.


### [Enhancing Atlas Online Archive With Data Expiration And Scheduled Archiving](https://yomu.fyi/post/enhancing-atlas-online-archive-with-data-expiration-and-scheduled-arch.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Published: Jun 7, 2022
- Type: Announcement

Atlas Online Archive moves infrequently accessed data from a cluster to a MongoDB-managed object store while enabling unified queries across both tiers. A new data expiration feature lets users set a deletion threshold for archived data, configurable from 7 to 9,125 days through the Atlas UI or Admin API, and rules can be edited after creation; previously, the only way to remove data was deleting the entire archive. Once data is deleted from the archive, it cannot be recovered, so rules must be defined carefully. Scheduled archiving addresses performance concerns by allowing users to define a recurring window—daily, weekly, or monthly—with a minimum duration of two hours, replacing the previous five-minute interval that could overload clusters during peak times. Users can edit the archive rule to specify both when to archive and when to delete, and both features are in preview to improve operational efficiency for Atlas customers.


### [The Annotated Diffusion Model](https://yomu.fyi/post/the-annotated-diffusion-model.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Niels Rogge, Kashif Rasul
- Published: Jun 7, 2022
- Type: Tutorial

Denoising Diffusion Probabilistic Models generate data samples by learning to reverse a discrete-time forward diffusion process that incrementally adds Gaussian noise to inputs. The forward process adds noise across a predefined schedule of variance parameters up to a fixed number of steps, transforming data into an isotropic Gaussian distribution. Because the true reverse conditional probability distribution is intractable, a neural network optimizes the variational lower bound to approximate the distribution mean at each step while keeping the variance fixed. Sampling subsequently proceeds by iteratively denoising pure Gaussian noise through successive network evaluations to reconstruct a clean data sample. Although diffusion models demonstrate high generation quality, their primary drawback remains the requirement of multiple forward passes during inference compared to models like generative adversarial networks.


### [Deep Q-Learning with Space Invaders](https://yomu.fyi/post/deep-q-learning-with-space-invaders.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Thomas Simonini
- Published: Jun 7, 2022
- Type: Explainer

Tabular Q-learning fails to scale to complex environments like Atari games due to massive observation spaces that make maintaining discrete state-action tables impractical. Deep Q-learning addresses this limitation by using a neural network to approximate Q-values for each possible action from input states. To process visual inputs effectively and capture temporal motion, raw game screens are grayscaled, resized to 84x84 pixels, and stacked in groups of four consecutive frames before passing through convolutional and fully connected layers. The training process stabilizes learning through experience replay buffers, fixed target networks that update periodically, and Double DQN architectures that decouple action selection from target evaluation. Agents can then be trained on environments like Space Invaders using frameworks such as RL-Zoo without tabular memory constraints.


### [Graph concepts and applications](https://yomu.fyi/post/graph-concepts-and-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenxiang Lu
- Published: Jun 2, 2022
- Type: Explainer

Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.


### [Supabase Beta May 2022](https://yomu.fyi/post/supabase-beta-may-2022.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Jun 1, 2022
- Type: Announcement

Supabase announced several platform updates and performance improvements across its database, hosting, and developer tooling ecosystem for May 2022. Developers managing preview deployments on Jamstack platforms like Vercel and Netlify can now configure wildcard redirect domains such as \*.mydomain.com/welcome. For serverless workflows, Edge Functions support a -no-verify-jwt CLI deployment flag to facilitate direct invocation via external webhooks without mandatory token validation. In addition, Supabase exposed a Prometheus-compatible metrics endpoint to all users for real-time project monitoring and alerting. Database security policies also gained a dedicated Target roles configuration, replacing older role check expressions and cutting query execution time from 46 seconds to 7 milliseconds in one customer deployment.


### [Logbook - 2022-06-01](https://yomu.fyi/post/logbook-2022-06-01.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 1, 2022
- Type: Announcement

Fly.io’s June 1, 2022 logbook records a week of platform, tooling, documentation, and dashboard changes, with forum community feedback driving several logged updates. The release adds rolling behavior to fly restart, extends fly status --all’s failed-VM visibility from two days to seven, supplies Laravel and Node.js examples, and introduces the first official Terraform provider with Machines API parity. Release commands receive 512MB of swap without increasing normal app RAM, while preview GraphQL endpoints fetch volumes, IPs, and certificates directly by ID to support Terraform imports and stateful tooling. Dashboard work adds Livebook upgrades, clearer billing and typography, individual Machine event tables, and fixes for IPv6 copying, organization-plan changes, cross-tab logout, billing URLs, misleading volume usage, and two-factor input limits.


### [Building a Form with Polaris](https://yomu.fyi/post/building-a-form-with-polaris.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 1, 2022
- Type: Tutorial

Shopify’s Polaris design system provides design and content guidelines and reusable React components for Shopify admin and third-party apps. The tutorial builds a basic product form with Title and Description fields, a Save button, and top navigation, while introducing React state, events, and handlers. It uses Form, FormLayout, TextField, Page, and Card, then wraps the application in the required AppProvider after MissingAppProviderError appears; AppProvider shares global settings, including translations, across child components. The form is configured as controlled, with separate useState values and onChange handlers for each field, an onSubmit path connected to the Page component’s onAction prop, and clearing logic after submission.


### [To Thread or Not to Thread: An In-Depth Look at Ruby’s Execution Models](https://yomu.fyi/post/to-thread-or-not-to-thread-an-in-depth-look-at-ruby-s-execution-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 31, 2022
- Type: Explainer

Threaded servers such as Puma and thread-based Sidekiq are popular Ruby deployment choices because threads can increase throughput without a proportional increase in memory. The post compares process and thread models using a memory model that separates static application memory from per-request processing memory, then refines it with Copy on Write (CoW), where forked processes share pages until writes force copies. It explains why RSS can double-count shared pages, recommends PSS and Linux’s /proc/$PID/smaps\_rollup for more meaningful measurements, and gives a production example in which about two-thirds of static memory is shared. The conclusion is metric-dependent: preload applications to improve CoW, prefer fewer larger containers when appropriate, consider Unicorn for reliable request timeouts, and start web workloads at two threads unless substantial I/O wait justifies more; job processors may benefit from higher counts.


### [Automated Experiment Analysis - Making experimental analysis scalable](https://yomu.fyi/post/automated-experiment-analysis-making-experimental-analysis-scalable.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Albert Cheng
- Published: May 30, 2022
- Type: Problem & solution

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.


[Newer posts](https://yomu.fyi/posts/page/207.md) · [Older posts](https://yomu.fyi/posts/page/209.md)