Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,535 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Supabase Beta May 2022
Supabase announced several platform updates and performance improvements across its database, hosting, and developer tooling ecosystem for May 2022. Developers managing preview deployments on Jamstack platforms like Vercel and Netlify can now configure wildcard redirect domains such as *.mydomain.com/welcome. For serverless workflows, Edge Functions support a -no-verify-jwt CLI deployment flag to facilitate direct invocation via external webhooks without mandatory token validation. In addition, Supabase exposed a Prometheus-compatible metrics endpoint to all users for real-time project monitoring and alerting. Database security policies also gained a dedicated Target roles configuration, replacing older role check expressions and cutting query execution time from 46 seconds to 7 milliseconds in one customer deployment.
Logbook - 2022-06-01
Fly.io’s June 1, 2022 logbook records a week of platform, tooling, documentation, and dashboard changes, with forum community feedback driving several logged updates. The release adds rolling behavior to fly restart, extends fly status --all’s failed-VM visibility from two days to seven, supplies Laravel and Node.js examples, and introduces the first official Terraform provider with Machines API parity. Release commands receive 512MB of swap without increasing normal app RAM, while preview GraphQL endpoints fetch volumes, IPs, and certificates directly by ID to support Terraform imports and stateful tooling. Dashboard work adds Livebook upgrades, clearer billing and typography, individual Machine event tables, and fixes for IPv6 copying, organization-plan changes, cross-tab logout, billing URLs, misleading volume usage, and two-factor input limits.
Building a Form with Polaris
Shopify’s Polaris design system provides design and content guidelines and reusable React components for Shopify admin and third-party apps. The tutorial builds a basic product form with Title and Description fields, a Save button, and top navigation, while introducing React state, events, and handlers. It uses Form, FormLayout, TextField, Page, and Card, then wraps the application in the required AppProvider after MissingAppProviderError appears; AppProvider shares global settings, including translations, across child components. The form is configured as controlled, with separate useState values and onChange handlers for each field, an onSubmit path connected to the Page component’s onAction prop, and clearing logic after submission.
To Thread or Not to Thread: An In-Depth Look at Ruby’s Execution Models
Threaded servers such as Puma and thread-based Sidekiq are popular Ruby deployment choices because threads can increase throughput without a proportional increase in memory. The post compares process and thread models using a memory model that separates static application memory from per-request processing memory, then refines it with Copy on Write (CoW), where forked processes share pages until writes force copies. It explains why RSS can double-count shared pages, recommends PSS and Linux’s /proc/$PID/smaps_rollup for more meaningful measurements, and gives a production example in which about two-thirds of static memory is shared. The conclusion is metric-dependent: preload applications to improve CoW, prefer fewer larger containers when appropriate, consider Unicorn for reliable request timeouts, and start web workloads at two threads unless substantial I/O wait justifies more; job processors may benefit from higher counts.
Automated Experiment Analysis - Making experimental analysis scalable
Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.
How Mike Lyndon is using Supabase to accelerate development of AllPullTogether
A collection of case studies outlines how diverse engineering teams adopt Supabase across various infrastructure, authentication, and database workloads. Companies like Good Tape and Shotgun achieved substantial cost reductions between 60% and 83% by migrating to Supabase managed Postgres and authentication services. For security and data access control, Quilia reduced development time by 75% leveraging Supabase Data API alongside Row Level Security features. In machine learning applications, organizations such as Firecrawl, Berri AI, and Quivr utilize Supabase Vector for PostgreSQL vector embeddings rather than standalone vector databases. Additional migrations include Mobbin transferring 200,000 users from Firebase for authentication, while teams like Replenysh implemented one-time password workflows in under 24 hours.
Logbook - 2022-05-26
The May 26, 2022 changelog records product, reliability, documentation, and web UI changes made while work continued toward the Fly Machines launch. Users can configure proxy protocol v2 headers, while Fly HA Postgres instances become read-only and fail health checks at 90% volume usage, creating time to respond before full disks cause crashes. The GraphQL API now names missing apps directly, API telemetry is more selective, and fixes address certificate and volume deletion errors, region-code mutation, DNS validation context, and delayed organization invitations. Documentation covers the Machines API, its reference and FaaS usage, fly.toml updates, and PlanetScale Portals, while redesigned dashboards, app and organization pages, paid plans, documents, and app-level views update the web interface.
Implementing Equality in Ruby
Ruby has several equality and comparison mechanisms, so custom classes need explicit implementations when instance identity is not the intended meaning. The post distinguishes value objects, whose equality depends on all attributes, from entities, whose equality depends on an explicit identifier; its Point and Employee examples also use class checks, while an entity with a nil ID is unequal to other entities. It defines reflexivity, symmetry, and transitivity as properties to preserve, notes NaN as an exception to reflexivity, and explains #equal?, #eql?, #hash, and #===. For ordered values, #<=> returns -1, 0, 1, or nil, while Comparable supplies relational operators and useful methods including #min, #sort, #between?, and #clamp.
Graphcore and Hugging Face Launch New Lineup of IPU-Ready Transformers
Graphcore and Hugging Face expanded the range of machine learning modalities and tasks available in Hugging Face Optimum. Developers can now access ten transformer models optimized for Graphcore IPUs across natural language processing, speech, and computer vision. The available architectures include BERT, ViT, GPT-2, RoBERTa, DeBERTa, BART, LXMERT, T5, HuBERT, and Wav2Vec2, complete with IPU configuration files and ready-to-use pre-trained or fine-tuned weights. The integration supports the Bow IPU processor, which uses 3D Wafer-on-Wafer stacking to achieve up to 350 teraFLOPS of AI compute. Optimum also integrates with the Poplar SDK 2.5, enabling compatibility with frameworks such as PyTorch, TensorFlow, Docker, and Kubernetes.
Introducing Pull Requests and Discussions 🥳
Hugging Face Hub released pull requests and discussions under a new community tab for all repository types, including models, datasets, and Spaces. This feature enables community members and repository owners to interact directly, propose model card updates, report biased generations, and contribute files. The pull request implementation departs from traditional Git hosting platforms by avoiding forks and standard branches. Instead, contributors push to custom ref branches stored directly on the source repository to bypass creating heavy forks for large machine learning artifacts. Furthermore, the Hub treats discussions and pull requests without hard distinctions, listing them together to streamline collaboration across machine learning assets.
Fly Machines: an API for fast-booting VMs
Fly Machines are virtual machines exposed through a fast REST API, with boot times of about 300ms, intended to support cost-effective standby capacity and scale-to-zero applications. Creation is deliberately separated from starting: it performs a centralized database preflight, reserves host capacity through NATS, records the machine, and prepares the OCI image locally. A later start request goes to the nearest regional API, which sends a direct message to the known host, avoiding database coordination and image pulls; examples cite roughly 10ms locally and 150ms across regions. Processes can exit when idle to leave machines intact for clean restarts, while fly-proxy can start them for incoming HTTP requests. The design trades simplicity for hardware pinning, so capacity or host failures may require fallback machines and redundancy; stopped machines currently incur image-storage charges, such as about $0.15 per month for a 1GB Docker image.
Our Favorite Moments From RailsConf 2022
RailsConf 2022 returned to an in-person format at Portland’s Oregon Convention Center after a two-year gap, with three days of keynotes, tutorials, and talks on technical and workplace topics. The 16th edition announced three new Rails Core team members, including Shopify’s Jean Boussier, and two new committers, while recurring themes included company investment in Rails, security, Active Records, “build vs buy,” diversity, and burnout. Shopify’s 15 Rails speakers contributed sessions and keynotes, and the recap gathers highlights on open-source staffing, long-term maintenance, developer wellbeing, Rails upgrades, supply-chain attacks, and modularization for “Big Rails.” Specific examples included regexp DOS risks, typosquatting and combosquatting, Rails upgrade automation, Ruby Archaeology, and the history of women in computer science. Because concurrent sessions and hallway conversations limited attendance, a virtual home edition and recordings were scheduled to make more talks available afterward.
Lessons Learned From Running Apache Airflow at Scale
Shopify describes operating Apache Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8 for data extraction, machine-learning training, Iceberg maintenance, and DBT modeling. In its largest environment, Shopify runs over 10,000 DAGs, averages over 400 tasks running at a given moment, and executes over 150,000 runs per day, while facing slow file access, metadata growth, ownership tracking, DAG-author control, traffic bursts, and resource contention. The authors describe replacing per-pod GCSFuse access with an in-cluster NFS volume synchronized to GCS, retaining GCS as the user-facing source of truth; they also use 28-day metadata retention, a YAML manifest, DAG policies, and scheduling controls including pools, priority weights, and Celery queues. These measures are presented as ways to improve file performance, limit database growth, identify owners, enforce constraints, and reduce workload interference, though the retention policy does not support durable history-dependent features such as long-running backfills.
Efficient Table Pre-training without Real Data: An Introduction to TAPEX
Conventional language model pre-training relies on massive natural text corpora and general-purpose objectives like masked language modeling, leaving a substantial gap when applied to structured table tasks. To bridge this gap efficiently, TAPEX (Table Pre-training via Execution) pre-trains a language model, such as BART, by learning a neural SQL executor over a synthetic corpus. The system pairs web tables with systematically sampled executable SQL queries and gathers target execution outputs via an off-the-shelf SQL engine. During downstream fine-tuning, the model consumes natural language questions and flattened tables to predict answers. TAPEX achieved new state-of-the-art results across four benchmark datasets, including WikiTableQuestions and TabFact, while achieving a pre-training speedup of nearly 50 times over TaBERT.
Asynchronous Communication is the Great Leveler in Engineering
After Shopify became a remote-first company in March 2020, globally distributed engineering teams had to replace office habits and spontaneous context-sharing with remote-focused ways of working. The article presents a spectrum of synchronousness, ranging from video calls and pair programming through chats, recorded video, email, written documents, wikis, and READMEs, with increasing permanence toward the asynchronous end. It argues for writing as a primary communication mode, replacing meetings where possible, recording and rebroadcasting events, documenting decisions, maintaining a single source of truth in the Vault, and defining team norms for overlap, deep work, code review, and shipping across time zones. It also cautions that not everything should become asynchronous: synchronous meetings, pair programming, social activities, and intentional in-person gatherings preserve human connectivity, while asynchronous practices provide flexibility, equal access to information, and durable archives.
An Introduction to Q-Learning Part 2/2
Q-Learning serves as a foundational off-policy reinforcement learning method for finding optimal action-value functions before advancing to deep reinforcement learning. The approach employs a tabular Q-function that updates state-action pairs at each step using temporal difference learning. Action selection balances exploration and exploitation via an epsilon-greedy strategy, where epsilon progressively decays as training proceeds. Updates compute a temporal difference target combining the immediate reward and the discounted maximum value of the subsequent state under a greedy policy. This discrepancy between the exploratory acting policy and the greedy updating policy classifies Q-Learning as an off-policy algorithm, enabling agents to learn optimal policies in environments like Frozen Lake and autonomous taxi navigation.
How Sempre Health is leveraging the Expert Acceleration Program to accelerate their ML roadmap
Sempre Health handles thousands of inbound patient text messages daily to manage medication adherence and prescription discounts. Although an existing rule-based system caught around 80% of inbound texts, the team required statistical machine learning to improve message parsing and reduce manual workload. Collaborating with Hugging Face through the Expert Acceleration Program, the engineers received guidance on labeling strategies, problem formulation, and model selection. They deployed a natural language processing pipeline into production to automatically classify and respond to incoming messages or initiate automated actions like refill requests. Since deployment, the new NLP system automatically handles nearly 20% of incoming messages that previously required operational support tickets.
Putting ethical principles at the core of the research lifecycle
Machine learning research risks introducing algorithmic biases, privacy violations, and malicious applications when ethical considerations are omitted from project lifecycles. To address these challenges, Hugging Face's multimodal learning group established an ethical charter at the inception of their multimodal project with input from ethics and governance experts. The charter defines content policies barring detrimental activities, the generation of personally identifiable information, and incautious applications in high-risk fields like medicine, law, and finance. Guiding values emphasize transparency, fairness through data and model bias reviews, self-critical curation, open reproducibility, and proper licensing attribution. Because ethical values like open data sharing and personal privacy can conflict, the team plans to iteratively evaluate trade-offs and track document revisions on GitHub.
Double Entry Transition Tables: How We Track State Changes At Shopify
Shopify needed a reliable way to count merchants using Shopify Balance over time, where usage requires both an active Shopify Balance account and an active Shopify account. The data team built accounts_transition_facts with double entry transition tables, a format that represents each state change as a -1 row for the previous state and a +1 row for the new state. Individual attribute tables use account_id, transition_at, and an index that breaks timestamp ties; the pipeline merges attributes, carries forward missing values, and generates net_change rows. Because net_change is additive, queries can sum it while filtering statuses, group results by date, or feed daily aggregates, while additional attributes can be added without rewriting existing SQL or PySpark. The initial build required substantial effort but produced a reporting structure intended to scale with Shopify Balance’s growing complexity.
An Introduction to Q-Learning Part 1
Value-based reinforcement learning centers on finding optimal policies indirectly through value functions. In these methods, practitioners define policy behavior manually, such as using greedy or epsilon-greedy policies, rather than training the policy directly. Value estimation relies on two primary formulations: the state-value function, which outputs expected returns from a state, and the action-value function, which evaluates state-action pairs. Because summing all future rewards across an entire trajectory is computationally demanding, algorithms adopt the Bellman equation alongside learning strategies like Monte Carlo or Temporal Difference (TD) learning. Monte Carlo updates value functions only after complete episodes using actual returns, whereas one-step TD learning bootstraps updates at each individual step using immediate rewards and discounted estimates of subsequent states.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.