Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,744 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Epsilon3 Self-Host Supabase To Revolutionize Space Operations
Epsilon3 digitizes paper procedures and telemetry data for high-stakes space missions and complex operational testing. Operating in heavily regulated environments requires strict security, ITAR compliance, and flexible deployment options ranging from AWS GovCloud to on-premises installations. To meet these demands without sacrificing developer velocity, Epsilon3 deployed a self-hosted Supabase configuration running against an Amazon RDS Postgres instance on AWS. This architecture gives the team real-time data streaming capabilities over Postgres so operators can track live procedure changes as they happen. In addition to reducing DevOps complexity, self-hosting Supabase enables Epsilon3 to leverage Postgres Row Level Security while ensuring full compliance across diverse deployment environments.
Supabase Community Day
Supabase announced a series of platform upgrades and open-source contributions centered around PostgreSQL and its broader tooling ecosystem. Every new project on the platform now deploys on PostgreSQL version 13.3, accompanied by marketplace images on Digital Ocean and AWS. PostgREST version 8.0 incorporates upstreamed performance enhancements, achieving up to 50% more throughput on GET requests by adopting prepared statements for SELECT queries and decreasing log verbosity. The release eliminates downtime during schema cache reloads and replaces the pg_listen requirement with standard NOTIFY commands for dynamic in-database configuration updates. Additionally, community-led efforts introduced beta releases of Flutter and Dart client libraries alongside a dedicated Discord server for developer discussions.
Supabase is now on Postgres 13.3
New Supabase projects now run on Supabase Postgres 13.3, transitioning the database platform forward from PostgreSQL 12.0. This release introduces features such as native UUID generation with gen_random_uuid, parallel index vacuuming, incremental sorting, smaller btree indexes, and extended statistics improvements. The underlying PostgreSQL binary was built from the ground up to enable Large System Extensions on AWS Graviton2 ARM instances, optimizing locking and synchronization performance. Alongside upgrading the underlying operating system to Ubuntu 20.04 for long-term support through 2025, the team integrated fail2ban to protect direct database connections on ports 5432 and 6543 against brute-force attacks. Finally, Supabase Postgres version numbering now mirrors upstream PostgreSQL major and minor versions to make tracking installed releases easier.
Protecting Personal Data in Grab's Imagery
Grab's KartaView platform collects geotagged street imagery across over 100 countries, requiring automated obfuscation of faces and licence plates to protect personal privacy. Because off-the-shelf solutions struggled with diverse global environments and equirectangular 360-degree camera formats, Grab built a custom machine learning pipeline. The system projects varied image formats into standardized planar views, applies a YOLOv4 object detection model to locate target regions, and transforms bounding coordinates back to the original imagery for blurring. Training the detector required iterative dataset updates to accommodate edge cases like face masks and mirror reflections, paired with offline view splitting and oversampling of scarce large bounding boxes. Assessments confirmed that obfuscating these regions had minimal negative impact on downstream map feature extraction services.
Supabase Launch Week II: The SQL
Supabase scheduled Launch Week II: The SQL for the week of Monday 26 July 2021, occurring four months after its initial launch week held in March. The five-day schedule opens by spotlighting the latest developments from the community alongside core ecosystem tools including Postgres and PostgREST. Tuesday introduces Storage Reloaded, representing four months of further engineering work following the Version 1 storage release. Wednesday focuses on authentication capabilities, specifically passwordless login features that build upon existing platform support for eight distinct OAuth providers. The week concludes with platform dashboard improvements alongside announcements regarding Supabase Functions updates, a community hackathon, and the official Supabase Swag Store.
A Guide to Running an Engineering Program
Shopify describes a playbook for running large engineering programs as platform complexity grows across products, tooling, and architecture. The framework begins with a Program Plan covering the problem statement, objectives, guiding principles, definition of done, risks, mitigation, staffing, scope, and a path to completion. Execution uses six-week cycles that revisit unfinished goals, regressions, emerging risks, and the planned route to completion; the described program spans 200 people, nine sub-organizations, and 92 projects. Supporting rituals include weekly updates, status checks, risk and escalation triage, cycle reviews, retrospectives, RFCs, and performance testing with Lua scripts orchestrated by Genghis. The post concludes that engineering program management must adapt to each organization and that a combination of scheduled and ad hoc rituals helped Shopify pursue its goals.
Processing ETL tasks with Ratchet
Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.
Last Mile Redis
Last Mile Redis presents a globally distributed caching design aimed at keeping backend responses under the 100-millisecond threshold by placing Redis near users. It argues that regional datasets overlap less than expected, making one Redis server per region a practical alternative to centralized sharding. The design combines standalone regional caches with a Dallas primary Redis and writable replicas configured with replica-read-only no, allowing local writes while replication distributes global changes. A DEL issued at the primary can therefore purge bad cache data across regions, while timestamped keys and expiration handle ordinary freshness; the same mechanism can also broadcast content or fan out queue messages.
Perspectives on React Native from Three Shopify Developers
Three Shopify developers—web developer AJ Robidas, Android developer Michelle Fernandez, and iOS developer Ash Furrow—describe how React Native changes mobile development for people from different backgrounds. They say its React-based model lets web developers build for iOS and Android, helps native developers create interfaces faster and reach a wider audience, and supports collaboration across web, Android, and iOS teams. The developers discuss surprises including React Native’s small set of building blocks, fast UI feedback, native bridge modules, existing SDK integration, and apps that can be difficult to distinguish from Swift or Objective-C software. They also identify challenges such as mobile-specific performance concerns across the RN Bridge, the Metro and Node ecosystem, and the large dependency stack. At Shopify, Visual Studio Code, command-line tooling, internal foundations, workshops, and cross-disciplinary teams support adoption.
Deep Learning over the Internet: Training Language Models Collaboratively
Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.
App Modularisation at Scale
Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.
Shopify-Made Patterns in Our Rails Apps
Shopify describes patterns and infrastructure adaptations used to keep its large Rails monolith maintainable as its organization and platform scale. Its dev command-line tool standardizes project setup, dependencies, services, environment variables, integrations, and command aliases across hundreds of active projects. Shopify Core uses pods to partition shops across independent databases and extends the shard concept across provisioning, deployment, load balancers, caching, and servers, while splitting code into domains without separate databases. The post also details safeguards including short-term Active Record migrations consolidated periodically into structure.sql, automated tests, a pod-aware maintenance task system, and a translation platform that commits translated strings through Git. Finally, Monorail imposes versioned JSON event schemas between Kafka producers and consumers and supports review and automated scrubbing of PII fields.
Welcome spaCy to the Hugging Face Hub
Hugging Face has integrated spaCy pipelines into the Hugging Face Hub to simplify the distribution, testing, and deployment of natural language processing models across the community. More than 60 canonical models from the spaCy 3.1 release are hosted in the spaCy organization, alongside discoverable community pipelines. Users can package and upload models directly through the command line using the spacy-huggingface-hub extension, which auto-generates required metadata and model cards. Hosted pipelines offer direct pip installation from live wheel URLs, interactive browser exploration, and production-ready HTTP endpoints through the Inference API. The platform provides immediate interactive widget support for named entity recognition components, with upcoming support planned for text classification and part-of-speech tagging pipelines.
Shopify's Path to a Faster Trino Query Execution: Infrastructure
Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.
High Availability by Offloading Work Into the Background
Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.
Deploy Hugging Face models easily with Amazon SageMaker
Hugging Face and Amazon introduced an inference solution integrating Hugging Face Transformers with Amazon SageMaker. The SageMaker Hugging Face Inference Toolkit and specialized Deep Learning Containers enable deployment of trained models or publicly available Hub models to managed production endpoints. Users configure endpoints with minimal code using the SageMaker Python SDK, standard pipelines, or custom inference scripts that override default behaviors. The architecture supports models stored in Amazon S3 as well as direct references via HF_MODEL_ID and HF_TASK environment variables. Deployments benefit from native AWS infrastructure features, including built-in monitoring, Identity and Access Management permission controls, and Virtual Private Cloud connectivity.
Reshaping Chat Support for Our Users
Grab transitioned from voice hotlines and third-party tools to an in-house native chat support system integrated into their CRM. The team validated the platform through an MVP and user shadowing to address session disconnections, agent context switching, and routing bottlenecks. To optimize support operations at scale, they introduced dynamic queue limits based on Little's law, machine learning autocomplete suggestions for agents, and duration timers with visual nudges. These enhancements reduced chat waiting times by 30%, unresponsive users by 7%, and overall chat handling duration by 22%.
Protecting reserved roles with PostgreSQL Hooks
Supabase required granting database customers the CREATEROLE privilege to manage custom roles while preventing them from dropping or altering backend service roles such as supabase_storage_admin. Because stock PostgreSQL lacks a native mechanism to define custom reserved roles, the team created the SupaUtils extension using PostgreSQL hooks. Loaded via shared_preload_libraries, SupaUtils intercepts utility statements by overriding the global ProcessUtility_hook function pointer. The extension inspects incoming statements like ALTER ROLE and DROP ROLE against a configurable list defined in postgresql.conf using DefineCustomStringVariable. If a targeted role matches the configured reserved roles, the hook raises an error, preserving backend infrastructure roles while permitting broader role administration.
Understanding GraphQL for Beginners–Part One
The article introduces GraphQL as an open-source data query and manipulation language for APIs, contrasting it with REST's fixed response shapes and multiple endpoint calls. GraphQL lets clients describe exactly which fields they need, returning that requested structure through a single endpoint rather than a cookie-cutter response. Using a Toronto Eagles example, the article explains how one GraphQL query can combine requests for a team's founding year, captain's name, and last championship, avoiding three separate RESTful trips whose latency can vary. It also uses burger and bank withdrawal analogies to explain over-fetching, under-fetching, flexible frontend iteration, and fewer endpoints, then points readers to a forthcoming Ruby on Rails implementation in part two.
Understanding GraphQL for Beginners–Part Two
The second part of this beginner tutorial uses a food-focused Ruby on Rails application to show how GraphQL fields expose database data. It reviews the graphql directory, where mutations hold data-changing classes, types define returned structures, and food_app_schema.rb defines available queries. It distinguishes root fields, which are GraphQL entry points that define response structure, from object fields, which represent attributes, then demonstrates generating Food and Nutrition types and adding an all_food root field with a non-null list return type. Through GraphiQL at localhost:3000/graphiql, it runs selective queries, adds nutrition to food, and introduces an id argument for retrieving one record; requests pass through graphql_controller’s execute method and return JSON responses.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents566
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.