Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,634 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Carving The Scheduler Out Of Our Orchestrator
Fly.io describes why it is carving scheduling out of its existing Nomad-based orchestrator and replacing it with a simpler system centered on flyd and AppsV2. It first reduces orchestration to a client-server supervisor, then explains centralized scheduling: workers register resources, while a server filters candidates by constraints and ranks them. The proposed design avoids jobs sitting pending or being placed in nearby but incorrect regions; requesting VMs in MAD yields MAD capacity or nothing. Policy-heavy behavior such as deployment spares, crash recovery across workers, and regional decisions moves into flyctl and other platform components rather than the scheduler. Fly Machines already exposes APIs to reserve, start, and stop individual VMs, with Nomad still used in January 2023 and flyd expected for deployments in December 2023.
Performance Testing At Scale—for BFCM and Beyond
Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.
From Farmer to Security Engineer: How Dev Degree Helped Me Find My Dream Job
Cailyn Edwards recounts how she moved from leaving high school and running an organic farm to a career in infrastructure security through Shopify’s Dev Degree program. Before entering the program, she completed a horticulture co-diploma, finished high school and the required math courses, and pursued computer science while needing to pay her mortgage and maintain work-life balance. Dev Degree’s training path and rotating placements exposed her to computer science fundamentals, pair programming, Ruby on Rails on the orders team, infrastructure security, and network foundations, leading her to discover a strong interest in cyber security and cloud infrastructure. After graduating in September 2021, she joined Infrastructure Security full-time and later reports projects, internal security talks, a promotion to Senior Infrastructure Engineer, and Kubernetes contributions including a contributor award.
Making Your React Native Gestures Feel Natural
The post describes techniques for making draggable elements in a React Native Sheet component feel more natural during open-and-close gestures. A gesture handler updates yPosition as the user’s finger moves, then chooses an open or closed destination when the drag ends, but a position-only decision can ignore a fast flick and a fixed 600 ms animation ignores distance. The revised interaction adds event.velocityY, weighted by a multiplier, to the destination decision and uses withSpring so distance and release velocity affect the animation. It also divides overdrag distance when the Sheet is already open, creating resistance that communicates the boundary, and reports that flicks become easier while movement feels more realistic; the examples are ultimately recommended for testing on a real device.
Ruby 3.2’s YJIT is Production-Ready
Ruby 3.2 makes YJIT, Shopify’s just-in-time compiler for CRuby, substantially more suitable for production after its experimental Ruby 3.1 release. The project was ported from C99 to Rust for stronger safety guarantees and better abstractions, while memory work reduced overhead to approximately one third of the Ruby 3.1 level through metadata optimization, machine-code garbage collection, and lazy page allocation. YJIT 3.2 speeds railsbench by about 38% over the Ruby 3.2 interpreter, or 57% over the Ruby 3.1.3 interpreter, and adds native ARM64 support for Apple M1/M2, AWS Graviton, and Raspberry Pi. Shopify has deployed it across its Storefront Renderer infrastructure, measuring real-world speedups of 5% to 10%, while identifying memory footprint and method-call optimization as continuing priorities.
How Good Documentation Can Improve Productivity
Internal documentation is presented as a productivity and scaling tool for engineering organizations, especially as people move between teams, projects, or companies and remote work makes writing a default channel for knowledge sharing. Shopify’s approach combines standardized writing practices with three primary locations: Markdown microsites built with Docusaurus, internal GitHub for issues, READMEs, setup guidance, and searchable code, and the Vault, a GitHub- and Markdown-powered source of contextual information. The article argues that documentation should be treated as a product: it should be searchable, user-friendly, kept fresh, integrated into CI/CD, and hosted where engineers already work. It also recommends making writing routine through small contributions, paired writing, peer review, and audience-aware context, concluding that accessible documentation speeds onboarding, reduces duplicated problem-solving, supports asynchronous and cross-team collaboration, and improves developer literacy.
Catching Anomalies Early in Mobile App Releases
Nextdoor deploys weekly mobile updates across iOS and Android using phased rollouts starting at 1% adoption to minimize blast radius. Standard aggregate observability cannot reliably detect early regressions because early adopters skew significantly more active than average users, obscuring silent drops in overall metric noise. To overcome this selection bias, Nextdoor built App Release Anomaly Detection using difference-in-differences causal inference. The methodology verifies pre-adoption trends with standard deviation bounds and fits a linear regression model to estimate the effect against an unobserved counterfactual. During a rollout of iOS version v1.234.5, this model identified a statistically significant app session decline at 1% adoption, isolating the regression 10 days earlier than week-over-week metrics.
From Ruby to Node: Overhauling Shopify’s CLI for a Better Developer Experience
Shopify rewrote its unified CLI from Ruby to TypeScript running on Node after growing dependencies on Node tooling, Hydrogen’s preference for npm-managed dependencies, and a broader fit with web technologies. The rewrite aims to reduce system requirements, improve terminal consistency, and make contribution easier, while using a modular architecture of domain NPM packages built on @shopify/cli-kit. The team established shared code, UI, convention, and principle foundations, addressed cross-OS support, used TypeScript for stronger contracts, and applied Cucumber for end-to-end testing. The Node CLI unified and streamlined app development and added extensibility such as functions, but Theme commands still use Ruby and removing that dependency remains future work.
Building Obsidian, Tinder’s Design System
As Tinder scaled over a decade, maintaining visual consistency across its client applications became unsustainable and increased engineering overhead. To address this, the UI Platform team developed Obsidian, a unified design system combining design tokens, reusable components, and documentation. Designers define base and context tokens in Figma using the Tokens Studio plugin, which are then transformed by Style Dictionary into native artifacts for iOS, Android, and Web. Web developers integrate these tokens using Atomizer to generate static stylesheets, while a Zeroheight documentation portal and a custom token lookup tool assist legacy migrations. The ongoing migration has streamlined collaboration between designers and engineers, resolved styling ambiguities, and improved color contrast accessibility across platforms.
Supabase Beta December 2022
Supabase's December 2022 beta update recaps Launch Week 6, Community Day, Postgres ecosystem work, and other recent releases. The announcements include redesigned documentation rebuilt with Next.js; Storage v2 features for image resizing, webhooks, and a Smart CDN; and multi-factor authentication with optional enforcement through Row Level Security. Supabase Wrappers extend PostgreSQL Foreign Data Wrappers to query external databases, systems, and third-party APIs using SQL, while The Vault enters beta for encrypted secrets and stored data. The update also announces custom domains, point-in-time recovery, SSO on the dashboard in preparation for SAML, and ecosystem projects including pg_graphql v1.0, a PostgREST 11 pre-release, and an experimental pg_crdt extension.
Graph service platform
Grab's GrabDefence team required a dedicated graph infrastructure to proactively identify mobile fraud patterns, such as multiple accounts operating on shared physical devices and suspicious financial loops. To address this, the team built a four-layer Platform as a Service that encapsulates graph database operations behind uniform RESTful APIs for OLTP search and OLAP analysis. The architecture utilizes Amazon S3 for raw data files, Amazon Neptune for graph storage, DynamoDB for schema and metadata configurations, and Kafka for streaming ingestion. When users trigger data loading tasks, the service validates entity attributes against schemas stored in DynamoDB before importing records into Neptune. This infrastructure allows investigators to traverse adjacent account IDs and visualize complex entity relationships without managing underlying database runtimes.
Reliving Your Happiest HTTP Interactions with Ruby’s VCR Gem
Ruby’s VCR library records HTTP interactions and replays them in test suites, verifying requests and returning predictable responses; the post presents it as a way to manage complex WebMock stubs and multi-step or unreliable API behavior. It contrasts recordings from a live API with manually authored stubs: cassettes preserve observed behavior at recording time but are harder to change and do not update when the API changes. The guidance covers VCR.use_cassette, configurable request matching, YAML cassettes, before_filter logging, filter_sensitive_data, and safer recording modes such as :once and :none instead of :new_episodes. It recommends VCR for integration-focused tests involving unpredictable sequences, multiple calls, or troublesome APIs, while favoring WebMock or decomposition for simple calls and specific states; cassette isolation and inspection add debugging value but can hide interactions.
WAL Mode in LiteFS
LiteFS v0.3.0 adds support for SQLite’s write-ahead log (WAL) alongside the previously supported rollback journal, allowing its distributed file system to replicate transactions from either journaling mode. The post explains how rollback journals and WAL differ, then shows how LiteFS observes filesystem calls, identifies changed pages, and emits a common LTX change-set format; in WAL mode, the final frame’s commit field marks transaction completion. Replica nodes validate LTX files, acquire SQLite-compatible locks, and apply the changes, while the new litefs import command transactionally replaces a database for bootstrapping or snapshot reversion. Version 0.3.0 also replaces previous-page checksum tracking with a per-page checksum map, costing about 3 MB of RAM per gigabyte and roughly 5 ms of CPU time per gigabyte per commit, and adds a trace log whose enabled use can double disk I/O.
Launch Week 6 Hackathon Winners
Supabase announced the winners of its sixth hackathon, held during Launch Week 6 after ending 2022 with many feature releases. Projects were judged across categories including Best Overall, Most Fun/Interesting, Best Flutter, Best Storage, and Best Edge Functions, with Deno participating in judging the edge category. Depulso, a CLI and web app for static site deployments with per-deployment .depulso.site subdomains and SSL, won Best Overall, while Supa SQL App was runner-up for converting human language into SQL queries. Other winners included HTTP Sense, a reverse proxy and network monitoring tool; supaquiz, a realtime multiplayer trivia game; Lookout, a metadata-based image discovery app; and Vista, a video transcoder that adds customizable subtitles from audio. Winners received keyboards or swag, and a Discord event was scheduled for January 5 at 7:00 AM PST.
Monte Carlo Simulations: Separating Signal from Noise in Sampled Success Metrics
Sampled success metrics can be difficult to interpret when the underlying population is too expensive or time-consuming to measure completely. The method presents a Monte Carlo simulation for estimating how sample size and an assumed trend affect the reliability of Shopify’s sampled “shop issue rate,” which estimates requirement violations for an average installed app. Its workflow establishes simulation metrics such as 1mDO, 2mDO, and 1qDO, measures MAPE, defines a sampling distribution, and runs repeated simulations across parameter combinations using Python and pandas. For the example, the issue count is modeled with a Poisson distribution, while sample size, lambda, and monthly decrease are varied to quantify uncertainty and collection cost. With 100 audits per month, an initial mean of 10 issues, and a 5-percent monthly decrease, the simulations predict monthly decreases 83 percent of the time and decreases over two months 97 percent of the time.
React Native Skia: A Year in Review and a Look Ahead
React Native Skia’s first year produced an initial alpha and 89 additional releases, evolving a React Renderer and Skia integration for declarative 2D graphics across iOS, Android, the Web, and Node.js testing. The project changed direction after identifying two bottlenecks: dependence on the JavaScript thread and the overhead of thousands of JSI allocations in large drawings. Skia DOM addresses them with a declarative drawing state that recomputes only necessary changes and executes drawings on the UI thread, with C++ implementations on iOS and Android and a JavaScript implementation on React Native Web. CanvasKit compatibility, Expo integration, Metal on iOS, and OpenGL on Android and the Web support the cross-platform design, while extension points enable modules such as Skottie and camera-frame effects. The library’s tests compare rendered examples across platforms, and planned work includes JavaScript-worklet and Reanimated animation integration plus advanced text layouts through Skia’s paragraph module.
Migrating our Largest Mobile App to React Native
Shopify’s largest mobile app, Shopify Mobile, had 300 screens per platform and more than six years of development, making a full rewrite impractical because it could take at least three years and halt feature work. The team first adopted React Native incrementally, but migration was projected to take four to five years while requiring three design-system implementations, three architectures, and costly native interoperability. It then switched to “Iterative Porting”: engineers were trained through the five-milestone RN Accelerator, surfaces were inventoried and prioritized with a RICE-based score, and eligible features were migrated behind flags, tested, monitored for two weeks, then completed by removing the native version. The effort also introduced Native Routes, modal deeplink handling on iOS, and component-level Safe Area fixes; with root screens ported and infrastructure in place, the team reports that ports are accelerating, although substantial work remains.
Optimizing Ruby’s Memory Layout: Variable Width Allocation
CRuby’s original garbage collector stores most Ruby objects in fixed 40-byte RVALUE slots, leaving differently sized data partly unused or forcing additional allocations through malloc. Shopify’s Variable Width Allocation project adds dynamic-sized allocation APIs and five fixed-size pools—40, 80, 160, 320, and 640 bytes—while increasing heap pages from 16 KB to 64 KB. The design targets cache locality, malloc overhead, memory utilization, and future garbage-collector experimentation; a new GC.stat_heap API reports size-pool statistics, and an escape hatch can disable the feature at build time on Ruby 3.2. Benchmarks on development Ruby 3.2.0 showed gains in most workloads, including 9.8% for psych-load, 5.3% for hexapdf, and 2.1% for railsbench, while mail was 1.7% slower.
Year in Review 2022: Tenderlove's Ruby and Rails Reflections and Predictions
Aaron Patterson’s 2022 review surveys developments in Ruby and Rails, pairing runtime performance work with tools intended to improve developer productivity. YJIT’s compiler was rewritten in Rust, enabling ARM and Apple Silicon support while producing faster machine code; Variable Width Allocation adjusts object slot sizes to keep instance variables embedded and improve data locality. Ruby 3.2’s Object Shapes use property order as a class-independent inline-cache key, while Ruby LSP adds editor services and Error Highlight adds column-level diagnostics that now appear on Rails error pages. Patterson predicts—more accurately, hopes—that languages, frameworks, and development environments will become more interconnected, providing better suggestions, context, and assistance so developers can spend less time coding and more time solving problems.
Automatically Rotating GitHub Tokens (So You Don’t Have To)
Shopify needed automated GitHub workflows that could clone from private repositories, update organization-level projects, and trigger downstream workflows but could not use GITHUB_TOKEN. Personal access tokens supplied those capabilities, yet introduced developer churn, user-attributed actions, and no organization-enforced rotation or expiry. The proposed system uses GitHub Actions and a GitHub App to generate repository secrets that rotate automatically while preserving PAT-like references in existing workflows. Its serverless design delegates scheduling, managed infrastructure, approvals, secret management, auditing, and break-glass operation to GitHub, while a custom TypeScript action encrypts and writes generated secrets. Prototyping exposed that parallel one-second jobs could accrue multiple billable minutes, prompting sequential execution in a consolidated workflow; the stated goal is improved resilience to developer churn and reduced leaked-secret blast radius.
Top topics
- Architecture832
- Performance814
- LLMs658
- Machine Learning620
- Open Source563
- AI Agents549
- AI465
- Developer Experience448
Weekly digest
The best engineering reads, once a week.