Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,535 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
How Shopify Manages API Versioning and Breaking Changes
Shopify describes how it versions APIs and manages breaking changes for an ecosystem of tens of thousands of partners. It releases date-based API versions every three months, with changes introduced in an unstable version and represented in the monolith through frozen records and a change file. Teams classify changes by whether third-party developers must migrate, use impact analysis and forward-compatibility testing, and protect uncertain changes with beta flags. For necessary breaking changes, tooling marks breaking or possibly breaking request paths, sends metadata through Monorail to a data warehouse, and produces reports on affected calls. Runtime conditionals then help teams manage old and new behavior, measure adoption, identify remaining risk, and communicate with developers.
How We Implemented Domain-Driven Development in Golang
Building GrabPlatform's partner integration self-service portal initially resulted in an unstructured codebase where individual files exceeded 500 lines and lacked proper segregation. Modifying existing functions carried high risks of breaking functionality across imported source collections. To resolve this, the team restructured the Go application using Domain-Driven Design principles in coordination with product domain experts. They mapped business rules into bounded contexts, identified entities and aggregate roots, introduced repository interfaces, and utilized domain events for cross-context communication. The refactoring distributed core functionality evenly, simplified onboarding, and aligned technical terminology with business concepts.
Successfully Merging the Work of 1000+ Developers
Shopify’s single monolith supports changes 40 times a day and roughly 400 commits to master daily, making safe trunk-based development difficult at the scale of 1,000+ developers. Its first Merge Queue limited deployment drift and batch size, but queued pull requests did not receive CI, soft conflicts could break master, and a browser extension enabled accidental direct merges. Merge Queue v2 replaces that workflow with GitHub comments and webhooks, programmatic branch protection, an emergency /shipit --emergency path, and a predictive branch where queued changes receive CI before merging. A reconciliation algorithm synchronizes that branch through GitHub’s GraphQL API, while failure-tolerance thresholds handle flaky tests and remove repeatedly failing pull requests. CI starts when requests enter the queue; batches of eight and three concurrently tested batches balance throughput, risk, and resource use, while the changes improved user experience, deployment safety, and deploy throughput.
Four Steps to Creating Effective Game Day Tests
Game Day tests deliberately trigger failure modes in production systems and examine whether automated behavior and human response match expectations. Shopify frames the practice as preparation for outages that can arise despite code checks, reviews, and other safeguards. The method starts by inventorying possible failures, then selecting scenarios by impact and simulation difficulty rather than estimated likelihood. Each scenario becomes a controlled experiment with prerequisites, testable hypotheses, expected alerts and downstream behavior, plus recovery steps recorded in a spreadsheet. During execution, teams track results, communication, and work distribution; afterward they fix gaps, review coordination, update response guidelines, and rerun tests to verify changes, with periodic testing keeping systems and people prepared.
Driving Southeast Asia Forward Through People-Focused Design
Designing digital products for Southeast Asia requires tailoring user experiences to unique regional constraints and consumer behaviors across diverse populations. Users in the region often operate low-end mobile hardware on congested networks while carefully rationing prepaid mobile data. Grab addresses these challenges by designing comprehensively for non-ideal UI stacks, implementing loading skeletons, and replacing heavy video tutorials with lightweight SVG animations to minimize bandwidth consumption. Furthermore, product teams adapt to mobile-only environments by prioritizing phone number and one-time-password registrations while avoiding legacy desktop-era iconography. Visual accessibility is validated by testing UI readability on dimmed, low-resolution screens under bright ambient sunlight.
Sam Saffron AMA: Performance and Monitoring with Ruby
Sam Saffron discusses Discourse’s approach to Ruby performance and monitoring, arguing that production memory constraints matter more than isolated microbenchmarks for many deployments. He favors clear code and selective optimization, sometimes bypassing ActiveRecord with MiniSql on performance-sensitive paths rather than broadly sacrificing Ruby’s readability. Discourse keeps performance under continuous observation, and Saffron describes budgets for dependencies, boot time, and high-profile pages, including alerts for query-count regressions. He identifies memory leaks as especially difficult to diagnose, describes bisecting an application to isolate a V8-Ruby interop leak, and says MRI remains the only feasible runtime for Discourse while memory profiling and analysis tooling remain substantially behind Java and .NET.
Static Typing in Ruby with a Side of Sorbet
An experiment introduced Sorbet into a greenfield Ruby codebase with four developers to test whether static typing could catch bugs earlier, ease refactoring, and improve code design. Sorbet combines static linting with runtime checks that verify method parameters and return values against signatures, while RBI files describe methods from gems and Rails models. Setup required sorbet init, running sorbet tc, and adding that command to CI; however, Rails metaprogramming required model signatures, and sorbet-rails could miss or mistype fields. Sorbet immediately exposed issues including repeated method calls, unintended nullable database columns, and unnecessary explicit returns or guard clauses. The team reports a very positive short-term experience, while noting that the project’s young age caused most of the bugs encountered and that long-term value remains uncertain.
Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily
Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds.
Automated Continuous Deployment at Heroku
Heroku’s Runtime team replaced a manual, coordination-heavy deployment process after team growth and Common Runtime sharding expanded work from one infrequently changed application in two environments to multiple services across 12 environments. Existing Heroku Pipelines deploy merged, CI-passing changes to staging, while the Go-based cedar-service-deployer reconciles differences between pipeline stages and promotes them when a series of safety checks pass. Those checks include a 30-minute age gate, deployment and incident controls, master-branch verification, Librato alerts, and HTTP 200 checks for configured URLs, including platform integration tests. The stateless system can handle concurrent rollouts, and the team added practices such as isolated early stages, feature flags, and separated database-migration changes. After about two years, it managed 23 services, allowing engineers to merge and rely on healthy systems for global release.
Make Great Decisions Quickly with TOMASP
Technical leaders and managers face high-impact decisions about hiring, technology, architecture, and priorities, but cognitive biases, short-term emotion, and overconfidence can distort judgment. TOMASP is a framework for making decisions timely, considering alternatives, defining decision criteria, examining uncertainty, and preparing to be wrong. It recommends timeboxing the decision, generating at least three options, making a meta-decision about optimization criteria, analyzing options, stepping back from emotion, and preparing to be wrong through prototypes and tripwires. In the Flutter rewrite example, Michelle compares alternatives against cross-platform consistency, performance, and development speed, then sets a two-week prototype and blind test. The framework treats decision quality as a function of how a choice is made and implemented, not simply its eventual outcomes.
Using Grab’s Trust Counter Service to Detect Fraud Successfully
Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs.
Five Common Data Stores and When to Use Them
The article compares five common data stores—relational databases, NoSQL databases, key-value stores, full-text search engines, and message queues—and connects each to appropriate design needs. Relational databases use schema-bound tables, primary and foreign keys, SQL, and ACID transactions for durable business-critical data, while NoSQL databases trade strict schemas for flexibility with document and wide-column models. In-memory stores such as Redis and Memcached provide fast access and configurable eviction for caching; Redis additionally supports richer data structures and disk persistence. Full-text search engines target word and substring searches, whereas Kafka uses a distributed append-only log and producer-consumer model to move data reliably between services. The article concludes that the right choice depends on whether data must be permanent, flexible, quickly accessed, searched, or transferred.
How to Write Fast Code in Ruby on Rails
Shopify’s guide presents performance advice for Ruby on Rails across Active Record, Rails, and Ruby, while treating speed as a feature rather than the first optimization priority. It recommends understanding Active Record’s lazy query execution, selecting fewer columns, avoiding unindexed queries, using safe indexing approaches for large tables, and treating query cache as short-lived rather than dependable. For Rails applications, it covers caching, throttling expensive or abusive operations, moving long-running work into Active Job-backed queues, and reducing dependency growth to limit boot time and memory use. Ruby-specific guidance includes limiting metaprogramming and indirection, choosing O(1) hash lookups over O(n) array searches when appropriate, and reducing allocations while avoiding harmful global mutation; benchmark figures show method-definition and invocation costs can differ.
How Shopify Manages Petabyte Scale MySQL Backup and Restore
Shopify manages a petabyte-scale fleet of MySQL servers across replica-sets, or shards, in three Google Cloud Platform regions, where file-based Xtrabackup archives made shard restores take more than six hours. To reduce that Recovery Time Objective, the team built Kubernetes CronJobs around the Compute Engine Persistent Disk snapshot API, scheduling snapshots every 15 minutes while accounting for regions, zones, instance roles, and MySQL consistency variables. Persistent Disk snapshots take about 20 minutes initially and typically under 10 minutes incrementally; restoring the latest snapshot to a new volume and starting MySQL, including InnoDB instance recovery and replication-lag recovery, typically brings RTO below 30 minutes. Retention tooling keeps the latest two snapshots plus policy-defined dailies and weeklies, while verification checks startup, GTID auto-positioning, and InnoDB page corruption; offsite tooling compresses, encrypts, and transfers exported data to another provider.
How Shopify Scales Up Its Development Teams
The article argues that scaling development is primarily about creating more effective teams, not simply increasing headcount. It first recommends checking whether an existing team prioritizes impactful work, minimizes supporting activity, and can ship code quickly while finding bugs; otherwise hiring may be premature. Shopify organizes product work into problem spaces and teams of five to nine, uses a multi-step process combining technical exercises with cultural assessment, and aims for at least as many onboarded members as onboarding members. New teams receive a baseline for work practices and technology, while transparent technical design, shared tooling with dashboards and embedded quality and testing standards, optional pairing, and frequent retrospectives support consistency and learning. The conclusion is that these practices, rather than hiring alone, enable development teams to scale more effectively.
Being a Principal Engineer at Grab
Grab's rapid growth resulted in roughly 350 microservices powering its superapp, creating the need for defined individual contributor career milestones. At Grab, a principal engineer oversees the architecture of an entire Tech Family, a sub-organisation containing over 50 engineers and 20 or more microservices. Responsibilities include translating broad, ambiguous problems into concrete projects, managing technical debt, and aligning multiple engineering teams across global R&D centres. The role demands continuous technical leadership through RFC design reviews, cross-functional communication, mentorship, and self-directed prioritization alongside engineering leadership. Ultimately, principal engineers amplify engineering quality and operational stability without directly managing people.
Want to Improve UI Performance? Start by Understanding Your User
Shopify’s team examined UI performance in the Marketing section of the Shopify admin, aiming to improve load time, perceived load time, and interactions after navigation. They tested the experience on a low-end Moto G3, profiled browser activity with Chrome Developer Tools and React DevTools, and used merchant workflows to reorder priorities. The work reduced unnecessary initial work by memoizing repeated date-formatting tasks, deferring the nonessential ImagePicker bundle, and prefetching that component and its data when a merchant hovered over its activator; memoization cut one component’s render time from about 64.7 ms to 0.5 ms. For loading states, the team recommends stable skeletons with static headings and immediately useful controls, such as the Create campaign button, so merchants can act before unrelated data arrives.
Up to 75% Faster Maintenances with Heroku Postgres and Redis Premium Plans
Heroku describes how it minimizes downtime during routine security and patching maintenances for high-availability Heroku Postgres and Heroku Redis plans. These plans run a writable primary alongside a hidden read-only standby, and maintenance requires moving application writes safely to the standby before promoting it. Previously, Heroku changed DATABASE_URL and restarted app dynos, then waited for the old primary to shut down and replicate remaining writes, which could extend downtime during long-running transactions. The new method redirects new connections to the new primary, kills existing connections to the old primary, and relies on reconnections plus a rolling restart to prevent further writes there. Heroku reports HA maintenances now complete in 15–40 seconds instead of 60 seconds or more, and says the behavior is default for Premium, Private, and Shield plans.
Building Resilient GraphQL APIs Using Idempotency
Shopify’s Payment Service uses API-level idempotency to make GraphQL payment mutations resilient to timeouts, connection failures, and ambiguous outcomes that could otherwise leave merchants unpaid or double-charge buyers. The design places a UUID-based idempotency key directly in each mutation input, identifies requests by client plus key, and locks concurrent duplicates so retries receive a 409 response while the original call is processing. An IncomingRequest record stores request state and completed recovery points, allowing completed responses to be replayed and incomplete attempts to recover before continuing. Handlers divide work into no-side-effect, transactional local-side-effect, and remote-side-effect steps, each optionally providing run and recover functions. The approach adds database-write overhead and requires careful compatibility testing when recovery points change, but the team reports that its structured handlers are clear and maintainable.
Designing for Accessibility: Contrast Ratio
Heroku’s accessibility initiative targeted WCAG 2.0 Level AA luminance contrast for its user interfaces, addressing reduced contrast sensitivity and color-vision deficiencies. The team audited its palette with Contrast Grid, proposed darker and revised colors, then chose a narrower proposal that preserved existing names and colors while meeting accessibility concerns within an affordable implementation effort. Because the palette affected GUIs built with Purple 3 and Malibu, pull requests generated Review Apps so teams could audit production-ready CSS and SVG assets across Dashboard components before release. Audits using axe-core 3.2.2 covered common and unusual elements, reducing color-contrast violations by an order of magnitude before rollout. The updated palette now spans the Heroku Dashboard, with planned accessibility testing through Ember tooling and further audits across other interfaces.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.