Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,538 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Meet Shopify Developer Kirsten Westeinde - Shopify
Kirsten Westeinde’s just-over-five-year Shopify career spans internal tools, partner-facing applications, and backend development on Kit as the company scaled. She began with Ruby on Rails, later helped extract partner software from Shopify Core using Elasticsearch, Kafka, Ruby on Rails, and the Shopify API, and now works with Data Engineering on real-time recommendations. The feature combines Go and Ruby and is designed to handle the current load while scaling to future demand. Her problem-solving practices include whiteboarding system interactions, taking mental breaks, and writing or explaining ideas to align collaborators. She also describes mentorship, pair programming, a women’s developer meetup, and f(empower) as ways to support growth and address exclusion, bias, and microaggressions in technology.
Seven Ways to Fortify Your Application
Applications can fail when a single upstream service goes down, prompting tools that shed load, fail gracefully, and protect struggling dependencies. The discussion frames this as application resilience for everyday failures, distinct from request-level retries, rollback strategies, and catastrophic disaster recovery. It presents maintenance mode as an easily toggled hard stop with a clear message and status-page link, read-only mode as a way to preserve access when modifications fail, and feature flags as controls for global or meaningful groups. Implementation examples include environment variables, relational databases, Redis, in-memory caches, and an S3-based check, with state stored primarily in Postgres and Redis at Heroku. The closing guidance says maintenance mode should be easy to toggle, while read-only mode can preserve user confidence and access to information.
Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting
Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.
Context Deadlines and How to Set Them
Microservice architectures operating under heavy network traffic require robust timeout handling to prevent slow or failing dependencies from causing cascading failures across services. Naive, static timeout configurations across a call chain often cause downstream components to waste compute effort on requests that upstream callers have already abandoned. To establish predictable timeout thresholds, engineers can align limits with service-level latency percentiles, pairing P99 limits with median latency retry allowances. Go's context package improves upon static network timeouts by propagating request-scoped deadlines and cancellation signals across service boundaries. This distributed context ensures downstream servers recognize remaining time budgets and terminate unneeded processing immediately when parent deadlines expire or callers manually cancel requests.
Shopify Devs Share Lessons on Self-Advocacy and Dealing with Adversity - Shopify
Shopify’s Behind The Code series profiles Stella Lee, Helen Lin, and Cathryn, three developers discussing self-advocacy, adversity, management, inclusion, and career changes in technology. Lee describes building Shopify Checkout Experience features in TypeScript and Ruby to reduce manual checkout fields, while leading the f(empower) women’s employee resource group; she also discusses imposter syndrome, mentorship, and turning initial “stupid” questions into better ones through timeboxing and research. Lin, a remote Themes team lead, emphasizes aligning projects with long-term company vision, managing stakeholders, staying connected, and directly requesting career opportunities while sharing accomplishments. Cathryn recounts moving from health sciences into programming and stresses evaluating personal goals, hiring a more diverse workforce, and using meeting-room paper pyramids to support inclusive discussion. The article closes with advice to embrace feedback and own both accomplishments and failures.
Recipe for Building a Widget: How We Helped to “Peak-Shift” Demand by Helping Passengers Understand Travel Trends
Transport demand spikes during regular commuting hours often outpace driver availability, resulting in passenger wait times and fare surges. To mitigate these imbalances, Grab created the Travel Trends Widget for its mobile feed to redistribute ride requests toward off-peak windows. The widget uses machine learning forecasting to present historical supply-demand patterns alongside pricing trends for the upcoming two hours. To handle anticipated high query rates across millions of database entries, engineers periodically load precomputed trend data into an in-memory data structure rather than querying the database per request. The feature rolled out to feeds in Singapore and Jakarta within four weeks of initial development.
PWA Tutorial: Turn Your Website Into a Progressive Web App
Progressive web apps let websites provide app-like experiences, including home-screen installation, offline access, push notifications, and some device hardware APIs, without separate platform-specific code or app-store approval. This tutorial converts an existing site by adding a JSON manifest, linking it from the base HTML, creating an offline page, implementing a service worker, registering that worker, and deploying over HTTPS. The service worker precaches important routes during installation and handles fetches by serving network responses, caching them, or falling back to cached content and offline.html; the example uses the "offline" cache and includes routes such as /blog/, /contact, and /resume. It also covers installation through Safari and Chrome and notes that iOS does not support background sync.
Structured Logging: The Best Friend You’ll Want When Things Go Wrong
Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.
How We Simplified Our Data Ingestion & Transformation Process
Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.
Reactive Programming with Salesforce Data
Salesforce Platform Events and Change Data Capture enable custom applications to react to activity in Salesforce, motivating an event-sourced design for responsive, durable integrations. The article demonstrates a Node.js app on Heroku that consumes the Salesforce Streaming API through jsforce 1.9.0 or later, whose Faye-based streaming client handles authentication, network connectivity, and message callbacks. It notes that a basic subscription loses its position on restart and cannot coordinate parallel processing, then uses Redis to persist each topic’s Replay ID and distribute messages through pub/sub or reliable queues. Because the Salesforce Streaming API lacks consumer-group delivery, a singleton stream_consumer feeds Redis while scalable web or worker processes handle work. The design also recommends a separate non-admin integration user and monitoring connection status, active clients, last message time, and processing duration.
Debugging in Ruby—Busting a Year-old Bug in Sprockets
Sprockets, the Ruby asset packaging system used in Rails’ asset-processing pipeline, had a bug that made assets:precompile fail after a project directory was renamed between runs. The debugging walkthrough reproduces the failure with CodeTriage by clearing caches, compiling assets, moving the project, and compiling again, yielding an error for an outdated absolute path. It then uses bundle open, source inspection, recursive grep, and IRB to examine Sprockets’ dependency cache; the IRB cache lookup does not work because the cache key differs from the cache ID, but the on-disk record exposes an absolute asset path where a relative path is expected. The proposed fix updates loader.rb’s to_load and to_link handling with compress_from_root, and the pull request adds a regression test.
Ten Ways to Secure your Applications
The post, adapted from Joe Kutner’s Devoxx 2018 talk, presents ten ways to strengthen web application security, with examples focused on Java but applicable to modern languages. It recommends keeping dependencies current, validating explicit payloads with schemas, checking regular expressions for ReDoS, throttling abusive requests, and keeping credentials outside the codebase. It also advises secure-first coding, certificate validation for outgoing connections, logging security-relevant actions without recording secrets, and monitoring suspicious behavior with tools such as AppSensor. The examples include Maven Versions Plugin, Snyk, Bucket4j, saferegex, environment variables, and EnvKeyStore, with a token-bucket algorithm proposed for request throttling. The conclusion is that security requires anticipating unintended functionality across network, platform, and physical domains, while these practices provide a starting point.
Deconstructing the Monolith - Shopify
Shopify describes why its decade-old Ruby on Rails monolith became difficult to change as the codebase and development organization grew. High coupling and absent boundaries made innocuous changes trigger unrelated test failures, slowed CI, and forced developers to absorb excessive domain context, while microservices would add deployment, infrastructure, network, and coordination costs. The chosen path was a modular monolith: retain one codebase and deployment unit while separating domains through strictly enforced component boundaries. Shopify’s Componentization effort was informed by a developer survey and uses Wedge to score components and report violations, including cross-component associations and calls that bypass explicitly public interfaces. Boundary enforcement at runtime remains a future goal, but dependency isolation has already enabled replacing a legacy tax engine with a new tax calculation system.
Understanding Supply & Demand in Ride-hailing Through the Lens of Data
Grab measures ride-hailing supply and demand across space and time to resolve geo-temporal allocation mismatches between moving drivers and ride-seeking passengers. The analytics pipeline defines supply as idle online drivers and demand as passengers checking fares within brief time slots, aggregating locations into geohashes. Each driver is mapped across neighbouring demand units and inversely weighted by straight-line distance, which yields the effective supply, supply-demand ratio, and supply-demand difference for each geographic polygon. Grab uses these aggregated metrics to identify marketplace imbalances, deploying driver heatmaps to shift excess supply and passenger travel trend widgets to defer time-insensitive ride requests.
Unifying Our GraphQL Design Patterns and Best Practices with Tutorials
Shopify describes how its GraphQL work grew from an early experiment for mobile APIs into contributions from hundreds of developers across teams and offices, including the Admin and Storefront APIs. As adoption expanded, sparse Markdown guidance left context inconsistently shared, while design guidelines were mixed with code-reference documentation and became difficult to extend for complex cases. The API Patterns Team, created in May 2017, addressed this by focusing on horizontal concerns, identifying common patterns, and documenting solutions for developers working across API domains. This work produced a living, in-depth design tutorial that explains Shopify’s GraphQL design philosophy through a practical example; its central rule is to understand the business domain and design the API to reflect it rather than the application’s implementation.
Engineering a Historic Moment: Shopify Gets Ready for Cannabis in Canada
Canada’s 2018 cannabis legalization required Shopify to support a new, tightly regulated retail industry while keeping personal information in Canada. Using Google Cloud Platform’s Montreal region, the team created a Canadian Shopify instance with Terraform-managed projects, regional clusters, segregated workloads, PCI-compliant payment infrastructure, and a regional data warehouse. Because consumer behavior and peak demand were uncertain, engineers modeled traffic scenarios, ran repeated load tests across storefront actions, tested bot attacks, held pre-mortems, and provisioned five times projected launch traffic and sales. Multiple safeguards—including application-level endpoint checks, firewalls, limited secrets, unique SSL certificates, monitoring, and alerting—were layered to prevent data from crossing jurisdictions. Merchants opened across time zones, and the first legal sale occurred on Shopify’s point of sale in Newfoundland at 12:01 NST, followed by launches with zero downtime, excellent performance, and secure Canadian data storage and transmission.
Attracting Local Talent And Building Mobile Apps: A Developer Hiring Initiative
Shopify’s “Build Things, Show Shopify” initiative used app development to reach local mobile developers beyond traditional recruitment channels and strengthen its talent pipeline. From May to August, workshops in Ottawa supported 75+ participants with help from more than 20 Shopify product managers, developers, designers, and computing educators, covering workflows, coding, code reviews, design, and presentation skills. At the September showcase, 13 developers presented 12 apps through rapid three-minute rotations, after which six promising projects advanced to a grand panel of Shopify leaders who provided technical questions and feedback. Participants left with fully developed apps, while a later Ottawa showcase connected them with angel investors and technical recruiters. The initiative also exposed a retention and inclusion problem: participation fell from 24 women among 77 at launch to 16 at the last event, with only two reaching the showcase, prompting consideration of daytime workshops.
How Shopify Uses Recommender Systems to Empower Entrepreneurs
Shopify uses recommender systems to help merchants find relevant insights, actions, apps, themes, services, and other resources across its commerce platform. Its approach represents implicit signals such as purchases, installations, clicks, and views as user-item interactions, then applies binary classification and linear methods including SLIM and LRec to learn an item-item similarity matrix and rank unseen items. The framework supports task-parallel training, interpretable coefficients, user-feature enrichment, and cold-start handling for new users or items, while evaluation considers precision alongside diversity, novelty, serendipity, and business-specific goals. In initial A/B tests, personalized app recommendations produced a 50% higher install rate, home feeds were up to 12% more likely to be rated useful, expert matching increased response and collaboration, and theme recommendations made merchants over 10% more likely to launch stores.
A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform
Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.
iOS Application Testing Strategies at Shopify
Shopify describes an iOS testing strategy for a monorepo containing multiple app projects and evolving through hundreds of weekly commits. The approach covers Models, Views, ViewModels, View Controllers, and user workflows with tests matched to each component’s role: regular unit tests for business rules, snapshot tests for visual behavior, and XCUITest UI automation for user-facing flows. UI automation selects elements through accessibility labels or strings, which can also improve accessibility for visually impaired users, while manual testing remains necessary for some animation and rendering issues. The article recommends a test pyramid because slower tests require more human effort and maintenance, and says suites must evolve as business requirements change.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI455
- Developer Experience446
Weekly digest
The best engineering reads, once a week.