Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,538 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Building a Service-oriented Architecture with Rails and Kafka
Service-oriented Rails applications can use Kafka to move high-volume events between producers and consumers without coupling downstream availability to an upstream request. Kafka persists events for a configured period, lets consumers manage offsets, and distributes replicated data across broker partitions, while ordering is guaranteed within a partition rather than across a topic. The post shows a Rails setup using the ruby-kafka gem, an asynchronous producer, JSON payloads, topics, and partition keys, then applies the event stream to order creation, email, and fulfillment workflows. It also addresses operational concerns including consumer scaling, lag metrics, duplicate delivery, idempotent processing with UPSERTs or event identifiers, and schema evolution, contrasting JSON’s accessibility with Avro’s stronger schema support.
Supporting Open Source: Our Donations for 2018
Kickstarter describes its 2018 effort to support the open-source ecosystem through financial contributions aligned with its PBC charter and stated values. After surveying engineers for nominations, the company split its yearly donation budget among two open-source projects, Babel and Webpack, and four organizations supporting learning or coding skills: Advent of Code, Rails Girls Summer of Code, Code Nation, and Code Cooperative. Babel converts ECMAScript 2015+ into backwards-compatible JavaScript, while Webpack bundles JavaScript used at Kickstarter. The other recipients support collaborative problem-solving, participation by women and non-binary individuals, technology pathways for students in under-resourced high schools, and technology education for formerly incarcerated people. The post also notes an engineer’s weekly Code Nation mentoring and invites engineers to join the team and help choose future recipients.
Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)
Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.
The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study
An Android monorepo expansion and larger team increased the number of tests run on master, exposing how flaky tests and infrastructure failures could erode CI confidence. Under an independence assumption, tests passing 99.95% each produced a theoretical full-suite pass rate of only 35% across 20 pipelines, while retrying each failed test once raised it to 99.95%. Shopify therefore added retries for setup failures, distinguishing failures retriable within a job from fatal failures requiring the test environment to reload, and retried failed tests up to three times. The system also flagged tests that passed after retry as warnings, since retries can hide reliability problems and slow builds. In practice, Android app pipeline pass rate rose from 31% to almost 90% after retry mechanisms and Gradle caching, while iOS rose from 67% to 97%.
Querying Big Data in Real-time with Presto & Grab's TalariaDB
Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.
Ruby 2.6 Released: Just-In-Time Compilation Is Here
Ruby 2.6 introduces an initial Just-In-Time (JIT) compiler, making the feature available through the --jit option and on Heroku. The implementation follows Takashi Kokubun’s YARV-MJIT approach, retaining Ruby’s existing YARV instructions instead of replacing them with the proposed RTL intermediate representation; it starts a compiler process at startup and supports GCC, Clang, and Microsoft Visual C++. On Heroku, applications can select Ruby 2.6 in the Gemfile and enable JIT through a Procfile flag or RUBYOPT, although setting RUBYOPT with heroku config is not yet supported. The optcarrot benchmark showed a reported 1.8x speed increase over Ruby 2.5, while Rails and Sidekiq were slower because compilation costs and roughly 2MB per compiled method create overhead. JIT compaction reduces memory use near the 1,000-method cache limit, but the source says further work is needed before JIT becomes the default.
Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)
Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic.
Preparing Shopify for Black Friday and Cyber Monday
Shopify’s Production Engineering team prepares its platform and 600,000+ merchants for Black Friday and Cyber Monday traffic without interrupting service. Preparation combines controlled fault injection “game days,” synthetic load tests, and staged freezes: a feature freeze begins several weeks ahead, followed by a code freeze allowing only critical fixes during the event. The tests probe dependencies such as shipping providers and payment gateways, including latency, fallback behavior, and responses to datastore failures. After the 2018 weekend, Shopify reported nearly 11,000 orders per minute, around 100,000 requests per second for extended periods, and overall metrics about 1.8 times 2017 levels. The account concludes that fault injection, load testing, and disaster role-play improved readiness, while third-party downtime remains Shopify’s responsibility to merchants and buyers.
Bug Bounty Year in Review 2018
Shopify’s 2018 bug bounty review reports its fifth year and most successful program to date, with faster handling and higher payouts. The security team cut average triage time from four days in 2017 to 10 hours in 2018 by assigning one of its 14 Application Security members to HackerOne triage each week. At Montreal’s H1-514 event, over 40 hackers produced more than 50 valid reports, supported by early submissions, disclosed resolved reports, incentive bonuses, expanded scope, and infrastructure shell access. Excluding event statistics, average bounty rose to $1,790, total payouts reached $155,750, 60% of resolved reports received a bounty, and triage, bounty, and resolution times improved, although resolved reports and participating hackers declined and some low-severity reports exceeded the one-month target.
How an Intern Released 3 Terabytes Worth of Storage Before BFCM
While preparing for Black Friday Cyber Monday (BFCM), Shopify’s Products team found that historical reporting records were causing MySQL to scan many InnoDB pages to return a single record. An intern built a Rails background job to delete the records safely, using Shopify’s job-iteration library to process small batches and an internal throttling enumerator to pause when database health or replication lag became a concern. Indexed fetch queries and cursor-based batching avoided repeated scans and timeouts, while the iterator deleted each batch. Run about two weeks before BFCM, the job deleted more than 5 billion records, peaked at approximately six million records per minute, and released three terabytes of storage. Afterward, the database scanned around three times fewer pages per returned record, improving resilience to flash-sale traffic and producing large cost savings.
Manage Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku
The post presents a Heroku-based system for handling high-volume product-usage events, combining near-real-time visualization with deeper SQL analytics. A Node.js producer sends simulated e-commerce events to an Apache Kafka topic, while a visualization consumer forwards selected data over WebSockets to a D3 stacked stream chart. A second consumer uses a Kafka consumer group, batches events, writes them to Amazon Redshift, and commits offsets after successful processing; Metabase then provides dashboards and ad-hoc SQL queries. The example uses Kafka, Redshift, and Metabase on Heroku, with Terraform described for provisioning Redshift and private connectivity, and notes that the setup incurs AWS and Heroku costs and requires Heroku Enterprise for that configuration. The source says the system can be provisioned on Heroku in 15 minutes.
Director of Engineering, Lawrence Mandel Talks Road to Leadership, Growth, and Finding Balance.
Lawrence Mandel, a Director of Production Engineering at Shopify, describes a career that moved from software development into technical leadership and the lessons behind that transition. He leads Developer Acceleration, a team that builds tools, conventions, and documentation so Shopify’s development organization can scale effectively. His path included web services, XML, open source, release engineering, project and release management, and self-taught Perl and CGI programming during a work term at IBM. Mandel says leadership required accepting that he could not know every technology or always be the most knowledgeable person in the room, while trusting others’ expertise. He also says prioritizing family and health before work improved his relationships and made his work more deliberate and efficient.
Orchestrating Chaos Using Grab's Experimentation Platform
Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts.
Developer Talks: How the Command Line Can Empower You (Webinar)
On 27 November 2018, Eric Fung, a Senior Data Scientist, presented “How the Command Line Can Empower You,” a 45-minute talk followed by 15 minutes of questions and answers. The presentation addresses work that often happens outside an IDE, including interacting with APIs, creating screenshots, and massaging data, and shows how command-line tools can simplify, improve, or automate those tasks. Using real-world examples, Fung covers typing less in the terminal, searching files, manipulating images and JSON files, and generating code automatically without point-and-click interaction. The presentation focuses on software for macOS, while noting that Linux and Windows users can benefit because many mentioned tools are cross-platform; the presentation can be watched online and speaker notes can be downloaded.
Handling Addresses from All Around the World
International address handling is difficult because countries and locales differ in field order, required fields, terminology, translations, and display conventions. Shopify models each address with fields such as names, address lines, zones, postal code, city, country code, and phone, then stores country-specific metadata in YAML-backed country data. A reusable format describes form and display order, allowing clients to reorder fields, prepopulate country and zone values, and add locale-specific characters such as Japan’s 〒 and 様. Label keys and translations handle terms such as province, prefecture, postcode, and pincode, while an API endpoint keeps applications synchronized and accepts locale information. The team also provides @shopify/address and higher-level components, while identifying caching as a remaining issue to reduce repeated network calls.
Running Apache Kafka on Kubernetes at Shopify
Shopify adopted Apache Kafka in 2014 as a data bus for reliable messaging, initially collecting events and aggregating logs, then expanded to multi-tenant regional and aggregate clusters delivering billions of messages daily. In 2017, it migrated Kafka infrastructure from data centers to the cloud with zero downtime by mirroring regional data to both aggregate clusters before moving producers and consumers to cloud clusters. Shopify chose Kubernetes over GCP virtual machines because StatefulSets provide ordered, graceful deployment and scaling, while Custom Resources and Controllers support cluster-specific automation. The implementation uses self-built Kafka images in an own registry, persistent volumes, node and inter-pod affinity controls, taints, rack-awareness across zones, readiness probes, and resource-monitoring sidecars to limit restart and availability risks. The migration completed successfully, and Shopify now runs multiple regional clusters plus an aggregate cluster that mirrors data for the data warehouse.
Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab
Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.
Building Shopify POS for Android Using MVVM
Shopify’s Android Point of Sale app adopted Model-View-ViewModel (MVVM) to replace an inconsistent codebase containing both MVC and MVP. Because the roughly three-and-a-half-year-old app could not be migrated at once, the team used MVVM for all new screens and converted legacy screens during significant changes. In the architecture, Fragments and sub-views report UI actions through contracts, ViewModels fetch and mutate data, and LiveData delivers updates to active observers. ViewModelProvider retains ViewModel instances across Activity or Fragment configuration changes, while Models handle databases, networks, and caches. Shopify reports more consistent development, easier onboarding and testing, lifecycle-safe UI updates, and a codebase that is easier to modify, although some MVC and MVP screens remain.
Creating Locale
Shopify’s move to support six additional languages exposed a challenge in its admin graphs: condensing numbers without distorting locale-specific conventions. The team needed locale-aware abbreviations for sales, visits, and customer data, because unshortened values could consume axis space. Existing JavaScript libraries either exceeded the requirement or conflicted with Shopify’s stack, while the Intl API lacked implemented number-condensing support. During Hack Days, the developers used abbreviation data from Unicode’s Common Locale Data Repository (CLDR) and currency information from Intl.js to build functions for locale-specific numbers and currencies. Their package handles examples such as Portuguese “3 mil” and non-Western systems including India’s lakhs and crores, and was released as condense-number on npm in July 2018.
Building a Data Table Component in React
Building a responsive data table in React requires preserving a rigid grid while accommodating narrow screens, long words, readable alignment, contextual headings, and accessible markup. The Polaris example starts with headings and rows passed as props, then maps them into a stateful table component and a stateless Cell subcomponent. To contain overflow, the table sits in a max-width: 100vw wrapper with horizontal scrolling, while an absolutely positioned, explicitly sized first column remains fixed and later cells receive matching left spacing. Because absolute positioning breaks natural row-height behavior, refs, clientHeight measurements, Math.max, componentDidMount(), and resize handling recalculate and apply the tallest height for each row. The finished component adds a caption and scope attributes for screen readers, and the described result is a responsive, accessible table for comparing and analyzing data, with the full Polaris version adding truncation, totals, sorting, and navigation.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI455
- Developer Experience446
Weekly digest
The best engineering reads, once a week.