Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,738 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Rails Asset Pipeline Directory Traversal Vulnerability (CVE-2018-3760)
All previously released versions of Sprockets, which powers the Rails asset pipeline, contained a directory traversal vulnerability, CVE-2018-3760. Rails applications were vulnerable when production runtime asset compilation was enabled with config.assets.compile = true; the default production setting was false, so apps using the default were not vulnerable. Sprockets handled requests by searching for rendered assets and compiling matches dynamically, and an attacker could exploit that process with a crafted URL based on a known absolute source-asset path to make it appear inside a permitted path. Patches were prepared for Sprockets versions 2, 3, and 4, with remediation via disabling runtime compilation or upgrading to versions 2.12.5, 3.7.2, or 4.0.0.beta8. Heroku also updated its Ruby Buildpack to fail affected builds, notified customers, and warned that compromised applications may require credential rotation.
Heroku CLI: Completing Autocomplete
Heroku CLI Autocomplete had to support a dynamic, plugin-based command set while completing values that often reside behind the Heroku API. Instead of static shell case statements, its cache builder iterates through installed commands and generates setters that supply command, flag, and value completions when the user presses Tab. A shim sourced from the user’s .bashrc or .zshrc limits profile changes, fails silently when unavailable, and permits updates without repeated edits. API-backed values are cached on disk, while oclif custom hooks invalidate or rebuild relevant caches after commands change data. The completion function also parses the partially entered command line to provide contextual results, such as add-on aliases for a specified app; the work is being incorporated into an oclif plugin.
Introducing Grab-Kit: Distributed Service Design at Grab
As Grab migrated from a monolith to microservices, maintaining consistency, coordination, and code quality across rapidly expanding teams became a major engineering challenge. To address this, the Developer Experience team built Grab-Kit, a Go framework that automates service scaffolding, code generation, and distributed system design patterns. The framework uses Protocol Buffer definition files as a single source of truth to generate data transfer objects, communication bindings, and standardized middleware for logging and profiling. Grab-Kit also features declarative metrics definitions that synchronize with the DataDog API to build and update service dashboards automatically. Adopting the framework reduced development time for creating new services by up to 70% in teams such as GrabFood while improving overall system stability.
Introducing the Merge Queue
Shopify describes a merge queue added to Shipit to reduce failures in its large monolithic Rails application, where growing pull request volume made manually checking whether master was safe unsustainable. The queue adds pull requests from GitHub, merges them when master is healthy, waits for required CI statuses, and can reject stale branches using configurable divergence thresholds. It is encouraged rather than enforced, while changing the browser extension’s visual emphasis helped make queued merging the default workflow; over 90% of core application pull requests use it. Shipit also handles unsafe deployment commits by allowing explicit marking and automatically detecting reverts of undeployed commits, marking the original and intermediate commits unsafe. These changes let deployment continue safely around reverted work without requiring developers to manually unlock the pipeline.
Securing Dependencies for Rails 5.2 Active Storage
Supporting Rails 5.2 Active Storage on Heroku required handling media files for PDF and video previews while addressing FFmpeg’s security and packaging constraints. Ubuntu Trusty’s lack of FFmpeg support, Xenial’s community-maintained package and missing vulnerability backports, and concerns about Libav led the team to build its own binary. Using a fork of sffmpeg and modified CMake configuration, the team created a static FFmpeg build that removed non-free software and unused components such as RTMP and xvid, reducing dependencies and attack surface. Docker-based CircleCI automation compiles the binary, packages it as a Debian archive, uploads it to AWS S3, and lets the buildpack install it without root access. The resulting package is about 30MB instead of roughly 200MB and can be updated in around 20 minutes after upstream security releases.
Modern Web Stack: Rails 5 API, ActiveAdmin, and Create React App on Heroku
Modern Rails applications can combine a Rails 5 API-only backend, ActiveAdmin for CMS capabilities, and a Create React App frontend in one Heroku-hosted project. The tutorial addresses the tension between Rails API mode, which omits views and middleware, and ActiveAdmin’s need to serve HTML and use cookies, flash, sessions, and assets. It restores those pieces by switching ApplicationController to ActionController::Base, adding forgery protection and middleware, reintroducing sprockets, and defining a separate ApiController that retains ActionController::API. The frontend uses Create React App, API proxying during development, React Router for client-side routes, and a build process that serves the compiled app from Rails on Heroku, including a React 404 for unmatched paths. A smoothie-recipes example demonstrates the resulting setup.
Rails 5.2 Active Storage: Previews, Poppler, and Solving Licensing Pitfalls
Rails 5.2’s Active Storage adds file uploads and attachments for Active Record models, with backing services such as AWS S3 and previews for PDFs and videos, but its beta PDF support depended on MuPDF. Security and legal review found MuPDF’s AGPL licensing could require commercial licensing, so Terence Lee updated Active Storage to support Poppler, an open-source PDF backend merged before release. Deployment guidance uses S3 through aws-sdk-s3, configures :amazon rather than the non-scalable local disk, and installs a custom buildpack containing FFmpeg and Poppler. The example Rails app attaches files to a Bulletin model, accepts uploads, and chooses PDF or video previews, image variants, or downloads according to available binaries and MiniMagick.
Shopify Interns Share Their Tips for Success
Shopify interns and a former intern describe how they approached interviews, daily work, and growth across backend, development, solutions engineering, iOS, and application security roles. Their routines include stand-ups, pull-request review, coding, meetings, pair programming, production monitoring, and regular breaks, with some teams organizing work around three-week release cycles. Interview advice centers on completing the application, explaining non-traditional backgrounds, demonstrating passion and adaptability, communicating technical solutions to different audiences, and being honest about knowledge gaps. Inside Shopify, contributors say success depends on asking questions early, seeking feedback and mentorship, taking initiative, talking beyond one’s team, and holding strong opinions weakly. The post also announces Fall 2018 intern hiring, with applications due May 11, 2018, at 9:00 AM EST.
Solving the N+1 Problem for GraphQL through Batching
GraphQL avoids REST’s over-fetching and under-fetching by letting clients request specific fields, but its field-specific resolvers can create the n+1 problem for nested data. With N authors, a server may make one datastore round trip for authors and N more for addresses, rather than loading all addresses in one additional trip. Shopify’s GraphQL Batch Ruby library addresses this through promises and batch loaders: resolvers request data without resolving immediately, similar loads are grouped, and the loaders fulfill them together. During execution, grouped promises are loaded and replaced with results, while nested fields can enter later batching rounds. The approach reduces datastore queries and computing power for the same requests, became general best practice at Shopify, and was released as open-source.
Event Sourcing made Simple
Event Sourcing stores application changes as immutable events, providing a history that can explain how current data came to be. The post introduces four components: Events record changes, Calculators update Aggregates from those events, and Reactors trigger side effects or create further events. Kickstarter implemented a minimal framework for d.rip using regular Active Record aggregates, a regular SQL database, synchronous atomic updates, and roughly 200 lines of home-made code. The approach supports replaying history to reconstruct past state, repair calculator bugs, backfill new aggregate fields, optimize read models, and forward events to analytics platforms, while presenting naming and mutation-to-event decomposition as important challenges.
MJIT: A Method Based Just-in-time Compiler for Ruby
Ruby programs are compiled into YARV instructions and executed by a stack-oriented virtual machine, while register-oriented representations can expose more opportunities for compiler optimization. The post explains how YARV represents a simple expression, contrasts stack and register architectures, and introduces Vladimir Makarov’s Ruby RTL, an intermediate representation distinct from GCC’s RTL. MJIT uses RTL generated by MRI to make speculative, operand-specific optimizations, then compiles methods to machine code rather than relying only on interpretation. The interview discusses deoptimization: when assumptions change, MJIT generates a new machine-code version and can switch between JITed execution and RTL interpretation. The participants say this overhead should be tiny and should not degrade performance overall, while acknowledging edge cases where MJIT is not faster and that benchmarking JITs is difficult.
Ruby CVE-2017-17405: Vulnerability in Ruby’s FTP Implementation
CVE-2017-17405 affected Ruby’s net/ftp implementation, where a filename returned by an FTP server could reach Ruby’s open function and enable command execution. The issue was found while investigating CVE-2017-8817 in libcurl and checking whether Ruby used libcurl or had a separate FTP implementation with comparable wildcard behavior. Reviewing Ruby 2.4.2 source identified the 1,496-line ftp.rb file and the gettextfile method, which called open(localfile, "w") without validating the value. A proof of concept made an FTP client request a file list containing a pipe-prefixed payload; Ruby passed it to the shell, producing remote code execution and creating a file containing id output. Ruby replaced open with File.open, released the fix in 2.4.3, and assigned CVE-2017-17405; affected versions included Ruby 2.2.8 and earlier, 2.3.5 and earlier, 2.4.2 and earlier, and 2.5.0-preview1.
Shopify’s Infrastructure Collaboration with Google
Shopify announces that it is building its Cloud with Google after more than a decade of operating the foundation of its commerce cloud through its own data centers and incremental public-cloud adoption. Its infrastructure evolved from database and application sharding, performance testing, automated failovers, and Docker-based immutable infrastructure toward Kubernetes orchestration, after the company initially built minimal container-management features itself. During the migration, Shopify created Shop Mover, a selective database migration tool that rebalances shops between shards with an average of 2.5s downtime per shop, and moved more than 50% of data-center workloads to Google Cloud. It also grew to more than 400 production services, consolidated them on Kubernetes through a platform-as-a-service, and contributed to and used Grafeas for software supply-chain auditing and governance.
Deconstructing Monolithic Apps with SHIFT Commerce
SHIFT Commerce describes its move from a monolithic application toward microservices, while arguing that architectural change should address concrete problems rather than follow fashion. The company identifies communication overhead, governance and high-risk deployments, and language or infrastructure limitations as reasons to extract functionality, including its Order Management Suite, admin panel, and integration system. SHIFT uses managed Apache Kafka on Heroku as an asynchronous event stream, with chronologically ordered, durable messages organized into topics so services can maintain local datasets. Its eight-step migration progressively adds producers, builds a consumer-backed database, tests and replicates logic, enables feature-flagged traffic, exchanges events, and removes deprecated monolith code. SHIFT reports more regular releases, lower latency on I/O-restricted API calls, and prevention of user-facing outages, while saying the migration remains ongoing.
A House of Cards: An Exploration of Security When Building Docker Containers
The post examines how a Docker-based, multi-tenant build environment can expose untrusted code when layered isolation is misconfigured. It traces a setup using AWS CodeBuild, Docker-in-Docker, and per-step build containers, then tests directory traversal through configurable Dockerfile paths before focusing on daemon networking. The vulnerability comes from starting dockerd with tcp://0.0.0.0:2375 and placing containers on a shared default network; build code can reach the daemon, issue Docker commands, mount the host filesystem, and retrieve container-managed AWS credentials. The excerpt reports a reverse shell, access to the CodeBuild filesystem, and temporary credentials whose impact depends on IAM permissions, while noting that the build must remain alive. The stated fix is to remove the TCP listener and retain the mapped Unix socket.
A Pods Architecture To Allow Shopify To Scale
Shopify outgrew its ability to scale by buying larger database servers, so it introduced database sharding in 2015. Sharding improved performance and horizontal scalability but made actions dependent on every shard, increasing the risk of platform-wide unavailability. In 2016, Shopify reorganized its runtime around pods: each pod groups shops on isolated datastores, while shared workers, application servers, and load balancers are restricted to one pod at a time. Sorting Hat assigns each request to a pod and location, while every web request and delayed job is kept within one pod; paired data centers and Pod Mover support recovery without dropping requests or jobs. The design makes a pod the unit for scalability, resiliency, and disaster recovery, and simplifies evacuating an entire data center.
Using HTTP Headers to Secure Your Site
A straightforward Static Buildpack app received an F, scoring 20/100 in Mozilla Observatory because several security response headers were missing, despite redirecting visitors to HTTPS with an Automated Certificate Management certificate. The walkthrough examines failures for Content Security Policy, HTTP Strict Transport Security, X-Content-Type-Options, X-Frame-Options and X-XSS-Protection, then configures them through the static.json headers collection. The CSP restricts resource origins, blocks inline scripts by default, and uses a SHA-256 hash to permit the Google Analytics inline script without adding unsafe-inline. HSTS uses max-age=63072000, includeSubDomains and optionally preload, while other headers supply MIME sniffing, iframe and legacy XSS protections. After the changes, the reported checks pass, the configuration reaches an A and is later described as A+; the final recommendation adds Referrer-Policy and omits opt-in HSTS preload.
How Grab Experimented with Chat to Drive Down Booking Cancellations
Post-allocation ride cancellations at Grab degrade the booking experience and create costly inefficiencies for both passengers and driver-partners. Internal user research and platform data confirmed that rides involving GrabChat conversations had significantly lower cancellation rates by reducing perceived wait times. To scale this interaction without extra cost, the team tested system-generated automated messages sent at varying delay intervals, styles, tones, and localized verbiage across different cities. Faster message delivery outperformed longer delays, and tailored prompts reduced booking cancellations by up to two percentage points across tested markets. The experiment demonstrated that high-quality, directed prompts solicited quick responses and improved pick-up efficiency even when overall message volume was lower than control groups.
Accelerating Android Talent Through Community Bootcamps
Shopify created Android Bootcamp to address its need for mobile talent while giving local developers an entry point into Android development. The two-day workshop targeted people with at least one year of development experience, especially newcomers to Canada, and taught participants to build a simple quiz app. The curriculum covered XML-defined views, Android lifecycles, MVC, JSON parsing, libraries, and remote data fetching, with six Shopify mentors supporting one primary teacher. After screening 33 applications, the team accepted 14 people—seven men and seven women—and received positive feedback, although participants found the room too small. Shopify reported 100% overall happiness, one attendee later joined full-time, another was set to join in May, and a second bootcamp was being planned for June.
Future Proofing Our Cloud Storage Usage
Shopify’s growth from 375,000 to over 600,000 merchants coincided with about 450,000 S3 operations per second and rising error rates plus read and write latency. Merchant-uploaded images and theme installations exposed a partitioning problem: S3 groups keys by shortest common asset prefixes, and abrupt throughput spikes could trigger SlowDown failures across a range of shops. Black-box observations suggested affected keyspaces were unlikely to recur, while newer shops were more often involved, though the latter was explicitly described as a shaky inference. Shopify prepended a hash-generated digest to new shops’ asset keys, distributing writes across more partitions instead of concentrating them under shop-specific prefixes. After the first post-change SlowDown event, exceptions nearly disappeared, while median latency for digested assets fell about 60% and the 95th percentile about 25%.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.