Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,738 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Dissecting Kubernetes Deployments
Kubernetes Deployments provide declarative updates for Pods and ReplicaSets, enabling application versions to roll out gradually rather than restarting every Pod at once. The Deployment controller uses informers and triggers for Deployment, ReplicaSet, and Pod events, then synchronizes state by creating ReplicaSets and scaling old and new sets according to replica counts, max surge, and available Pods. During a rolling update, it waits for new Pods to respond before terminating old ones; progress deadlines help detect stalled rollouts, including crash loops. After completion, old ReplicaSets are cleaned up but a configured number is retained for rollback, which requires changing image tags rather than continually using :latest. The post also shows a Go informer that watches Deployment creation and update events and explains that the trigger-based design supports custom controllers.
2017 Bug Bounty Year in Review
Shopify’s 2017 bug bounty review reports higher payouts, four-hour average first responses, four-day triage, and changes intended to improve researcher participation. The program paid $67,550 outside the H1-415 event, with an average bounty of almost $1,100, while 121 reports were resolved and 63.1% of submissions were closed as not applicable. At H1-415, five hackers helped resolve 15 bugs for $42,000, and the program’s largest individual payout, $20,000, addressed unauthorized collaborator access caused by account-conversion logic. Shopify also acknowledges fewer resolved reports and delayed low-severity cases, then announces a $500 minimum payment at triage, clearer scope and payout guidance, a $25,000 typical maximum, and earlier access to beta functionality for eligible whitehat partner accounts.
Implementing ChatOps into our Incident Management Procedure
Shopify’s incident management process separates coordination from technical remediation: the incident manager on call (IMOC) leads response, while component experts fix production issues and a Support Response Manager handles public communication. The process follows a simplified Incident Command System hierarchy and moves from detection and confirmation through communication, mitigation, verification, documentation, investigation, and root cause analysis. Its ChatOps chatbot, Spy, runs in Slack through Lita and integrates with PagerDuty, StatusPage, and GitHub via webhooks, providing commands for paging, incident startup, status communication, mitigation, and stopping incidents. Spy binds incidents to a shared #war-room, records notes and event times, generates service disruption documents, sends status-page reminders, and can summon the IMOC squad when an incident lasts beyond a specified period. The result is a more consistent response that reduces context switching and post-incident toil while supporting detailed post-mortems.
Deep Dive into Database Timeouts in Rails
Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout_timeout, connect_timeout, and read_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.
Scaling ipify to 30 Billion Requests and Beyond on Heroku
ipify began as a free API that returns a caller’s public-facing IP address, created to help discover cloud instances’ external addresses without management APIs. Its initial implementation was a sub-50-Loc Node service deployed on one Heroku dyno for about $7/month; rising usage caused slow responses and 503s. Investigation showed roughly 10 rps, while one Node process and string manipulation around X-Forwarded-For limited throughput; clustering doubled capacity, and the implementation later used Go. Heroku supplied load balancing, automatic restarts, and horizontal dyno scaling as traffic expanded through bursty demand and malicious use; the operator chose manual capacity rather than paid autoscaling. The service surpassed 30B requests per month, routinely serving 2k–20k RPS with 1–20ms average response times at $150–$200 monthly; ownership later passed to whoisxmlapi.com, whose work includes an improved UI and more data endpoint.
Dealing with the Meltdown Patch at Grab
AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.
Kafka Streams on Heroku
Kafka Streams on Heroku shows how to organize, deploy, and scale Java stream-processing services using Kafka Streams, Apache Kafka, and the Heroku Runtime. The example contains a Text Processor that splits input from a textlines topic into words, an Aggregator that counts words in 10-second windows before writing to Postgres, and an independent Anomaly Detector that routes 10-second login-failure counts to urgent alerts or non-urgent email based on thresholds. A Gradle multi-project setup packages each service as an executable and uses a Procfile to run them as separate Heroku workers, while managed Kafka supplies the event-streaming infrastructure. Scaling is explained through Kafka partitions and Kafka Streams Stream Tasks: Stream Threads provide vertical scaling, dyno counts provide independent horizontal scaling, and Kafka Streams redistributes tasks when workers change or fail.
Jekyll on Heroku
Jekyll can be deployed on Heroku to use its plugin system, but relying on jekyll serve would use Ruby’s single-threaded WEBrick rather than a production-oriented web server. The setup installs Ruby with ruby-install and chruby, then installs Bundler and Jekyll, creates a site, and tracks it in Git. Heroku’s official Ruby buildpack handles dependencies and runs a Rake assets:precompile task that executes jekyll build, while the Static buildpack serves the generated _site/ directory according to static.json. The example pins Ruby 2.4.2, adds Rake, enables clean URLs and HTTPS, and completes with a successful deployment of the generated site to a Heroku URL.
GrabShare at the Intelligent Transportation Engineering Conference
Grab presented a technical paper on the construction of its real-time ridesharing service, GrabShare, at the Intelligent Transportation Engineering Conference in Singapore. The platform pairs passengers heading along similar routes with drivers immediately while handling network drops, volatile supply and demand, and heavy traffic conditions in Southeast Asian cities. To deliver accurate pairings, the scheduling system generates and filters through hundreds of travel time estimates for each candidate match before finalizing an itinerary. Operational teams on the ground evaluate complaints about poor matches, enabling engineers to refine the online matching systems. Over the course of one month, the service cut more than 4.5 million kilometers of driving distance and brought in over 100,000 new users within two weeks.
Grabbing Growth: A Growth Hacking Story
Grab established a dedicated Growth Hacking team within its Technology organization to pursue high-risk, niche initiatives and scale impact across 68 million regional users. Positioning the team within engineering allowed rapid A/B testing and simultaneous multi-market deployments driven by structured growth loops. The team prioritizes and evaluates all initiatives using a Growth Factor metric, calculated as the increase in rides divided by the increase in costs. To improve driver engagement, the team implemented a Spin-to-Win game based on B.F. Skinner's variable ratio reinforcement principles, delivering probabilistic monetary and merchandise rewards upon meeting daily ride thresholds. Ongoing regional experiments continue to evaluate metrics including driver acceptance, cancellation rates, and driver ratings.
Namespacing Actions for Redux
Kickstarter’s front-end team encountered a Redux action-namespacing problem when multiple instances of the same functional React component responded to actions intended for only one instance. They rejected local state because the application used functional components and redux-loop, which routes state updates and side effects through reducers, and they sought programmatic namespacing compatible with combineReducers and a low-surface-area component API. The team compared nested reducers, higher-order reducers, and a closure-based module pattern; the latter met the stated constraints with namespaced actions, wrapper components, reducers, and a namespace returned by a factory that defaults to a UUID. Namespaces must be preserved when reducers trigger further actions, including redux-loop results, and complex composition can lose the required binding. The approach has worked for most reusable and compound components, while possible performance costs from many reducers remain a future concern.
How Shopify Merchants can Measure Retention
Shopify addresses customer retention for merchants operating non-contractual businesses, where churn is not directly observable. Common rules that label customers inactive after N days create arbitrary binary states, so the post proposes probabilistic modeling with the BG/NBD (Beta Geometric / Negative Binomial Distribution) model. It represents each customer with a purchase rate and churn probability, estimates their population distributions, and uses age, recency, and repeat-purchase frequency as sufficient statistics. Shopify fits a modified implementation in lifetimes using Apache Spark DataFrames, subsampling up to 50k customers per shop for performance, and produces activity probabilities and expected future purchases for more than 500 million historical merchant customers. The model is interpretable and supports customer ranking and targeted campaigns, but it handles seasonality poorly and does not easily incorporate additional variables such as country or products purchased.
FRP at Kickstarter and Kotlin
Kickstarter’s open-source iOS and Android apps use Functional Reactive Programming (FRP) patterns to keep views simple and move mutation and complex logic into thoroughly tested ViewModels. The post describes how this discipline helped a developer ramp up quickly on the Android codebase after working on Kickstarter’s backend and payments teams. It also introduces the Android team’s interest in Kotlin, a JetBrains language that runs on the JVM, while noting that dependency concerns had prevented conversion of core functionality. For a 2017 Functional Swift Conference talk in Berlin, the author used Kotlin to add a simple Android screen following the team’s established FRP patterns, presenting the implementation as live coding and an introduction to FRP and Kotlin.
Upgrading Kickstarter to Rails 5
Kickstarter upgraded its main Rails application to Rails 5, alongside payments app Rosie, to support site security and performance while moving a nine-year-old legacy codebase forward. The team planned a three-month project, upgraded Ruby 2.2.5 to 2.4.x, classified release-note changes, resolved dependency conflicts, used Rails’ upgrade script, and moved through twelve documented steps. Work included merging Rails 4.2-compatible fixes, repairing nearly 4,000 CI test failures from a roughly 70 percent coverage baseline, removing deprecations, smoke-testing major flows, backporting more than thirty pull requests, and preparing reviewers and deployment contingencies. The upgrade met its timeline without rollbacks or major downtime, although untested bugs increased Zendesk tickets and exposed issues involving ActionController::Parameters, tracking events, and mobile applications. A retrospective emphasized planning, documentation, cross-team communication, and broader native-team participation for future upgrades.
How We Enable Our Interns to Make an Impact
Shopify describes an internship model in which interns work on projects affecting merchants, partners, and developers, including merchant pain points such as selling across channels. The program combines autonomy comparable to that of full-time employees with mentorship, regular pair programming, unfamiliar challenges, and cross-team collaboration. Examples include launching the Amazon Channel, changing an Android app design pattern across multiple codebases while limiting effects on third-party developers, backfilling affiliate data with Elasticsearch and Ruby on Rails, and generating Java and Swift models from a strongly typed GraphQL schema. Interns are encouraged to investigate open-ended problems, consult documentation and stakeholders, and develop informed recommendations with mentor support. Returning interns can change teams or roles, while tools such as Slack enable direct communication across the organization; the stated result is practical experience, deeper technical knowledge, and potential full-time opportunities.
Tell Your Stories: The Benefits of Strategic Engineering Communications
Shopify created engineering communications in early 2016 as rapid growth and expansion from four to five offices made informal knowledge sharing insufficient. The program serves internal and external audiences through weekly technical talks, an internal newsletter, engineering-blog posts, conference presentations, and an editorial calendar, with Shopifolk developers treated as a primary audience. Communications staff hunt for stories by building relationships with project leads, then coach speakers and work alongside developers while preserving their voices; coaching covers narrative, breath control, body language, and calls to action. The reported effects include less risk of duplicated work or mistakes, cross-pollination of ideas, more confident speakers, increased interest in Shopify from some attendees, and a recurring pipeline in which storytelling generates further stories.
The Data and Science Behind GrabShare Part I: Verifying Potential and Developing the Algorithm
Expanding from point-to-point dispatch services to dynamic carpooling requires matching independent passenger requests traveling in similar directions without causing unacceptable delays. Grab evaluated the feasibility of its GrabShare service by analyzing historical trip data with DBSCAN clustering on coordinates projected into a Universal Transverse Mercator system. This analysis demonstrated that 35% to 46% of rides across typical daytime windows fell into tight geographic clusters with near-identical pickup and drop-off coordinates. The resulting matching framework adapts the baseline dispatch flow by searching for in-transit drivers and enforcing real-time seat reservation constraints. Route assignment decisions subsequently evaluate detour times, trip angles, and expected arrival times to ensure driver utilization improves while total driving distance decreases.
How Shopify Governs Containers at Scale with Grafeas and Kritis
Shopify describes using Grafeas and Kritis to govern container software across a platform handling more than 6,000 container builds per day and 330,000 images in its primary registry. Grafeas serves as a central store for metadata about container origins, builds, base images, layers, registry pulls, installed packages, vulnerabilities, and deployment status, supporting auditing and lifecycle strategies. During CI, audits produce attestations, which Shopify signs with PGP; Kritis uses named attestation authorities and Kubernetes policies to require specific attestations before deployment. Policies can require containers to be built by Shopify, sourced from a trusted repository, free of newly introduced vulnerabilities, tested, non-root, and deployed with the appropriate security context. Shopify also generates attestations from kubeaudit results for workload manifests, helping catch runtime-security regressions before deployment.
The Art of Hiring Good Engineers
Building and scaling engineering teams requires adapting recruitment strategies across different organizational sizes, from initial hires to hundreds of engineers. Teams often begin with small groups of battle-tested engineers before designing formal hiring pipelines. The recruitment approach outlines four key steps: sourcing top-tier talent from platforms like GitHub and LinkedIn, assessing technical fit through tailored screens or assignments, evaluating cross-functional and culture fit via group interviews, and optimizing the process. This optimization includes routing candidates strategically to strict bar-raiser interviewers to prevent burnout and expedite decisions.
In the Cloud, No One Can Hear Your OutOfMemoryError
Heroku introduces Heroku Exec and Language Runtime Metrics to improve production monitoring, inspection, and debugging for applications running on dynos. Heroku Exec creates secure TCP and SSH tunnels for SSH sessions, port forwarding, remote debugging, and diagnostic tools, while the Heroku Java CLI integrates jmap and VisualVM. A synthetic Java memory-leak example shows VisualVM tracking heap growth and jmap producing a downloadable HPROF heap dump that Eclipse Memory Analyzer identifies as Servlet-held FakeLeak objects. The metrics feature, enabled through the Heroku Metrics buildpack on paid Standard-or-above dynos, displays heap, non-heap, and garbage-collection activity in the dashboard through a Java agent using Prometheus; JVM-language support is generally available, while Go support is in beta.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.