Loading…
Browse by type
Explainer reads
Explainer posts on Yomu, summarised newest first.
- 1,382 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Explainer posts
Getting to Know Python 3.7: Data Classes, async/await and More!
Python 3.7 is presented as an upgrade for developers seeking new language features, library improvements, and performance gains over earlier Python 3 releases. The article focuses on data classes, where the @dataclass decorator inspects typed attributes and generates an init method, reducing boilerplate for data-oriented classes. It also covers asyncio.run for simpler asynchronous execution, breakpoint() for entering pdb, module-level getattr for lazy loading, and context variables that keep values separate across execution contexts. Dictionary insertion order becomes part of the language specification in 3.7, while OrderedDict remains available for advanced behavior; reported improvements include 10–30% faster startup, 45–70% faster common sorting cases, 5.5-times-faster dict.copy(), and 4–6-times-faster namedtuple creation.
Building Shopify’s Application Security Program
Shopify's application security program is designed to preserve safety while products and services scale quickly across a platform serving more than 800,000 merchants. It organizes the work around scaling secure applications, scaling security teams, and scaling security interactions, with a culture that treats security as a non-negotiable priority. Standardizing on Ruby on Rails and other shared technologies lets tooling go deep, while service tracking provides visibility into applications, dependencies, code, and automated dependency updates. The security organization grew from one employee into specialized teams, supplemented by penetration testing, bug bounties, and audits, while incident response remains shared across the company. At the code level, tripwires detect dangerous patterns early; Shopify renamed html_safe to dangerously_output_as_html, links flagged pull requests to XSS training, and alerts Application Security, shifting review toward automation.
Turn Your Code into Docker Images with Cloud Native Buildpacks
The post presents Cloud Native Buildpacks (CNB) as an alternative to hand-written Dockerfiles for turning application source code into Docker images. It argues that Dockerfiles make developers manage language runtimes, system packages, dependency caching, cleanup, and repeated maintenance, creating a leaky abstraction between application and deployment concerns. CNB uses app-aware buildpacks to detect a project’s language from files such as Gemfile or pom.xml, reuse work from previous builds, download dependencies, prepare the application, and export an OCI-compliant image. The beta buildpacks.io tooling provides pack build and a Heroku builder supporting Ruby, Node.js, Java, Python, PHP, and Go; pack rebase can update an image’s toolchain without rebuilding it. The stated conclusion is that this approach reduces operational complexity while retaining compatibility with Docker tooling and the broader container ecosystem.
Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering
Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.
Bug Bounties and Black Swans: How Heroku Expects the Unexpectable
Heroku frames security as an engineering discipline that must challenge happy-path assumptions, test continuously, and prepare for low-predictability, high-consequence “black swan” events. Its threat-modeling guidance focuses on challenging defaults, recognizing that attackers have resources and organizational support, and planning for unexpected failures such as a zero-day exploit. A control-plane instrument for Heroku Postgres detected privilege escalation to a superuser role, and the Data team isolated the instance within 17 minutes while investigating the activity. The event was traced to researcher Andrew Krasichkov (“buglloc”), whose testing exposed a vulnerability in the dblink and postgres_fdw extensions. Through the Bugcrowd bug bounty program, Heroku worked with him to understand the issue, reward the finding, and defend against the previously unknown flaw before a patch was available.
Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks
Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services.
Meet Shopify Developer Kirsten Westeinde - Shopify
Kirsten Westeinde’s just-over-five-year Shopify career spans internal tools, partner-facing applications, and backend development on Kit as the company scaled. She began with Ruby on Rails, later helped extract partner software from Shopify Core using Elasticsearch, Kafka, Ruby on Rails, and the Shopify API, and now works with Data Engineering on real-time recommendations. The feature combines Go and Ruby and is designed to handle the current load while scaling to future demand. Her problem-solving practices include whiteboarding system interactions, taking mental breaks, and writing or explaining ideas to align collaborators. She also describes mentorship, pair programming, a women’s developer meetup, and f(empower) as ways to support growth and address exclusion, bias, and microaggressions in technology.
Seven Ways to Fortify Your Application
Applications can fail when a single upstream service goes down, prompting tools that shed load, fail gracefully, and protect struggling dependencies. The discussion frames this as application resilience for everyday failures, distinct from request-level retries, rollback strategies, and catastrophic disaster recovery. It presents maintenance mode as an easily toggled hard stop with a clear message and status-page link, read-only mode as a way to preserve access when modifications fail, and feature flags as controls for global or meaningful groups. Implementation examples include environment variables, relational databases, Redis, in-memory caches, and an S3-based check, with state stored primarily in Postgres and Redis at Heroku. The closing guidance says maintenance mode should be easy to toggle, while read-only mode can preserve user confidence and access to information.
Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting
Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.
Context Deadlines and How to Set Them
Microservice architectures operating under heavy network traffic require robust timeout handling to prevent slow or failing dependencies from causing cascading failures across services. Naive, static timeout configurations across a call chain often cause downstream components to waste compute effort on requests that upstream callers have already abandoned. To establish predictable timeout thresholds, engineers can align limits with service-level latency percentiles, pairing P99 limits with median latency retry allowances. Go's context package improves upon static network timeouts by propagating request-scoped deadlines and cancellation signals across service boundaries. This distributed context ensures downstream servers recognize remaining time budgets and terminate unneeded processing immediately when parent deadlines expire or callers manually cancel requests.
Shopify Devs Share Lessons on Self-Advocacy and Dealing with Adversity - Shopify
Shopify’s Behind The Code series profiles Stella Lee, Helen Lin, and Cathryn, three developers discussing self-advocacy, adversity, management, inclusion, and career changes in technology. Lee describes building Shopify Checkout Experience features in TypeScript and Ruby to reduce manual checkout fields, while leading the f(empower) women’s employee resource group; she also discusses imposter syndrome, mentorship, and turning initial “stupid” questions into better ones through timeboxing and research. Lin, a remote Themes team lead, emphasizes aligning projects with long-term company vision, managing stakeholders, staying connected, and directly requesting career opportunities while sharing accomplishments. Cathryn recounts moving from health sciences into programming and stresses evaluating personal goals, hiring a more diverse workforce, and using meeting-room paper pyramids to support inclusive discussion. The article closes with advice to embrace feedback and own both accomplishments and failures.
Reactive Programming with Salesforce Data
Salesforce Platform Events and Change Data Capture enable custom applications to react to activity in Salesforce, motivating an event-sourced design for responsive, durable integrations. The article demonstrates a Node.js app on Heroku that consumes the Salesforce Streaming API through jsforce 1.9.0 or later, whose Faye-based streaming client handles authentication, network connectivity, and message callbacks. It notes that a basic subscription loses its position on restart and cannot coordinate parallel processing, then uses Redis to persist each topic’s Replay ID and distribute messages through pub/sub or reliable queues. Because the Salesforce Streaming API lacks consumer-group delivery, a singleton stream_consumer feeds Redis while scalable web or worker processes handle work. The design also recommends a separate non-admin integration user and monitoring connection status, active clients, last message time, and processing duration.
Ten Ways to Secure your Applications
The post, adapted from Joe Kutner’s Devoxx 2018 talk, presents ten ways to strengthen web application security, with examples focused on Java but applicable to modern languages. It recommends keeping dependencies current, validating explicit payloads with schemas, checking regular expressions for ReDoS, throttling abusive requests, and keeping credentials outside the codebase. It also advises secure-first coding, certificate validation for outgoing connections, logging security-relevant actions without recording secrets, and monitoring suspicious behavior with tools such as AppSensor. The examples include Maven Versions Plugin, Snyk, Bucket4j, saferegex, environment variables, and EnvKeyStore, with a token-bucket algorithm proposed for request throttling. The conclusion is that security requires anticipating unintended functionality across network, platform, and physical domains, while these practices provide a starting point.
Understanding Supply & Demand in Ride-hailing Through the Lens of Data
Grab measures ride-hailing supply and demand across space and time to resolve geo-temporal allocation mismatches between moving drivers and ride-seeking passengers. The analytics pipeline defines supply as idle online drivers and demand as passengers checking fares within brief time slots, aggregating locations into geohashes. Each driver is mapped across neighbouring demand units and inversely weighted by straight-line distance, which yields the effective supply, supply-demand ratio, and supply-demand difference for each geographic polygon. Grab uses these aggregated metrics to identify marketplace imbalances, deploying driver heatmaps to shift excess supply and passenger travel trend widgets to defer time-insensitive ride requests.
How Shopify Uses Recommender Systems to Empower Entrepreneurs
Shopify uses recommender systems to help merchants find relevant insights, actions, apps, themes, services, and other resources across its commerce platform. Its approach represents implicit signals such as purchases, installations, clicks, and views as user-item interactions, then applies binary classification and linear methods including SLIM and LRec to learn an item-item similarity matrix and rank unseen items. The framework supports task-parallel training, interpretable coefficients, user-feature enrichment, and cold-start handling for new users or items, while evaluation considers precision alongside diversity, novelty, serendipity, and business-specific goals. In initial A/B tests, personalized app recommendations produced a 50% higher install rate, home feeds were up to 12% more likely to be rated useful, expert matching increased response and collaboration, and theme recommendations made merchants over 10% more likely to launch stores.
iOS Application Testing Strategies at Shopify
Shopify describes an iOS testing strategy for a monorepo containing multiple app projects and evolving through hundreds of weekly commits. The approach covers Models, Views, ViewModels, View Controllers, and user workflows with tests matched to each component’s role: regular unit tests for business rules, snapshot tests for visual behavior, and XCUITest UI automation for user-facing flows. UI automation selects elements through accessibility labels or strings, which can also improve accessibility for visually impaired users, while manual testing remains necessary for some animation and rendering issues. The article recommends a test pyramid because slower tests require more human effort and maintenance, and says suites must evolve as business requirements change.
Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)
Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.
Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)
Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic.
Preparing Shopify for Black Friday and Cyber Monday
Shopify’s Production Engineering team prepares its platform and 600,000+ merchants for Black Friday and Cyber Monday traffic without interrupting service. Preparation combines controlled fault injection “game days,” synthetic load tests, and staged freezes: a feature freeze begins several weeks ahead, followed by a code freeze allowing only critical fixes during the event. The tests probe dependencies such as shipping providers and payment gateways, including latency, fallback behavior, and responses to datastore failures. After the 2018 weekend, Shopify reported nearly 11,000 orders per minute, around 100,000 requests per second for extended periods, and overall metrics about 1.8 times 2017 levels. The account concludes that fault injection, load testing, and disaster role-play improved readiness, while third-party downtime remains Shopify’s responsibility to merchants and buyers.
Director of Engineering, Lawrence Mandel Talks Road to Leadership, Growth, and Finding Balance.
Lawrence Mandel, a Director of Production Engineering at Shopify, describes a career that moved from software development into technical leadership and the lessons behind that transition. He leads Developer Acceleration, a team that builds tools, conventions, and documentation so Shopify’s development organization can scale effectively. His path included web services, XML, open source, release engineering, project and release management, and self-taught Perl and CGI programming during a work term at IBM. Mandel says leadership required accepting that he could not know every technology or always be the most knowledgeable person in the room, while trusting others’ expertise. He also says prioritizing family and health before work improved his relationships and made his work more deliberate and efficient.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.