Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
Loading…
Topic
Testing
270 posts about Testing. Every summary links to the original.
- 270 posts
- 42 companies
- 1 explainer
See the idea
Testing explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Accessibility for real-time web apps
Accessibility for real-time web apps argues that accessibility should be treated as an ongoing design and engineering process rather than a late checklist, using blind developer Nolan Darilek’s experience to frame the subject. It explains why reducing interaction barriers benefits everyone, why disability can be temporary or acquired with age, and why teams should plan for access needs from the beginning. Recommended practices include automated checks such as Deque’s Axe DevTools, live audits, testing with people who have specific access needs, and hiring people with disabilities without making one employee solely responsible. The post then identifies Phoenix LiveView challenges in LiveBeats: dynamically loaded routes need transition announcements, live regions need careful updates, and dropdowns and modals require keyboard-accessibility work.
RubyConf 2021: The Talks You Might Have Missed
RubyConf 2021: The Talks You Might Have Missed compiles talks presented by Shopify’s Ruby and Rails infrastructure engineers at the November conference in Denver, offering ways to revisit sessions for attendees who missed them. The program covers Ruby implementation topics including compiler history, Aaron Patterson’s pure-Ruby JIT compiler, YJIT’s incremental integration inside CRuby, parser generators, variable-width allocation, and Ruby memory layout. Other sessions address gradual Sorbet adoption, native extension compilation and security, pair programming, memoization performance, executable code as data, Ruby archaeology, fast CI, and Ractor-based parallel testing. The roundup reports early YJIT performance results, describes a CI environment with 2.8 million Rails monolith lines and 210,000 Ruby tests, and notes variable-width allocation passed Shopify’s Rails CI while serving more than 500 million requests in a week of limited production traffic.
Cloud, Load, and Modular Code: What 2022 Looks Like for Shopify
Shopify’s 2021 BFCM four-day event exposed the scale its commerce platform must absorb: peak app-server traffic exceeded 32 million requests per minute, load balancers surpassed 34 million RPM, and a flash sale used more than 20% of total computing capacity. The post frames larger, spikier sales, mobile and social commerce, and expanding buying environments as enduring engineering challenges for supporting millions of merchants. Planned work includes evolving the large Rails codebase into modular components, moving storefront rendering out of the monolith, expanding Spin cloud development environments, and making in-house load testing part of product development. The storefront split and refactoring produced four-times-faster cache fill rates and five-times-faster page rendering, while YJIT ran 31% faster than interpreted CRuby and 26% faster than MJIT; Shopify also remains committed to Rails and React Native.
Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox
Shopify built a real-time buyer-signal pipeline for Shopify Inbox after noting that 70 percent of its conversations involved customers making a purchasing decision. The system surfaces cart actions and order-completion information as chronological conversation events, covering events from up to 14 days before a conversation and post-conversation events while its seven-day lifecycle remains active. It uses Kafka through Monorail and Change Data Capture, Apache Beam on Google Cloud Dataflow, filtering jobs for mission-critical transactional events, and stateful aggregation with timers and a Global Window to retain buyer activity. In an equal-group A/B test, buyer signals increased merchant response rate by two percentage points and attributed conversion rate by 0.7 percentage points, while response time showed no significant change.
Faster Python Tests
The title, “Faster Python Tests,” identifies test speed as the intended subject. The accompanying material consists of HTML metadata, favicon and RSS declarations, a JavaScript bundle reference, analytics setup, and a notice that JavaScript is required to run the app. It does not state which Python test framework, execution bottleneck, configuration, code change, or measurement was involved. Consequently, no implementation approach, benchmark, performance figure, or conclusion about faster tests is provided in the available text. The only technical details exposed are the page’s JavaScript application shell and its Segment and Google Analytics initialization, neither of which explains how Python tests were accelerated.
Building an App Clip with React Native
The Shop team built an iOS App Clip to reduce the drop-off between a buyer’s checkout thank-you page and opening the full Shop app, where App Store download and onboarding added friction. App Clips are invoked through actions such as QR scans, links, or Smart App Banners and must stay below 10MB, while the team wanted React Native support for order tracking and related functionality. A Hello World build initially measured 28MB; explicitly selecting native dependencies in the Podfile reduced the baseline to 4.3MB, and a scrappy feature build reached 6.5MB. The team used early testing, invocation URL handling through React Native Linking, and shared data containers for app upgrades, ultimately releasing an App Clip at 9.1MB; extensive code sharing proved too complex and some code was copied.
Shopify's Path to a Faster Trino Query Execution: Custom Verification, Benchmarking, and Profiling Tooling
Shopify needed safer ways to evaluate Trino configuration changes and software upgrades as its forked, hundreds-node cluster handled more data and data scientists depended on fast queries. The Data Reliability team built a lightweight Python library that centralizes Trino query orchestration, then exposed it through PyTest fixtures for repeatable verification, including correctness checks and candidate-versus-control comparisons. The broader framework also supports benchmarking and profiling concepts such as replaying historical queries, reproducing traffic patterns, and testing autoscaling, although the project ultimately focused most of its performance-testing scope on verification. Extensive correctness tests helped the Data Foundations team complete a successful Trino upgrade, while p95 query execution time remained stable over the reported period; the library was designed to leave room for more advanced interfaces later.
Try Out YJIT for Faster Rubying
YJIT is Shopify’s just-in-time compiler for CRuby, designed to convert frequently used code into optimized native machine code. Early results include a 20% or more improvement on a simple hello, world Rails benchmark, while production-app benchmarking and one real web service showed about a 6% speedup on a small percentage of traffic. The guide covers building a YJIT-enabled Ruby from source with Autoconf, make, OpenSSL, and GCC or Clang, then switching to it with a Ruby version manager and enabling YJIT explicitly. It shows how to verify installation, compare test and benchmark runs, and collect runtime statistics about generated code, compiled ISEQs, and interpreter versus YJIT instruction execution. The implementation ran Shopify’s full unit tests, similar GitHub tests, and limited production traffic, but remains in progress and invites issue reports about crashes, errors, and speed.
Journey Through a Dev Degree Intern’s First Placement
During her first 12-month Dev Degree placement, the author worked as a back-end developer on Shopify’s Docs & API Libraries team, which builds libraries, tools, and documentation for developers. She describes an initially steep learning curve through work on Shopify CLI, including a change allowing Rails flags such as --db=DB during project scaffolding, and a Shopify Admin API Library for Node written in TypeScript. To handle uncertainty, she kept an Achievement Log, learned by adapting existing code and tests, paired with colleagues, asked for help, and reviewed draft pull requests with teammates. The placement challenged stereotypes about “real” developers and showed that imposter syndrome can coexist with genuinely needing more knowledge. She concludes that learning is collaborative, representation can strengthen inclusion, and becoming comfortable with discomfort is part of development.
10 Lessons Learned From Online Experiments
After completing ten online experiments over the past year, the author describes recurring statistical and implementation traps that can distort experiment results. The discussion stresses choosing independent randomization units, examining whether the analysis unit matches the randomization unit, and using simulations to test suspected false-positive effects before changing an experiment plan. It also shows how request-level traffic analysis and close collaboration with engineering can avoid unnecessary implementation work, while generous logging, A/A tests, and investigation of sample ratio mismatches can expose redirect failures or qualification bugs. The author reports that switching one experiment’s analysis unit from session to user changed its result from negative to positive, and argues for monitoring results without stopping early, investigating anomalies, and slicing averages across important user segments.
Shipit! Presents: How We Write React Native Apps
A May 19, 2021 Shipit! event brought together Colin Gray, Haris Mahmood, Guil Varandas, and Michelle Fernandez, developers setting Shopify’s React Native standards, to discuss writing performant apps. The Q&A covers accessibility and design guidance through Restyle and Polaris documentation; their responsive layouts avoid fixed sizing where possible and use breakpoints for fonts and spacing. They describe React Native state as the usual source of truth while native modules manage internal state and send updates through EventEmitter, with SQLite in Point of Sale as an exception that still has one source of truth. Jest handles unit and integration tests, Buildkite manages CI builds, nightly builds are automatically distributed to testers, and failing feature-branch tests block merges to main.
How Shopify Dynamically Routes Storefront Traffic
Shopify’s Storefront Renderer team rewrote the storefront implementation beginning in 2019 and needed a way to shift traffic between the legacy and new versions while migrating features. Using nginx with OpenResty, Lua modules, a control plane, and the spy chatbot, they moved routing rules from a Lua file into JSON payloads that could be created, updated, enabled, disabled, and deleted without a CI or deployment cycle. Rules selected shops and request subsets through filters, then controlled rendering and verification rates, including forward verification, reverse verification, and self-verification in the nearest region. During nginx request processing, matching occurs in the rewrite phase; verification is scheduled with timers in the log phase so duplicate requests run asynchronously after the client response path. The system supported gradual rollouts, parity checks, and resilience testing under Redis failures, while the team planned to simplify routing after the new storefront served almost all traffic.
Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms
Shopify’s Help Center serves more than 2 million users monthly, making search relevance important across setup, marketing, and troubleshooting queries. The post presents a three-step framework for evaluating new search algorithms against an existing system, combining data collection, offline metrics, and online A/B testing. It contrasts Vanilla Pagerank, which boosts articles by total views across searches, with Query-specific Pagerank, which uses click frequencies from historic searches containing the search term. Kafka event streams are modeled into search facts for near-real-time monitoring and experiment assignment, while Support-team annotations provide relevance labels for offline evaluation, with dataset staleness requiring recurring updates. In the reported A/B test, Query-specific Pagerank users clicked more, relied less on follow-up searches and later pages, and reached a lower average clicked rank; the algorithm replaced Vanilla Pagerank.
Remove Circular Dependencies by Using Dependency Injection and the Repository Pattern in Ruby
An internal Ruby pricing gem used by Shopify Core and Storefront Renderer needed consumer-owned data while retaining shared pricing knowledge, creating a circular dependency. The proposed design puts calculation logic and shared domain types in a stateless gem, while each consumer implements a repository contract for retrieving and returning the required data. Constructor injection passes a PricingRepositoryInterface implementation into PricingEngine::Engine, and Sorbet interfaces and function signatures enforce implemented methods and expected return types. Testing separates gem-isolated tests using repository mocks, consumer unit tests for repository behavior, and integration tests confirming the gem works within each consumer. The result, according to the post, is removal of the circular dependency and a typed contract that makes consumers responsible for data access.
Updates on Shopify's Bug Bounty Program 2021 - Shopify
Shopify’s 2021 bug bounty update reviews a busier 2020 program and outlines planned improvements for hackers and internal responders. Its first virtual live hacking event brought together 38 hackers from seven countries, generated 83 valid reports, and paid more than $220,000 in bounties plus $54,000 in bonuses. Shopify also raised its maximum bounty to $50,000, surpassed $2 million in total awards, replaced its bounty table with CVSS-based scoring and a public calculator, and published a GraphQL hacking guide. To handle volume, the Application Security team created a dedicated HackerOne team, built a dashboard for follow-up questions, and planned more resources and a revised private Experiments program. In 2020, Shopify triaged 215 reports, paid over $460,000, and reported an average first-response time of 25 hours, while average triage and bounty times increased.
4 Tips for Shipping Data Products Fast
Shipping data products under tight deadlines requires teams to stay focused while avoiding overengineering and building something users will not use. Shopify’s Data Science team recommends four practices: run a timeboxed design sprint with a small team and a designated Champion, prototype quickly—often with spreadsheets—and gather feedback throughout development. For first iterations, the post advises using simple approaches instead of machine learning when complexity would delay integration, polish, and user conversations; a sales forecasting prototype used linear regression with two input variables before a more complex model was built. User research should use specific questions and a diverse set of users, as feedback changed a confusing 0–100 scoring display to easier-to-interpret letter grades. Together, these practices provide a clearer path to launching useful products quickly.
Keeping Developers Happy with a Fast CI
Shopify’s Test Infrastructure team tackled a 45-minute 95th-percentile CI time for its Core monolith, which had more than 170,000 tests and ran on every git push. They instrumented Buildkite commands and prioritized agent preparation, dependency building, and test execution using command frequency and duration. Disk I/O from writing large caches slowed Docker startup, so the team increased disk capacity and write speed, mounted most caches read-only, skipped unchanged database and asset work through hashes, parallelized setup, and expanded changed-file test mappings. These changes reduced container-start p95 from 90 to 25 seconds, raised builds avoiding the full suite from 45% to over 60%, improved test stability from 88% to 97%, and lowered overall CI p95 to 18 minutes.
Using Betas to Deploy New Features Safely
Continuous deployment makes changing a running system risky, especially for a platform serving more than a million merchants and changing code multiple times daily. The post presents beta flags as a control mechanism: per-Subject BetaFlags support explicit activation, while BetaRollouts use a percentage and a digest-based modulo calculation to assign a stable, growing subset as rollout increases. It then places a higher-level Feature abstraction above those primitives, allowing direct targeting, percentage rollout, per-Subject opt-outs, and a 100% kill switch that can halt a feature without deleting thousands of database records. The guidance also covers testing permutations, notifying owners through BetaIdentifier metadata, and removing stale flags after rollout. These patterns provide operational control and safer deployment, while requiring substantial implementation and ongoing cleanup.
Reflecting on the Five Years of Bug Bounty at Grab
Grab launched a private bug bounty programme on HackerOne before opening it publicly in 2017 to complement internal product security efforts. Prior to going public, the team conducted security sweeps, established policies, and expanded the testing scope to prepare for an influx of reports. Operationally, the team opted to pay full bounties immediately upon report triage rather than waiting for vulnerability resolution to incentivize security researchers. Noise reduction was handled using HackerOne Triage, Human-Augmented Signal, and by blocking automated scanning networks targeting Grab's infrastructure. In addition, rotating security engineers weekly and integrating HackerOne APIs with PagerDuty ensured fast initial response times.
Resiliency Planning for High-Traffic Events
Shopify’s preparation for Black Friday Cyber Monday (BFCM) centers on building and maintaining resiliency practices for a high-traffic platform and large development teams. Regular load and stress tests verify required capacity, probe limits, detect regressions, and compare behavior across differently configured shops. Teams document expected user experiences and dependency failure modes in a resiliency matrix, then use game days to test that model against reality, including alerts, paging, operator readiness, and response procedures. Incident analysis treats disruptions as learning opportunities rather than stopping at a single “root cause”; nearer to BFCM, Shopify slows feature and service changes while increasing work on performance, reliability, scale, runbooks, tools, and bottlenecks. The source says these practices support a platform that powered over $5.1 billion in sales.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.