Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
Testing
270 posts about Testing. Every summary links to the original.
- 270 posts
- 42 companies
- 2 explainers
See the idea
Testing explainers
Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Avoiding silent errors in your Stripe integration
Stripe integrations can appear healthy while API and webhook failures accumulate unnoticed when applications lack robust logging and alerting. The post demonstrates Stripe Workbench, a Dashboard view that stores available information by default, obfuscates sensitive data, and provides searchable logs without code changes or additional software. In the example, about 50% of daily API calls fail; the Errors tab groups failures and connects each type to sample requests and related logs, revealing repeated card-verification errors and a request associated with an “Attacker Engineering Test Key” whose Go user-agent conflicted with the stated .NET backend. The investigation identifies a team testing attack-like API calls in production and recommends stopping production-key testing, avoiding test requests in production, and using Sandboxes.
Developing and investigating subscription data flow
The article presents Stripe Sandboxes as isolated environments for developing and investigating subscription systems without affecting production data or settings. It explains how to create subscriptions, inspect dashboard actions through Workbench, and review API requests, JSON resource data, and webhook events such as customer.subscription.created and invoice.created. Test Clocks allow developers to advance time and examine subscription cancellation behavior, including the customer.subscription.deleted event after a scheduled cancellation. Workbench’s Shell tab, API Explorer, and Print SDK Request feature support testing requests in the browser and converting them into implementation code, while the article stresses testing plan changes, trial expirations, cancellations, and payment failures.
Preparing for Stripe API upgrades
Upgrading a third-party API can bring new features and fixes but may also introduce breaking changes, so applications should be evaluated before production changes. Stripe Sandboxes provide isolated test accounts with separate API keys and access controls, optionally copying production configuration without copying products, customers, or transaction data. The workflow uses Dashboard Workbench to inspect and upgrade the sandbox API version, then runs the application with sandbox credentials and forwards webhook events through the Stripe CLI. In the example, Stripe.net 41.0.0 rejected an event because it expected API version 2022-11-15 while the sandbox delivered 2024-09-30.acacia; upgrading the SDK to 46.0.0 resolved the mismatch.
Optimize payment flow while reducing code complexity with Stripe's A/B Testing
Payment method choices affect ecommerce conversion, and the post notes that 81% of surveyed customers frequently abandon carts when their preferred method is unavailable. It explains how Stripe’s dynamic payment methods in Elements and Dashboard settings can apply country, region, and amount rules without embedding those conditions in PaymentIntent code. The proposed workflow uses Dashboard experiments to enable methods such as Klarna, set display rules and traffic allocation, then uses Workbench Inspector, Logs, and Events tabs to inspect presented payment_method_types, API requests, and webhook history. A Klarna example reports a 16% increase in average order value, while new payment methods were deployed within four days.
Easily debug your 3DS authentication with Stripe Workbench
Stripe Workbench provides browser-based tools for investigating EMV 3DS authentication and related payment flow behavior in the Stripe dashboard. It lets developers inspect Payment Intent and Charge data, including payment_method_details.card.three_d_secure.authentication_flow and result, to determine whether authentication used a challenge UI and whether it completed successfully. The Inspector also exposes API history and webhook events, while the Events tab can filter historical payment_intent.requires_action events and show their frequency over a selected period. In the Shell tab, stripe trigger payment_intent.requires_action simulates a 3DS-related flow and generates event data without installing additional tools, allowing webhook handling to be tested and investigated from the dashboard.
Testing subscriptions with Stripe Test Clocks and Workbench
Stripe subscriptions require testing scenarios that unfold across billing cycles, and the post presents Test Clocks and Workbench as no-additional-cost test-mode tools for doing so without waiting real time. It provisions three monthly recurring product tiers, creates a test clock with a Unix Epoch FrozenTime, and attaches newly created customers and subscriptions to that clock using the Stripe .NET SDK. Advancing the clock by one month updates time-dependent objects, allowing Workbench Inspector logs and events such as invoice.created and customer.subscription.updated to be examined alongside payment and subscription state. A second simulation attaches pm_card_chargeCustomerFail, makes it the default payment method, and advances to the next billing period. The resulting failed invoice, canceled subscription, and customer.subscription.deleted cancellation_details show how these tools reveal the events and properties an integration should test.
Choosing the right sandbox strategy for your organization
Stripe sandboxes provide isolated environments for testing Stripe features without affecting a live integration, with access controls for the people who use them. Each live account can have up to five sandboxes, which admins can create, name, configure by copying selected live-account settings, and delete when no longer needed. The guide describes a staged setup for startups and small teams, using separate environments for development, integration testing, user acceptance testing, and spare capacity, while larger organizations may assign sandboxes to departments and rotate them between projects. It also presents an iterative model in which teams prototype, collect stakeholder feedback, retest changes, and document results. The recommended strategy depends on team size, organizational structure, workflow, and development methodology.
Testing a React App in Chrome with Heroku CI
The article presents a walkthrough for automating end-to-end browser tests for a React application in Heroku CI rather than relying on manual local testing. The sample app uses Jest and Puppeteer to check navigation to Puppeteer Documentation, retention of text entered into a form, and the greeting shown after submission, with headless Chrome supplied by the Chrome for Testing Heroku Buildpack. Local execution initially exposes a mismatch between the expected and received greeting, then passes after the application is fixed; a streamlined command starts the server and runs the suite. Heroku CI is configured with an app.json file containing Chrome for Testing and Node.js buildpacks, plus a test script that runs start-server-and-test and the Jest/Puppeteer tests. Pushes to the connected GitHub repository trigger the suite.
Avoiding test mode tangles with Stripe Sandboxes
Stripe integrations can become difficult to test when an account has only one test mode, because unrelated work can interfere and sensitive configuration must remain protected. The post presents Stripe Sandboxes as separate testing areas and shows a Node.js Express application authenticating with a sandbox secret key stored in a .env environment variable. It creates payment intents through /create-payment-intent and retrieves account details through /account, then uses Jest and supertest to verify the sandbox identity, including the expected display name dev-sandbox. A GitHub Actions workflow injects STRIPE_API_KEY from repository secrets, installs dependencies, and runs tests on pushes to the main branch, while .gitignore excludes environment files. The conclusion is that isolated sandboxes and automated checks reduce test tangles and help prevent calls to the wrong environment.
Stripe Developer Digest (August 2024): Workbench is now available to everyone
Stripe’s August 2024 developer update announces that Workbench is available to everyone across every Dashboard surface, addressing developers’ request for in-context guidance and debugging tools during API integration work. Workbench is a developer home in the Dashboard that provides an at-a-glance view of integration behavior, lets users inspect API and event history, prototype integrations, and receive critical account alerts. As developers build, it highlights errors, helps resolve them, and recommends ways to improve integration efficiency; it can be opened with a single keystroke. The update also places Sandboxes and Event Destinations in public beta, expands Tap to Pay on iPhone with Stripe to seven countries, and notes community resources for Stripe Tax, the Stripe CLI, Payment Links, and related integrations.
Observing immediate versus delayed payments with Stripe Workbench
Stripe payment methods differ in how quickly they confirm attempted payments, with cards reporting status immediately and bank debits, bank transfers, and cash-based vouchers requiring additional processing time. The post uses Stripe Workbench and its in-browser CLI shell to create Checkout sessions in test mode, submit card and ACH Direct Debit payments, and inspect emitted events. For cards, charge.succeeded and payment_intent.succeeded signal successful payment, while integration code must inspect checkout.session.completed payment_status rather than confuse it with the session status. With delayed methods, checkout.session.completed can show unpaid before checkout.session.async_payment_succeeded or checkout.session.async_payment_failed arrives; the former changes payment_status to paid, and production processing can take days.
How we evaluated the business impact of marketing campaigns
Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.
Grab Experiment Decision Engine - a Unified Toolkit for Experimentation
Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.
Macramé: Untangling the Knot on the Etsy Android Listing Screen
The Etsy Android listing screen had accumulated nearly 200 experiments in three years, leaving a 4,000-plus-line Fragment, inconsistent patterns, low test coverage, and costly, brittle tests. Macramé replaced that legacy design with immutable state propagated through StateFlow to a reactive UI, transforming screen data into smaller ListingUiModels rendered by RecyclerView and ListAdapter. Sealed Events move through a single Dispatcher to focused Handlers, which update State, start asynchronous work, or emit SideEffects for operations such as navigation and logging. After a five-month rewrite, the new screen’s Time to First Content fell 18%, from 1,585 ms to 1,298 ms, while listings viewed rose 2.4% and business-logic test coverage reached 76%. The architecture still has wiring boilerplate, debugging challenges around Handler interactions, and possible overkill for simpler screens, but its use expanded to Cart and Shop.
Increasing velocity by modernizing a Python codebase
An expanding monorepo of machine learning workflows became difficult to change safely as contributors and projects multiplied, undermining the velocity it was intended to support. The team replaced requirements.txt with Poetry, using pyproject.toml for direct dependencies and poetry.lock to record transitive dependencies and hashes, while virtual environments simplified setup. Ruff standardized formatting, linting, and CI enforcement; pytest addressed corner cases and bad assumptions, and mypy added annotations and stricter checks that could expose incompatible changes across call sites. To adopt these controls incrementally, the team used ignore comments and staged remediation, including a script for mypy exceptions, while an ignored revision preserved useful git blame history. The resulting codebase was kept in better shape, enabling faster iteration with greater confidence.
Horizontally scaling the Rails backend of Shop app with Vitess
Shop app’s hockey-stick growth pushed its Ruby on Rails backend and MySQL primary datastore toward capacity, with multi-terabyte storage, week-long schema migrations, and throttled background jobs. After splitting databases and scaling surrounding systems, the team adopted Vitess to shard and re-shard MySQL while retaining SQL compatibility and self-managed primary datastores. They organized user-owned data into a sharded “users” keyspace using user_id, kept other data in an unsharded “global” keyspace, and built application-layer query verifiers to validate routing, distribution, and transaction correctness before migration. The move made schema migrations take hours instead of weeks, removed the reported capacity-driven throttling, and enabled further scaling by adding shards, while introducing concepts such as Sequences, Vindexes, and VSchemas that developers must learn.
Championing CyberSecurity: Grab's bug bounty programme in 2023
Grab reviewed the performance and milestones of its security bug bounty programme for 2023, marking eight years of running the initiative. Operating quarterly campaigns through HackerOne alongside dedicated internal teams, the company processed approximately 1,000 vulnerability submissions over the year, including 400 reports during Q1 alone. Cumulative payouts to external security researchers surpassed $700,000 since the programme's 2015 launch. To adapt to an expanding service portfolio, Grab broadened its scope to encompass joint ventures and acquisitions while introducing diversified financial and recognition-based rewards.
Router 2.0 Common Runtime Router: The Road to Beta
Heroku’s Router 2.0 is a beta replacement for the Common Runtime’s Erlang-based “Hermes” router, which is planned to be replaced in 2024. The project addresses the difficulty of safely delivering upgrades from a legacy custom HTTP server and aims to share routing code with the Go-based Private Spaces router; common functionality has been extracted into an internal HTTP library. Architecturally, Router 2.0 is an L7 router running in memory behind the existing L4 proxy, alongside Hermes, allowing app-level traffic to move from the old path without a big-bang cutover. The team added continuous load testing that exercises scaling policies and blocks deployments missing internal service-level objectives, and introduced Network Error Logging to measure browser-observed failures. Feature parity is still incomplete, with HTTP/2 planned next; apps can opt into the beta using a Heroku Labs command.
What I learned taking Ramp Bill Pay from 0 to N
An engineer recounts lessons from taking Ramp’s Bill Pay from its first code in April 2021 to a publicly launched product that eventually became a product with multibillion annual total payment volume. For 0-to-1 work, the account emphasizes protecting core abstractions, including canonicalized payment statuses that reduce varied payment states to five, using mocked functions to coordinate parallel engineering, and reaching beta customers quickly. Customer feedback, session replay, and persistent iteration are presented as necessary because the first customer will usually expose substantial flaws. For 1-to-N growth, manual processes and duct tape must be prioritized for replacement according to user impact, error risk, and workload. Growing stakes also require testing, monitoring, contingency plans, gradual rollouts, QA, and documentation that helps engineers self-service onboarding.
Contributing support for a Wasm instruction to Winch
Adding a WebAssembly instruction to Winch, Wasmtime’s baseline non-optimizing compiler, requires coordinated changes across code generation, register management, ISA-specific emission, and tests. The contribution described adds visitor support, delegates through CodeGenContext and MacroAssembler, emits x86_64 instructions through Cranelift-backed assembler methods, and registers the instruction for differential fuzzing. CodeGenContext keeps the value stack, register allocator, and function frame consistent, while spilling can move values to linear-memory offsets; MacroAssembler handles ISA differences, register constraints, and CPU feature fallbacks such as lzcnt versus bsr-based clz. Filetests inspect generated machine code without executing it, while differential fuzzing compares Winch results with another WebAssembly engine across randomly generated modules and inputs. The workflow also includes running fuzzing, incorporating draft-PR feedback, rebasing, and submitting the public Wasmtime pull request.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.