Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
Testing
270 posts about Testing. Every summary links to the original.
- 270 posts
- 42 companies
- 2 explainers
See the idea
Testing explainers
Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How we improved our iOS CI infrastructure with observability tools
Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions.
Evolution of quality at Grab
As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%.
Responsibly mid-size: What Matters, Suffers
Moving into a mid-size company requires adopting processes and attitudes piecemeal rather than suddenly operating like a large enterprise, while preserving the ability to deliver customer value. The post challenges “clean code” as a universal standard, arguing from Sandi Metz’s discussion of churn and semantic complexity that important, frequently changed files are often large and complicated, making “bad code” an emergent feature of successful systems. It also uses Joel Spolsky’s case against rewrites to explain that ugly code may contain accumulated fixes for real bugs and operating conditions. The resulting advice is to distinguish code that manages responsibilities from code that does not, develop engineering judgment, and avoid letting aesthetic reactions damage collaboration, team cohesion, or delivery.
The Hardest Part of Writing Tests is Getting Started
The article presents getting started with automated testing as the main obstacle, especially when a full suite requires helper tools, stubs, and fixtures that can overwhelm beginners. It argues that tests serve as executable documentation, preserve intended behavior and edge cases, and expose regressions before changes reach production. The recommended progression is to install a framework and verify both passing and failing tests, cover isolated functions, add stubs with sanitized API-response fixtures, and then build end-to-end tests while monitoring coverage. In an ETL example, API stubbing avoided expensive, slow calls, while database checks validated loaded and transformed data; the resulting tests covered more than 80 percent of the codebase and saved time for the author and team. The article also proposes E2E tests as a way to safely approach legacy code before refactoring it.
Performance Testing At Scale—for BFCM and Beyond
Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.
From Ruby to Node: Overhauling Shopify’s CLI for a Better Developer Experience
Shopify rewrote its unified CLI from Ruby to TypeScript running on Node after growing dependencies on Node tooling, Hydrogen’s preference for npm-managed dependencies, and a broader fit with web technologies. The rewrite aims to reduce system requirements, improve terminal consistency, and make contribution easier, while using a modular architecture of domain NPM packages built on @shopify/cli-kit. The team established shared code, UI, convention, and principle foundations, addressed cross-OS support, used TypeScript for stronger contracts, and applied Cucumber for end-to-end testing. The Node CLI unified and streamlined app development and added extensibility such as functions, but Theme commands still use Ruby and removing that dependency remains future work.
Reliving Your Happiest HTTP Interactions with Ruby’s VCR Gem
Ruby’s VCR library records HTTP interactions and replays them in test suites, verifying requests and returning predictable responses; the post presents it as a way to manage complex WebMock stubs and multi-step or unreliable API behavior. It contrasts recordings from a live API with manually authored stubs: cassettes preserve observed behavior at recording time but are harder to change and do not update when the API changes. The guidance covers VCR.use_cassette, configurable request matching, YAML cassettes, before_filter logging, filter_sensitive_data, and safer recording modes such as :once and :none instead of :new_episodes. It recommends VCR for integration-focused tests involving unpredictable sequences, multiple calls, or troublesome APIs, while favoring WebMock or decomposition for simple calls and specific states; cassette isolation and inspection add debugging value but can hide interactions.
React Native Skia: A Year in Review and a Look Ahead
React Native Skia’s first year produced an initial alpha and 89 additional releases, evolving a React Renderer and Skia integration for declarative 2D graphics across iOS, Android, the Web, and Node.js testing. The project changed direction after identifying two bottlenecks: dependence on the JavaScript thread and the overhead of thousands of JSI allocations in large drawings. Skia DOM addresses them with a declarative drawing state that recomputes only necessary changes and executes drawings on the UI thread, with C++ implementations on iOS and Android and a JavaScript implementation on React Native Web. CanvasKit compatibility, Expo integration, Metal on iOS, and OpenGL on Android and the Web support the cross-platform design, while extension points enable modules such as Skottie and camera-frame effects. The library’s tests compare rendered examples across platforms, and planned work includes JavaScript-worklet and Reanimated animation integration plus advanced text layouts through Skia’s paragraph module.
Flutter Authorization with RLS
This tutorial extends a Flutter real-time chat app by adding authorization and one-to-one conversations backed by Supabase. It introduces a rooms page for starting conversations and listing existing rooms, while flutter_bloc coordinates profile and message data across the rooms and chats pages. The database changes add rooms and room_participants tables, connect messages to rooms, expose create_new_room for reusing or creating a two-user room, and register deep links for email confirmation on iOS and Android. SQL RLS policies then restrict room, participant, and message reads to participants and require message inserts to use the caller's profile ID, producing a real-time chat app with authentication and authorization.
Supabase Beta October 2022
Supabase’s October 2022 update reports a broad set of product, documentation, and community releases, while announcing that Launch Week 6 will run from December 12–16. The new supabase-js v2 and supabase-flutter v1 SDKs reached full release, alongside a revamped Next.js quickstart with pre-built Auth UI and Auth Helpers and a Next.js 13 example for fetching and caching data from the Serverless API. Documentation covers Supabase Auth’s server-side rendering flow, and a database-testing framework uses pgTAP and pg_prove; Edge Functions also gained support for GET, PUT, PATCH, and DELETE requests. Additional changes include functions support in supabase-py, session refreshing, provider refresh tokens, request-location detection through X-Forwarded-For, expanded logging, and storage downloads via a query parameter, alongside three featured community apps and hiring announcements.
What is PostgreSQL commitfest and how to contribute
PostgreSQL Commitfest is the community's month-long review process for patches intended for merging into core. Contributors submit patches to pgsql-hackers, create a Git branch from master, consolidate changes, generate a versioned patch, and associate its mailing-list thread with an open Commitfest entry. The process expects authors to explain rationale, respond to review, rebase or revise patches, and review a comparable number of other patches. Submitted changes are checked by CFBot across architectures and operating systems, while statuses such as "Needs review," "Ready for committer," and "Waiting on author" track progress. Commitfests occur five times yearly, with the March event tied to feature-freeze for the October major release.
How to Write Code Without Having to Read It
The article argues that developers can minimize code reading when fixing bugs or changing features by focusing on information not already checked by tools. It recommends locating a user-visible entrypoint, then making a deliberately breaking edit and climbing the callstack through build errors rather than reading each caller. Strict types, tagged unions, exhaustive switches, linters, and unit tests expose incompatible inputs, missing cases, deprecated patterns, and affected consumers; pure functions separated from side-effect code keep tests simple and informative. After the tooling-driven passes, developers manually confirm behavior, revisiting the design when runtime issues remain and making future changes more "unread"-able.
Safely Adding NOT NULL Columns to Your Database Tables
Shopify’s Database Migrations team investigated whether Large Hadron Migrator (LHM) can safely add NOT NULL columns to existing MySQL tables with minimal downtime. LHM creates a shadow table, installs triggers for INSERT, UPDATE, and DELETE operations, and copies records in batches before renaming the tables. The experiments varied whether the new column had a DEFAULT value or UNIQUE index and whether MySQL used strict or non-strict mode, evaluating trigger compatibility and record counts. A NOT NULL column with a DEFAULT was backward compatible, but adding a UNIQUE index could cause data loss; without a DEFAULT, strict mode broke compatibility while non-strict mode used implicit defaults, which may be undesirable.
Shopify’s Technical Interview Process: What to Expect and How to Prepare
Shopify’s guide outlines the interview process for individual-contributor technical roles, while noting that stages can vary by role and team and that leadership hiring may add team-specific interviews. Candidates may have a one-hour recruiter “Life Story” conversation, a 40-minute CoderPad coding exercise, a 75–90-minute remote pair-programming session, and a 60-minute Technical Deep Dive focused on previous projects. The coding exercise assesses algorithmic thinking and problem solving, while pairing emphasizes communication, implementation, edge cases, scalability, limitations, and testability. The deep dive asks candidates to discuss impact, architecture, technical decisions, trade-offs, failures, successes, lessons learned, teamwork, and mentorship; the guide also recommends clarifying questions, running code early, using permitted references, and choosing projects they know well.
Partial data dumps using Postgres Row Level Security
Dumping full production databases onto local development machines creates severe security issues once applications hold real user data. To safely generate local seed data, developers can leverage PostgreSQL Row Level Security to restrict data access. The process involves provisioning a dedicated database user with select privileges on the target schema and tables. Next, administrators enable row-level security policies on those tables to filter records using criteria such as specific primary keys, email domain patterns, date intervals, or boolean flags. Finally, running pg_dump with the restricted user credentials and the --enable-row-security flag produces a sanitized seed.sql file containing only approved records.
Announcing Evaluation on the Hub
Hugging Face introduced Evaluation on the Hub, a no-code tool designed to streamline the evaluation of machine learning models across diverse datasets. Traditional evaluation approaches often suffer from reproducibility issues, implementation inconsistencies, and cumbersome workflows across varied metrics and datasets. Powered by AutoTrain and Hugging Face Spaces, the service allows users to configure tasks, map dataset columns, select evaluation metrics, and run tests directly from dataset pages. Completed evaluations automatically open pull requests on the respective model cards to encode standardized verification metadata and update public dataset leaderboards. This system establishes consistent benchmarking pipelines across tasks like image classification, text summarization, and named entity recognition without requiring local code execution.
Director of Machine Learning Insights [Part 3: Finance Edition]
Machine learning leaders in financial institutions face significant operational hurdles when integrating automated models into production environments. Practitioners must navigate complex legacy architectures, strict regulatory oversight, and privacy mandates tied to personally identifiable financial records. Integrating models often fails when organizations lack clear communication buffers, deploy models using improper prediction windows, or treat systems as opaque black boxes without verifying underlying mechanics. Despite these challenges, financial applications increasingly rely on machine learning for anti-money laundering compliance, automated fraud screening, and loan underwriting. Success in this regulated sector demands comprehensive evaluation across representative input spaces, robust ongoing model monitoring, and explainable decision outputs.
Building a Business System Integration and Automation Platform at Shopify
Shopify needed a maintainable way to connect increasingly modular business systems whose interfaces ranged from SOAP and FTP to REST and GraphQL. The company evaluated Mulesoft but found that its proprietary XML and Java virtual machine stack isolated the team from Shopify’s development ecosystem and complicated access to internal services and infrastructure. It instead built a Ruby on Rails integration platform around minimal validation, transparent data transformations, flexible client support, security, reusable components, documentation, and request-level testing with WebMock. The implementation favors direct manipulation of JSON hashes because that keeps mappings visible, while acknowledging risks such as typos and complex mock management. After a year, the platform had 18 integrations, had migrated all Mulesoft applications, and enabled other teams to build integrations, with ongoing work on testing and onboarding.
Faster Pull Request Merges
Ramp’s growing Python pytest suite took over 12 minutes, while pull requests could wait up to one to two hours between merge queueing and production. Previously, Kodiak kept pull requests in a merge queue, requiring each PR to be up-to-date with master and rerun the full suite after every merge. Removing that requirement exposed a failure mode: parallel branches could introduce Alembic migrations with multiple heads, making the database migration history invalid. Ramp instead added migrations/migration-hash.txt, a hash of migration filenames that creates merge conflicts between migration-changing PRs while leaving other PRs free to merge; the deploy pipeline still runs tests on master before deployment. This reduced average merge time from more than an hour to 12 minutes.
Test Budget: Time Constrained CI Feedback
Shopify’s core monolith contains more than 170,000 tests, and variable CI feedback times create delays and context switches as the test pool grows. The Test Infrastructure team evaluated a Test Budget approach that builds prioritized orders on top of deterministic test selection, then stops execution at a predetermined time to increase the chance of finding failures early while accepting some risk. Six criteria were compared—failure_rate, avg_duration, churn, coverage, complexity, and default/random—using Time to First Failure, APFD, and Convergence Index; historical results were stored and queried with Redis Sorted Sets. The analysis found failure_rate had the best detection rate, while churn and complexity were inappropriate; running 60% of the suite detected 80% of failures in the mean case, and 70% of the test-selection suite detected 50% at the fifth percentile.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.