Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
Testing
270 posts about Testing. Every summary links to the original.
- 270 posts
- 42 companies
- 2 explainers
See the idea
Testing explainers
Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Towards Model-based Verification of a Key-Value Storage Engine
MongoDB developers formalized the interface boundary between their distributed transactions protocol and the underlying WiredTiger key-value storage engine using a compositional TLA+ specification. This modular model established an abstract contract to verify that the storage engine's concrete timestamp-based operations matched the semantics expected by the distributed transaction layer. To check conformance, a modified version of the TLC model checker generates a complete graph of reachable states for finite parameters and computes path coverings. Each identified path is translated into an automated test case consisting of a sequence of storage engine API calls. For a finite model with two keys and two transactions, this workflow generated 87,143 test cases that executed against WiredTiger in approximately 40 minutes.
2,000 robots walk into a shop: Simulated A/B testing (2026) - Shopify
Shopify’s SimGym addresses the difficulty of evaluating storefront changes when merchants lack enough traffic for conventional A/B tests, by simulating shopping sessions with LLM-guided robots. Each robot runs in a cloud Chromium browser with a persona, budget, and intent, while a counterpart tests the alternate theme; the system uses real page representations rather than a mocked DOM. To support bursty, sequential workloads with 89K–127K accumulated tokens per session, Shopify self-hosted gpt-oss-120b on 48 NVIDIA B200 GPUs, combining MXFP4 expert quantization, FP8 KV-cache, FlashInfer kernels, async scheduling, and guided decoding. Production measurements report 80K to 88K tokens/second per B200 from async scheduling, a 57% reduction in HTTP/gRPC overhead, and MIG experiments yielding about 20% lower latency and 12% higher daily throughput without quality regression; speculative decoding is validated but queued.
Closing the code review loop with Bugbot Autofix
Bugbot Autofix extends the code review agent so it can find issues in pull requests and automatically propose fixes. It spawns cloud agents in independent virtual machines that test software, and more than 35% of its changes are merged into the base PR. Now out of beta and available to all Bugbot users, the feature adds proposed fixes to the pull requests Bugbot reviews once enabled. Bugbot’s average issues identified per run has nearly doubled over six months, while the user resolution rate rose from 52% to 76%, indicating more bugs are caught and fewer false positives are flagged. Future work includes configurable automations, self-verification, deeper research, and continuous codebase scanning.
Announcing major updates to Procedures and Simulations: Enabling Fin to handle complex queries
Intercom announces 12 updates to Fin’s Procedures and Simulations, the mechanisms it presents for handling complex customer queries while keeping teams in control. Procedures combine natural-language instructions, deterministic controls such as conditional steps, data connectors, code snippets, checkpoints, and fully agentic behavior; new features include AI drafting, reusable Sub-procedures, explicit switching rules, targeted knowledge-hub references, internal handoff notes, automatic switching, and structured extraction from PDFs and images. The Procedures UI also adds improved authoring and reporting on triggers, resolutions, and handoffs. Simulations support full multi-turn conversations, image uploads, clearer visibility into Fin’s reasoning, and AI-generated tests stored in a central library that can be rerun after changes. Procedures and Simulations are available now, with the stated goal of helping teams train, test, deploy, analyze, and refine complex workflows before and after launch.
We proactively fixed ~100 security issues in 6 days with 0 humans
Ramp Security Engineering describes a six-day experiment in which home-grown agents found, validated, and fixed nearly 100 latent backend security issues, including some high-severity findings, without human involvement until pull-request review. The issues had not been uncovered by penetration testing, bug bounty testing, static analysis, or trials of 10+ code-scanning vendors, and all were patched within a week of discovery. The pipeline used specialized vulnerability detectors, adversarial manager agents that rejected 40% of initial proposals in sample testing, integration tests for validation, and an internal coding agent that generated patches. Live interactive validation struggled with complex preconditions, so test-based validation became the main approach; a human team member still reviewed and landed each pull request.
Minions: Stripe’s one-shot, end-to-end coding agents—Part 2
Stripe’s minions are an unattended agentic coding flow whose pull requests are human-reviewed but contain no human-written code; more than 1,300 merge weekly. The system runs agents in AWS EC2-based devboxes that are prewarmed to become usable within about 10 seconds, providing parallel, predictable, and isolated environments. Its custom harness uses code-defined blueprints that combine deterministic nodes, such as linting and pushing changes, with agent loops for tasks such as implementation and fixing CI failures. Minions receive automated feedback through local linting, one full CI iteration, and a possible second repair attempt, while devbox isolation and security controls limit destructive actions; Stripe presents the approach as an extension of long-running investments in human developer productivity.
Trusting the Untestable: Validation and Diagnostics for the Doubly Robust Models
When randomized A/B tests are infeasible for evaluating partnerships or long-term effects, Lyft applies Augmented Inverse Propensity Weighting (AIPW) doubly robust models to estimate treatment effects from observational data. The quasi-experimentation platform mandates rigorous confounder selection prior to treatment exposure and adjusts for downsampling bias via propensity score conversions and outcome reweighting. When comparing AIPW against experimental ground truths, initial estimates understated effects because trimming extreme propensity scores removed key users and unobserved confounders persisted. To evaluate model health and external validity, the platform incorporates diagnostic scorecards featuring common support checks, Marginal Sensitivity Models, and covariate comparisons between trimmed and untrimmed cohorts.
OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments
OpenEnv is an open-source framework from Meta and Hugging Face for evaluating AI agents against real systems rather than simulations, with stateful environments that connect to tools and APIs. Turing's production-grade Calendar Gym applies this model to calendar operations, combining access control lists, partial visibility, temporal reasoning, and multi-step workflows in isolated sessions. Agents often handled individual actions but became less reliable on longer or ambiguous tasks: success was close to 90% with explicit calendar identifiers and roughly 40% when requests used natural-language descriptions. More than half of failures involved malformed arguments or incorrect action ordering despite selecting the correct tool, while structured validation, permission, and datetime errors provided concrete recovery signals.
Minions: Stripe’s one-shot, end-to-end coding agents
Stripe’s Minions are homegrown, fully unattended coding agents designed to complete tasks in a single run, beginning with an invocation such as a Slack thread and ending with a pull request ready for human review. More than 1,000 pull requests merged weekly at Stripe are minion-produced and contain no human-written code, although engineers review them and can provide follow-up instructions. Each run uses an isolated, pre-warmed devbox and a customized fork of Block’s coding agent goose, combining agent loops with deterministic operations for version control, linting, and testing. Minions use curated MCP tools, including Stripe’s Toolshed with more than 400 tools, to retrieve internal context before and during work. Local checks and selective CI provide feedback, with no more than two CI rounds, balancing correction against token, compute, and time costs.
Community Evals: Because we're done trusting black-box leaderboards over the community
Hugging Face is introducing Community Evals, a system for decentralized and transparent reporting of model evaluation results on the Hub. It responds to saturated benchmarks and inconsistent scores across model cards, papers, and evaluation platforms, while acknowledging that benchmark performance does not necessarily reflect real-world capabilities. Benchmark dataset repositories can register as benchmarks, define reproducible eval specifications in eval.yaml, aggregate scores from model repositories and community submissions, and display leaderboards in dataset cards. Model results are stored in .eval_results/*.yaml, while users can submit sourced scores through pull requests that remain visible as community results, with Git history and APIs supporting tracking and further aggregation. The beta feature aims to expose what was evaluated, how, when, and by whom, but it does not claim to solve benchmark saturation, test-set training, or the benchmark-reality gap.
We Got Claude to Build CUDA Kernels and teach open models!
Domain-specific coding tasks like writing optimized CUDA kernels often require capable, expensive frontier models, leading to high operational costs. The upskill tool addresses this challenge by capturing specialized workflows from teacher models such as Claude Opus 4.5 into portable skill files that can be consumed by smaller or locally hosted models. The framework automatically generates input-output test cases from execution traces to benchmark student model accuracy and token efficiency with and without the skill. In an evaluation targeting HuggingFace kernel-builder development, a local GLM-4.7 Flash model improved its pass rate from 40% to 85% when supplied with the generated skill. Measuring token usage across different models remains essential, as skills can lower token consumption for some architectures while introducing unnecessary overhead for others.
Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs
Standard Arabic language model evaluations predominantly assess Modern Standard Arabic while overlooking regional spoken dialects that feature distinct syntax, vocabulary, and cultural grounding. To address this evaluation gap, researchers created Alyah, a benchmark comprising 1,173 manually curated multiple-choice questions collected from native Emirati speakers across seven cultural and linguistic categories. Distractor answer choices were synthetically generated using language models and manually reviewed for semantic plausibility. Evaluating base and instruction-tuned Arabic-native, adapted, and multilingual models revealed that instruction-tuned systems consistently outperformed base variants, with falcon-h1-arabic-7b-instruct achieving the top instruction accuracy at 82.18%. However, questions involving implicit meanings, rare expressions, and specific dialect phenomena remained difficult across all tested model families and sizes.
AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality
AssetOpsBench is an evaluation framework designed to benchmark agentic AI systems within industrial Asset Lifecycle Management operations, such as chillers and air handling units. The benchmark incorporates 2.3 million sensor telemetry points, 4,200 work orders, 53 structured failure modes, and over 140 curated scenarios spanning four agent roles. Systems are assessed across six qualitative criteria, including task completion, retrieval accuracy, result verification, sequence correctness, clarity and justification, and hallucination rate. Evaluation traces are analyzed using a trajectory-level diagnostic pipeline called TrajFM, which extracts and clusters failure patterns. In community tests covering over 300 agents, no evaluated model family reached the 85-point readiness threshold.
Best practices for coding with agents
Cursor's guide presents practices for using coding agents that can run for hours, complete multi-file refactors, and iterate until tests pass. Planning first is emphasized: Plan Mode researches the codebase, asks clarifying questions, creates a reviewable implementation plan, and waits for approval. The guidance recommends letting the agent discover context, starting new conversations when focus declines, and using Rules for persistent project instructions while Skills provide dynamic workflows. It also describes hooks for long-running loops, worktrees for isolated changes, parallel model runs, cloud agents, and Debug Mode's hypothesis-and-instrumentation process. Careful diff review, specific prompts, and verifiable goals remain necessary because generated code can be subtly wrong.
Demystifying user journeys: Revolutionizing troubleshooting with auto tracking
Troubleshooting critical issues in the Grab app was hindered by incomplete user-journey data, because manually instrumenting hundreds of UI components left interaction gaps and delayed diagnosis. AutoTrack is a new SDK that records application state, user interactions, and UI screens in real time, creating an end-to-end session chronicle from launch through navigation. It tracks lifecycle transitions and launch mechanisms such as deeplinks and push notifications, while UIViewController, Activity, and Fragment lifecycle events identify the displayed screen. Screen metadata includes the full hierarchy as JSON; after personal information is redacted for GDPR/PDPA compliance, that data is used to train an AI model that generates an HTML representation of the screen and interactions. AutoTrack supports reproducing elusive bugs, automated issue assignment, UI test generation, and analytics-trigger validation, with future work adding network and performance signals, privacy guardrails, and automated triage.
The Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator
Assessing whether large language model improvements stem from genuine advances or underspecified evaluation conditions remains a major challenge across the industry. Most published model evaluations omit critical execution parameters, prompt templates, harness versions, and runtime configurations. In response, NVIDIA released Nemotron 3 Nano 30B A3B alongside its complete, reproducible evaluation recipe built with the open-source NeMo Evaluator library. The library serves as an orchestration layer that unifies diverse benchmark harnesses under standard configurations while decoupling evaluation logic from underlying inference backends. Developers can execute the identical evaluation pipeline against local deployments or hosted endpoints using published YAML configurations and structured logging.
Stablecoin payments for Stripe developers: zero crypto knowledge required
The guide explains how Stripe developers can add USDC stablecoin payments without managing private keys, gas estimation, network selection, or wallet connection flows. Stripe supports USDC on Ethereum, Solana, Polygon, and Base, settles transactions as fiat in the merchant’s Stripe balance, and charges a 1.5% fee; the described feature is currently limited to US businesses. Integration can use payment_method_types: ['card', 'crypto'], although Stripe recommends automatic_payment_methods for Dashboard-managed configuration, filtering, ordering, and testing. Stripe Checkout and Elements expose crypto without frontend changes, while confirmation, webhook handling, monitoring, and familiar failure processing remain aligned with existing payment flows. The guide also describes Sandbox testing without testnets or fake cryptocurrency and presents crypto payments as a way to offer global settlement and potentially lower processing costs.
Treating Data as Code at Two Sigma
Scaling research and trading platforms across thousands of data sources created bottlenecks at Two Sigma, where reliance on database snapshots and fragmented infrastructure slowed dataset delivery to data scientists. To address operational costs and architectural complexity, the data engineering team adopted software development principles by treating data as code. The organization migrated to Google BigQuery's serverless architecture, standardized SQL transformations using dbt, and defined declarative pipelines with Terraform under continuous integration workflows. Internal tooling was introduced to track directed acyclic graph dependencies, automate anomaly detection, and streamline data discovery. These shifts eliminated manual data movement routines, reduced operational overhead, and enabled formal data contracts to safeguard downstream consumers and emerging language model integrations.
Building world-class product search at Shopify: Where C++ excellence meets ML innovation
Shopify describes a product-search stack built for multi-tenant commerce, where real-time indexing, retrieval, ranking, experimentation, and merchant controls must operate at global scale. Commerce ranking combines classical information retrieval with transformers, neural rankers, LightGBM, Catboost, and signals such as relevance, purchase popularity, brand trust, and navigational intent, while supporting privacy, multi-language, and accessibility requirements. RankFlow, a Python-like domain-specific language, lets data scientists change ranking logic without C++ expertise, while TurboDSL compiles and executes it with type safety, vectorized operations, cache optimization, predictable latency, and efficient memory use. The team first shipped SimScorerDSL in C++, then built TurboDSL without changing the DSL syntax, enabling continued experimentation during the transition. TurboDSL achieved a 48% speedup in ranking feature computation, and automated pull-request analysis compares component timings, statistical significance within a ±2% tolerance, and side-by-side results to detect performance or accuracy regressions.
Headlines from Pioneer 2025: Fin 3, the vision for a unified Customer Agent, and what’s next for customer experience
Pioneer 2025 presented Intercom’s latest customer-service AI plans, centered on Fin 3 and a longer-term vision for a single Customer Agent covering the entire customer experience. Fin 3 adds Procedures for complex, policy-controlled resolutions, Simulations for pre-release testing, Slack and Discord deployment, improved Voice, and Insights features such as CX Score Reasons, Topic Trends, and AI-powered Suggestions. The proposed Customer Agent would combine specialized roles, user-defined goals, persistent memory, business knowledge, and interoperability across tools, systems, and channels. Customer examples described phased rollouts, rapid feedback, broad scaling, and support-team involvement, while Anthropic reported two specialists enabling nearly 50 full-time employees’ worth of coverage in its volume. The event framed AI as a force for converging products, roles, and organizational structures, with the Customer Agent future described as arriving quickly.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.