Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
Testing
271 posts about Testing. Every summary links to the original.
- 271 posts
- 42 companies
- 2 explainers
See the idea
Testing explainers
Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Headlines from Pioneer 2025: Fin 3, the vision for a unified Customer Agent, and what’s next for customer experience
Pioneer 2025 presented Intercom’s latest customer-service AI plans, centered on Fin 3 and a longer-term vision for a single Customer Agent covering the entire customer experience. Fin 3 adds Procedures for complex, policy-controlled resolutions, Simulations for pre-release testing, Slack and Discord deployment, improved Voice, and Insights features such as CX Score Reasons, Topic Trends, and AI-powered Suggestions. The proposed Customer Agent would combine specialized roles, user-defined goals, persistent memory, business knowledge, and interoperability across tools, systems, and channels. Customer examples described phased rollouts, rapid feedback, broad scaling, and support-team involvement, while Anthropic reported two specialists enabling nearly 50 full-time employees’ worth of coverage in its volume. The event framed AI as a force for converging products, roles, and organizational structures, with the Customer Agent future described as arriving quickly.
Introducing Cursor 2.0 and Composer
Cursor 2.0 introduces Composer, described as the product’s first coding model, alongside an interface designed for running multiple agents in parallel. Composer is built for low-latency agentic coding, completes most turns in under 30 seconds, and was trained with tools including codebase-wide semantic search to improve work in large codebases. The new interface centers on agents and outcomes rather than files, while still allowing users to open files in a redesigned layout or return to the classic IDE; parallel runs are isolated through git worktrees or remote machines. The release also targets review and testing bottlenecks by simplifying change inspection and adding a native browser tool that lets Cursor test its work and iterate, and Cursor 2.0 is now available.
BigCodeArena: Judging code generations end to end with code executions
Assessing AI-generated code via static inspection is error-prone and fails to reflect whether code functions correctly across real-world environments. BigCodeArena addresses this evaluation bottleneck as a human-in-the-loop platform that executes model outputs in real time inside isolated sandbox environments across ten programming languages and eight execution frameworks. Users interact directly with live applications, refine implementations through multi-turn conversations, and submit pairwise preference votes to generate Elo ratings via the Bradley-Terry model. In five months of community testing covering over 14,000 conversations, proprietary models including o3-mini and o1-mini led overall Elo rankings, while open models like Qwen2.5 and Llama-3.3-70B trailed closed competitors. The project has open-sourced its full evaluation pipelines, conversation datasets, and related benchmarks to support community-driven code generation assessment.
Introducing RTEB: A New Standard for Retrieval Evaluation
Public embedding benchmarks often fail to measure true generalization because training sets overlap with public evaluation data, rewarding memorization over general search quality. The Retrieval Embedding Benchmark (RTEB) beta introduces a standardized evaluation framework designed for real-world enterprise retrieval applications. To prevent overfitting while preserving transparency, RTEB combines fully public datasets with private evaluation sets managed exclusively by MTEB maintainers. The suite spans 20 languages across domains like healthcare, law, finance, and code, measuring ranked retrieval quality using NDCG@10 on datasets of at least 1,000 documents. While currently limited to text-only retrieval and containing repurposed question-answering datasets, the benchmark provides clear signals when models degrade on unseen data.
Gaia2 and ARE: Empowering the community to study agents
Existing AI agent evaluation environments are often tightly coupled to specific tasks and fail to model real-world challenges such as API failures, spontaneous events, and asynchronous conditions. To address this limitation, the Gaia2 benchmark and the Meta Agents Research Environments (ARE) framework introduce interactive read-and-write evaluation. Gaia2 incorporates 1,000 human-created scenarios spanning multi-step execution, cross-source search, ambiguity handling, adaptability, temporal reasoning, agent collaboration, and noise tolerance. Using a simulated smartphone interface equipped with 101 tools, evaluations showed that GPT-5 with high reasoning scored highest overall, while Kimi K2 was the leading open-source model. The results demonstrated that instruction following and search do not reliably predict performance on closer-to-real-world tasks.
Democratizing AI Safety with RiskRubric.ai
Cloud Security Alliance and Noma Security introduced RiskRubric.ai to provide standardized, transparent risk assessments across the open AI model ecosystem. The framework evaluates AI models across six pillars—transparency, reliability, security, privacy, safety, and reputation—using over 1,000 reliability tests, 200 adversarial security probes, automated code scanning, and harmful content evaluations. Each model receives 0–100 scores and A–F letter grades, supplemented by specific vulnerability findings and recommended mitigation strategies to assist deployment filtering. Initial benchmark results across models showed composite scores ranging from 47 to 94 with a median of 81, revealing polarized safety distributions and indicating that security hardening directly correlates with reduced safety risks.
Migrating to React Native's New Architecture
Shopify successfully migrated two largest apps, Shopify Mobile and Shopify Point of Sale, to React Native’s New Architecture while continuing weekly releases for millions of merchants. The strategy prioritized minimal code changes, dual-architecture compatibility, and performance and stability parity, with TopHat generating both architecture builds for every pull request. The team upgraded React Native first, updated or removed incompatible dependencies, and changed only native modules that failed under the new architecture rather than migrating all 40-plus modules to TurboModules. Migration issues included state batching, UIManager-related blank screens, shadow-tree manipulation, view flattening, and main-thread legacy module initialization; the reported outcome was maintained development velocity with no feature-development disruption, followed by planned optimization work.
Building production-ready agentic systems: Lessons from Shopify Sidekick
Shopify’s Sidekick evolved from a simple tool-calling system into an agentic assistant that helps merchants manage stores through natural language, exposing scaling challenges in architecture, evaluation, and training. As its inventory grew beyond 50 tools, the system prompt accumulated conflicting special cases and became difficult to maintain; JIT instructions instead return relevant guidance with tool data. For evaluation, the team uses Ground Truth Sets based on production distributions, human labels from at least three product experts, statistical agreement measures, specialized LLM judges calibrated against humans, and merchant simulation. Judge correlation rose from Cohen’s Kappa 0.02 to 0.61 against a human baseline of 0.69, while syntax validation improved from about 93% to 99% after reward-hacking fixes. GRPO training exposed opt-out, tag, and schema hacking, and the post recommends modularity, procedural plus semantic validation, user simulation, and iterative judge refinement for production agentic systems.
Context engineering case studies: Etsy-specific question answering
The post examines whether prompt engineering can provide truthful Etsy-specific answers for AI-assisted onboarding without fine-tuning an LLM, using Travel & Entertainment (T&E) policy questions and an external Etsy community-forum use case. For the internal pilot, T&E documents were indexed, embedding-based search retrieved relevant sections, and those sections were added to each question; a manually curated 40-question evaluation found approximately 86% correct or satisfactory answers, with 14% wrong or misleading. Explicit uncertainty instructions sometimes avoided hallucinations but also missed available answers, while asking for an explanation or chain-of-thought reasoning could improve results, though complex reasoning required careful setup. The experiments conclude that prompt engineering is a practical alternative to parameter-based fine-tuning, but requesting specific source snippets remains important for flagging potential hallucinations.
The Future Isn't Model Agnostic
The post argues that AI product teams should stop treating model agnosticism as a default priority because users value reliable, familiar behavior more than model flexibility. It says improving models are converging, while differentiation shifts to product experiences shaped by prompts, edge-case handling, workflows, and model-specific behavior. Swapping models therefore means more than changing an endpoint: teams must rewrite prompts, rerun evaluations, and absorb changes users notice in how the product feels. Claude Code and Qwen 3 Coder, which the post says outperforms Claude in benchmarks, illustrate how tightly model behavior, interface, rituals, trust, and expectations can become coupled. The recommended approach is to choose one model, specialize deeply, and treat evaluation as architecture, with games such as an AI Town environment offered as a way to test models.
Building financial operations agents with Hypermode and Stripe
Hypermode Agents and Stripe can be combined to create domain-specific financial operations agents from natural-language descriptions rather than conventional code. The workflow uses Hypermode’s Concierge to generate a system prompt, then connects the agent to Stripe through the Stripe Model Context Protocol (MCP) server and optionally to services such as Notion, Slack, and Google Calendar. With a restricted Stripe API key and sandbox data, the agent can retrieve policies, inspect customer and transaction history, apply refund rules, process payments, create invoices, and analyze subscriptions for metrics such as MRR. The examples describe partial refund decisions, onboarding workflows, campaign suggestions, and populated test environments, while the guidance recommends prompt iteration, human approval for high-value transactions, and monitoring and logging for compliance and improvement.
How To Build Agents Users Can Trust
Ramp describes how it built and shipped LLM-backed agents for expense management, focusing on the trust risks created by low-quality or unexpected finance outputs. It recommends choosing ambiguous, high-volume problems with asymmetric upside, then combining reasoning and citations grounded in product or user-provided context with an explicit “Needs review” path when the agent is unsure. The policy agent uses expense policies to approve, reject, or escalate expenses, while deterministic rules such as dollar limits, vendor blocklists, and category restrictions constrain autonomy. Ramp reports that more than 65% of approvals are fully handled by the agent, while users can progressively move from suggestions to autonomous actions through workflow-based controls and evaluation.
Measuring Commercial Impact at Scale at Canva
Canva needed a scalable, standardized way to calculate the commercial impact of thousands of annual experiments on key business metrics like Monthly Active Users and Annual Recurring Revenue. Previously, disparate teams spent over six hours per experiment performing manual, error-prone calculations across fragmented spreadsheets and inconsistent data models. To resolve this, Canva built the IMPACT app using Snowflake, Streamlit, Snowpark, and Cortex to provide a self-serve platform tied directly to its central finance model. The application scales local uplift by actual audience exposure and supports pre-experiment scenario modeling alongside post-experiment tracking. A custom deployment workflow generates pull-request-isolated Streamlit environments in Snowflake stages, reducing time-to-insight to under ten minutes while enabling multiple developers to safely build and demo features in parallel.
Introducing Roast: Structured AI workflows made easy
Shopify’s Augmented Engineering Developer Experience team built and open-sourced Roast to make AI-assisted developer workflows more reliable than free-form agents operating across large codebases. Roast is a convention-oriented orchestration framework that combines non-deterministic AI behavior with ordinary code execution through YAML workflows, Markdown prompts, shell commands, Ruby steps, and parallel or conditional control flow. Its shared context, built-in tools, Claude Code integration through CodingAgent, and automatic session replay support workflows that are version-controlled, testable, resumable, and adaptive within defined guardrails. Shopify reports using it to analyze test quality at scale, add Sorbet annotations through the Boba workflow, aggregate competitive intelligence, and research code history, while the project is distributed as a Ruby gem and supports AI providers through Raix.
Sessions 2025 Developer Track resources
Stripe’s Sessions 2025 developer track recap presents new products, APIs, and enhancements for building and managing financial infrastructure. Stripe’s Workflows provides visual automation for real-time events, with more than 600 triggers, branching logic, dynamic fields, and built-in error handling, while the MCP server and agent toolkit connect AI tools and agents to Stripe documentation, account data, APIs, and financial capabilities. Accounts v2 unifies connected-account management, and fully localized Connect Mobile Embedded Components bring onboarding to iOS and Android; the Payment Element also gains support for the Checkout Sessions API. New Stripe Apps UI components, generally available Sandboxes, Organization-wide sharing of customers and payment methods, and conversational documentation extend customization, testing, and collaboration.
Building rock-solid Stripe integrations: A developer's guide to success
The guide presents ten practices for building production-ready Stripe integrations, covering payment forms, subscriptions, and operational concerns. It recommends verifying webhook signatures with the Stripe-Signature header and endpoint secret, recording event IDs for idempotency, and returning a fast 2xx response while processing asynchronously. Test and live API keys, webhook endpoints, logs, and monitoring should remain separate, while error handling should distinguish card, request, and API failures and use exponential backoff for retries. The guide also recommends testing successful, declined, insufficient-funds, and 3DS scenarios with Stripe test cards, alongside simulated webhook events. Its conclusion emphasizes edge-case planning, logging, monitoring, and ongoing review as foundations for reliable, secure payment processing that can scale with business needs.
New to Stripe? Learn the key concepts for software developers.
The guide introduces software developers to Stripe’s main objects and workflows for integrating payments, recurring billing, and related account activity. It begins with Payment Intents, which specify an amount and currency, track a payment’s lifecycle, and produce a client secret for completing the frontend flow, then covers Payment Methods and Customers, including Setup Intents and attaching a default payment method. For subscriptions, it distinguishes Products, Prices, and Subscriptions, and explains using events and webhooks to react to asynchronous payment and subscription changes while verifying webhook signatures. It also describes Disputes, refunds, and expanded API responses, including retrieving related resources in one call. The guide recommends starting in test mode and using dashboard logs while consulting Stripe’s documentation and changelog as the platform evolves.
Crush errors with Sandbox testing
Payment integrations can encounter errors caused by external events, such as network issues or outages, or by bugs in application code, and unhandled failures can stall checkout. The post presents Stripe Sandboxes as isolated environments for reproducing live-integration failures without changing live payment settings or traffic, optionally copying settings when a sandbox is created. Developers can use Workbench’s Errors and Logs views to diagnose a card_declined response, then route test requests through sandbox keys and a test card. On the Python backend, a try/except implementation catches Stripe errors and generic exceptions and returns an error response instead of leaving the payment page stalled. Sandboxes remain separate after creation, but each Stripe account can have at most five, so unused environments may be deleted or have their test data cleared while settings are preserved.
Testing Connect onboarding with Sandboxes
Stripe Connect testing can become unwieldy as platforms cover payment methods, US states, international countries, and merchant verification conditions, while the existing test mode stays synchronized with live settings. Sandboxes provide an isolated environment that can replicate a platform’s live configuration without affecting live payment traffic. The workflow covers creating a Sandbox, confirming that Connect settings such as negative-liability handling were preserved, and onboarding test connected accounts through Stripe-provided or custom flows. To reproduce a website-verification failure, a custom account can use “https://inaccessible.stripe.com”; after testing an update to “https://accessible.stripe.com” in the Sandbox, the platform can apply the fix to its production onboarding flow.
Managing multiple Stripe test environments from your AWS-hosted application
Stripe sandboxes provide isolated test accounts that let teams manage multiple environments from one account. Unlike legacy test mode, each sandbox has separate data, its own API keys, and configurable user access, while up to five sandboxes can exist per account. They support simulated external events, fake balances, Test Payouts with API v2 keys, and CLI or SDK access by changing keys. For AWS-hosted applications, the recommended pattern stores sandbox and production keys in AWS Secrets Manager, using secret names as environment-neutral aliases rather than embedding environment logic in code. Teams can combine Stripe sandbox permissions with AWS IAM resource policies and directory groups to restrict which users or accounts can retrieve keys and prevent accidental production access.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.