Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
Testing
271 posts about Testing. Every summary links to the original.
- 271 posts
- 42 companies
- 2 explainers
See the idea
Testing explainers
Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts
Large language models frequently exhibit human-like behaviors such as expressing emotions, building relationships, and setting boundaries, yet empirical guidance on appropriate usage remains limited. To address this gap, researchers analyzed 21,000 multi-turn conversations across four models: gpt-4o, gpt-4.1-mini, claude-sonnet-4.6, and gemini-2.5-flash. The study combined LLM-as-a-judge techniques with human evaluation to measure behavior prevalence, perceived appropriateness, and prompt controllability across diverse user goals and profiles. Human evaluators judged LLM self-referential and relationship-building behaviors as less appropriate than identical human actions, whereas boundary-maintaining behaviors were rated as more appropriate for models. System prompting effectively steered these behaviors, though the authors emphasize that rigorous evaluation remains necessary to prevent unintended side effects.
How AI Coding Agents Can Unlock Materials Simulation with NVIDIA ALCHEMI Toolkit
Atomistic simulation combines scientific judgment, compute-efficient implementation, and an accessible interface to the simulation stack, but the MLIP ecosystem lacks the mature interfaces available for classical force fields. NVIDIA ALCHEMI Toolkit addresses implementation with composable, PyTorch-native, GPU-accelerated building blocks and in-flight batching, while its agent skills and reference files supply API patterns to coding agents. The post benchmarks 45 generated pipelines across silicon equation of state, copper adsorption, and lithium self-diffusion, checking code features and running representative workloads on NVIDIA H200 GPUs. All 45 scripts used batched GPU execution, 38 completed screening, and all 15 production representatives completed; however, agents did not assess physical validity, and thermostat choices could suppress measured lithium diffusion by 3–5×, so independent validation against DFT, experiment, or another reference remains necessary.
17,600 Actions: Agent Security Is a Systems Problem
Security incidents involving AI agents, such as the OpenAI and Hugging Face event where attackers executed 17,600 actions, demonstrate that system security cannot rely entirely on manual human review. Managing agent velocity requires dedicated system-level controls to properly constrain, observe, and govern their behaviors during execution. To secure autonomous development workflows, Docker Sandboxes allow AI agents to operate within GitHub Actions, run Testcontainers tests, adjust code, and automatically generate draft pull requests inside isolated environments. In addition, escalating software supply-chain threats and the increased volume of AI-generated code drive the adoption of Docker Hardened Images and Docker Extended Lifecycle Support. These updates incorporate software built from source into container images, extend security patch coverage past end-of-life milestones, and move organizational policy enforcement directly onto developer machines.
Coding Agent Horror Stories: The Command You Already Approved
The OpenAI and Hugging Face security incident involved 17,600 attacker actions, demonstrating that AI agent security cannot rely exclusively on manual human review. Modern development workflows require systems controls to constrain, observe, and govern autonomous agents operating at high speeds. To address related isolation and automation requirements, developers can run AI agents inside GitHub Actions using Docker Sandboxes to execute Testcontainers tests, fix code, and generate draft pull requests. In addition, Docker Extended Lifecycle Support provides continued security coverage past software end of life while moving policy enforcement directly to developer workstations. Finally, Docker Hub has transitioned Docker Verified Publisher applications to a self-serve model, enabling publishers to distribute verified content directly to developers.
How to Evaluate Production AI Agents: Measure System Outcomes, Not Conversations
Evaluating AI agents solely on conversational transcripts fails when agents must invoke tools to modify external production systems. Language models can convincingly narrate that an action occurred while completely omitting the underlying tool call, leaving database or billing records unchanged. Outcome-based evaluation addresses this divergence by treating external system mutations rather than generated dialogue as the actual source of truth. Frameworks such as CRMAgentBench implement persistent, stateful environments with strict all-or-nothing validation, verifying tool arguments, execution sequence, final record modifications, and the total absence of unauthorized actions. Furthermore, measuring repeated success across independent executions using pass^k metrics exposes operational variance and inconsistency that standard single-shot scores and pass@k metrics conceal.
Reading the signals: Helping your AI-built payment integration understand business intent
AI agents can quickly produce Stripe payment flows that create a PaymentIntent, render the Payment Element, and pass an end-to-end demo, yet a correct implementation can still violate the business intent behind the money. The post argues that payment expertise and business decisions are often missing from integrations built from API references, causing silent failures such as terminal declines being retried, intents stuck in requires_action, uncaptured authorizations, restricted connected accounts, and asynchronous payments being refunded. It recommends defining business intent before configuring the PaymentIntent, then writing those rules into code comments, metadata, documented requirements, or custom objects. A verifier agent can check rules on every build, either through an agent skill, automated tests, or CI, while shared ownership remains with the human who decides what correct means.
What are AI Hallucinations?
AI hallucinations are coherent, confident outputs that are factually wrong, fabricated, or unsupported, appearing in chatbots, image generators, and multimodal systems. Generative models predict likely continuations rather than retrieve guaranteed facts, while training incentives, limited or flawed data, generation settings, and error compounding in reasoning models can increase the risk. Examples include Google Bard’s incorrect JWST claim, Air Canada’s nonexistent bereavement policy, Microsoft Sydney’s erratic responses, and fabricated legal citations that led to sanctions. Hallucinations create enterprise risks involving safety, liability, compliance, security, reputation, and trust; recommended controls include guardrails, source verification, adversarial testing, monitoring, human review, retrieval-augmented generation, governed data, and evaluation throughout development and deployment.
When Can LLMs Replace Humans in A/B Tests?
Replacing human participants in A/B testing with large language model predictions eliminates causal identification guarantees by design, forcing reliance on surrogate endpoint assumptions. An empirical evaluation using the Upworthy dataset revealed that raw gpt-4o-mini click-through rate predictions exhibited systematic attenuation bias, capturing only 39 percent of observed human treatment effects. To address this discrepancy, researchers formalized surrogacy and comparability conditions, finding that nonlinear machine learning calibrators like random forests and gradient-boosted trees successfully recovered treatment effects where linear ordinary least squares failed. In addition, averaging multiple sample draws per experimental unit mitigated variance and zero-bias caused by model generation temperature. Ultimately, because comparability cannot be verified for novel interventions that depart significantly from historical experiments, calibrated models cannot entirely replace human testing for genuine product innovations.
What We Learned by Reproducing 2,200 papers from ICML
An ICML 2026 reproduction hackathon examined whether coding agents could audit research at conference scale, amid 23,918 submissions and 6,352 accepted papers. From July 15 to August 2, 1,221 participants published 6,816 Trackio logbooks covering 2,226 papers and 35,908 claims; an open-weights GLM-5.2 judge assigned per-claim verdicts, while participants used cloud jobs and toy reproductions when full experiments were impossible. At least one claim was independently verified in 51% of examined papers, including 266 fully reproduced papers, while 23% had a falsified or contested claim and 242 papers produced opposing verdicts across teams. Adversarial re-checks confirmed issues including a paging-theorem error, a reverse-versus-forward-KL mismatch, and padded evaluation tokens, but also caught false falsifications, leading to the conclusion that agents expand auditing capacity while humans remain necessary to steer experiments and assess cases such as image quality.
How Agentforce-Powered AI Security Workflows Accelerate Incident Response
Salesforce engineers expanded Security Center from a basic conversational interface into an Agentforce-powered stateful investigations platform for incident response. Managing enterprise security incidents required handling long-running lifecycles, disparate telemetry formats, and finite LLM context windows that risk reasoning pocket collapse. To prevent overflowing context limits, the architecture partitions telemetry into contextual segments and uses AI-driven summarization pipelines before feeding data to the agent. Because non-deterministic LLM outputs break traditional deterministic testing, the team implemented AI-driven evaluation pipelines using simulated customer interactions to evaluate response behavior rather than exact wording. Ongoing work focuses on grounding public foundation models with curated, Salesforce-owned security knowledge to improve platform-specific remediation guidance.
How a major freight railroad scaled pipeline creation with Genie Code
A major Canadian freight railroad with roughly 20,000 route miles needed to modernize a decades-old analytical estate while handling hundreds of pipelines and rising demand for real-time analytics and AI. Using Databricks Genie Code, Unity Catalog, custom Agent Skills and a Streamlit Databricks App, the team created a metadata-grounded pipeline-generation workflow driven by compact YAML prompts. The workflow discovers schemas, maps source fields, and emits six artifacts—DDL, historical and streaming ingestion, incremental merges, and automated tests—using deterministic patterns for audit columns, deduplication, change-sequence guards and soft-delete reconciliation. Human-reviewed Source-to-Target Mapping preserves business interpretation, while Genie Code handles discovery, orchestration and artifact generation within governed Databricks execution. The reported result was more than 90% automation for new table ingestion, reducing delivery from days per table to minutes and supporting single, multi-table and bulk generation.
Welcoming Mateus Resende, our new Software Engineer
Resend introduces Mateus Resende as a Software Engineer based in Belo Horizonte, Brazil. He began working in software in 2015, first as a Ruby and Ruby on Rails developer before moving across domains including e-commerce recommendations, vehicle tracking, fintech, and email systems. A snowstorm during his exchange at SUNY Buffalo led him to discover MIT OpenCourseWare, code for hours, and later switch from Electrical Engineering to Computer Science. At Resend, he says its remote and async culture, extensive documentation, and developer-oriented product and company shaped his decision to join. His approach to engineering emphasizes balancing quality with speed and tailoring solutions to users and reviewers, while he recommends fundamentals such as testing, Domain-Driven Design, and Evolutionary Architectures over relying on hyped LLM coding techniques.
How we raised mobile end-to-end test stability to 98%
Shopify’s largest mobile app had become blocked by flaky end-to-end tests, which failed when screens took longer to render and had been removed from pull-request checks. Since 2023, tests used Appium through WebdriverIO and React Native Test IDs, allowing timing shortcuts and assertions about component-tree presence rather than what merchants could see or use. The rebuild hides Appium behind a strict builder-style wrapper that requires an assertion after every action; computer vision uses screenshots, PaddleOCR for text, and OpenCV for icons, with Test IDs as an opt-in fallback. After promotion into blocking CI, stability reached 98% of individual test successes over total runs, up from 50% with the old API, on two platforms. A pre-promotion flakiness gate rejects tests that exceed a failure threshold across repeated runs, while remaining failures largely involve network problems or simulator boot failures.
Grab Bench: Evaluating AI on Grab-shaped production work
Grab Bench is a configurable evaluation harness for assessing AI systems on production-shaped work where plausible outputs can violate metric definitions, tool contracts, evidence boundaries, or hidden code invariants. It uses YAML configuration, task plugins, provider and modality checks, row-level records, model summaries, and deterministic scorers or LLM judges according to each task. Synthetic or redacted cases preserve production-relevant difficulty while protecting live data; profile reasoning requires ontology-valid JSON claims and faithful evidence IDs, while coding tasks exercise hidden tests and anti-gaming checks. The harness also records tokens, latency, skip reasons, configurations, and failure tags so teams can diagnose whether errors reflect unsupported claims, parameter drift, hidden invariants, cost, or capability gaps. It concludes that synthetic evaluations test contract adherence under controlled pressure but do not establish production uplift.
amazon.science
A decade of mathematical certainty: Reflections on the Automated Reasoning Group
Launched in 2016, the AWS Automated Reasoning Group applies mathematical logic, formal verification, and satisfiability modulo theories solvers to prove the correctness and security of cloud systems at scale. Early research prototypes such as Tiros and Zelkova evolved into production tools like Reachability Analyzer, Amazon Inspector, S3 Block Public Access, and IAM Access Analyzer to evaluate network reachability and access policies. Internally, the group has verified foundational infrastructure, including cryptographic libraries, boot code, the Nitro Confidentiality Engine, and an authorization engine handling one billion API calls per second. Requiring precise formal specifications frequently uncovered simpler, more maintainable system designs while establishing guarantees that traditional scenario-based testing cannot provide. Recent initiatives integrate automated reasoning into generative AI workflows, powering Amazon Bedrock Guardrails, agent compliance policies, and requirement verification in Kiro.
How We Moved 1,500 Android Screenshot Tests to Roborazzi
Thumbtack migrated 1,500 Android screenshot tests from Firebase Test Lab to Roborazzi to eliminate slow and error-prone testing on remote emulators. The engineering team ported test suites from androidTest to test in batches of 20, adjusting test rules and annotations for Robolectric. Because Roborazzi runs in a single JVM process with parallel execution, the team had to debug shared font cache pollution and cross-platform shadow rendering inconsistencies between macOS and Debian. They resolved configuration repetition by mapping Robolectric annotations into module-level robolectric.properties files and avoided stale cached images by clearing Roborazzi's cache via a custom Gradle task. The migration significantly decreased CI runtimes while making local test recording and verification easier.
Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning
OfficeQA Pro V2 benchmarks whether AI agents generalize grounded reasoning to unfamiliar enterprise document collections rather than specialize in one corpus or task distribution. It contains 90 questions grounded in roughly 120,000 pages of U.S. Treasury accounting records spanning 1793–2024, including dense tables, charts, revised values, and changing reporting conventions. Questions were generated with asynth from seeded inputs, corpus evidence searches, traceable reasoning, and quality gates, then evaluated with deterministic exact match at 0.0% tolerance. Out-of-the-box frontier agents averaged 37.5% in one matched comparison, while competition systems averaged 41.1% and the winning team reached 63.3%; Databricks Genie raised matched-model accuracy by 24.0 percentage points on average, although failures remained in parsing, temporal reconciliation, and entity or category interpretation.
Databricks joins the Open Secure AI Alliance to advance AI safety and security
Databricks announces that it is a founding member of the Open Secure AI Alliance, formed with NVIDIA and more than 75 organizations to advance openly shared AI safety, AI security, and AI-enabled cyber defense research. The alliance’s scope extends beyond model weights to an open execution stack covering runtimes, guardrails, agent harnesses, risk frameworks, and governance. Databricks contributes Omnigent, an Apache 2.0 meta-harness supporting 13+ harnesses with contextual policies, spend caps, and sandbox isolation; DASF 3.0, which maps 97 risks across 13 components to 73 controls; DAGF; and BlackIce, a containerized toolkit bundling 14 red-teaming tools. Its Lakewatch Security Lakehouse applies the same openness to governed, agentic detection and response over portable data formats.
Are Supabase docs agent-friendly? We didn't know, so we checked
Supabase found that coding agents increasingly use its documentation to configure projects, creating risks when pages omit or bury critical instructions. To measure this, the team built an evaluation for the Row Level Security (RLS) guide, running an agent through realistic tasks and grading outcomes with checklist-based and agent-based judges. The initial runs found unauthenticated anon access with write privileges across tables and showed that test guidance was skipped when separated from the policy code. Supabase responded by adding procedural revocation steps, explicit policy snippets, and colocated test examples; the eval then passed the affected checks. The project concludes that agents and humans both benefit from selective-reading-aware structure, clear sections, explicit signposting, and recurring documentation evaluations, with the RLS scenario moving into the public supabase/evals benchmark.
Introducing Supabase Evals
Supabase has open sourced supabase/evals, a benchmark and framework for testing how AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks such as schema creation, Edge Function debugging, and RLS policy repair, scoring behavior with deterministic checks and LLM-as-a-judge. The framework executes scenarios in hosted-like and local containerized environments through the actual MCP server and CLI, with separate breadth-focused benchmark and depth-focused regression suites, plus a retry after failure. Initial findings show many agents pass broad tasks without skills, while skills improve some scores and documentation use; agents still often choose hand-written migrations, miss newer libraries, and use documentation inconsistently across harnesses.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.