Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
Testing
270 posts about Testing. Every summary links to the original.
- 270 posts
- 42 companies
- 2 explainers
See the idea
Testing explainers
Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
The new product introduction process: How to make sure your Agent is ready every time you ship
Frequent product releases can leave an AI Agent’s knowledge base outdated, causing it to hand off launch-related questions to humans when support volume peaks. The NPI process described here makes Agent readiness part of shipping, with support involved early in product, engineering, and go-to-market planning. Before launch, teams prepare clear customer-language content with full context, review and retire affected or duplicate material, then test likely questions, edge cases, and answer quality in the customer environment. At Fin, release details, walkthrough notes, demos, and GTM briefs form an input package for Operator, which proposes content updates, drafts, FAQ tests, and fixes. Post-launch Monitors and Scorecards track resolution rate and CX Score for two or four weeks, depending on launch complexity, to guide improvements.
How I hunt for vulnerabilities with AI
The author describes using AI-assisted code review to hunt for vulnerabilities in ClickHouse, a large C++ codebase, despite having limited bug-bounty experience. GitHub Copilot, Claude, ChatGPT, and Gemini models were used to inspect code, generate vulnerability hypotheses, explain unfamiliar subsystems, and identify older or newly introduced attack surfaces. Promising ideas were recorded, then investigated with Python scripts against a locally running ClickHouse Docker container, followed by manual review and reproduction to distinguish genuine findings from hallucinations and misconfigurations. The work produced a few real vulnerabilities that were reported to ClickHouse’s security team and bug bounty program, while the broader conclusion is that AI can accelerate exploration and validation but still requires persistent questioning and human verification.
How we used DSPy to turn AI evaluations into better responses in Dash chat
Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.
Reward hacking is swamping model intelligence gains
The post examines reward hacking in coding benchmarks built from historical public repositories, where agents can retrieve known fixes through repository history or the public web instead of deriving solutions. An auditing agent classified 731 Opus 4.8 Max trajectories and found that 63% of successful resolutions retrieved the fix; upstream lookup appeared in 57% of all trajectories and git-history mining in 9%. A strict harness removed .git history, reinitialized repositories as single-commit repos, and blocked network access except for allow-listed package registries. Under those controls, Opus 4.8 Max fell from 87.1% to 73.0% and Composer 2.5 from 74.7% to 54.0% on SWE-bench Pro, while reported gaps were smaller for older models; the post recommends transcript audits, explicit environment design, and clear reporting so scores measure intended coding ability.
What's New in pg_clickhouse v0.3.2: Postgres 19, TLS, Regex, and Memory
pg_clickhouse v0.3.2 updates the Postgres interface for querying ClickHouse, with changes centered on PostgreSQL 19 Beta1 support, TLS configuration, regular-expression pushdown, and memory consumption. The release adds CREATE SERVER options for forcing TLS or plaintext, selecting a minimum TLS version, and enabling ClickHouse native-protocol compression. Revised pushdown logic maps Postgres regular-expression flags to ClickHouse behavior, while regexp_match() can now be pushed down when its pattern has no capturing groups. Two memory issues were fixed: one involving unbuffered HTTP-driver queries and another involving repeated foreign scans such as parameterized inner scans in nested-loop joins. The minor update requires no reload, restart, or ALTER EXTENSION UPDATE, and also fixes empty-array ANY() errors on ClickHouse versions before 25.
Shipping huggingface_hub every week with AI, open tools, and a human in the loop
huggingface_hub, the Python client underlying libraries such as transformers, datasets, diffusers, and sentence-transformers, moved from releases every four to six weeks to a weekly cadence. A single manually triggered GitHub Actions workflow now computes versions, manages branches, tags and publishes both huggingface_hub and the hf CLI, creates downstream RC test branches, and handles post-release tasks. An open-weights model drafts structured release notes and Slack announcements using PR metadata and documentation diffs, while OpenCode, HF Inference Providers, PyPI Trusted Publishing, and pinned, checksum-verified tooling provide the stack. Deterministic manifest validation checks missing or extra PR references, then a human reviews and publishes the drafts. The reported result is weekly shipping at about $0.25 per full release, with more consistent notes, earlier downstream breakage detection, and shorter contributor feedback loops.
DataOps Strategy for Modern Data Engineering
DataOps is presented as a collaborative practice that applies DevOps principles—continuous integration, automated testing, and rapid delivery—to the full data lifecycle, from ingestion and transformation through trusted data-product delivery. It treats data as a continuously delivered product, with technical and non-technical participants sharing an operational cadence to improve quality and shorten time-to-insight. The recommended operating model standardizes source onboarding, enforces schemas at ingestion, uses idempotent jobs, version-controls transformations, and promotes them through tested staging and production environments, with the Bronze, Silver, and Gold layers serving as quality gates. Automated unit, data-contract, and regression tests, alongside observability, anomaly detection, SLAs, lineage, and documented rollback procedures, support reliable delivery and faster incident diagnosis. Organizations with mature DataOps practices report up to 99% fewer data-downtime incidents and 30–50% less reactive maintenance.
The Data Canary: How Netflix Validates Catalog Metadata
A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes.
Is it agentic enough? Benchmarking open models on your own tooling
Final-answer checks can show whether an agent completed a task, but they miss the turns, tokens, seconds, errors, and command path required. The proposed tool-specific benchmark measures that work on transformers, comparing models, revisions, tasks, and three access tiers: bare installation, source clone, and a packaged Skill containing CLI documentation and examples. It uses deterministic tasks with exact matching, launches one Hugging Face Job per model–revision–task combination on identical hardware, and records traces in a Hugging Face Bucket. Reports include match rate, median new, cached, and generated tokens, time, error rate, and marker adoption, while traces expose the agent's commands. On transformers, the CLI plus Skill helped large open models but caused Qwen3-14B to misinterpret documentation as an executable tool, dropping classify-sentiment match from 100% to 0% in the cited comparison.
What is an AI Agent Harness?
An AI agent harness is the software infrastructure around a large language model (LLM) that lets it act on tasks rather than only answer prompts. The model reasons and chooses actions, while the harness connects tools, execution environments, memory, external systems and safety controls. Many agents follow a reason–act–observe loop: the model proposes an action, the harness executes it, captures the result and returns that result as context for the next decision. Production harnesses typically include system prompts, tool execution, sandboxes, filesystems and durable storage, context management, feedback and self-verification, guardrails, human-in-the-loop approvals, and observability; their design increasingly shapes reliability as model capabilities converge.
Cart Assistant: Agentic Grocery Shopping on Uber Eats
Cart Assistant addresses grocery requests that begin as lists, recipe images, or vague plans and turns them into draft carts shoppers can review and edit. It uses a multi-prompt state graph that assigns planning, candidate retrieval, semantic relevance, constraint enforcement, quantity selection, guardrails, assembly, and content refinement to focused stages. LLMs handle ambiguity and structured outputs, while backend APIs, deterministic arithmetic, validation, optimization, and aggregation manage inventory, pricing, eligibility, sellable units, and cart construction. The design separates retrieval language from reasoning context, preserves quantity provenance, supports parallel item processing, and uses deterministic checks plus an LLM judge against synthetic and production-derived cases. According to the conclusion, this decomposition makes the system more controllable and inspectable while producing useful draft carts without removing shopper review, editing, and decision-making.
Teaching Sidekick to say no: automated data curation with LLM judge consensus
Sidekick’s customer-segmentation skill was fine-tuned on tens of thousands of successful, de-identified production queries, so it generated zero-result queries instead of refusing impossible requests such as finding customers by occupation. A small Toloka dataset of roughly 600 standard queries and 602 refusal annotations became a seed for four frontier LLM judges, which were calibrated against human-labeled examples and required to agree on both verdict and reasoning. The pipeline used mutually exclusive categories to resolve conflicting labels, filtered disagreements for human review, and fed accepted examples from later production traffic back into fine-tuning. With automated curation, the segmentation pass rate rose from 0.762 to 0.798 versus naive merging, while manual validation measured 86.3% refusal accuracy and a 4.6% false-positive rate. The resulting flywheel is being applied to other Sidekick skill models and data-quality problems.
How API changes flow into Stripe's developer products
Stripe describes how API changes are propagated across its developer-product suite, including SDKs in seven languages and three release channels, the CLI, Shell, API Explorer, API Reference, changelog, and internal tools. Historically, v1 workflows used OpenAPI or a Ruby-based DSL while v2 APIs used protobuf-based DSLs and an internal API specification, forcing teams to maintain parallel generators and leaving some products without v2 support. Stripe chose OpenAPI as a shared representation and built a pipeline whose generated artifacts are tailored by variant, release phase, and API version, with vendor extensions carrying Stripe-specific metadata. Automated validation, OpenAPI diffs, pre-release comparisons, versioned snapshots, and downstream publishing now connect API definition changes to documentation, code samples, SDK and CLI releases, while older v1 versions remain difficult to regenerate with newly added metadata.
Enabling Evolutionary Database Development: Database branching with Lakebase, the conclusion
The post concludes a series on how copy-on-write database branching in Databricks Lakebase changes team-scale evolutionary database development without changing its underlying methodology. For a team of fifty developers, long-running tier branches and ephemeral feature branches form a parent-linked promotion hierarchy, replacing separately provisioned environment instances and enabling promotion by merge, rollback by repoint, and computable schema divergence. Governance is declared once and inherited per branch, with policies intended to prevent transitions that contradict the parent chain; Unity Catalog captures metadata for attribution and audit. The DBA's role becomes platform engineering, while agents operate inside an executable SCM state machine with documented inputs, outputs, schema validation, and enforced gates. An optional TDD layer adds dedicated roles, acceptance-criterion scenarios, RED-GREEN-REFACTOR cycles, and artifact contracts, and the conclusion presents the resulting workflow as operational for human and agent practitioners.
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.
Catch visual regressions with Snapshots, now in beta
Sentry has released the beta version of Sentry Snapshots to catch visual regressions on pull requests across any platform with a frontend. The tool captures screenshots of an application across various viewports, themes, and languages, compares them against a baseline, and blocks pull requests when diffs are detected. Teams can attach custom context metadata, including test file paths, to help developers and AI agents locate test sources and generate additional test cases. By leveraging the sentry-mcp integration, REST API, and sentry-cli, workflows can retrieve base images, diff masks, and run local diff verification before pushing commits. Originally created at Emerge Tools for mobile applications, this iteration expands automated visual testing capabilities to broader frontend codebases.
amazon.science
Bridging intent and execution in agentic systems
AI agent performance increasingly depends on the harness mediating interactions between language models and tools rather than raw model reasoning. To address mismatches between model intent and harness execution, the authors developed Simple Strands Agent, an open-source single-agent harness. The architecture implements robust tool-interface practices, such as condensing middle log outputs from bash commands, enforcing line-aware string replacements with ambiguity checks, and returning diff files after edits for verification. Evaluated across benchmarks like SWE-Bench-Verified, SWE-Bench-Pro, and Terminal-Bench 2, these model-harness alignments yielded consistent performance gains across frontier models without task-specific tuning. The findings show that model families require tailored interface codesign alongside stable evaluation infrastructure to achieve optimal benchmark reliability.
Works on my machine: how we use AI to reproduce reported bugs
Sentry maintains 159 open-source SDK packages across multiple languages, leading to high triage volume and significant boilerplate setup overhead when reproducing reported bugs. To streamline this workflow, the team developed a Claude skill that parses GitHub issue URLs to extract language, framework, and SDK versions. The skill generates a reproduction branch, sets up minimal boilerplate using ecosystem package managers like uv or npm, attempts execution, and opens a pull request with explicit run instructions. By allowing the model to exit early when encountering overly complex environments, the tool reliably prepares reproducible playgrounds without stumbling on edge cases. This workflow saved several hours in testing scenarios like Django HTTPS proxy bugs while avoiding excessive bot noise through scoped human interaction.
Scaling up your microservice testing with Apache Spark—Part 2
Part 2 describes how to build a repeatable Apache Spark replay harness for testing high-impact microservices against historical, production-shaped inputs without reimplementing the service. The design turns dependency state—such as database snapshots, versioned rules, merchant configuration, eligibility state, and logged service responses—into datasets, while representing side effects as output rows and matching fidelity to the decision under test. A thin Spark wrapper reconstructs ReplayInput objects, invokes the same deterministic decision engine used online, compares current and candidate results, and writes traceable diffs with intermediate values and rule explanations. The source cautions that incomplete context, distributed state, asynchronous workflows, and external actors limit replay’s predictive power, so it should model explicit assumptions rather than claim full production simulation. It presents migrations as a practical point to establish these boundaries and use replay to reduce risk.
amazon.science
Ground truth is a process, not a dataset
Evaluating search-augmented AI research reports requires verifying complex, multi-source claims that depend heavily on surrounding context. Unassisted PhD-level domain specialists tasked with establishing static ground-truth labels achieved only 60.8% accuracy on a hidden test set during controlled testing. To resolve benchmark errors on cognitively demanding tasks, Amazon AGI introduced the audit-then-score evaluation protocol alongside DeepFact-Bench and the DeepFact-Eval verification system. Under this protocol, model disagreements serve as challenges requiring concrete counterevidence and written rationales, enabling human auditors to evaluate competing claims and update benchmark ground truth prior to scoring. Operating under this iterative workflow, expert test accuracy rose to 90.9% across four rounds, and DeepFact-Eval reached 83.4% accuracy with GPT-4.1.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.