Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
Testing
271 posts about Testing. Every summary links to the original.
- 271 posts
- 42 companies
- 2 explainers
See the idea
Testing explainers
Assist or Delegate? Choosing an AI Development Tool
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding
Resolve rate alone can hide how coding agents reach the same outcome. This evaluation compares Claude Opus 4.7, Gemini 3.5 Flash, GPT-5.5, and Qwen 3.6 27B FP8 through a Junie pipeline across four benchmark datasets, combining test results with trajectory, cost, efficiency, patch quality, and process measures. In one private benchmark, Opus and Gemini solved the same number of tasks, but Opus averaged 184 steps at USD 2.79 per run versus Gemini's 271 steps at USD 1.24. Across 523 tasks, Opus resolved 267 (51.1%) and Gemini 254 (48.6%); shared outcomes masked differences in navigation, validation, redundancy, hallucination, and patch scope. The wider comparison found GPT-5.5 had the highest resolve rate and always ran an executable check, Opus led patch-quality metrics, and Qwen delivered lower cost with lower resolution, while the authors caution that profiles depend on the Junie scaffold and LLM judges are diagnostic rather than ground truth.
The State of Django 2026: Boring is so back
The fifth annual Django Developers Survey, produced by Django Software Foundation and PyCharm, draws on responses from nearly 3,500 developers in more than 40 countries. It reports that Django’s core remains stable, with PostgreSQL used by 76–79% of respondents across five years, templates around 80%, and 43% already on Django 6.0. At the same time, AI is routine but fragmented: 58% of AI users work with it daily, while 56% use it only for chat and advice and just 27% use autonomous multi-step tasks. The survey also describes rapid consolidation around uv and Ruff, growing adoption of type hints, and a testing and CI/CD base that can verify agent-generated changes, although 19% write no automated tests. Django supports both server-rendered and API/frontend patterns, while its stable foundation lets developers change surrounding tools and workflows without replacing the framework.
Agent Seer: Synthesizing Scenarios from Specification Understanding
Agent Seer addresses the problem of creating realistic, scalable test scenarios for AI agents that use external tools, where hand-built cases require domain expertise and static benchmarks cannot track changing APIs. It synthesizes scenarios from a single Model Context Protocol (MCP) specification containing function names, natural-language descriptions, and typed parameter schemas, without examples, live tool access, or domain-specific tuning. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded, multi-turn dialogues, then evaluates tool-calling correctness and conversational coherence across seven MCP specifications spanning domains and tool-suite sizes. It achieves strong quality across all domains and complete tool coverage on small and medium specifications; parameter schema complexity correlates most strongly with quality variation, while tool-suite size has a smaller orthogonal role. Among imperfect scenarios, argument value accuracy is the dominant failure mode, which coarse-grained name-match metrics do not reveal.
The CLion Roadmap: What’s Coming Between Now and Late 2026
The CLion development roadmap outlines planned updates for minor releases and the stable 2026.3 release. Development priorities center on agentic workflows, embedded development support, and debugger enhancements. Planned AI features include agent-assisted project setup and a specialized skill for ARM Cortex-M HardFault debugging with register, memory, and disassembly access. Debugger enhancements introduce JSON-based debug profiles, import capabilities from tools such as VS Code, a dedicated Lauterbach TRACE32 profile, and LLDB 21 for Windows MSVC users. Additional additions span QNX project support, default resolve configurations for unlinked files, ISPC language support, bundled toolchain upgrades, and improvements to IDE responsiveness and Dev Container workflows.
How to Train a Cross-Embodiment Robot Navigation Policy with AI Agents
The tutorial presents an agent-driven workflow for adapting robot navigation across embodiments and scenes, addressing the repeated cost of collecting data, preparing simulation assets, training, diagnosing, and evaluating each robot-scene pair. Using Boston Dynamics Spot, it applies COMPASS to a built-in warehouse and a SAGE-10K scene, with NVIDIA Omniverse NuRec described as an option for captured environments. COMPASS reuses the pretrained NVIDIA X-Mobility navigation policy and trains a residual reinforcement-learning specialist that corrects base actions for a selected robot and environment; specialists can later be distilled into a shared policy. Repository skills guide dependency checks, asset preparation, smoke testing, training, checkpoint comparison, and packaging, while human gates control scene acceptance, smoke-test continuation, and checkpoint promotion. Runtime integration converts camera images, goals or routes, and speed from odometry into velocity commands, with cuVSLAM available separately when compatible odometry and transforms are unavailable.
amazon.science
When LLM judges agree, should we believe them?
Multijudge panels in LLM-as-a-judge evaluations often suffer from correlated errors when models share prompt templates, training lineages, or evaluation rubrics. To account for these interdependencies without human reference labels, a method presented at ICML uses Ising models to simultaneously infer judge reliability and pairwise correlation. The unsupervised algorithm iteratively updates the soft probability of each item's true latent label while estimating both individual judge skill and pairwise dependence structures. Evaluated across retrieval relevance, toxicity classification, and summarization benchmarks with ten zero-temperature judges, the approach achieved test accuracies of 0.912, 0.792, and 0.806. These dependence-aware aggregation scores consistently outperformed weighted majority voting baselines by 9% to 14% on standard metrics across all three tasks.
ClickGap: Autonomous QA for ClickHouse
ClickGap addresses defects that survive ClickHouse’s extensive CI, human review, and in-house automated review, including untested behavior, silent resource waste, and performance regressions. The autonomous QA daemon examines eligible merged pull requests against verified builds, uses separate bug and coverage analysts, and requires hypotheses to become executed tests. An adversarial reviewer and ten evidence gates check consequences, source citations, complete-file review, reproducibility, and other conditions before a report can be filed. Five months after launch, ClickGap had filed roughly 500 issues and opened around 200 coverage or fix pull requests; more than half of the issues were closed, with over 200 linked to a fix PR. The system’s central design goal is to make reports trustworthy rather than merely prolific.
Your alt text passes automated checks. That doesn’t mean it’s any good.
Automated accessibility checkers reliably flag missing alt text attributes but frequently miss unhelpful descriptions like raw filenames or repetitive labels. To evaluate image description quality without generating excessive false positives, GitHub built an alt text plugin for the GitHub Accessibility Scanner. The tool combines five deterministic, zero-credential rules that evaluate strings and visual layout spacing with an opt-in vision model check for subjective context. Page context including headings and surrounding prose is extracted alongside images to guide the model using structured outputs and explicit anti-nitpick instructions. While deterministic checks catch unwritten text, the model-driven rule serves as an opt-in prompt for human review rather than an absolute verdict.
Responsible AI adoption needs developer workflow design
Responsible AI adoption struggles when organizations rely on static policy documents instead of integrating guardrails directly into engineering workflows. Developers under delivery pressure inevitably turn to unapproved shadow AI when sanctioned routes introduce friction. Stack Overflow survey data reveals that 84% of respondents use or plan to use AI tools, yet more developers distrust AI accuracy than trust it. Operational controls must be embedded inside repositories, build pipelines, and pull request reviews to match specific failure modes like those listed by OWASP. Organizations achieve dependable adoption by pairing automated controls with clear role ownership, psychological safety, role-specific training, and outcome-based engineering metrics.
How to Evaluate Live & Voice Agents in ADK
ADK now supports native evaluation of live, voice-based agents, addressing regressions in tool use, context continuity, turn-taking, timing, and recovery that may not appear in demos. The walkthrough builds a graph-based workflow of three single-purpose agents running gemini-live-2.5-flash-native-audio, with a mocked date-of-birth tool between stages and session state carried across the open audio stream. It shows how to author scenario-based or fixed conversations, synthesize simulated user turns with Gemini TTS, configure live mode and rubric-based multi-turn trajectory judges, and run evaluations from the CLI or AgentEvaluator. ADK Web then reconstructs the audio as transcripts with playable clips, while configurable voices, languages, invocation limits, thresholds, and per-turn metrics support repeatable debugging and CI/CD regression checks.
amazon.science
SOP-Bench: A new benchmark for evaluating AI agents on real business procedures
AI agents often struggle to execute standard operating procedures because real-world business workflows contain implicit domain knowledge and procedural ambiguity that standard benchmarks overlook. To evaluate agent reliability, researchers introduced SOP-Bench, an open framework featuring more than 2,000 runnable tasks across 12 industrial domains paired with functioning tool interfaces and ground-truth answers. Domain experts authored the workflows, while an Anthropic Claude 3.5 Sonnet v2 model generated mock APIs, tool code, and edge-case datasets before expert verification. Experiments evaluating function-calling and reasoning agents across 11 frontier models revealed that newer models did not consistently outperform older versions, and providing 20 extraneous tools nearly halved task success. The benchmark, baseline agents, and evaluation code are released on GitHub and HuggingFace for reproducible testing.
Dispatches from O'Reilly: The right amount of spec for agentic development
As software agents make implementation faster and cheaper, engineering difficulty shifts away from typing code toward defining correctness and verifying behavior reliably. Omitting specifications forces human reviewers into expensive correction loops, while overspecification clutters model context windows with stale prose and competing instructions. To find the optimal middle ground, teams must validate specifications before coding, using one agent to draft minimal constraints and another to attack ambiguities, hidden dependencies, and missing failure modes. Multi-agent architectures demand formal interface contracts, schemas, and executable checks because interpretive drift compounds rapidly across unbounded agent handoffs. Core Extreme Programming and Agile practices such as continuous integration, small release batches, and rapid feedback loops remain essential error-detection mechanisms during agentic development.
Running AI agents in GitHub Actions with Docker Sandboxes
GitHub Agentic Workflows added support for Docker Sandboxes as an agent runtime, allowing CI coding agents to operate within microVM isolation. Useful AI agents require broad capabilities like executing arbitrary shell commands and launching containers, which elevates risk on shared CI runners. With the docker-sbx runtime, agents receive root privileges and a private Docker daemon inside a dedicated microVM while external repository access and network traffic remain restricted. In a demonstration, an agent ran on an Ubuntu runner, executed Java 21 integration tests with PostgreSQL via Testcontainers, identified an email case-sensitivity defect, applied a code fix, and opened a draft pull request. The complete workflow executed in eleven minutes and sixteen seconds while confining modifications strictly to source files.
Measuring benchmark optimization in speech recognition
Public voice AI benchmark scores can overstate speech-recognition ability when models learn benchmark-specific patterns rather than improve underlying transcription. Traditional evaluations may miss reliability, naturalness, contextual appropriateness, and other real-world conditions, motivating held-out sets in Real World VoiceEQ, the Open-ASR Leaderboard, and the Far-field ASR Leaderboard. The study introduces three tests and evaluates 11 widely used open-source ASR models, finding that several reproduce VoxPopuli and LibriSpeech reference transcripts when audio contradicts them, words are silenced, or two written forms sound identical. Models also sometimes use acoustic cues to identify dataset membership and select its expected transcription policy, while fresh post-cutoff data and other interventions often restore more audio-faithful output; the findings support fully held-out evaluation and broader metrics than a single public WER score.
Why AI-Generated Code Is Easy but Engineering Trust Is Hard
At Salesforce, engineering teams discovered that AI coding agents could produce internally consistent code and passing test suites that still failed to meet actual requirements. To ensure correctness before writing code, the team developed an agentic Spec-Driven Development workflow centered on explicit specifications and gated verification. The system separates evidence-based repository lookups handled by agents from subjective judgment calls escalated to human engineers. Implementation plans must cite repository evidence reviewed by a Skeptic Agent, while a Compliance Matrix traces each success criterion directly to executable test proof. Finally, a multi-agent review system evaluates the implementation under an asymmetric rule allowing an independent judge to downgrade passing checks but never override failed gates.
PyCharm for AI-assisted Django Workflows
According to the 2026 Django Developers Survey, ninety percent of respondents incorporate artificial intelligence into their weekly or daily workflows. While agents generate code rapidly, developers remain accountable for understanding, evaluating, and shipping changes within their applications. PyCharm addresses these demands by integrating support for external agents, local models via Ollama and LM Studio, and customizable agent skills across codebases. The IDE maintains version-specific framework assistance, including auto-completion for Django 6.0 template partials, alongside architectural inspection tools and visual diffs. Furthermore, developers can trace application components, inspect API endpoints, execute HTTP requests, and query database migrations directly through built-in tooling.
Developing NVIDIA Holoscan Applications with CLI, Skills, and AI Coding Agents
NVIDIA Holoscan is a platform for real-time AI applications at the edge, and HoloHub provides reference applications and components. This post describes an engineer-guided workflow in which a coding agent builds a separate Python endoscopic tool segmentation application from the existing MONAI model, sample video, and Holoscan patterns. The Holoscan CLI, invoked through ./holohub, exposes shared commands for scaffolding, running, modes, tests, and benchmarking, while development skills and documentation guide the agent through project-specific operations. The application connects video replay, preprocessing, TensorRT inference, segmentation postprocessing, telemetry, and HoloViz visualization, with visual, smoke, and benchmark modes; the post reports that the CLI-plus-skills-plus-docs/examples configuration produced a separate application in 40 minutes using 11M tokens, and concludes that this combination offered the lowest resource overhead among the compared configurations.
Post-Train NVIDIA Cosmos 3 Edge for On-Device Robot Control
Deploying large world models on robots is constrained by onboard memory and the control latency needed for continuous manipulation. The tutorial post-trains NVIDIA Cosmos 3 Edge, a 4B omni-model with a 2B NVIDIA Nemotron-based reasoner, into a DROID action policy using 76,000 successful teleoperated trajectories, then serves it through an OpenPI-compatible WebSocket server on Jetson Thor. The configuration predicts 32 future 8-dimensional joint-position actions at 15 Hz, while a receding-horizon loop executes a prefix and replans from fresh camera and robot-state observations. On an NVIDIA Jetson AGX Thor T5000, inference takes about 1.53 seconds per action chunk covering roughly 2.13 seconds of motion, and the policy reaches 22.9% success in closed-loop RoboLab evaluation; the source contrasts this with Cosmos 3 Nano’s 36.8%.
Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
NVIDIA SkillEvaluator measures whether packaged agent skills improve agent performance by comparing task runs with and without a skill installed. It combines three evaluation tiers: static safety and structure checks, embedding-based distinctiveness analysis, and live evaluations in isolated Harbor sandboxes. In the August 12, 2026 benchmark snapshot, more than 300 verified skills spanning over 30 NVIDIA products were tested across Claude Code and Codex, with macro-averaged Skill Lift of +31 points across all dimensions and +39 excluding Security. Correctness rose from 46 to 87 and Effectiveness from 39 to 78, while Security moved from 97 to 98; token and runtime effects varied by skill, with one reducing both and another increasing them.
Accelerate automation with AI and the Ansible development tools MCP servers
Ansible automation development often involves frequent context switching and struggles with AI assistants generating syntactically valid code that hallucinates module parameters or misses local dependencies. To address this gap, the Ansible development tools Model Context Protocol server connects AI assistants to external development tooling through structured interfaces. The server wraps command-line utilities, enabling AI models to deterministically run ansible-lint with automatic fixes, scaffold projects via ansible-creator, manage environments, and build execution environment container images. By surfacing local workspace schemas, documentation, and organizational linting standards, the system grounds AI responses in real environment configurations. The integration operates across multiple MCP-compatible clients without requiring a specific model provider.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.