---
title: "Building an agentic harness that outlasts the model"
description: "Shopify’s Application Security team built an agentic harness that scans software for vulnerabilities, proves candidates with real tests, and opens Shopify-specific remediation pull requests. Its Dispatch orchestrator partitions repositories, runs category-specific Hunting agents in parallel, verifies findings sequentially with a different model, and persists architecture artifacts for later diff-based scans. Test oracles require realistic integration or functional coverage, including cross-tenant fixtures for IDOR checks, while unproven candidates are rejected or downgraded. Across more than 80 applications and thousands of scans, the workflow produced over 300 findings, including two rated Critical by Shopify’s severity calculator, with full scans costing $50–$300 and incremental scans $5–$50. The post concludes that durable value lies in the harness’s verification, partitioning, cross-model review, and deterministic infrastructure rather than in any single model."
---

# Building an agentic harness that outlasts the model

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Jul 29, 2026

**Type:** Explainer

## Summary

Shopify’s Application Security team built an agentic harness that scans software for vulnerabilities, proves candidates with real tests, and opens Shopify-specific remediation pull requests. Its Dispatch orchestrator partitions repositories, runs category-specific Hunting agents in parallel, verifies findings sequentially with a different model, and persists architecture artifacts for later diff-based scans. Test oracles require realistic integration or functional coverage, including cross-tenant fixtures for IDOR checks, while unproven candidates are rejected or downgraded. Across more than 80 applications and thousands of scans, the workflow produced over 300 findings, including two rated Critical by Shopify’s severity calculator, with full scans costing $50–$300 and incremental scans $5–$50. The post concludes that durable value lies in the harness’s verification, partitioning, cross-model review, and deterministic infrastructure rather than in any single model.

## Context

Shopify’s security-tuned frontier models found more candidate vulnerabilities than baseline models, but increased the volume of false positives and issues requiring confirmation. Manual, generalist multi-agent workflows also produced inconsistent results, theoretical findings, and frequently incorrect fixes when they lacked clear vulnerability categories and reliable validation.

## Approach / What changed

Dispatch orchestrates staged scans: test bootstrap, architecture documentation, file cataloging, partitioning, parallel category-specific hunting, sequential cross-model verification, post-processing, reporting, and per-finding remediation. It stores reusable artifacts in a Rails backend, uses diff-based scans for later commits, embeds validations in existing application tests, and uses deterministic scripts for structured operations such as JSON generation, credentials, Git, and storage.

## Takeaways

- Verifier agents author and execute tests in the target application; findings that cannot demonstrate exploitability under defined constraints are rejected or downgraded to Low or Medium.
- Partitioning files into coherent groups sized at roughly 20–30% of the model context window improved accuracy and recall while keeping scanning costs reasonably low in comparison tests.
- The workflow scanned more than 80 applications and generated over 300 findings across thousands of scans; full scans cost about $50–$300, while incremental diff scans cost roughly $5–$50.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/building-an-agentic-harness-that-outlasts-the-model)
- Source URL: https://shopify.engineering/building-an-agentic-harness-that-outlasts-the-model
- Ingested by Yomu: 2026-08-30T13:17:23.775Z

[Read original post](https://shopify.engineering/building-an-agentic-harness-that-outlasts-the-model)