Model vs. Harness: Where AI Agent Controls Belong
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
A case study sells a peer's outcome; a benchmark proves a repeatable delta. This side-by-side shows what each kind of evidence establishes, what it hides, and how to pick the right proof before you commit.
Case studies persuade by narrative: a relatable protagonist, the tension before the change, and the resolution after it, which is why early-majority buyers trust a peer reference over a vendor claim. Benchmarks persuade differently, by fixing a workload, comparing a baseline against a candidate, and reporting a measured delta such as a 34x cut in cluster compute or a halved p99 time to first byte. The two answer different questions, so an engineer should use the narrative to decide whether a problem is real and worth attempting, then check the measured conditions—workload shape, tolerance, and cost—before assuming the result transfers to their own system.
Start with the case study: a peer narrative tells you the problem is real, what the old workflow cost, and that someone comparable solved it. Then use a benchmark to check the delta under conditions close to yours. The story narrows your option set; the numbers decide whether it holds.
A benchmark fixes one workload, so it hides whether your workload has the same shape, and it often omits the migration pain. The Spark declarative pipeline comparison cut cluster compute roughly 34x while raising token cost about 2.3x, a trade-off that shifts as data volume grows. Narrative surfaces that tension and the effort.
Check the conditions the number was measured under: baseline, workload shape, accuracy tolerance, and hardware or software stack. TensorRT tuning results stay valid only until the model, input shapes, hardware, or stack change, so re-tuning is the stated best practice. Treat any case study number as a hypothesis to re-measure, not a guarantee.
The posts behind it
AI agents now create over 70% of Spark data pipelines on Databricks, making cheap iteration and readable code critical. A benchmark comparing procedural PySpark with declarative pipelines tested 22 tasks across 12 datasets each, totaling 528 builds using Opus 4.8 for agent coding and an automated grader. Declarative pipelines enable dry-run validation that catches structural errors before execution, reducing cluster compute time by roughly 34x compared to procedural pipelines, which used 596 executor-seconds versus 17. Declarative code was about half the size (median 64 vs 131 lines) with less variability, while silent error rates were similar within about 15%. The declarative approach trades 2.3x higher LLM token costs ($0.50 vs $0.26 median) for eliminating wasted cluster time, a trade-off that improves as data volumes grow because compute waste scales while token costs are fixed.
Naive byte-range caching in CDNs can waste huge bandwidth when a small client request triggers a large origin fetch—for example, a seek request for a few megabytes could pull 128 MiB. Cloudflare’s previous implementation used fixed range sizes that created tradeoffs; the size was part of the cache key, so updates invalidated entries, and it could not combine with concurrent streaming, forcing downloads before serving. The new Origin Range Requests feature, a self-service option with no tuning, uses Pingora’s fixed-size chunks to align client ranges to chunk boundaries and fetch only the aligned chunk. A coordinator merges bytes from cache, in-flight downloads, and origin responses, checking entity tags and refusing to combine across representation changes. Enabling Origin Range Requests halved p99 time to first byte for R2, cut tiered cache traffic by roughly 6 percent, reducing petabyte-scale daily data movement.
The Global Performance Tuner in TensorRT 11.1 provides automatic, accuracy-aware, end-to-end performance tuning, replacing reliance on a single default scheduling strategy with explicit build routes. Each build route is a concrete combination of compiler knobs controlling scheduling decisions; TensorRT 11.1 exposes over 200 such knobs. The tuning loop iterates over candidate routes, builds each engine, measures its accuracy and end-to-end performance against user-supplied references, and excludes any that fail to meet a tolerance threshold. This process decouples performance optimization from accuracy constraints, addressing the performance-accuracy entanglement problem described in Part 1. In evaluated workloads, tuning delivered 1.14x to 1.63x speedups over the default route, and best practices include re-tuning after any change to the model, input shapes, hardware, or software stack.
Shipsidekick, a logistics platform, replaced Neon with ClickHouse Managed Postgres after struggling with frequent outages, pool exhaustion, and slow support. The migration took a single weekend, with ClickPipes replicating 500 GB of data in 4-5 hours. Since the move, CPU usage dropped from 100% to 5-10%, cache hit rate rose from 30-35% to 99%, and queries run about 10x faster. The team resolved a partial unique index conflict during replication by dropping and re-creating the index. They also implemented timeouts on a read-only role to prevent AI agent queries from impacting primary performance.
Case studies help B2B SaaS companies persuade the majority of potential customers who are not technology enthusiasts and are averse to revolutionary change by de-risking the purchase decision. Effective case studies follow a proven story formula with a relatable protagonist from an aspirational company, strong tension depicting pre-product challenges, and a resolution showing how the product replaced the old workflow. Geoffrey Moore’s Crossing the Chasm prescribes that the only suitable reference for an early majority customer is another early majority member, so you should profile non-innovator customers. Customer interviews should spend most time on tension and resolution, using questions about how they started using the product, what tools they used before, and what jobs the product solves, both qualitatively and quantitatively. Well-crafted case studies can be repurposed across sales, demand generation, events, and communications, and the process deepens the marketer’s empathy for customers.
More like this
A comparison of what AI models should learn versus what agent infrastructure must enforce. See how reasoning, permissions, isolation, budgets, and observability split across the boundary.
Compare two places to keep Claude's working state: curated files on your own machine versus a durable session store or isolated cloud computer, and learn which one fits your deployment.
CIMD uses an HTTPS URL as an OAuth client identifier that points to a metadata document. This comparison shows when to pre-register clients manually and when to let dynamic discovery handle agent onboarding.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.