---
title: "Case Study vs. Benchmark: Which Proof Transfers?"
description: "A case study sells a peer's outcome; a benchmark proves a repeatable delta. This side-by-side shows what each kind of evidence establishes, what it hides, and how to pick the right proof before you commit."
---

# Case Study vs. Benchmark: Which Proof Transfers?

**Type:** [Comparison](https://yomu.fyi/visual-explainers/category/comparison.md)

![Diagram explaining Case Study vs. Benchmark: Which Proof Transfers?.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/99fcddc0-9a3a-4d43-b38c-c084d0a91473/image-cea876be-3ddd-49a7-80af-ecde5c0c9a30.webp)

> A case study sells a peer's outcome; a benchmark proves a repeatable delta. This side-by-side shows what each kind of evidence establishes, what it hides, and how to pick the right proof before you commit.

## Explanation

Case studies persuade by narrative: a relatable protagonist, the tension before the change, and the resolution after it, which is why early-majority buyers trust a peer reference over a vendor claim. Benchmarks persuade differently, by fixing a workload, comparing a baseline against a candidate, and reporting a measured delta such as a 34x cut in cluster compute or a halved p99 time to first byte. The two answer different questions, so an engineer should use the narrative to decide whether a problem is real and worth attempting, then check the measured conditions—workload shape, tolerance, and cost—before assuming the result transfers to their own system.

## FAQ

### When should I trust a case study over a benchmark?

Start with the case study: a peer narrative tells you the problem is real, what the old workflow cost, and that someone comparable solved it. Then use a benchmark to check the delta under conditions close to yours. The story narrows your option set; the numbers decide whether it holds.

### What does a benchmark hide that a case study reveals?

A benchmark fixes one workload, so it hides whether your workload has the same shape, and it often omits the migration pain. The Spark declarative pipeline comparison cut cluster compute roughly 34x while raising token cost about 2.3x, a trade-off that shifts as data volume grows. Narrative surfaces that tension and the effort.

### How do I know a peer's result will transfer to my system?

Check the conditions the number was measured under: baseline, workload shape, accuracy tolerance, and hardware or software stack. TensorRT tuning results stay valid only until the model, input shapes, hardware, or stack change, so re-tuning is the stated best practice. Treat any case study number as a hypothesis to re-measure, not a guarantee.

**Tags:** [Benchmark](https://yomu.fyi/topic/benchmark.md), [Case Studies](https://yomu.fyi/topic/case-studies.md), [Decision-Making](https://yomu.fyi/topic/decision-making.md), [Evaluation](https://yomu.fyi/topic/evaluation.md), [Mental Models](https://yomu.fyi/topic/mental-models.md)

## Related reading

### [Built for Agents: The Case for Spark Declarative Pipelines](https://yomu.fyi/post/built-for-agents-the-case-for-spark-declarative-pipelines.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Lisa Cao, Sudhanva Huruli
- Published: Oct 23, 2026
- Type: Benchmark

AI agents now create over 70% of Spark data pipelines on Databricks, making cheap iteration and readable code critical. A benchmark comparing procedural PySpark with declarative pipelines tested 22 tasks across 12 datasets each, totaling 528 builds using Opus 4.8 for agent coding and an automated grader. Declarative pipelines enable dry-run validation that catches structural errors before execution, reducing cluster compute time by roughly 34x compared to procedural pipelines, which used 596 executor-seconds versus 17. Declarative code was about half the size (median 64 vs 131 lines) with less variability, while silent error rates were similar within about 15%. The declarative approach trades 2.3x higher LLM token costs ($0.50 vs $0.26 median) for eliminating wasted cluster time, a trade-off that improves as data volumes grow because compute waste scales while token costs are fixed.


### [Origin Range Requests: Serving large files without range-request tuning](https://yomu.fyi/post/origin-range-requests-serving-large-files-without-range-request-tuning.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Matthew Gumport
- Published: Oct 23, 2026
- Type: Announcement

Naive byte-range caching in CDNs can waste huge bandwidth when a small client request triggers a large origin fetch—for example, a seek request for a few megabytes could pull 128 MiB. Cloudflare’s previous implementation used fixed range sizes that created tradeoffs; the size was part of the cache key, so updates invalidated entries, and it could not combine with concurrent streaming, forcing downloads before serving. The new Origin Range Requests feature, a self-service option with no tuning, uses Pingora’s fixed-size chunks to align client ranges to chunk boundaries and fetch only the aligned chunk. A coordinator merges bytes from cache, in-flight downloads, and origin responses, checking entity tags and refusing to combine across representation changes. Enabling Origin Range Requests halved p99 time to first byte for R2, cut tiered cache traffic by roughly 6 percent, reducing petabyte-scale daily data movement.


### [Scaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 2](https://yomu.fyi/post/scaling-nvidia-tensorrt-performance-optimization-beyond-a-single-build-2.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Oct 23, 2026
- Type: Explainer

The Global Performance Tuner in TensorRT 11.1 provides automatic, accuracy-aware, end-to-end performance tuning, replacing reliance on a single default scheduling strategy with explicit build routes. Each build route is a concrete combination of compiler knobs controlling scheduling decisions; TensorRT 11.1 exposes over 200 such knobs. The tuning loop iterates over candidate routes, builds each engine, measures its accuracy and end-to-end performance against user-supplied references, and excludes any that fail to meet a tolerance threshold. This process decouples performance optimization from accuracy constraints, addressing the performance-accuracy entanglement problem described in Part 1. In evaluated workloads, tuning delivered 1.14x to 1.63x speedups over the default route, and best practices include re-tuning after any change to the model, input shapes, hardware, or software stack.


### [How Shipsidekick replaced Neon with ClickHouse Managed Postgres for 10x faster queries and better reliability](https://yomu.fyi/post/how-shipsidekick-replaced-neon-with-clickhouse-managed-postgres-for-10.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Oct 23, 2026
- Type: Problem & solution

Shipsidekick, a logistics platform, replaced Neon with ClickHouse Managed Postgres after struggling with frequent outages, pool exhaustion, and slow support. The migration took a single weekend, with ClickPipes replicating 500 GB of data in 4-5 hours. Since the move, CPU usage dropped from 100% to 5-10%, cache hit rate rose from 30-35% to 99%, and queries run about 10x faster. The team resolved a partial unique index conflict during replication by dropping and re-creating the index. They also implemented timeouts on a read-only role to prevent AI agent queries from impacting primary performance.


### [Using case studies to tell your story for you - The Intercom Blog](https://yomu.fyi/post/using-case-studies-to-tell-your-story-for-you-the-intercom-blog.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Published: Oct 8, 2026
- Type: Explainer

Case studies help B2B SaaS companies persuade the majority of potential customers who are not technology enthusiasts and are averse to revolutionary change by de-risking the purchase decision. Effective case studies follow a proven story formula with a relatable protagonist from an aspirational company, strong tension depicting pre-product challenges, and a resolution showing how the product replaced the old workflow. Geoffrey Moore’s Crossing the Chasm prescribes that the only suitable reference for an early majority customer is another early majority member, so you should profile non-innovator customers. Customer interviews should spend most time on tension and resolution, using questions about how they started using the product, what tools they used before, and what jobs the product solves, both qualitatively and quantitatively. Well-crafted case studies can be repurposed across sales, demand generation, events, and communications, and the process deepens the marketer’s empathy for customers.
