Many regression tests verify what developers intended a system to do, not what it actually does, leaving a critical gap. Hyrum’s Law states that once a system has enough users, every observable behavior becomes depended on, so unplanned quirks and accidental defaults are a common source of regressions. Characterization tests, coined by Michael Feathers, address this by capturing the current actual behavior of code as a test, regardless of whether that behavior is correct. They are rarely written because writing them is a slog—requiring testing many inputs manually—and because it feels wrong to assert behavior that appears buggy. A powerful method is to record production traffic at the network level, capturing requests, responses, and side effects like database calls, then replaying those recordings as a repeatable test suite that flags any behavior change, and helps during refactoring, framework upgrades, rewrites, and AI-assisted changes.
Context
Traditional regression tests only cover intended behavior, leaving accidental dependencies unprotected, which causes regressions when accidental behaviors change.
Approach / What changed
Write characterization tests that capture current actual behavior, often by using golden master/approval testing or recording production traffic at the network level for replay.
Takeaways
1
Characterization tests pin what code actually does, providing a safety net for accidental behavior changes that intent-based tests miss.
2
Recording real production traffic at the network level captures a service’s full behavior, including side effects, without modifying application code.
3
Reviewing characterization test diffs in pull requests turns behavior changes from accidents into deliberate decisions.
FAQ
How do characterization tests differ from ordinary regression tests?
Ordinary regression tests assert what the code should do; characterization tests assert what the code currently does, regardless of correctness, to flag any behavior change.
What is a practical way to create characterization tests for a service?
Record production traffic at the network level, capturing all requests, responses, and downstream calls, then replay that recording as a test suite to create a repeatable baseline.
How do you handle noise from timestamps and IDs in characterization tests?
Ignore specific fields like timestamps and generated IDs that change every run, while keeping the rest of the output fully compared.
Cekura generates simulated caller scenarios to test voice-agent prompts, so failures surface before real calls. The trade-off: fast, repeatable iteration covers only the failures you thought to script.
A five-stage walkthrough of how CBRN risk evaluations are commissioned, designed, graded, and converted into safety-level decisions for frontier AI models.
A career ladder turns vague seniority into a repeatable flow: define skill buckets, set level expectations, collect real evidence, calibrate across managers, then decide. Here is how each stage works.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.