A modular regression testing framework built on AWS replays a sample of production traffic against new code and compares results to validate no behavior changes before deployment. It uses Step Functions for orchestration, Lambda for replay, DynamoDB for capturing requests and responses, and CloudWatch for logging and alarming on match fraction. The design is minimally invasive, requiring no changes to stateless systems and only a side-effect skip for stateful ones, and can be inserted at API, service, database, or message queue boundaries. In over six months of production use, the framework caught five regressions with zero false positives, using a 99.9% match fraction alarm to block problematic deployments via a deployment gate. The post covers three use cases—refactoring, feature toggle deployments, and impact boundary validation—and provides code examples for capture, replay, and evaluation functions.
Context
AWS blog post by an engineer detailing a modular regression testing framework using AWS services for large-scale systems.
Approach / What changed
The framework captures production requests and responses to DynamoDB, replays them via Lambda against new code in a pre-production environment, and compares results. It uses Step Functions to orchestrate start/stop, SSM to store simulation state, and CloudWatch alarms on match fraction to gate deployments.
Takeaways
1
The framework replays production traffic, not synthetic data, making regression detection accurate.
2
It is minimally invasive: stateless systems need no changes; stateful systems add a check to skip side effects.
3
In production, it caught five regressions with zero false positives over six months.
FAQ
How does the framework handle stateful systems?
Stateful systems add a check that recognizes replayed requests and skips side effects, preventing duplicate records or notifications.
What match fraction threshold is used to block deployments?
Teams commonly set alarms at 99.9% match fraction to block potentially problematic deployments.
What types of systems work best with this framework?
Computation-heavy, stateless systems like pricing calculators, validation engines, and data transformation pipelines work best.
A layered architecture view of fine-tuning a 3D generation model: which flow DiT branches get LoRA adapters, which stay frozen, and how data, compute, and evaluation wrap the adapted core.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.