Beyond pass or fail: The new era of chaos engineering

Summary

Binary pass/fail uptime testing no longer suffices for cloud-native deployments because a cluster can remain operational while dropping packets, breaching SLOs, or suffering performance degradation. Krkn addresses this with a trifecta of telemetry: a weighted Resiliency Score, continuous HTTP Health Checks, and KubeVirt Checks. The Resiliency Score is a 0–100% macro-health metric that evaluates live Prometheus data against pre-defined SLOs using custom weights, allowing engineers to prioritize business-critical components like authentication or payment gateways. HTTP Health Checks probe application endpoints at configurable intervals, capturing exact downtime start and end times alongside HTTP status codes, enabling teams to track self-healing duration. KubeVirt Checks monitor virtual machine instances via real-time SSH connectivity and record node, IP address changes, and exact down-time, with a final post-chaos check that flags VMs remaining unreachable after the experiment concludes.

Takeaways

1. 1

   The Resiliency Score uses a weighted scoring model where standard warning alerts are worth 1 point and critical outages are worth 3 points by default, but engineers can assign custom weights to individual SLOs to prioritize business-critical components.

2. 2

   HTTP Health Checks capture exact downtime start time, end time, and duration during chaos experiments, and support bearer tokens and credential tuples for monitoring protected API endpoints.

3. 3

   Running identical Krkn chaos scenarios against successive application versions in CI/CD provides quantifiable evidence of architectural regressions by comparing Resiliency Scores, HTTP endpoint failure durations, and VM outage durations.

FAQ