Improved failure reports on Red Hat OpenShift with the event-driven diagnostic operator

Summary

During major infrastructure incidents, critical system logs and diagnostic evidence are frequently overwritten, lost, or erased by emergency recovery efforts before engineers can investigate. To solve this visibility gap in multi-cluster environments, an open-source event-driven diagnostic operator was developed as an automatic first information report collector. Deployed on a central hub cluster, the operator monitors Kubernetes warning events rather than relying on heavy polling. When an event matches a predefined regular expression pattern, the operator identifies the target spoke cluster, copies its kubeconfig secret, and spins up an independent must-gather job. Diagnostic data is written to off-site or shared storage with a configurable retention window, preserving forensic context without human intervention.

Takeaways

1. 1

   The operator acts on Kubernetes Type=Warning events on the hub cluster rather than relying on continuous polling or manual triage.

2. 2

   Diagnostic jobs mount ReadWriteMany persistent volumes to collect logs and rely on native TTL controllers for automatic cleanup after a retention period.

3. 3

   Target spoke cluster names are extracted from events using a multi-strategy priority parser evaluating InvolvedObject.Kind, namespace prefixes, or regex patterns.