# Improved failure reports on Red Hat OpenShift with the event-driven diagnostic operator

[Red Hat](https://yomu.fyi/company/red-hat) · Periyamaruthu Mohanraj · Aug 21, 2026

**Type:** Problem & solution

## Summary

During major infrastructure incidents, critical system logs and diagnostic evidence are frequently overwritten, lost, or erased by emergency recovery efforts before engineers can investigate. To solve this visibility gap in multi-cluster environments, an open-source event-driven diagnostic operator was developed as an automatic first information report collector. Deployed on a central hub cluster, the operator monitors Kubernetes warning events rather than relying on heavy polling. When an event matches a predefined regular expression pattern, the operator identifies the target spoke cluster, copies its kubeconfig secret, and spins up an independent must-gather job. Diagnostic data is written to off-site or shared storage with a configurable retention window, preserving forensic context without human intervention.

## Context

During production incidents on OpenShift, critical logs and failure triggers are often overwritten, lost, or erased by emergency recovery actions, hindering root cause analysis and post-incident reviews.

## Approach / What changed

An event-driven diagnostic operator running on a central hub cluster watches for Kubernetes Type=Warning events, matches event messages against regex rules, extracts target spoke cluster identities, copies their kubeconfig secrets, and executes independent must-gather jobs saving logs to external storage.

## Takeaways

- The operator acts on Kubernetes Type=Warning events on the hub cluster rather than relying on continuous polling or manual triage.
- Diagnostic jobs mount ReadWriteMany persistent volumes to collect logs and rely on native TTL controllers for automatic cleanup after a retention period.
- Target spoke cluster names are extracted from events using a multi-strategy priority parser evaluating InvolvedObject.Kind, namespace prefixes, or regex patterns.

**Tags:** [Incident Response](https://yomu.fyi/topic/incident-response), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Observability](https://yomu.fyi/topic/observability), [Open Source](https://yomu.fyi/topic/open-source), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://developers.redhat.com/articles/2026/08/21/improved-failure-reports-on-red-hat-openshift-with-the-event-driven-diagnostic-operator)