Improved failure reports on Red Hat OpenShift with the event-driven diagnostic operator

Summary

During major infrastructure incidents, critical system logs and diagnostic evidence are frequently overwritten, lost, or erased by emergency recovery efforts before engineers can investigate. To solve this visibility gap in multi-cluster environments, an open-source event-driven diagnostic operator was developed as an automatic first information report collector. Deployed on a central hub cluster, the operator monitors Kubernetes warning events rather than relying on heavy polling. When an event matches a predefined regular expression pattern, the operator identifies the target spoke cluster, copies its kubeconfig secret, and spins up an independent must-gather job. Diagnostic data is written to off-site or shared storage with a configurable retention window, preserving forensic context without human intervention.

Context

During production incidents on OpenShift, critical logs and failure triggers are often overwritten, lost, or erased by emergency recovery actions, hindering root cause analysis and post-incident reviews.

Approach / What changed

An event-driven diagnostic operator running on a central hub cluster watches for Kubernetes Type=Warning events, matches event messages against regex rules, extracts target spoke cluster identities, copies their kubeconfig secrets, and executes independent must-gather jobs saving logs to external storage.

Takeaways

• The operator acts on Kubernetes Type=Warning events on the hub cluster rather than relying on continuous polling or manual triage.
• Diagnostic jobs mount ReadWriteMany persistent volumes to collect logs and rely on native TTL controllers for automatic cleanup after a retention period.
• Target spoke cluster names are extracted from events using a multi-strategy priority parser evaluating InvolvedObject.Kind, namespace prefixes, or regex patterns.

Related reading

• Red Hat · Aug 24, 2026

  Run LoRA fine-tuning on Red Hat OpenShift AI with Ray

  Red Hat OpenShift AI 3.5 integrates Ray cluster runtime images with preinstalled Training Hub packages to support distributed fine-tuning algorithms including LoRA, SFT, OSFT, and GRPO. Using the CodeFlare software development kit within a Jupyter workbench, developers can provision an elastic single-GPU Ray cluster attached to shared ReadWriteMany persistent storage containing model weights and training datasets. The tutorial demonstrates fine-tuning the Qwen2.5-1.5B-Instruct model on the public sql-create-context dataset to produce direct, bare SQL output without explanatory conversational text. After training completes, the workbench loads and merges the resulting LoRA adapter checkpoints with the base model to evaluate output format correctness against sample database schema questions. The RayCluster automatically tears down resources once the job concludes, while the shared pipeline architecture allows switching between supported fine-tuning algorithms by modifying dataset inputs and configuration parameters.

  Fiona Waters

• Red Hat · Aug 26, 2026