Run LoRA fine-tuning on Red Hat OpenShift AI with Ray

Summary

Red Hat OpenShift AI 3.5 integrates Ray cluster runtime images with preinstalled Training Hub packages to support distributed fine-tuning algorithms including LoRA, SFT, OSFT, and GRPO. Using the CodeFlare software development kit within a Jupyter workbench, developers can provision an elastic single-GPU Ray cluster attached to shared ReadWriteMany persistent storage containing model weights and training datasets. The tutorial demonstrates fine-tuning the Qwen2.5-1.5B-Instruct model on the public sql-create-context dataset to produce direct, bare SQL output without explanatory conversational text. After training completes, the workbench loads and merges the resulting LoRA adapter checkpoints with the base model to evaluate output format correctness against sample database schema questions. The RayCluster automatically tears down resources once the job concludes, while the shared pipeline architecture allows switching between supported fine-tuning algorithms by modifying dataset inputs and configuration parameters.

Context

Distributed fine-tuning of large language models across enterprise infrastructure often requires complex dependency management, manual pip installations, and custom cluster provisioning across multi-stage machine learning pipelines.

Approach / What changed

Red Hat OpenShift AI 3.5 incorporates Ray CUDA runtime images with preinstalled Training Hub libraries. Using the CodeFlare SDK from a Jupyter workbench connected to ReadWriteMany shared persistent storage, a Ray cluster is dynamically configured and launched to run a parameter-efficient LoRA fine-tuning job on Qwen2.5-1.5B-Instruct using the sql-create-context dataset.

Takeaways

• Training Hub comes preinstalled in Red Hat OpenShift AI 3.5 Ray CUDA runtime images, enabling fine-tuning without pip installations or external dependencies in air-gapped environments.
• Shared persistent volume claims require ReadWriteMany access mode so that the workbench control plane and Ray cluster pods can concurrently access model weights, datasets, and checkpoints.
• LoRA fine-tuning on the sql-create-context dataset trains Qwen2.5-1.5B-Instruct to output direct, executable SQL queries rather than wrapping answers in conversational preamble or Markdown formatting.

Related reading

• Red Hat · Aug 26, 2026

  GRPO fine-tuning on Red Hat OpenShift AI: Reinforcement learning from verifiable rewards with Training Hub

  Supervised fine-tuning for structured reasoning tasks like tool calling often demands expensive, labor-intensive dataset curation. Group Relative Policy Optimization (GRPO) resolves this by using programmatic reward functions to score multiple candidate responses without requiring human labels or a dedicated reward model. On Red Hat OpenShift AI, developers can implement GRPO using Training Hub, Kubeflow Trainer, and the Kubeflow SDK to fine-tune models like Qwen3-4B. The setup utilizes vLLM for candidate generation alongside LoRA weight updates, sharing storage via ReadWriteMany persistent volume claims. In a practical tool-calling test, five training iterations raised task accuracy from 33% to 67%, teaching the model to emit parseable function invocations rather than fabricated conversational text.

  Fiona Waters

• Red Hat · Aug 26, 2026

  Try the Ansible playbook generation lab with Gemini and OpenAI