Orchestrate production RAG with OpenShift AI

Summary

This guide shows how to move a streaming retrieval-augmented generation (RAG) workflow from a monolithic Ray Data script to modular AI pipelines on OpenShift AI. It addresses failure recovery and run traceability by separating parsing and chunking, ingestion, model download, and deployment into five reusable components. Parsed and chunked JSONL is stored in S3-compatible storage so KFP can rerun ingestion without repeating document processing, and each run records parameters, timestamps, logs, and status. The data chain uses Docling’s HybridChunker and Ray map_batches with ActorPoolStrategy, while the model chain downloads a Hugging Face model and deploys it through vLLM and KServe in parallel. For embedding ingestion, the workflow supports local CPU processing with granite-embedding-125m-english or a dedicated GPU vLLM service for larger collections and offers notebooks for deployment and query validation.

Context

A single streaming RAG script requires rerunning the entire workflow after a late-stage failure, makes previous run parameters difficult to recover, and couples parsing, chunking, embedding, and Milvus writing. The guide addresses the need for reproducible, maintainable production workflows with independent retries, run history, and parallel execution.

Approach / What changed

The workflow uses Kubeflow Pipelines on OpenShift AI to coordinate five components: parse_and_chunk, ingest_to_milvus, deploy_embedding_model, download_model, and model_deployment. It writes intermediate JSONL output to S3-compatible storage, runs data and model chains in parallel, distributes Docling processing with Ray and KubeRay, stores vectors in Milvus, and serves models through vLLM and KServe.

Takeaways

• S3-compatible JSONL intermediates let the Milvus ingestion component be rerun independently after parsing has completed.
• Docling’s DocumentConverter runs in separate subprocesses within Ray actors, providing crash isolation for malformed PDFs.
• Embedding can use local CPU inference with granite-embedding-125m-english for smaller collections or a dedicated GPU vLLM service for larger ones.

Related reading