Loading…
Topic
OpenShift AI
10 posts about OpenShift AI. Every summary links to the original.
- 10 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Build an AI matching engine with Docling, Granite & OpenShift AI
PeopleMesh uses Docling and Granite4:3B LLM on Red Hat OpenShift to extract structured data from PDF resumes and match people across multiple dimensions. Docling converts documents into a DoclingDocument with semantic labels using an AI model trained on 150,000 annotated samples to recover reading order from PDF’s print-oriented encoding, then exports to Markdown for LLM-driven JSON extraction via a system prompt. For matching, PeopleMesh decomposes a user query with an LLM to extract intent, entity type, country, and language, then applies structured filtering followed by semantic vector search via cosine similarity on distilled query embeddings. To mitigate embedding dilution from concatenated concepts, PeopleMesh uses a weighted scoring algorithm that reranks top 100 candidates using 7 information channels including nice-to-have skills and penalty multipliers for missing must-have and negative skills. The scoring provides relative ordering only, not normalized probabilities, and default weights are adjustable via the SearchOptions Java record.
Add NeMo Guardrails to a LangGraph agent on OpenShift AI
LangGraph agents can call tools and follow prompts but lack safety guardrails, risking unsafe responses like explaining fraud. NeMo Guardrails acts as a transparent proxy between the agent and LLM, inspecting every request and response without requiring agent code changes. On Red Hat OpenShift AI, the guardrails service is deployed via a NemoGuardrails custom resource managed by the TrustyAI Operator, and the agent’s BASE_URL points to this service instead of the model endpoint. The example demonstrates three rail types: regex filtering for jailbreak patterns, content safety classification against S1-S13 categories, and topic safety to keep the agent within a banking domain, with optional tracing using MLflow and OpenTelemetry. Rails execute sequentially, with the first failure short-circuiting the chain, and the separation lets safety policy be owned by guardrails while the agent handles reasoning and tools.
Distributed training on OpenShift AI 3.4 with Kubeflow Trainer v2
Red Hat OpenShift AI 3.4 introduces Kubeflow Trainer v2, a unified training API that replaces framework-specific CRDs like PyTorchJob and TFJob with a single TrainJob resource. Under the hood, each TrainJob is translated into a JobSet, a Kubernetes-native API for managing groups of Jobs as a single unit, which handles headless services for pod discovery and automatic failure recovery. The post validates the stack through two phases: manual validation with a minimal NCCL all-reduce test across 2 nodes, and real-world training of an embedding-based neural network on a fraud detection dataset using 1, 2, and 4 NVIDIA A10G GPUs. Benchmarking shows near-linear Distributed Data Parallel scaling at 95% efficiency, successful checkpoint recovery after pod kills, and GPU utilization confirmed via DCGM Exporter. The TrainJob API simplifies distributed training by automatically configuring torchrun, environment variables, and pod coordination, while raw JobSets offer more control for multi-stage pipelines and differentiated failure handling.
What did AI cost you this quarter?
AI infrastructure costs are often billed to a platform team as one lump sum, leaving Finance without a departmental breakdown and teams without feedback on their usage. The post presents a working demo on Red Hat OpenShift AI that attributes requests and token consumption through department-specific MaaS API keys, shared model servers, Prometheus metrics, Perses dashboards, and MLflow traces. In its example, three departments generate 79,420 inference requests and consume 634 million tokens against a $14,200 quarterly bill, while Marketing’s 32% GPU utilization reveals roughly $34,000 in annual waste. The system combines aggregate usage filters with request-level details such as prompts, token counts, latency, and calculated cost, and the post argues that showback helps identify inefficient allocations and model mismatches rather than police teams.
From token consumer to token provider: Building your org's AI API
Enterprise AI usage can become costly and fragmented when teams rely on separate provider accounts, personal keys, and independently operated inference stacks, leaving spending, workloads, and governance difficult to see. The post presents Model-as-a-Service (MaaS) in Red Hat OpenShift AI 3.4 as a shared service in which a platform team deploys approved models and exposes them through a standard, OpenAI-compatible API. Administrators can use the model catalog, distributed inference through llm-d on NVIDIA L40s, scoped API keys, rate limits, and dashboards for token use, latency, errors, GPU utilization, and queue length. Developers obtain self-service endpoints and can connect tools such as Claude Code, OpenCode, or Continue with existing clients. The architecture combines OpenShift, vLLM, an AI gateway, Prometheus, Grafana, and Jaeger to centralize inference, governance, and cost attribution across local and external model providers.
Deploy NeMo Guardrails on Red Hat OpenShift AI
This article shows how to deploy an evaluated NeMo Guardrails configuration on a Red Hat OpenShift AI cluster, continuing a series on local development and dataset-based evaluation. It requires OpenShift AI 3.5 or later and namespace write access, while OpenShift AI 3.2 or newer can be configured to use the 3.5 NeMo Guardrails image through TrustyAI service-operator changes. Deployment uses a ConfigMap for the configuration, a NemoGuardrails custom resource, and an authentication annotation that protects the server route; an empty rails.co file is sufficient when no custom algorithms are defined. The deployed server accepts v1/guardrail/checks requests with a bearer token, and the demonstrated classifier permits a benign prompt, blocks a prompt injection, and rejects an invalid token. The article notes that LLM proxying and LLM-based guardrailing are outside its scope.
Personalize your product's text-to-speech voice for any language: Fine-tuning with Kubeflow Trainer on Red Hat OpenShift AI
Training an English-centered text-to-speech model for Turkish can produce fluent but hard-to-understand pronunciation for native speakers. This tutorial fine-tunes unsloth/orpheus-3b-0.1-pretrained with about 81,000 Turkish text-audio pairs on Red Hat OpenShift AI using a Kubeflow Trainer TrainJob distributed across two nodes with two A100 GPUs each. LoRA freezes the 3-billion-parameter base and trains 97.3M adapter parameters, while preprocessing converts text to Llama-3 tokenizer tokens and audio to three-layer SNAC codes, masking text labels during scoring. The merged model is evaluated against the base on ten Turkish sentences using Whisper transcription and WER/CER metrics. Mean WER falls from 2.44 to 0.35 and CER from 1.59 to 0.09, though the authors describe the small test as directional.
Evaluate AI agents with IBM CLEAR & EvalHub on OpenShift AI
Reliable AI agents can fail in production through incorrect tool use, off-topic reasoning, or plausible but incorrect answers that unit tests, static benchmarks, and logs may not expose. This tutorial builds an eval-driven trace-evaluation pipeline on Red Hat OpenShift AI 3.4 or later, using MLflow to store structured agent traces and evaluation artifacts, IBM CLEAR to score reasoning quality, relevance, and tool usage with an LLM judge, and EvalHub to orchestrate OpenShift jobs. It uploads eight prerecorded multistep research-agent traces, each containing at least 30 spans, then verifies them in MLflow before deploying EvalHub, registering the ibm-clear provider, and configuring judge-model credentials. Standard mode evaluates steps holistically, while SPARC mode gives tool calls separate scoring rows; the resulting workflow produces critiques and ranked failure patterns with frequency, severity, and step or agent attribution.
LoRA backdoor threat: How OpenShift AI mitigates the risk
LoRA adapters can carry semantic backdoors even when the underlying open-weight model is clean, creating a supply-chain risk for customized AI assistants. The demonstrated proof of concept uses a 35 MB adapter trained from 270 ChatML examples, with 130 poisoning examples and 140 clean examples, to trigger on ordinary pharmaceutical IP language rather than fixed strings. With Qwen/Qwen2.5-0.5B-Instruct served by vLLM 0.25.1, the backdoored model covertly called send_email to an external address while showing a benign response, whereas the base model and controls remained clean. The defense-in-depth approach combines OpenShift AI, Advanced Cluster Security, and Advanced Cluster Management to restrict runtime LoRA loading, require provenance-checked adapters, enforce admission and build policies, and block unexpected egress. Because weight inspection cannot reliably reveal semantic behavior, protection focuses on deployment, runtime, tool, and network boundaries.
Orchestrate production RAG with OpenShift AI
This guide shows how to move a streaming retrieval-augmented generation (RAG) workflow from a monolithic Ray Data script to modular AI pipelines on OpenShift AI. It addresses failure recovery and run traceability by separating parsing and chunking, ingestion, model download, and deployment into five reusable components. Parsed and chunked JSONL is stored in S3-compatible storage so KFP can rerun ingestion without repeating document processing, and each run records parameters, timestamps, logs, and status. The data chain uses Docling’s HybridChunker and Ray map_batches with ActorPoolStrategy, while the model chain downloads a Hugging Face model and deploys it through vLLM and KServe in parallel. For embedding ingestion, the workflow supports local CPU processing with granite-embedding-125m-english or a dedicated GPU vLLM service for larger collections and offers notebooks for deployment and query validation.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.