From token consumer to token provider: Building your org's AI API

Summary

Enterprise AI usage can become costly and fragmented when teams rely on separate provider accounts, personal keys, and independently operated inference stacks, leaving spending, workloads, and governance difficult to see. The post presents Model-as-a-Service (MaaS) in Red Hat OpenShift AI 3.4 as a shared service in which a platform team deploys approved models and exposes them through a standard, OpenAI-compatible API. Administrators can use the model catalog, distributed inference through llm-d on NVIDIA L40s, scoped API keys, rate limits, and dashboards for token use, latency, errors, GPU utilization, and queue length. Developers obtain self-service endpoints and can connect tools such as Claude Code, OpenCode, or Continue with existing clients. The architecture combines OpenShift, vLLM, an AI gateway, Prometheus, Grafana, and Jaeger to centralize inference, governance, and cost attribution across local and external model providers.

Takeaways

1. 1

   MaaS exposes approved models through an OpenAI-compatible /v1/chat/completions endpoint and can route requests to locally served vLLM models or external providers such as AWS Bedrock, Azure, and Anthropic.

2. 2

   API keys are generated for team access, scoped to policy, and revocable; admin-defined rate limits return HTTP 429 responses that standard clients can handle with backoff.

3. 3

   Prometheus, Grafana, and Jaeger support tracking token consumption, request rates, latency, errors, GPU utilization, and distributed traces.