Loading…
Build On-Device AI Companions with the NVIDIA ACE Game Agent SDK and Unreal Engine 5 Plugins
NVIDIA Developer BlogPhillip Singh
Summary
Game developers integrating on-device artificial intelligence companions face challenges such as conversational latency, state synchronization, and loop prevention. NVIDIA announced the open-source ACE Game Agent SDK alongside a suite of Unreal Engine 5 plugins to streamline native, hardware-accelerated character pipelines. The SDK offers Agent, Chat, and RAG APIs, enabling characters to execute multi-step tool-assisted reasoning and query game databases. Unreal Engine 5 plugins introduce local runtime models including nemo-conformer-ctc-120m for speech recognition, Qwen 3.5 4B for language generation, and Chatterbox Turbo 350M for speech synthesis. Additional tooling expands to motion generation via Animotive Kimodo and rendering updates with the DLSS 4.5 Unreal Engine plugin.
Context
Building on-device AI non-player characters requires overcoming conversational latency, synchronizing global and dynamic game states, avoiding infinite loops, and mitigating cloud compute costs.
Approach / What changed
NVIDIA released the open-source ACE Game Agent SDK in beta along with native Unreal Engine 5 plugins for automatic speech recognition, small language models with GGUF support, and text-to-speech, running locally on RTX hardware.
Takeaways
- The NVIDIA ACE Game Agent SDK provides Agent, Chat, and RAG APIs for stateful reasoning, direct inference control, and hybrid knowledge retrieval.
- NVIDIA ACE Unreal Engine 5 plugins include local runtime AI models such as nemo-conformer-ctc-120m ASR, Qwen 3.5 4B SLM, and Chatterbox Turbo 350M TTS.
- The DLSS 4.5 Unreal Engine plugin integrates Dynamic Multi Frame Generation, Multi Frame Generation 6X Mode, and a second-resolution transformer for Super Resolution.
Related reading
Q&A: How KRAFTON Built PUBG Ally, a Co-Playable Character Powered by NVIDIA ACE
KRAFTON developed PUBG Ally, an on-device AI co-playable character for PUBG: BATTLEGROUNDS, using NVIDIA ACE tooling and models. Conventional non-playable characters rely on fixed dialogue, but PUBG Ally processes live game states and player voice input in real time. The interactive pipeline combines automatic speech recognition, a quantized Mistral-NeMo-Minitron-2B small language model, and a custom text-to-speech model running locally on client GPUs with at least 8GB of VRAM. To minimize latency, the architecture separates reactive gameplay into a behavior-tree System 1 layer while routing strategic decisions and speech generation to the language model. KRAFTON tuned the non-deterministic system across multi-layer evaluations and large-scale playtests with over a thousand players.
Elizabeth Goodman