# Building AI Agents for AR Glasses and XR Devices with NVIDIA XR AI

[NVIDIA Developer Blog](https://yomu.fyi/company/nvidia-developer-blog) · Greg Barbone · Jun 16, 2026

**Type:** Announcement

## Summary

Building AI experiences for augmented reality and wearable devices requires bridging hardware with live media streams, multimodal models, enterprise tools, and runtime infrastructure. NVIDIA XR AI provides an open-source, modular framework connecting extended reality headsets and smart glasses to GPU-accelerated services across cloud, edge, and workstations. In this architecture, camera frames and microphone audio ingest into an XR Media Hub that routes data while keeping raw video pixels in shared memory to minimize overhead. The ecosystem uses NVIDIA Cosmos models for vision-language grounding, NVIDIA Nemotron models for reasoning and tool invocation, and the Model Context Protocol for enterprise integrations. Optional agent orchestration via NVIDIA NeMo Agent Toolkit and spatial streaming through NVIDIA CloudXR support complex workflows across healthcare and manufacturing.

## Context

Developers building AI for AR glasses and wearable hardware face infrastructure hurdles in integrating live audio and video feeds, multimodal models, enterprise data, and device runtimes.

## Approach / What changed

NVIDIA introduced the open-source XR AI beta, a modular architecture that routes live media through an XR Media Hub, uses Cosmos and Nemotron models for perception and reasoning, connects to tools via Model Context Protocol, and streams spatial content using CloudXR.

## Takeaways

- XR AI keeps video pixels in shared memory while passing lightweight metadata to reduce unnecessary model inference and data movement across the system.
- Model Context Protocol servers handle domain integrations, visual question answering, spatial scene manipulation, and enterprise data retrieval.
- Production deployments can pair a smaller model for rapid acknowledgments with a larger model running deeper background reasoning and tool calls.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://developer.nvidia.com/blog/building-ai-agents-for-ar-glasses-and-xr-devices-with-nvidia-xr-ai)