---
title: "Building Federated Multimodal AI Workflows with NVIDIA FLARE"
description: "Federating vision-language models (VLMs) across institutions requires keeping images, text, and prompts local while coordinating training and combining updates from sites with different task and modality mixes. NVIDIA FLARE addresses this workflow through server-scheduled rounds, client-local execution, and an explicit update contract, while supporting large-object externalization, pull-based tensor streaming, and disk-backed aggregation for large payloads. The post presents FedUMM as an adapter-based design in which clients train LoRA adapters over a frozen BLIP3o backbone and send only adapter updates, while full-model training remains possible when required. In reported eight-client experiments, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round, improved VQA v2 by 0.7 points, and retained about 97% of centralized performance on VQA v2 and GenEval; the evaluation used simulated sites and does not establish clinical performance or formal privacy guarantees."
---

# Building Federated Multimodal AI Workflows with NVIDIA FLARE

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Tanya Lenz · Aug 19, 2026

**Type:** Explainer

## Summary

Federating vision-language models (VLMs) across institutions requires keeping images, text, and prompts local while coordinating training and combining updates from sites with different task and modality mixes. NVIDIA FLARE addresses this workflow through server-scheduled rounds, client-local execution, and an explicit update contract, while supporting large-object externalization, pull-based tensor streaming, and disk-backed aggregation for large payloads. The post presents FedUMM as an adapter-based design in which clients train LoRA adapters over a frozen BLIP3o backbone and send only adapter updates, while full-model training remains possible when required. In reported eight-client experiments, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round, improved VQA v2 by 0.7 points, and retained about 97% of centralized performance on VQA v2 and GenEval; the evaluation used simulated sites and does not establish clinical performance or formal privacy guarantees.

## Context

VLM training data may be distributed across institutions that cannot centralize raw records. Federated workflows must accommodate clients with different task and modality mixes, while full-model updates can impose substantial serialization, network, and server-memory costs.

## Approach / What changed

NVIDIA FLARE coordinates server-side rounds and client-local training through an explicit update contract. Large objects can be externalized, PyTorch tensors streamed incrementally, and incoming updates offloaded to disk during aggregation. FedUMM reduces communication further by freezing a BLIP3o backbone and federating only locally trained LoRA adapters.

## Takeaways

- The federated update contract should define what remains local, which model components clients may update, what leaves each site, and how component-level updates are combined.
- FLARE Tensor Downloader streams requested PyTorch tensor chunks, while tensor disk offload stores incoming FedAvg updates as temporary safetensors files to limit server CPU-memory growth during aggregation.
- In eight-client FedUMM experiments, adapter-only federation reduced communication from 28.6 GB to 0.094 GB per client per round and improved VQA v2 by 0.7 points over full-model FedAvg.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [NVIDIA](https://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare)
- Source URL: https://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare
- Ingested by Yomu: 2026-08-28T20:01:04.976Z

[Read original post](https://developer.nvidia.com/blog/building-federated-multimodal-ai-workflows-with-nvidia-flare)