Building Federated Multimodal AI Workflows with NVIDIA FLARE

Summary

Federating vision-language models (VLMs) across institutions requires keeping images, text, and prompts local while coordinating training and combining updates from sites with different task and modality mixes. NVIDIA FLARE addresses this workflow through server-scheduled rounds, client-local execution, and an explicit update contract, while supporting large-object externalization, pull-based tensor streaming, and disk-backed aggregation for large payloads. The post presents FedUMM as an adapter-based design in which clients train LoRA adapters over a frozen BLIP3o backbone and send only adapter updates, while full-model training remains possible when required. In reported eight-client experiments, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round, improved VQA v2 by 0.7 points, and retained about 97% of centralized performance on VQA v2 and GenEval; the evaluation used simulated sites and does not establish clinical performance or formal privacy guarantees.

Takeaways

1. 1

   The federated update contract should define what remains local, which model components clients may update, what leaves each site, and how component-level updates are combined.

2. 2

   FLARE Tensor Downloader streams requested PyTorch tensor chunks, while tensor disk offload stores incoming FedAvg updates as temporary safetensors files to limit server CPU-memory growth during aggregation.

3. 3

   In eight-client FedUMM experiments, adapter-only federation reduced communication from 28.6 GB to 0.094 GB per client per round and improved VQA v2 by 0.7 points over full-model FedAvg.