---
title: "Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio"
description: "MLX enables running large language models locally on Apple silicon by exploiting the unified memory architecture of Mac Studio. The latest MLX release supports distributed inference across multiple Macs using RDMA over Thunderbolt, with pipeline or tensor parallelization schemes, and uses the jaccl communication collective. Benchmarks compare M3 Ultra and M5 Ultra across three models: Qwen 3.8 (27B) and Mistral Medium 3.5 (128B) fit on a single 512GB Mac, while the newly released Muse Spark Open (816B) requires parallelization across up to four machines at its 4-bit NVFP4 quantization. The M5 Ultra’s GPU Neural Accelerator delivers up to 4.6 times faster time-to-first-token than the M3 Ultra, while its 1.2 TB/s memory bandwidth improves generation throughput by 1.6 times for dense models. On four M5 Ultra Mac Studios with tensor parallelism, Muse Spark Open achieves over 36 tokens per second."
---

# Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio

[Apple](https://yomu.fyi/company/apple) · Oct 15, 2026

**Type:** Benchmark

## Summary

MLX enables running large language models locally on Apple silicon by exploiting the unified memory architecture of Mac Studio. The latest MLX release supports distributed inference across multiple Macs using RDMA over Thunderbolt, with pipeline or tensor parallelization schemes, and uses the jaccl communication collective. Benchmarks compare M3 Ultra and M5 Ultra across three models: Qwen 3.8 (27B) and Mistral Medium 3.5 (128B) fit on a single 512GB Mac, while the newly released Muse Spark Open (816B) requires parallelization across up to four machines at its 4-bit NVFP4 quantization. The M5 Ultra’s GPU Neural Accelerator delivers up to 4.6 times faster time-to-first-token than the M3 Ultra, while its 1.2 TB/s memory bandwidth improves generation throughput by 1.6 times for dense models. On four M5 Ultra Mac Studios with tensor parallelism, Muse Spark Open achieves over 36 tokens per second.

## Context

MLX is an open-source array framework by Apple for running machine learning models efficiently on Apple silicon. It exploits the unified memory architecture to allow large models to run on personal hardware. The post benchmarks recent LLMs on Mac Studio M5 Ultra and M3 Ultra, including distributed setups via RDMA over Thunderbolt.

## Approach / What changed

Using MLX’s mlx.launch command with the jaccl backend for RDMA over Thunderbolt to parallelize large models across multiple Mac Studios via pipeline or tensor parallelization.

## Takeaways

- MLX on M5 Ultra achieves up to 4.6x faster time-to-first-token than M3 Ultra due to the GPU Neural Accelerator for compute-bound matrix operations.
- Muse Spark Open (816B) requires distributed inference across multiple Macs; with tensor parallelism on four M5 Ultra machines it reaches over 36 tokens/s at 8k context.
- MLX uses RDMA over Thunderbolt 5 for inter-Mac communication, supporting both pipeline and tensor parallelization via the jaccl collective.

## FAQ

### Which models run on a single Mac Studio with 512GB?

Qwen 3.8 (27B) and Mistral Medium 3.5 (128B) fit comfortably on a single Mac Studio with 512GB RAM, using up to 68GB and 158GB respectively at 65k context. Muse Spark Open (816B) does not fit and requires distributed inference.

### How is distributed inference set up with MLX?

Use the mlx.launch command with the jaccl backend, specifying a hostfile with RDMA-over-Thunderbolt connection. Models can be split via pipeline parallelization (split by layers) or tensor parallelization (split by width).

### What is the power consumption of Mac Studio during LLM inference?

Peak power consumption is about 300W for the M5 Ultra and 200W for the M3 Ultra during inference, which is a fraction of a household appliance like a toaster.

**Tags:** [Apple Silicon](https://yomu.fyi/topic/apple-silicon), [LLMs](https://yomu.fyi/topic/llm), [MLX](https://yomu.fyi/topic/mlx), [RDMA](https://yomu.fyi/topic/rdma)

- Source: [Apple](https://machinelearning.apple.com/research/mlx-distributed-inference)
- Source URL: https://machinelearning.apple.com/research/mlx-distributed-inference
- Ingested by Yomu: 2026-10-15T18:00:11.843Z

[Read original post](https://machinelearning.apple.com/research/mlx-distributed-inference)