---
title: "Local AI"
description: "3 posts about Local AI, summarised, each linking to the original."
---

# Local AI
> 3 posts about Local AI, summarised, each linking to the original.

## Articles

### [Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples](https://yomu.fyi/post/build-local-ai-apps-with-c-and-nvidia-tensorrt-rtx-samples.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Luca Spindler
- Published: Oct 1, 2026
- Type: Explainer

DIN Deploy is an open-source collection of C++ samples combining ONNX Runtime with NVIDIA TensorRT RTX for local AI inference on Windows and Linux. It separates model conversion from deployment: Python exports models to ONNX, then native C++ runs inference via ONNX Runtime session and tensor APIs. Supported workloads include automatic speech recognition (Whisper offline, Parakeet TDT streaming), interactive segmentation (SAM 2.1), and prompt-driven image generation (FLUX.2-klein-4B). The FLUX.2 sample includes Vulkan and DirectX graphics interop and demonstrates post-training quantization with NVIDIA Model Optimizer, producing quantized ONNX models as drop-in replacements without code changes. On DGX Spark, GPU acceleration provides speedups: Whisper large-v3-turbo achieves 58.5x real-time factor, Parakeet TDT 206.41x, and SAM 2.1 runs at 38.3 FPS on GPU versus 0.5 FPS on CPU.


### [Transformers now runs llama.cpp quants](https://yomu.fyi/post/transformers-now-runs-llama-cpp-quants.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Marc Sun, Arthur Zucker, Lysandre
- Published: Sep 22, 2026
- Type: Announcement

Transformers now supports running GGUF quantized models locally, reusing llama.cpp’s ggml kernels for efficient inference on Apple Silicon. The GGUF format packages weights, tokenizer, and chat template in one file, with quantization levels like Q4\_K\_M that trade precision for memory. Users can load a checkpoint from the Hub with from\_pretrained and generate text using the standard Transformers API, or serve it via an OpenAI-compatible endpoint. The initial focus is on the Qwen3.5 architecture, with benchmarks on a MacBook Pro M2 Max showing token-generation rates close to llama.cpp, including prefill time. The integration uses packed quantization kernels and generation-loop optimizations to reduce overhead, though it is currently limited to single conversations on MPS without batching support.


### [Jun Kim, oMLX creator and maintainer, joins Hugging Face to support the MLX community](https://yomu.fyi/post/jun-kim-omlx-creator-and-maintainer-joins-hugging-face-to-support-the.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Pedro Cuenca, Lysandre, Victor Mustar, Julien Chaumond, Jun Kim
- Published: Sep 22, 2026
- Type: Announcement

Jun Kim, creator and maintainer of oMLX, has joined Hugging Face to support the MLX community. MLX is Apple’s framework for local AI, optimized for Apple Silicon, and Hugging Face is the Hub for MLX models. oMLX, staying Apache 2.0, will gain stability and faster development as a funded, maintained project, allowing Jun to guide contributors and build for the long term. The goal is to unblock the community to run local AI in any form, with oMLX serving as a testbed leveraging dependencies like mlx-lm and mlx-vlm, and collaborating with projects such as LMStudio. A key focus is streamlining the transition from transformers model definitions to reference MLX implementations for different engines, so each can focus on unique features.