Transcribing on Fly GPU Machines

Summary

Fly GPU Machines are used to run Whisper Webservice for audio transcription, with the post noting that Fly GPUs are deprecated and unavailable after August 1. The setup uses a GPU-friendly Docker image, an a100-pcie-40gb Machine, and a Fly Volume mounted at /root/.cache/whisper so model files download and persist outside the image; ASR_MODEL=large and ASR_ENGINE=faster_whisper configure the service. It shows flyctl commands for creating the app, GPU-hosted volume, Machine, public IPs, and HTTPS routes, then verifies GPU availability with nvidia-smi and exposes the web UI and OpenAPI specification. A second path automates volume and Machine creation through the Machines API, requiring cpu_kind plus valid cpus and memory_mb values. It also demonstrates a custom Ubuntu 22.04 image with NVIDIA CUDA libraries, FFmpeg, Python, and Whisper, concluding that GPU Machines can support composable AI workloads.

Takeaways

1. 1

   Fly GPU Machines offer a100-pcie-40gb and a100-sxm4-80gb options; the stated distinction is 40 GB versus 80 GB of GPU memory.

2. 2

   Creating the volume with --vm-gpu-kind places it on a GPU host, and using the same region and GPU kind for the Machine can place both on the same host.

3. 3

   Machines API requests need matching cpu_kind and gpu_kind settings; Machine creation also requires valid cpus and memory_mb values for performance Machines.