Loading…
Fly.io
Global application platform enabling developers to deploy and scale applications with edge computing capabilities.
- 199 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Picture This: Open Source AI for Image Description
Nolan Darilek presents a self-hosted image-description service intended to help blind screen-reader users understand photos, diagrams, menus, and other visual material. Humans often provide inadequate alt text, and the post contrasts that history with increasingly capable LLM-generated descriptions from apps and browser add-ons. The implementation combines Ollama, running the Apache-licensed LLaVA 34B multimodal model, with a PocketBase API over SQLite and a small Python client; collection hooks send images and follow-up questions, retain chat history, and enforce per-user access rules. In testing, a cold request on an A100-40GB Fly Machine took about 45 seconds, while stopping idle GPU Machines reduced resource use at the cost of startup delay. The example produced useful scene descriptions but could not reliably identify leafless tree species, illustrating both the prototype’s value and its limitations.
JIT WireGuard
Fly.io describes how its gateways provisioned WireGuard peers for flyctl and why the old design became unreliable and slow. NATS delivery losses caused failed provisioning, while CI-generated peers accumulated because gateways never removed them, leaving hundreds of thousands of stale kernel entries that slowed reloads and contributed to kernel panics. The replacement detects handshake-initiation packets with BPF or a WebSockets packet hook, runs enough Noise cryptography to recover the initiator’s public key, then fetches matching peer data over an internal HTTP API and installs it on demand. A rate-limited SQLite cache supports cleanup jobs, and the gateway can use the observed four-tuple to initiate back toward flyctl and accelerate handshakes. After weeks in production, stale peers rounded to none, with faster setup and rebooting that no longer required reloading unused entries.
Fly Kubernetes does more now
Fly Kubernetes is moving from early access to a closed beta as a managed Kubernetes service backed by Fly.io infrastructure. It provides the Kubernetes API while mapping core primitives to Fly resources: containers become Firecracker microVMs through flyd and Firecracker, pods become Fly Machines, networking uses an internal WireGuard mesh, services use the Fly Proxy, and storage uses Fly Volumes. Virtual Kubelet, running alongside K3s, receives pod requests and deploys them through the Fly Machines API, removing user-managed node capacity planning and hardware management. The beta supports workloads, including GPUs, across available regions, with Deployments/ReplicaSets, Services, DNS, and ephemeral and persistent volumes, but only part of the Kubernetes API. Missing capabilities include multi-container pods, StatefulSets, network policies, horizontal pod autoscaling, and emptyDir volumes; pricing is free during beta and planned at $75 per cluster monthly afterward, excluding resource costs.
Globally Distributed Object Storage with Tigris
Globally distributed object storage is presented as a way to avoid the latency and application complexity that arise when modern apps route uploads and downloads through a single-region S3 bucket. Tigris addresses this with redundant FoundationDB clusters for object tracking, Fly.io NVMe volumes as a first-level raw-byte cache, and a queueing system modeled on Apple’s QuiCK paper that sends data to replicas, in-demand regions, and third-party stores such as S3. Objects smaller than about 128 kilobytes are instantly global by default, while the platform also provides inter-region routing, consistency guarantees, instant purge, and relay regions. Its S3-compatible API means existing S3 libraries can be reused, and fly storage create injects the required AWS configuration and secrets. Fly.io says the partnership delivers object storage under its billing system without requiring a separate account.
GPUs on Fly.io are available to everyone!
Fly.io has made GPU instances available to everyone, adding datacenter-grade accelerators to its globally distributed compute platform. Users can select Ampere A100 variants with 40GB or 80GB of memory, or a Lovelace L40s with 48GB, by setting an app or process’s vm.size to the corresponding GPU kind. The post demonstrates an Ollama deployment using an L40s, a 100GB persistent volume mounted at /root/.ollama, and HTTP service settings that allow machines to stop and start automatically with zero minimum running machines. These instances are positioned for large language model inference, text transcription, and image generation; lower-cost A10 GPUs are planned for the next few weeks, while some organizations may need additional verification based on account age and payment history.
Event Driven Machines
Event Driven Machines presents Lambdo, a pattern for using Fly Machines as short-lived, event-triggered compute rather than relying on HTTP activation. It polls an SQS queue for JSON events, batches the messages and metadata, then creates fast-booting Fly Machines with the events written to /tmp/events.json; the Machine is destroyed after processing finishes. Application code can come from an existing codebase or Lambdo’s Node and PHP runtime images, with handlers or commands processing each event. SQS message attributes select the required image and can specify VM size and command, while the approach supports arbitrary code and even GPU-enabled Machines, giving each event control over its execution environment.
Delegating tasks to Fly Machines
Resource-intensive web tasks such as resizing high-resolution images can require substantially more memory or CPU than ordinary request handling, making an always-powerful web server inefficient. The article presents a framework- and language-agnostic delegation pattern using Fly Machines, the private network, and two process groups that run the same application with different Machine specifications. A Next.js route calls a wrapper that executes the task when FLY_PROCESS_GROUP is worker; otherwise, it sends a POST request to the identical endpoint through the worker’s internal hostname, returning the worker’s response. The example uses a lighter web group and a performance-4x worker group, with local development using separate ports and production routing over .internal addresses; the author calls the implementation naïve and suggests queues and worker timeouts to improve resilience and control worker availability.
Macaroons Escalated Quickly
Fly.io built a Macaroons-based security token system after finding its API tokens too powerful for expressing scoped permissions and roles. The design starts with a user-linked, HMAC-signed bearer token and chains caveats through HMAC, while checking every caveat independently so each added restriction can only weaken authorization. It models organizations, apps, machines, and volumes with first-party caveats, and also includes third-party caveats, service-token transformations, machine locking, nonce-based revocation, and expiration handling across customer workloads. The resulting system lets users create roles without changing the platform and supports SSO, plugins, and traceable inter-service authorization, although the author acknowledges extra complexity and awkward ways to express some roles.
How Yoko Li makes towns, tamagoes, and tools for local AI
Yoko Li’s local AI work spans playful agent experiments and practical tools for running language-model workflows on personal hardware. AI Town uses conversations among AI agents to produce changing relationships and emergent behavior, while AI Tamago feeds game state and user inputs to an unmodified LLaMA 2 7B model and uses Ollama’s JSON mode to generate pet actions. The Local AI Starter Kit packages document ingestion, vector storage, and language-model responses into a private, local “chat with a knowledge base” pipeline, avoiding setup across at least four API providers. The post also describes unreliable JSON generation, hallucinations, and grammar-guided workarounds, concluding that randomness is a liability for factual outputs but useful for game behavior and creative experimentation.
Deploy Your Own (Not) Midjourney Bot on Fly GPUs
This tutorial builds a Discord image-generation bot that uses Fooocus, an open-source interface for Stable Diffusion, to provide a Midjourney-like /imagine command. It separates the system into a normal Fly Machine running a Python Pycord bot and a GPU Machine running Fooocus-API, with Flycast connecting them over Fly.io’s private network. The bot defers Discord responses, polls for the generated image each second, and uploads the resulting file, while the post notes that this blocking flow will not perform well for hundreds of simultaneous users. Fly GPU rootfs supports the nearly 12GB image, and an L40S setup costs about $3.20/hr when running, with scale-to-zero intended to avoid charges for unused compute. GPU activity is verified with nvidia-smi, and the source code can be extended with additional commands and image-to-image features.
Fly With Alpine
Alpine Linux can reduce container image sizes compared with Debian slim, but switching distributions involves compatibility and ecosystem trade-offs. The post compares Fly.io’s Debian-based baseline across Bun, Node, Python, and Ruby, then reports smaller Alpine base and application images, including 304MB versus 428MB for a minimal Rails/Postgresql/esbuild application. Migration consists of changing Dockerfile base images from -slim to -alpine, replacing apt-get with apk, and reviewing package-name differences; recent Bun, Node, and Rails generators can make these changes with an --alpine option. Alpine uses musl rather than Debian’s glibc, and issues can involve DNS, missing commands or packages, precompiled binaries, insufficient testing of release candidates, and software such as Chrome. Fly.io will retain Debian as its default while keeping Alpine available.
Introducing Fly Kubernetes
Fly.io introduces Fly Kubernetes (FKS), an implementation of Kubernetes that runs on its globally distributed container platform for users who want Kubernetes tooling and declarative orchestration. Fly.io says its earlier Nomad-based approach struggled with global scheduling at more than 200,000 applications and with placing workloads in users’ requested regions, so it moved state to individual workers and exposed the Fly Machines API, whose placement requests fail quickly rather than retry elsewhere. FKS combines K3s with Virtual Kubelet, mapping pod lifecycle operations to Fly Machines API calls while running K3s and Virtual Kubelet on one Fly Machine. A generated kubeconfig enables kubectl access, and Pods run on separate Fly Machines as the platform scales the cluster. The design provides declarative replica, autoscaling, and volume definitions plus automatic recovery for failed Machines, while retaining Fly Machines’ trade-off of failing when requested placement is unavailable.
Fly.io has GPUs now
Fly.io announces GPU support for its cloud, extending its near-user deployment model to AI and machine-learning workloads at the edge. The offering attaches Nvidia A100 GPUs with 40GB or 80GB of RAM, CUDA support, eight AMD EPYC CPU cores, and volumes up to 500GB; L40s are also listed as coming soon and then noted as available. A sample fly.toml deploys Ollama with a 100GB model volume, while auto-start and auto-stop settings can reduce min_machines_running to zero when idle. GPU apps can scale across supported regions, retaining the same public IP address and TLS certificates, with an Amsterdam example using fly scale count 2 --region ams. Listed prices are $2.50 per hour for A100 40GB and $3.50 for A100 80GB, with reserved-instance and dedicated-host discounts available.
What are these "GPUs" really?
The post explains how graphics hardware evolved from CPU-offloaded video rendering into modern GPUs and why those devices now matter for AI/ML. It traces the path from the Commodore 64’s VIC-II and 3D accelerator cards through Quake, Tomb Raider, integrated graphics, and increasingly parallel hardware, while describing components such as framebuffers, command processors, shader hardware, memory, and display outputs. It then connects shader and CUDA cores to workloads including GPT-2, Stable Diffusion, and speech or text processing, and presents model chaining as more useful than treating models as isolated tools. A conference-video workflow combines ffmpeg, Whisper, subtitle segmentation, a large language model, Markdown, and human review to produce accessible written material. The conclusion distinguishes graphics-capable cards from A100 accelerators, which the post says cannot process graphics or encode video but are highly effective for AI/ML workloads.
Scaling Large Language Models to zero with Ollama
Running GPU nodes for language-model inference can be costly because they often sit idle between requests, so this guide configures Ollama on a Fly.io Machine that scales to zero when idle. Ollama is described as a wrapper around llama.cpp, enabling self-hosted models with GPU acceleration, while about 70 lines of Go add authentication to protect the endpoint from unauthorized use. Deployment uses an A100-40GB GPU, a private Flycast address, automatic start and stop settings, and a 100GB persistent volume mounted at /root/.ollama so downloaded models survive stopped Machines. A stopped Machine no longer incurs GPU charges, but scaling to zero suspends it on the same host and may fail to wake during capacity issues. The setup supports CLI and API requests, WireGuard access, custom Modelfiles, and JSON-mode function-calling workflows.
Rethinking Serverless with FLAME
Rethinking Serverless with FLAME presents a pattern for running selected application functions on short-lived copies of the full app instead of moving code into bespoke FaaS services. FLAME, short for Fleeting Lambda Application for Modular Execution, uses FLAME.call to send a function and closed-over values to a runner pool, while retaining access to the application’s database connections and other code. The Elixir flame library includes LocalBackend and FlyBackend adapters, and Fly.io can boot a same-image Machine in the parent’s region in about three seconds. Runners may stay warm, scale to zero, or grow elastically based on configuration. The post also describes process-based streaming with LiveView and supervision and monitoring mechanisms for transient runners.
The risks of building apps on ChatGPT
The post argues that applications with critical AI features should avoid relying on proprietary, externally hosted models such as OpenAI’s ChatGPT and consider self-hosted open source alternatives. It identifies single-provider, regulation or policy, financial, and governance risks, illustrating the first with a Personal AI Fitness Trainer that failed during an OpenAI outage on November 8, 2023. When the AI is the app’s primary value proposition, an outage, changed behavior, denied usage, price increase, or provider leadership crisis can make the service unusable or threaten the business; less-central integrations may instead cause only degraded experience. Self-hosting a model can insulate an application from provider interruptions, licensing or usage-fee changes, regulator- or company-driven behavior changes, and leadership turmoil, while introducing GPU capital and operating costs with rented cloud GPU access available as an alternative to owning GPUs.
Print on Demand
Print on Demand presents a resource-conscious alternative to scaling an application by separating PDF generation from the main server. The post explains how a Fly.io PDF appliance uses Fly-Replay to route a PDF request to a machine running headless Chrome, which requests the original page without its “.pdf” extension and converts the returned HTML into a PDF. Each machine reuses one Chrome instance across requests, while auto_stop_machines allows idle machines to stop and restart on demand. Integration requires cloning the GitHub repository, creating and configuring an app, deploying it, and returning a fly-replay response header for PDF-directed requests. The extracted production code is packaged for deployment with existing applications, supports session, cookies, and basic authentication through passed-back HTTP headers, and can serve applications using different languages and frameworks.
Launching to Victory
Fly.io introduces a streamlined version of its fly launch command for deploying applications near their users. The CLI now infers desired infrastructure from application source code and offers opinionated defaults or alternative options when it cannot apply the same shape. Previously, the launch flow asked a series of questions; the change reduces guesswork by folding varied frameworks, languages, and toolchains into a few basic infrastructure shapes. A Rails application using SQLite is given an example set of defaults that developers can build from, while other applications can be configured through a confirmation webpage. Once settings are accepted, the app is built and deployed unless --no-deploy is supplied, and the CLI provides a link to the running app.
How I Fly
This article describes how Xe Iaso uses Fly.io to host a blog, CDN, and experimental personal projects while deploying services near users. The blog backend runs in Toronto, while XeDN runs in 35 datacenters, fetches missing files from Backblaze B2, and uses caching and traffic shifting to tolerate individual node failures. Arsène runs every 12 hours, asking the ChatGPT API for posts, using Stable Diffusion for illustrations through an Automatic1111 API on a private network, then sending images to XeDN for global delivery. The setup emphasizes reusable, interoperable building blocks rather than isolated projects, with Fly.io handling deployment, networking, and scaling concerns. Xe reports that three 256 MB Machines ran XeDN for a year, while Arsène still runs on one, and says applications can scale to zero when idle.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.