--- title: "Fly.io" description: "Global application platform enabling developers to deploy and scale applications with edge computing capabilities." --- # Fly.io > Global application platform enabling developers to deploy and scale applications with edge computing capabilities. ## Articles ### [My AI Skeptic Friends Are All Nuts](https://yomu.fyi/post/my-ai-skeptic-friends-are-all-nuts.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Jun 2, 2025 - Type: Explainer The essay argues that skepticism about LLMs for software development often reflects outdated experience with chat-based code generation rather than current agent workflows. Modern coding agents inspect repositories, edit files, invoke Unix tools and MCP calls, use Git, and run linters, compilers, tests, and model checkers, turning feedback into iterative repairs. The author says these systems can automate tedious typing, lookup, bookkeeping, test refactoring, and edit-compile-test-debug cycles while developers retain responsibility for curation, judgment, direction, and reviewing merged code. Hallucinations are presented as substantially mitigated by tool-based validation, although the author notes that agents remain unsuitable for some trusted operations and that effectiveness depends on language, tooling, prompts, indices, and guardrails. The conclusion is that AI-assisted programming is already materially changing developer productivity, despite its noise and unresolved limitations. ### [Using Kamal 2.0 in Production](https://yomu.fyi/post/using-kamal-2-0-in-production.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: May 29, 2025 - Type: Announcement Sam Ruby describes publishing an online guide to using Kamal 2.0 for production after finding existing material insufficient for deploying a real project. The guide grew from research for the Deployment and Production chapter of Agile Web Development with Rails 8, then was converted from Pragmatic Programming’s XML grammar to Markdown with GitHub Copilot and proofread with it. It presents an opinionated starter list with defaults and alternatives, covering seven initial requirements alongside Docker builders, container repositories, secret management, hosting choices, load balancing, and managed databases. It also treats production as including database backups, searchable logs, monitoring, and application security, and invites community corrections or additions under a CC0 license. ### [parking\_lot: ffffffffffffffff...](https://yomu.fyi/post/parking-lot-ffffffffffffffff.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: May 28, 2025 - Type: Incident / postmortem Fly.io’s fly-proxy, a Rust-based Anycast router, suffered lockups while managing rapidly changing routing state across thousands of servers. An earlier outage came from an if let expression that kept a read lock through both branches, while later regionalization and lazy loading increased write contention in the Catalog. The investigation traced the new failures to parking\_lot’s try\_write\_for wakeup path: after a waiting writer timed out, it attempted to clear WRITER and WRITER\_PARKED together even though one bit had already been cleared, and wrapping arithmetic produced the corrupted lock value 0xFFFFFFFFFFFFFFFE. The parking\_lot team fixed the defect, after which the lockups did not recur; the team also audited Catalog locking, removed if let patterns and RAII lock guards, and added lock-timing metrics, labeled slow-write logs, and holder tracking. ### [Litestream: Revamped](https://yomu.fyi/post/litestream-revamped.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: May 20, 2025 - Type: Announcement Litestream’s redesign aims to make SQLite suitable for more deployment scenarios while preserving its embedded, application-oblivious model. The original tool continuously streamed WAL pages to S3-compatible object storage, but restores could be slow because they replayed every change since the last snapshot. The new design adopts LiteFS’s transaction-aware LTX format, sorted changesets, and compaction, enabling faster point-in-time restores with fewer duplicate pages; conditional writes provide time-based leases that prevent overlapping instances from confusing a destination. Planned additions include a VFS-based read-replica layer that fetches and caches pages from object storage, plus replication across hundreds or thousands of database files. The remote read path may be less efficient than local SQLite, but caching and prefetching are expected to provide strong performance for suitable use cases. ### [Launching MCP Servers on Fly.io](https://yomu.fyi/post/launching-mcp-servers-on-fly-io.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: May 19, 2025 - Type: Announcement The post presents Fly.io’s approach to simplifying remote deployment of Model Context Protocol (MCP) servers, which otherwise require client-specific JSON configuration, local processes or HTTP hosting, and potentially cumbersome OAuth 2.1 setup. It uses the Slack MCP Server as a demonstration, converting its npx command into a fly mcp launch invocation with --claude, --server, and secret flags for the bot token and team ID. Running this requires flyctl v0.3.125 or later, and the command supports Fly.io options including auto-stop, file contents, flycast, regions, volumes, secrets, and VM sizes. Built-in configuration support covers Claude, Cursor, Neovim, VS Code, Windsurf, and Zed, with bearer-token authentication enabled by default on both server and client. The broader beta support includes multiple transports, Machines API deployment, separate Machines or containers, in-application deployment, and HTTP Authorization, WireGuard tunnels, flycast, or reverse proxies. ### [Provisioning Machines using MCPs](https://yomu.fyi/post/provisioning-machines-using-mcps.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: May 7, 2025 - Type: Announcement Fly.io’s flyctl MCP server lets LLMs such as Claude or Cursor use tools to inspect and modify application resources, demonstrated by creating a volume and adding support for all volume commands. The author describes asking Claude to identify unattached volumes and delete the oldest, while noting that flyctl’s mounted-volume safeguards provided an error if destruction was unsafe. Compared with writing against the Machines API, using flyctl directly, or navigating the dashboard, the conversational workflow surfaced an observation and turned it into an action with less manual effort. The post outlines a future in which an LLM proposes deployment plans, examines build errors and logs, and acts through MCP servers on Fly.io’s private network. Users can try the current open-source server with fly version upgrade to v0.3.117, configure an LLM, or run the MCP Inspector; apps, logs, machines, orgs, platform status, and volumes are roughed in. ### [30 Minutes With MCP and flyctl](https://yomu.fyi/post/30-minutes-with-mcp-and-flyctl.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Apr 10, 2025 - Type: Explainer Kurt Mackey describes building a minimal MCP server for flyctl in 30 minutes, motivated by testing how an LLM could inspect deployed applications. MCP connects an LLM client to server-exported tools through JSON exchanges; this implementation uses stdio mode, a Go MCP library, and exposes only fly logs and fly status, producing roughly 90 lines of code. Existing JSON modes in flyctl commands made the tools easier to drive from automation. When pointed at unpkg through Claude, the server surfaced a globally distributed deployment, identified two critical machines, and connected repeated restarts and oom\_killed events to Bun using about 3.7 GB of a machine’s 4 GB allocation. The post calls the result surprisingly useful while warning that local MCP servers give cloud-hosted Claude access to native programs and would be safer in isolation. ### [Our Best Customers Are Now Robots](https://yomu.fyi/post/our-best-customers-are-now-robots.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Apr 8, 2025 - Type: Explainer Fly.io reports that users driving the platform’s most recent growth are robots rather than people, prompting a reconsideration of developer experience (DX) as the primary product lens. The post describes how vibe-coding workflows use Fly Machines—Docker containers running as hardware-isolated virtual machines—to create, modify, stop, and rapidly restart environments without paying during stopped periods. These workflows favor incremental filesystem state, object storage, and controllable routing for MCP connections, including potentially long-lived SSE connections that should reach the same stateful instance. Fly.io also proposes exposing its API through MCP and using sealed, tokenized secrets so isolated machines can substitute real OAuth credentials only when needed. It concludes that a useful robot experience (RX) will require more than an MCP interface, while managed Postgres remains the company’s most important engineering priority and the platform is still primarily designed for humans. ### [Operationalizing Macaroons](https://yomu.fyi/post/operationalizing-macaroons.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Mar 27, 2025 - Type: Explainer Operationalizing Macaroons at Fly.io requires treating them as online-stateful bearer tokens whose nonces must be checked against database state, while keeping root secrets away from the primary API cluster. The team built tkdb, a roughly 5,000-line Go service backed by encrypted SQLite managed with LiteFS and Litestream, deployed on isolated hardware across the US, Europe, and Australia. Its HTTP/Noise interface supports verification and restricted signing, while FlyCast routing, retries, and client-side caching reduce the impact of unreliable global connectivity; verification caching exceeds 98%. Revocation uses blacklist state, and third-party caveats let services such as flyd obtain narrowly scoped secret-reading authority through tkdb rather than holding broad credentials. OpenTelemetry, standard metrics, and a retained OpenSearch audit trail provide visibility, while the post concludes that Macaroons’ internal infrastructure benefits outweigh limited user adoption. ### [Taming A Voracious Rust Proxy](https://yomu.fyi/post/taming-a-voracious-rust-proxy.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Feb 26, 2025 - Type: Incident / postmortem Fly edge proxies began reporting elevated HTTP errors and sharply higher CPU utilization on several IAD hosts, with restarting fly-proxy providing only temporary relief. Profiling an affected process showed Rust tracing and repeated Future polls dominating samples while little time was spent in the kernel, pointing to an async busy-loop. Investigation narrowed the fully qualified polling type to a stack containing the proxy’s Duplex and Rustls’s TlsStream, whose mishandled Waker behavior after a TLS CloseNotify could spin when buffered data remained underneath. Tigris Data’s load test, involving tens of thousands of connections that sent small HTTP bodies and terminated early, triggered the condition; stopping the test and deploying the fix ended the spin-outs. The incident prompted plans to instrument spurious wakeups and reinforced that dependency updates require deliberate tracking, testing, and prioritization. ### [We Were Wrong About GPUs](https://yomu.fyi/post/we-were-wrong-about-gpus.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Feb 14, 2025 - Type: Incident / postmortem Fly.io’s GPU bet was intended to give internet-facing apps fast AI/ML inference through Fly GPU Machines, which pair a hardware-mapped Nvidia GPU with a Fly Machine. Building them required dedicated GPU servers, Intel Cloud Hypervisor with PCI passthrough, driver work, orchestrator changes, security assessments, and handling large model-weight files. The product has not found product-market fit because most software developers want LLM APIs and services such as OpenAI, Anthropic, Claude, GPT, Replicate, or RunPod rather than raw GPUs, while serious GPU users seek much larger clusters and lightweight users may be too uncertain a market. L40S demand exists, but those GPUs are not driving the core business. Fly.io will keep existing GPU Machines but does not plan a near-term v2, concluding that the bet was worth making without compromising its isolation or developer experience. ### [The Exit Interview: JP Phillips](https://yomu.fyi/post/the-exit-interview-jp-phillips.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Feb 12, 2025 - Type: Announcement JP Phillips leaves after four years at Fly.io, saying the company’s increasing focus on Managed Postgres and LLMs does not match the direction he hoped to see: a move toward cloud provision and larger established customers. He says his original goal—building Fly Machines to replace HashiCorp Nomad—has been accomplished, with the platform now capable of supporting the next iteration of products. His central engineering contribution was the Fly Machines API and flyd, an independently running orchestrator whose finite-state-machine invocations use deterministic, durable execution and log each step in BoltDB. He also discusses corrosion2, a Rust service that uses SWIM gossip and a CRDT-structured SQLite database to distribute near-real-time machine state. Phillips criticizes GraphQL, Elixir, inconsistent communication and shifting priorities, while defending BoltDB and praising Rust’s productivity. ### [Did Semgrep Just Get A Lot More Interesting?](https://yomu.fyi/post/did-semgrep-just-get-a-lot-more-interesting.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Feb 10, 2025 - Type: Explainer Thomas Ptacek considers whether LLM-driven coding agents could make Semgrep more useful by closing the loop between code generation, execution, and automated repair. He describes Semgrep as a semantics-aware code-search tool whose symbolic variable placeholders and ordinary code can express vulnerability patterns, while noting that teams rarely have time to build extensive rule libraries. Drawing on Cursor’s rules feature, including rules that help organize or generate other rules, he proposes having an agent observe runtime errors, generalize fixes into unit tests, and generate Semgrep rules for the underlying bug. The proposed workflow would test each rule with positive and negative controls, save it, scan the codebase, and fix newly found instances. The post concludes that this adaptive process may be more compelling than predicting every failure in advance. ### [A Blog, If You Can Keep It](https://yomu.fyi/post/a-blog-if-you-can-keep-it.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Feb 10, 2025 - Type: Announcement Thomas Ptacek announces that Fly.io is moving away from tightly choreographed EffortPosts and toward a more conventional blog. Its Hacker News strategy used technical pieces over 2,000 words, runnable code, research surveys, minimal promotion, selected topics, and style rules, making publishing a primary outreach tool. The process became an impediment because an editorial function serialized the team, writers endlessly refined drafts, and fear of wearing out their welcome limited them to perhaps one or two Hacker News opportunities per month. After Kurt pushed him to publish BFAAS, involving 500 lines of Go code and a 4,500-word draft, Ptacek reconsidered that theory and plans more frequent, less optimized posts with room for other team members. ### [VSCode’s SSH Agent Is Bananas](https://yomu.fyi/post/vscode-s-ssh-agent-is-bananas.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Feb 7, 2025 - Type: Explainer VSCode’s remote-editing flow over SSH is examined as a concern for running LLM-driven agent workflows on remote machines. The motivation is to keep iterative code generation and execution away from a development laptop, ideally on a clean-slate Linux instance that starts instantly and cannot damage the host. Unlike Emacs Tramp, which operates through commands on the remote connection, VSCode uses a Bash snippet stager to download an agent, including a binary Node installation. That agent communicates over port-forwarded SSH through a WebSockets connection to the VSCode front end and can browse the filesystem, edit arbitrary files, launch shell PTYs, and persist itself. The author says this raises concerns for development servers and especially production incident work, while noting a custom Fly Machine connection avoids needing this mechanism. ### [AI GPU Clusters, From Your Laptop, With Livebook](https://yomu.fyi/post/ai-gpu-clusters-from-your-laptop-with-livebook.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Sep 24, 2024 - Type: Explainer Livebook, FLAME, and the Nx stack combine Elixir notebooks, elastic execution, and GPU-backed machine learning into a workflow that can move from a laptop to remote infrastructure. Livebook can start runtimes on Fly Machines, connect to existing Erlang/Elixir applications, and synchronize notebook dependencies and modules across nodes while preserving access to private resources within a Fly organization. FLAME wraps arbitrary code in Flame.call, manages executor pools, controls node limits and concurrency, streams results, and shuts down idle nodes or the cluster after disconnection. In the demonstrated AI workflow, video stills are processed on GPU Fly Machines with Llama, then Mistral summarizes the returned descriptions. The same setup also drives a 64-GPU hyperparameter run for BERT, with each L40s node preparing dependencies, loading data, fine-tuning, and streaming chart results to Livebook. ### [Accident Forgiveness](https://yomu.fyi/post/accident-forgiveness.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Aug 21, 2024 - Type: Announcement Public cloud billing can turn accidental deployments, configuration errors, runaway VM allocation, or leaked CI/CD environments into unexpectedly enormous charges, making teams hesitant to experiment. Fly.io announces an initial “Accident Forgiveness” policy: customers with any support contract can request refunds for unexpected charges, almost without questions, while the company reserves the right to cut off abuse. The longer-term goal is an automated billing system with line-item vetoes, but fraud—especially cryptocurrency mining—requires an incremental rollout; the first version uses manual follow-up reminders to investigate incidents. A rebuilt billing system is intended to support future accident protection, including detecting month-over-month spikes, flagging unusual deployments before billing, and identifying opportunities for reserved pricing and backdated reservations. ### [We're Cutting L40S Prices In Half](https://yomu.fyi/post/we-re-cutting-l40s-prices-in-half.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Aug 15, 2024 - Type: Announcement Fly.io lowered the hourly price of NVIDIA L40S GPUs to $1.25, making them available at the same price as A10s after customer data showed that A10s—not the more powerful cards—were its most popular GPU. The post argues that most users adding GPUs to fast networks are running inference rather than training, making networking, memory, object storage, and request latency more important than access to an H100. The L40S is described as an AI-optimized L40 with AI compute performance comparable to an A100, while the L40 retains graphics hardware and is designed for rack use with more memory and lower energy consumption than gaming GPUs. The intended combination places model parameters, datasets, and compute near one another on Fly.io’s Anycast network and Tigris object storage. Examples include Llama 3.1 70B, Flux, Whisper, SegAlign, and DOOM Eternal. ### [Making Machines Move](https://yomu.fyi/post/making-machines-move.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Jul 30, 2024 - Type: Problem & solution Fly.io needed to restore the ability to drain physical workers after attached NVMe volumes made stateful Fly Machines tied to particular servers. Snapshot restoration could lose data, while killing a machine before copying its volume caused unacceptable downtime; copying first risked losing writes. The solution was an asynchronous clone operation built around Linux device mapper’s dm-clone: a new volume is created immediately, reads of unhydrated blocks are fetched from the original over iSCSI, writes go to the destination, and background copying rehydrates the rest. The work also required orchestration across flyd and a fix for Fly Postgres configurations containing literal 6PN IPv6 addresses. After cautious early migrations, the infrastructure team could drain hosts without much ceremony by summer 2024, though routine automated rebalancing remained a future goal. ### [AWS without Access Keys](https://yomu.fyi/post/aws-without-access-keys.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Jun 19, 2024 - Type: Problem & solution The post describes how Fly Machines can access AWS resources such as an S3 bucket without distributing long-lived AWS access keys. It uses OpenID Connect (OIDC): AWS IAM trusts Fly.io’s OIDC identity provider, a role grants narrowly scoped permissions, and the app receives the role ARN as an environment variable. At boot, Fly’s Rust init requests a machine-specific OIDC token through the /.fly/api Unix socket, writes it to /.fly/oidc\_token, and sets AWS\_WEB\_IDENTITY\_TOKEN\_FILE and AWS\_ROLE\_SESSION\_NAME so the AWS SDK can call STS AssumeRoleWithWebIdentity. AWS verifies the token, issues short-lived credentials, and evaluates the role’s IAM policy before serving the requested object. The trust policy can constrain tokens by audience and subject, including an organization, Fly App, or machine; the same pattern can authenticate against other OIDC-compliant providers, although Azure and GCP lack equivalent init integration. [Newer posts](https://yomu.fyi/company/fly-io.md) · [Older posts](https://yomu.fyi/company/fly-io/page/3.md)