Fly.io
Daniel Botha argues that conventional benchmarks reveal little about how models behave in long-context, interactive situations or whether they deliver the tone a product requires. He presents games as a more observable evaluation setting, because success signals are unambiguous and gameplay exposes strategic reasoning, long-term planning, and adaptation with opponents or collaborators. The post focuses on a fork of a16z-infra’s AI Town, a pixel-art simulation inspired by Generative Agents: Interactive Simulacra of Human Behavior, where characters remember conversations, maintain relationships, react to events, and stay in character. A single deploy script runs the project on Fly Machines with scale to zero, cost and performance optimizations, and support for OpenAI-compatible services, Together.ai models, and custom embedding models configured through secrets.