---
title: "Games as Model Eval: 1-Click Deploy AI Town on Fly.io"
description: "Daniel Botha argues that conventional benchmarks reveal little about how models behave in long-context, interactive situations or whether they deliver the tone a product requires. He presents games as a more observable evaluation setting, because success signals are unambiguous and gameplay exposes strategic reasoning, long-term planning, and adaptation with opponents or collaborators. The post focuses on a fork of a16z-infra’s AI Town, a pixel-art simulation inspired by Generative Agents: Interactive Simulacra of Human Behavior, where characters remember conversations, maintain relationships, react to events, and stay in character. A single deploy script runs the project on Fly Machines with scale to zero, cost and performance optimizations, and support for OpenAI-compatible services, Together.ai models, and custom embedding models configured through secrets."
---

# Games as Model Eval: 1-Click Deploy AI Town on Fly.io

[Fly.io](https://yomu.fyi/company/fly-io) · Aug 11, 2025

**Type:** Announcement

## Summary

Daniel Botha argues that conventional benchmarks reveal little about how models behave in long-context, interactive situations or whether they deliver the tone a product requires. He presents games as a more observable evaluation setting, because success signals are unambiguous and gameplay exposes strategic reasoning, long-term planning, and adaptation with opponents or collaborators. The post focuses on a fork of a16z-infra’s AI Town, a pixel-art simulation inspired by Generative Agents: Interactive Simulacra of Human Behavior, where characters remember conversations, maintain relationships, react to events, and stay in character. A single deploy script runs the project on Fly Machines with scale to zero, cost and performance optimizations, and support for OpenAI-compatible services, Together.ai models, and custom embedding models configured through secrets.

## Context

Conventional benchmarks are described as insufficient for assessing how models behave in the wild, especially with long contexts, dynamic interactions, strategic decisions, and the tone or feel required by a product experience.

## Approach / What changed

A fork of a16z-infra’s AI Town provides a single deploy script for Fly Machines, with scale to zero, cost and performance optimizations, and configuration for OpenAI-compatible services, Together.ai models, and custom embedding models through secrets.

## Takeaways

- Games provide observable, unambiguous success signals while testing strategic reasoning, long-term planning, and dynamic adaptation.
- AI Town evaluates conversational behavior through characters that remember conversations, maintain relationships, respond to new situations, and remain in character.
- The fork supports scale to zero on Fly Machines and can test OpenAI-compatible services, Together.ai models, and custom embedding models.

**Tags:** [AI Town](https://yomu.fyi/topic/ai-town), [Fly Machines](https://yomu.fyi/topic/fly-machines), [Model Evaluation](https://yomu.fyi/topic/model-evaluation)

- Source: [Fly.io](https://fly.io/blog/games-as-model-eval)
- Source URL: https://fly.io/blog/games-as-model-eval
- Ingested by Yomu: 2026-09-03T01:23:27.680Z

[Read original post](https://fly.io/blog/games-as-model-eval)