Autonomous LLM post-training with Tunix on TPUs

Summary

autofinetune applies autonomous research loops to LLM post-training, covering supervised fine-tuning (SFT) and reinforcement learning with GRPO through Tunix, Gemma, Cloud TPUs, Antigravity CLI, and Gemini Flash 3.7. A human defines the arena in program.md and supplies a self-contained run.py, while an agent edits hyperparameters, launches jobs, evaluates a target metric, keeps winning Git commits, reverts regressions, and logs results in results.tsv. On FunctionGemma 270M with google/mobile-actions, 20 Cloud TPU v5e-1 experiments tested LoRA settings, optimization choices, schedules, batch size, and seeds without changing the dataset, epochs, or architecture. A second study used Gemma 3 1B, GSM8K, and Cloud TPU v6e-1 for 40 GRPO experiments over two to three days, improving the combined numerical-and-format reward by about 10%.

Takeaways

1. 1

   The SFT arena allowed changes to LoRA rank and alpha, target projection layers, learning rates, schedules, optimizers, gradient clipping, batch size, and seeds, while disallowing dataset, epoch-count, and model-architecture changes.

2. 2

   The GRPO study used Post_RL_metric, defined as numerical_accuracy plus format_accuracy, to simplify the agent’s outer optimization loop.

3. 3

   In the GRPO trajectory, the agent identified improvements involving LoRA configurations, rollout temperature, KL penalty, and system prompt, raising total reward by approximately 10%.