Autoresearch isn’t just for training models

Summary

Repeated visual-regression failures made Polaris CI feedback cycles take about 30 minutes, prompting an attempt to optimize build time rather than fix individual tasks. The approach uses Pi to create an Autoresearch extension that repeatedly forms a hypothesis, measures the build against a 19.1-second baseline, and keeps only changes that improve the metric, discarding crashes and slower runs. The loop identified wasted VRT work: the full component pipeline ran before Storybook recompilation, while TypeScript transformed all 580 component files although 105 needed it. Removing those inefficiencies made the build 65% faster, although the system also produced unacceptable hacks that were discarded. The extension later gained multi-metric support, consistent iteration scripts, auto commits, and open-source adoption, with reported internal improvements including unit tests running 300 times faster and Playwright tests becoming faster.

Context

Every small Polaris change triggered random visual regression failures, requiring 30 minutes of CI before revealing another problem and creating repeated waiting cycles. The motivation was to reduce the underlying build time that all CI pipelines depended on.

Approach / What changed

An Autoresearch extension for Pi focuses an agent on a measurable metric. It records a baseline, generates and tests a hypothesis each iteration, keeps faster successful changes, discards crashes and regressions, and repeats until stopped or out of context. The extension presents iterations in a custom UI and was later expanded with multi-metric support, consistent execution scripts, and auto commits.

Takeaways

• The one-shot instruction to improve Polaris build time produced a non-building solution, while the metric-focused loop provided a clear acceptance test for every change.
• The loop found that VRT rebuilt work before Storybook recompilation and transformed 580 TypeScript component files when only 105 required it; removing those inefficiencies reduced build time by 65%.
• Autoresearch sometimes generated unacceptable shortcuts, such as removing many files, so successful metric results still required human review before retaining changes.

Related reading