Flow generation through natural language: An agentic modeling approach

Summary

Shopify built a tool-calling agent for Sidekick that converts natural-language requests into Shopify Flow workflows, addressing the cold-start problem by working backward from thousands of anonymized production workflows. The team generated synthetic queries and tool trajectories, fine-tuned Qwen3-32B, and changed Flow’s nested JSON DSL into an equivalent Python representation connected by a round-trip transpiler. Using Python improved syntactic correctness by 22 points and semantic correctness by 13 points on a 300-example benchmark, while matching production tool names, ordering, responses, and prompts proved essential to avoid degradation. Production evaluation exposed a 35% gap masked by benchmark parity, so Shopify built a weekly feedback loop that scores conversations, filters training data, identifies slice gaps, retrains, and deploys. The resulting agent serves most production traffic, runs 2.2 times faster at 68% lower cost, and outperforms closed models.

Takeaways

1. 1

   Representing Flow workflows as Python instead of nested JSON improved syntactic correctness by 22 points and semantic correctness by 13 points on the same training data.

2. 2

   Training and serving details such as tool names, tool order, response formatting, and system prompts materially affected accuracy even when the underlying task logic was unchanged.

3. 3

   The production feedback loop uses an LLM judge and tagged slice analysis to route high-quality conversations into training, quarantine low-scoring examples, and guide weekly retraining and deployment.