Flow generation through natural language: An agentic modeling approach

Summary

Shopify built a tool-calling agent for Sidekick that converts natural-language requests into Shopify Flow workflows, addressing the cold-start problem by working backward from thousands of anonymized production workflows. The team generated synthetic queries and tool trajectories, fine-tuned Qwen3-32B, and changed Flow’s nested JSON DSL into an equivalent Python representation connected by a round-trip transpiler. Using Python improved syntactic correctness by 22 points and semantic correctness by 13 points on a 300-example benchmark, while matching production tool names, ordering, responses, and prompts proved essential to avoid degradation. Production evaluation exposed a 35% gap masked by benchmark parity, so Shopify built a weekly feedback loop that scores conversations, filters training data, identifies slice gaps, retrains, and deploys. The resulting agent serves most production traffic, runs 2.2 times faster at 68% lower cost, and outperforms closed models.

Context

Sidekick needed to generate Shopify Flow workflows from plain-English requests, but fine-tuning initially lacked production conversations because the feature had not yet been deployed. Flow’s native JSON-based DSL was also unlike the program-like formats common in model pretraining, and benchmark results did not fully predict performance with real users.

Approach / What changed

Shopify sampled validated production workflows, generated corresponding natural-language queries and ideal tool-call trajectories, and fine-tuned Qwen3-32B on the resulting synthetic dataset. It introduced Python as an intermediate representation, using a round-trip transpiler between Python and Flow’s JSON DSL, mirrored production tools and prompts in training data, and established a weekly loop for scoring conversations, filtering examples, finding coverage gaps, retraining, and deploying.

Takeaways

• Representing Flow workflows as Python instead of nested JSON improved syntactic correctness by 22 points and semantic correctness by 13 points on the same training data.
• Training and serving details such as tool names, tool order, response formatting, and system prompts materially affected accuracy even when the underlying task logic was unchanged.
• The production feedback loop uses an LLM judge and tagged slice analysis to route high-quality conversations into training, quarantine low-scoring examples, and guide weekly retraining and deployment.

Related reading