---
title: "Flow generation through natural language: An agentic modeling approach"
description: "Shopify built a tool-calling agent for Sidekick that converts natural-language requests into Shopify Flow workflows, addressing the cold-start problem by working backward from thousands of anonymized production workflows. The team generated synthetic queries and tool trajectories, fine-tuned Qwen3-32B, and changed Flow’s nested JSON DSL into an equivalent Python representation connected by a round-trip transpiler. Using Python improved syntactic correctness by 22 points and semantic correctness by 13 points on a 300-example benchmark, while matching production tool names, ordering, responses, and prompts proved essential to avoid degradation. Production evaluation exposed a 35% gap masked by benchmark parity, so Shopify built a weekly feedback loop that scores conversations, filters training data, identifies slice gaps, retrains, and deploys. The resulting agent serves most production traffic, runs 2.2 times faster at 68% lower cost, and outperforms closed models."
---

# Flow generation through natural language: An agentic modeling approach

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Apr 22, 2026

**Type:** Explainer

## Summary

Shopify built a tool-calling agent for Sidekick that converts natural-language requests into Shopify Flow workflows, addressing the cold-start problem by working backward from thousands of anonymized production workflows. The team generated synthetic queries and tool trajectories, fine-tuned Qwen3-32B, and changed Flow’s nested JSON DSL into an equivalent Python representation connected by a round-trip transpiler. Using Python improved syntactic correctness by 22 points and semantic correctness by 13 points on a 300-example benchmark, while matching production tool names, ordering, responses, and prompts proved essential to avoid degradation. Production evaluation exposed a 35% gap masked by benchmark parity, so Shopify built a weekly feedback loop that scores conversations, filters training data, identifies slice gaps, retrains, and deploys. The resulting agent serves most production traffic, runs 2.2 times faster at 68% lower cost, and outperforms closed models.

## Context

Sidekick needed to generate Shopify Flow workflows from plain-English requests, but fine-tuning initially lacked production conversations because the feature had not yet been deployed. Flow’s native JSON-based DSL was also unlike the program-like formats common in model pretraining, and benchmark results did not fully predict performance with real users.

## Approach / What changed

Shopify sampled validated production workflows, generated corresponding natural-language queries and ideal tool-call trajectories, and fine-tuned Qwen3-32B on the resulting synthetic dataset. It introduced Python as an intermediate representation, using a round-trip transpiler between Python and Flow’s JSON DSL, mirrored production tools and prompts in training data, and established a weekly loop for scoring conversations, filtering examples, finding coverage gaps, retraining, and deploying.

## Takeaways

- Representing Flow workflows as Python instead of nested JSON improved syntactic correctness by 22 points and semantic correctness by 13 points on the same training data.
- Training and serving details such as tool names, tool order, response formatting, and system prompts materially affected accuracy even when the underlying task logic was unchanged.
- The production feedback loop uses an LLM judge and tagged slice analysis to route high-quality conversations into training, quarantine low-scoring examples, and guide weekly retraining and deployment.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

- Source: [Shopify](https://shopify.engineering/fine-tuning-agent-shopify-flow)
- Source URL: https://shopify.engineering/fine-tuning-agent-shopify-flow
- Ingested by Yomu: 2026-08-30T13:18:00.952Z

[Read original post](https://shopify.engineering/fine-tuning-agent-shopify-flow)