Loading post…
ShopifyMingyu ZhaoExplainer
ShopGym: Realistic, reproducible sandboxes for shopping agents (2026) - Shopify
Summary
ShopGym converts live storefronts into self-contained, resettable sandbox shops for training and evaluating shopping agents, addressing challenges from changing products and bot detection. Its ShopArena layer separates exploration from generation using anonymized specifications and coding agents with execution-verification loops that repair failures without regenerating the whole shop. ShopGuru then synthesizes benchmark tasks across seven skill categories using rule-based and LLM generators grounded in each shop’s catalog, navigation, and policies. Validation over 224 tasks across six shops, three synthetic and three real, showed synthetic shops retain key structural properties and agent performance positively correlates with live storefronts when tested with BrowserGym and an internal system. The framework aims to provide reproducible environments for agents to fail and learn before interacting with live stores, with plans to expand to more retail domains and task types.
Context
Shopify developed ShopGym to address the difficulty of training and evaluating shopping agents due to changing products, prices, and store layouts, as well as bot detection and throttling that interrupt runs.
Approach / What changed
ShopGym uses ShopArena to convert live storefronts into anonymized sandbox shops via a staged generation process with coding agents and execution-verification loops, and ShopGuru to synthesize benchmark tasks across seven skill categories using rule-based and LLM generators.
Takeaways
ShopGym separates storefront exploration from sandbox generation using an anonymized shop specification as the only interface between phases.
Synthetic shops generated by ShopArena preserve key structural properties of live storefronts, including similar numbers of interface states and page complexity.
Agent performance on synthetic shops positively correlates with performance on the live storefronts they mirror, especially on multi-step tasks.
FAQ
How does ShopGym ensure sandbox shops remain anonymous?
ShopArena's exploration agents omit brand names, product names, people's names, and absolute URLs when writing the specification, so every later stage stays anonymous without a separate redaction pass.
What validation did ShopGym undergo?
The framework was validated with graph-based structural analysis and agent-based behavioral evaluation over 224 tasks across six sandbox shops, three synthetic and three real, showing structural fidelity and behavioral alignment.
What types of tasks does ShopGuru generate?
ShopGuru creates tasks across seven skill categories, including single-skill tasks like finding a product or locating a return policy, and multi-step journeys that combine several steps like filtering, sorting, and adding to cart.
Source and freshness
- Source
- shopify.engineering
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.