---
title: "ShopGym: Realistic, reproducible sandboxes for shopping agents (2026) - Shopify"
description: "ShopGym converts live storefronts into self-contained, resettable sandbox shops for training and evaluating shopping agents, addressing challenges from changing products and bot detection. Its ShopArena layer separates exploration from generation using anonymized specifications and coding agents with execution-verification loops that repair failures without regenerating the whole shop. ShopGuru then synthesizes benchmark tasks across seven skill categories using rule-based and LLM generators grounded in each shop’s catalog, navigation, and policies. Validation over 224 tasks across six shops, three synthetic and three real, showed synthetic shops retain key structural properties and agent performance positively correlates with live storefronts when tested with BrowserGym and an internal system. The framework aims to provide reproducible environments for agents to fail and learn before interacting with live stores, with plans to expand to more retail domains and task types."
---

# ShopGym: Realistic, reproducible sandboxes for shopping agents (2026) - Shopify

[Shopify](https://yomu.fyi/company/shopify) · Mingyu Zhao · Oct 1, 2026

**Type:** Explainer

## Summary

ShopGym converts live storefronts into self-contained, resettable sandbox shops for training and evaluating shopping agents, addressing challenges from changing products and bot detection. Its ShopArena layer separates exploration from generation using anonymized specifications and coding agents with execution-verification loops that repair failures without regenerating the whole shop. ShopGuru then synthesizes benchmark tasks across seven skill categories using rule-based and LLM generators grounded in each shop’s catalog, navigation, and policies. Validation over 224 tasks across six shops, three synthetic and three real, showed synthetic shops retain key structural properties and agent performance positively correlates with live storefronts when tested with BrowserGym and an internal system. The framework aims to provide reproducible environments for agents to fail and learn before interacting with live stores, with plans to expand to more retail domains and task types.

## Context

Shopify developed ShopGym to address the difficulty of training and evaluating shopping agents due to changing products, prices, and store layouts, as well as bot detection and throttling that interrupt runs.

## Approach / What changed

ShopGym uses ShopArena to convert live storefronts into anonymized sandbox shops via a staged generation process with coding agents and execution-verification loops, and ShopGuru to synthesize benchmark tasks across seven skill categories using rule-based and LLM generators.

## Takeaways

- ShopGym separates storefront exploration from sandbox generation using an anonymized shop specification as the only interface between phases.
- Synthetic shops generated by ShopArena preserve key structural properties of live storefronts, including similar numbers of interface states and page complexity.
- Agent performance on synthetic shops positively correlates with performance on the live storefronts they mirror, especially on multi-step tasks.

## FAQ

### How does ShopGym ensure sandbox shops remain anonymous?

ShopArena's exploration agents omit brand names, product names, people's names, and absolute URLs when writing the specification, so every later stage stays anonymous without a separate redaction pass.

### What validation did ShopGym undergo?

The framework was validated with graph-based structural analysis and agent-based behavioral evaluation over 224 tasks across six sandbox shops, three synthetic and three real, showing structural fidelity and behavioral alignment.

### What types of tasks does ShopGuru generate?

ShopGuru creates tasks across seven skill categories, including single-skill tasks like finding a product or locating a return policy, and multi-step journeys that combine several steps like filtering, sorting, and adding to cart.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Benchmark](https://yomu.fyi/topic/benchmark), [Evaluation](https://yomu.fyi/topic/evaluation), [Retail](https://yomu.fyi/topic/retail)

- Source: [Shopify](https://shopify.engineering/shopgym)
- Source URL: https://shopify.engineering/shopgym
- Ingested by Yomu: 2026-10-02T18:00:14.211Z

[Read original post](https://shopify.engineering/shopgym)