---
title: "2,000 robots walk into a shop: Simulated A/B testing (2026) - Shopify"
description: "Shopify’s SimGym addresses the difficulty of evaluating storefront changes when merchants lack enough traffic for conventional A/B tests, by simulating shopping sessions with LLM-guided robots. Each robot runs in a cloud Chromium browser with a persona, budget, and intent, while a counterpart tests the alternate theme; the system uses real page representations rather than a mocked DOM. To support bursty, sequential workloads with 89K–127K accumulated tokens per session, Shopify self-hosted gpt-oss-120b on 48 NVIDIA B200 GPUs, combining MXFP4 expert quantization, FP8 KV-cache, FlashInfer kernels, async scheduling, and guided decoding. Production measurements report 80K to 88K tokens/second per B200 from async scheduling, a 57% reduction in HTTP/gRPC overhead, and MIG experiments yielding about 20% lower latency and 12% higher daily throughput without quality regression; speculative decoding is validated but queued."
---

# 2,000 robots walk into a shop: Simulated A/B testing (2026) - Shopify

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Feb 27, 2026

**Type:** Benchmark

## Summary

Shopify’s SimGym addresses the difficulty of evaluating storefront changes when merchants lack enough traffic for conventional A/B tests, by simulating shopping sessions with LLM-guided robots. Each robot runs in a cloud Chromium browser with a persona, budget, and intent, while a counterpart tests the alternate theme; the system uses real page representations rather than a mocked DOM. To support bursty, sequential workloads with 89K–127K accumulated tokens per session, Shopify self-hosted gpt-oss-120b on 48 NVIDIA B200 GPUs, combining MXFP4 expert quantization, FP8 KV-cache, FlashInfer kernels, async scheduling, and guided decoding. Production measurements report 80K to 88K tokens/second per B200 from async scheduling, a 57% reduction in HTTP/gRPC overhead, and MIG experiments yielding about 20% lower latency and 12% higher daily throughput without quality regression; speculative decoding is validated but queued.

## Context

Conventional storefront A/B tests can require days or weeks to reach statistical significance, and smaller merchants may not have enough traffic for testing to converge. SimGym is intended to provide simulated test results even for stores with zero real traffic.

## Approach / What changed

SimGym sends persona-driven robots through a merchant’s storefront in cloud browsers, with each robot choosing actions such as clicking, scrolling, adding to cart, or leaving. A twin runs the same process on an alternate theme. Shopify self-hosted gpt-oss-120b and optimized its serving stack with NVIDIA/CentML, vLLM, FlashInfer, quantization, scheduling, caching, and GPU partitioning for the workload.

## Takeaways

- Input tokens account for 94% of SimGym’s token usage. A roughly 20% reduction in average LLM latency reduced cost per merchant run by about 10% and increased daily throughput by about 12%.
- Replaying SimGym’s production traffic shapes produced 11K tokens per second per H200 versus 57K per B200, a 5.2x speedup that supported choosing Blackwell hardware.
- Reducing reasoning effort cut session duration by about 75%, but error rates rose from 0.5–0.75% to 4.5–10.9% across trials, so the change was rejected.

**Tags:** [Deployment](https://yomu.fyi/topic/deployment), [LLMs](https://yomu.fyi/topic/llm), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/simgym)
- Source URL: https://shopify.engineering/simgym
- Ingested by Yomu: 2026-08-30T13:18:19.425Z

[Read original post](https://shopify.engineering/simgym)