Sidekick's continual learning loop

Summary

Frontier models can accelerate an AI product launch, but their serving cost and latency can become impractical as usage grows, while their weights do not learn from production failures. Shopify’s proposed flywheel captures low-quality conversations, defines quality with a rubric and calibrated judges, improves the surrounding harness through autoresearch, and then converts repaired trajectories into supervised fine-tuning and GRPO training signal. A panel of frontier reasoning models critiques failures, an arbiter produces repair instructions, and unresolved cases go to expert annotators; gist compression then shortens the agent’s static prompt without measured judge-quality loss. In production, the GraphQL agent serves up to 2,000 requests per minute and reportedly achieves a 96% serving-cost reduction versus an estimated frontier-model cost, with lower latency and higher throughput. The approach is presented as a continual-learning system that turns production experience into model-weight updates.

Context

As usage grows, frontier models can become too slow and expensive to serve every request at scale. They are general-purpose and do not automatically learn from user corrections, rejected outputs, or recurring production failures, leaving product knowledge in prompts, retrieval examples, routing rules, and harness code while the model weights remain unchanged.

Approach / What changed

The flywheel defines quality with a rubric, calibrates focused judges against expert annotations and production behavior, and uses those judges to optimize the existing prompts, tool definitions, and orchestration through autoresearch. It then mines hard negatives from anonymized production traffic, repairs them with frontier reasoning models and expert annotators, distills successful trajectories through supervised fine-tuning, applies GRPO, and compresses the system prompt into learned gist tokens.

Takeaways

• The quality rubric scores completeness, execution, response quality, and safety; Cohen’s kappa on 25 independently annotated random samples is used to identify ambiguity before calibration.
• Gist compression reduced the GraphQL agent’s static system prompt from roughly 6,000 tokens to about 1,500 learned gist tokens, with no measured quality loss on the judge.
• At 350 requests per minute, gist compression reduced time-to-first-token by about 19% and end-to-end latency by about 38%, while increasing requests per second by about 16% on identical GPUs.

Related reading