Unlocking Faster Insights with Experimenter-Defined Segmentations

Summary

At Etsy, Catapult supported only 13 hard-coded segmentations, forcing experimenters who needed other cuts to rely on slow, ad-hoc SQL analysis outside the platform. They replaced that fixed model with a Segmentation Library and creation tool where experimenters define reusable segmentations as SQL queries plus join columns connecting segment data to experimental units. Each definition is validated before production, then processed through a configuration-driven pipeline that detects new segmentations and generates independent tasks, improving failure isolation and allowing downstream work to wait only for required results. In the first three months, teams added 34 segmentations, increasing support from 13 to 47, while 78% of experimenters reported saving 2–5 hours monthly through custom analysis in Catapult.

Takeaways

1. 1

   Experimenters define segmentations with SQL that supplies segment data and assigns segment values, plus join columns such as user IDs to connect that data to experimental units.

2. 2

   Dynamic task generation processes each segmentation independently, isolating failures and allowing downstream jobs to begin when their specific required segmentations are ready.

3. 3

   Within three months, supported segmentations grew from 13 to 47; 78% of experimenters reported saving 2–5 hours per month using custom segmentations in Catapult.