Loading…
Topic
14 posts about Experimentation. Every summary links to the original.
See the idea
A/B testing trades traffic, time, and exposure risk for evidence. This explainer maps the benefit of a controlled comparison, the real cost of running one, and the condition that decides when measuring beats shipping on judgment.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Brian Balfour, VP of Growth at HubSpot, defines a growth team as distinct from product and marketing, focusing on getting the target audience to experience core product value quickly and often. He argues that the lines between marketing and product have blurred due to technical, data-driven marketing and accelerating growth speeds, necessitating a new team structure. Growth teams should not build core product value but rather optimize the funnel, such as onboarding, referral, and engagement, using a rapid experimentation process. Balfour warns against chasing vanity metrics and emphasizes that growth is a mentality and process, not just tactics, and that the right time to build a growth team is after clear product-market fit, typically around Series B. He recommends getting executive buy-in, educating across teams, and securing early wins with data to overcome skepticism and establish the growth team’s value.
A/B testing can help teams prioritize projects by forcing them to estimate effect sizes and required sample sizes before committing resources. The most common mistake is calling tests too early based on statistical significance without sufficient statistical power, leading to unreliable results. To run a test correctly, teams should estimate the expected KPI improvement, use a sample size calculator to determine test duration, and decide if the wait is worthwhile. For example, a complete signup flow redesign expected to boost conversion from 10% to 15% requires 24 times less data than a color change expected to lift it to 11%, and a non-significant result can still be useful. Startups with limited traffic should focus on large, high-potential changes to avoid working towards a local maximum, while larger companies with high traffic can profitably test small optimizations.
HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.
Auth0 Identity Conversion Suite addresses identity friction across the visitor-to-user journey, where forced account creation and complicated checkout are cited as contributors to abandonment. It combines Anonymous Sessions, which uses OAuth 2.0 access tokens to track pre-authentication activity and transfer session data into an authenticated profile, with Experiment Center, which supports controlled authentication-flow tests. Visitors can be assigned to percentage- or segment-based variants, kept in the same variant during their journey, and measured through exposure, success-rate, and time-to-auth events across supported Auth0 surfaces. The Early Access release is available to Enterprise-plan Auth0 customers and is presented as a way to improve personalization, onboarding, guest checkout, and authentication decisions without changing application APIs or code.
Resend welcomes Roberto Braga as its new Growth Marketer, adding a leader with more than 10 years of experience building and scaling technology businesses. He previously co-founded and served as CMO of YC-backed BX Blue, led growth through its acquisition by PicPay, ran organic growth at PicPay with SEO and ASO, and most recently was Reddit’s Country Growth Lead in Brazil. At Reddit, he helped double the platform’s DAU through content and partnership-led growth strategies; at Resend, he plans to create the Growth discipline and systematize the factors driving user acquisition and retention. Braga describes experimentation, channel mixing, email, and user conversations as central to his work, and advises growth marketers to focus on user pains rather than the newest tools or frameworks.
The Anthropic Economic Futures Research Fund commits $200 million to external research on interventions that could help society prepare for AI’s economic impacts. It prioritizes five areas: workplace integration, support for AI-driven transitions, income support for displacement, worker stakes in AI-driven growth, and evidence on public investments. The fund will favor large-scale randomized controlled trials, ambitious pilots, and program evaluations, with particular interest in projects that share findings at key milestones and can scale rapidly if promising; typical awards will be $5 million to $30 million, with nothing below $1 million funded directly. Eligible applicants include accredited degree-granting institutions, independent research and policy organizations, and nonprofits with experience running field experiments at scale, while the fund remains global despite US-centric examples.
A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test.
The Driver app is piloting a new offline heatmap on its home screen in select US cities. It shows where couriers can go online instantly and indicates demand levels intended to help them find trips more quickly. Each outlined delivery zone is color-coded red for high demand, orange for medium demand, yellow for low demand, or grey for areas open only to Platinum and Diamond couriers. The map refreshes every 10 minutes; couriers can tap a zone to see its name or navigate there, then see nearby busy restaurants and stores after going online. The announcement lists participating locations across the United States.
Uber describes how AI-assisted prototyping is changing product development by making interactive flows available within hours instead of weeks of coordination. In one example, a Merchant team customized a prototype to a specific catalog, producing immediate actionable feedback and rapid internal alignment. Across experiments, teams used Lovable, Figma Make, Claude Code, and Cursor to explore multiple concepts before PRDs and architecture decisions hardened; one PM explored six concepts in about 20 minutes. The pattern was earlier exploration, faster alignment, and clearer MVP boundaries, while PRDs and design reviews remained necessary for intent, tradeoffs, constraints, and durable decisions. Uber also opened prototyping to engineering and operational teams, while examining how prototypes should inform production without being mistaken for shippable solutions.
Etsy’s Ads Search ranking model needed a stronger purchase-intent signal because clicks are noisy and purchases are sparse later in the buyer journey. It retained its shared feature-representation and interaction layers, then replaced the shared feed-forward network with Multigate Mixture of Experts (MMoE), using task-specific gates and towers for click-through rate (CTR) and post-click conversion rate (PCCVR). Add-to-cart was added as an auxiliary task, while favorites were rejected after producing negative transfer. Offline, the resulting model increased Purchase and Click PR AUCs by average 3.5% and 1%, respectively, and Purchase ROC AUC by 0.5%. Online, it improved purchases and purchase calibration, supplied better inputs to auto-bidding, kept model size flat through pruning, and reduced inference cost.
Etsy describes how its experimentation platform evolved from CUPED to CUPAC to shorten the time needed to reach statistically powered results. CUPED uses pre-experiment visitor data as covariates in linear regression, while CUPAC first uses a nonlinear model to predict the outcome and then uses that prediction in the variance-reduction correction. Etsy trained and tuned LightGBM models on more than 100 pre-experiment features in Vertex AI, generating predictions through parallel Dataflow jobs and applying the correction with Spark within an Airflow-orchestrated pipeline. CUPED reduced variance by 7% on average, whereas CUPAC achieved 27%, shortening average experiment duration by almost three additional days, although results ranged from 2% to 77% depending on the metric and experiment; Etsy is balancing CUPAC’s impact against its training and maintenance costs.
At Etsy, Catapult supported only 13 hard-coded segmentations, forcing experimenters who needed other cuts to rely on slow, ad-hoc SQL analysis outside the platform. They replaced that fixed model with a Segmentation Library and creation tool where experimenters define reusable segmentations as SQL queries plus join columns connecting segment data to experimental units. Each definition is validated before production, then processed through a configuration-driven pipeline that detects new segmentations and generates independent tasks, improving failure isolation and allowing downstream work to wait only for required results. In the first three months, teams added 34 segmentations, increasing support from 13 to 47, while 78% of experimenters reported saving 2–5 hours monthly through custom analysis in Catapult.
Medium’s Daily Digest recommendations were less relevant than the homepage despite identical story sourcing, ranking model, and features, because digest filtering operated with less email-reporting information and was too aggressive. The “Sent in Opened Digest” filter excluded all 15 stories after an email-open signal, even when a reader saw only the first five; Apple Mail Privacy Protection also caused servers to preload messages, making many open events unreliable. Medium removed that filter and reduced the backoff from seven to four days, accepting more repetition while relying on other filtering rules and organic recommendation changes to diversify digests. An A/B test found that experiment users were 10% more likely to convert to paying members and read 7% more on Medium. The post says these results led to more filter investigations, including Bloom Filters, “Hard vs Soft Filtering,” and cross-digest diversification, while recommending audits of products that depend on email-open tracking.
Medium’s Daily Digest team investigated complaints that recommendations had become repetitive, especially for power users, after earlier filtering changes. Data showed some stories were sent up to eight times to one user, while affected users saw digest posts repeated twice on average; the team also had to account for Apple Mail Privacy Protection, which prevents reliable open tracking. They therefore tracked digest email IDs when readers clicked a story, marking emails with at least one click as “definitely opened,” and added a recommendation-stack filter that counts prior occurrences in those digests. Testing supported a threshold of one occurrence: complaining users would lose nearly five stories per digest, whereas free users would lose fewer than one. An A/B test produced a statistically significant increase in reading time, with a larger rise among members; the team reported no further complaints and suggested soft filtering and email-position rules as possible refinements.
The best engineering reads, once a week.
