10 Lessons Learned From Online Experiments

Summary

After completing ten online experiments over the past year, the author describes recurring statistical and implementation traps that can distort experiment results. The discussion stresses choosing independent randomization units, examining whether the analysis unit matches the randomization unit, and using simulations to test suspected false-positive effects before changing an experiment plan. It also shows how request-level traffic analysis and close collaboration with engineering can avoid unnecessary implementation work, while generous logging, A/A tests, and investigation of sample ratio mismatches can expose redirect failures or qualification bugs. The author reports that switching one experiment’s analysis unit from session to user changed its result from negative to positive, and argues for monitoring results without stopping early, investigating anomalies, and slicing averages across important user segments.

Context

Controlled experiments can be affected by traps, biases, and nuances that lead teams away from the true effect of a change. The examples address dependent sessions, mismatched randomization and analysis units, sample ratio mismatches, redirect behavior, user skew, qualification errors, and averages that conceal segment-level effects.

Approach / What changed

The author describes choosing user-level randomization when sessions could receive different experiences, simulating experiments to evaluate false-positive concerns, filtering traffic according to the experiment’s actual request scope, and using A/A tests and Kafka logging to validate redirect behavior. The examples also use ongoing result checks, code and qualification-logic investigation, and segment-level analysis to diagnose anomalies and interpret outcomes.

Takeaways

• Randomizing by session can expose a user to both experiences, allowing carryover effects to influence later sessions and violate the assumption of independent randomization units.
• A simulation using the population and planned user-level randomization found no higher false-positive rate when session conversion remained the primary analysis metric.
• An SRM investigation found that one group’s returning sessions were disqualified by a cookie; switching analysis from session to user changed the experiment result from negative to positive.

Related reading