Controlled Experimentation (or A/B testing) has evolved into a powerful tool for driving product strategy and innovation. The dramatic growth in online and mobile content, media, and commerce has enabled companies to make principled data-driven decisions. Large numbers of experiments are typically run to validate hypotheses, study causation, and optimize user experience, engagement, and monetization.
The popularity of controlled experimentation has given birth to several companies that specialize in offering tools and services to effectively setup, manage, and monitor large-scale tests. While the simplicity of this concept is disarming, there are several pitfalls that can stymie progress and occasionally lead to poor decisions.