A/B testing
A valid test starts with a hypothesis, primary metric, target population, sample-size plan, and stopping rule. Random assignment helps isolate the effect of the change from other factors. Teams should also monitor guardrail metrics and practical significance, not only statistical significance. Repeated peeking, overlapping experiments, biased exposure, or changing the metric after seeing results can produce misleading conclusions.
What is A/B testing?
A/B testing is a controlled experiment that randomly assigns users to two or more variants to estimate how a change affects a defined outcome.
Why it matters
A valid test starts with a hypothesis, primary metric, target population, sample-size plan, and stopping rule. Random assignment helps isolate the effect of the change from other factors. Teams should also monitor guardrail metrics and practical significance, not only statistical significance. Repeated peeking, overlapping experiments, biased exposure, or changing the metric after seeing results can produce misleading conclusions.
Example
Half of eligible users see the current checkout button and half see a revised label. The team compares completed purchases while monitoring errors and refunds.