Summary of A/B test course by Google on Udacity (Part 1)

Common threat for A/B testing validity — Novelty effect. Novelty effect is the tendency for performance to initially improve when new technology is instituted, not because of any actual improvement in learning or achievement, but in response to increased interest in the new technology (Wikipedia).

The binomial distribution is a probability distribution that summarizes the likelihood that a value will take one of two independent values under a given set of parameters or assumptions. Three underlying assumption of binomial distribution include: 2 types of outcomes, independent events (mutually exclusive) & identical distribution (same probability).

Binomial distribution curve. Source: from course's video
Binomial distribution curve. Capture from course's video.

Note that: The experiment result might be statistical significant but not practical significant. That is why the launching of experiment need to check with business standpoint.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store