Summary of A/B test course by Google on Udacity (Part 3)

  • Ethical considerations: Since actual people are involved in the experiment as experimental units, it’s very important to give careful considerations to the ethics of the experiment. Some ethical considerations are risk, benefit and privacy. If the risk exceeds the threshold for the minimal risk, i.e. it encompasses physical, psychological, emotional, social or economic concerns, then getting an informed consent becomes crucial. If the users would benefit post completion of the study, then stating the benefits is important. If the internal processes for collections of new data are well in place, then privacy won’t be a huge issue, but if not, additional safety measure would be needed.
  • Variability of metric: the choice of unit of diversion can greatly impact the variability of a metric. The variability of the metric is much higher if the unit of diversion is broader as compared to the unit of analysis. Unit of analysis is denominator of metrics. If click-through-probability metrics is defined as no of clicks/no of pageview then no of pageview is unit of analysis. Analytic & empirical estimation will be close if unit of diversion and unit of analysis be the same. If unit of diversion and unit of analysis is different, it makes the estimated analytical variability is much higher than empirical one so you might want to move to use empirical one. It is because when you actually compute the variability analytically, you assume about the distribution of data and what is considered to be independent. You draw randomly and whether they are independent or not. When you use event-based diversion, the single event is a different random draw so your independent assumption is valid. When you use cookies or user-id diversion, that independence assumption is no longer valid because you are diverting group of events. And so they are actually correlated.
  • Examine user retention
  • Want to increase user activity
  • Anything requiring user to be established
  • How long and how often users see the changes.
  • For safety reason as above refer to, as we make bigger change, it potentially lead to bigger risk and it might be harder for users to adapt.

Feral cat trying to make sense people and the world around