How Web Analytics evolved by adopting terms and concepts from other disciplines and making them their own
Despite all the things we do not know we usually maintain that there is a standard of quality to which we can hold our data, and that by this standard we can judge the results we arrive at based on this data. This idea is known as validity.
1. In statistics, the validity of a measurement tool is considered to be the degree to which the tool measures what it claims to measure.
2. In logic, an argument is valid if and only if it takes a form that makes it impossible for the premises to be true and the conclusion nevertheless to be false.
There is a very boring side to validity, and that is the technical stuff. All our pages need to be tagged, sample sizes need to be determined correctly and so on and so. Apart from the fact that we cannot really recognize our users, or know how long they stayed on our page and so on we have data collection pretty much nailed down. The interesting things happen when we take multiple observations from our data and arrive at, hopefully valid, conclusions. There is, however, sometimes a problem with the way we are doing this. I will demonstrate this with one of our favourite marketing tools, remarketing.
Remarketing is meant to re-engage a user who has looked at your products before. He is marked with a third party cookie and a tracking tag sends additional data – was this a category page or a product page, a highly priced or cheap item. If the user buys the item the cookie is removed. If not he remarkting vendor displays ads on third party websites that show the item previously viewed.
To reduce the theory behind this to its absurd minimum: users (supposedly) show their interest in your products by not buying them. And the more information they get before not buying it the larger their interest (supposedly) is. This theory is backed up by a seemingly valid conclusion:
Premise: We do some remarketing
Premise: There is some uplift in Revenue*
Conclusion: Remarketing works
(* there usually is some uplift if you start throwing money around)
But are we sure that our web analytics tool really measured what it was supposed to measure ?
Because that people do not buy your stuff because they like it so much is not the only possible explanation. There is another theory that would explain the observation of people not buying your merchandise, and that is that they do not like your stuff and do not want to have it. In fact we could rephrase one of our premises and still arrive at a valid conclusion:
Premise: We penetrate people with BS
Premise: There is some uplift in Revenue
Conclusion: BS works
This is not an idle concern. Doing remarketing right is really hard. Your remarketing pixel might be implemented on the wrong pages, or send the wrong data, or people might share computers so you do not have clean datasets and so on and so on.
This really happened:And even if you do everything right your ads have to compete with a lot of advertising, so are you sure your visitors will recognize your carefully crafted signal among all the noise ? Maybe it’s just enough to maintain a strong presence and to hope that users will eventually click something.
The point is not (just) to make fun of remarketing. The point is that we have, or had, a built-in bias that let us favour hypotheses that support opinions we held in the first place. To come to a really valid conclusion you always have to test multiple competing hypothesis to see if you have actually measured what, as an Analyst, you have claimed to measure. After all you want to attribute your success to the correct cause.
Despite our best efforts there is a certain fudge factor built in
Always test multiple conflicting hypotheses to arrive at valid conclusions