On page 8 of Numbersense (link), I wrote:
Web logs are a messy, messy world. If two vendors are deployed to analyze traffic on the same website, it is guaranteed that their statistics would not reconcile, and the gap can be as high as 20 or 30 percent.
Insiders will nod their heads; for those who aren’t familiar with Web data, take a look at this recent post on The Verge about the metrics of Web traffic, whimsically titled "Yahoo is bigger than Google, but Google is bigger than Yahoo." This post is a reaction to the recent announcement that Yahoo sites have surpassed Google sites in terms of traffic--well, according to one way of counting.
Note that Web companies like Google, Yahoo!, and Facebook live and die by these numbers. All these companies offer “free” services to users, and they generate revenues by selling (out) these users to advertisers. Advertisers want “reach,” which is typically measured by such metrics as unique visitors and unique visits. When you hear people make the claim that digital advertising is more “measurable” than traditional advertising, they are referring to this type of metrics.
The Verge reporter compares three measurement services: Comscore (the dominant player), Compete and SimilarWeb. These services could not agree if Google or Yahoo! was bigger or if they were the same size. Moreover, their numbers differ by orders of magnitude.
Read between the lines, and you realize that these services don’t agree on the definitions of those metrics either. What is a unique visit? Conceptually, a visit is a browsing session. In reality, it’s not easily nailed down. Say, you go home from work and start watching a movie on Netflix. Half an hour later, your Mum calls, and you pause the movie. Another 30 minutes later, you finish the call, and continue watching the movie. If you ask me, I’d say only one Web visit should be counted. On the other hand, if after the call, you abandoned your movie, and checked your email, I’d argue you started a second unique visit.
The reality is Comscore, etc. does not have a CCTV installed in your home. All they have is the Web log, which has an entry for every page you viewed and the time you went there. How much idle time should occur before one declares a new visit? If you set this to 30 minutes, then in the example above, you’d have counted two unique visits. If you set it to 180 minutes, you’d have counted only one visit.
If you believed the hype that online businesses are inherently more measurable, think again. They may be measurable in the sense that you can more easily collect data, but the data is only approximate, and vary wildly by who’s collecting it.
PS. I’m not against subjectivity in measurement. In fact, I believe every metric has subjective elements. I’m just saying you should find out how things are measured. This is one of the key takeaways from Numbersense (link).