The headline says it all. The Hollywood Reporter reported that "'The Witcher' Breaks Netflix Series Viewership Records -- with One Big Catch."
This should come as no surprise to my readers. It proves yet again you can use data to prove anything you want. Apparently, Netflix switched how it counts "viewers" in 2020.
Those not knee-deep in data work might not appreicate how measuring viewership can be so controversial. How hard is it to count how many people viewed a show?
In this case, Netflix announced 76 million households watched the series during the first four weeks of release. This seemingly simple statement includes a load of unspoken assumptions:
- Does each houshold have to watch all four weeks or one out of four weeks? (Nothing prevents one from requiring each household to watch two or three out of four weeks, either!)
- How much of each show must be watched in order to count as a "view"? (Hint: if the show is 60 minutes, absolutely no one will count 60 minutes. If you want to guess, select a number closer to 0 than to 60. Correct answer is disclosed below.)
- How many episodes must be be "viewed" to consider the series "viewed"? (The Witcher Season 1 has eight episodes, all released at the same time. I don't think you need a hint for this one.)
- Should each device and each sub-account be aggregated or not?
The above list isn't even comprehensive; there are even more subtle issues:
- What is being measured is not "viewing" but "streaming". The video player sends back regular signals to Netflix saying the player is still streaming a particular show. To measure actual viewing, the TV would have to track the viewers' eyes. (This is technically feasible although I think these companies aren't doing this.)
- Do repeat minutes count? If the rule is 10 minutes of at least two episodes, how do we deal with someone who watched 10 minutes of the same episode, twice?
- How to deal with fast forward, skipping, etc.?
***
As you look at those questions, think about whether there is a principled way to make these decisions. How does one argue that one out of four weeks is not a view but two out of four is? How does one argue that 10 minutes constitute viewing but 5 minutes don't?
It's pretty clear why you can use data to prove anything you want.
Still, what Netflix did is jaw-dropping. The previous definition of a view is "watching at least 70 percent of one episode of a series or 70 percent of a feature film". The new definition of a view - watching at least 2 minutes of one episode. You are reading this right; the 76 million households have "viewed" one Watcher episode for at least 2 minutes.
The first definition is already quite generous. The rule of the game is to get enough people to turn on the first episode, and it doesn't matter that they never return. Since Netflix has the famous recommendation engine, it's got the tool to drive traffic to the first episode of pretty much anything. The audience have to stick around for 70 percent of the running time, though.
How does Netflix justify dropping the bar from 70 percent of one episode to 2 minutes?
From the Hollywood Reporter:
Our new methodology is similar to the BBC iPlayer in their rankings based on 'requests' for the title, 'most popular' articles on The New York Times, which include those who opened the articles, and YouTube view counts," the company notes in its earnings report. "This way, short and long titles are treated equally, leveling the playing field for all types of our content including interactive content, which has no fixed length."
Netflix is right about Youtube. Youtube counts 30 seconds of run time as "viewing". Also, Youtube is autoplay by default. Once you start playing one video, it performs a version of "doom scrolling" so it keeps playing the next clip. I'm surprised people actually suggest that such "autoplay" views are not counted as views. Doing so would disqualify a ton of legitimate viewing as not views - and Youtube makes money from generating views!
The most objectionable part of the quote is "short and long titles are treated equally". According to Netflix, treating someone who watched 2 minutes of a 10-minute clip the same as someone who watched 2 minutes of a 240-minute movie demonstrates fairness.
You know it's a problem when the agreeable press not only noticed the data offense but also decided to stick it in the headline rather than burying it two paragraphs from the end of the article.
***
The issue of definitions is everywhere in data analysis. Covid19 tests and deaths. Flight delays. Mean tweets. Twitter users. Facebook video view time. Wind chill.
In Chapter 2 of Numbersense (link), I featured some doctors who believed that changing the definition of obesity will solve the obesity problem in the U.S.
P.S. [10/20/2020:
Mark Palko alerted me to his take on this news also. I think what he's saying is Netflix knows (of course) how much time each device "viewed" each minute of each episode, it just chooses not to publicize that data. Great point!
Here is another post by Mark that covers how Netflix uses "deceptive messaging and suspicious statistics".
]
Comments