Does being underrated have statistical meaning? (I don't have an answer.)

While I was reading Eater's article on "NYC's Most Underrated Restaurants" (link), a thought keeps troubling me. Here, the editors conducted a poll of the readers to discover which restaurants are underrated.

That sounds innocuous enough but something is amiss. A poll measures the average opinion. An underrated restaurant is one in which the average opinion is purportedly wrong. Something about using an instrument for discovering the average opinion to disprove the average opinion doesn't sit right.

***

So I set about thinking how one might mathematically describe the idea of being underrated.

The most obvious choice seems to be a gap between an intrinsic rating and the average opinion of the rating. The intrinsic rating can never be measured, and we use polls to estimate its value (really, a set of values). The underrated restaurant receives a poll rating that is below its intrinsic rating.

Something doesn't sound right about that too. The intrinsic rating simply doesn't exist. There is no higher being to tell us which restaurant is better than which other restaurant.

So, perhaps we can compare a poll rating with a "population" rating. The population rating is defined as what the rating would have been if everyone who cares to rate restaurants is polled. But now the gap between the two ratings is just the error in the sample. If the sample size of the poll is large enough, the concept of underrated (or overrated) would dissolve.

(One could argue that the population rating is the intrinsic rating because if one must find a value for the intrinsic rating, a good choice would be the population rating.)

***

Maybe underrated is a function of a subset of the population. An object is considered underrated by a subset of a population if the rating by that subset is substantially lower than the population rating. Put simply, it's just a cliche of people, like readers of Eater, voicing their dissent against the average opinion.

But one can find such a subset in any sample since there must be people who rate a given restaurant lower than average. Can there ever be a consensus of an underrated something?

Let me know if you have any thoughts on this.

 

Comments

[Wayne]

I think John S is on the right track.

Think of a "critically acclaimed" movie: a highly visible set of reviewers say it's great, but the vast majority think it's overrated. Or think of the restaurant that's just starting out in a nondescript strip mall: those who bother to stop in love it, but the vast majority of people simply have a bad impression when they drive by based on the wrong things (location) rather than what a restaurant would generally be rated on (food, service, atmosphere).

So you could look for restaurants that have a fanatical following, but flags of poor location, etc, that would cause many people to unfairly downrate it. Or look for restaurants which are highly rated by highly visible/influential people who are not representative of the general customer base.

If you're talking under/over-rated to a particular subgroup and you're a member of that subgroup speaking to that subgroup -- say a blogger -- you'd simply find places that were highly rated by your subgroup, but not by the general population. The consensus of being underrated is not a general consensus, but would be with "everybody" you know. Like the Yogism, "No one goes there anymore, it's too crowded."

[Jonathan Baron]

Perhaps Prelec's "Bayesian truth serum" would be useful here. It gives greater weight to opinions that are "surprisingly common" given the predictions of others' opinions. I'm sure it would show something, but I'm not sure that this would correspond to being "underrated".

[Nick F.]

If we define underrated as "not rated (often) enough" as opposed to "not rated highly enough." I think we get at a truer statistical definition. This metric might look something like a discrepency between average yelp user rating, and the amount of people who have reviewed it, after we adjust for both neighborhood traffic and restaurant capacity.

[Kaiser]

Thanks for all your comments. I have summarized them in a new post here with some further thoughts.