Does being underrated have statistical meaning? (I don't have an answer.)

While I was reading Eater's article on "NYC's Most Underrated Restaurants" (link), a thought keeps troubling me. Here, the editors conducted a poll of the readers to discover which restaurants are underrated.

That sounds innocuous enough but something is amiss. A poll measures the average opinion. An underrated restaurant is one in which the average opinion is purportedly wrong. Something about using an instrument for discovering the average opinion to disprove the average opinion doesn't sit right.

***

So I set about thinking how one might mathematically describe the idea of being underrated.

The most obvious choice seems to be a gap between an intrinsic rating and the average opinion of the rating. The intrinsic rating can never be measured, and we use polls to estimate its value (really, a set of values). The underrated restaurant receives a poll rating that is below its intrinsic rating.

Something doesn't sound right about that too. The intrinsic rating simply doesn't exist. There is no higher being to tell us which restaurant is better than which other restaurant.

So, perhaps we can compare a poll rating with a "population" rating. The population rating is defined as what the rating would have been if everyone who cares to rate restaurants is polled. But now the gap between the two ratings is just the error in the sample. If the sample size of the poll is large enough, the concept of underrated (or overrated) would dissolve.

(One could argue that the population rating is the intrinsic rating because if one must find a value for the intrinsic rating, a good choice would be the population rating.)

***

Maybe underrated is a function of a subset of the population. An object is considered underrated by a subset of a population if the rating by that subset is substantially lower than the population rating. Put simply, it's just a cliche of people, like readers of Eater, voicing their dissent against the average opinion.

But one can find such a subset in any sample since there must be people who rate a given restaurant lower than average. Can there ever be a consensus of an underrated something?

Let me know if you have any thoughts on this.

 

Comments

[Kris]

Very good question. A similar idea exists in finance where people strive to find "undervalued" stocks. However, in finance there exists much more information that past stock price (i.e. past ratings). Seeing that most facts are known by everyone, insider information is very useful. (But regulated by the government, so don't do it)

I propose that a restaurant can be 'rated,' (i.e. valued) using a number of public metrics:

Age of Restaurant - (long lasting ones are probably better than new ones)
Rent of Building - (ambiance, price etc.)
Prices on Menu - (low price with high rents means we're getting a good deal and/or volume is high, indicating many people eat there)
# of different meats - (options, variety)
# of Vegetarian/Vegan options - (again options)
Demographics of the zip-code - (population etc.)

One thing that would be very good to discover is where the ingredients come from and how recently they were ordered. Luckily companies are becoming more open to their sourcing and it may be possible to ID that. I know of a local seafood company: http://www.goincoastalseafood.com/ that specifically tells the diners were the seafood came from and when it was caught. Could text mining of the menu + demographics give a better idea than 'ratings'?

Hmmmm maybe I'll get cracking on this idea.

[Datacooker]

When I rank schools or restaurants or towns for Boston/NY/Philly magazines, I usually fit a model and compare (for example) predicted mean/median house prices in a town with actual prices so the towns that have prices below the prediction are 'underrated' or 'best bang for the buck.'

I don't think a survey would be a good way to do this.

[Casey]

I would argue that "underrated" doesn't mean what it sounds like, "less well rated than it deserves to be," but rather the slightly different "less often mentioned or thought about than it deserves to be." You see this interpretation reflected in a lot of the responses at your link: "You know what people aren't talking about any more? Pig foot heaven..." "When it comes to sushi these days, you only hear about..." "Yes, they got a good review in the Times, but they've been ignored otherwise." It's not that people don't think they are bad restaurants, it's that they don't think of them at all.

Overrated thus seems to reflect a particular imbalance between popularity and quality, possibly focused on being popular among a subset of relative experts. While quality is one factor of popularity, especially among experts, so too is location, trendiness of cuisine, novelty, existing hype, and lots of other factors. The statistical meaning that I could imagine, then, would be something to do with number of reviews, blog posts, and tweets about a restaurant, or maybe even a questionnaire about restaurant recommendations, (perhaps along with number of bookings or lunch turnover for, say, a food cart) versus the rated quality of food among those places. You'd then want to look places with a quality consistently as high as the best, but a low outlier in terms of discussion frequency.

[Kenny]

I agree with Casey's definition of underrated. Asking people what restaurants they feel are "less discussed" eliminates well-known restaurants from the discussion. Therefore, the population of "all restaurants in NYC" is effectively reduced and your "average" opinion is therefore not a true population average.

[Avner Shahar-Kashtan]

One method of identifying underrated items is used by the AnimeNewsNetwork site. It maintains a rating of anime movies and TV shows based on user rankings, but normalizes the results based on the number of votes. If a show has fewer votes, it is pulled towards the average score for a show. The more votes, the more the score is the true mathematical average of votes. This serves to prevent a small but rabid fan base from pushing a show up, but it also serves to create the Underrated metric: a show that has few votes, but those votes are very high, is considered underrated. The premise is that this show isn't very well known, but very well loved by those that do.

Here's the link to the data:
https://www.animenewsnetwork.com/encyclopedia/ratings-anime.php#most_underrated

[Jerzy]

The population rating can't be the intrinsic rating.
If I *personally* think something's underrated, that means I believe it deserves a higher (intrinsic) rating than the current (population) rating, whether the latter's estimated from a poll or counted by a census.

So "underrated" can only have a consensus meaning if there is a consensus on how to measure the intrinsic rating. That can't be measured by the same census/poll used to get the population rating.

Instead, let's say everyone agreed that you can measure restaurant quality by taking a panel of trusted expert food critics and forcing them to reach unanimous agreement.
(Or say we all agreed "volume of food per dollar" was a good metric, or whatever.)
You could do this; then take a poll or census among the general population; and see if they differ.

As for sub-populations: If you can't get a general consensus on the metric to use, at least you could find subgroups who agree on a metric, and then say "people who use metric X think it's underrated but people who use metric Y think it's overrated."

(Also, great points above: it's underrated if it "deserves" to be rated high, but most people rate it low *or don't think about it at all*. The population poll would have to treat "I don't know, I never think about it" as a useful response category.)

I'm really not sure how to describe what the Eater poll is measuring.

[zbicyclist]

Two meanings come to mind:

Under-rated = under-hyped (more attention should be paid)

My personal rating is higher than the average rating

[Jonathan]

Can it be one of the options that is not the ideal choice from the subset of all ideal choices?

[The Econometrician]

Kaiser: Interesting question! I wrote up a few thoughts of mine at my blog, casualinference.com. (Shameless plug-but I'm hoping there are complementarities between our readership). I assume that there's a true ranking though, which helps a lot!

[John S.]

One way this could happen is if the rating measures the wrong thing, or is influenced by factors other than the most important ones. For example, a restaurant close to the city center could be rated higher than an identical place in an outlying suburb due to a halo effect from the surrounding area. The effect of sample size would also suggest that frequently rated restaurants have more accurate ratings than restaurants that are less frequently rated. If you could discover these hidden variables, you could possibly control for them and discover some restaurants that are underrated.