How to read survey results: chocolate milk edition

Apparently, the Washington Post decided to assist the dairy industry in its latest advertising campaign by publishing a weird survey result, which claims that some adults believe that chocolate milk comes from brown cows. (link)

Just start by thinking about the survey design. In order for people to express this opinion, the survey had to contain a choice of "brown cows." If they had done an open-ended survey, the result will likely be quite different.

The Innovation Center for the dairy industry, whose name is attached to the survey result, does not publish any data or anything else that can provide context to understanding this result. The only thing I can find is this page, which is labeled "infographic" but contains nothing of the sort.

I'm happy to see that this writer for Huffington Post, which also reported on the survey, posed some questions to the Innovation Center. The spokesman claimed that Edelman Intelligence conducted the online survey. "The center, though, was unable to provide a full copy of the survey. And when asked about the survey’s methodology, [the spokesman] said it was 'conducted online'.”

There are some other plausible explanations for the 7% who supposedly said they thought chocolate milk came from brown cows. For example:

• There weren't any other reasonable answers. Notably, in none of the reporting on this survey are we told what the correct answer was, or what the other choices were.
• There were bots or people that were just guessing. We do not know if the answer choices were randomized for each visitor, nor do we know how many choices were presented to each respondent.
• There were bots programmed to select a fixed choice. We do not know if the answer choices were fixed for each visitor.
• There were people offended by the idiocy of the "brown cows" answer who decided to usurp the survey.

We do not know whether the question is multiple-choice, and if so, how many answer options were displayed, or indeed if the survey was open-ended.

Nor do we know where they placed the survey, how they recruited respondents, whether these respondents truly represent American adults, etc. Responding to the Huffington Post request, they claimed that they had representation across all 50 states. That raises even more questions because they would have neede to run a stratified sample overweighting the small states if they only had 1000 respondents.

The Washington Post isn't exhibiting much statistical literacy when its reporter concludes: "If you do the math, that works out to 16.4 million misinformed, milk-drinking people." This reporter naively believes that the 1000 respondents are truly representative of all American adults. I also do not see where the Innovation Center discloses that they pre-select for "milk drinking" people.

So, don't believe this story unless they are willing to publish the data. This sort of thing gives survey research a bad name.

 

Comments

[Tim]

The milk that comes from brown cows is used to make chocolate milk. You take the milk from brown cows and mix it with chocolate syrup.

So, depending on how the survey was worded, it could have been very misleading.

[Ken]

There is a lot of this happening. Surveys quoted that don't have reports available, articles about medicine where they don't give the paper, and universities putting out press releases because something is presented in a conference but isn't available. It should be a good journalistic practice that the source is always publicly available.