Today's short post is a tour of the imaginary Museum of Scholarly Misconduct that Andrew Gelman proposed here. If you've been reading his blog in the last few years, he's been dogged in his campaign to expose research fraud. Sadly, there is a lot more research fraud than we think. Academics are capable of doing the most questionable things. It's not just the junior professors trying to make a name; the senior, most decorated stars of the field have been tarred. (I don't have room in this post to discuss the other side of the issue, which is the closing of ranks when such frauds are exposed, and the complete lack of accountability. Also, how "peer review" fails repeatedly.)
Along the way, I'll tell you how I got hoodwinked too by a study that has now attracted infamy.
***
Our first objet d'art is "Shredder with the missing teeth" by Dan Ariely, Duke professor (formerly MIT). Readers of this blog probably know Prof. Ariely through his series of best-sellers, starting with Predictably Irrational and the Honest Truth about Dishonesty.
This shredder featured in one of his key studies on human dishonesty. He described the experiment in this interview (my bolding):
At the beginning of 2002, we started a series of studies called The Matrix Experiments. We gave people 20 simple math problems: each one was a matrix of numbers where people had to find two numbers that added up to ten. It was a simple enough exercise that anyone could do, but we didn’t give them enough time. At the end of five minutes, people had to put their pencils down and write on another piece of paper how many they solved correctly. They then put the original test paper in the shredder, so nobody would know the true number they had solved. They received $1 for each problem they claimed to have solved correctly.
What they didn't know is that we modified the shredder! We only shredded the sides of the page, whereas the body of the page remained intact. Over 40,000 people, from all walks of life, participated in The Matrix Experiments.
I decided to quote the entire two paragraphs and give him all the space to explain the experiment. It sounds like an ingenious experiment. If I had heard it for the first time, I'd have thought what a smart design.
Some years later, another researcher attempted to replicate the experiment. He and his graduate students simply couldn't find a way to "modify the shredder". They even enlisted a mechanical engineer and still they failed. They asked Ariely how he did it. Ariely left them a voice mail (which you can hear yourself here), in which he said get a regular shredder from Home Depot, and take out the teeth in the middle using a screwdriver. Needless to say, this instruction didn't pan out.
I own a "Home Depot shredder" and I'd be surprised if I could use a screwdriver to knock out its middle teeth.
What's always surprising in these suspicious cases is that the researchers cannot produce evidence to support their prior results. Does Ariely or his team not have any pictures of the shredders used? It's like they are their own best lawyers ("missing emails", "missing text messages" come to mind.)
Over the years, many of Ariely's studies have been found to be fraudulent. I blogged about his work before when he responded to one of the controversies by claiming that he just now learned the importance of examining one's data before analyzing them - something he apparently did't learn in many years of teaching at top-tier universities and publishing in top journals.
***
The second objet d'art is the "bottomless scorpion bowl" by Brian Wansick, Cornell professor at the famous Ag school.
While I didn't hear about the Ariely's shredder until it's been exposed, I remember very well my immediate impression when a conference speaker retold Wansick's experiment on portion sizes of food. I thought it was genius! For years, I used this example in some of my talks.
The scorpion bowl - for my non-American readers - is described in Wikipedia as "a communally shared alcoholic tiki drink served in a large ceramic bowl traditionally decorated with wahine or hula-girl island scenes and meant to be drunk through long straws." It's something you find in college bars all over America. So, Wansick's idea is to make a bottomless scorpion bowl, one that is refilled from the bottom as people drink through straws.
In the experiment, half the college students were led to one room with the bottomless bowl, and the other half to another room with the non-refilling bowl. After the party is over, the students were asked how much they have drunk, and surprisingly, people in both rooms gave similar answers (the detail escaped me but it was portrayed as a slam dunk effect).
This research proved that humans have a poor sense of how much food we ingest, and large portion sizes served in American restaurants are a major cause of American obesity.
It was just the perfect research soundbyte - similar in nature to Ariely's shredder that revealed people's dishonesty.
Fast forward to today. Most of the research that came out of Wansick's lab have been discredited. In many cases, he said he couldn't find the data. Some researchers have questioned the existence of this bottomless bowl (link). It seems hard to make - especially because the device has to react to the rate of consumption. Nevertheless, Wansick has made various videos and pictures of bottomless bowls, and even relayed stories of failed attempts, so maybe the lab did make such a device.
It doesn't have to be so hard to prove that a study was performed. That's why Gelman's idea of a museum is a good one. It doesn't even have to be just for Scholarly Misconduct. It can be a Hall of Fame of ingenious experiments, with a Wall of Shame for retracted studies.
***
Even more amusing. The scorpion bowl apparently is a whole-cloth invention of the conference speaker from whom I heard this story. The bottomless bowl in the experiment contained Campbell tomato soup, not cocktails, so it's a soup bowl, not a scorpion bowl. Presumably, tomato soup is boring and scorpion bowls are fun.
Even a skeptic like myself is sometimes not skeptical enough. Given that Wansick most likely gave out credit to his students for participating in his lab's experiments, serving alcohol during that experiment would seem to violate college rules. Maybe the speaker said they served "virgin" drinks during the experiment - I don't quite recall.
Comments