For this week's Statbusters (link), we opine on that astounding report from a few weeks ago about how Google could manipulate the next elections by biasing search results. We walk you through our vetting process, starting with face validity ("the magnitude of the reported effect is too large to be believed!").
The crux of the article is about the experimental design. You start with a group of people who have no prior opinion of the candidates (e.g. showing Americans Australian candidates), then give them only one source of data about those candidates, then check to see if the data affected their opinions. Besides, the manipulation was rigged for maximum impact - the base setting is that the entire first page of results favors one candidate.
As we said, the result of these experiments is still interesting but not as interesting as the breathless headlines proclaim.
One issue that didn't make it to the article but I should mention here is the ethics of one of the experiments, which sought to manipulate a real-life election in India. Recall the outcry surrounding the Facebook experiments that supposedly manipulated the emotions of people via feed messages. I didn't think much of that controversy (link). However, I'm surprised there isn't a bigger outcry surrounding this Indian experiment, which manipulated political voting behavior. The researchers said the IRB accepted their argument that their sample size is tiny relative to the size of the Indian electorate. I am not comfortable with the concept that ethics is relative to sample size.
Read our Statbusters column here.