The New York Times reported on a bit of controversy about the use of statistical techniques to prove the existence of ESP. Here's a profile of the researchers who claimed to have proven ESP, and here's an overview of the statisticians who disagree. The statistical discussion is rather technical, and Andrew Gelman has a good post on it. My view is similar to Andrew's: the real issue is how to use statistical methods to measure effects that are really small. Indeed, is this a solvable problem? If the "noise" in the data is much stronger than the signal we're trying to detect, is it possible to detect the signal with a reasonable level of accuracy?
***
You are not alone in noticing that Google search results have been invaded by spam. It is now often the case that the first-page results consist of nothing but commercial websites, spam websites, and unreliable content created by several companies set up specifically to game Google's algorithms. Google is either losing the battle with spammers or content to make money from whomever.
See Paul Kedrosky's pieces here and here. Unlike Paul, I don't think curation is a standalone solution; it is not scalable to the multitude of possible searches. Smart algorithms with human input is our only option. This is in its essence the same problem as the email spam problem.
Here is a summary of three articles on this subject.
In Chapter 2 of Numbers Rule Your World, I briefly mentioned why it is a horrible idea for regulators to force banks to reveal their credit-scoring algorithms. Doing so would lead to the kind of gaming that has harmed Google's search algorithms.
Comments