Carl Bialik, i.e. the Numbers Guy at WSJ, wrote a nice piece (link) trying to explain something that is very difficult to explain to a general audience... the notion of statistical significance. He discusses this in relation to the experiments that have supposedly proved the existence of the Higgs boson.
I won't repeat his entire piece here. I have these thoughts while reading the piece:
The physicists talk about LEE - the look elsewhere effect. In statistics, we call this "multiple comparisons". We are typically looking for something out of the ordinary, say the agent that causes an illness. But patients do fall ill by themselves. So when the event occurs, we have to determine if it is explained by something extraordinary or is it just a normal event? We want to reduce the chance of a false positive finding. The harder you look, the more likely we will discover something that turns out to be false.
In Chapter 2 of Numbers Rule Your World, I talked about epidemiologists using questionnaires to help unlock the source of e-coli infections. The theory is that if food X is the cause of the outbreak, then a much higher proportion of those patients who fell ill (the cases) should have consumed X than those people who did not fall ill (the controls). The key issue is what is food X? The "multiple comparisons" issue is that if the epidemiologists asked about every possible variation of all food items, to the depth of say Brand Y green-red spinach with 2-inch stems packed in 10-ounce clear plastic bags, then they run the risk of identifying the wrong culprit.
Imagine finding the average case and the average control and seating them on the same stage. We ask them how many eggs they have eaten last week. Case said 2 and control said 3. That's not a big enough difference to say eggs caused the case to get ill and not the control. We then ask about hamburgers. Case said 1 and control ate none. Imagine going from food to food to food. Eventually we will chance upon some food item in which one side ate a lot more/less than the other side. Does this difference prove that that food item caused the e-coli infection? Or is it that these two groups of people have certain differences in eating habits regardless of their e-coli status?
Still not convinced. Imagine finding the average person wearing a bracelet/wristband and the average person who isn't wearing one. Ask them about what food they ate and go through all the same food items. If the list is long enough, we will surely find something that differentiates them. Is the difference caused by the wristband? That's the danger of "look elsewhere".
You will notice that we don't have a "solution" for this problem. All we do is to make the standard of evidence tougher. We just accept that this is a workplace hazard.
As Carl pointed out, the medical context is in some ways diametrically opposite to the particle physics context. In epidemiology, data is extremely scarce, and we must look for very big differences to be comfortable about the result. In particle physics, we have lots of data that are generated in controlled experiments, and we are looking for tiny differences. This explains why the standard for accepting a finding is so much higher in physics than in medicine.
The other reason is that we believe that the laws of physics are immutable so even tiny deviations can disprove a theory. Biology (economics, psychology, etc.) does not have immutable laws - in fact, there are lots of causes acting together for almost anything, and when it comes to psychology (unless you don't believe in free will), even the same person is likely to act differently when presented with the same scenario twice. This is to say, small variations do not destroy such theories.