Can a test prove two things are identical?

I have a (possibly irrational) fear of genetically modified/engineered foods. I thought I would disclose this before I plunge into this topic, agitated by this Washington Post article about the FDA stance that (1) does not require labeling of genetically modified foods; and (2) forbids labeling of non genetically modified foods as such.

***

How is this topic related to statistics? Consider how you might answer the question: is this new food going to harm us? Clearly, some form of testing is needed to compare the natural version and the genetically modified version. Are they the same or are they different?

According to the FDA, this is an open-and-shut case. Their scientists have determined that genetically modified salmon is not "materially" different from other salmon, by which they mean there are no "biologically relevant" variations between the two versions of salmon.

The level of confidence expressed in this conclusion is astounding and, I have to say, un-statistical. I seriously wonder if there are statisticians involved in this decision at all.

***

Notice that the scientists issued a qualified statement: no "biologically relevant" variations instead of no variations at all. What do they mean by "biologically relevant"? The article does not tell. 

In fact, "biologically relevant" is according to whatever metrics these scientists deem important to examine. Fat content, mercury content, etc. Presumably anything that they consider to have health implications. Therein lies the rub. This analysis presumes that we know exactly which parts of salmon are related to our health, and which parts aren't. In my opinion, there is no way to know because the genetically modified salmon is an innovation.

The general lesson is that tests won't tell us about things that they are not designed to measure. The design phase of testing is even more important than any subsequent analysis.

***

One wonders if a properly designed randomized controlled trial is being planned, and if not, why not. One branch would only eat natural salmon and the other branch would only eat modified salmon. Would we find differences in clinical outcomes?

The value of such trials is that if the clinical outcomes turn out to be different, we know something unexpected is happening, and we can go back and search for the biological cause of such a difference. Without such trials, our knowledge is limited by what dimensions the scientists attempted to test. But what we fear is not the usual suspects but the surprises.

***

Under the traditional statistical testing framework (discussed in Chapter 5 of my book), one would set up a hypothesis that the two versions are the same, and venture to seek significant evidence of difference to reject that so-called "null hypothesis". If insufficient evidence exists for rejection, the conclusion is that there is insufficient evidence to prove that the two versions are different. This statement is not the same as that the versions are identical.

There is a reason why we do not like to assert that two things are the same. The two things can be different along any number of dimensions -- a fish is a complex organism; one cannot possibly have exhausted all such comparisons.

One dimension was clearly not considered, namely, the genetic makeup of the salmons. Given that one version has been given a gene (which quickens its growth), it cannot possibly be the case that the two versions are identical. This difference is judged to be not biologically relevant by someone in a position of power.

***

Even if one trusts the FDA scientists, there is still absolutely no reason why marketers of natural salmon should be forbidden from advertising their fish as not having genetic modification. In the article, we were fed a bunch of legalistic, Orwellian mumbo jumbo about why this is so, none of which convincing. The last section of the article ("No 'Hormone free' either") is a must-read, and unintentionally hilarious.

If the scientific evidence is as clearcut as the FDA is saying, they should have no reason to fear a public rebuke unless they think the public is stupid. Withholding information and taking choice away from consumers are horrible ideas.

Comments

[Tom]

According to the LA Times, http://www.latimes.com/health/la-na-salmon-fda-20100921,0,5887640.story, the FDA's findings are only preliminary, and at least one FDA Advisory Committee is recommending additional study over concerns about the sample size of some studies.

Given the history of "biologically irrelevant" products (e.g. DDT), "safe" GMO (e.g. Monsanto's GM canola) and the complex interactions in a food source, I find it remarkable that the FDA could reasonably approve GMO salmon without performing several longitudinal studies lasting up to twenty years.

[Ken]

The assumption of course is that the salmon was OK to eat in the first place. Most foods have not been shown to be safe, simply because the sample sizes needed to show that they have at most a small effect on mortality would not be worth the investment. Showing that something didn't add more than say a 1 in 1000 per year risk would require large samples and large costs as everyone involved would need to be supplied with their own salmon.

Why do this when we know that the salmon doesn't have anything in it that wasn't in some other food. There are problems with GM foods but they relate to putting things into plants or animals that would not otherwise be eaten.

On the question of proving identical, not from a statistical perspective. What can be done is to show that the 95% confidence interval on the difference is within what we would describe as similar.

[Rosie Redfield]

I agree with Ken.

Given that we don't know of any reason why GMo salmon would be less safe, we expect any harm to be quite small. Because we can't control for the other stochastic risks in the study population, detecting such a risk would require a very large and expensive trial. The money would be better spent on interventions or trials with larger predicted payoffs.

The reasons that merchants want to put "not-GMO" labels on have nothing to do with enabling consumers to make informed decisions. Instead they (and the regulators) expect that such labels will sway uninformed consumers into paying a higher price.

[Kaiser]

Ken and Rosie: Thanks for offering up the other side of the argument. If one starts by assuming no harm, then there really is no point in testing anything.

In terms of labeling, either you believe that the "not GM" label is a lie or you believe that the "not GM" label leads customers to have incorrect perceptions. I think "not GM" is a statement of fact. A merchant is not responsible for what customers perceive the product to be. There are lots of foods that are labeled "no artificial coloring", or "100% juice" that also "misinform" consumers if we use the same logic. There is little scientific evidence that colors or <100% juice will lead to harm, and yet the FDA does not forbid those labels.

Even more generally, why do we allow Coke and Pepsi to label their sodas differently when consumers in taste tests cannot tell them apart? They are both carbonated sugar water. In fact, using this logic, we should just ban psychics, diets, nutritional supplements, luxury-brand fashion, Ferraris, etc. since to a larger/smaller extent, consumers who buy any of these things are uninformed.

As you can tell, I see no justifiable reason to ban labeling, except a kowtow to lobbyists.

[Ken]

Kaiser, I agree with you on the labelling: it should be allowed to make it clear how the product was or was not produced. After all, the point of using the GM is to reduce the cost. If I go into a supermarket and see two products and the cheaper is GM then I can make my choice base on price and prejudice. If I choose the non-GM, and in fact it doesn't give a benefit then I don't think the world will end because I made an irrational decision. The worst aspect of this, I expect, is that the aim of the business world is not to pass on the cost savings through GM.

[Daniel]

I bought some milk recently that said "Hormone Free*" and I looked for the asterisk to find the disclaimer, "FDA states: No significant difference in milk from cows treated with artificial growth hormones."

A claim of "significance" certainly implies some statistical method (dubious as that may be) but I'm happy they were allowed to write "hormone free" on the label. I'm with Kaiser on this one: if they're stating a fact, let the consumer decide how to interpret it. It's a sad condition when people are no longer allowed to make informed choices about their own food.

[Egg Syntax]

On a related note, Huffington Post presents a study (http://laudyms.wordpress.com/2010/04/22/genetically-modified-soy-linked-to-sterility-infant-mortality-in-hamsters/) in which GMO soy is linked to dramatic negative outcomes in hamsters. Any thoughts from a statistical perspective about this study?

[Kaiser]

Daniel: Good to know there is the asterisk solution to this problem. I'm fine with "Not GMO*".

Egg Syntax: On first look, the studies are standard and credible. However, they are not clinical trials that directly address the question of whether GM soy affects us or not. All the studies together form a reasonable basis to believe that GM soy has effects on us that natural soy doesn't. The studies are clinical trials on rats or observational studies on humans. One wonders if the FDA has a point of view on these studies.