The New York Times publishes another advertisement as news

The New York Times strikes again. Here is another article which makes you wonder if it is a press release by companies selling personal DNA test kits, or is it serious science/health reporting?

The article claims that there are two new studies which support removing restrictions to the sales of personal DNA test kits to the public. It turns out the studies are far from convincing (the first one is utterly worthless while the second one is hard to judge based on what is reported). Also, the argument has nothing to do with science. Egregiously, the entire article contains not one statistic even though the argument is purpurtedly supported by "new research", "two separate studies of genetic tests".

***

First, let's look at the argument:

Most people say they'll pay for genetic tests even if the predictions are sometimes wrong, and most people don't seem to be traumatized even when they receive bad news.

Later, we are told that:

Most people [surveyed] said they'd be willing to spend money out of their own pocket for the test.

So the Times thinks that it is ok for medical tests to tell lies just because people don't care. And that if people are willing to pay for something, they should get it. You'd have expected to read about science but that's not what we get. For example, the entire article says not one word about how predictive DNA information is for various types of diseases. (I don't have first-hand knowledge but I suspect that for most diseases, genetics is only mildly predictive and thus, this data is likely to be worth little.)

***

When we think about the argument, we have to wonder whether the studies are done right. It's puzzling why people who pay hundreds of dollars for tests would not care that the tests tell lies.

There are two types of lies (Chapter 4 of Numbers Rule Your World): being told one is at a genetic risk for some disease when in fact one does not have elevated risk for it; and being told one is not at risk when in fact one has elevated risk for some disease. From the perspective of the buyer of this test, the first lie is much easier to detect than the second lie: if you're told you have risk of colon cancer, you can take a screening test to confirm whether you have it or not.

Any test, including these DNA test kits, can be tuned to allow more of the false positive errors, or more of the false negative errors. This also affects the perception of customers to the product since the costs of the two errors are asymmetric. So, we can't interpret these studies without knowing how they tuned the tests.

But we don't even need to go there. The first study has a list of glaring problems that should lead anyone to discount the findings immediately:

•  the participants are 100% "self-selected", people who make a choice to buy DNA test kits; recall that these people then tell us they like the DNA test kits. (The Times reporter noted this half way through the article but apparently this flaw did not stop him from using this study as a key source.)
• the study was done by the Scripps Translational Science Institute, whose mission statement says it "aims to replace the status-quo of one-size-fits-all-medicine with individualized health care that is based on the known genetic factors influencing health and disease and that takes advantage of advances in digital technology for real-time health monitoring."
• the test kits used in the study was sold at a "steep discount" to participants by Navigenics, a company that sells such tests. The reporter tells us that such vendors want to get doctors out of the way so that these tests could become less expensive to the patients but again, knowing this does not affect his opinion of the study.
• the study has no control group, or even any group for comparison at all. It looks at 2000 people who bought the product at a steep discount. (How did it get published in the New England Journal of Medicine?)

And those are just glaring flaws based only on the information available in the Times article.

***

What most disturbs me about this article is the language used by the reporter, who drops the pretense of being an objective observer. It is the voice of one who has an agenda; these are exactly the words that one might find in the marketing materials of STSI or Navigenics. Here are some samples:

Members of a national advisory committee concluded that personal DNA testing needed to be carefully supervised by experts like themselves.

What if the would-be guardians of the public overestimated the demand for their supervisory services?

Supervised? Experts like themselves? Would-be guardians? Demand for supervisory services?

In two separate studies of genetic tests, researchers have found that people are not exactly desperate to be protected from information about their own bodies.

Protected from information about their own bodies?

For now residents of New York still can't be trusted to buy these tests directly.

The paternalists argue that it's still unclear how to interpret some of these genetic tests.

Can't be trusted? Paternalists? 

And none of these quotes is presented as a quote of someone. So, pray tell whether the Times is a news organization or a billboard company selling ad space.

Comments

[DavidC]

Seems to me like it would be nice if the tests presented probabilities using Bayesian reasoning. Could even explain the reasoning! I don't think it would have to be too complicated for that.

Let people weigh the cost of false positives/negatives for themselves.

[HarlanH]

Direct-to-consumer SNP tests such as 23AndMe provide increased/decreased risk estimates as just that, posterior probabilities of risk given the customer's DNA. It's not correct to say 'any test, including these DNA test kits, can be tuned to allow more of the false positive errors, or more of the false negative errors. ' That's only true if the tests were saying "you have cancer" or "you won't get Alzheimer's." That is, if the risk estimates were being collapsed to binary predictions. But they're not saying that.

This isn't necessarily an argument that these tests should be sold to the public, or that the Times reported the story well. But your criticism is misleading in this particular respect.

[Kaiser]

HarlanH: Thanks for the comment; it makes me think a bit more about what I wrote, and also to read the Wiki entry on genetics testing. While it is not as straightforward as my sentence implies, I still think these DNA tests have errors. For example, the Wiki says "it is possible that the test missed a disease-causing genetic alteration because many tests cannot detect all genetic changes that can cause a particular disorder". Even the presence or absence of a given chemical is not likely to be error-free since trace amounts can always be found which would be interpreted as absence if under some threshold.

HarlanH/DavidC: I'm going to poke the Bayesian readers now. Specifying posterior probabilities is a way to skirt the problem but it doesn't solve the problem. In essence, you pass the ball to the end-user and it is now the decision-maker who makes the false positive or false negative errors. Someone who gets the DNA test result will want to make a decision on whether to seek medical help - that is a binary decision, and behind that decision is an implicit cutoff probability.

[Janos]

But behind that implicit cutoff probability are the costs of getting tested vs the costs of late diagnosis. These can depend both on the patient's preferences/resources and on the state of medicine for that particular disease; if you give the patient a +/-, you're implicitly including some assumptions about this information.

[Zubin]

You must not be familiar with John Tierney's work, he's a big time libertarian and former Op-Ed columnist. I don't think much of Tierney's views most of the time, but he is technically writing a column in the science section so by journo standards he's allowed to express his opinion.
Tierney wiki page: http://goo.gl/5rM7k

[DavidC]

@Kaiser: yeah, the user still has to decide what to do, but I generally have a knee-jerk view that informing people is good, and misleading them is bad. That's really the only reason for my comment.

[Ebenezer Scrooge]

Let me rephrase Zubin's comment less delicately:
In writing this column, John Tierney did not act as a ho. Instead, he acted as a freak.