Big Data, Plainly Spoken (aka Numbers Rule Your World)

How averages get usurped by policymakers

This WSJ reporter doesn't much like statisticians, growth charts, and CDC (link). Her headline is "Is Baby Too Small? Growth Charts Make It Hard to Tell." This is her lede:

Elias Thorsteinsson weighed 6 pounds, 11 ounces, when he was born in January 2010. That put him in the 25th percentile of U.S. babies, meaning 25% of babies were smaller than he was at birth. But he didn't gain weight as quickly as other infants and dropped to the 1st percentile on the pediatric growth chart when he was 6 months old...Elias gained weight rapidly once he started eating solid foods. His head grew even faster—to the 99th percentile on the head circumference chart. His pediatrician wanted to do an MRI to rule out brain abnormalities. But Elias was developing normally, so Mrs. Stebbins refused. "We'd been through the wringer with these percentiles," she says. "I said, 'I have a big head. His father has a big head. He'll grow into it.' "

Is this damned lies and statistics or is this calling everything that has four legs a pig?

She goes on to complain, among other things, the existence of CDC and WHO standards which are not identical, and the existence of pediatricians who don't "always take the time to fully explain what the growth charts mean". (my italics)

***

The history of growth charts (height and weight tables) is fascinating, and not widely known.

They were pioneered by actuaries rather than doctors. In other words, they were first created for commercial reasons -- to enhance corporate profits of insurers -- rather than for health reasons.

The first charts were descriptive. It merely gave the body shape for the "average" person of a certain age and gender. Eventually, these charts became prescriptive - people who are not average are rated as "overweight", "obese", "underweight", etc. It's a big difference between knowing you're 50% above the average and being told you have to slim down, which is to say, you should be average.

Now, it appears that the growth charts may become detached from reality. The reporter includes an unattributed quote from the CDC: "The CDC says it doesn't plan to adjust its charts because it doesn't want the ever-more-obese population to become the new norm." This sounds like the standards will now be detached from the data, and simply be whatever the CDC wants us to weigh.

 

PS. This is not an anti-CDC post. There is an interesting problem here relating to using indices. If we keep using average BMI as the "target", and we are failing to revert to the mean, then the "target" will keep moving.

 

Do to other as you would have them do to you

The New York Times wrote about how the "Big Data" industry is trying to transform education (link). This is amusing and creepy by turns.

All of these may be well-intentioned, but what strikes me is how unscientific the arguments are given in favor of these data-driven methods. You'd expect the same data-driven approach to be used to justify their new solutions but you find almost none of that.

***

For example:

Arizona State’s initial results look promising. Of the more than 2,000 students who took the Knewton-based remedial course this past year, 75 percent completed it, up from an average of 64 percent in recent years.

What does it mean by "completing" the course? Is completion the same as competence? How do we know that the students were comparable from year to year? What is the variability of completion rates from year to year? Were there any changes in the admission rules or criteria for completion in the course? Were there any changes in the contents of the course?

Where is the control group? Andrew Gelman has written a number of times about experimentation in education. It would seem like companies like Knewton should take the lead in this type of evidence-gathering.

***

Elsewhere:

Mr. Lange and his colleagues had found that by the eighth day of class they could predict, with 70 percent accuracy, whether a student would score a “C” or better.

I don't know what the distribution of grades is at this school (Rio Salado) but grade inflation in US colleges has generally moved most if not all grades to "C" or better so I'd consider a 70 percent accuracy predicting "C" or above to be poor. Also, the issue is not whether one can diagnose the cases but whether there is a solution that would improve the underperforming students' grades. That would depend on the reason for underperforming. There will be cases where the students deserve "C" or worse.

Reading the article, I feel like much more deep thinking is needed to figure out why we would want to change in these ways.

***

Change is not always good. I have been teaching a course at NYU for many years. About two or three years ago, the course evaluation form went online. It used to be that I'd dedicate 15 minutes of the last class handing out evaluation forms, leave the classroom, and dedicate a student to collect the forms and drop them in the mail. Now, students are reminded by email towards the end of the semester to fill out an online survey.

Not surprisingly, the number of students responding has plunged. It was almost 100% when it was filled in during class. Now it's rarely above 30%. In order to encourage higher response rates, the emails that go out to students (and faculty) have become more frequent and they start earlier and earlier in the semester. The first email that opens the survey window is now sent not long after the midpoint of the course. As a result, students can comment on the class based on half or two-thirds of the experience.

The nature of responses has also changed. I now see mostly extreme opinions. The people who care to write evaluations either love you or hate you. (The irony is that all students think they deserve an A, a standard they don't apply when evaluating professors.) Students who are in the middle don't bother to give feedback.

It is absolutely true that putting the form online is more efficient, saves class time, and creates a data source for future data mining but the quality of the data has drastically declined.

***
These all go back to the issue of measuring intangible things. It's very difficult to do right. See my related post here.

The causation creep

If you're reading this blog, you probably have heard that correlation does not imply causation. Apparently, health-beat reporters and medical researchers in peer-reviewed journals know this too. They often explicitly warn us that the studies only show correlation, and they couldn't explain why the results are as they are. All is well if they stop there. But they don't.

All too often, they succumb to "causation creep". All of a sudden, they interpret the results in a way that presumes causation.

Here are two particularly egregrious examples that appeared on the Vitals blog at MSNBC.

In the first example (link), we are told:

Researchers followed 397 children from pregnancy (sic) through their first year of life, and found that those living with dogs developed 31 percent fewer respiratory tract symptoms or infections, 44 percent fewer ear infections and received 29 percent fewer antibiotic prescriptions.

And then comes the disclaimer:

the researchers acknowledged that couldn't account for all such factors [other than living with dogs that can explain the finding], and noted that they found a correlation, not a cause-and-effect relationship.

Finally, causation creep happens:

For heathlier kids, get a cat or dog, study suggests.

This is a cause-effect statement, there is no getting around it. It's saying that if you get a cat or a dog, your kids will be healthier. If the researchers truly believe that they found a correlation, then drawing this causal conclusion is unconscionable. Chances are, they just thought this is a fun thing to say (or at least the headline writer thinks so), allowing causation to creep in.

PS. The first person who commented on the article suggests that perhaps families that own pets also spend more time outdoors (playing with pets) and it could be the outdoor exposure that causes the observed effect. We don't know why but one thing is for sure: families who keep pets are not at all the same as families who don't keep pets.

PPS. It may be cruel at this point to point out that the so-called "cat" effect is 2 percent fewer antibiotics, which amounts to nothing. Also, no urban families were present in this analysis.

***

The second example (link) is if anything worse.

We are told:

Katzmarzyk and colleagues analyzed information from five earlier studies involving more than 167,000 adults that looked at the link between sitting and risk of dying from any cause over the next four to 14 years... About 27 percent of deaths in the studies could be attributed to sitting, and 19 percent to television viewing, the researchers said.

The disclaimer:

The researchers noted their study assumed a cause-effect link between sedentary behavior and risk of dying, which further research should validate, they said.

Finally, a blossom of colorful statements all of which presume causation, which they assumed without proof:

Sit less than 3 hours a day, add 2 years to your life (sic)

Reducing the daily average time that people spend sitting to less than three hours would increase the U.S. life expectancy by two years (sic)

The study adds to a growing body of evidence suggesting that sitting itself is deadly. (sic)

***

These researchers should realize they are not doing Freakonomics. It's true Levitt and Dubner also sometimes succumb to causation creep, especially in the more casual pieces. But in the original book, where they described how abortion policy could have caused crime to fall, they went through a vigorous explanation to rule out many other possible explanations. None of the studies cited here (or many other similar ones) have the care that is required to claim causation. We should note that the abortion finding has been debunked - that just goes to show how hard it is to prove causation with data that is conveniently collected.

 

The look elsewhere effect

Carl Bialik, i.e. the Numbers Guy at WSJ, wrote a nice piece (link) trying to explain something that is very difficult to explain to a general audience... the notion of statistical significance. He discusses this in relation to the experiments that have supposedly proved the existence of the Higgs boson.

I won't repeat his entire piece here. I have these thoughts while reading the piece:

The physicists talk about LEE - the look elsewhere effect. In statistics, we call this "multiple comparisons". We are typically looking for something out of the ordinary, say the agent that causes an illness. But patients do fall ill by themselves. So when the event occurs, we have to determine if it is explained by something extraordinary or is it just a normal event? We want to reduce the chance of a false positive finding. The harder you look, the more likely we will discover something that turns out to be false.

In Chapter 2 of Numbers Rule Your World, I talked about epidemiologists using questionnaires to help unlock the source of e-coli infections. The theory is that if food X is the cause of the outbreak, then a much higher proportion of those patients who fell ill (the cases) should have consumed X than those people who did not fall ill (the controls). The key issue is what is food X? The "multiple comparisons" issue is that if the epidemiologists asked about every possible variation of all food items, to the depth of say Brand Y green-red spinach with 2-inch stems packed in 10-ounce clear plastic bags, then they run the risk of identifying the wrong culprit.

Imagine finding the average case and the average control and seating them on the same stage. We ask them how many eggs they have eaten last week. Case said 2 and control said 3. That's not a big enough difference to say eggs caused the case to get ill and not the control. We then ask about hamburgers. Case said 1 and control ate none. Imagine going from food to food to food. Eventually we will chance upon some food item in which one side ate a lot more/less than the other side. Does this difference prove that that food item caused the e-coli infection? Or is it that these two groups of people have certain differences in eating habits regardless of their e-coli status?

Still not convinced. Imagine finding the average person wearing a bracelet/wristband and the average person who isn't wearing one. Ask them about what food they ate and go through all the same food items. If the list is long enough, we will surely find something that differentiates them. Is the difference caused by the wristband? That's the danger of "look elsewhere".

You will notice that we don't have a "solution" for this problem. All we do is to make the standard of evidence tougher. We just accept that this is a workplace hazard.

As Carl pointed out, the medical context is in some ways diametrically opposite to the particle physics context. In epidemiology, data is extremely scarce, and we must look for very big differences to be comfortable about the result. In particle physics, we have lots of data that are generated in controlled experiments, and we are looking for tiny differences. This explains why the standard for accepting a finding is so much higher in physics than in medicine.

The other reason is that we believe that the laws of physics are immutable so even tiny deviations can disprove a theory. Biology (economics, psychology, etc.) does not have immutable laws - in fact, there are lots of causes acting together for almost anything, and when it comes to psychology (unless you don't believe in free will), even the same person is likely to act differently when presented with the same scenario twice. This is to say, small variations do not destroy such theories.