Big Data, Plainly Spoken (aka Numbers Rule Your World)

The Musk sampling plan, thought through

Elon Musk likes to stir things up, and last Friday he claimed Twitter exaggerated its user statistics by counting "spam" accounts. Of course, Musk has made a bid to buy the social-media company, and he may just be looking for a discount.

He posted a tweet asking for ideas of how to estimate the proportion of spam/fake accounts on Twitter.

Later he suggested "ignore the first 1,000 followers, then pick every 10th".

Let's play this out a bit. When I'm done, you'll realize why sampling is a profession.

***

Each twitter account has a user name (mine is @junkcharts), and no two users can have the same user name. Each user has two basic counts: followers and following. The typical user has more following than followers - that would be true of most content consumers. If one is a content producer, the opposite is hopefully true: more followers than following. Elon Musk, for example, has over 90 million followers but he only follows 100 or so accounts.

So what happens when the first 1,000 followers are ignored, as Musk stipulated? Presumably he believes the earlier followers of any account are unrepresentative. It's not clear this exclusion does what he intends. The exclusion rule won't affect his own account as 1,000 is a rounding error. The set of Musk followers - with or without the first 1,000 - is essentially the same.

Applying this exclusion to other people's accounts may be problematic because 1,000 followers is a pretty high bar. 2838, 70, 133, 564, 3432, 608, 16K, 10K, 982, 4648. Those are the follower counts for the first 10 accounts shown on Musk's feed. Half of those accounts have fewer than 1,000 followers. After removing the first 1,000 from these smaller accounts, there is no follower left to pick from. The next three accounts have between 3000-5000 followers; excluding a third to a fifth of their followers from sampling seems severe. Thus, the Musk sampling plan introduces a large-account bias.

***

The second step of the Musk sampling plan is described as pick every tenth follower. So, take Musk's 90 million followers minus the first 1,000, and we'll get a gigantic sample of 9 million.

Here comes the hard part. What he wants to know is the proportion of those 9 million accounts that are spam accounts. Who is going to decide whether each of the 9 million accounts are "spam", and how?

Nine million is too big a sample to handle. He may be applying the "rule of thumb" that a random sample should be 10% of the population. That's a myth busted in any intro Stats class. We typically only need to interview 1,000 Americans to generalize the sample responses to the entire population of 300 million, which is far, far, far, far smaller than 10%.

The required sample size is not a fixed number. If one can tolerate a larger margin of error, the sample can be reduced. What's our tolerable margin of error? If the true proportion of spam accounts is 5%, as Twitter management asserted, we may want a rather precise estimate, within plus or minus 2%. Reviewing Stats 101, we learn that translates to a standard error of 1%, and a sample size of 475.

What that means is if we take a random sample of 475 of Musk's followers, and learn that 5% (24) of those are spam accounts, then we can conclude that 5% of his followers are spam accounts, plus or minus 2%. (Others may disagree.)

It would be much easier to verify 500 accounts than 9 million.

***

Perhaps Musk didn't intend to pick every 10th follower until the list of 90 million is exhausted. What he might have in mind is to pick every 10th until the required sample size is reached. Let's say we want a sample of 500, so we stop the random sampling procedure after 500 picks, or having gone through 5,000 followers.

Can you see what's wrong with this? In effect, rather than being randomly picked from the entire list of followers, the 500 all came from a narrow slither of 5,000 followers, precisely followers #1001 to #6001 after expelling the first 1,000. But... the list of followers is ordered not at random, but chronologically. Thus, all 500 followers in the sample would have started following Musk during a specific narrow time window.

Musk appears to be imagining that he has a list starting with the earliest followers through to the most recent. (In practice, it may be easier to obtain the reverse chronological list starting with the most recent.)

***

Notice that the sampling plan involves two steps: first sample from all users, then within each selected user, sample from followers. As stated, the Musk plan is incomplete, as we have no idea what he plans to do for step 1. Aside from his own account, what other users does he include, and how many?

Even if the first step involves a random sample from all users, this unbiased sample is later tainted by the exclusion of the first 1,000 followers, because it flushes out small accounts from the sample.

The first step is unlikely to be a simple random sample; some intentional bias may be inserted. Like most social-media platforms, I'm surmising there is a highly skewed distribution of followers. Maybe 10 percent of the biggest accounts attract 90 percent of all followers. Thus, a simple random sample will contain lots of small accounts with few followers, not an ideal situation given the exclusion rule that subsequently flushes out small accounts. (Bias is not always a bad thing!)

As explained above, a sample size of 475 followers gives a good estimate of the proportion of spam accounts among Musk's followers; I'd be hesitant to generalize it to other twitter accounts. That's because Elon Musk's is no ordinary user.

Combining the spam proportion in Musk's account with similar in other selected accounts is tricky. If intentional bias is used in the first step, then a weighted average is required. Weighting does not cure the large-account bias though.

The two-step sampling seems unnecessary. Why not just sample users directly? After all, a follower must be a user.

***

The above considerations form a starter's list.

Here are further topics:

• If one wants a random sample, then every user should have equal probability of being picked. Such a sample is likely to contain a majority of small accounts.
• Some users are lurkers. They have few followers.
• Some accounts have been abandoned but not deleted.
• Newer accounts have fewer followers and following.
• One person can have multiple twitter accounts. Spammers probably have a higher average number of accounts than humans
• The more accounts a user follow, the more likely the user will appear in the random samples.
• A follower may be one of the first 1,000 followers of Elon Musk and thus excluded but the same person will be post-1000th followers of other accounts and included if randomly selected.
• Are all bot accounts "spam"? What is the definition of "spam"?
• All social media have tons of spam accounts. Is it better to answer the inverse question of how many humans use Twitter?

Reading Statistical Analytics Illustrated

A few months ago, Prof. Kottemann sent me a copy of his new book, "Statistical Analysis Illustrated: Foundations You Should Know" (link). On a recent flight, I finally found time to give it a quick read.

The book covers a central section of frequentist statistical theory, which forms the core of most traditional Statistics 101 courses. This is the sequence that runs through random sampling, sampling distributions, law of large numbers, central limit theory, confidence intervals, and hypothesis testing (tests of significance).

Kottemann adopts an approach that deemphasizes mathematical derivation, and delists any theorem while favoring verbal and visual explanation. This is an approach that I also prefer in introductory statistics courses so I flip the pages eagerly.

With the stated goal of helping readers appreciate a "way of thinking" about data, Kottemann improves upon most textbooks in the following manner:

1. He concentrates on the case of sample proportions (e.g. x% of respondents felt this way), which can be easily appreciated given their common usage in surveys. This use case avoids one of the tricky turns that confuse statistics students, namely, learning the difference between the population standard deviation, the sample standard deviation and the standard error.
2. He smartly abandons the usual bottoms-up presentation of the central sequence. In the first chapter, he goes straight into sampling distributions, confidence intervals and significance testing without having defined a random variable, a normal distribution, the law of large numbers, the standard error or the null hypothesis (all of which will appear in later chapters).
   
   
3. The visuals in the book are improved versions of typical textbook diagrams as he removed unnecessary and confusing details.

This central sequence of frequentist statistics has proven challenging to generations of students because it's not a simple, straightforward argument. It's not easy because generalizing from samples of data is a complex undertaking. Kottemann's scope is narrower than the typical Statistics 101 textbook but what he covers is essential.

Most of the writing by Kotteman is lucid. The complexity of the materials - when explained in words - only got the better of him on a few occasions. Above Figure 1.1, he writes "[This histogram] shows the frequency (out of 1,000) with which we should expect to get random samples that yield the various possible values for the percent-in-favor sample statistic." I quote it only to illustrate the type of tongue-twisters statistics teachers run into when covering this material (and so he may rephrase it in the next edition); the sentence is an outlier in a very readable book.

The hypothesis testing framework has been under constant attack from other "ways of thinking". In the last chapter of the book, Kottemann briefly covers the Bayesian way of thinking. I hope readers appreciate that everything is different, from which question is answered to what assumptions are made to how decisions are framed.

The frequentist viewpoint has been widely abused. Journal editors reject any studies that do not reach a 95% confidence level, effectively ordaining publication bias, and motivating scholars to "p-hack," which is a general term that means engineering the "p-value" to hit 95%. In practice, people may "d-hack" which is to review a large array of outcomes, and select the one that happens to reach 95%; or they may "n-hack," which is to make the sample size so big that even tiny signals are declared statistically significant. I might even add "ps-hack" which is to pre-specify a large array of outcomes to cover up "d-hacking". 

If there is one gap in Kotteman's presentation, it's in the transition from the case of proportions to "the rest of the (frequentist) iceberg". If the reader is asked to do a test of difference in means at the 10% significance level, I'm not sure s/he can figure it out.

I'd also like to see a few worked-out examples of analyzing real-life opinion polls or market research surveys, and some exercises.

If you're learning statistics, and having trouble with the central sequence from random sampling to statistical testing, or if your sole interest is understanding polls or surveys, give Kottemann's book a try. It may help unblock you.

More case-counting puzzles

The Johnson & Johnson vaccine is being treated as a third-class citizen due to some safety concerns (link). Interestingly, such concerns did not surface in the "final" analysis of the single-shot Covid-19 vaccine, published in February, 2022 in New England Journal of Medicine (link). In this report, the scientists concluded "Ad26.COV2.S was associated with mainly mild-to-moderate adverse events, and no new safety concerns were identified". Since in the interim report, the FDA said "the analysis supported a favorable safety profile with no specific safety concerns," we were supposed to believe that the J&J vaccine is completely safe.

On the efficacy side, the final report headlined 56%. This number is meaningless unless we know how VE is defined. The 56% number comes from a counting window that starts 14 days after the shot (1D+14). During the interim report, the headline number was 67% (link).

However, these numbers are not comparable for reasons I'll get into in this post. The FDA has allowed pharmas to keep shifting the goal posts so nothing can really be compared like for like. The situation illustrates clearly why statisticians won't give you an answer unless we know all the details about how things have been measured!

***

First, the interim report was conducted when only about half of the participants have reached 2 months of follow-up - note that the largest number on the time axis was 126 days.

In the final analysis, about half the participants have reached 6 months of follow-up - the largest number on the time axis is now about 280.

So, the comparable value from the final report is not the headline VE which concerns up to 280 days of follow-up but the VE at day 126... and that number is 55% (1 - (436-100)/(938-185)). According to Table 2B (final report), the error bar around this number is 20% to 70% so the trial did not enroll enough participants to give a precise estimate of VE.

Harmonizing the follow-up period is not sufficient. Between the initial and final analyses, J&J ditched the requirement that all cases be "centrally confirmed". In the interim report, cases were counted only if a central laboratory confirmed the local PCR test positives. In the final report, cases accrued based on local test results.

***

Another major development is letting the placebo group get vaccinated. After J&J's vaccine was granted EUA at the end of February 2021, all participants reaching 6 months of follow-up were effectively unblinded so that those in the placebo group could get the vaccine if they so chose. Almost everyone did.

Since enrollment ended October 2020, and the data cutoff for the final analysis occurred in July 2021, everyone could have been followed for at least 6 months, if the above plan were implemented to the letter.

But that was not what happened. Some participants were unblinded prior to reaching the 6-month point because they requested getting one of the by-then authorized Covid-19 vaccines.

As a result, only 23% of the study participants had an observation window of at least 6 months. (By design, 100% should.) In fact, half of the participants in this trial had a follow-up period of less than four months. Because the follow-up period was truncated by the placebo participants receiving the vaccination, effectively destroying the placebo group, there will be no further useful data.

Refer back to the case curve shown above, notice that the at-risk population has dropped by more than half by day 126. The further out one went in time, the fewer participants contributed to the incremental case numbers, the wider the error bar. Such dropoff does not happen in a normal clinical trial. 

Those who reached the longest follow-up period are also the earliest enrollees in the trial. In most vaccine trials, the pharmas first enrolled the lowest risk people - in the case of J&J, the earliest batch of enrollees were young people with no comorbidities. So the right hand side of the case curve lacks both validity and reliability.

***

When I wrote about the J&J trial, I pointed out that it was one of the few double-blind trials - the Pfizer and Moderna trials were single-blind. Blinding removes a source of potential bias - a kind of self-fulfilling prophecy in which the investigators or the participants behave in a way that advantages the treatment against the placebo.

We now have solid evidence that blinding did not work well, and confirmation bias was a real concern. To begin with, given that about 30% of Americans have obstinately refused to get the vaccine, we know that this subset is not represented in these vaccine trials. When the placebo groups in these trials were offered the vaccine, almost all of the participants took advantage, implying they were vaccine enthusiasts, not skeptics.

Blinding is supposed to neutralize partisan feelings about vaccines but it didn't work as planned. In Figure S1 (in the appendix), we learned that 1,222 placebo participants "received other vaccine before unblinding", compared to 670 on the vaccine arm. That's 80% more on the placebo arm than the vaccine arm. If the treatment groups were properly randomized, and if participants were properly blinded, they would not know whether they received the placebo, and so the proportion of people who got another vaccine against trial rules should be balanced.

The number of participants who withdrew during the trial was also disproportionately higher on the placebo arm (338 vs 224), about 50% more. These are huge gaps.

***

One of the big mysteries of Covid-19 vaccine trials is what counts as a case. The case-counting window is one variable I've discussed often. But there's more. Most of these early trials were designed to measure symptomatic, PCR-confirmed cases. J&J added a definition of mild, moderate and severe cases, although as I pointed out before, the definition of "mild" disease is meaningless since almost every case has been classified as moderate or severe.

In most studies, including the J&J trial, all participants are given serological tests at regular time intervals. Seropositivity indicates that someone has been infected in the past, and thus, it is presumed that said infections were asymptomatic. This device has been used by scientists to claim the vaccines also prevented asymptomatic infections.

Apparently, J&J now believes that someone without a positive result from either PCR or serological testing can also be infected. This is how I interpret one of the footnotes from Table 1 that corresponds to a concept called "any SARS-CoV-2 infection", which includes:

"undetected cases that were subsequently detected through a positive serological result (according to the clinical severity adjudication committee), which did not count as either symptomatic cases (becqause they were RT-PCR-negative) or asymptomatic cases"

I'm not sure how the adjudicators got wind of the "positive serological result" since if such a result was present, the case should have counted as "asymptomatic", which is a different line in Table 1.

(Also, the severity adjudication committee is a misnomer as the committe members concerned themselves here with asymptomatic infections that would be milder than "mild".)

What is shocking about this line of numbers is the magnitude. There were 1,038 such infections in the vaccine group and 1,699 in the placebo group. These two numbers are much higher than the counts that made it to the headline VE calculation. In particular, only 42% of the vaccine infections, and only 52% of the placebo infections, ended up as "cases". This calculation also shows that an "any" infection on the placebo group has a higher chance of being counted as a case than an "any" infection on the vaccine group.

 

 

 

 

 

 

 

 

 

Know your data 32: location data from mobile apps

Vice discovered that one can spend $160 to buy information about people who have visited abortion clinics. Let's unpack this situation.

Is the data accurate?

The reporter also learned that among the results returned on a search of "Planned Parenthood", not all such clinics offer abortion services. Someone who buys this dataset thinks everyone counted on the list have gone to abortion clinics but in reality, only a portion of those have.

This illustrates one of the problems with today's data marketplace. Incorrectly matched data frequently get traded. These errors are unlikely to be corrected because correcting them usually reduces the size of the search results, which reduces the potential revenues that can be earned on them. Besides, the "victims" of inaccurate data have no clue that such data have been collected, or that such data have been assigned to them, or that such data have been used against their interests.

I don't think the vendor allows, or should allow, a more targeted search such as "abortion clinics" or "Planned Parenthood abortion clinics". So the inaccuracy may also be a deliberate business decision.

Abortion is a highly sensitive subject, and we may want to switch to something less emotional when thinking about the underlying issues. Facebook banned searches based on targeting college students. But if I were marketing to college students, I can buy location data capturing anyone who visits the campus bookstore, or the pizzerias, etc. I'm almost surely going to get a good coverage of the student population, but I will also get "noise": professors, parents of students, visitors, prospective students, etc. In marketing, the cost of missing a potential customer is typically much higher than the cost of spamming.

Who wants the data?

In the case of abortion clinics, the obvious, alarming use case are vigilantes in states empowered by recent laws to hunt down women who have had abortions.

But there are always two sides to a story. Abortion clinics themselves might desire this data for market research purposes, and pro-abortion researchers might like to analyze such data.

The entire surveillance capitalism industry missed a golden opportunity during the Covid-19 pandemic to shine a light on the positive aspects of data collection. They could have offered up location-based data for the common good. Such data would have been very useful to characterize the contact rates at local levels, and such public health use cases require only aggregated data, unlike marketing or law enforcement use cases, for which individuals must be identified.

The particular dataset acquired by Vice article is at the census block level, which means it shows, for example, which census block people who visited a specific Planned Parenthood location live. This is aggregated data but not really. The census block is quite precise, especially in less dense areas. There are over 10 million blocks in the U.S. In a city, it resembles a city block. When we filter by time and space, we may end up with one or only a handful of individuals in a block. Then, most likely, we just need one other data element to fully recover people's identity.

Don't believe this? Just pick five names from your contact list. Can you find one thing that is unique for each of these five people? Perhaps only one likes sports, perhaps only one is a Buddhist, perhaps only one speaks a foreign language, perhaps only one is over 50 years old, etc.

Why is the data so cheap?

Big Data is not scarce data. Any company collecting your location data can sell such datasets. Who's collecting your location data? If you have a iPhone, just go to Privacy/Location Services and see which apps are "using" your location data. That would be the starting point.

Apps that require location, like maps, navigation, share rides, and weather, obviously have your location data. But any app you install can collect your location data. Some years ago, Yelp - the app about restaurants - was discovered to be retrieving your location continuously throughout the night (a good way to learn where you live.) Your location at any time is one value but that same value is collected by lots of companies who then resell it to lots of other companies.

Then read the Data notice of those apps, and you'll learn that there is a buzzing marketplace in which lots of third parties piggyback on these apps to get access to your data. They repackage the data and sell them.