Big Data, Plainly Spoken (aka Numbers Rule Your World)

Are 81 percent of Elon Musk's twitter followers fake?

If we believe the analysis by two companies in the business of detecting fake twitter accounts, around 81 percent of Elon Musk's twitter followers are "spam" or "fake" (link). With 93 million followers, that is a shocking 75 million "spam"/fake accounts, enough to keep the anti-fake crusader Musk awake at night.

But what the analysis really tells us is that counting "spam"/fake accounts is a highly subjective exercise. Remember when I said in my previous post:

Here comes the hard part. What he wants to know is the proportion of those 9 million accounts that are spam accounts. Who is going to decide whether each of the 9 million accounts are "spam", and how?

We have two companies eager to tell us what "spam" is. Here are some of what they included in the 75 million spam/fake accounts:

• "inactive" accounts that have not tweeted in the past 90 days (67 million, 71% of Musks's followers)
• accounts using Twitter's default profile images (up to 24 million, 26%)
• accounts that have a "suspiciously small number" of followers (up to 78 million, 83%)
• accounts on an "unusually small number" of lists (up to 89 million, 95%)
• accounts using "spam-correlated keywords" in profile descriptions (up to 68 million, 73%)
• accounts specifying a location that is not a known place name (up to 66 million, 71%)
• some number of accounts that are "protected" (i.e. private): these accounts cannot be analyzed by data analysts since the data are locked up. In the binary classification of fake or not fake, the private accounts ended up on the fake side of the ledger

What's the problem? Many of us will disagree with how they define a spam/fake account.

By their own admission, a bot account that automatically shares frontpage posts from the Hacker News website counts as "fake". I doubt Musk wanted to get rid of this type of accounts. Some twitter users (lurkers) may never tweet and they count also as not authentic. I doubt users who don't tweet or retweet are responsible for spreading fake news.

The modelers claim that they are "conservative", "We intentionally biased to missing some fake/spam accounts rather than accidentally marking any real accounts incorrectly." So they claim that the algorithms almost never classify real accounts as "spam"/"fake".

That's a bit hard to believe. Consider a typical human lurker, someone who is a content consumer but not a content producer. They may have used a fake location, their accounts most likely would not be on anyone's list, they likely have very few followers, they may have retained Twitter's default profile image, etc. etc. They may therefore accumulate enough signals to be classified as spam.

***

But let's take their word and assume that the chance of a false positive is close to zero. The chance of a false negative - classifying a spam/fake account as human - is 35%, which they disclosed. This means, for every 100 spam accounts, 35 would be called human. So... the number of spam accounts is actually more than reported, since some of the spam accounts are erroneously labeled humans. I'm not sure which analysis has this false negative rate. I'm going to assume it's the one in which they applied the model to only the active Musk followers (i.e. 27 million rather than the full 93 million).

The model asserts that 23% of active Musk followers are spam accounts. Since they claim true humans are not misclassified as spam, then all 6 million are true spam accounts. But 35% of spam accounts are misclassified as humans. So the 21 million "humans" comprise of true humans plus misclassified spam accounts. 35% * # spam accounts + # humans = 21 million. Thus, # spam accounts = (6)/(1-35%) = 9 million.

In other words, 9 out of 27 million of active Musk followers are spam accounts, out of which 6 million are found by the algorithm, while 3 million are erroneously classified as humans. Thus, the total number of accounts classified as humans is 18 + 3 = 21 million, as expected.

If we adjust the model results by the claimed false negative/false positive rates, the total number of real Musk followers is 18 million out of 26.8 million active accounts (67%), or 18 million out of 93.4 million (19%) if inactives are considered spam and no humans are inactive.

The report authors concluded that 23% of the active Musk followers are spam but they failed to adjust for false negatives. After adjustment, it's 33% spam among actives (and almost 80% of total).

 

 

 

The Musk sampling plan, thought through

Elon Musk likes to stir things up, and last Friday he claimed Twitter exaggerated its user statistics by counting "spam" accounts. Of course, Musk has made a bid to buy the social-media company, and he may just be looking for a discount.

He posted a tweet asking for ideas of how to estimate the proportion of spam/fake accounts on Twitter.

Later he suggested "ignore the first 1,000 followers, then pick every 10th".

Let's play this out a bit. When I'm done, you'll realize why sampling is a profession.

***

Each twitter account has a user name (mine is @junkcharts), and no two users can have the same user name. Each user has two basic counts: followers and following. The typical user has more following than followers - that would be true of most content consumers. If one is a content producer, the opposite is hopefully true: more followers than following. Elon Musk, for example, has over 90 million followers but he only follows 100 or so accounts.

So what happens when the first 1,000 followers are ignored, as Musk stipulated? Presumably he believes the earlier followers of any account are unrepresentative. It's not clear this exclusion does what he intends. The exclusion rule won't affect his own account as 1,000 is a rounding error. The set of Musk followers - with or without the first 1,000 - is essentially the same.

Applying this exclusion to other people's accounts may be problematic because 1,000 followers is a pretty high bar. 2838, 70, 133, 564, 3432, 608, 16K, 10K, 982, 4648. Those are the follower counts for the first 10 accounts shown on Musk's feed. Half of those accounts have fewer than 1,000 followers. After removing the first 1,000 from these smaller accounts, there is no follower left to pick from. The next three accounts have between 3000-5000 followers; excluding a third to a fifth of their followers from sampling seems severe. Thus, the Musk sampling plan introduces a large-account bias.

***

The second step of the Musk sampling plan is described as pick every tenth follower. So, take Musk's 90 million followers minus the first 1,000, and we'll get a gigantic sample of 9 million.

Here comes the hard part. What he wants to know is the proportion of those 9 million accounts that are spam accounts. Who is going to decide whether each of the 9 million accounts are "spam", and how?

Nine million is too big a sample to handle. He may be applying the "rule of thumb" that a random sample should be 10% of the population. That's a myth busted in any intro Stats class. We typically only need to interview 1,000 Americans to generalize the sample responses to the entire population of 300 million, which is far, far, far, far smaller than 10%.

The required sample size is not a fixed number. If one can tolerate a larger margin of error, the sample can be reduced. What's our tolerable margin of error? If the true proportion of spam accounts is 5%, as Twitter management asserted, we may want a rather precise estimate, within plus or minus 2%. Reviewing Stats 101, we learn that translates to a standard error of 1%, and a sample size of 475.

What that means is if we take a random sample of 475 of Musk's followers, and learn that 5% (24) of those are spam accounts, then we can conclude that 5% of his followers are spam accounts, plus or minus 2%. (Others may disagree.)

It would be much easier to verify 500 accounts than 9 million.

***

Perhaps Musk didn't intend to pick every 10th follower until the list of 90 million is exhausted. What he might have in mind is to pick every 10th until the required sample size is reached. Let's say we want a sample of 500, so we stop the random sampling procedure after 500 picks, or having gone through 5,000 followers.

Can you see what's wrong with this? In effect, rather than being randomly picked from the entire list of followers, the 500 all came from a narrow slither of 5,000 followers, precisely followers #1001 to #6001 after expelling the first 1,000. But... the list of followers is ordered not at random, but chronologically. Thus, all 500 followers in the sample would have started following Musk during a specific narrow time window.

Musk appears to be imagining that he has a list starting with the earliest followers through to the most recent. (In practice, it may be easier to obtain the reverse chronological list starting with the most recent.)

***

Notice that the sampling plan involves two steps: first sample from all users, then within each selected user, sample from followers. As stated, the Musk plan is incomplete, as we have no idea what he plans to do for step 1. Aside from his own account, what other users does he include, and how many?

Even if the first step involves a random sample from all users, this unbiased sample is later tainted by the exclusion of the first 1,000 followers, because it flushes out small accounts from the sample.

The first step is unlikely to be a simple random sample; some intentional bias may be inserted. Like most social-media platforms, I'm surmising there is a highly skewed distribution of followers. Maybe 10 percent of the biggest accounts attract 90 percent of all followers. Thus, a simple random sample will contain lots of small accounts with few followers, not an ideal situation given the exclusion rule that subsequently flushes out small accounts. (Bias is not always a bad thing!)

As explained above, a sample size of 475 followers gives a good estimate of the proportion of spam accounts among Musk's followers; I'd be hesitant to generalize it to other twitter accounts. That's because Elon Musk's is no ordinary user.

Combining the spam proportion in Musk's account with similar in other selected accounts is tricky. If intentional bias is used in the first step, then a weighted average is required. Weighting does not cure the large-account bias though.

The two-step sampling seems unnecessary. Why not just sample users directly? After all, a follower must be a user.

***

The above considerations form a starter's list.

Here are further topics:

• If one wants a random sample, then every user should have equal probability of being picked. Such a sample is likely to contain a majority of small accounts.
• Some users are lurkers. They have few followers.
• Some accounts have been abandoned but not deleted.
• Newer accounts have fewer followers and following.
• One person can have multiple twitter accounts. Spammers probably have a higher average number of accounts than humans
• The more accounts a user follow, the more likely the user will appear in the random samples.
• A follower may be one of the first 1,000 followers of Elon Musk and thus excluded but the same person will be post-1000th followers of other accounts and included if randomly selected.
• Are all bot accounts "spam"? What is the definition of "spam"?
• All social media have tons of spam accounts. Is it better to answer the inverse question of how many humans use Twitter?

Reading Statistical Analytics Illustrated

A few months ago, Prof. Kottemann sent me a copy of his new book, "Statistical Analysis Illustrated: Foundations You Should Know" (link). On a recent flight, I finally found time to give it a quick read.

The book covers a central section of frequentist statistical theory, which forms the core of most traditional Statistics 101 courses. This is the sequence that runs through random sampling, sampling distributions, law of large numbers, central limit theory, confidence intervals, and hypothesis testing (tests of significance).

Kottemann adopts an approach that deemphasizes mathematical derivation, and delists any theorem while favoring verbal and visual explanation. This is an approach that I also prefer in introductory statistics courses so I flip the pages eagerly.

With the stated goal of helping readers appreciate a "way of thinking" about data, Kottemann improves upon most textbooks in the following manner:

1. He concentrates on the case of sample proportions (e.g. x% of respondents felt this way), which can be easily appreciated given their common usage in surveys. This use case avoids one of the tricky turns that confuse statistics students, namely, learning the difference between the population standard deviation, the sample standard deviation and the standard error.
2. He smartly abandons the usual bottoms-up presentation of the central sequence. In the first chapter, he goes straight into sampling distributions, confidence intervals and significance testing without having defined a random variable, a normal distribution, the law of large numbers, the standard error or the null hypothesis (all of which will appear in later chapters).
   
   
3. The visuals in the book are improved versions of typical textbook diagrams as he removed unnecessary and confusing details.

This central sequence of frequentist statistics has proven challenging to generations of students because it's not a simple, straightforward argument. It's not easy because generalizing from samples of data is a complex undertaking. Kottemann's scope is narrower than the typical Statistics 101 textbook but what he covers is essential.

Most of the writing by Kotteman is lucid. The complexity of the materials - when explained in words - only got the better of him on a few occasions. Above Figure 1.1, he writes "[This histogram] shows the frequency (out of 1,000) with which we should expect to get random samples that yield the various possible values for the percent-in-favor sample statistic." I quote it only to illustrate the type of tongue-twisters statistics teachers run into when covering this material (and so he may rephrase it in the next edition); the sentence is an outlier in a very readable book.

The hypothesis testing framework has been under constant attack from other "ways of thinking". In the last chapter of the book, Kottemann briefly covers the Bayesian way of thinking. I hope readers appreciate that everything is different, from which question is answered to what assumptions are made to how decisions are framed.

The frequentist viewpoint has been widely abused. Journal editors reject any studies that do not reach a 95% confidence level, effectively ordaining publication bias, and motivating scholars to "p-hack," which is a general term that means engineering the "p-value" to hit 95%. In practice, people may "d-hack" which is to review a large array of outcomes, and select the one that happens to reach 95%; or they may "n-hack," which is to make the sample size so big that even tiny signals are declared statistically significant. I might even add "ps-hack" which is to pre-specify a large array of outcomes to cover up "d-hacking". 

If there is one gap in Kotteman's presentation, it's in the transition from the case of proportions to "the rest of the (frequentist) iceberg". If the reader is asked to do a test of difference in means at the 10% significance level, I'm not sure s/he can figure it out.

I'd also like to see a few worked-out examples of analyzing real-life opinion polls or market research surveys, and some exercises.

If you're learning statistics, and having trouble with the central sequence from random sampling to statistical testing, or if your sole interest is understanding polls or surveys, give Kottemann's book a try. It may help unblock you.

More case-counting puzzles

The Johnson & Johnson vaccine is being treated as a third-class citizen due to some safety concerns (link). Interestingly, such concerns did not surface in the "final" analysis of the single-shot Covid-19 vaccine, published in February, 2022 in New England Journal of Medicine (link). In this report, the scientists concluded "Ad26.COV2.S was associated with mainly mild-to-moderate adverse events, and no new safety concerns were identified". Since in the interim report, the FDA said "the analysis supported a favorable safety profile with no specific safety concerns," we were supposed to believe that the J&J vaccine is completely safe.

On the efficacy side, the final report headlined 56%. This number is meaningless unless we know how VE is defined. The 56% number comes from a counting window that starts 14 days after the shot (1D+14). During the interim report, the headline number was 67% (link).

However, these numbers are not comparable for reasons I'll get into in this post. The FDA has allowed pharmas to keep shifting the goal posts so nothing can really be compared like for like. The situation illustrates clearly why statisticians won't give you an answer unless we know all the details about how things have been measured!

***

First, the interim report was conducted when only about half of the participants have reached 2 months of follow-up - note that the largest number on the time axis was 126 days.

In the final analysis, about half the participants have reached 6 months of follow-up - the largest number on the time axis is now about 280.

So, the comparable value from the final report is not the headline VE which concerns up to 280 days of follow-up but the VE at day 126... and that number is 55% (1 - (436-100)/(938-185)). According to Table 2B (final report), the error bar around this number is 20% to 70% so the trial did not enroll enough participants to give a precise estimate of VE.

Harmonizing the follow-up period is not sufficient. Between the initial and final analyses, J&J ditched the requirement that all cases be "centrally confirmed". In the interim report, cases were counted only if a central laboratory confirmed the local PCR test positives. In the final report, cases accrued based on local test results.

***

Another major development is letting the placebo group get vaccinated. After J&J's vaccine was granted EUA at the end of February 2021, all participants reaching 6 months of follow-up were effectively unblinded so that those in the placebo group could get the vaccine if they so chose. Almost everyone did.

Since enrollment ended October 2020, and the data cutoff for the final analysis occurred in July 2021, everyone could have been followed for at least 6 months, if the above plan were implemented to the letter.

But that was not what happened. Some participants were unblinded prior to reaching the 6-month point because they requested getting one of the by-then authorized Covid-19 vaccines.

As a result, only 23% of the study participants had an observation window of at least 6 months. (By design, 100% should.) In fact, half of the participants in this trial had a follow-up period of less than four months. Because the follow-up period was truncated by the placebo participants receiving the vaccination, effectively destroying the placebo group, there will be no further useful data.

Refer back to the case curve shown above, notice that the at-risk population has dropped by more than half by day 126. The further out one went in time, the fewer participants contributed to the incremental case numbers, the wider the error bar. Such dropoff does not happen in a normal clinical trial. 

Those who reached the longest follow-up period are also the earliest enrollees in the trial. In most vaccine trials, the pharmas first enrolled the lowest risk people - in the case of J&J, the earliest batch of enrollees were young people with no comorbidities. So the right hand side of the case curve lacks both validity and reliability.

***

When I wrote about the J&J trial, I pointed out that it was one of the few double-blind trials - the Pfizer and Moderna trials were single-blind. Blinding removes a source of potential bias - a kind of self-fulfilling prophecy in which the investigators or the participants behave in a way that advantages the treatment against the placebo.

We now have solid evidence that blinding did not work well, and confirmation bias was a real concern. To begin with, given that about 30% of Americans have obstinately refused to get the vaccine, we know that this subset is not represented in these vaccine trials. When the placebo groups in these trials were offered the vaccine, almost all of the participants took advantage, implying they were vaccine enthusiasts, not skeptics.

Blinding is supposed to neutralize partisan feelings about vaccines but it didn't work as planned. In Figure S1 (in the appendix), we learned that 1,222 placebo participants "received other vaccine before unblinding", compared to 670 on the vaccine arm. That's 80% more on the placebo arm than the vaccine arm. If the treatment groups were properly randomized, and if participants were properly blinded, they would not know whether they received the placebo, and so the proportion of people who got another vaccine against trial rules should be balanced.

The number of participants who withdrew during the trial was also disproportionately higher on the placebo arm (338 vs 224), about 50% more. These are huge gaps.

***

One of the big mysteries of Covid-19 vaccine trials is what counts as a case. The case-counting window is one variable I've discussed often. But there's more. Most of these early trials were designed to measure symptomatic, PCR-confirmed cases. J&J added a definition of mild, moderate and severe cases, although as I pointed out before, the definition of "mild" disease is meaningless since almost every case has been classified as moderate or severe.

In most studies, including the J&J trial, all participants are given serological tests at regular time intervals. Seropositivity indicates that someone has been infected in the past, and thus, it is presumed that said infections were asymptomatic. This device has been used by scientists to claim the vaccines also prevented asymptomatic infections.

Apparently, J&J now believes that someone without a positive result from either PCR or serological testing can also be infected. This is how I interpret one of the footnotes from Table 1 that corresponds to a concept called "any SARS-CoV-2 infection", which includes:

"undetected cases that were subsequently detected through a positive serological result (according to the clinical severity adjudication committee), which did not count as either symptomatic cases (becqause they were RT-PCR-negative) or asymptomatic cases"

I'm not sure how the adjudicators got wind of the "positive serological result" since if such a result was present, the case should have counted as "asymptomatic", which is a different line in Table 1.

(Also, the severity adjudication committee is a misnomer as the committe members concerned themselves here with asymptomatic infections that would be milder than "mild".)

What is shocking about this line of numbers is the magnitude. There were 1,038 such infections in the vaccine group and 1,699 in the placebo group. These two numbers are much higher than the counts that made it to the headline VE calculation. In particular, only 42% of the vaccine infections, and only 52% of the placebo infections, ended up as "cases". This calculation also shows that an "any" infection on the placebo group has a higher chance of being counted as a case than an "any" infection on the vaccine group.