Big Data, Plainly Spoken (aka Numbers Rule Your World)

In times of paranoia, seek out data and stick to facts

All around the world, people are getting very afraid of the coronavirus. What's happening now is a live version of a disease outbreak investigation, which is discussed in Chapter 2 of Numbers Rule Your World (link).

To understand what's going on, you should see this as the exact opposite of "big data". We have a new flu virus. There is almost no direct data on this virus. The data are dribbling in as cases are being reported. We have a couple hundred deaths.... in a country with almost 1.5 billion people. Public health officials must make key decisions now before more data become available. But we have already passed the most dangerous phase, because they already found what causes the disease.

I saw a graphic last night, shown during an interview with an expert who investigated the SARS epidemic. Supposedly the mortality rate for the coronavirus is about 3 percent. This number has a large margin of error at this stage because of the shortage of data. This shortage is due to the newness of the phenonemon.

The closest relative to the coronavirus is the SARS virus which had a mortality rate of about 10 percent, which is three times higher than what we currently know about the coronavirus.

We also know that the coronavirus spreads faster than SARS. Most of the news reports I read mention the number of infected but rarely the number of deaths. This is a major oversight.

What's fearful is a deadly disease. A disease that infects lots of people but acts like the common cold is not a public-health crisis. So as data roll in, we should keep an eye on the mortality rate.

The mortality rate is the number of deaths divided by the number of infected. Both counts are growing each day. If the growth of infected is faster than the growth of deaths, the mortality rate will go down (not up!). The mortality rate will go up only if the growth rate of deaths exceeds the growth rate of infected.

In fact, by the time I finish this post, the latest report said the coronavirus death rate is 2 percent, so it has already been adjusted downwards because the flu is spreading fast. (The death rate of regular flu is about 1 percent, according to the graphic that was shown last night. This level acts as a control for interpreting the mortality rate.)

That report contains a typically alarmist quote by a medical professional: "There are likely to be many times more cases in Wuhan than officially confirmed". Okay, but if that is true, the real mortality rate is many times smaller than what is official! In terms of missing data, it's the unreported deaths that are more significant than unreported infections.

Also note the network effect. It's very unlikely that the growth rate of deaths can be higher than the growth rate of infected because infection grows exponentially due to transmissions. Deaths don't transmit.

It would also be useful to know what the cure rate is - because some of the infected will be in the treatment stage. During the report last night, the journalist wrongly asserted that 97 out of 100 infected will survive. This interpretation is correct only if we exclude those infected that are currently in treatment. (The technical term is right-censored data. We don't yet know how many in this subgroup will survive or not.)

While one should be cautious because this coronavirus is new, in times of paranoia, one should seek out the data. Ignore the unfounded rumors, innuendo and folklore. Stick to the facts. So far, the data aren't that alarming.

***

I n Chapter 2 of Numbers Rule Your World (link), I describe two modes of statistical modeling. Disease outbreak investigation illustrates one mode, in which the chief concern is establishing cause-effect relationships. This is always hard if you have observational data - without controls. But it's even more challenging in these crisis situations, with small drips of data. If you understand what statistical tools are available, and what the limitations are, you have a better grasp of what's going on right now.

 

Image credit: By https://www.scientificanimations.com - https://www.scientificanimations.com/wiki-images/, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=86358105

 

P.S. [2/3/20] See also my post on Junk Charts about a graphic used by the New York Times in a report on the virus.

The absurdity of predictive policing

Big (surveillance) data is changing everything - that is undeniable. But is Big Data changing everything for the better? That is what you have to think about.

As someone who's done data analytics for a long time, I can say confidently that the potential for harm is at least as large, but probably larger, than the potential for good.

A case in point is data as related to crime and policing. This post picks up from the perceptive article in the New York Times, "How Reporting Practices can Skew Crime Statistics" (link).

The article points out something really important - data is as data is defined. Different definitions result in different analytical results, frequently contradictory. No one should interpret analytical results without first learning the data definitions.

The author describes two extremely popular styles of "big data" analytics: top N ranking, and trend analysis. In crime statistics, sample insights are (1) "South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people." and (2) "Violent crime has worsened in South Bend since mayor Pete got elected in 2012."

With the volume of data available today, those two simple analyses still account for the vast majority of "analytics" out there.

South Bend, Indiana is the 27th worst city for violent crime rate among all cities with over 100,000 people.

Here is a list of definitions that affect the interpretation of this statement, most of which are described in the NYT article:

• whether the suburbs are included as part of the "city", or more broadly, how the boundaries of the city are drawn. This matters because the crime rates in the suburbs are different from the crime rates in the city centers.
• whether the error in the count of population is comparable across all cities under consideration
• what is considered a crime
• what proportion of crime is reported
• what crime is classified as "violent"
• are there any exceptions

You might think the population of a city is an objective fact but it isn't. Definitions matter, and they matter a lot when you're comparing different cities.

Violent crime has worsened in South Bend since mayor Pete got elected in 2012.

In addition to the points listed above, we also need to know if any of those definitions changed between 2012 and 2019. Changing how something is defined almost always breaks the continuity of any analysis, thus undermining any trend analysis. I raised this point in my book Numbersense (link) in the chapter on obesity - if the medical community changes how we measure obesity, then all historical data are rendered useless so such proposals must be considered using great caution.

Further, we should find out whether violent crime was increasing or decreasing prior to 2012 in South Bend, whether violent crime was increasing or decreasing in other cities comparable to South Bend since 2012, whether violent crime was increasing or decreasing in the nation as a whole since 2012, etc.

***

Some readers of my book Numbersense (link) might be mystified why I spent so much energy discussing definitions - of grad school admissions statistics, of U.S. News ranking, of obesity measures, of unemployment rates, of inflation rates, of fantasy football metrics, etc., etc.

A key to numbersense - to judging the statistics and data thrown at you - is to understand clearly how the data are collected, and in particular, how any metric is defined.

The discussion of crime statistics by the New York Times makes this case cogently.

***

Now let me get to the absurdity of predictive policing.

A particularly popular and cited application of Big Data "for social good" is predictive policing. The goal of predictive policing is to prevent crime from happening.

The ideal of "Minority Report" is often hoisted on us. Wouldn't it be great if some AI or machine learning or predictive model can figure out who will be burglarizing your home next year, and have the police lock the would-be burglar up before s/he could commit the crime?

I have done a good amount of predictive modeling in my career so I am not saying what I'm saying because I'm a luddite. It's because of that experience that I am leading you down the following rabbit hole:

There are not many burglaries. According to Statista, the worst state (New Mexico) suffered 770 burglaries per 100,000 residents in 2018. This means that if the analyst is given a list of 100,000 New Mexicans chosen at random, a perfect model should label 770 of those as potential burglars. That is to say, we want the model to pick out 0.77 percent of the list.

A perfect model does not exist. Any model generates false positives. Because burglars are rare, any reasonably predictive model would need to red-flag many more than 0.77 percent of the list to have any chance of capturing all 770 potential burglars. Let's say the model points the finger at 1 percent of the list. That would mean 1,000 potential burglars. Since there should be only 770 burglars, the first thing we know is that at least 230 innocent people would be flagged by this model.

Next, think about what happens to the 1,000 flagged people. In the ideal of predictive policing, police would knock on their doors and arrest them for their future crimes. These people who haven't burglarized anyone would be duly jailed by courts who buy into predictive policing. (How would they defend themselves against an allegation of future crime?) 

We could say with certainty that those 1,000 did not subsequently commit burglary - given that they were incarcerated.

So humor me, and try to answer the following question: Of the 1,000 people flagged by this model, how many of these were accurately predicted?

 

 

 

Response to Kohavi's note on non-significance in A/B testing

Last week, I posted some comments on Ron Kohavi's note on A/B testing, which appeared on his Linkedin feed. Kohavi's chief concern is what to do when the A/B test comes back inconclusive, i.e. no significant difference between the test and control treatments. As any practitioners know, most tests come back inconclusive (!)

Kohavi correctly points out that for a variety of reasons, people push for "shipping flat", i.e. adopting the Test treatment even though it did not outperform Control in the A/B test. His note carefully lays out these reasons and debunks most of them. When he publishes the final version, I'll post it here.

The first section deals with situations in which Kohavi would accept "shipping flat". He calls these "non-inferiority" scenarios. My response to those scenarios were posted last week. I'd prefer to call several of these quantification scenarios, in which the expected effect is negative, and the purpose of A/B testing is to estimate the magnitude.

This post delves into the second section, in which Kohavi explains why many of the popular reasons for "shipping flat" should be rejected. I am in general agreement with most of these points.

His very first point is the most important. If you learn nothing else from the note, here is the part that you should pay attention to:

Null hypothesis statistical significance testing only allows us to reject the null hypothesis (in this case commonly of no treatment effect). We cannot accept that a p-value above the threshold (commonly 0.05) implies that there is no treatment effect. It could very well be that the test is underpowered. The problem is exacerbated when the first iteration shows stat-sig negative, tweaks are made, and the treatment is iterated a few times until we have a non-stat sig iteration. In such cases, a replication run should be made with high power to at least confirm that the org is not p-hacking.

What he's saying is that A/B testing is about the signal-to-noise ratio. When this ratio is too small, our test is inconclusive. This could be due to a weak signal, too much noise, or both. If "too much noise" is suspected, the test should be re-run with a larger sample size ("higher power"). If it then works out as expected, i.e., with dampened noise, the signal comes through, there will be no need to "ship flat".

A couple of other points Kohavi made are worth emphasizing here.

The "ship flat or not" decision should be based on a cost-benefit analysis. It may be true that both Test and Control lead to indistinguishable results (say, similar advertising revenues) but switching to the Test treatment incurs additional costs (e.g. the need to apply new tags across the website). In this case, the clear answer is to reject the Test treatment.

The "sunk cost" fallacy may be in play. Sometimes, labor or other investments may have been expended in order to develop the Test treatment. In the example I gave in the last post, if the Test involves a new file uploader module on a website, that new uploader application would have to be developed in order to be testable. Economists would argue that these costs should be treated as "sunk" i.e. irrelevant by the start of the A/B test. In practice, though, such costs are used to justify shipping flat.

Next is the strategic argument. The CEO is committed to this strategy, and no test result would overturn it. In this case, the purpose of the A/B test has been wrongly specified. The true purpose is to quantify the impact of the switch in strategy - as opposed to determining which strategy is better.

The appropriate follow-on question is whether the CEO would change his/her mind after receiving the new information that the strategic switch would incur costs of implementation while generating no benefits, thus negatively impacting profits. Few executives will make a decision that hurts profits while not adding revenues, knowingly.

No amount of scientifically-sound data can topple unsubstantiated speculation that some unmeasurable other effect would overwhelm the observable outcomes. Examples given by Kohavi include (a) arguing that the Test treatment enables other features or strategies that could well generate the positive results; (b) arguing that some other system such as a machine learning model would adapt to the Test treatment so that an unobserved interaction effect would materialize to justify it; (c) arguing that while the Test treatment did no better on the key performance indicator, it should be better on some unmeasured other indicator (say, long-term customer satisfaction).

In these arguments, the propenent attempts to use unobserved assumptions to override carefully measured data, and any data-driven decision-maker should reject them.

 

 

 

 

Response to Kohavi's note on non-inferiority in A/B testing

Ron Kohavi is on a mission to reform A/B testing culture within the big tech companies, and that's a good thing. On Linkedin, he posted a note about thoughts on "shipping flat". For the statistically-minded readers here, that is a term to mean switching treatment to the Test even though in the A/B test, the difference between the Test and Control treatments was judged to be statistically insignificant. (Admittedly, I heard this term for the first time myself from Kohavi although I can attest that it is common industry practice to launch the Test because it is not worse than the Control.)

More concretely, let's say the engineering team developed a new file uploader for the website. When tested against the existing file uploader, it did not improve the performance metric, say the daily frequency of uploads by users.

In the old days of direct marketing, people would just reject the new variant. We had a notion of the "Champion". Challengers don't knock off the Champion unless there is definitive proof (statistical significance) that they are better. In sports, world records do not get beaten when equaled. The first person still retains the title until someone beats (not equals) the record.

Nowadays, there are plenty of arguments made to "ship flat". Kohavi laid out many of those arguments, and presents why they are flawed. I will discuss those in the next blog post (This is now available.).

***

In this post, I want to respond to Kohavi's list of situations where "non-inferiority testing" is justified. These are situations in which he believes that tests can be designed with the hope of proving that the new treatment (Test) does no worse than the existing Control. In other words, one can accept that the Test and Control are a statistical tie.

I believe in most of the situations he outlined, non-inferiority is wishful thinking. It is better to acknowledge that those changes are expected to impact performance metrics negatively - the purpose of the A/B test in those scenarios is to quantify the extent of the damage.

The simplest case to explain this point of view is Kohavi's scenario (c): GDPR and CCPA are pro-privacy legislation that will almost surely hurt websites with advertising business models. For example, websites may have to ask for opt-in instead of opt-out. Ideally, the website would love to come up with something that would not reduce their advertising income. If they succeeded in doing so, then the legislation is proven toothless! The value of the A/B test here isn't to show no statistical difference, but to show management the likely magnitude of the harm.

Scenario (d) called "competitive parity" is the most interesting. In Kohavi's words: 

The competition has the feature, and an important comparison is coming out by a third-party where our product will be missing the checkmark. Note that if the competition is a solid data-driven company, it is likely that they shipped this feature because it was positive to the OEC.   A flat treatment hints at a poor implementation that could be improved.

I can even hear this argument in my head so it's great that Kohavi laid it out for discussion. I find it hard to accept it as a valid case for "shipping flat". 

Here is what I don't like about the competitive parity argument:

1. The competitor - the solid data-driven company - is also likely to have "shipped flat" due to any of the reasons listed by Kohavi in the same note. So the fact that they shipped the new feature does not mean it positively improved performance.
   
   
2. Test results are usually highly specific to the context. Even if the new feature wins at the competitor's site, it is a big stretch to assume it would win at our own site, given our own other features, technologies, set of users, etc. Even within one's own site, a feature that worked on one page does not necessarily work on a different page.
   
   
3. The performance metric used by the competitor to evaluate the test may not be the same as ours.
   
   

In a case like this, I might accept that we need that feature for strategic reasons - and the purpose of the A/B test is to measure its impact on our site. The test should be designed as a "two-tailed" test, meaning that we don't have an expectation of whether the new feature would be beneficial or harmful.

P.S. [1/28/20] The second post is now up. This new post addresses the scenarios under which Kohavi advises against "shipping flat".

A student skipping class might be grievously depressed. Claims like these discredit data science

This is Part 3 of a multi-part post on student surveillance technologies. You many want to read Part 1 and Part 2 first.

In the Washington Post article on student surveillance, one of the featured use cases is predictive analytics. In short, the location data are merged with other data to “score” students. Scoring is a form of rating by means of a formula.

The most famous example of predictive scoring is the ubiquitous and consequential credit scoring system. The credit score is a prediction of whether an individual is likely to repay the credit card company which is lending him or her money to purchase something today (without any collateral). I consider it a fundamental example of big data analytics, and have an extensive discussion of credit scoring in Chapter 2 of Numbers Rule Your World (link).

[Image credit: Frankieleon on Flickr]

***

Misunderstanding predictive analytics is flooding the marketplace with lots of defective products that harm people. I'm quite concerned by the language used by the business folks in the Post article to sing the praises of predicting student well-being. These statements reveal unfounded optimism about the accuracy of predictive analytics, and a shallow grasp of cause-effect relationships.

One vendor presented how their predictive model works: “a student avoiding the cafeteria might suffer from food insecurity or an eating disorder”. This kind of overly simplistic analysis is giving data science a bad name.

In reality, an extreme minority of people who avoid the cafeteria will be suffering from an eating disorder. Most scoring algorithms work with not one but multiple predictive factors, and even then, such an app will trigger many false alarms.

Even more shocking is the deduction that “a student skipping class might be grievously depressed”. It’s not the lack of modeling skill that concerns me here; it is the lack of common sense. For each student who skips class because of “grievous” depression, there are tons of non-depressed students who skip class. The insertion of “grievous” makes it even worse.

Behind any such deduction is a cause-effect model. It should be clear that Skip Class -> Grievously Depressed does not fly.

This may be a fallacy of transitivity. Grievously depressed -> Skip class is likely to hold. But reversing the causal order does not work.

***

Flawed data science products have consequences. Welfare checks are wasted on large numbers of healthy students. Students rightfully demand to be treated as adults. The truly grievously depressed students meanwhile may be missed.  

***

In Chapter 4 of Numbers Rule Your World (link), and Chapter 5 of Numbersense (link), I discuss ways to examine predictive models, and dealing with their inaccuracy.

New data on vaping

In my video about vaping (link to video, link to blog post), I explained why the e-cigarette industry’s claim that vaping saves lives is a half-truth. Their argument assumes people who vape would have died from smoking traditional cigarettes if they hadn't chosen to vape instead.

That’s true only when all of the following conditions apply:

1. The person would have smoked cigarettes if e-cigarettes were not available
2. The person switched completely to e-cigarettes
3. The person would have died from smoking-related illnesses had s/he smoked cigarettes
4. The person would not have died from vaping-related or other illnesses had s/he vaped
5. The person would not have undertaken any other measure, available now or in the future, to mitigate the risk of death

(1) fails for every vaper who would never have smoked cigarettes.

We expect this category to be sizeable – precisely because powerful marketing asserts that vaping is safe. The potential harm of a new product is hard to quantify at the start but we now know that people have died from it.

What’s easier to establish is how big this problem is. A recent Wall Street Journal article gives us the data: according to a federal survey, 20% of all middle- and high-school students in the U.S. “recently used e-cigarettes”, compared to 4% who used cigarettes. If the prevalence of cigarettes is stable, then four out of five vapers in school would never have shouldered the risk of smoking-related death but are newly exposed to possible harm through e-cigarettes.

***

In thinking about point (3), keep in mind that not all smokers die from smoking-related illnesses. You should only count the preventable deaths, that is to say, the person would have lived longer had s/he not smoked. What proportion of smokers actually die from smoking-related illnesses?

This question is harder to answer than one thinks. The basic analysis would be identifying who's the smoker, and whether the person died from a disease associated with smoking such as lung cancer. However, there are few diseases with a single cause so even if a smoker died from lung cancer, it's not assured that smoking is the cause of death. The counterfactual question we're interested in is whether this same person would have died from that disease had s/he not smoked. To complicate this further, research showed that if someone quit smoking early enough, the risk factor goes down by 90 percent. CDC has this page that shows some relevant numbers.

***

My video takes a deeper look into the different groups of people that may be impacted by vaping.

 

All about standardizing data

As some of you may know, I have been experimenting with Youtube videos. The latest one is about standardizing data, and you can view it now. 

This video belongs to a series I call "Data Science: The Missing Pieces" (link). Standardizing data is embedded into almost all data modeling pipelines, whether you are building deep learning, machine learning or traditional statistical models. Everyone does it but just like much in data science, it's not well explained why you standardize the data. In this series, I intend to cover these types of gaps and help you understand why you are doing something you've been told to do. "Because it works" is not a sufficient answer!

 

This video is an extension of a recent blog post in which I explained why standardizing data is not about turning data into a normal distribution, contrary to what you may have been told. I get into why we standardize data, the basic formula for standardization, and when you shouldn't be using it.

***

If you have a topic you'd like me to cover in the future, please comment below or send me an email.

 

 

Election polls politicking

As the U.S. enters election year, the press will be reporting on polls. Sadly, it is still not mainstream to prominently display margins of error, or to draw conclusions based on interpreting these margins. So readers must take this matter into their own hands.

Polling is an exercise in generalization: the goal is to learn the average opinion of a large group of people without having to solicit each person's opinion. If one conducted the same poll twice on different random samples of people, the results would vary since different individuals would have answered the poll. Margins of error account for such variability, and provide a simulation of what the range of results might have been if the poll were run repeatedly.

If the difference between two candidates is less than twice the margin of error, then the two candidates are in a "dead heat". A dead heat just means that this particular poll can't tell those two candidates apart - with the required degree of confidence.

***

The media routinely makes a mess of poll numbers.

This last week, Reuters led with the headline "Sanders leads, with Warren, Buttigieg, Biden chasing in Iowa democratic poll" (link). The poll results:

Sanders - 20%
Warren - 17%
Buttigieg - 16%
Biden - 15%

More specifically, the reporter said "Sanders received support from 20% of respondents in the poll, with the next three candidates - U.S. Senator Elizabeth Warren, former South Bend, Indiana Mayor Pete Buttigieg and former Vice President Joe Biden - in a statistical tie behind him in the poll." (my bolding)

This is a rare example of news media declaring a difference as insignificant. But don't lose sight of the pairwise differences: Sanders-Biden is 5%, Sanders-Warren is 3% and Buttigieg-Biden is 1%.

Later in the article, the margin of error is reported as 3.7%. Twice the margin of error is about 8%. So the poll really can't distinguish between any of the top 4 candidates.

The candidates outside the top 4 are said to have scored below 10%. For a candidate at 10%, this poll gives Sanders a difference of 10% which is significant; against Biden, the difference is only 5% which is a statistical tie.

***

Compare the above report with an earlier report also by Reuters, from August 2019 (link). This article led with "The latest Reuters/Ipsos poll finds Biden leads Democrats as minorities favor most electable candidate versus Trump". 

What are the poll numbers that prompted this headline?

Biden - 22%
Sanders - 18%

Margin of error? 5%. So twice the margin of error is 10%. A 4% difference against 10% is definitely a "statistical tie". And yet, unlike the first article, this one ignored the statistical tie, and declared Biden the front-runner.

***

It is preferable to interpret collections of polls instead of individual polls. Over at FiveThirtyEight, Nate Silver doesn't make headlines based on one poll. See his latest view on the Democratic race.

 

P.S. It's unclear to me whether the margin of error cited is for an estimate of single proportions or difference in proportions. If the former, then these polls are even less precise than indicated above. Perhaps we have readers who are polling experts who can clarify...

 

 

 

The day totes of smartphones invaded the lecture halls

This is Part 2 of a multi-part post. Read Part 1 here. In short, some colleges are purchasing technology based on smartphones to track student movements.

At the end of Part 1 on student surveillance, I wonder how this tricky issue will play out as schools and students find common ground.

In this post, I focus on another side of the conversation, which is how disruptive cellphones have been to the classroom learning environment. Technology Review has an article touching upon this recently, and I’ve also heard off and on from colleagues complaining about these trends.

The philosophy professor who wrote the article enticed 12 students to give up their phones for nine days. (The fact that they had to be bribed with extra credit to do so is itself revealing.) One of the students wrote: “One of the worst and most common things people do nowadays is pull out their cell phone and use it while in a face-to-face conversation. This action is very rude and unacceptable, but yet again, I find myself guilty of this sometimes because it is the norm.”

This behavior is also spreading to the classrooms. The professor estimates that 70 percent of the students in his classes at any time are using their phones. How does the professor feel when students start texting each other in the middle of a lecture? Are the students expressing confusion or worse, boredom? Has s/he misspoken, or said something unintentionally funny? One of the main advantages of in-class learning is the continuous feedback through body language and facial expressions – this is lost when students are preoccupied with, or snickering about, something unrelated to the class.

One drastic solution is to ban the phones. Are there less extreme measures?

***

In the same way that “smart” phones and devices reveal locations, with the appropriate software, they can also reveal whether students are using their phones during class, how frequently they are doing it, and what apps they are running. The first version of tracking software might inform the professor "Student D came to class late by 2 minutes” but the next version will say “Student D spent 16 minutes in your class today playing League of Legends”.

The debate around where to place the limits should be based on what these technologies can do, not just what actions are built into current versions of the software.

***

For data analysts reading this, realize that even if the data say someone was present in the auditorium, you don’t know that for a fact. You only know that the “smart” phone was there -- the person who owns the phone might not have been.

“Smart” phones are already disrupting lectures. But watch out for the day when totes of smartphones show up in lecture halls!

P.S. As a statistician, I can’t help but point out the iffy science behind this Technology Review article. The professor created an extra-credit assignment because he surmised that cellphone usage depressed his students’ learning ability. He then “paid” volunteers to conduct this exercise. The paid volunteers reported that being without the phones made them more productive, confirming the expected effect. This methodology is like an airline sending you a customer satisfaction survey after paying you for volunteering to vacate your seat on their overbooked flight, and then offering frequent-flyer “status” as a reward for filling out the survey.

Since I’m sympathetic to Professor Srigley’s conclusion, my skepticism didn’t come as fast or as strongly as when I disagree with the finding. This is a type of confirmation bias that data scientists have to guard against.

P.S. [1/23/2020] Part 3 is now up. In this new post, I deal with the other use case of student surveillance data - predicting student behavior.

Is it ok for colleges to use smartphones to follow students everywhere they go?

As the new year arrives, I will be writing about some topics that are likely to enter the national conversation in 2020.

First up is establishing limits around what data science businesses are allowed to do. The dam really broke on this issue after the Cambridge Analytica scandal. (See my coverage here.) Since then, the mainstream press has done a commendable job speaking on behalf of the silent plurality (?) who have concerns about how far the big data industry has pushed. These journalists continue to uncover new examples of businesses trying to hide or confuse users about what data they are collecting on them and for what purpose.

In these discussions, it is important to distinguish between what is already implemented, what can be implemented, where industry insiders say they draw the line, whether we can trust what they tell us, and whether and why government regulators are silent or slow to act.

***

The Washington Post published a well-researched article describing how some colleges and universities in the U.S. have started using smartphones to amass databases of student locations. I’m hopeful that the youngsters may finally re-assess their relationship with “smart” technologies. The double-edged sword raises many difficult questions, and not just for students, since the same surveillance technology affects the rest of us.

For those who want to skip reading the Post article, here is a summary:

Some college administrators have purchased technology that tracks students everywhere they go through their smartphones. Several uses of the location data are described. Some professors use the data to take class attendance. The software reveals who walks in to class two minutes late (in reality, location tracking is precise down to fractions of a second.) Athletic coaches use the data to ensure student-athletes remain in compliance. Vendors analyze the data to generate early warnings of “troubled” students [my added quotation marks]. Data privacy advocates, some educators and some students are voicing concerns about the negative implications of such technologies.

There is so much to discuss here that I am breaking it up into multiple posts.

In Part 1, I spotlight a few myths around the topic of data privacy that this Washington Post article forcefully debunks. 

Bluetooth and Wifi expose your whereabouts.

Smart phones and smart devices collect our precise locations, not merely through GPS or cell towers. The preferred location tracking technologies today are Bluetooth and Wifi. Google engineers realized this long time ago – remember when they were caught pulling down Wifi network names and passwords using the vans that take photos for Google Maps? (You might not remember this episode so here is a link to get started. Also, I made a video about this.)

Typically, apps are necessary to operationalize location tracking. Many apps are free only because they are quietly collecting location data from users that are then sold to the highest bidders. (Most famous among these are the weather apps, many of them now owned by IBM. Here is pre-woke mainstream reporting on the IBM acquisition, and here is post-woke reporting, three years late but still welcome.)

We are far from the world of “data exhaust” in which our location is revealed because our cell phone talks to nearby cell towers as a basic function. What’s happening today is that specific technologies (e.g. the Bluetooth beacons mentioned in the Post) are created to harness location data, upon which entire businesses are built.

See my previous post on data sleaze for more on this topic.

***

Location tracking is not anonymous.

Consider the following excerpt: “the system can email a professor or advisor automatically if a student skips class or walks in more than two minutes late.” Later, the journalist noted: “Advisors text some freshmen athletes if they don’t show up within five minutes of class.”

These fragments show that the buyers of location data want to effect specific behavioral changes: the coach calls out individual athletes to get them to class pronto! You should always assume that data collection is neither anonymous nor anonymized. Why? If the student’s identity were not known, the software couldn’t have traced it to the coach, nor could the coach have texted the specific student!

In the same way, marketers who pay for location data want to send promotional offers to specific shoppers deemed likely to respond to them. Political parties target “persuadables” ahead of elections. Personalizing services or actions requires the individuals’ identities as well as their contact information. They simply can’t be anonymous or anonymized.

In an otherwise excellent article, the journalist failed to realize that the vendors implicitly admitted they collect personally identifiable information (PII) and that the data are not anonymous. The next time you hear a tech executive justifying data collection with anonymity, you know s/he isn’t serious about data privacy.

***

Location tracking is not aggregated.

What else isn't true? The claim that businesses only sell “aggregated” data.

You can’t aggregate the data when your goal is to monitor individual students. Aggregating data eliminates privacy concerns but the reality is that many buyers of location data attempt to change individual behavior, which requires individual-level data.

Here’s the bottom line: data are collected at the individual level. Aggregating data is extra work. Buyers of data spend higher sums for individual-level data. So data aggregation is higher cost and lower revenue. If your job is to grow profits, what do you think you’ll be selling?

The decision to throw away personal identifiers and only sell anonymous or aggregated data is a conspicuous choice to pursue a less profitable path.

It's not that no one sells aggregated data. Anyone who is "personalizing" anything for real isn't aggregating the data.

P.S. Note that I said location tracking is not aggregated. Location databases can be aggregated, if the vendor so chooses to, but the raw data feeding such databases are definitely not aggregated.

***

Trade-offs.

And yes, you can use a “burner” phone that cannot be traced back to you. Then, you’ll feel the double-edged sword of “smart” technologies. If you are looking for a restaurant, and the phone doesn’t know your location, you’d have to tell the software where you are. It’s a few more keystrokes, or clicks – for some, convenience is worth the price of giving up their location privacy; for others, it’s not.

For the students, having the smartphones on campus clearly enriches their lives in many ways. Now, they learn that these devices are informing school administrators of their daily activities and movements. How will they react? It's good that the conversation is happening now.

 

P.S. [1/9/20] Part 2 of this series is now live. See why totes of smartphones may invade the lecture hall!

P.S.[1/23/20] Part 3 is now up. In this new post, I examine the other use case of student surveillance data - predicting student behavior.