Big Data, Plainly Spoken (aka Numbers Rule Your World)

Lies, damned lies, and tech

In the second half of 2018, in the aftermath of the Cambridge Analytica-Facebook story, the news media have broken out of a stupor, and started realizing that the world of the Web and mobile is filled with fake data. I've written about this problem quite a bit. For example, my 2017 piece on the birth of the "fake news business" is still worth your time.

I also urged data analysts to recognize the amount of fake data that we are analyzing while looking the other way. This post describes some real examples of fake data discovered by journalists. Even earlier, in 2015, I authored a piece for HBR discussing Augustine Fou's work on uncovering ad fraud - driven by automated bots.

New York magazine just noticed that a lot of Web data are fake. This is a nice piece discussing the current state - it has only gotten worse. Among the trove of lies and damned lies mentioned in this article are fake web traffic, fake websites, fake clicks, fake mouse movements, fake social network accounts, fake cookies, fake video views, fake time on site, fake subscribers, fake video viewers, fake AI assistants, fake Instagram influencers, fake sponsored content (i.e. ads that pretend to be news), fake people, ... One inexplicable omission is the fake product review, which is probably the most commonly encountered species.

The tech industry is driving a lot of this and has not taken the proper actions to contain this problem before it goes out of hand. The fake data problem will evolve into a trust problem; in the last half of 2018, we start to hear rumblings of discontent.

 

 

The tradeoff between privacy and convenience

The Gillian Brockell letter (see previous post) can be read as a consumer complaint letter but also a love letter to the tech industry. 

Reader Antonio R. who forwarded the column to me via Twitter raised this interesting question: 

Her conclusion seems to be: more relevant ads is better than no ads at all. What future is waiting for cheated fe/males? A warning "Be careful to your partner" or a reassuring "All is well" to choose in advance among app settings?

Gillian is someone who totally buys into the tech industry's "big data" pitch - that the more you share, the more you gain. She's writing tags that cue algorithms to send her relevant ads. Presumably, when she was pregnant, she was satisfied with the ads that at the time were selling her relevant products.

She's mad that the algorithm is not all-knowing, personalized and omnipotent. She expects that Facebook, Instagram, Amazon, etc. tracks her every move, and optimizes her experience just for her. She's angry when it makes mistakes.

And, if one reads behind the lines, her proposed solution is for the tech industry to be even more creepy, gather even more personal data, be even more personalized. She wants ads, just not the ones she doesn't like.

This solution is not radical at all. In fact, it is exactly what tech firms have been doing for 10 years. The "theory" is: data make ads more relevant, and if ads are not relevant enough, it is because they do not have enough personal data. In this sense, Gillian's column is a love letter to the tech industry.

***

The overlooked solution is to have less relevant ads or no ads at all.

In the Charles Duhigg story about Target's pregnancy prediction model (see Numbersense), one of the curious nuggets we learned is that the data scientists deliberately mixed random products in between the pregnancy goods being marketed to the women predicted to be pregnant. The official explanation was to make the brochures appear less creepy.

In the book, I suggested a different explanation for that decision. In a predictive model like that, there are likely to be multiples more false positives (i.e. women wrongly predicted to be pregnant and thus sent irrelevant materials) than true positives (i.e. women correctly predicted to be pregnant). I also speculated that many true positives would act like Gillian did - appreciating the pregnancy product ads as relevant rather than creepy. However, I believe that the false positives will complain that the pregnancy product ads are irrelevant, maybe even somewhat offensive. 

Mixing in other products lessens the bad of the wrong predictions - but simultaneously, it will also soften the impact of the correct predictions. What's in the balance is consumer interests versus advertiser business goals.

 

 

 

 

 

What the Dear tech companies article says about predictive algorithms

Gillian Brockell at the Washington Post published a heart-felt essay pointing out the inhumanity of algorithmic (i.e. programmatic) advertising on social-media platforms like Facebook and Instagram (owned by Facebook). It's getting some attention in tech circles, which is a good thing. (Thanks to reader Antonio R. for tweeting it to me.)

Here's a tl;dr of her story: she works at the Post as an video editor, and is very active on social media, especially Facebook. Sadly, she recently suffered a stillborn. She found that Facebook continued to bombard her with advertising of pregnancy products, which kept bringing up sad memories. When she clicked on the "I don't want to see this ad" icon, the advertising did not stop but just switched gears, now assuming that she needs post-childbirth products.

She's social-media savvy enough to realize that the advertising platforms perfected by Facebook and Google are inhuman, driven by algorithms, based on silent, pervasive collecting of personal details that are either shared publicly, shared within semi-private communities, or transacted privately. Advertisers are relying on the scoop of data to auto-pilot their advertising campaigns, most of which define the goal as ad clicks.

***

The zeal of the tech industry to sell more servers, more processors, more boxes, more tools has led to a level of over-confidence in predictive algorithms that underly Gillian's dissatisfactory experience. Such enthusiasm is matched by the advertisers' hunger for more clicks, and more sales while ignoring the tenuous link between those clicks and sales. As data scientists, it's easy to get caught up in the adulation and, using Nassim Taleb's phrase, get fooled by randomness.

It is widely held that predictive algorithms are all-knowing, personalized, and omnipotent. The reality is not so clearcut. Let's take a deeper look at how algorithms work, and from there, maybe the data-science community will come up with alterations that can reduce the chance of dissatisfaction.

***

ALL-KNOWING

Algorithms are not all-knowing despite our tendency to believe they are so. Each algorithm is driven by a fixed set of data inputs. In Gillian's article, she suggested some possibilities, such as the tags on her Instagram, the contents of her Facebook posts, or her friends' posts, Google searches, "metadata" on Amazon wishlists, etc.

All of those could be inputs in someone's algorithm but each such item has to be meticulously captured by code. Collecting every additional item requires more coding. How much is collected depends on the culture of the development team. Some are extremely zealous, others draw the line.

Every algorithm then weighs the importance of different data inputs. These weights ultimately control the actions of the algorithms. Their determination is a guessing game. It's like what factors you'd use to decide who should be a Hall of Famer in sports. No two people can agree, neither can two algorithms.

***

PERSONALIZED

No algorithm can be truly "personalized", as in one-to-one personalized. One-to-one is incompatible with Big Data. If the algorithm tailors recommendations to you just based on what it knows about you, then it likely will make lots of mistakes. Data available at the individual level is sparse, and riddled with holes.

All algorithms leverage statistical averages. It's much easier to predict what movie the average teenager would watch than the preference of each particular teenager. Most algorithms work like this: if the teenager discloses all the movies that s/he watched in the past, then that personal history will form the basis for the recommendations, which can be quite accurate; but for most people, that level of detail is not present, so the algorithm falls back on statistical averages - what the average teenager watches.

Even within individuals, taste changes over time. So even for the data-hoarding teen, the recommendation derives from a "moving" average of his or her past record, in other words, the average over time of his or her past viewings. Because of this, the longer the history, the less sensitive is the algorithm to recent data. The GPA is a good analogy. It's much harder to move your GPA in your senior year than in your sophomore year because of the accumulation of grades.

(PS. This is a call for the industry to take up the issue of right to be forgotten. Deleting old data not only pays respect to your users but also removes a source of error from these algorithms!)

***

OMNIPOTENT

The most damaging myth about predictive algorithms is their supposed omnipotence. Almost every report in mainstream media about predictive algorithms makes assertions such as "Google can predict when you will die", and "IBM can predict who's a good employee". What do they mean by "can predict"? Readers are led to believe that "can predict" means "will predict", or "will predict accurately".

In reality, there is no black-or-white, no can or cannot, no accurate or not accurate. Accuracy is important, is much lower than advertised, and is not a binary state but a progression.

Predictive algorithms have been around for a long long time. Think weather forecasts. If I say, "the weather channel can predict the weather", you do not presume they will predict every day accurately. Some of us might even think "well, they can barely predict the weather. They have the ability to issue forecasts, but that doesn't mean the forecasts are accurate or useful."

***

There are several other important aspects of predictive algorithms, which I delve into in my two books. For example, accuracy is measured several ways, and always involves a trade-off between false positives and false negatives. The algorithms behave as per the incentives of their owners. The chapter on steroids testing, terrorist prediction, polygraphs in NUMBERS RULE YOUR WORLD (link) tackles this.

Also in NUMBERSENSE (link), the chapter on Target's model to predict pregnancy walks through how one measures the accuracy of predictive models, and how incentives play a role. Nate Silver's Signal and Noise book is also recommended - he has a section discussing the accuracy of weather forecasts, and how they are systematically biased (due to incentives).

 

 

 

 

Know your data 24: trust eroding

Big Tech is in serious danger of losing our trust when it comes to our data. For the longest time, they have sold consumers on a bargain: in return for providing convenience and services, they claim our personal data and sell them to the highest bidders.

That bargain requires consumers to trust Big Tech as good custodians of the data.

A number of recent revelations has eroded that trust. And I'm not talking about Cambridge Analytica.

Here are two recent examples that have not received the attention that they deserve.

Android phones found to track users even if they turn off Location History.

The Associated Press first broke this story, which was later verified by Princeton researchers. Android users who choose to turn off Location History obviously do so intending that they not be tracked.

In the same story, it is also disclosed that iPhone users who have installed Google apps are also being tracked at all times. When these users disable location tracking for the Google app, the app stops saving locations in the folder called Location History and saves the location data in a different folder instead. (Note also: the phone itself would not know how to do this; software engineers wrote code to enable this feature.)

This revelation is highly damaging. There have always been suspicion that your phone (or TV or other devices) is spying on you even if it is turned off. There are no technical obstacles to making this capability. The only reason why customers are not worried is that they trust Big Tech when they claim their phones (or other devices) do not have that capability.

Google does not deny this is happening, and in fact, argues that it is transparent in dealing with users.

***

Facebook found to have taken users' phone numbers provided for "two-factor" authentication and sold them to advertisers.

For a number of years now, Big Tech and small tech alike have bombarded us with security warnings, and made claims that "two-factor" authentication is the ultimate solution to online security issues. Setting up two factor requires the user to provide a cell phone number, which immediately removes any semblance of online anonymity (unless you get your hands on a burner phone).

It's been known in the marketing world forever that the key to unlocking anyone's data is your phone number. That's because the cell phone is almost always a personal device.

I don't typically consent to two factor for fear that the phone number could then become marketing fodder. This story, by Gizmodo, confirms that my worst fear is real. Similar to Google above, Facebook confesses to coopting those phone numbers, and even maintains that users have consented to such use.

This revelation is also highly damaging. In fact, it damages the reputation of the security industry. How are we to trust them when they tell us to use things like phone numbers, fingerprints, eye scans, etc. for security purposes when we cannot trust that such private data would not be transferred to other entities?

In the same story, it is disclosed that Facebook also harvests phone numbers for advertisers under a variety of other pretexts. For example, when you provide your phone number to receive alerts about new log-ins to your account, that number ends up on marketing lists. Further, when you upload your contact list to Facebook, you have exposed all of your friends' phone numbers, turning them into fair game for Facebook's advertising clients!

This further relevation is even more damaging. Lots of websites ask for our phone numbers under the pretext of servicing our accounts, but we can no longer trust any of them that they would not turn our personal data over to advertisers.

 

***

Judging by their subsequent pronouncements when caught red-handed, we fear that Big Tech is not sensing the importance of "trust" in their business model. If the trust goes away, these businesses will be in for a nasty surprise.

New Facebook data breach, and the importance of Business Intelligence

 

On Friday, Facebook announced that hackers have gained access to personal data of at least 50 million users (see here for example). Analysts immediately connected this incident to the Cambridge Analytica scandal. How is this data breach different from the Cambridge scandal?

How the data was breached

One should take any announcement of how a data breach occurred with a grain of salt: it's obvious that companies will not publish anything that attracts lawsuits; plus, it's unclear that there is a penalty for lying about the real reasons for a data breach. There is no external auditing, and the companies control the narratives and statistics surrounding these events.

Based on what Facebook told us, the Cambridge Analytica scandal involves Facebook partners gaming the system to obtain data about Facebook users. At worst, it is a violation of community standards, i.e. claiming to be doing academic research. The research firm uses tools provided by Facebook to obtain the data. (The firm maintains that it disclosed to users that it was using the data for non-research purposes.)

In the present case, Facebook claims that a combination of coding bugs (i.e. unintended features) enabled unknown partners to gain access to the user's entire account. Not even that, but any accounts on third-party apps or websites that the user signs on to using Facebook.

 

How the breach was discovered

The current scandal demonstrates the value of the business intelligence/business analytics functions within companies. We were told that Facebook first realized that certain metrics were showing unusual trends, and upon investigation, they discovered the bugs.

This is entirely believable. That's what happens when you have good data reports. They surface anomalies. These then have to be investigated. These investigations are extremely tricky because all you know is the trends are different. There are a thousand reasons for the shift. The analyst's job is to establish a cause-effect. Especially since the development community adopted "agile" practices, all kinds of self-imposed changes are occurring all the time with no warnings. For a site as large and complex as Facebook, it takes a huge effort just to get a list of all site changes within some specified time window!

What complicates this situation more is that the vulnerability was traced to multiple bugs, not just one. I could imagine the twists and turns and the false alarms that were generated during the investigation.

The nature of this problem is no different from an investigator trying to chase down an e-coli outbreak, which is detailed in Chapter 2 of Numbers Rule Your World (link).

The data science community is guilty of talking down on the business intelligence function. There is a misperception that BI is for less skilled people doing boring things. The reality is there is more science in BI than in so-called data science (defined here as software engineering). Science, after all, is about figuring out why things are as they are. Engineers, by contrast, use our understanding of science to change the way things are.

 

You've been warned

In my debut Youtube video, released a few weeks ago, I explain how Facebook collects data about you. My biggest tip about protecting yourself is to not use Facebook to log onto other websites. Convenience is the drug that lures you into the trap.

Think about those one-password services. Instead of hundreds of password, you have one. So the thief needs only one password to assess hundreds of sites. Using Facebook to log onto other websites makes Facebook the centralized point of attack by the bad players.

Note that this is not limited to Facebook. Using Google, Yahoo, Amazon, etc. to log onto other services is no different!

If it's more convenient for you, it's also more convenient for your enemies. Remember that.

Also in the video, you will learn

• why did Facebook invent the Like button
• how do Facebook know what you are doing outside Facebook
• why not clicking doesn't mean you're not being tracked
• why you shouldn't use Facebook to log into other services

See the video here, and send me ideas for future episodes!

Think twice before you Waze

Among many commuters, Waze (acquired by Google) has become a staple - some might say, a drug. Waze is an app that finds you the "best" (i.e. shortest) route to your destination. It is famous for routing people through strange, local roads, and intuitively dubious routes (e.g. it once told me to drive through an airport). Because it takes crowd-sourced congestion data as one of its inputs, there are reports that some residents of previously quiet neighborhoods are sending fake signals into the system to discourage Waze from sending traffic onto their streets.

***

If you are a fan of Waze, let me pose you a few questions to think about.

Let's say you are trying to get from point A to point B. 

Let's say there are 50 different possible routes to get from A to B. Some of these are marginally different (e.g. take a left turn one block later) while others are substantively different (e.g. avoid a stretch of highway). 

At any moment in time (for example, at the time you enter your destination), one of those 50 routes is the shortest route between A (where you are) and B (where you are going).

Also, there are a bunch of other drivers also trying to get from A to B. 

If Waze sends all the drivers to this shortest route, then this route will no longer be the shortest route. 

So some cars will be sent down a route that is not currently the shortest route. Do these drivers know they are not on the shortest route? (Do you?)

Do Waze users know how the algorithm decides who to send down which route?

Should the software vendor disclose how they make decisions like this?

***

Of course, these GPS route planners have transformed our driving experiences in a majorly positive way. Gone are the days where I am juggling and folding a paper map while trying to steer the car and paying attention to the traffic around me. 

When you compare Waze to say a standard GPS route planner, what is the real difference?

***

P.S.

(a) If we allow the road system to run for a long term, at "equilibrium," one should expect all 50 routes to have the same travel time (but different amounts of traffic). If there is an identifiable shortest route, then some drivers will shift there, making some other route better.

(b) Almost all press coverage of the "bias" of algorithms are related to identity politics. However, the truly difficult issues related to bias have nothing to do with race, income, gender, etc. Here is an example of a fairness issue that should get much more attention.

Good job Bloomberg, and some comments on ad measurement

I'm continuing to see important articles as our journalists shine light on the tech companies. This Bloomberg article reveals a previously secret relationship between Google and Mastercard. It is important also as a reminder that the intense focus on Facebook is misguided - as other big tech companies engage in similar practices.

I'd recommend reading the article in full. Here I'd like to bring out a few key observations for you to chew on.

1) The payment processors have become part of the surveillance network that has been constructed by the tech industry. In the past (when I was working in that industry almost two decades ago), payment processors played the role of toll booths that happened to be a very lucrative business collecting cash as the transactions stream through. Users' purchase data were safeguarded and remained private.

Sometime in the last 5 to 10 years, the payment processors decided that the purchase data is a goldmine. They now monetize this data through mostly secret deals with all kinds of companies. Most card-users are not aware that their purchases are being disclosed to third parties.

An analogy is Gmail and other web-based email systems. Gmail in particular pioneered the concept of reading users' emails to extract information. It is now known that Gmail allows selected partners to extract all kinds of information from emails, such as shopping receipts, boarding passes, etc. We knew this because of the scandal about the Unroll.me service in which the service provider extracted shopping information from emails under the pretense of finding email lists from which users can be unsubscribed from.

 

2) Next time reporters hear the tech companies talk about "anonymized" data or hiding of "personally identifiable information", they really need to press the companies for details. Look out for clever word play. It is simply not possible to get value from these datasets without access to information about each person.

In this instance, Google is merging two datasets, the clickstream (what ads people clicked on) and Mastercard purchases, with the goal of claiming that person A purchased item X after clicking on some Google ad. The Mastercard data contain your card number, and/or your name. The clickstream data contain one or more of the following: your Google "cookie", your Google user name, or your email. You can't merge the datasets unless there is a common field. In this case, it will most likely be your name or email address. How could that be "anonymous"? Good question if you're asking it. And the reporters should be pushing on this point.

Further, if the data are fully anonymized, the potential utility of the data is much lower, and the price marketers are willing to pay for them is much lower. If I don't have your name and address, how can I send you a piece of junk mail?

 

3) While the article points out the violation of privacy of this partnership, it inadvertently legitimizes the highly dubious science behind matching the clickstream data and the Mastercard purchase data. The reporter described the program as "powerful" and "potent" but should have talked to third-party experts to understand this science better. I'll just list several major problems with this approach:

• As far as I know, Mastercard does not have line-item level details on our purchases. It may know that you spent $500 at Best Buy, but it doesn't know that that total includes an iPad and a TV, for example. So if the ad shows a TV, how does Google know that Best Buy purchase included a TV? The reporter claims that Google knows that someone bought "red lipstick" with her Mastercard. I find that dubious.
• Do you remember what ads you clicked on 30 days ago? 15 days ago? 5 days ago? This system counts any purchases made on Mastercard as long as 30 days ago as "caused" by an ad click.
• Does Google know what these users are doing on non-Google websites? For example, does Google know that the user who saw a Google ad 10 days ago also saw a Facebook ad for the same product 3 days ago?
• Does Google know anything about the users' interactions offline? For example, the user who clicked on the Google ad 10 days ago may have heard a radio ad, seen a TV ad, talked to a friend about the product, been handed a flyer at the mall, saw a piece of direct mail, etc.

The reporter suggested that Facebook, the other digital advertising behemoth, is also pursuing similar deals with payment processors. If Facebook were to adopt this same method for measuring ad performance, then both Facebook and Google would have claimed credit for the same purchase, as each would have matched that purchase to its own clickstream data. One solution to this particular issue is to merge the Facebook and Google data, which simply merges two surveillance networks into one extra-large surveillance network.

 

 

The side effect of automation is fake data and theft

There is hope in our business journalism! That's the message I'm getting as I delight in the current stream of articles that take a critical look at the inherent unaccountability of technology giants. These stories have been around for years, even decades, but better late than never.

Fake views/plays on Youtube videos (New York Times) This is a companion piece to a previous story about fake social-media followers (New York TImes magazine)

Uber drivers gaming the pricing algorithm (Wall Street Journal)

Google's Android tracks you everywhere whether you allow it or not (Bloomberg)

Finally some coverage of the shame of IBM's Marketing (Wall Street Journal)

As blog readers know, most companies in the Web/mobile ecosystem use an advertising-supported model. Even the most popular products like Youtube, Google Maps, and Facebook have not found it possible to charge users anything (when in theory, users derive value from such products). Such digital advertising, since the beginning, has been heralded as inherently more accountable because it was supposedly more measurable. Just look at the mountains of data that are generated by the very act of putting up ads on a website. You can even trace a user's final action, say the purchase, back to various interactions prior to the action. (This data is called the "click stream".)

The marketing on digital marketing centers on its "efficiency," achieved through automating the process, taking human beings out of the equation, and trusting the black box machines. This system is flawed even in its most theoretical form, assuming "complete data". The clickstream and other digital data have multiple problems. Assigning causality is highly suspect. I wrote about this years ago - for example, here.

The opacity of the black box machines offers not just efficiency but also shelter for fraud. Any kind of verification, quality checking, common-sense making slow the system down. The more efficient is the system, the easier and faster it is to commit fraud!

What kind of fraud are we talking about? Fake everything! Fake clicks, fake impressions, fake ads, fake viewers, fake plays, fake followers, fake accounts, fake likes, fake webpages, you name it. Since advertisers are paying money for clicks, impressions, ads, viewers, plays, followers, accounts, etc., and most advertisers have neither the incentives nor the nerves to look under the hood, they are paying real dollars for all the fakes.

The much-lauded efficiency of the black-box advertising model sits on a pile of fake data. The system is seriously flawed assuming complete data; it is worse when we admit that not only do we not possess complete data, but we are drowning in fake data.

Now, data scientists are hard at work building even more black-box machines that promise to suss out the fake stuff. To understand how this might work, think about spam emails, an analogous problem. In the machine-learning world, spam detection is thought of as a "solved" problem. Spam filtering algorithms have existed for decades. Think about how many fake emails continue to sit in your inbox, and in addition, how many not-fake emails land in your spam folder - and you can begin to see why technical solutions are not potent enough to drive out fraudsters.

***

Below are some comments on selected parts of the latest New York Times article on fake Youtube views (here).

• Youtube stars can make millions from advertisers by publishing viral videos. Google which owns Youtube makes tons of money by having people upload their content to the platform and charging rent for it. Advertisers ultimately pay for those views. Youtube stars, and by extension, Google, can make lots of money from fake views, i.e. bots (software) pretending to play the video. Third parties approach wannabes to sell them fake views and take a cut of the action.
• It's a cat and mouse game that the fraudsters are winning. They have the upper hand because the products are often designed without assuming nefarious players gaming the system. Adding software to deal preemptively with nefarious players slows down tech startups trying to build an initial market. Preventing bad players from acting badly imposes (the externality of) restrictions for good players, which is another disincentive to tackle the problem.
• Youtube officially claims: "fake views represent just a tiny fraction of the total." (later defined as 1 percent). This is a guess masquerading as fact. To know what fraction of the views on Youtube is "fake" requires knowing whether each view is fake or not. In theory, one can review a sample of the views, and then infer the fraction in the entire catalog from that sample. In practice, it is not easy to prove if a view is fake just from the data collected about the viewer's browser, OS, etc. A machine-learning algorithm will "predict" if a view is fake or not but such predictions should not be used as "ground truth".
• Legitimate organizations buy views, including marketing agencies, news organizations and musicians. As a former employee said, "View count manipulation will be a problem as long as views and the popularity they signal are the currency of YouTube." It's not just YouTube - we know that when teachers are compensated based on student test scores, they become complicit in abetting their students to cheat.
• This underground economy is quite well developed now, with wholesalers and resellers of fake views. If you are a reseller, you don't even need to know how to generate fake views. You just buy fakes from the wholesalers and charge a markup!
• The sellers of fake views are so sure they can evade Youtube's detection technologies that they are openly talking about their clients, business models, etc. to the press. Either they know those technologies don't work, or they know that Youtube isn't serious about cracking down.
• Several buyers interviewed by the Times said they truly believed the marketing spin of the fake-view sellers, that they were driving real users to watch their videos. This is a tad disingenuous. As one writer who purchased views pointed out, those purchased views led to no sales, and she couldn't get any data about who viewed the videos.

It's actually very easy to demonstrate fraud as the fake views are barely hidden. The article came with a great graphic, which does not appear to be addressed directly in the text. So let me explain.

The reporter purchased fake views from the vendor called "BuyYoutubViews". The top chart shows that close to 100% of those views ("in campaign") played the entire video (over 5 minutes long). The bottom chart shows that when they turned off the "campaign" i.e. stopped buying fake views, almost no one plays the video for more than a few seconds. (What is not shown is that the number of views probably plunged after the fake-views campaign was shut down.)

Raise your hand if you think "BuyYoutubViews" is finding real people to watch the video!

 

 

 

 

What about daughters of politicians?

Sometimes, the story is so good there is no need to embellish.

A medical school in Japan blatantly reduces test scores of women to reduce the proportion of admitted women.

It also awards bonus points to the politically well-connected.

The practice started in 2006, not 1906.

Read the story here.

What's wrong with the Dropbox research?

In a mailing list I subscribe to, some users were not happy about the academic research using Dropbox data collected on academics, as written up by the co-authors in Harvard Business Review (link).

In a nutshell, the researchers obtained "anonymized" "project-folder-related" data from Dropbox on university-affiliated accounts, and did some simplistic bivariate correlations, and proceeded to draw several conclusions about "best practices" on "successful team collaborations." This type of research is very common in this "Big Data" age, and I have already written extensively about its many challenges.

OCCAM Data

This Dropbox dataset has all five descriptors of the OCCAM characteristics of "BIg Data". It is Observational, seemingly Complete, with no Controls, Adapted from its original use, and Merged (with data from Web of Science). These characteristics cause many problems with the analysis, which I describe below. For more on OCCAM data, check out this post, and my other posts on OCCAM data.

Found Data

Implicit in their analysis - and most other uses of "found data" - is the assumption of complete data. The authors believe that because their data consist of tens of thousands of researchers and 500,000 projects, they must have all the data. In this case, the authors knew that there are other platforms out there but they waved away the inconvenience. This implies they believe they have all of the "informative" data.

The authors also assumed that (a) all relevant collaborative research involves putting all relevant files on one of the major online platforms (e.g. nothing on Slack or emails) and (b) all project collaborators use one and only one platform. Further, they assume that everyone has highly organized and structured folder directories within Dropbox for which an external person who knows nothing about the projects, or a machine, can infer its contents. These problems arise because the researchers did not start with a research question and design the data collection. They chose to adapt "found data" to their own objectives.

Ecological Fallacy, and Story Time

A typical conclusion is "People at higher-performing universities seemed to share work more equally, based on the frequency of instances that collaborators accessed project folders." Top universities are ranked by the aggregate performance of all teams (that use Dropbox, and are identified correctly). It does not follow that every team at a top university is a top team.

This is an instance of "story time." A piece of data is offered about something related, then while the reader is dozing off, it is linked to a conclusion that is not directly supported by the data. That conclusion is elaborated by a lot of words hypothesizing why it must be true. In this case, they say "It’s likely that more frequent collaborations led to positive spillover of information, insights, and team dynamics from one project to another." But they provide no evidence at all for this last statement. That's just a story.

Proxy Unmasking

Here's another one of the conclusions: "People at higher-performing universities seemed to share work more equally, based on the frequency of instances that collaborators accessed project folders." They make a conclusion about work allocation between different collaborators but what they actually measured was the relative frequency at which collaborators accessed the project folders in Dropbox. That's a proxy measure, and it's convenient based on "found data", but not a good proxy.

Xyopia

It does not appear that a multiple regression model was run. The presentation walks through apparently a series of bivariate analyses. The word control does not appear in the entire article. So this work suffers from xyopia - in each analysis, the one explanatory variable being analyzed is presumed to be the chief and only variable that influences the outcome.

Causation Creep, and More Story Time

The authors made no attempt to establish causality at all. They just interpreted every correlation as causal. So every conclusion is "story time". They print one analysis of the data, then they draw a causal conclusion that one would believe only when half-asleep.

***

People are also upset about data privacy.

1. It does not appear that the academic users understood that using Dropbox means they are part of research studies.
2. People don't believe the data are truly anonymized. It's pretty clear that the anonymization can be easily reversed. Just take the HBR article for example. If they removed the names of the authors but retained information about the authors as: one junior faculty at Northwestern Univ. Business School, one senior faculty at Northwestern Univ. Business School, and one employee of Dropbox - I don't think you can find another article that fits those criteria. So is that anonoymous?
3. It's unclear if how they "anonymized folders" or analyzed them. There are folders with highly descriptive names, there are folders with partially descriptive names that only the collaborators may be able to decipher, and there are folders with names that do not identify the project (e.g. old_work). If they converted all folder names to alphanumeric strings, then all information about the contents of the folders is lost. If they don't convert those names, then there are clearly privacy concerns.

It's clear that some kind of IRB review is necessary to approve Big Data research projects to make sure privacy is protected.