Big Data, Plainly Spoken (aka Numbers Rule Your World)

Uber data collection makes news again

This week, we are celebrating a company for not doing something outrageous.

There are reports (e.g. NPR) that Uber decided to pull the plug on one of its more invasive data collection schemes. If you had the Uber app, it was tracking your location for an additional five minutes after you finish your Uber trip, even if the app is closed. Now Uber says it will stop doing it.

Uber previously argued that the data of your whereabouts five minutes after you get off the Uber car can be used to "improve pickups, drop-offs, customer service, and to enhance safety."

This episode reveals a few things about mobile app technology. We learn that

• Apps are not really off when they are turned off
• Apps may be collecting all kinds of data about you and your phone 24/7/365, whether it is on or off (Previously, Uber has been caught tracking users even after they delete the app.)
• Unless the app developer discloses the information, the app user does not know what data are being collected by the app at any given time. This is a similar situation to not knowing whether the camera on your laptop is filming you or not.
• Won't do and couldn't do are two different things. For example, it's not a technical limitation that Uber did not track you beyond those five minutes. If they wanted to, they could have. So, users better trust the app developer when they say they won't do something.

As app developers say, if you don't like their behavior, don't use their service. In the case of Uber, it is a not an essential app so that statement is true.

The things you learn from a coffee menu footnote

The other day, I noticed that the Peet's Coffee near me added a footnote to their menu sign.

I'm talking about this note at the bottom:

Milk-based beverage calories calculated using 2% milk, except for Havana Cappuccino and Black Tie.... 2,000 calories a day is used for general nutrition advice, but calorie needs vary...

These few sentences really speak to why an analyst must know how metrics are defined before he or she can interpret them.

What I learned about measurement from this calorie estimation footnote:

• The number of calories for the same beverage depends on assumptions, such as whether one is using 2% milk, or no-fat milk, or whole milk
• Definitions have exceptions: here, Havana Cappuccino and Black Tie are exceptions - they didn't say in what way but I know those two drinks use condensed milk
• There may be other hidden assumptions
• The calorie metric only attains meaning when a reference level (daily calorie need) is provided
• The reference level is provided as a one-size-fits-all number; even the writer of this note realizes is too generalized
• The reference level is defined with another pile of hidden assumptions
• I wonder if the calorie counts come from formulas or some kind of lab measurement

Whether one believes the calorie estimation on restaurant menus depends on whether one knows the assumptions used to produce those estimates. Also, trusting a metric is just one step; one must then figure out how to use the metric for a purpose. 

Naive analysis is naive: debate about college admissions

Right now, the home page of New York Times online shows the following headline for me:

[Link to article]

The chart is an example of great data visualization. It is concise without being over-elaborate. For example, the axes are not labeled in detail but it doesn't take much effort to understand the message - that the proportion of whites and Asians are "over-represented" in "top" colleges while that of Hispanics and Blacks are "under-represented." The representation gap is "worsening" over time.

I have placed some words in quotes because those words are subjective even though they look like they are supported by data analysis. A good instance demonstrating that data analysis cannot be separated from the assumptions underlying them.

A key issue is the definition of this representation gap. The gap is defined using an underlying reference level, which is the ethnic composition of the aggregate U.S. population. Under this perspective, a college is at fault if the proportion of Whites or Asians in its student body is higher than the respective proportions in the population at large.

But the U.S. population is not an appropriate reference level. A more relevant reference level might be all college applicants or all college-age Americans.

The authors actually switches the reference level in the first few paragraphs. Right after the chart showing proportions of aggregate population, they showed a different set of charts showing proportions of college-age population. For example:

[The terms "elite" schools and "top" schools are not explained in the article.]

***

Which schools are included in the list of "top" colleges affects the conclusion. The detailed charts shown in the article itself show a lot of diversity amongst the schools. For example:

[Note that these charts show that even the numerator of the proportion is subject to definitional variability. The proportions are based on first-year students, not all students.]

Stanford and Berkeley are both located in the Bay Area so they are comparable from a geographical perspective. They are not comparable in other ways. One is a private university; the other is public. Berkeley has for a number of years based admissions on more objective metrics like test scores (after a ban on affirmative action was imposed) while Stanford uses more "holistic" criteria.

There are clear differences between ethnic composition by geography. For example, this analysis of 45 major cities by Priceonomics shows the proportion of whites ranging from less than 10% to over 50%, the proportion Asians ranging from 0% to 30%, and the proportion of Hispanics from under 10% to over 60%. These numbers are all based on aggregate population, rather than college-age population, or college applicants.

***

Aggregate statistics are useless when investigating this issue. This sort of naive analysis is frequently used in political discourse. But naive analysis is naive. In fact, naive analysis can be quite dangerous as it promotes an over-simplified view of the world.

In the case of college admissions, there is another side to it. For a number of years now, some Asian parents have been suing top colleges for "under"-representing Asians. In this perspective, they are unfairly treated because the admission rate of students with comparable credentials should be roughly equal, and evidence seems to show that Asians of comparable credentials have a lower admission rate than other ethnicities.

The crux of the analysis is driven by how one defines which groups to compare.

Chapter 3 of Numbers Rule Your World covers statistical concepts useful in thinking about this debate.

 

Gelman digested read

It's hard to keep up with Andrew Gelman, so let me point to some interesting recent posts from his blog.

Readings on philosophy of statistics (link): Andrew has a bunch of links of (mostly his own) writings about deep statistical issues. Science is about understanding how the world works, which involves questions of cause and effect, and randomness and unexplained variability. Data that can be observed are almost never sufficient to establish cause decisively but statistical theories can be drawn upon to make careful, principled conjectures. These statistical methods are not infallible, and are subject to abuses, both malign and unintentional. Recent work has uncovered that lots of results from all kinds of fields (psychology, social psychology, evolutionary psychology, medicine, cancer studies, etc.) cannot be replicated, raising concerns about abuses. Andrew - as well as commentators - compile a list of readings for those interested in this ongoing controversy.

An elementary error showing up in JAMA (link): misinterpreting p-values - every elementary textbook warns against such erroneous claims

Something for bird lovers (link) and for cat lovers (link)

Another one on Gelman's favorite subject - the garden of forking paths leading to over-confident statistical conclusions. I once summarized his arguments in a series of posts: 1, 2, 3

Some commentary on Mechanical Turk and the general issue of measurement and data quality (link) This is an important topic in Big Data. I will be writing about a study that looks at weather as the explanatory variable. Weather is derived by looking up someone's IP address and then the weather report at that IP address. One should ask how accurate the measurement of weather was for this study.

Did web scraping just receive a legal boost?

Something truly game-changing might have happened. According to TheNextWeb, a judge has told Linkedin (now owned by Microsoft) that it cannot stop people from using scripts to scrape public data off the Linkedin website.

I have mentioned this issue before - web scraping is in some kind of legal gray area. Web scrapers appear like human beings to a website, and these bots just exist to collect data off the website. It's clear that owners of many websites (Kayak, Linkedin, ESPN, etc.) do not like web scraping. They implement technologies to block such bots.

There are several legitimate business reasons for opposing web scraping. For example, a retailer like Amazon does not want competitors to know all of its pricing. The downside of convenience - 24/7 shopping - is that it reduces information asymmetry. In the pre-Internet days, a store can run special prices in certain locations without word spreading to other locations - but this is no longer the case. Information asymmetry allows profits to materialize.

Besides, the data might have been purchased for real money from some vendor. In this case, the vendor will forbid the buyer from publishing all the data, otherwise the vendor loses revenues when some potential customer scrapes it off the buyer's website. The buyer also has an incentive not to give way the data assets, say, to a potential competitor.

From the web scraper's perspective, the data are publicly displayed so why should there be a restriction on usage? For social-media sites, the data are shared by users, not generated by the site owners, and so ownership of the data is confused. When site owners include a prohibition on web scraping on their terms and conditions (knowing full well that most people do not bother reading them), this creates at minimum an ethical issue, and possibly a legal issue. However, with the Linkedin ruling, the cloud may be clearing.

 

 

A call for less automation, more transparency in digital advertising

The Facebook Split Testing feature is an example of nice math not producing nice results when released in the "wild". More on my experience in a prior blog post here.

I should be quite upset about the absurd results of my tests, not just because of the waste of time, but also because of a waste of money. During those botched tests, I paid Facebook dollars for each ad impression rendered. If I were a larger advertiser, I'd be exponentially more irritated.

The testing platform can be made instantly more useful by offering users some degree of control:

(a) let users impose a waiting period before the optimization algorithm kicks in

(b) let users manage the "learning rate", i.e. the speed at which the algorithm responds to the response rate differential

[Note: One issue with letting user control the running of this type of algorithms is that many users would fall prey to a narrative fallacy. It is fairly easy to convince oneself that any of-the-moment result conforming to one's belief is credible. This is the reason why many experimenters discourage "peeking." What I am saying is zero control is worse.]

The fact that the Facebook testing platform doesn't allow any user control is symptomatic of a destructive bargain between mainstream software designers and average software users. Facebook is by no means the only company following this paradigm.

Many software designers chase full automation. The ultimate goal of this paradigm is to remove human beings from processes. But human beings are nosy and meddlesome. If users are allowed to monitor the details of algorithms, they can't help but offer suggestions and corrections. To really get rid of human interference, the designers make the system opaque.

Some users like opaqueness. It releases them from accountability. You take the reports coming out of the opaque system, and regurgitate them to your supervisor. So long as these reports show good news, few will be questioned. If there are doubts, you just shrug since you don't have access to the data, or have any knowledge of its implementation.

***

The lack of user control frustrates me about the Facebook platform. I also have zero understanding of which specific algorithm is being deployed, or which allocation shifting rules are being implemented, or how the learning rate has been set. Their documentation does not explain much. I am pretty sure the algorithm failed me because it hasn't been tuned properly but I have zero knowledge of how it has been tuned, and with what data. The software designers dictated all of those things, and if you want to advertise on Facebook, that's what you must live with.

In effect, Facebook is asking users to give blind trust to an opaque process.

The pricing model is equally opaque. For example, an advertiser can run ads with the explicit goal of generating "leads". However, the advertiser is not allowed to pay Facebook based on the number of leads generated - the only setting allowed here is to pay by the number of impressions served. This model is not pay for performance, but pay for "effort."

I won't get into the controversy surrounding serving of ads - let's just say that there are many "impressions served" that no human beings could see. A recent Google experiment (link, tip from Augustine Fou) confirms that impressions are being counted even when the "programmatic" advertising platform is turned off! This is an industry-wide crisis, and it's not my intention to single out Facebook.

If the advertiser accepts the pay-for-effort structure, then he or she runs into another wall. How much will the advertiser be charged per (thousand of) impressions served? The answer is: the technocrats can't tell you - there is no list price, and the actual price varies widely. In my tests, the cost per thousand impressions paid was different every day despite everything else the same.

If pressed, Facebook and Google and pretty much everyone in the business will bring out their "weapon of math destruction" (to steal a term from Cathy "Mathbabe" O'Neill): the price is determined by a real-time auction.

Is there anything the advertiser can know about this auction? No. You can read some theoretical papers about generalized second-price auctions (which in the generalized case, do not reveal true values). But you shall not see any data related to the specific auctions that determined the specific prices that you are being invoiced. And of course, you shall not review the code used to implement such auctions. Under these conditions, the advertiser can be charged whatever price, and there is simply no way to contest it.

***

The digital platforms (Facebook, Google, everyone else) all report on themselves. They hold on to the data for dear life. How many impressions were sent? How many people click on the ads? How many people shared, liked, commented, forwarded, etc.? Lots of metrics are shown to users, none of which are verifiable. (If the ad is supposed to drive sales, and the sale transaction occurs on the advertiser's own website, and the advertiser's internal web operations team keeps track of such transactions, then those specific end-points can be verified, although the connection of any given transaction to an ad view or ad click is still tenuous.)

These platforms create their own virtual currencies (Facebook likes, twitter follows/retweets, etc.), they sell these virtual coins in exchange for real dollars, and they determine the exchange rate via an obscure auction.

***

Digital advertising as currently practiced falls far short of the measurability, accountability and efficiency promises made. If traditional (TV) advertising is considered wasteful, most of the money pouring into digital ads are also wasted.

The industry needs to reform by shifting the focus from automation to transparency.

For example, if one were to implement a real-time optimization platform for A/B testing, users ought to have access to the following:

1. Some degree of control of tuning parameters of the underlying algorithm
2. The run-time statistics of the algorithm should be reported, not just on the outcomes but also on operating metrics
3. Detailed data should be available, not just data summaries
4. The code for the algorithm should be reviewable

 

 P.S. This rant by an early investor in Facebook and Google is very timely and a great complementary read.

 

 

 

Know your data 21: another example of data sleaze, straight from your home

I first mentioned "data sleaze" in reaction to the article disclosing that Uber was secretly collecting certain user data. In my previous post, I define data sleaze as:

Data sleaze is the data about one's own customers that are obtained secretly by businesses, and then sold to the highest bidders, also in secret transactions. The production of data sleaze is frequently justified by giving services away for "free." However, running a business as a "free service" fronting a profitable espionage operation is a choice made by management teams, not an inevitability. Indeed, many businesses that have a proper revenue model also produce data sleaze.

Data sleaze is going to explode with the introduction of so-called Internet of Things (in short, tech firms intend to have every device in your home beam data back to home base all day long). A warning of what's to come is in this Gizmo article about Roomba, the vacuum-cleaning robot. Gone are the days when customers cough up their data in exchange for some "free" service. Roomba's product is not free. The robot collects data about homes while it roams around performing its advertised function - cleaning the floors. The device creates a detailed map of the home of its user. Roomba then turns the data over to third parties for a fee - that was what the CEO told the investment community until it got some blowback and he seemingly backtracked, claiming that he has been misunderstood, and Roomba merely "share the maps for free with customer consent".

The CEO said the company will never sell the data but its privacy policy allows it to do so. Consumers cannot verify his statement because business contracts are not shared publicly. Some models of Roomba have a camera and the company states that "[the camera] is separated from any wireless or wired transmission." It did not say that video images or clips of people's homes are not transmitted beyond the device. Video files can still be transmitted even if the camera itself is not connected to a transmitter directly. The concept of "customer consent" has been irreparably compromised by the prevalence of forced consent ("if you don't want to consent, we won't serve you.")

The article confirms a few things about Roomba:

• The company has collections of maps about people's homes through cameras and other means. Such maps are beamed to the "cloud."
• The maps that Roomba possesses are more detailed than those that are shown to users via the Roomba app
• Roomba is talking to multiple third parties about selling or sharing data with them
• The data will survive the dissolution of Roomba - as it is free to sell the data to an acquiring firm

Another good article about this is in the San Francisco Chronicle (link).

Burglars and criminals everywhere are drooling uncontrollably.