Big Data, Plainly Spoken (aka Numbers Rule Your World)

Power pose relapse

The "power pose" believers will simply not rest.

Daniel Engber at Slate reviews the latest attempt to kill the messengers - an article in the Boston Globe by a Harvard biologist. Sounds like the NYT Magazine article by Susan Dominus that I discussed here.

The common threads are (a) the unscientific use of selected anecdotes to paint a picture of "mobs" which an easy Web search will quickly disprove, as Engber did; (b) the citation of a few colorful adjectives as the entire proof of bad behavior, while conveniently ignoring similar language used to denigrate the reformers (again easily found online), a practice known as cherry-picking and widely seen as unscientific; (c) using personal attacks to condemn others of personal attacks; (d) no response to the scientific substance being debated while focusing on personalities.

From the start, the big problem with "power pose" is that its most important scientific claims cannot be replicated. Nothing has changed despite the many thousands of words used to "call off the revolutionaries."

 

A paradox of data collection

In talking to people about the invasive and pervasive data collection by businesses and governments these days, I often encounter the following view points:

1. Businesspeople who believe in the power of data believe that the data collection is for the benefit of their customers and humankind. For example, the carmakers think that the data will save lives - and at the minimum, the data help drivers manage their car's performance.
2. Many consumers say that they don't care about their data being collected, they have "nothing to hide", they are powerless anyway so they accept the situation.

If those views hold, then the paradox of data collection is: why is there a need to hide data collection practices? Why is consent hidden and coerced? Why are sensors hidden and silent?

***

If those views hold, then the following practice seems possible: businesses disclose clearly how and what data are collected, consumers are given a choice of whether they want their data to be collected or not, those consumers who don't care will sign the consent and benefit from the data collection; those who care about their privacy do not sign the consent, will continue to be served, but will not benefit from the data collection.

So what is the problem?

Automakers and data sleaze

Last year, I defined "data sleaze" as

the data about one's own customers that are obtained secretly by businesses, and then sold to the highest bidders, also in secret transactions. (link)

Washington Post just published a very useful article about auto-makers who are doing the following:

Carmakers have turned on a powerful spigot of precious personal data, often without owners' knowledge, transforming the automobile from a machine that helps us travel to a sophisticated computer on wheels that helps us travel to a sophisticated computer on wheels that offers even more access to our personal habits and behaviors than smartphones do.

That is exactly what I was talking about. Your new car is spying on you.

The volume of personal data being tracked is staggering - for example, they can monitor your body weight by putting a sensor in your driver seat. It's all in the article, go read it.

***

There are many problems about data sleaze that our society has failed to recognize and address so far. 

The data will be used against you.

Many companies claim that the data collection benefits you, while they make a profit selling your data to third parties. For example, the carmakers tell Washington Post that "the data is used to improve performance and enhance vehicle safey. The information, they add, will soon be able to reduce traffic accidents and fatalities, saving tens of thousands of lives."

How? According to Wikipedia, fatalities total tens of thousands per year so these carmakers are suggesting that having the data will eliminate all motor vehicle deaths! Is your car going to brake automatically because the sensor learns that you're speeding? If your car is skidding on ice, how is the data going to avert the crash?

If the sole purpose of data sleaze is car safety, then the data won't be sold to third parties for profit. Insurers and marketers are not involved in car safety. In fact, the data can reside within your car; why do the data need to be stored and sold, and tied to your personal information?

What harm can the data do you? Take as an example, you drive through high-risk neighborhoods to and from work each day - if the auto insurer learns that, it will hike your premium. That's not a benefit to you.

A retailer learns that you buy a gallon of OJ every week. How will the retailer monetize this data? You aren't getting a 30% off coupon from them since they know you are loyal. If they did that, they forego 30% of their sales for no reason. Instead, the retailer wants to increase your total purchase value so they might send you coupons for other items, or upsell you to a more pricey brand of OJ. The problem is that your interest and their interest don't usually align.

***

Data sleaze occurs behind customers' back.

The enabling device is the "terms and conditions." The carmakers say that their customers have given them "explicit permission" in language buried in thick service agreements. Everyone knows that few read these lengthy documents so in effect, permission is presumed but not granted.

Because the trading of the data is in the shadows, and because the businesses profiting from such data do not explain how they are using it (especially if they profit at our expense), consumers might believe mistakenly that the data collection is harmless.

***

The false claim of anonymity

Most businesses shy away from the word anonymity these days; and if they didn't, they probably are lying. Data sleaze is almost always tied to individuals. Imagine you are the insurer or the marketer paying the carmakers to use this data. You are using the data to adjust premium rates or send marketing materials. You must have personally identifying data, such as addresses, emails and phone numbers, to communicate these actions to the customers.

 

 

Can't say it better

Andrew Gelman wrote:

If you find a cool data pattern and present it as such, you probably won’t get much attention. But if you wrap it in the garb of scientific near-certainty, there’s a chance you could hit the media jackpot. The incentives are all wrong.

Neatly summarizes the state of academic publishing, and why "published in a peer-reviewed journal" doesn't carry the weight that it used to.

The new airline re-booking policy

 

With the snowstorms hitting the East Coast hard this year, lots of travelers are stranded at various cities away from home when their connecting flights are cancelled,. In recent years, the airlines have changed the way they re-book passengers on alternate flights.

Here's what happens today: the passengers get a notification from the airline, maybe before or while they are getting off the plane, that their connecting flight was cancelled due to weather. Don't worry, the notice proclaims, the airline has taken care of their rebooking needs proactively. What follows is their revised itinerary.

The process used to be different: stranded passengers formed a mob around the gate agents who patiently worked through the options to identify their most desired alternate flights. Seasoned travelers scrambled to reach the gate agents as quickly as possible because the best options (e.g. direct flights) would be snapped up in a second.

***

The new method is more civilized. It prevents the mob scene, and the protracted negotiations with the agents to find the least inconvenient option.

The new method shifts the task of re-booking from the agents to a software program running an algorithm. This solution has major advantages from an engineering perspective. All passengers are treated as pawns that can be moved around a chessboard. The software has the bird's-eye view of everybody's destinations, the future schedule of flights, the number and types of seats available on each flight, and so on.

The algorithm finds the best rebooking assignments at the system level, instead of the individual level. In the old world of first-come, first-served, agents made "greedy" moves, satisfying the current passenger, possibly at the expense of others still in queue. For example, after letting the current passenger onto a specific flight, an agent might later have to split up a family.

In the new method, passengers are not given a chance to fight for better options. They are coerced to accept the outcomes of the algorithm. The software optimizes for the “average customer.”

***

So is the new method better? It's not as clear-cut as the engineers would like us to think.

Accountability

There is one myth that both engineers and consumers share - the reification of software. Software (read: AI) is treated as a person with the ability to think for itself. In reality, the decisions did not pass from human agents to this AI “person”; the AI is an algorithm designed by software engineers to behave in a certain way. So the algorithm designers have effectively taken over the job of re-booking passengers.

But if you ask the algorithm developers, they would deny that they accepted that role. They live far from the scene of the chaos. At the airport, passengers could not complain to these engineers. They would be lucky if they found a gate agent to whom they could plead their case.

The new method is opaque. It feels like coercion. A friend recently told me that the notification even disclosed that passengers might end up in a fare class lower than the one they paid for – without possibility of getting compensated. One problem with opaque systems is that the feedback mechanism that is crucial for ongoing improvement is curtailed.

Whose interests

A software does not necessarily make better decisions than human agents. A real concern is what is being optimized? There are many possible "objective functions" for this problem, such as:

• maximizing profit for the airlines
• minimizing inconvenience for passengers, e.g. by minimizing average delay to final destination
• avoiding fines, or compensation for inconvenience caused
• satisfying a segment of the passengers that the airlines deem more important, such as those who pay more for the ticket, veterans, etc.

Further, an algorithm can be tuned to work well on average; or it can be made to create the most good for the most passengers; or any number of configurations.

The software explicitly or implicitly settles all these issues. But the opaqueness discussed above ensures that most if not all personnel involved have only a weak understanding of whose interests are being served. The airlines loudly proclaim that they are doing what’s best for passengers. But it is naïve to think that what’s best for passengers always align with what’s best for the airline’s business results. More often than not, those two objectives are in conflict.

Just take for example, the airline wants to avoid paying passengers any kind of compensation for the inconvenience – and actual cost – of an unplanned stay at the layover cities. Passengers prefer that airlines cover the unplanned costs. In the past, passengers could plead with the gate agents. That is no longer true when the passengers are sent one-way instructions from a remote computer.

Special Needs

A system-level optimization basically serves the "average user" well. Anyone with special needs will suffer. For example, if you are rushing to go to a funeral, the software will not know it; and even if it knows it, the engineer may not have anticipated the needs of funeral-goers. In the old world, the human agent might succeed in working out a solution, perhaps with the assistance of other passengers.

Fairness

The system is fair only to the extent that one understands and agrees to the trade-offs implicitly coded into the algorithm. Since those design decisions are almost never made public, customers have no way of knowing if they are fairly treated. This creates an environment in which unfair treatment can be covered up.  

***

From a mathematical perspective, the new way of rebooking passengers through a behind-the-scene algorithm is more orderly and optimal from a system-level perspective. However, this automated approach treats passengers as cattle, and can cause real pain because some passengers are bound to have individual needs. Also, the software approach lets businesses hide behind opaque processes, and could be exploited by those demanding special privileges.

2017 Review and 2018 Outlook on Cybersecurity

Another key recurring news item in 2017 was cybersecurity. This issue gets tied up with data because data, particularly personal data collected by businesses and organizations, have value, and criminals are using stolen data as ransom.

What happened in 2017

Throughout 2017, high-profile companies admitted to losing huge piles of private customer data. Yahoo! led off the parade, disclosing that billions of account data were stolen. At the time, the wayward Internet pioneer was selling itself to Verizon, which shrewdly used the negative publicity about the data breach to re-negotiate the acquisition price. In a matter of months, Verizon itself copped up to losing data of millions of customers. The next ground-breaking leak ensnared Equifax, one of several corporate giants making a business out of compiling and selling profiles on every American. This data theft is egregious as it involves much more private data than usual, such as driver’s license numbers and social security numbers. Not to be outdone, the Republican National Committee left the cyber-door open on voter data of all 200 million voters in the U.S. By year’s end, the bad kid on the block, Uber, revealed that it had paid off hackers to hide the theft of data on close to 60 million customers.

And those were just the big ones, the ones we are told about. The known incidents are the tip of the iceberg (this site has a long list of 2017 data breaches) as there appears to be no legal jeopardy for the screw-ups. Government regulators are absent from the crime scenes. Businesses and the cybersecurity industry are shaping the entire narrative.

Also worth mentioning is the cover-up and profiteering in the aftermath of data breaches. In several cases, executives are suspected of personally profiting from these mishaps by selling shares prior to the public announcements. If the allegations are true, the executives violated insider-trading laws but don’t count on law enforcement.

 

What to expect in 2018

Some first questions might be:

• Will our cybersecurity improve in 2018?
• Will there be fewer data breaches?
• Will businesses win the war against cyber-criminals?

These questions are too easy -- the answers are no, no and no. A better question is: Will 2018 be the year in which businesses finally get serious about cybersecurity?

Truth be told, I did not write much about cybersecurity on this blog in 2017 but I will be covering the topic in 2018. To give you a taste of what’s to come, I see cybersecurity as a problem created by the tech industry, therefore unsolvable from a technological standpoint, thus the demand for cybersecurity solutions is bottomless. No company can get serious about cybersecurity because it’s not a solvable problem.

Here’s a useful analogy: they are selling you flood abatement solutions all the while they are clearing the vegetation, which causes more flooding. They say that the devegetation is for the greater good, as it makes space for building new homes, which relieves over-crowding. They sell you a dam, which you build. Subsequently, as the flooding gets worse, you have to reinforce the dam, over and over again.

Long time ago, the technology industry decided that our cybersecurity is secondary to their other objectives such as convenience, automation, speed, utility, profits. Many technological decisions involve a trade-off between cybersecurity and one or more of those other objectives.

One of the biggest digital infrastructure trends in recent years is the migration of data to “cloud” storage. Instead of businesses storing our data in proprietary computers they own and maintain, they upload the data to huge networks of shared computers run by third parties such as Amazon, Microsoft and Google.

This paradigm shift yields a wide array of benefits to consumers. For example, the data are located closer to the end-user, reducing time we spend waiting for things to load (equivalently, computations to finish). Tasks can be split up among several computers, and run in parallel, accelerating computations, and thus decisions. It is also much easier to exchange data between companies since these cloud services have open interface protocols, unlike proprietary computers that have heavily guarded access points. The cloud also supports a high degree of automation – computers can communicate via bots without humans intervening.

But those same features create opportunities for cyber-criminals. If it’s easy and convenient for the good bots, it is easy and convenient for the bad bots as well. Cloud services require data to be replicated and stored on multiple computers, and they make the data travel. The more copies there are, the more trips they take, the higher chance of data being stolen.

Cloud services are not the sole cause of data breaches. I am just using it as an illustration of the two-sidedness of technological decisions. The tech industry has made a bet that consumers will accept those benefits at the expense of cybersecurity.

So, in 2018, we should see more data breaches, and an expanding cybersecurity industry. Don’t hold your breath for the ultimate solution. The Wall Street Journal recently reports that governments might lay down the law - it sounds like Europe might have to take the lead. Governments, though, are compromised by their own desire to get a hand on that data.

 

2018 Outlook on Fake News

Happy New Year to my blog readers!

I am starting the new year with a series of posts: each reviews a key issue covered on this blog in 2017, and I will discuss what might be in store for 2018.

First up is FAKE NEWS.

What happened in 2017

If I had a vote, I’d have chosen “fake news” as the phrase of 2017. Fake news is like that famous non-definition of pornography, you know it when you see it. It is this rare chameleon that pleases both parties equally. First, it was the Clinton supporters who claimed that voters were duped by “fake news” generated by Republicans or related entities. Then, when Trump became President, he seized the phrase, and made it his own – now, liberal-leaning news organizations are the “fake news media” who disseminate false information to the American public.

The term “fake news media” doesn’t make much sense – how does one “fake” a medium? But by adding the third word, the President has shifted the context of the issue from the contents of the news to the creators of the news. He portrays an awfully naïve, black-and-white world in which a media company is either “fake” or not “fake”, where all news reported by an alleged “fake” media company are “fake news.”

The emphasis on authorship runs against the recent current in media consumption. The invention of social media and user-generated content – much of which written pseudonymously or by people without name recognition – has tapered our awareness and appreciation of who’s doing the writing. When we consume Yelp or Amazon reviews, we barely register who authored them. Mainstream name-brand media has been withering before the President ironically saved them from the quicksand of online anonymity.

Suddenly, the technology giants also found themselves in the spotlight – for their alleged role in spreading “fake news.” In particular, Facebook and Google act as gatekeepers of online content. As startups, they cultivated a brand image of being objective, non-commercial, judgement-free “organizers” of information.

The “fake news” controversy has spotlighted that in later life, Google and Facebook morphed into gigantic advertising agencies, reliant on earning advertising revenues from commercial partners. The “eyeballs” of their users are the new oil. They must keep users coming back and sticking around, like drug addicts.

Advertisers want eyeballs, and Google and Facebook are extremely efficient at delivering them, through a number of tools such as “collaborative filtering” algorithms that control what individual users see when they search the Web or open their social media accounts, and A/B tests used to discover the conditions under which users interact with advertising messages. Needless to say, a lot of advertising messages can properly be called “fake news.”

Machine-learning algorithms fundamentally measure “success” based on view counts, which can easily amplify “fake news” through an well-known echo-chamber effect. Additionally, an army of “optimization experts” is available for hire to manipulate the output of those algorithms. The various futile attempts to “get rid of fake news” has awoken users to the amount of control Facebook, Google and similar companies exercise over one’s reading materials.

What to expect in 2018

The controversy over “fake news” and “fake news media” will not end in 2018. The President will continue to call out anyone daring to disagree with him as “fake news media.” Facebook and Google will roll out and roll back various “solutions” to combat “fake news” but no silver bullet will be discovered. The two tech giants will remain gargantuan advertising agencies seeking to monetize our attention: they won’t find enough paying customers to wean themselves off from advertisers.

The good news: the failed attempts to control “fake news” will teach us a lot about the nature of the search/discovery algorithms that control our online lives. Developers of these algorithms will benefit greatly from the increased scrutiny, and need to ply open the black box. We will appreciate that algorithms are neither objective nor judgment-free.

Three dead bodies will be resurrected: fact-checking, authorship, and expertise, in that order. Media companies will come under pressure to reinvest in fact-checking; authorship will become a key weapon in the fight against lies; and consumers will re-discover the value of experts, hopefully before the end of the year.