Big Data, Plainly Spoken (aka Numbers Rule Your World)

The hate campaign against negative reviews

A friend sent me this New York Times article about a "hate campaign" orchestrated on the book reviews website, GoodReads (acquired by Amazon a while back, no surprise), that allegedly suppressed sales of a new book.

I'm definitely a biased observer here since I have authored two books, and am affected by online book reviews. I also have emotional responses towards negative reviews, and like other humans, I exercise far less scrutiny of positive reviewers. It's a difficult issue to grapple with, and instead of offering conclusions, I want to suggest some topics for discussion.

(This is an example of a 1-star review on Amazon. When I wrote this post, the GoodReads site was malfunctioning and no reviews would load for me on multiple browsers. In my experience, GoodReads reviewers give fewer stars relative to Amazon reviewers.)

 

(A) The gripe that the negative reviewers published their opinions even before the book's publication date.

By itself, this is an invalid complaint, because publishers routinely send out advanced copies to friendly reviewers who are expected to help generate buzz prior to and leading up to the publication date. The real gripe is that they don't want to hear negative buzz, but if in an alternate universe, it's not a "hate campaign" but a "love campaign", I don't think this gripe would emerge.

Should there be an embargo on all reviews before the publication date?

The novel was published on June 6, 2023. The first review on Amazon was on June 6, 2023. Of the nine reviews posted on the first day, four were 5 stars, three 4 stars, and none below 3 stars. As of this writing, about 25% of the total reviews are below 3 stars. Who are the first-day reviewers? Should there be a reasonable X number of days to allow people to buy the book, read it, and write a review?

 

(B) The gripe that the negative reviewers objected merely to the outline of the book, and have not read the book itself.

According to the journalist, after the friendly reviewers put up a summary of the plot of the book on GoodReads, some unsettled readers wrote bad reviews because they hate the concept. This gripe is more valid. But... realistically, how many of the people who write reviews (including positive ones) have read the book? How much of the book must be read to be considered "qualified" to have an opinion? Did the famous people who put up blurbs on the book covers read the books first page to last?

The same question can be asked of not just early reviewers but any reviewers.

 

(C) The gripe that GoodReads do not ban these allegedly invalid reviews or reviewers.

The problem is how to set the censorship criteria. Should the authors and their publicists have the right to decide which reviewers or reviews are invalid?

Take, for example, the criterion that GoodReads should ban reviewers because they have not read the book. How is GoodReads going to know that? Do the publishers send GoodReads a list of names to whom they have sent advanced copies? What if one of the advanced readers pass the book on to someone else, who then write a review after reading the book?

If someone puts up a positive review after reading just the first 10 pages, is the review valid?

Take the other example of banning all reviews before the publication date. Surely, the publishers didn't mean that because they actively seek out advance praise. Emphasis on the word praise.

My own favorite censorship criterion is to delete all content-empty reviews. You routinely see reviews that have no value at all - "This ia a wonderful book", "This book is terrible. I hate it!" Having read the review, I have gained no appreciation for whether the reviewer read one page of it. But be careful what I reap! There are at least as many content-empty positive reviews as there are negative ones.

Here are two examples from Amazon:

Should online platforms censor content? This is a challenging question.

***

The journalists lamented that the book in question sold only 1,000 copies in the first 10 days, attributing the supposedly poor performance to the "hate campaign". Too bad the readers who purchased the book did not organize a "love campaign" on GoodReads.

 

 

Book review: Don't Trust Your Gut

Seth Stephens-Davidowitz has a new book out early this year, "Don't Trust Your Gut", which he kindly sent me for review. The book is Malcolm Gladwell meets Tim Ferriss - part counter intuition, part self help. Seth tackles big questions: how to find love? how to raise kids? how to get rich? how to be happier? He invariantly believes that big data reveal universal truths on such matters.

In 2013, I speculated about the challenges faced by big data analysts in this piece for Significance. Big Data has become a real thing in the decade since its publication. Seth's book interests me as a progress report on the state of "big data analytics".

If there is a common core of this style of analytics, as demonstrated by Seth's examples, I'd include the following elements:

a) Big Data is less defined by the amount of data, but by several characteristics I previously labeled OCCAM (see this post).

The data are typically collected by passive observation (e.g. tax records, dating app usage, artist exhibit schedules). Meaningful controls are absent (e.g. no non-app users, no failed artists). The dataset is believed to be complete. The data aren't specifically collected for the analysis (an important exception is the happiness data collected from apps for that specific purpose). Several datasets are merged to investigate correlations.

b) Much - though not all - of the analyses use the most rudimentary statistics, such as statistical averages.

This can be appropriate, if one insists one has all the data, or "essentially" all. An unstated axiom is that the sheer quantity of data crowds out any bias. This is not a new belief: as long as Google has existed, marketing analysts have always claimed that Google search data are fully representative of all searches since Google dominates the market. (Seth's previous book covers his analyses of Google search results.)

c) If the analyst incorporates model adjustments, these adjusted models are treated as full cures of all statistical concerns.

The last few chapters on activities that cause happiness or unhappiness report numerous results from adjusted models of underlying data collected from 60,000 users of specially designed mobile apps. The researchers broke down 3 million logged events by 40 activity types, hour of day, day of week, season of year, location, among other factors. For argument's sake, let's say the users came from 100 places, ignore demographic segmentation, and apply zero exclusions. Then, the 3 million points fell into 40*24*7*4*100 = 2.7 million cells... unevenly but if evenly, each cell has an average of 1.1 events. That means many cells contain zero events. By the magic of model adjustments, all cells have non-zero estimated activities and associated happiness quotients. The estimates in many cells reflect an underlying model that hasn't been confirmed with data - and the credibility of these estimates rests with the reader's trust in the model structure.

I observed a similar phenonmenon when reading the well-known observational studies of Covid-19 vaccine effectiveness. Many of these studies adjust for age, an obvious confounder. Having included the age term, which quite a few studies proclaimed to be non-significant, the researchers spoke as if their models are free of any age bias. Taking this line of logic further, such a belief implies that no one should bother with running expensive randomized clinical trials because a self-selected trial coupled with post-hoc regression adjustments is a perfect substitute!

d) A blurred line barely delineates using data as explanation and as prescription.

Take, for example, the revelation that people who own real estate businesses have the highest chance of being a top 0.1% earner in the U.S., relative to other industries. This descriptive statistic is turned into a life hack, that people who want to get rich should start real-estate businesses. Nevertheless, being able to explain past data is different from being able to predict the future.

A kind of feedback loop is also at play. The average odds of getting rich surely depends on how many people are entering the industry. So if people read Seth's book, and rush into real estate, the data may no longer rank this industry as best.

e) Most of the featured big-data research aim to discover universal truths that apply to everyone.

For example, an eye-opening chart in the book shows that women who were rated bottom of the barrel in looks have half the chance of getting a response in a dating app when they messaged men in the most attractive bucket... but the absolute response was still about 30%. This produces the advice to send more messages to presumably "unattainable" prospects.

Such a conclusion assumes that the least attractive women are identical to the average women on factors other than attractiveness. It's possible that such women who approach the most attractive-looking men have other desirable assets that the average woman does not possess.

It's an irony because with "big data", it should be possible to slice and dice the data into many more segments, moving away from the world of "universal truths," which are statistical averages, said differently. In the last chapter of the book, Seth describes one study that tackles this problem.

***

I wish Seth had devoted more space to explaining model adjustments and statistical issues, and discussing whether the remedies are sufficient. His target audience is clearly people with minimal mathematical training, and I understand his decision not to open such cans of worms. The references at the end should satisfy those interested in exploring further.

As with Gladwell, I recommend reading this genre with a critical eye. Think of these books as offering fodder to exercise your critical thinking. Don't Trust Your Gut is a light read, with some intriguing results of which I was not previously aware. I enjoyed the book, and have kept pages of notes about the materials. The above comments should give you a guide should you want to go deeper into the analytical issues.

I think there is a lot more that can be done with big data, we are just seeing the tip of the iceberg. So I agree with Seth that the potential is there. Seth is more optimistic about the current state than I am.

Some things of interest

My review of Jim Albert's book Visualizing Baseball is on the sister blog. As I mentioned in the review, I thought a more appropriate title should be "Fast Intro to Baseball Analytics". Thus, the book may be of interest to some of the readers here.

***

Over the past year, I've been experimenting with the video medium. Here are some clips on various topics:

Analyzing observational data, containing a good example of using critical thinking

Alarming college admissions statistic, a nice example of data driving behavior in a feedback loop

Did you betray your friend today?, which explores how data flow through businesses

Two-part exploration of route optimizing apps like Waze: why they might not give you the fastest route, and why these apps are still great

To keep track of future releases, subscribe to our channel.

 

The discontent of engineering uncertainty

Andrew Gelman likes us to "accept uncertainty and embrace variation" (link).

I was lucky to earn my Operations Research degree in a department filled with professors who have advanced our thinking about stochastics, learning from the likes of Erhan Cinlar, John Mulvey, Warren Powell, and Rob Vanderbei. In operations research, much of the early seminal work deals with "deterministic" problems in which the data are assumed to be fixed and available, but most real life problems involve data that are highly variable or missing. Replacing the distribution of values by their averages allows those deterministic solutions to be computed although it is widely known that such solutions are not optimal. In other words, we must embrace variation.

These thoughts came to mind as I started reading the Algorithms to Live By book by Christian and Griffiths. In the first 20 pages, they kept returning to the altar of deterministic, precise engineering solutions.

The book begins with the classic optimal stopping problem, in which one is faced with a sequential search problem at each step, one can either make the decision and stop searching, or delay the decision and keep searching. House buying is an example of such a problem. The authors pronounce that there is a provably optimal solution of spending first 37 percent of the house search period delaying the decision, after which one waits for the next house that is better than the previously seen.

They eventually do admit that this 37-percent solution is optimal only under a set of highly unrealistic assumptions, such as that one can never bid on a house one passed over before, and that the seller will always accept one's offer.

It's not the over-selling of the math that annoys me. Assumptions can be relaxed and complexity added to the base scenario. It's the constant harping on 37 percent. It's not 35 percent, it's not 39 percent. It is precisely 37 percent, and you better believe it!

No one should believe it because another huge assumption in these "provably optimal" solutions is that the data supplied to the problem are available, and precise. That's far from the truth! In a variant of the optimal stopping problem which the authors also describe, the decision-maker is supposed to have "full information" about the value of the house, that is to say, one knows the value of each house being viewed, expressed as the precise percentile in the distribution of all house values!

This leads to the hair-raising moment in which the authors declare on page 21, "if the cost of getting another offer is only a dollar, we’ll maximize our earnings by waiting for someone willing to offer us $499,552.79 and not a dime less." (By now, the problem setting has flipped to selling rather than buying a home.)

As formulated, the problem yields a "closed-form" formula, which spits out an answer to two decimal places. If one accepts uncertainty and embraces variation, one would not care about dimes.

***

I still plan on finishing the book, and I will write a detailed review when I finish. This post is not a refutation of the entire book. It's quite common to take formulas as god-given objects. Between unrealistic assumptions and uncertain data, one can only hope for general guidance and approximate solutions to these problems.

 

Book review: Everybody Lies by Seth Stephens-Davidowitz

Seth Stephens-Davidowitz has written a fascinating book calling for social scientists to use data collected by Google or Facebook in their research. This is a controversial issue, and if it weren’t so, it wouldn’t warrant writing a full-length book about it. Google does not release publicly its search data, but provides some pre-processed and aggregated statistics through services such as Google Trends and Adwords. Researchers who use this data do not have control over its collection or processing. (Seth previously worked at Google, and has written some columns for New York Times.) Facebook does not publish its data either, although social media users have a lower expectation of privacy.

Such big datasets come with a set of knotty problems, which I have previously summarized as OCCAM. In addition to having little control over its origin, the researcher’s purpose typically diverges from that of the data collector. The data is found or observed and not usually experimental. It is often treated as “complete” or essentially complete by the researcher, which is an assumption, not a fact. In stages of merging in other data, the researcher introduces inevitable errors. Here is my previous post about OCCAM datasets (link). It is not surprising that classically trained scientists have reservations about such datasets, especially if they are interested in causal mechanisms. Nevertheless, I agree with Seth that we can make progress on solving these problems if we start taking them seriously.

In writing the book, Seth carried out a number of mini-studies using mostly Google Trends data. Here are 8 things I learned from reading Everybody Lies:

1. Some people use search engines as confessionals. They type complete sentences like “I am sad.” or open-ended questions like “Is my daughter ugly?”
2. People assume machines (like the Google search engine) will keep their secrets. For sensitive topics, Google may generate more honest data than surveys. There are many questions asked to Google that I’m sure people won’t pose to a librarian.
3. Google searches for “Obama” is frequently paired with “kkk” and the “n” word. The prevalence of racist searches does not exhibit a North-South divide – it’s East-West.
4. As President of Harvard, Larry Summers spent quite a bit of time brainstorming with Economics PhD students on how to beat the stock market using new data. (And they came up empty-handed, or so they say.)
5. Anthony Weiner got rejected from Stuyvesant High School (famous NYC public school), missing the cutoff score in the admissions test by one point
6. Some economists found that going to Stuyvesant conferred no meaningful benefit to one’s career – at least, this is the case for those who attain a score close to the cutoff in the admissions test.
7. There are 6,000 searches on Google a year for “how to kill your girlfriend” while there are 400 murders of girlfriends.
8. “Big data” does not provide any insights that surveys can’t at the aggregate level so people slice and dice the data to examine “micro” segments, which means they are analyzing a huge collection of small data sets

The research mentioned in Seth’s book come out of the Economics discipline primarily, and can be considered in the tradition of Freakonomics (2005). There are examples of natural experiments, as popularized by Steven Leavitt. Seth brings the coverage up to the current trends, describing regression discontinuity, field experiments, and other techniques favored by econometricians at the moment. For those interested in what happens after Steven Leavitt, this is a good place to start.

Reading Everything is Obvious by Duncan Watts

In his book, Everything is Obvious (Once You Know the Answer): Why Common Sense Fails, Duncan Watts, a professor of sociology at Columbia, imparts urgent lessons that are as relevant to his students as to self-proclaimed data scientists. It takes only nominal effort to generate narrative structures that retrace the past, Watts contends, but developing lasting theory that produces valid predictions requires much more effort than common sense.

Watts’s is a perfect foil to the pop sociology books such as The Tipping Point and Outliers. Many a time while reading Everything is Obvious, I was reminded of Malcolm Gladwell by a passage here or there but the lessons drawn by Watts are decidedly more complex and nuanced.

Several chapters in the book begin with stories that could easily have come from Gladwell’s pen. In Chapter 9, Watts recalled the night an NYPD veteran, James Gray, eventually killed a family while driving drunk.

What happened next isn’t completely clear, but the record indicates that as Officer Gray drove north on Third Avenue, under the Gowanus Expressway overpass, he ran a red light. Definitely not good, but also perhaps not a big deal. On any other Saturday evening, he might have sailed right on through and gotten safely to Staten Island…

Using this case study, Watts shows how society cares more about outcomes than processes. For to replay the scenario in one’s mind’s eye is to realize that the unfortunate victims would have been safe if any number of events would have intervened causing Officer Gray to be “late” to the scene by just one second.

I imagine Gladwell would have embraced the James Gray story and folded it into one of his crowd-pleasing narratives wrapped around some simple truth about humans. Gladwell’s universe of truths such as “stars are made, not born.” One anecdote, a couple of research papers, a trifle of interviews - Gladwell’s ammo has dazzled and swayed the public, who crave broad-brush descriptors of our world.

Thankfully for this reader, Watts does not go there. The milieu of Everything is Obvious is far more complex. Watts resists reducing sociological phenomenon to simple terms. His stories tend to leave readers hanging; we learn that researchers have not quite figured out the goings-on. The James Gray story, Watts explains, is about how bad deeds can have harmless outcomes, good deeds can have bad outcomes, bad outcomes can arise from harmless deeds, and so on. Watts challenges us to think more broadly, behind the outcomes unto the processes. On this very issue, I wish sportscasters would read Everything is Obvious, and then they might cease calling for the coach’s head when he ordered a pass play that failed on the last play of a football game, with his team inches from the goal line. The coach made a bad call because the play failed but it would have been celebrated as the clinching moment on the coach’s resume had a touchdown been the outcome.

Watts is not afraid to admit when sociologists have no answers. Such indeterminacy may frustrate some readers, or startle them. Popularizers of science, following the tradition of Gladwell, have invented a world high on consensus and low on uncertainty. In Everything is Obvious, Watts, who converted to sociology from physics, repeatedly describes failed theories and false starts.

The book also works well as an overview of the field of sociology, its key research areas and avenues. Many of the topics covered are thoroughly modern, such as the rise of crowd-sourced data, the running of large-scale experiments, the complexity of systems, and the futility of chasing after a “grand unifying theory” of sociology (and by extension, economics, and other social sciences).

Watts and his associates made valuable contributions to some of these subjects. He holds a research position at Yahoo! Research Labs at the time the book was written, and thus has access to industrial-grade systems and datasets. Several chapters of the book are concerned with the nature of making predictions. I particularly enjoy the materials about choosing one’s battles - the idea that some things are just not meant to be predicted. This important lesson has been laid to waste in the Big Data age. There are too many data scientists who issue pointless predictions just because they have datasets that can be fed to algorithms.

I share Watts’s obsession about measuring the accuracy of predictions. One of his research papers investigated “predictive markets,” concluding that they were not meaningfully better than experts, contradicting the unwarranted hype about predictive markets (and its crowd-sourcing ilk). The only surprise here is the miserly mention by the media, who otherwise eagerly publish any hearsay promotion about wisdom of crowds.

The title of the book is worth another moment’s thought. Everything is obvious, Watts says, when viewed in hindsight. The hyperventilation after each “terrorist” attack is instructive. Inevitably, a reporter files a note, disclosing that the accused killer has left a warning on his Facebook account. Such hindsight is taken as evidence of law enforcement’s failure to secure the nation. If right now, we were to flag every Facebook message that contains a threat of violence, we might have to investigate thousands, if not hundreds of thousands, of people, nearly all of whom will never become real killers. The menacing Facebook note becomes known only after its writer has committed a heinous act.

By its nature, Watts’s book does not read as quickly as your average pop sociology book. It contains quite a bit of philosophising, and urges readers not to take things at face value. Common sense is helpful, but limited.

Numbersense, in Chinese and Japanese

This is a cross-post on my two blogs.

The new year brings news that my second book, Numbersense: How to Use Big Data to Your Advantage has been translated into Chinese (simplified) and Japanese. Here are the book covers:

In Chinese, the title reads: "Say No to Fake Big Data". Captures the sentiment of the book pretty well, I must say.

I have no idea what the Japanese title means. Perhaps a reader can help me out here.

***

The Japanese version is available here or here.

The Chinese version is here.

The English version is here.

Two new reviews of Numbersense

I've learned one thing about book readers. There is a lag between buying a book and reading it. In fact, I imagine an author has two battles to win: one is at the bookstore (or Amazon) getting you to purchase the book; the next is to get you to pull out the book from your shelves and start reading it. I mean, what am I to say? I own shelves of unread books.

Anyway, reviews of Numbersense are trickling in. The following two long reviews make me happy, as these readers get the points I'm trying to get across.

Here's Thinkanator's review (link). He apparently is tired of the heavily-groomed media coverage of Big Data, and wanted the world to know about "Real Big Data," the less glamorous side of the job. He noted: "In the epilogue Kaiser shares one of his own Real Data Science stories and I found myself nodding my head and saying, “Yup, that’s how I spent several days in the last couple of weeks!”

Thinkanator probably wrote a better summary of the book than I have: "Numbersense is a wonderful and accessible book that consists of a series of stories about data that illustrate how to think about the kinds of statistics you read about on a daily basis. The emphasis isn’t mathematical, it’s more about when you should think, “Hmmm… that doesn’t sound right”, when you hear some statistics thrown around."

***

Veronique Frizzell wrote not one but three posts about Numbersense.

Post 1, Post 2, Post 3

In Post 1, she highlights the sections of the book that she enjoys most. In Post 2, she applies the Groupon analysis to her own business. In Post 3, she draws lessons from the chapter on unemployment statistics.

***

Get your copy of Numbersense here.

 

 

Imprecise machines mess with me

Just a little while ago, I showed an example of imprecise algorithms and how it causes incorrect historical facts to be promulgated. The point is not that algorithms are scary things but that we should not confuse efficiency with accuracy (or truth).

So this past week, I have another encounter with imprecise machines, and this time, it's personal.

***

If you go to Amazon right now, and search for my name in quotes "Kaiser Fung", you will get several versions of my 2010 book Numbers Rule Your World, including the recently published Chinese translation but you will not see my 2013 book Numbersense at all.

If you instead search for Kaiser Fung without quotes, the first match is Numbersense, followed by the older book.

To me, this is a clear mistake.

However, Amazon doesn't think so. This is what the customer service rep wrote me:

I understand your recent book "Numbersense" is not appearing in the search results when you search with your name "kaiser fung", including the quotes.

When you use our search engine to look for books, our system attempts to find the products you're most likely to be looking for based on the words you entered. Our search methods go beyond simple keyword matching and may also be using information not visible on the search results page, including attributes provided by the publisher.

So apparently, people who search for my name in 2013-4 are looking for my 2010 book instead of my 2013 book. In addition, my publisher has given them attributes to suppress my recent book from the search results.

Search results for books may be based on the text of each book, not just its title. That's why you may sometimes see results you weren't expecting.

I don't even understand what this sentence could mean, in the context of the name of the author.

I regret that we haven't been able to address your concerns to your satisfaction.

We won't be able to provide further insight or assistance for your request.

Thank you for contacting us.

This is rather shocking coming from the gatekeeper of most book sales.

***

Needless to say, this type of error costs authors as some people won't find the book. Yet, Amazon is unwilling/unable to fix the issue. Are there any readers here who have insight into why this is happening and how I might be able to correct this error?

Numbersense News

Here are a few links to recent coverage of Numbersense:

Arati Mejdal has an interesting list of 20 books, which should interest my readers: link

Albert Cairo selects it as one of his favorites of 2013: link. Pay attention to his recommendations for visual design books.

Rakesh Arya writes a review of the book, concluding: "The examples are quite apt with the topics and easily understood as they are explained in simple language without using the statistical jargons. It’s a good-read for those who are interested in behind-the-scene-theories and like to crunch numbers to find out the truth. But obviously it’s not a casual read." link.

***

In related news, my first book, Numbers Rule Your World, appeared in China (Mainland) this fall. I'm proud to be published by the press of Renmin University, one of the top schools in China.

There is a really nice article about the book. I'll translate it for you when I have time. If you can read Chinese, it's here. The title of the article is: "In the Big Data era, how should you analyze data?"

The book was chosen as one of the top Finance/Business books in the Chinese press. (link)

***

To gift Numbersense or Numbers Rule Your World to your data-loving friends for the holidays, you can click any of these links:

 

Amazon: Numbersense

Barnes and Noble: Numbersense

 

Amazon: Numbers Rule Your World

Barnes and Noble: Numbers Rule Your World

 

Barnes and Noble: Nook Two-Book Bundle

Amazon: Kindle Two-Book Bundle