Big Data, Plainly Spoken (aka Numbers Rule Your World)

Aging

According to Business Insider (link), IBM is being sued for age discrimination. There apparently exist incriminating emails in which executives disparaged older workers.

What caught my attention was the following defense by IBM's spokesman:

In 2020, the median age of IBM's US workforce was 48, the same as it was ten years prior, he added.

Hmm. I don't think he thought this one through.

If the statement were about gender equality, then it might be convincing. Say, the proportion of women in the workforce was 40% in 2020, and also 40% in 2010. They would be saying gender inequality did not worsen in those 10 years.

But... age and gender are not the same attributes - despite both being popular demographic variables. Gender (traditionally speaking) is a simple variable with primarily two values (male, female) and immutable. The age of each person changes each year; and in aggregate, the average age of a population also changes. In most developed economies, populations are aging.

This Statista chart shows that the median age of Americas increased from 37.2 in 2010 to 38.4 in 2019. In other words, we should expect the median age of IBM's workforce to increase, not stay the same if IBM were age-neutral. If no one were hired or fired, the entire workforce should have aged 10 years from 2010 to 2020. If the median age stayed the same over 10 years, it might suggest that older workers were being replaced by younger ones.

It begins with the data

We've reached the third part discussing the Covid Symptom Tracker app, launched in the U.K. by a King’s College team, and supported in the U.S. by Harvard and Stanford.

In the first post, I explained how the researchers analyzed their data to draw two attention-grabbers: that around 13 percent of the U.K. population has been infected with SARS-CoV-2 by the end of March, and that the loss of taste and smell is a better predictor of having the virus than the usual suspects like persistent cough. I also pointed out how both those conclusions are questionable. Some issues include the unwarranted assumptions of unbiased data, perfect model accuracy and perfect test accuracy.

In the second post, I explained how the research team pre-processed the app data. Pre-processing is absolutely necessary when we're working with observational data for which we exercise minimal control. Some of these steps could introduce additional bias into the dataset. Certain information such as levels of severity of symptoms and the sequencing of symptoms is dropped, which is acceptable for a quick first-pass analysis; it should be good practice to describe how those decisions were arrived at.

At this stage of the pandemic, we are fully aware of the importance of knowing how the data came into being. This is the focus of today's post.

***

#1 Self-selection

The biggest issue with any tracker app is self-selection. Mobile app users do not give a random sample of the U.K. population. For example, Covid-19 especially impacts the elderly, who are far less likely to use smartphones. Some of the bias can be corrected by appropriate statistical adjustments.

It's clear that certain groups of the population have an inclination or disinclination to use a tracking app. Will someone who has recovered sign up and send daily reports to the app? Will healthy people with no symptoms use the app? The more symptoms someone has, the more severe those symptoms, the more likely someone will find out about, and use the Symptom Tracker app. In a country where tests are rationed, if one has been rejected from testing, one is more likely to use the Symptom Tracker.

It appears that the app users are more likely to be infected than non app users, which adversely affects the goal of determining the prevalence in the general population.

#2 Self-reporting

Everything – demographics, symptoms and test results – is self-reported. There is little app developers can do to verify user-submitted responses. Reported symptoms are likely to be self-diagnosed. Users also select which pieces of information to provide. Missing data present analytical challenges described in the previous post.

#3 Outreach

The authors are very proud of the 1.6 million app downloads, and mentioned this fact five times in the short paper. The data were collected between 24 and 29 March 2020. The app was launched on 24 March 2020. (Assuming they disqualified some app users for various reasons, the total downloads must have exceeded 1.6 million in five days.) A couple of app traffic experts I spoke to felt that such an impressive number requires explanation. What marketing tactics created awareness of the app? Where did they advertise to drive app downloads? What motivated users to adopt the app? Other than random selection (with 100% response rate), any method of outreach introduces selection bias.

#4 Analytical Sample

In any case, the 1.6 million statistic is completely meaningless. In an accompanying table of the preprint, we learn that 412,000 users “answered questions on their symptoms”. The other 1.1. million played no role in the analysis. To catch a glimpse of them, we must visit the top few rows of Table 1 showing demographic summaries. Of the 412,000, about 1,200 reported their PCR test results. Only 1,200 made it to the predictive model. The remaining 410K were later scored by the model, leading to the 13% prevalence claim.

#5 Selection by triage testing

In addition to self-selection discussed in #1, the analytical sample is also biased by the U.K. government’s triage testing policy. The analytical sample retains only people who have test results. Those who have been tested equals the group that qualified for testing. During March, the tested subpopulation consisted primarily of people who have severe symptoms of Covid-19. Because of triage testing, the subgroup who have test results are those with severe symptoms, while the subgroup who haven’t been tested have no or mild symptoms. They are essentially disjoint, which presents extreme analytical challenges.

#6 Aided questions

Those familiar with marketing research know the difference between aided and unaided brand awareness. To measure aided awareness, the questionnaire may ask if someone has heard of the brand Snap. By mentioning Snap, the respondent has been reminded of the brand while filling out the survey, and thus brand awareness may be over-estimated. An unaided question may ask the respondent to list the names of some brands of social media websites they know of. This sort of question produces a messy pile of data.

In the Covid Symptom Tracker App, every question is aided. The questionnaire designer works off a list of known symptoms, asking app users if they experienced each symptom on the list.

#7 Default answers

Looking at the screengrabs, I noticed the pre-populated answers. To allay my doubts, I found and view of a few Youtube videos of people using the Covid Symptom Tracker app. I was surprised to see that the questions about symptoms come with default answers. The defaults appear to be “no”. By having these defaults, it is impossible to distinguish between the user choosing to skip the question, and the user specifically answering no. There will be a massive bias toward default answers. This is, in a word, fatal.

#8 Default answers super-sized

In one of those videos, the host found out that after indicating he was feeling fine, the daily report immediately ended, with no need to answer individual questions about symptoms. From what I could tell, the app then filled out every symptom question “no” by default. (If that wasn’t the behavior, then this user would have zero reports of symptoms, which would result in his exclusion from the analytical sample based on the criteria laid out by the research team.)

So, unintentional bifurcation occurred here. Users who say they feel fine are offered an "unaided" assessment of symptoms while users who say they feel not right are given an aided questionnaire. (See #6 for the definition of aided vs unaided.) Further, the adverse effect is magnified among those who have tested positive, and presumably recovered – as this subgroup is most likely to be feeling healthy.

#8 Timing of symptoms

It’s also crucial to ask the users who reported test results to only list symptoms they experienced prior to taking the test. One can’t predict based on information not known until the predicted event has passed. However, with the design of this app, it's not clear how to attach symptoms to an earlier date, which is not the date of reporting.

***

It's important to say that the presence of bias is a fact of life when you have observational data. Bias does not automatically invalidate an analysis. Recognizing its existence is the first step; taking measures to size up the direction and magnitude is next; and finally, designing appropriate adjustments to mitigate bias is key. 

I should applaud the research team for making very responsible disclosure of the limitations, referencing many of the issues I’ve raised in this and my previous posts.

After acknowledging that all their data are self-reported, that the anosmia might appear after other symptoms or after recovery, that testing positive is not identical to being infected, that triage testing means their sample is “not fully representative”, the research team then, in the very next sentence, recommended that WHO adds loss of smell and taste to their symptom list. (which they did along with various other symptoms)

It’s a curious practice of academic science that making such disclosure is an act of exorcism, after which authors are free to make recommendations while ignoring the disclosed deficiencies.

 

 

[P.S. See my previous comments on the data pre-processing and the analytical methods of the Covid Symptom Tracker study.]

The list of worst college majors, and why you should ignore it

Business Insider reports on the bottom 20 college majors, with the warning "what not to study". The list looks extremely random including everything from visual and performing arts to international relations to math and computer science (!).

The author explains these majors have the highest unemployment rates.

Here are a few reasons why you can and should ignore this study.

1. Presumably the journalist is advising college students, who will soon become new college graduates. But the analysis references college graduates which include anyone with a college degree, ranging from a new college graduate to someone who's worked for 40 years. New college graduates have a much higher unemployment rate than all college graduates. (See this recent Huffington Post article.)
   
   
2. Students do not randomly select a college major. Thus, students who choose to major in international relations are not the same type of people who decide to major in engineering. If an international relations student were to switch major to engineering, it would not follow that this person's employment prospect had suddenly brightened. It might even worsen because (a) the student could be relatively less competitive against other engineering majors; and (b) the student and the major might now be mismatched.
   
   
3. If everyone followed this journalist's advice, then each college would need a very small number of majors. Unless the economy suddenly produced a bonanza of jobs, where would the unemployed go? You got it... the unemployment rate would accrue to the several remaining majors, instead of being spread out over hundreds of majors. The unemployment rate of these "good" majors would rise to the level of the average unemployment rate. Oops.
   
   
4. Not everyone needs or wants a job. Changing majors doesn't change that reality.

***

There is an even bigger howler in the article. The analyst at Bankrate equated "job stability" with low unemployment rate, using language such as "Having a high-paying job doesn't necessarily mean you'll have job stability, and vice versa." So, majors with high unemployment rates are bad because people job-hop.

But the unemployment rate is not a measure of job tenure, and thus not an indicator of job stability. In fact, if people job-hop a lot, the unemployment rate will be relatively low, because the same job can be held by multiple people in the course of a year.

Let's imagine a country of 10 people with 5 jobs. The unemployment rate is 50%. If no one changes jobs, the same five people are employed always, and the rate is stable at 50%. The government stipulates that no person can hold on to the same job for longer than 6 months. We put the 10 people in a circle, and every other person is given a chair and seated. Every 6 months, each person moves clockwise by one slot: the five previously seated no longer have a seat, the five seats now occupied by the five previously standing. Everyone is now employed 6 months out of the year. The employment rate is now 100% (counting part-timers).

Thus, unemployment rate of zero can coincide with high job instability.

 

 

Some practice case interview questions for data science

In Part 1 of my KDnuggets article, I explained what hiring managers mean when they look for critical thinking in the arena of data science and analytics. These requirements relate to the nature of data problems found in industry and business settings. The datasets are generally observational, self-selected, non-random, with hidden biases, and increasingly OCCAM (link); the business leaders have high-level objectives ("we want to increase customer loyalty"). The data scientist/analyst is the person in the "middle," trying to figure out how to make the problem precise, and solvable by a systematic analysis of available data.

In Part 2, I offer some practice case interview questions, based on three recent news events

• the college admissions scandal
• IPOs of ride-sharing companies like Lyft and Uber
• the Blue Apron post-IPO doldrums.

Long a staple of the management consulting hiring process, the case interview is a free-flowing dialogue between the interviewer and the interviewee. The interviewer is holding back some data to simulate what is known at the beginning of a data analysis process. The interviewee must be willing to probe, digging out more data, and shaping the structure of the analysis. The end product is an analytical framework. No one knows if the framework would be successful until it is implemented.

Those who do well in case interviews are good at (a) thinking on their feet (b) embracing uncertainty e.g. by making appropriate assumptions (c) listening to the interviewer's hints and (d) persuading.

***

As I mentioned in Part 2, the best way to practice is to form a group of 3-5 people, and interview each other. If there is enough interest, we can start a group in the comments below.

What is the hardest part of the data science job search?

When I ask job-seekers what their biggest obstacle is to finding a job in data science and analytics, one of the most frequent answers is performing during the interview. Some of them are stumped by technical interviews (coding) while even more are worried about the case interviews.

The purpose of the case interview is to test critical thinking. It is as challenging for the job candidate as for the hiring manager! Technical questions have pretty standard answers, and it's easy to score the answers. Case interviews are like essays - the hiring manager has to make judgment calls.

My piece on critical thinking is featured at the KDNuggets blog, which I've followed since I was an analyst. In this first part, I explain the two aspects of critical thinking that the case interviewer is typically looking for. There will be a part 2 in which I provide some practice examples.

 

P.S. [5/1/2019] This piece from TED is relevant.

 

Purdue's Math, Data Science and Industry Conference

This past weekend, I found my way to West Lafayette, Indiana, to speak at the Math, Data Science and Industry Conference, organized by Math Prof. Aaron Yip and Drew Swartz. I was very impressed with the quality and diversity of the talks there. They managed to strike a nice balance between academic talks and industry talks, and the BS quotient was minimal.

I will first outline my own talk, and then in a next post, I will highlight some things from the other talks I attended.

The goal of my talk is to paint a broad-brush picture of the scope of jobs that are part of the current Data Revolution, and to give a flavor for the nature of the work so that graduate students may decide for themselves whether this "data science" industry is a good fit for them.

One key takeaway is the distinction between research jobs and industry jobs. Research jobs lead to innovative research that can be published in scholarly journals. Most industry jobs demand short-term results that impact the business, and it does not matter whether the methods used are innovative. The boom in data jobs, however, is in industry jobs. Only large corporations in cash-rich industries can afford research jobs, and even at those firms, there are hundreds if not thousands of industry jobs for each research position. Math graduates can totally get hired for industry positions, if they put in a little effort to prepare for this career path.

Within industry jobs, I like to think of three job types.

1. Data science jobs - these are the headline-catching jobs because they are disproportionately found in the high-tech industry. Think of these as software developers with advanced database skills. The culture here is automation, removing human beings from the process.
2. Business analytics jobs - these jobs are tethered to business teams, such as marketing, finance, operations and customer service. They are the champions of embedding data analyses in the everyday decision-making processes. They interact constantly with business managers, providing a form of consulting service.
3. Data IT jobs - these people keep the data flowing in the organization, so to speak. They are also responsible for "data governance" and standardizing the formats, definitions, quality, etc. of the data. This sector is experiencing rip-roaring growth.

There is a huge need for scientific thinkers and data-savvy people in all three job types but at least half the open positions are in "business analytics." I discuss two particular gaps in skills that hiring managers often complain about in university graduates: (a) inability to develop the question and (b) not knowing how to question the data.

There is a clear reason why such gaps exist. A typical question we pose to students in a problem set first lays out the problem to be solved, then presents the set of data to be used, and finally challenges the students to plug the data into an appropriate method or framework so that the solution to the problem drops out.

The professor is not going to look kindly on the student if s/he criticizes or revamps the question or points out flaws in the data! (University classes teach theory, models and frameworks so this is not surprising.)

This brings me full circle to the distinction between research and industry jobs. In research, you can "choose your battles" by making certain assumptions to move past obstacles. For example, you assume that the (biased) dataset that you obtained is representative - lots of research papers that use observed social-media data do this. You just argue that bias correction is a separate problem to be tackled at some other time, perhaps by some other research team.

In industry, you don't have that luxury. A great solution to the biased problem may turn out to be a horrible solution to the unbiased problem. When I was at SiriusXM, we had some data on people's online listening patterns but almost nothing on their in-car listening. Building great models using the online data isn't going to do much good because most of the listening happens in the car, and people who listen online are quite different from those who listen in the car.

Towards the end of the talk, I pointed out that in order to do well in these data jobs, one must be comforable to live in the "gray" areas. There is the gray between science and social science, between models and heuristics, between data and intuition.

People were very friendly and we had some fine conversations at a bar after the day was over. I'm happy to report that at least a few people have indicated that they want to pursue these industry jobs.

 

 

 

 

 

 

How to break into business analytics

I have been contributing to Andrew's thread on how to get into the data science field. A recent college grad with a degree in environmental science and minor in statistics wants a job. Andrew suggests getting a job in industry - which I think is an excellent suggestion.

Here is my advice:

1. Figure out what he enjoys doing – is it coding or is it problem solving? Those are two different jobs, one is software engineering, the other is more statistics and analyses. If he is in NYC, come to one of my public lectures at NYPL in which I explain how to pick a career path within this wide and exciting field. [The next one is on the May schedule.]
2. Once he has picked an area, and hopefully also an industry, then he needs to reach out and talk to as many people in industry as possible. Go to networking events and meetups.
3. Then apply to jobs. The job search is a job in itself; keep applying until someone gives you a chance. You will encounter lots of rejection but keep trying.
4. If nothing is working, consider going to a bootcamp. They are set up to give you practical skills that appeal to hiring managers. Talk to the bootcamp organizers to get a sense of what their vision is, and see if it’d help you make your case.

One reason I have organized a bootcamp is that for some, it will be very difficult to break into the field without extra help - both filling knowledge gaps and making industry connections. I give the above advice to my students as well. They need to find a job that matches their temperament, and then work hard at convincing hiring managers to take a chance.

***

Next Tuesday, we are hosting an Open House.

If you're interested in learning about our vision, drop by and say hello.

Reserve your spot here.

An email from LinkedIn

Last week, I got served a dose of predictive analytics. I got an email solicitation from LinkedIn, presenting a list of jobs that they think I might be interested in. This email is algorithmically generated, and LinkedIn tells me that I received it because I clicked on a job posting for a senior data analyst position at Ogilvy & Mather, a top advertising agency based in Manhattan.

Yes, I did click on that job posting while preparing for the information session for Principal Analytics Prep a few days before the email arrived. A podcast of the event is available here.

Here is what the email looks like.

There are several things one can learn from this email:

1. The field of analytics is absolutely exploding. There are six pages of jobs related to the one job I clicked on. Most of these jobs are junior positions (“senior analyst”), because the ad I clicked on is at that level.

2. Many top companies are hiring. The competitors of Ogilvy – AKGA, MRM/McCann, OMD USA, RAPP/Omnicom, J Walker Thompson – are also competing for talent. Not just advertising but other related industries are also hiring senior analysts. I recognize Shazam (famous mobile song-recognition app), frog (top design agency), A+E network, Etsy (noteworthy startup retailer), Luxottica (high-end makers of glasses), AIG, Mercer (top management consultancy), Mastercard, S&P, and Burtch Works (a top executive recruiter in the data space – also our guest at the info session!) So, the jobs are at  top companies, startups and small businesses.

3. The jobs are spread out over all industries. Just in that small, nonrandom sample, we have representation of advertising, retail, technology, media and entertainment, e-commerce, credit cards, finance and insurance, human resources, management consultancy, and graphic design.

4. Analytics are needed in all job functions. The job seeker should also consider unconventional career paths e.g. the analyst at Burtch Works is a recruiter but with a specialty in data science and analytics. Another unusual path is to become an account manager at a digital advertising agency – you may not be running analyses all day long but if you have superior analytical skills, you’d be much better at explaining results and data-driven recommendations to your clients. Similarly, a salesperson for an analytics product company should definitely use foundational analytics knowledge.

One of the key reasons I started Principal Analytics Prep is to open doors in the job market for people of diverse backgrounds and diverse career paths. Analytics and data jobs are not limited to technical people who are coders and engineers. There are plenty of exciting job opportunities across all industries and job functions for data wizards with unconventional backgrounds. Please contact us if you want to learn more about how we can help guide you to your next career in data.

Collaboration with New York Public Library

For many years now, the field of Data Science and Business Analytics has been booming, and hiring managers are finding a severe dearth of high-quality job-seekers. Meanwhile, there are a good number of people interested in entering the field but keep bumping into walls. Hiring managers like to hire experienced people for a host of reasons, including the fear of other hiring managers poaching their trained employees. For a number of years now, I have been interested in solving this problem.

In the next two weeks, I am collaborating with the New York Public Library's Job Search Central unit to offer two free events.

The first is a public lecture on "How to Start a Data Science and Analytics Career." I will be spending significant time explaining what this field is about, and what data science and analytics teams do. I will also discuss the structure of the job market, and provide tips for how to land a job. You can find more information about it here. The talk is scheduled for April 1.

The second is a resume review workshop. From my experience as a hiring manager, I know that many job-seekers fall short in selling themselves to organizations. It is particularly challenging for those who are seeking to entering this field, and thus do not have direct experience to call upon. This is a hands-on workshop in which I will offer advice on individual resumes. The event is free but obviously, we have limited space, so you must pre-register here. Note that you will need to upload your resume to complete the registration. The Workshop is being held on April 8.

***

I have updated the Events list on the right column of the blog. This is where you learn whether I will be appearing in a city near you. You can check out future and past events here as well.

 

The idol worship of objective data is damaging our discipline

In class last week, I discussed this New York Times article with the students. One of the claims in the article is that the U.S. News ranking of colleges is under threat by newcomers whose rankings are more relevant because they more directly measure outcomes such as earnings of graduates.

This specific claim in the article makes me head hurt: "If nothing else, earnings are objective and, as the database grows into the millions, reliable."

The entire Chapter 1 of Numbersense (link) is devoted to blowing apart the myth that school rankings are "objective." In fact, I go on to assert that even objective-sounding metrics like company revenues are not objective at all... if you know GAAP and the games accountants play with those numbers. If someone buys a car on eBay from another person for $20,000, does eBay book $20,000 in revenues or just x% of the revenues that the seller pays to eBay?

Objective implies there is a ground truth that can be verified. There is no true school ranking, nor is there true revenue.

Where does this "objective" earnings data come from? Apparently a company called Payscale, whose methodology is explained here. They say their data come from "individuals who fill out the PayScale Salary Survey." How did these people discover the survey? We don't really know. Are these people representative of the universe of employed people? Most likely not but we again do not know.

Do people give out their real salaries voluntarily? Not in my experience. In fact, none of my co-workers in my 15+ years in the corporate world has ever told me how much money they make. PayScale claims that the salary number "combines base annual salary or hourly wage, bonuses, profit sharing, tips, commissions, overtime, and other forms of cash earnings, as applicable." Do people have their aggregate salaries at their finger tips? I highly doubt it (unless they make just a base salary).

In return for filling out the surveys, the individuals receive a free salary report. Does this encourage people to make up fake data just to obtain the salary report? Take a guess.

PayScale claims that it "rigorously tests and verifies" the data. Given that most employers wouldn't even do salary verification for other employers, I don't believe the salary data can be verified, and absolutely not "every data point" as claimed in their marketing materials.

The other ludicrous claim is that the "reliability" of the data improves with scale. This is only true if the data is a proper random sample of the population of all salaries, which it clearly isn't. Dumping more garbage on top of garbage is still garbage.

Consider this sequence of scenarios: if

(a) you are the Devious Dean of Admissions at a college, and
(b) improving your college's ranking is on your annual performance management plan, and
(c) you know that the Economist ranking is largely based on the PayScale Salary Report, and
(d) PayScale's data come from the Salary Surveys which do not require explicit identity verification, and
(e) you have access to various devices that can access these Salary Surveys

why are you not sending in a bunch of fake reports of outsized compensation?

Oh, my Mom reads this blog so let me not promote unethical behavior. You don't need to fake data. You can just target a bunch of alumni who have had successful careers, and encourage them to send in their true reports.

***

The other key sentence in the article is: "[The Economist] took the College Scorecard earnings data and performed a multiple regression analysis to assess how much a school’s graduates earn compared with how much they might have made had they attended another school."

I asked the students what are the explanatory (X) variables that might be found in such a regression. Some of the answers were: job title, gender, location of job, GPA, college major, number of years of work experience, family background. Essentially the "all else equal" requires a lot of covariates.

It turns out PayScale has a product called MarketMatch (link) which gives us some hints. Here is a description of this product:

The MarketMatch algorithm uses a two-step process for producing compensation data in a PayScale report. The first step is to understand which of our more than 250 compensable factors are important when it comes to pricing a job and how that job's pay is affected by these compensable factors. This is done in order to define a pay distribution for this job. The mix of compensable factors and their effect on pay is highly dependent upon the job. For example, coding languages and locations are important compensable factors for a Software Developer, while average sales prices and annual sales are important for an Account Executive.

This description leads to a very complex multiple regression model, with "more than 250" covariates, and a host of interaction effects (e.g. allowing the effect of location to depend on the job title). This model has at least 250 main effects. If it has all pairs of 2-way interactions, the regression equation has over 31,000 more terms.

PayScale did earlier impress us with their 1.4 million salary profiles (which, for the following discussion, we assume to be objective and reliable.) This, they say, translate to anywhere from 50 to 4000 profiles per school. While the lower limit is 50 students, PayScale actually does not publish results for schools with fewer than 325 profiles.

Even with 4000 profiles, you can't estimate tens of thousands of regression coefficients with any resemblance of accuracy. If each of the 250 factors were binary ("Yes"/"No"), you would have created 2^250 unique types of individuals. That number has 75 zeroes in it, and you only have 4,000 observations. For overwhelming majority of these types of individuals for which you are issuing predicted salaries, you have zero data.

Is the resulting salary predictions "reliable"? You decide.