Inside elevators in commercial buildings in New York, there are frequently screens that pump out commercials or infomercials all day long. I took a snapshot of this one that showed up last week:
These poll results annoy me in many ways. Let me explain.
The conclusion "free snacks is the key to happiness" is clearly wrong, even if the data were trustworthy (I'll talk about the data in a future post). I call this "xy myopia" or xyopia, in short. The writer of the poll has narrowly framed the problem: in this xyopic world, happiness hinges on just one factor - whether one's employer offers free food. In the real world, the happiness of an employee depends on a host of factors, of which free food is likely to be a minor one.
Xyopia arises from the reasonable concept of ceteris paribus ("all others equal"). Ceteris paribus is of little value in data analysis - it is often implicitly assumed by the analyst, but the assumption is false, which leads to bad conclusions.
In this case, the analyst just assumes that the two groups - employees getting free food, and those not getting free food - are equal on all other dimensions. So not true! The employer who offers free food is more likely to treat employees better across the board, such as providing more vacation days, better benefits, allowing work from home, etc. Besides, free food is only relevant in office settings (think Starbucks baristas). Certain demographic groups may value free food more than others.
Unless it is established that the respondents who got free food are identical to the respondents who didn't in all other ways, the conclusion that free food is key results because it is the only factor you analyzed. This is xypoia.
(Not to mention, "free snacks" is not the same thing as "free food.")
***
Now, think about what question the poll writer asked that led to the headline "Free snacks is the key to happiness". It appears that they asked a pair of questions: "Does your employer offer free food?" and "Rate your happiness at work". Asking the first question frames your answer to the second question.
I'm going to make up my own poll:
A) Does your employer provide financial support for your personal development?
B) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).
Based on this poll, I intend to conclude that paying for personal development is the key to employee happiness.
***
If you extend your vision to the basket of factors that affect employee happiness, such as their pay, benefits, treatment, work-life balance, family life, etc., you'll recognize more problems with the data analysis.
The poll result of 67% versus 56% is a strong result only if the gap is wider than for other factors, such as the poll I created above. I might find that 75% of the employees who get reimbursed for professional development answer extremely or very happy.
Also, consider a poll with three questions:
A) Does your employer provide free food?
B) If answer to A) is yes, do you consume the free food provided by your employer?
C) Rate your happiness at work (1 = extremely unhappy, 5 = extremely happy).
This creates three subsets of respondents (No - NA; Yes - No; and Yes - Yes) and leads to a better understanding of the causal mechanism.
***
Polls are easy to do but hard to do well.
Comments