Why you must know how analytical results were obtained

A friend sent me to this Vice article, admitting abashedly that she got clickbaited (link).

[right here is supposed to be an image showing the article's headline - but it's not here because Typepad's image loader is failing yet again!]

How did she get baited?

Of course, the MTA did not "deliberately slow down buses to punish riders". What the public bus operator did was to suspend so-called all-door boarding. All-door boarding was being tested on a small number of "Select" bus routes: on these routes, all passengers are required to pay using machines installed on the sidewalk before boarding, and thus, they no longer line up at the front of the bus, and can board through any of the buses' doors. As expected, Select bus rides experienced lower delays.

When this experiment was suspended, the Vice reporter "interpreted" this as MTA deliberately slowing down buses.

Why was the experiment stopped? Certainly not because the operator thought NYC buses were running too fast. Clue: what might happen when passengers are allowed to enter buses through the back doors without having to show they paid the fare?

You got it. The MTA is forced to suspend the all-door boarding test because of fare evasion.

***

The Vice reporter dismissed that possibility by citing an earlier MTA study which concluded that all-door boarding reduced fare evasion. It's as counterintuitive a finding as one could imagine. In fact, that study found that fare evasion was 10 times lower on Select bus routes than on regular bus routes!

[right here is supposed to be an image showing a data graphic from the study - but it's not here because Typepad's image loader is failing yet again!]

This is the kind of analytical finding that smells like your fridge after you return from 15 days' vacation.

The last column suggested that fare evasion was almost 20% on all non-Select bus routes. The table below showed fare evasion went down to 2% on SBS bus routes. In the famous words of your favorite Covid-19 talking head, SBS is the miracle vaccine that has 90% effectiveness in curing fare evasion!

Want to get rid of fare evasion? Install more machines on the sidewalk, and let everyone board without showing proof of purchase!

What does one do when the stats stank like rotting food?

How about turning to the "methodology" section?

There, I learn that two completely different methods were used to collect data on "Select" buses and regular (front-door boarding) buses.

The method used for regular buses appeared rigorous and intentionally designed:

A randomized quarterly sample of observation surveys is generated; peak and non-peak, high and low volume stations, all boroughs represented

The sample consists of approximately 180 station control areas and 140 bus routes per quarter

Staff visit several assigned subway stations / bus routes each day to observe and record evasion of different types, e.g. illegal turnstile / service gate entry, entering bus through back door, etc.

Here is how the study authors described the sampling method for "Select" buses:

Eagle teams conduct periodic “surge” exercises, where in addition to enforcement activity they count paid vs. unpaid passengers on board the bus

So, what are some of the differences?

The regular bus routes have daily data collection while the Select bus routes are sampled "periodically" (with an undisclosed frequency and cadence).

The regular bus routes are represented by a random sample from all routes. We have no idea if every Select bus route is sampled (it's possible since there are not that many). However, since these "eagle teams" had other "enforcement activities", one wonders if the frequency of sampling depended on the frequency of "enforcement activities".

Having to both conduct "enforcement activities" and data collection suggests that the data collectors for regular bus routes are properly trained and specialized in data collection while those for Select bus routes may have less training and experience, and are responsible for multiple tasks.

Let's see: enforcement taking place alongside data collection. What about having police officers standing on every street corner, and then measuring the prevalence of crime in those areas at those times? Well - they knew about this bias because they listed it under "limitations":

The mere presence of field staff may limit evasion.

This is what I call a "professional foul". This type of language appears on almost every journal article - important biases that likely invalidate results are simply ignored after the authors acknowledge that they exist. Most analysts keep quiet since everyone else follows the same practice.

***

I'm also struggling to imagine how these "Eagle teams" collected the fare evasion data on Select bus routes. I can see two possibilities:

(a) They position themselves at the bus stops, and observe passengers as they board the bus. In this case, they can see who paid at the roadside machines and who didn't. (In recent years, though, the MTA also installed scanners that allow people to board through back doors using Applepay or chipped credit cards so that significantly complicates the counting. The study was conducted in 2018 which may have preceded this new payment method.) I'm not sure what they mean by "enforcement actvity" - could it mean enforcing fare payment? If so, this is truly Orwellian.

(b) They position themselves on the buses, presumably performing their "enforcement activities". At each stop, a crowd of people enters from all doors. None of them is required to show tickets, in the name of speeding up boarding. How do they figure out what proportion paid? Do they do a show of hands? Do they ask every individual rider to show proof of purchase? If so, do they hold up the buses and not allow anyone to get off until they have finished tallying?

The language used in the Methodology section suggested (b) since they "counted" "passengers on board the bus".

Readers - maybe I'm missing the obvious. How do you think they collected the data?

***

Even if the same sampling methodology was applied, and the data were properly collected, the next challenge is that the Select bus routes serve a small portion of the area served by all bus routes, and the demographics cannot be assumed as identical.