« Update: reviews, translations | Main | A significant mystery »


Feed You can follow this conversation by subscribing to the comment feed for this post.

Cosma Shalizi

I can't bring myself to read the paper, but it sounds like they showed each subject multiple chart pairs, say k pairs, which would mean they had 20*k observations and not just 20. Still a hell of a multiple comparisons problem, though. Of course, the real problem here is that the experimental procedure can be fairly paraphrased as "someone with an ax to grind asked leading questions until he was happy with the answers".


Cosma: Not really. They are not randomizing treatment. Every one of 20 participants inspected both sets of charts. They should have used a paired-difference test but they didn't. The comparison is for the difference between the sum total of scores for chart type A and the sum total of scores for chart type B, replicated 20 times. So I think you have to call that 20 observations.


Are you sure? The paper says, "Each participant saw only one version of each chart, either Holmes or plain."


Jerzy: I know it's confusing because the way they set it up is super complicated. Each participant saw 14 charts, 7 are of the Holmes variety, 7 are of the plain variety. They alternate between Holmes and plain charts, and insert a blank chart to provide a "visual break".
So each participant is exposed to both "treatments" in alternating fashion.
However, for any given chart (better described as given data set), each participant sees only once. So in the above figure, you either saw the Holmes version or the plain version. But if you saw the Holmes version for the Diamond chart, your next chart would be the plain version.

Cosma Shalizi

So I think you have to call that 20 observations.

If that's what they did, then yes, it's just n=20. Wow (and not in a good way). "We can perhaps say what the experiment died of."

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)


Link to Principal Analytics Prep

See our curriculum, instructors. Apply.
Business analytics and data visualization expert. Author and Speaker. Founder of Principal Analytics Prep, MS Applied Analytics at Columbia. See my full bio.

Next Events

May: 2 New York Marketing Association Big Data Workshop, NYC

May: 5 NYPL Analytics Careers Talk, NYC

May: 8 Data Visualization Seminar, Denver, CO

May: 15 Data Visualization Seminar, Cambridge, MA

May: 17 Data Visualization Seminar, Philadelphia, PA

May: 22 Data Visualization Seminar, San Ramon, CA

Past Events

See here

Future Courses (New York)

Summer: Statistical Reasoning & Numbersense, Principal Analytics Prep (4 weeks)

Summer: Applied Analytics Frameworks & Methods, Columbia (6 weeks)

Junk Charts Blog

Link to junkcharts

Graphics design by Amanda Lee


  • only in Big Data