« The charting process | Main | Google trends »



Isn't it unsurprising that around 5% of the 1000 simulations turned out to have a p-value of <0.05? This is an artefact of choosing 0.05 as the significance level.


You raise a good point; p-values are of course not the end-all of any analysis.

The point to note is that if each simulation contains say 50,000 players instead of 761, then we'd expect every line to look flat.

Unfortunately, when sample size is large, p-values or t-tests do not help us. Using those, we'd still have to conclude that some of the lines are not flat; however, if plotted, it'd be obvious that all of the lines are essentially horizontal. That's the "practical significance" conundrum rearing its head again.


thank you this is enlightening.

The comments to this entry are closed.


Link to Principal Analytics Prep

See our curriculum, instructors. Apply.
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR.

See my Youtube and Flickr.

Book Blog

Link to junkcharts

Graphics design by Amanda Lee

The Read

Keep in Touch

follow me on Twitter