The Conversation: Seven deadly sins of statistical misintepretation, and how to avoid them
The Conversation recently published a nice piece by Louis and Chapman on common statistical misinterpretations and how they can be avoided. Here is summary of the main points:
| Problem | Reason | Solution |
|---|---|---|
| 1. Assuming small differences are meaningful | Most small differences are due to chance, not meaningful differences | Ask for the margin of error (ie. half of the 95% CI): if the difference observed is smaller than the margin of error, the difference is probably due to random fluctuations in the data. |
| 2. Equating statistical significance with real world significance | Generalisations about between-group differences often ignore within-group variability or between-group similarities. | Ask for the effect size of the difference between groups and its precision (eg. mean difference and 95% CI) |
| 3. Neglecting to look at extremes | For Normally distributed data, a small change in the group average accentuates differences at the extremes of the distribution more than differences within most of the bell shape. | When you’re dealing with group averages (which is most of the time), small group differences don’t matter much. |
| 4. Trusting coincidence | You can nearly always find an interesting pattern or correlation if you massage the data hard enough. The authors cite a correlation between no. of drownings from falling into pools and films Nicholas Cage appeared in. | Ask how reliable the observed association is: has this association only happened once or has it happened before? Can future associations be predicted? |
| 5. Getting causation backward | Correlation does not equal causation: Did drownings in pools cause Nicholas Cage to appear in more films, or did Nicholas Cage appearing in more films cause more drownings? | Remember to think about reverse causality when you see an association. |
| 6. Forgetting to consider outside causes | Known as confounding: eg. people who eat at restaurants appear to be healthier, but they are often healthier because they are richer and can afford better health care. | Remember to think about potential outside causes: when investigating a certain cause, think about what, in turn, causes that cause. |
| 7. Deceptive graphs | (This is a biggie.) Eg. vertical axis scaling can accentuate differences between groups even though differences are small. | Check graph labels along axes. Be very skeptical about unlabeled graphs. |
Reference
Louis and Chapman (2017) The seven deadly sins of statistical misinterpretation, and how to avoid them. The Conversation.
Scientifically Sound 