Statistical issues

Here is a collection of statistical issues and misunderstandings you often will encounter in empirical research. My plan is to add more examples in the future.

Issue Description Source
Absence of evidence fallacy No evidence for a finding should not be interpreted as there is evidence of its absence. Altman and Bland (1995)
Berkson’s paradox When conditioning on a variable creates a spurious correlation (i.e., collider bias, the conditioning on a collider). Berkson (1946)
Cronbach’s alpha People often misunderstand the coefficient. There is not a particular level of alpha that is desired or adequate. Hoekstra et al. (2018)
Garbage can regression Adding too many independent variables to your regression model (i.e. a kitchen-sink approach). Achen (2004)
Garden of forking paths When researchers conduct multiple analyses but only end up reporting a subset of these (data-dependent analysis). Gelman and Loken (2014)
Moderation vs. Mediation A moderator is a variable that affects the direction and/or strength of the relation between two variables – not the same as mediation. Baron and Kenny (1986)
Multivariate vs. Multivariable A multivariate model is a model with multiple dependent variables. Mustillo et al. (2018)
p-value as a probability The p-value is not the probability that the null hypothesis is true. Greenland et al. (2016)
Prosecutor’s fallacy Incorrectly assuming that Pr(A|B) = Pr(B|A). Westreich et al. (2014)
Simpson’s paradox A trend in the data can disappear or reverse when looking at subgroups in the data. Simpson (1951)
Spurious correlation When two variables correlate but are not casually related. Simon (1954)
Statistical power The importance of having sufficient data to estimate the effect size of interest. Cohen (1992)

Recommended readings

Kennedy, P. E. 2002. Sinning in the Basement: What are the Rules? The Ten Commandments of Applied Econometrics. Journal of Economic Surveys 16(4): 569-589.

Makin, T. R., and J. O. de Xivry. 2019. Science Forum: Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. eLife 8:e48175.

Motulsky, H. J. 2014. Common Misconceptions about Data Analysis and Statistics. Journal of Pharmacology and Experimental Therapeutics 351(1): 200-205.

Schrodt, P. A. 2014. Seven deadly sins of contemporary quantitative political analysis. Journal of Peace Research 51(2): 287-300.