Halsey et al. raise issue with the often cited but infrequently analyzed P value. The authors use random sampling to show that experiments with low statistical power often will give false negative results, a regime often found in most scientific studies. In the rare case of a statistically significant event in an experiment of low statistical power, the effect size is exaggerated, termed the “winner’s curse.” Though no approach is always best, the authors recommend the use of effect sizes and 95% confidence intervals to improve statistical analysis, such that “the importance and relevance of the effect can be judged.” -KJM
Halsey LG, Curran-Everett D, Vowler SL, and Drummond GB. 2015.
The fickle P value generates irreproducible results. Nature Methods 12, 179–185. doi:10.1038/nmeth.3288.