A small reform in the data analysis in psychology: a small p is not enough, effect size is needed too

Abstract

The main objective of this paper is to point out the limitations and problems that occur when relying on conventional tests of statistical significance in presenting the results of empirical research. The following misinterpretations of p values are emphasized in the paper: a) p value is the probability that the result was due to sampling error; b) p value represents the probability of wrong decisions in the event of rejecting the true null hypothesis; c) p value is the probability of the null hypothesis being true given the data d) 1-p is the probability that a replication attempt will also reject the null hypothesis, and e) 1-p is the probability that the alternative hypothesis is true given the data. Effect sizes and confidence intervals can be used as additional indices in the process of statistical inference. A large number of the effect size indexes can be classified as standardized mean difference indices, such as Cohen’s d, Hedges’ g, Glass’s δ and Cohen’s f2, and variance-accounted-for indices, such as R2, and η2 and ηp2. Suggestions for approximate evaluations of certain indicators, as well as the manner of their interpretation in the context of specific research designs are given.