Research Design and Statistical Consulting

George M. Diekhoff, Ph.D.

I was asked why a measure of effect strength was needed *in addition to* the *p*-value associated with a significance test. Wouldn’t the *p*-value decrease (e.g., from .10 to .001) as the strength of the effect increased? Why does one need both *p*and Cohen’s *d* in a *t*-test for example? My answer was this: The value of *p* does reflect the magnitude of the effect, but it is also influenced by the size of the sample, *n*. With a small sample, an effect of a given strength might show a large *p*-value (e.g., *p* = .10), whereas with a large sample, an effect of the same magnitude would show a smaller *p*-value (e.g., *p* = .001). In contrast, measures of effect strength are unaffected by sample size. One value of including a measure of effect strength with every significance test is that it can alert the reader to the fact that a very weak effect that might lack any real *practical* significance has reached *statistical* significance only because a large sample size was used. Or it can alert the reader to the fact that an effect was large and potentially very important even though it only produced a marginal *p*-value because the sample was small. In addition, measures of effect strength are used in meta-analysis by researchers to evaluate effects across studies that vary in their sample sizes.