Research Design and Statistical Consulting
George M. Diekhoff, Ph.D.

Why Do We Need Both p-Values and Measures of Effect Strength?

I was asked why a measure of effect strength was needed in addition to the p-value associated with a significance test. Wouldn’t the p-value decrease (e.g., from .10 to .001) as the strength of the effect increased? Why does one need both pand Cohen’s d in a t-test for example? My answer was this: The value of p does reflect the magnitude of the effect, but it is also influenced by the size of the sample, n. With a small sample, an effect of a given strength might show a large p-value (e.g., p = .10), whereas with a large sample, an effect of the same magnitude would show a smaller p-value (e.g., p = .001). In contrast, measures of effect strength are unaffected by sample size. One value of including a measure of effect strength with every significance test is that it can alert the reader to the fact that a very weak effect that might lack any real practical significance has reached statistical significance only because a large sample size was used. Or it can alert the reader to the fact that an effect was large and potentially very important even though it only produced a marginal p-value because the sample was small. In addition, measures of effect strength are used in meta-analysis by researchers to evaluate effects across studies that vary in their sample sizes.