Blog

Statistical “significance” is a dirty word

Date: 2026.02.09

The term “statistically significant” is completely arbitrary in the modern age. It reduces the results from a research process to a simple fact about whether the data meets an arbitrary alpha value.

Let’s say, for example, that we have a p-value (which is in itself not very useful) for one experiment is 0.02 different from that in another experiment, we might reasonably say that there isn’t a big difference between the significance of the results of the two. The difference 0.041 and 0.043, is inconsequential, and so is the difference between 0.085 and 0.087. Most any statistician or scientist would agree. What if the two p-values were 0.049 and 0.051? Why does it suddenly make a difference? Because we have set 0.05 as a common “goalpost”, this arbitrary number becomes the litmus test that results are held up to. Some may argue that 0.05 is not the right number to judge results against, but that misses the mark as well. Regardless of the number chosen as an alpha value, it is still arbitrary, whether it is 0.05, 0.01, or 0.1.

What should be done, as recommended by many statisticians in the modern age, is a reporting of the effect size and a thoughtful description of what it means, and whether the size is large enough to be important. When it comes down to it, research in any form is nuanced and requires a more in-depth explanation of what results mean than simply that they are (or aren’t) “significant”. Rebecca Betensky makes this point clear with her argument in the article “The p-Value Requires Context, Not a Threshold”. She suggests that for each experiment, a threshold should be determined based on the experiment itself. For example, if a certain difference between two means is found to be a threshold for clinical significance, this should serve as the test of for the importance of results.

This helps to address two key issues that plague statistical analysis:

One is the reproducibility crisis, which is rampant in medical research. By setting a more realistic threshold, one can improve the chance that the results hold up.
The other issue is an issue less discussed, but I would argue far more important. Some results which are far from “clinically significant” are shown to be “statistically significant”, or vice versa. By marrying the two concepts, we know that any important findings elucidated in data analysis will be much more likely to be appreciated in practice.

This determination of threshold should be made on a case by case basis and requires the consideration and explanation in publication of the research team, which fosters an improvement in analytic quality and scientific communication overall. Thus, everyone wins.

To read more from the special issue of the American Statistician that inspired this post, visit this link. The special edition features a variety of articles on what to do and what NOT to do with regard to statistical significance in the modern day. The article by Betensky listed above is a key part of this issue.

This short blog post was written as part of a course assignment for PQHS 432: Statistical Methods II taken at Case Western Reserve University in Spring of 2026.