Last week, the American Statistical Association (ASA) released a special issue of its journal, The American Statistician, with 43 articles addressing the issue of “statistical significance.” If you are on the ASA’s mailing list, you received an email announcing that
“the lead editorial calls for abandoning the use of ‘statistically significant’, and offers much (not just one thing) to replace it. Written by Ron Wasserstein, Allen Schirm, and Nicole Lazar, the co-editors of the special issue, ‘Moving to a World Beyond ‘p < 0.05’ summarizes the content of the issue’s 43 articles.”
In 2016, the ASA issued its “consensus” statement on statistical significance, in which it articulated six principles for interpreting p-values, and for avoiding erroneous interpretations. Ronald L. Wasserstein & Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” 70 The American Statistician 129 (2016) [ASA Statement] In the final analysis, that ASA Statement really did not change very much, and could be read fairly only to state that statistical significance was not sufficient for causal inference.1 Aside from overzealous, over-claiming lawyers and their expert witnesses, few scientists or statisticians had ever maintained that statistical significance was sufficient to support causal inference. Still, many “health effect claims” involve alleged causation that is really a modification of a base rate of a disease or disorder that happens without the allegedly harmful exposure, and which does not invariably happen even with the exposure. It is hard to imagine drawing an inference of such causation without ruling out random error, as well as bias and confounding.
According to the lead editorial for the special issue:
“The ASA Statement on P-Values and Statistical Significance stopped just short of recommending that declarations of ‘statistical significance’ be abandoned. We take that step here. We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term ‘statistically significant’ entirely. Nor should variants such as ‘significantly different’, ‘p < 0.05’, and ‘nonsignificant’ survive, whether expressed in words, by asterisks in a table, or in some other way.”2
The ASA (through Wasserstein and colleagues) appear to be condemning dichotomizing p-values, which are a continuum between zero and one. Presumably saying that a p-value is less than 5% is tantamount to dichotomizing, but providing the actual value of the p-value would cause no offense, as long as it was not labeled “significant.”
So although the ASA appears to have gone “whole hog,” the Wasserstein editorial does not appear to condemn assessing random error, or evaluating the extent of random error as part of assessing a study’s support for an association. Reporting p < 0.05 as opposed to p = a real number between zero and one is largely an artifact of statistical tables in the pre-computer era.
So what is the ASA affirmatively recommending? “Much, not just one thing?” Or too much of nothing, which we know makes a man feel ill at ease. Wasserstein’s editorial earnestly admits that there is no replacement for:
“the outsized role that statistical significance has come to play. The statistical community has not yet converged on a simple paradigm for the use of statistical inference in scientific research—and in fact it may never do so.”3
The 42 other articles in the special issue certainly do not converge on any unified, coherent response to the perceived crisis. Indeed, a cursory review of the abstracts alone suggests deep disagreements over an appropriate approach to statistical inference. The ASA may claim to be agnostic in the face of the contradictory recommendations, but there is one thing we know for sure: over-reaching litigants and their expert witnesses will exploit the real or apparent chaos in the ASA’s approach. The lack of coherent, consistent guidance will launch a thousand litigation ships, with no epistemic compass.4
1 “The American Statistical Association’s Statement on and of Significance” (March 17, 2016); “The American Statistical Association Statement on Significance Testing Goes to Court – Part I” (Nov. 13, 2018).
2 Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar, “Editorial: Moving to a World Beyond ‘p < 0.05’,” 73 Am. Statistician S1, S2 (2019).
3 Id. at S2.
4 See, e.g., John P. A. Ioannidis, “Retiring statistical significance would give bias a free pass,” 567 Nature 461 (2019); Valen E. Johnson, “Raise the Bar Rather than Retire Significance,” 567 Nature 461 (2019).
