Under the heading of “falsehood flies,” we have the attempt by the American Statistical Association (ASA) to correct misinterpretations and misrepresentations of “statistical significance,” in a 2016 consensus statement.[1] Almost before the ink was dried, lawsuit industry lawyers seized upon the ASA statement to proclaim a new freedom from having to exclude random error.[2] Those misrepresentations were easily enough defeated by the actual text of the ASA statement, as long as lawyers bothered to read it carefully.
In 2019, Ronald Wasserstein, the ASA executive director, along with two other authors wrote an editorial, which explicitly called for the abandonment of using “statistical significance.” Although the piece, published in the American Statistician, was labeled “editorial,”[3] I predicted that Wasserstein’s official title, which appears in the editorial, and the absence of a disclaimer that the piece was not an ex cathedra pronouncement, would lead to widespread confusion, abuse, and further misrepresentations of the ASA’s views.[4]
Some people pooh-poohed the danger of confusion, but I was doubtful, given the experience with what happened with the anodyne 2016 ASA statement. What I did not realize until recently was that the Wasserstein editorial was misunderstood to be an official policy statement by the ASA’s own publication, Significance!
Significance is a bimonthly magazine on statistics for educated laypeople, published jointly the ASA and the Royal Statistical Society. In August 2019, the editor of Significance, Brian Turran, published an editorial that clearly reflected that Turran interpreted the Wasserstein editorial as an official ASA pronouncement.[5] Indeed, Turran cited the Wasserstein 2019 editorial as the ASA “recommendation.”
Donald Macnaughton, President of MatStat Research Consulting Inc., in Toronto, wrote a letter to point out Turran’s error.[6] Macnaughton noted that Wasserstein had disclaimed an official imprimatur for his ideas in various oral presentations, and that the editors of the New England Journal of Medicine had explicitly rejected the editorial’s call for abandoning statistical significance.[7]
In reply, Tarran graciously acknowledge the mistake, and pointed to an ASA press release that had led him astray:
“Thank you for this clarification. Our mistake was to give too much weight to the headline of a press release, ‘ASA Calls Time on “Statistically Significant” in Science Research’ (bit.ly/2UBWKNq).”
Inquiring minds might wonder why the ASA allowed such a press release to go out.
In 2019, then President of the ASA, Karen Kafadar, wrote on multiple occasions, in AmStat News, to correct any confusion or misimpression created by Wasserstein’s editorial:
“One final challenge, which I hope to address in my final month as ASA president, concerns issues of significance, multiplicity, and reproducibility. In 2016, the ASA published a statement that simply reiterated what p-values are and are not. It did not recommend specific approaches, other than ‘good statistical practice … principles of good study design and conduct, a variety of numerical and graphical summaries of data, understanding of the phenomenon under study, interpretation of results in context, complete reporting and proper logical and quantitative understanding of what data summaries mean’.
The guest editors of the March 2019 supplement to The American Statistician went further, writing: ‘The ASA Statement on P-Values and Statistical Significance stopped just short of recommending that declarations of “statistical significance” be abandoned. We take that step here. … [I]t is time to stop using the term “statistically significant” entirely’.
Many of you have written of instances in which authors and journal editors – and even some ASA members – have mistakenly assumed this editorial represented ASA policy. The mistake is understandable: The editorial was coauthored by an official of the ASA. In fact, the ASA does not endorse any article, by any author, in any journal – even an article written by a member of its own staff in a journal the ASA publishes.”[8]
Kafadar did not address the hyperactivity of the ASA public relations’ office, but her careful statement of the issues should put the matter to bed. There are now citable sources to correct the incorrect claim that the ASA has recommended the complete abandonment of significance testing.
——————————————————————————————————————–
[1] Ronald L. Wasserstein & Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” 70 The Am. Statistician 129 (2016); see “The American Statistical Association’s Statement on and of Significance” (March 17, 2016).
[2] “The American Statistical Association Statement on Significance Testing Goes to Court – Part I” (Nov. 13, 2018); “The American Statistical Association Statement on Significance Testing Goes to Court – Part 2” (Mar. 7, 2019).
[3] Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar, “Editorial: Moving to a World Beyond ‘p < 0.05’,” 73 Am. Statistician S1, S2 (2019).
[4] “Has the American Statistical Association Gone Post-Modern?” (Mar. 24, 2019); “American Statistical Association – Consensus versus Personal Opinion” (Dec. 13, 2019). See also Deborah G. Mayo, “The 2019 ASA Guide to P-values and Statistical Significance: Don’t Say What You Don’t Mean,” Error Statistics Philosophy (June 17, 2019); B. Haig, “The ASA’s 2019 update on P-values and significance,” Error Statistics Philosophy (July 12, 2019).
[5] Brian Tarran, “THE S WORD … and what to do about it,” Significance (Aug. 2019).
[6] Donald Macnaughton, “Who Said What,” Significance 47 (Oct. 2019).
[7] See “Statistical Significance at the New England Journal of Medicine” (July 19, 2019); See also Deborah G. Mayo, “The NEJM Issues New Guidelines on Statistical Reporting: Is the ASA P-Value Project Backfiring?” Error Statistics Philosophy (July 19, 2019).
[8] Karen Kafadar, “The Year in Review … And More to Come,” AmStat News 3 (Dec. 2019) (emphasis added); see Kafadar, “Statistics & Unintended Consequences,” AmStat News 3,4 (June 2019).
