Sander Greenland is a well-known statistician, and no stranger to the courtroom. I first encountered him as a plaintiffs’ expert witness in the silicone gel breast implant litigation, where he testified for plaintiffs in front of a panel of court-appointed expert witnesses (Drs. Diamond, Hulka, Kerkvliet, and Tugwell). Professor Greenland has testified for plaintiffs in vaccine, neurontin, fenfluramine, anti-depressant and other pharmaceutical cases. Although usually on the losing side, Greenland has written engaging post-mortems of several litigations, to attempt to vindicate his positions he took, or deconstruct positions taken by adversary expert witnesses.
In one attempt to “correct the record,” Greenland criticized a defense expert witness for stating that Bayesian methods are rarely used in medicine or in the regulation of medicines. Sander Greenland, “The Need for Critical Appraisal of Expert Witnesses in Epidemiology and Statistics,” 39 Wake Forest Law Rev. 291, 306 (2004). According to Greenland, his involvement as a plaintiff’s expert witness in a fenfluramine case allowed him to observe a senior professor in Yale University, who served as Wyeth’s statistics expert, make a “ludicrous claim,” id. (emphasis added), that
“the Bayesian method is essentially never used in the medical literature or in the regulatory environments (such as the FDA) for interpreting study results. . . .”
Id. (quoting from Supplemental Affidavit of Prof. Robert Makuch, App. Ex. 114, ¶5, in Smith v. Wyeth-Ayerst Labs., 278 F.Supp. 2d 684 (W.D.N.C. 2003)). Greenland criticizes Professor Makuch’s affidavit as “provid[ing] another disturbing case study of misleading expert testimony regarding current standards and practice.” 39 Wake Forest Law Rev. at 306.
“Ludicrous,” “disturbing,” “misleading,” and “demonstrably quite false”? Really?
Greenland notes, as a matter of background, that many leading statisticians recommend and adopt Bayesian statistics. Id. (citing works by Donald Berry, George Box, Bradley Carlin, Andrew Gelman James Berger, and others). Remarkably, however, Greenland failed to cite a single new or supplemental drug application, or even one FDA summary of safety or efficacy, or FDA post-market safety or efficacy review. At the time Greenland was preparing his indictment, there really was little or no evidence of FDA’s embrace of Bayesian methodologies. Six years later, in 2010, the agency did promulgate a guidance that set recommended practices for Bayesian analyses in medical device trials. FDA Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials (February 5, 2010); 75 Fed. Reg. 6209 (February 8, 2010); see also Laura A. Thompson, “Bayesian Methods for Making Inferences about Rare Diseases in Pediatric Populations” (2010); Greg Campbell, “Bayesian Statistics at the FDA: The Trailblazing Experience with Medical Devices” (Presentation give by Director, Division of Biostatistics Center for Devices and Radiological Health at Rutgers Biostatistics Day, April 3, 2009). Even today, Bayesian analysis remains uncommon at the U.S. FDA.
Having ignored the regulatory arena, Greenland purported to do a study of the biomedical journals, “to check the expert’s claim in detail.” 39 Wake Forest Law Rev. at 306. Greenland searched on the word “Bayesian” in the Journal of Clinical Oncology for issues published from 1994-2003, and “found over fifty publications that contain the word in that journal alone.” Greenland does not tell us why he selected this one journal, which was not in the subject matter area of the litigation in which he was serving as a partisan expert witness. For most the time surveyed, the Journal of Clinical Oncology published 24 issues a year, and occasional supplements. Most volumes contained over 4,000 pages per year. Finding 50 uses of the word “Bayesian” in over 40,000 pages hardly constitutes resounding evidence to support his charges of “ludicrous,” “misleading,” “disturbing,” and “quite false.” Greenland further tells us looking at these 50 or so articles “revealed several,” which “had used Bayesian methods to explore statistically nonsignificant results.” 39 Wake Forest Law Rev. at 306-07 & n.61 (citing only one paper, Lisa Licitra et al., Primary Chemotherapy in Resectable Oral Cavity Squamous Cell Cancer: A Randomized Controlled Trial, 21 J. Clin. Oncol. 327 (2003)). So in over 40,000 pages, Greenland found “several” Bayesian analyses, apparently post hoc looks to explore results that did not achieve pre-specified levels of statistical significance. Given the historical evolution of Bayesian analyses at FDA, and Greenland’s own evidence, the posterior odds that Greenland was correct in his charges seem to be disturbingly low.
Greenland tells us that the number of Bayesian analyses could be increased by looking at additional journals, and the Bayesian textbooks he cites. No doubt this is true, as is his statement that respected statisticians, in prestigious journals, have called for Bayesian analyses to replace frequentist methods. Of course, increasing the scope of his survey, Greenland would be dramatically increasing the denominator of total journal papers with statistical methods. Odds are that the frequency would remain very low. Greenland’s empirical evidence hardly contradicts his bête noire for making the quoted purely descriptive statement about the infrequent use of Bayesian analysis in biomedical journals and in regulatory applications.
In lodging charges of ludicrousness, Greenland might have presented a more balanced view from more carefully conducted surveys of the biomedical literature, in the relevant time period. See, e.g., J. Martin Bland & Douglas G. Altman, “Bayesians and frequentists,” 317 Brit. Med. J. 1151, 1151 (1998) (“almost all the statistical analyses which appear in the British Medical Journal are frequentist”); David S. Moore, “Bayes for Beginners? Some Reasons to Hesitate,” 51 The Am. Statistician 254, 254 (“Bayesian methods are relatively rarely used in practice”); J.D. Emerson & Graham Colditz, “Use of statistical analysis in the New England Journal of Medicine,” in John Bailar & Frederick Mosteler, eds., Medical Uses of Statistics 45 (1992) (surveying 115 original research studies for statistical methods used; no instances of Bayesian approaches counted); Douglas Altman, “Statistics in Medical Journals: Developments in the 1980s,” 10 Statistics in Medicine 1897 (1991); B.S. Everitt, “Statistics in Psychiatry,” 2 Statistical Science 107 (1987) (finding only one use of Bayesian methods in 441 papers with statistical methodology).
Perhaps the balance between frequentist and Bayesian analysis is shifting today, but when Professor Makuch made his affidavit in 2002 or so, he was clearly correct, factually and statistically.
In the legal arena, Bayesian analyses are frequently used in evaluating forensic claims about DNA, paternity, lead-isotopes, and other issues of identification. Remarkably, Bayesian analyses play virtually no role in litigation of health effects claims, whether based upon medicines, or upon occupational or environmental exposures. In searching Google scholar and Westlaw I found no cases outside of forensics. Citations to black-swan cases are welcomed.
