TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

Access to a Study Protocol & Underlying Data Reveals a Nuclear Non-Proliferation Test

April 8th, 2024

The limits of peer review ultimately make it a poor proxy for the validity tests posed by Rules 702 and 703. Published peer review articles simply do not permit a very searching evaluation of the facts and data of a study. In the wake of the Daubert decision, expert witnesses quickly saw that they can obscure the search for validity by the reliance upon published studies, and frustrate the goals of judicial gatekeeping. As a practical matter, the burden shifts to the party that wishes to challenge the relied upon facts and data to learn more about the cited studies to show that the facts and data are not sufficient under Rule 702(b), and that the testimony is not the product of reliable methods under Rule 702(c). Obtaining study protocols, and in some instances, underlying data, are necessary for due process in the gatekeeping process. A couple of case studies may illustrate the power of looking under the hood of published studies, even ones that were peer reviewed.

When the Supreme Court decided the Daubert case in June 1993, two recent verdicts in silicone-gel breast implant cases were fresh in memory.[1] The verdicts were large by the standards of the time, and the evidence presented for the claims that silicone caused autoimmune disease was extremely weak. The verdicts set off a feeding frenzy, not only in the lawsuit industry, but also in the shady entrepreneurial world of supposed medical tests for “silicone sensitivity.”

The plaintiffs’ litigation theory lacked any meaningful epidemiologic support, and so there were fulsome presentations of putative, hypothetical mechanisms. One such mechanism involved the supposed in vivo degradation of silicone to silica (silicon dioxide), with silica then inducing an immunogenic reaction, which then, somehow, induced autoimmunity and the induction of autoimmune connective tissue disease. The degradation claim would ultimately prove baseless,[2] and the nuclear magnetic resonance evidence put forward to support degradation would turn out to be instrumental artifact and deception. The immunogenic mechanism had a few lines of potential support, with the most prominent at the time coming from the laboratories of Douglas Radford Shanklin, and his colleague, David L. Smalley, both of whom were testifying expert witnesses for claimants.

The Daubert decision held out some opportunity to challenge the admissibility of testimony that silicone implants led to either the production of a silicone-specific antibody, or the induction of t-cell mediated immunogenicity from silicone (or resulting silica) exposure. The initial tests of the newly articulated standard for admissibility of opinion testimony in silicone litigation did not go well.[3]  Peer review, which was absent in the re-analyses relied upon in the Bendectin litigation, was superficially present in the studies relied upon in the silicone litigation. The absence of supportive epidemiology was excused with hand waving that there was a “credible” mechanism, and that epidemiology took too long and was too expensive. Initially, post-Daubert, federal courts were quick to excuse the absence of epidemiology for a novel claim.

The initial Rule 702 challenges to plaintiffs’ expert witnesses thus focused on  immunogenicity as the putative mechanism, which if true, might lend some plausibility to their causal claim. Ultimately, plaintiffs’ expert witnesses would have to show that the mechanism was real by showing that silicone exposure causes autoimmune disease through epidemiologic studies,

One of the more persistent purveyors of a “test” for detecting alleged silicone sensitivity came from Smalley and Shanklin, then at the University of Tennessee. These authors exploited the fears of implant recipients and the greed of lawyers by marketing a “silicone sensitivity test (SILS).” For a price, Smalley and Shanklin would test mailed-in blood specimens sent directly by lawyers or by physicians, and provide ready-for-litigation reports that claimants had suffered an immune system response to silicone exposure. Starting in 1995, Smalley and Shanklin also cranked out a series of articles at supposedly peer reviewed journals, which purported to identify a specific immune response to crystalline silica in women who had silicone gel breast implants.[4] These studies had two obvious goals. First, the studies promoted their product to the “silicone sisters,” various support groups of claimants, as well as their lawyers, and a network of supporting rheumatologists and plastic surgeons. Second, by identifying a putative causal mechanism, Shanklin could add a meretricious patina of scientific validity to the claim that silicone breast implants cause autoimmune disease, which Shanklin, as a testifying expert witness, needed to survive Rule 702 challenges.

The plaintiffs’ strategy had been to paper over the huge analytical gaps in their causal theory with complicated, speculative research, which had been peer reviewed and published. Although the quality of the journals was often suspect, and the nature of the peer review obscure, the strategy had been initially successful in deflecting any meaningful scrutiny.

Many of the silicone cases were pending in a multi-district litigation, MDL 926, before Judge Sam Pointer, in the Northern District of Alabama. Judge Pointer, however, did not believe that ruling on expert witness admissibility was a function of an MDL court, and by 1995, he started to remand cases to the transferor courts, for those courts to do what they thought appropriate under Rules 702 and 703. Some of the first remanded cases went to the District of Oregon, where they landed in front of Judge Robert E. Jones. In early 1996, Judge Jones invited briefing on expert witness challenges, and in face of the complex immunology and toxicology issues, and the emerging epidemiologic studies, he decided to appoint four technical advisors to assist him in deciding the challenges.

The addition of scientific advisors to the gatekeeper’s bench made a huge difference in the sophistication and detail of the challenges that could be lodged to the relied-upon studies. In June 1996, Judge Jones entertained extensive hearings with viva voce testimony from both challenged witnesses and subject-matter experts on topics, such as immunology and nuclear magnetic resonance spectroscopy. Judge Jones invited final argument in the form of videotaped presentations from counsel so that the videotapes could be distributed to his technical advisors later in the summer. The contrived complexity of plaintiffs’ case dissipated, and the huge analytical gaps became visible. In December 1996, Judge Jones issued his decision that excluded the plaintiffs’ expert witnesses’ proposed testimony on grounds that it failed to satisfy the requirements of Rule 702.[5]

In October 1996, while Judge Jones was studying the record, and writing his opinion in the Hall case, Judge Weinstein, with a judge from the Southern District of New York, and another from New York state trial court, conducted a two-week Rule 702 hearing, in Brooklyn. Judge Weinstein announced at the outset that he had studied the record from the Hall case, and that he would incorporate it into his record for the cases remanded to the Southern and Eastern Districts of New York.

Curious gaps in the articles claiming silicone immunogenicity, and the lack of success in earlier Rule 702 challenges, motivated the defense to obtain the study protocols and underlying data from studies such as those published by Shanklin and Smalley. Shanklin and Smalley were frequently listed as expert witnesses in individual cases, but when requests or subpoenas for their protocols and raw data were filed, plaintiffs’ counsel stonewalled or withdrew them as witnesses. Eventually, the defense was able to enforce a subpoena and obtain the protocol and some data. The respondents claimed that the control data no longer existed, and inexplicably a good part of the experimental data had been destroyed. Enough was revealed, however, to see that the published articles were not what they claimed to be.[6]

In addition to litigation discovery, in March 1996, a surgeon published the results of his test of the Shanklin-Smalley silicone sensitivity test (“SILS”).[7] Dr. Leroy Young sent the Shanklin laboratory several blood samples from women with and without silicone implants. For six women who never had implants, Dr. Young submitted a fabricated medical history that included silicone implants and symptoms of “silicone-associated disease.” All six samples were reported back as “positive”; indeed, these results were more positive than the blood samples from the women who actually had silicone implants. Dr. Young suggested that perhaps the SILS test was akin to cold fusion.

By the time counsel assembled in Judge Weinstein’s courtroom, in October 1996, some epidemiologic studies had become available and much more information was available on the supposedly supportive mechanistic studies upon which plaintiffs’ expert witnesses had previously relied. Not too surprisingly, plaintiffs’ counsel chose not to call the entrepreneurial Dr. Shanklin, but instead called Donard S. Dwyer, a young, earnest immunologist who had done some contract work on an unrelated matter for Bristol-Myers Squibb, a defendant in the litigation.  Dr. Dwyer had filed an affidavit previously in the Oregon federal litigation, in which he gave blanket approval to the methods and conclusions of the Smalley-Shanklin research:

“Based on a thorough review of these extensive materials which are more than adequate to evaluate Dr. Smalley’s test methodology, I formed the following conclusions. First, the experimental protocols that were used are standard and acceptable methods for measuring T Cell proliferation. The results have been reproducible and consistent in this laboratory. Second, the conclusion that there are differences between patients with breast implants and normal controls with respect to the proliferative response to silicon dioxide appears to be justified from the data.”[8]

Dwyer maintained this position even after the defense obtaining the study protocol and underlying data, and various immunologists on the defense side filed scathing evaluatons of the Smalley-Shanklin work.  On direct examination at the hearings in Brooklyn, Dwyer vouched for the challenged t-cell studies, and opined that the work was peer reviewed and sufficiently reliable.[9]

The charade fell apart on cross-examination. Dwyer refused to endorse the studies that claimed to have found an anti-silicone antibody. Researchers at leading universities had attempted to reproduce the findings of such antibodies, without success.[10] The real controversy was over the claimed finding of silicone antigenicity as shown in t-cell or the cell-mediated specific immune response. On direct examination, plaintiffs’ counsel elicited Dwyer’s support for the soundness of the scientific studies that purported to establish such antigenicity, with little attention to the critiques that had been filed before the hearing.[11] Dwyer stuck to his unqualified support he had expressed previously in his affidavit for the Oregon cases.[12]

The problematic aspect of Dwyer’s direct examination testimony was that he had seen the protocol and the partial data produced by Smalley and Shanklin.[13] Dwyer, therefore, could not resist some basic facts about their work. First, the Shanklin data failed to support a dose-response relationship.[14] Second, the blood samples from women with silicone implants had been mailed to Smalley’s laboratory, whereas the control samples were collected locally. The disparity ensured that the silicone blood samples would be older than the controls, which was a departure from treating exposed and control samples in the same way.[15] Third, the experiment was done unblinded; the laboratory technical personnel and the investigators knew which blood samples were silicone exposed and which were controls (except for samples sent by Dr. Leroy Young).[16] Fourth, Shanklin’s laboratory procedures deviated from the standardized procedure set out in the National Institute of Health’s Current Protocols in Immunology.[17]

The SILS study protocol and the data produced by Shanklin and Smalley made clear that each sample was to be tested in triplicate for t-cell proliferation in response to silica, to a positive control mitogen (Con A), and to a negative control blank. The published papers all claimed that the each sample was tested in triplicate for each of these three response situations (silica, mitogen, and nothing).[18] Shanklin and Smalley described their t-cell proliferation studies, in their published papers, as having been done in triplicate. These statements were, however, untrue and never corrected.[19]

The study protocol called for the tests to be run in triplicate, but they instructed the laboratory that two counts may be used if one count does not match the other counts, which is to be decided by a technical specialist on a “case-by-case” basis. Of data that was supposed to be reported in triplicate, fully one third had only two data points, and 10 percent had but one data point.[20] No criteria were provided to the technical specialist for deciding which data to discard.[21] Not only had Shanklin excluded data, but he discarded and destroyed the data such that no one could go back and assess whether the data should have been excluded.[22]

Dwyer agreed that this exclusion and discarding of data was not at all a good method.[23] Dwyer proclaimed that he had not come to Brooklyn to defend this aspect of the Shanklin work, and that it was not defensible at all. Dwyer conceded that “the interpretation of the data and collection of the data are flawed.”[24] Dwyer tried to stake out a position that was incoherent by asserting that there was “nothing inherently wrong with the method,” while conceding that discarding data was problematic.[25] The judges presiding over the hearing could readily see that the Shanklin research was bent.

At this point, the lead plaintiffs’ counsel, Michael Williams, sought an off-ramp. He jumped to his feet and exclaimed “I’m informed that no witness in this case will rely on Dr. Smalley’s [and Shanklin’s] work in any respect.” [26] Judge Weinstein’s eyes lit up with the prospect that the Smalley-Shanklin work, by agreement, would never be mentioned again in New York state or federal cases. Given how central the claim of silicone antigenicity was to plaintiffs’ cases, the defense resisted the stipulation about research that they would continue to face in other state and federal courts. The defense was saved, however, by the obstinence of a lawyer from the Weitz & Luxenberg firm, who rose to report that her firm intended to call Drs. Shanklin and Smalley as witnesses, and that they would not stipulate to the exclusion of their work. Judge Weinstein rolled his eyes, and waved me to continue.[27] The proliferation of the t-cell test was over. The hearing before Judges Weinstein and Baer, and Justice Lobis, continued for several more days, with several other dramatic moments.[28]

In short order, on October 23, 1996, Judge Weinstein issued a short, published opinion, in which he granted partial summary judgment on the claims of systemic disease for all cases pending in federal court in New York.[29] What was curious was that the defendants had not moved for summary judgment. There were, of course, pending motions to exclude plaintiffs’ expert witnesses, but Judge Weinstein effectively ducked those motions, and let it be known that he was never a fan of Rule 702. It would be many years later, before Judge Weinstein allowed his judicial assessment see the light of day. Two decades and some years later, in a law review article, Judge Weinstein gave his judgment that

“[t]he breast implant litigation was largely based on a litigation fraud. …  Claims—supported by medical charlatans—that enormous damages to women’s systems resulted could not be supported.”[30]

Judge Weinstein’s opinion was truly a judgment from which there can be no appeal. Shanklin and Smalley continued to publish papers for another decade. None of the published articles by Shanklin and others have been retracted.

[1] Reuters, “Record $25 Million Awarded In Silicone-Gel Implants Case,” N.Y. Times at A13 (Dec. 24, 1992) (describing the verdict returned in Harris County, Texas, in Johnson v. Medical Engineering Corp.); Associated Press, “Woman Wins Implant Suit,” N.Y. Times at A16 (Dec. 17, 1991) (reporting a verdict in Hopkins v. Dow Corning, for $840,000 in compensatory and $6.5 million in punitive damages); see Hopkins v. Dow Corning Corp., 33 F.3d 1116 (9th Cir. 1994) (affirming judgment with minimal attention to Rule 702 issues).

[2] William E. Hull, “A Critical Review of MR Studies Concerning Silicone Breast Implants,” 42 Magnetic Resonance in Medicine 984, 984 (1999) (“From my viewpoint as an analytical spectroscopist, the result of this exercise was disturbing and disappointing. In my judgement as a referee, none of the Garrido group’s papers (1–6) should have been published in their current form.”). See also N.A. Schachtman, “Silicone Data – Slippery & Hard to Find, Part 2,” Tortini (July 5, 2015). Many of the material science claims in the breast implant litigation were as fraudulent as the health effects claims. See, e.g., John Donley, “Examining the Expert,” 49 Litigation 26 (Spring 2023) (discussing his encounters with frequent testifier Pierre Blais, in silicone litigation).

[3] See, e.g., Hopkins v. Dow Corning Corp., 33 F.3d 1116 (9th Cir. 1994) (affirming judgment for plaintiff over Rule 702 challenges), cert. denied, 115 S.Ct. 734 (1995). See Donald A. Lawson, “Note, Hopkins v. Dow Corning Corporation: Silicone and Science,” 37 Jurimetrics J. 53 (1996) (concluding that Hopkins was wrongly decided).

[4] See David L. Smalley, Douglas R. Shanklin, Mary F. Hall, and Michael V. Stevens, “Detection of Lymphocyte Stimulation by Silicon Dioxide,” 4 Internat’l J. Occup. Med. & Toxicol. 63 (1995); David L. Smalley, Douglas R. Shanklin, Mary F. Hall, Michael V. Stevens, and Aram Hanissian, “Immunologic stimulation of T lymphocytes by silica after use of silicone mammary implants,” 9 FASEB J. 424 (1995); David L. Smalley, J. J. Levine, Douglas R. Shanklin, Mary F. Hall, Michael V. Stevens, “Lymphocyte response to silica among offspring of silicone breast implant recipients,” 196 Immunobiology 567 (1996); David L. Smalley, Douglas R. Shanklin, “T-cell-specific response to silicone gel,” 98 Plastic Reconstr. Surg. 915 (1996); and Douglas R. Shanklin, David L. Smalley, Mary F. Hall, Michael V. Stevens, “T cell-mediated immune response to silica in silicone breast implant patients,” 210 Curr. Topics Microbiol. Immunol. 227 (1996). Shanklin was also no stranger to making his case in the popular media. See, e.g., Douglas Shanklin, “More Research Needed on Breast Implants,” Kitsap Sun at 2 (Aug. 29, 1995) (“Widespread silicone sickness is very real in women with past and continuing exposure to silicone breast implants.”) (writing for Scripps Howard News Service). Even after the Shanklin studies were discredited in court, Shanklin and his colleagues continued to publish their claims that silicone implants led to silica antigenicity. David L. Smalley, Douglas R. Shanklin, and Mary F. Hall, “Monocyte-dependent stimulation of human T cells by silicon dioxide,” 66 Pathobiology 302 (1998); Douglas R. Shanklin and David L. Smalley, “The immunopathology of siliconosis. History, clinical presentation, and relation to silicosis and the chemistry of silicon and silicone,” 18 Immunol. Res. 125 (1998); Douglas Radford Shanklin, David L. Smalley, “Pathogenetic and diagnostic aspects of siliconosis,” 17 Rev. Environ Health 85 (2002), and “Erratum,” 17 Rev Environ Health. 248 (2002); Douglas Radford Shanklin & David L Smalley, “Kinetics of T lymphocyte responses to persistent antigens,” 80 Exp. Mol. Pathol. 26 (2006). Douglas Shanklin died in 2013. Susan J. Ainsworth, “Douglas R. Shanklin,” 92 Chem. & Eng’g News (April 7, 2014). Dr. Smalley appears to be still alive. In 2022, he sued the federal government to challenge his disqualification from serving as a laboratory director of any clinical directory in the United States, under 42 U.S.C. § 263a(k). He lost. Smalley v. Becerra, Case No. 4:22CV399 HEA (E.D. Mo. July 6, 2022).

[5] Hall v. Baxter Healthcare Corp., 947 F. Supp. 1387 (D. Ore. 1996); see Joseph Sanders & David H. Kaye, “Expert Advice on Silicone Implants: Hall v. Baxter Healthcare Corp., 37 Jurimetrics J. 113 (1997); Laurens Walker & John Monahan, “Scientific Authority: The Breast Implant Litigation and Beyond,” 86 Virginia L. Rev. 801 (2000); Jane F. Thorpe, Alvina M. Oelhafen, and Michael B. Arnold, “Court-Appointed Experts and Technical Advisors,” 26 Litigation 31 (Summer 2000); Laural L. Hooper, Joe S. Cecil & Thomas E. Willging, “Assessing Causation in Breast Implant Litigation: The Role of Science Panels,” 64 Law & Contemp. Problems 139 (2001); Debra L. Worthington, Merrie Jo Stallard, Joseph M. Price & Peter J. Goss, “Hindsight Bias, Daubert, and the Silicone Breast Implant Litigation: Making the Case for Court-Appointed Experts in Complex Medical and Scientific Litigation,” 8 Psychology, Public Policy &  Law 154 (2002).

[6] Judge Jones’ technical advisor on immunology reported that the studies offered in support of the alleged connection between silicone implantation and silicone-specific T cell responses, including the published papers by Shanklin and Smalley, “have a number of methodological shortcomings and thus should not form the basis of such an opinion.” Mary Stenzel-Poore, “Silicone Breast Implant Cases–Analysis of Scientific Reasoning and Methodology Regarding Immunological Studies” (Sept. 9, 1996). This judgment was seconded, over three years later, in the proceedings before MDL 926 and its Rule 706 court-appointed immunology expert witness. See Report of Dr. Betty A. Diamond, in MDL 926, at 14-15 (Nov. 30, 1998). Other expert witnesses who published studies on the supposed immunogenicity of silicone came up with some creative excuses to avoid producing their underlying data. Eric Gershwin consistently testified that his data were with a co-author in Israel, and that he could not produce them. N.A. Schachtman, “Silicone Data – Slippery and Hard to Find, Part I,” Tortini (July 4, 2015). Nonetheless, the court-appointed technical advisors were highly critical of Dr. Gershwin’s results. Dr. Stenzel-Poore, the immunologist on Judge Jones’ panel of advisors, found Gershwin’s claims “not well substantiated.” Hall v. Baxter Healthcare Corp., 947 F.Supp. 1387 (D. Ore. 1996). Similarly, Judge Pointer’s appointed expert immunologist Dr. Betty A. Diamond, was unshakeable in her criticisms of Gershwin’s work and his conclusions. Testimony of Dr. Betty A. Diamond, in MDL 926 (April 23, 1999). And the Institute of Medicine committee, charged with reviewing the silicone claims, found Gershwin’s work inadequate and insufficient to justify the extravagent claims that plaintiffs were making for immunogenicity and for causation of autoimmune disease. Stuart Bondurant, Virginia Ernster, and Roger Herdman, eds., Safety of Silicone Breast Implants 256 (1999). Another testifying expert witness who relied upon his own data, Nir Kossovsky, resorted to a seismic excuse; he claimed that the Northridge Quake destroyed his data. N.A. Schachtman, “Earthquake Induced Data Loss – We’re All Shook Up,” Tortini (June 26, 2015); Kossovsky, along with his wife, Beth Brandegee, and his father, Ram Kossowsky, sought to commercialize an ELISA-based silicone “antibody” biomarker diagnostic test, Detecsil. Although the early Rule 702 decisions declined to take a hard at Kossovsky’s study, the U.S. Food and Drug Administration eventually shut down the Kossovsky Detecsil test. Lillian J. Gill, FDA Acting Director, Office of Compliance, Letter to Beth S. Brandegee, President, Structured Biologicals (SBI) Laboratories: Detecsil Silicone Sensitivity Test (July 15, 1994); see Gary Taubes, “Silicone in the System: Has Nir Kossovsky really shown anything about the dangers of breast implants?” Discover Magazine (Dec. 1995).

[7] Leroy Young, “Testing the Test: An Analysis of the Reliability of the Silicone Sensitivity Test (SILS) in Detecting Immune-Mediated Responses to Silicone Breast Implants,” 97 Plastic & Reconstr. Surg. 681 (1996).

[8] Affid. of Donard S. Dwyer, at para. 6 (Dec. 1, 1995), filed in In re Breast Implant Litig. Pending in U.S. D. Ct, D. Oregon (Groups 1,2, and 3).

[9] Notes of Testimony of Dr. Donnard Dwyer, Nyitray v. Baxter Healthcare Corp., CV 93-159 (E. & S.D.N.Y and N.Y. Sup. Ct., N.Y. Cty. Oct. 8, 9, 1996) (Weinstein, J., Baer, J., Lobis, J., Pollak, M.J.).

[10] Id. at N.T. 238-239 (Oct. 8, 1996).

[11] Id. at N.T. 240.

[12] Id. at N.T. 241-42.

[13] Id. at N.T. 243-44; 255:22-256:3.

[14] Id. at 244-45.

[15] Id. at N.T. 259.

[16] Id. at N.T. 258:20-22.

[17] Id. at N.T. 254.

[18] Id. at N.T. 252:16-254.

[19] Id. at N.T. 254:19-255:2.

[20] Id. at N.T. 269:18-269:14.

[21] Id. at N.T. 261:23-262:1.

[22] Id. at N.T. 269:18-270.

[23] Id. atN.T. 256:3-16.

[24] Id. at N.T. 262:15-17

[25] Id. at N.T. 247:3-5.

[26] Id. at N.T. at 260:2-3

[27] Id. at N.T. at 261:5-8.

[28] One of the more interesting and colorful moments came when the late James Conlon cross-examined plaintiffs’ pathology expert witness, Saul Puszkin, about questionable aspects of his curriculum vitae. The examination was revealed such questionable conduct that Judge Weinstein stopped the examination and directed Dr. Puszkin not to continue without legal counsel of his own.

[29] In re Breast Implant Cases, 942 F. Supp. 958 (E.& S.D.N.Y. 1996). The opinion did not specifically address the Rule 702 and 703 issues that were the subject of pending motions before the court.

[30] Hon. Jack B. Weinstein, “Preliminary Reflections on Administration of Complex Litigation” 2009 Cardozo L. Rev. de novo 1, 14 (2009) (emphasis added).

Posted in Data Sharing, Expert Witness Hall of Shame, Rule 702, Rule 703, Scientific Misconduct, Scientific Publishing | 68 Comments »

Peer Review, Protocols, and QRPs

April 3rd, 2024

In Daubert, the Supreme Court decided a legal question about the proper interpretation of a statute, Rule 702, and then remanded the case to the Ninth Circuit of the Court of Appeals for further proceedings. The Court did, however, weigh in with dicta about some several considerations in admissibility decisions.  In particular, the Court identified four non-dispositive factors: whether the challenged opinion has been empirically tested, published and peer reviewed, and whether the underlying scientific technique or method supporting the opinion has an acceptable rate of error, and has gained general acceptance.[1]

The context in which peer review was discussed in Daubert is of some importance to our understanding its holding peer review out as a consideraton. One of the bases for the defense challenges to some of the plaintiffs’ expert witnesses’ opinions in Daubert was their reliance upon re-analyses of published studies to suggest that there was indeed an increased risk of birth defects if only the publication authors had used some other control group, or taken some other analytical approach. Re-analyses can be important, but these reanalyses of published Bendectin studies were post hoc, litigation driven, and obviously result oriented. The Court’s discussion of peer review reveals that it was not simply creating a box to be checked before a trial court could admit an expert witness’s opinions. Peer review was suggested as a consideration because:

“submission to the scrutiny of the scientific community is a component of “good science,” in part because it increases the likelihood that substantive flaws in methodology will be detected. The fact of publication (or lack thereof) in a peer reviewed journal thus will be a relevant, though not dispositive, consideration in assessing the scientific validity of a particular technique or methodology on which an opinion is premised.”[2]

Peer review, or the lack thereof, for the challenged expert witnesses’ re-analyses was called out because it raised suspicions of lack of validity. Nothing in Daubert, or in later decisions, or more importantly in Rule 702 itself, supports admitting expert witness testimony just because the witness relied upon peer-reviewed studies, especially when the studies are invalid or are based upon questionable research practices. The Court was careful to point out that peer-reviewed publication was “not a sine qua non of admissibility; it does not necessarily correlate with reliability, … .”[3] The Court thus showed that it was well aware that well-ground (and thus admissible) opinions may not have been previously published, and that the existence of peer review was simply a potential aid in answering the essential question, whether the proponent of a proffered opinion has shown “the scientific validity of a particular technique or methodology on which an opinion is premised.”[4]

Since 1993, much has changed in the world of bio-science publishing. The wild proliferation of journals, including predatory and “pay-to-play” journals, has disabused most observers that peer review provides evidence of validity of methods. Along with the exponential growth in publications has come an exponential growth in expressions of concern and out-right retractions of articles, as chronicled and detailed at Retraction Watch.[5] Some journals encourage authors to nominate the peer reviewers for their manuscripts; some journals let authors block some scientists as peer reviewers of their submitted manuscripts. If the Supreme Court were writing today, it might well note that peer review is often a feature of bad science, advanced by scientists who know that peer-reviewed publication is the price of admission to the advocacy arena.

Since the Supreme Court decided Daubert, the Federal Judicial Center and National Academies of Science have provided a Reference Manual for Scientific Evidence, now in its third edition, and with a fourth edition on the horizon, to assist judges and lawyers involved in the litigation of scientific issues. Professor Goodstein, in his chapter “How Science Works,” in the third edition, provides the most extensive discussion of peer review in the Manual, and emphasizes that peer review “works very poorly in catching cheating or fraud.”[6]  Goodstein invokes his own experience as a peer reviewer to note that “peer review referees and editors limit their assessment of submitted articles to such matters as style, plausibility, and defensibility; they do not duplicate experiments from scratch or plow through reams of computer-generated data in order to guarantee accuracy or veracity or certainty.”[7] Indeed, Goodstein’s essay in the Reference Manual characterizes the ability of peer review to warrant study validity as a “myth”:

“Myth: The institution of peer review assures that all published papers are sound and dependable.

Fact: Peer review generally will catch something that is completely out of step with majority thinking at the time, but it is practically useless for catching outright fraud, and it is not very good at dealing with truly novel ideas. … It certainly does not ensure that the work has been fully vetted in terms of the data analysis and the proper application of research methods.”[8]

Goodstein’s experience as a peer reviewer is hardly idiosyncratic. One standard text on the ethical conduct of research reports that peer review is often ineffective or incompetent, and that it may not even catch simple statistical or methodological errors.[9] According to the authors, Shamoo and Resnik:

“[p]eer review is not good at detecting data fabrication or falsification partly because reviewers usually do not have access to the material they would need to detect fraud, such as the original data, protocols, and standard operating procedures.”[10]

Indeed, without access to protocols, statistical analysis plans, and original data, peer review often cannot identify good faith or negligent deviations from the standard of scientific care. There is some evidence to support this negative assessment of peer review from testing of the counter-factual. Reviewers were able to detect questionable, selective reporting when they had access to the study authors’ research protocols.[11]

Study Protocol

The study protocol provides the scientific rationale for a study, clearly defines the research question, the data collection process, defines the key exposure and outcomes, and describes the methods to be applied, before commencing data collection.[12] The protocol also typically pre-specifies the statistical data analysis. The epidemiology chapter of the current edition of the Reference Manual for Scientific Evidence offers blandly only that epidemiologists attempt to minimize bias in observational studies with “data collection protocols.”[13] Epidemiologists and statisticians are much clearer in emphasizing the importance, indeed the necessity, of having a study protocol before commencing data collection. Back in 1988, John Bailar and Frederick Mosteller explained that it was critical in reporting statistical analyses to inform readers about how and when the authors devised the study design, and whether they set the design criteria out in writing before they began to collect data.[14]

The necessity of a study protocol is “self-evident,”[15] and essential to research integrity.[16] The International Society of Pharmacoepidemiology has issued Guidelines for “Good Pharmacoepidemiology Practices,”[17] which calls for every study to have a written protocol. Among the requirements set out in this set of guidelines are descriptions of the research method, study design, operational definitions of exposure and outcome variables, and projected study sample size. The Guidelines provide that a detailed statistical analysis plan may be specified after data collection begins, but before any analysis commences.

Expert witness opinions on health effects are built upon studies, and so it behooves legal counsel to identify the methodological strengths and weaknesses of key studies through questioning whether they have protocols, whether the protocols were methodologically appropriate, and whether the researchers faithfully followed their protocols and their statistical analysis plans. Determining the peer review status of a publication, on the other hand, will often not advance a challenge based upon improvident methodology.

In some instances, a published study will have sufficiently detailed descriptions of methods and data that readers, even lawyers, can evaluate their scientific validity or reliability (vel non). In some cases, however, readers will be no better off than the peer reviewers who were deprived of access to protocols, statistical analysis plans, and original data. When a particular study is crucial support for an adversary’s expert witness, a reasonable litigation goal may well be to obtain the protocol and statistical analysis plan, and if need be, the original underlying data. The decision to undertake such discovery is difficult. Discovery of non-party scientists can be expensive and protracted; it will almost certainly be contentious. When expert witnesses rely upon one or a few studies, which telegraph internal validity, this litigation strategy may provide the strongest evidence against the study’s being reasonably relied upon, or its providing “sufficient facts and data” to support an admissible expert witness opinion.

[1] Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579, 593-594 (1993).

[2] Id. at 594 (internal citations omitted) (emphasis added).

[3] Id.

[4] Id. at 593-94.

[5] Retraction Watch, at https://retractionwatch.com/.

[6] Reference Manual on Scientific Evidence at 37, 44-45 (3rd ed. 2011) [Manual].

[7] Id. at 44-45 n.11.

[8] Id. at 48 (emphasis added).

[9] Adil E. Shamoo and David B. Resnik, Responsible Conduct of Research 133 (4th ed. 2022).

[10] Id.

[11] An-Wen Chan, Asbjørn Hróbjartsson, Mette T. Haahr, Peter C. Gøtzsche, and David G. Altman, D. G. “Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles,” 291 J. Am. Med. Ass’n 2457 (2004).

[12] Wolfgang Ahrens & Iris Pigeot, eds., Handbook of Epidemiology 477 (2nd ed. 2014).

[13] Michael D. Green, D. Michal Freedman, and Leon Gordis, “Reference Guide on Epidemiology,” in Reference Manual on Scientific Evidence 573 (3rd ed. 2011) 573 (“Study designs are developed before they begin gathering data.”).

[14] John Bailar & Frederick Mosteller, “Guidelines for Statistical Reporting in Articles for Medical Journals,” 108 Ann. Intern. Med. 2266, 268 (1988).

[15] Wolfgang Ahrens & Iris Pigeot, eds., Handbook of Epidemiology 477 (2nd ed. 2014).

[16] Sandra Alba, et al., “Bridging research integrity and global health epidemiology statement: guidelines for good epidemiological practice,” 5 BMJ Global Health e003236, at p.3 & passim (2020).

[17] See “The ISPE Guidelines for Good Pharmacoepidemiology Practices (GPP),” available at <https://www.pharmacoepi.org/resources/policies/guidelines-08027/>.

Posted in Data Sharing, Expert Witness Discovery, Rule 702, Rule 703, statistical evidence, Underlying Data | 12 Comments »

QRPs in Science and in Court

April 2nd, 2024

Lay juries usually function well in assessing the relevance of an expert witness’s credentials, experience, command of the facts, likeability, physical demeanor, confidence, and ability to communicate. Lay juries can understand and respond to arguments about personal bias, which no doubt is why trial lawyers spend so much time and effort to emphasize the size of fees and consulting income, and the propensity to testify only for one side. For procedural and practical reasons, however, lay juries do not function very well in assessing the actual merits of scientific controversies. And with respect to methodological issues that underlie the merits, juries barely function at all. The legal system imposes no educational or experiential qualifications for jurors, and trials are hardly the occasion to teach jurors the methodology, skills, and information needed to resolve methodological issues that underlie a scientific dispute.

Scientific studies, reviews, and meta-analyses are virtually never directly admissible in evidence in courtrooms in the United States. As a result, juries do not have the opportunity to read and ponder the merits of these sources, and assess their strengths and weaknesses. The working assumption of our courts is that juries are not qualified to engage directly with the primary sources of scientific evidence, and so expert witnesses are called upon to deliver opinions based upon a scientific record not directly in evidence. In the litigation of scientific disputes, our courts thus rely upon the testimony of so-called expert witnesses in the form of opinions. Not only must juries, the usual trier of fact in our courts, assess the credibility of expert witnesses, but they must assess whether expert witnesses are accurately describing studies that they cannot read in their entirety.

The convoluted path by which science enters the courtroom supports the liberal and robust gatekeeping process outlined under Rules 702 and 703 of the Federal Rules of Evidence. The court, not the jury, must make a preliminary determination, under Rule 104, that the facts and data of a study are reasonably relied upon by an expert witness (Rule 703). And the court, not the jury, again under Rule 104, must determine that expert witnesses possess appropriate qualifications for relevant expertise, and that these witnesses have proffered opinions sufficiently supported by facts or data, based upon reliable principles and methods, and reliably applied to the facts of the case. (Rule 702). There is no constitutional right to bamboozle juries with inconclusive, biased, and confounded or crummy studies, or selective and incomplete assessments of the available facts and data. Back in the days of “easy admissibility,” opinions could be tested on cross-examination, but limited time and acumen of counsel, court, and juries cry out for meaningful scientific due process along the lines set out in Rules 702 and 703.

The evolutionary development of Rules 702 and 703 has promoted a salutary convergence between science and law. According to one historical overview of systematic reviews in science, the foundational period for such reviews (1970-1989) overlaps with the enactment of Rules 702 and 703, and the institutionalization of such reviews (1990-2000) coincides with the development of these Rules in a way that introduced some methodological rigor into scientific opinions that are admitted into evidence.[1]

The convergence between legal admissibility and scientific validity considerations has had the further result that scientific concerns over the quality and sufficiency of underlying data, over the validity of study design, analysis, reporting, and interpretation, and over the adequacy and validity of data synthesis, interpretation, and conclusions have become integral to the gatekeeping process. This convergence has the welcome potential to keep legal judgments more in line with best scientific evidence and practice.

The science-law convergence also means that courts must be apprised of, and take seriously, the problems of study reproducibility, and more broadly, the problems raised by questionable research practices (QRPs), or what might be called the patho-epistemology of science. The development, in the 1970s, and the subsequent evolution, of the systematic review represented the scientific community’s rejection of the old-school narrative reviews that selected a few of all studies to support a pre-existing conclusion. Similarly, the scientific community’s embarrassment, in the 1980s and 1990s, over the irreproducibility of study results, has in this century grown into an existential crisis over study reproducibility in the biomedical sciences.

In 2005, John Ioannidis published an article that brought the concern over “reproducibility” of scientific findings in bio-medicine to an ebullient boil.[2] Ioannidis pointed to several factors, which alone or in combination rendered most published medical findings likely false. Among the publication practices responsible for this unacceptably high error rate, Ioannidis identified the use of small sample sizes, data-dredging and p-hacking techniques, poor or inadequate statistical analysis, in the context of undue flexibility in research design, conflicts of interest, motivated reasoning, fads, and prejudices, and pressure to publish “positive” results.  The results, often with small putative effect sizes, across an inadequate number of studies, are then hyped by lay and technical media, as well as the public relations offices of universities and advocacy groups, only to be further misused by advocates, and further distorted to serve the goals of policy wonks. Social media then reduces all the nuances of a scientific study to an insipid meme.

Ioannidis’ critique resonated with lawyers. We who practice in health effects litigation are no strangers to dubious research methods, lack of accountability, herd-like behavior, and a culture of generating positive results, often out of political or economic sympathies. Although we must prepare for confronting dodgy methods in front of jury, asking for scientific due process that intervenes and decides the methodological issues with well-reasoned, written opinions in advance of trial does not seem like too much.

The sense that we are awash in false-positive studies was heightened by subsequent papers. In 2011, Uri Simonsohn and others showed that by using simulations of various combinations of QRPs in psychological science, researchers could attain a 61% false-positive rate for research outcomes.[3] The following year saw scientists at Amgen attempt replication of 53 important studies in hematology and oncology. They succeeded in replicated only six.[4] Also in 2012, Dr. Janet Woodcock, director of the Center for Drug Evaluation and Research at the Food and Drug Administration, “estimated that as much as 75 per cent of published biomarker associations are not replicable.”[5] In 2016, the journal Nature reported that over 70% of scientists who responded to a survey had unsuccessfully attempted to replicate another scientist’s experiments, and more than half failed to replicate their own work.[6] Of the respondents, 90% agreed that there was a replication problem. A majority of the 90% believed that the problem was significant.

The scientific community reacted to the perceived replication crisis in a variety of ways, from conceptual clarification of the very notion of reproducibility,[7] to identification of improper uses and interpretations of key statistical concepts,[8] to guidelines for improved conduct and reporting of studies.[9]

Entire books dedicated to identifying the sources of, and the correctives for, undue researcher flexibility in the design, conduct, and analysis of studies, have been published.[10] In some ways, the Rule 702 and 703 case law is like the collected works of the Berenstain Bears, on how not to do studies.

The consequences of the replication crisis are real and serious. Badly conducted and interpreted science leads to research wastage,[11] loss of confidence in scientific expertise,[12] contemptible legal judgments, and distortion of public policy.

The proposed correctives to QRPs deserve the careful study of lawyers and judges who have a role in health effects litigation.[13] Whether as the proponent of an expert witness, or the challenger, several of the recurrent proposals, such as the call for greater data sharing and pre-registration of protocols and statistical analysis plans,[14] have real-world litigation salience. In many instances, they can and should direct lawyers’ efforts at discovery and challenging of the relied upon scientific studies in litigation.

[1] Quan Nha Hong & Pierre Pluye, “Systematic Reviews: A Brief Historical Overview,” 34 Education for Information 261 (2018); Mike Clarke & Iain Chalmers, “Reflections on the history of systematic reviews,” 23 BMJ Evidence-Based Medicine 122 (2018); Cynthia Farquhar & Jane Marjoribanks, “A short history of systematic reviews,” 126 Brit. J. Obstetrics & Gynaecology 961 (2019); Edward Purssell & Niall McCrae, “A Brief History of the Systematic Review,” chap. 2, in Edward Purssell & Niall McCrae, How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students 5 (2020).

[2] John P. A. Ioannidis “Why Most Published Research Findings Are False,” 1 PLoS Med 8 (2005).

[3] Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn, “False-Positive Psychology: UndisclosedFlexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” 22 Psychological Sci. 1359 (2011).

[4] C. Glenn Begley and Lee M. Ellis, “Drug development: Raise standards for preclinical cancer research,” 483 Nature 531 (2012).

[5] Edward R. Dougherty, “Biomarker Development: Prudence, risk, and reproducibility,” 34 Bioessays 277, 279 (2012); Turna Ray, “FDA’s Woodcock says personalized drug development entering ‘long slog’ phase,” Pharmacogenomics Reporter (Oct. 26, 2011).

[6] Monya Baker, “Is there a reproducibility crisis,” 533 Nature 452 (2016).

[7] Steven N. Goodman, Daniele Fanelli, and John P. A. Ioannidis, “What does research reproducibility mean?,” 8 Science Translational Medicine 341 (2016); Felipe Romero, “Philosophy of science and the replicability crisis,” 14 Philosophy Compass e12633 (2019); Fiona Fidler & John Wilcox, “Reproducibility of Scientific Results,” Stanford Encyclopedia of Philosophy (2018), available at https://plato.stanford.edu/entries/scientific-reproducibility/.

[8] Andrew Gelman and Eric Loken, “The Statistical Crisis in Science,” 102 Am. Scientist 460 (2014); Ronald L. Wasserstein & Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” 70 The Am. Statistician 129 (2016); Yoav Benjamini, Richard D. DeVeaux, Bradly Efron, Scott Evans, Mark Glickman, Barry Braubard, Xuming He, Xiao Li Meng, Nancy Reid, Stephen M. Stigler, Stephen B. Vardeman, Christopher K. Wikle, Tommy Wright, Linda J. Young, and Karen Kafadar, “The ASA President’s Task Force Statement on Statistical Significance and Replicability,” 15 Annals of Applied Statistics 1084 (2021).

[9] The International Society for Pharmacoepidemiology issued its first Guidelines for Good Pharmacoepidemiology Practices in 1996. The most recent revision, the third, was issued in June 2015. See “The ISPE Guidelines for Good Pharmacoepidemiology Practices (GPP),” available at https://www.pharmacoepi.org/resources/policies/guidelines-08027/. See also Erik von Elm, Douglas G. Altman, Matthias Egger, Stuart J. Pocock, Peter C. Gøtzsche, and Jan P. Vandenbroucke, for the STROBE Initiative, “The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement Guidelines for Reporting Observational Studies,” 18 Epidem. 800 (2007); Jan P. Vandenbroucke, Erik von Elm, Douglas G. Altman, Peter C. Gøtzsche, Cynthia D. Mulrow, Stuart J. Pocock, Charles Poole, James J. Schlesselman, and Matthias Egger, for the STROBE initiative, “Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration,” 147 Ann. Intern. Med. W-163 (2007); Shah Ebrahim & Mike Clarke, “STROBE: new standards for reporting observational epidemiology, a chance to improve,” 36 Internat’l J. Epidem. 946 (2007); Matthias Egger, Douglas G. Altman, and Jan P Vandenbroucke of the STROBE group, “Commentary: Strengthening the reporting of observational epidemiology—the STROBE statement,” 36 Internat’l J. Epidem. 948 (2007).

[10] See, e.g., Lee J. Jussim, Jon A. Krosnick, and Sean T. Stevens, eds., Research Integrity: Best Practices for the Social and Behavioral Sciences (2022); Joel Faintuch & Salomão Faintuch, eds., Integrity of Scientific Research: Fraud, Misconduct and Fake News in the Academic, Medical and Social Environment (2022); William O’Donohue, Akihiko Masuda & Scott Lilienfeld, eds., Avoiding Questionable Research Practices in Applied Psychology (2022); Klaas Sijtsma, Never Waste a Good Crisis: Lessons Learned from Data Fraud and Questionable Research Practices (2023).

[11] See, e.g., Iain Chalmers, Michael B Bracken, Ben Djulbegovic, Silvio Garattini, Jonathan Grant, A Metin Gülmezoglu, David W Howells, John P A Ioannidis, and Sandy Oliver, “How to increase value and reduce waste when research priorities are set,” 383 Lancet 156 (2014); John P A Ioannidis, Sander Greenland, Mark A Hlatky, Muin J Khoury, Malcolm R Macleod, David Moher, Kenneth F Schulz, and Robert Tibshirani, “Increasing value and reducing waste in research design, conduct, and analysis,” 383 Lancet 166 (2014).

[12] See, e.g., Friederike Hendriks, Dorothe Kienhues, and Rainer Bromme, “Replication crisis = trust crisis? The effect of successful vs failed replications on laypeople’s trust in researchers and research,” 29 Public Understanding Sci. 270 (2020).

[13] R. Barker Bausell, The Problem with Science: The Reproducibility Crisis and What to Do About It (2021).

[14] See, e.g., Brian A. Noseka, Charles R. Ebersole, Alexander C. DeHavena, and David T. Mellora, “The preregistration revolution,” 115 Proc. Nat’l Acad. Soc. 2600 (2018); Michael B. Bracken, “Preregistration of Epidemiology Protocols: A Commentary in Support,” 22 Epidemiology 135 (2011); Timothy L. Lash & Jan P. Vandenbroucke, “Should Preregistration of Epidemiologic Study Protocols Become Compulsory? Reflections and a Counterproposal,” 23 Epidemiology 184 (2012).

Posted in Expert Witnesses, Rule 702, Rule 703, Scientific Evidence, Scientific Publishing | 37 Comments »

Nullius in verba

March 29th, 2024

The 1975 codification of the law of evidence, in the Federal Rules of Evidence, introduced a subtle, aspirational criterion for expert witness opinion – knowledge. As originally enacted, Rule 702 read:

“If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise.”[1]

In case anyone missed the point, the Advisory Committee Note for the original Rule 702 emphasized that the standard was an epistemic standard:

“An intelligent evaluation of facts is often difficult or impossible without the application of some scientific, technical, or other specialized knowledge. The most common source of this knowledge is the expert witness, although there are other techniques for supplying it.”[2]

Perhaps we should not be too surprised that the epistemic standard was missed by most judges, and even by most lawyers. For a very long time, the common law set out a minimal test for expert witness opinion testimony. The expert witness had to be qualified by training, experience, or education, and the opinion proffered had to be logically and legally relevant to the issues in the case.[3] The enactment of Rule 702, in 1975, barely made a dent in the regime of easy admissibility.

Before the Federal Rules of Evidence, there was, of course, the famous Frye case, which involved an appeal from the excluded expert witness opinion based upon William Marston’s polygraph machine. In 1923, the court in Frye affirmed the exclusion of the expert witness opinion, based upon the lack of general acceptance of the device’s reliability, with its famous twilight zone language:[4]

“Just when a scientific principle or discovery crosses the line between the experimental and demonstrable stages is difficult to define. Somewhere in this twilight zone the evidential force of the principle must be recognized, and while courts will go a long way in admitting expert testimony deduced from a well-recognized scientific principle or discovery, the thing from which the deduction is made must be sufficiently established to have gained general acceptance in the particular field in which it belongs.”

With the explosion of tort litigation fueled by strict products liability doctrine, lawyers pressed Frye’s requirement of general acceptance into service as a bulwark against unreliable scientific opinions. Many courts, however, limited Frye to novel devices, and in 1993, the Supreme Court, in Daubert,[5] rejected the legal claim that Rule 702 had incorporated the common law “general acceptance” test. Looking to the language of the rule itself, the Supreme Court discerned that the rule laid down an epistemic test, not a call for sociological surveys about the prevalence of beliefs.

Resistance to the spirit and text of Rule 702 has been widespread and deep seated. After Daubert, the Supreme Court decided three more cases to emphasize that the epistemic standard was “exacting” and that it would not go away.[6] Since Daubert was decided in 1993, Rule 702 was amended substantively, in 2000, to incorporate some of the essence of the Supreme Court’s quartet,[7] which required the proponent of expert witness opinion to establish that proffered testimony is based upon sufficient facts or data, is the product of reliable principles and methods, and the result of reliably applying those reliable principles and methods to the facts of the case.

The change in the law of expert witnesses, in the 1990s, left some academic commentators well-nigh apoplectic. One professor of evidence law at a large law school complained that the law was a “conceptual muddle containing within it a threat to liberty and popular participation in government.”[8] Many federal district and intermediate appellate courts responded by ignoring the language of Rule 702, by reverting to pre-Daubert precedent, or by inventing new standards and shifting the burden to the party challenging the expert witness opinion’s admissibility. For many commentators, lawyers, and judges, science had no validity concerns that the law was bound to respect.

The judicial evasion and avoidance of the requirements of Rule 702 did not go unnoticed. Professor David Bernstein and practicing lawyer Eric Lasker wrote a paper in 2015, to call attention to the judicial disregard of the requirements of Rule 702.[9]  Several years of discussion and debate ensued before the Judicial Conference Advisory Committee on Evidence Rules (AdCom), in 2021, acknowledged that “in a fair number of cases, the courts have found expert testimony admissible even though the proponent has not satisfied the Rule 702(b) and (d) requirements by a preponderance of the evidence.”[10] This frank acknowledgement led the AdCom to propose amending Rule 702, “to clarify and emphasize” that gatekeeping requires determining whether the proponent has demonstrated to the court “that it is more likely than not that the proffered testimony meets the admissibility requirements set forth in the rule.”[11]  The Proposed Committee Note written in support of amending Rule 702 observed that “many courts have held that the critical questions of the sufficiency of an expert’s basis, and the application of the expert’s methodology, are questions of weight and not admissibility. These rulings are an incorrect application of Rules 702 and 104(a).”[12]

The proposed new Rule 702 is now law,[13] with its remedial clarification that the proponent of expert witness opinion must show the court that the opinion is sufficiently supported by facts or data,[14] that the opinion is “the product of reliable principles and methods,”[15]  and that the opinion “reflects a reliable application of the principles and methods to the facts of the case.”[16] The Rule prohibits deferring the evaluation of sufficiency of support or reliability of application of method to the trier of fact; there is no statutory support for suggesting that these inquires always or usually go to “weight and not admissibility,” or that there is a presumption of admissibility.

We may not have reached the Age of Aquarius, but the days of “easy admissibility” should be confined to the dustbin of legal history. Rule 702 is quickly approaching its 50th birthday, with the last 30 years witnessing the implementation of the promise and potential of an epistemic standard of trustworthiness for expert witness opinion testimony. Rule 702, in its present form, should go a long way towards putting validity questions squarely before the court under Rule 702. Nullius in verba[17] has been the motto of the Royal Society since 1660; it should now guide expert witness practice in federal court going forward.

[1] Pub. L. 93–595, §1, Jan. 2, 1975, 88 Stat. 1937 (emphasis added).

[2] Notes of Advisory Committee on Proposed Rules (1975) (emphasis added).

[3] See Charles T. McCormick, Handbook of the Law of Evidence 28-29, 363 (1954) (“Any relevant conclusions which are supported by a qualified expert witness should be received unless there are other reasons for exclusion.”)

[4] Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923).

[5] Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579 (1993).

[6] General Electric Co. v. Joiner, 522 U.S. 136 (1997); Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999); Weisgram v. Marley Co., 528 U.S. 440 (2000).

[7] See notes 5, 6, supra.

[8] John H. Mansfield, “An Embarrassing Episode in the History of the Law of Evidence,” 34 Seton Hall L. Rev. 77, 77 (2003); see also John H. Mansfield, “Scientific Evidence Under Daubert,” 28 St. Mary’s L.J. 1, 23 (1996). Professor Mansfield was the John H. Watson, Jr., Professor of Law, at the Harvard Law School. Many epithets were thrown in the heat of battle to establish meaningful controls over expert witness testimony. See, e.g., Kenneth Chesebro, “Galileo’s Retort: Peter Huber’s Junk Scholarship,” 42 Am. Univ. L. Rev. 1637 (1993). Mr. Chesebro was counsel of record for plaintiffs-appellants in Daubert, well before he became a convicted racketeer in Georgia.

[9] David Bernstein & Eric Lasker, “Defending Daubert: It’s Time to Amend Federal Rules of Evidence 702,” 57 Wm. & Mary L Rev. 1 (2015).

[10] Report of AdCom (May 15, 2021), at https://www.uscourts.gov/rules-policies/archives/committee-reports/advisory-committee-evidence-rules-may-2021. See also AdCom, Minutes of Meeting at 4 (Nov. 13, 2020) (“[F]ederal cases . . . revealed a pervasive problem with courts discussing expert admissibility requirements as matters of weight.”)], at https://www.uscourts.gov/rules-policies/archives/meeting-minutes/advisory-committee-evidence-rules-november-2020.

[11] Proposed Committee Note, Summary of Proposed New and Amended Federal Rules of Procedure (Oct. 19, 2022), at https://www.uscourts.gov/sites/default/files/2022_scotus_package_0.pdf

[12] Id. (emphasis added).

[13] In April 2023, Chief Justice Roberts transmitted the proposed Rule 702, to Congress, under the Rules Enabling Act, and highlighted that the amendment “shall take effect on December 1, 2023, and shall govern in all proceedings thereafter commenced and, insofar as just and practicable all proceedings then pending.” S. Ct. Order, at 3 (Apr. 24, 2023), https://www.supremecourt.gov/orders/courtorders/frev23_5468.pdf; S.Ct. Transmittal Package (Apr. 24, 2023), < https://www.uscourts.gov/sites/default/files/2022_scotus_package_0.pdf>.

[14] Rule 702(b).

[15] Rule 702(c).

[16] Rule 702(d).

[17] Take no one’s word for it.

Posted in Rule 702, Scientific Evidence | 30 Comments »

Dipak Panigrahy – Expert Witness & Putative Plagiarist

March 27th, 2024

Citing an IARC monograph may be in itself questionable, given the IARC’s deviations from good systematic review practice. Taking the language of an IARC, monograph, and passing it off as your own, without citation or attribution, and leaving out the qualifications and limitations stated in the monograph, should be disqualifying for an expert witness.

And in one federal court, it is.

Last week, on March 18, Senior Judge Roy Bale Dalton, Jr., of Orlando, Florida, granted defendant Lockheed Martin’s Rule 702 motion to exclude the proffered testimony of Dr. Dipak Panigrahy.[1] Panigraphy had opined in his Rule 26 report that seven substances[2] present in the Orlando factory cause eight different types of cancer[3] in 22 of the plaintiffs. Lockheed’s motion asserted that Panigrahy copied IARC verbatim, except for its qualifications and limitations. Judge Dalton reportedly found Panigraphy’s conduct so “blatant that it represents deliberate lack of candor” and an “unreliable methodology.” Although Judge Dalton’s opinion is not yet posted on Westlaw or Google Scholar,[4] the report from Legal Newsline quoted the opinion extensively:

“Here, there is no question that Dr. Panigrahy extensively plagiarized his report… .”

“And his deposition made the plagiarism appear deliberate, as he repeatedly outright refused to acknowledge the long swaths of his report that quote other work verbatim without any quotation marks at all – instead stubbornly insisting that he cited over 1,100 references, as if that resolves the attribution issue (it does not).”

“Indeed, the plagiarism is so ubiquitous throughout the report that it is frankly overwhelming to try to make heads or tails of just what is Dr. Panigrahy’s own work – a task that neither he nor Plaintiffs’ counsel even attempts to tackle.”

There is a wide-range of questionable research practices and dubious inferences that lead to the exclusion of expert witnesses under Rule 702, but I would have thought that Panigraphy was the first witness to have been excluded for plagiarism. Judge Dalton did, however, cite cases involving plagiarism by expert witnesses.[5] Although plagiarism might be framed as a credibility issue, the extent of the plagiarism by Panigraphy represented such an egregious lack of candor that it may justify exclusion under Rule 702.

Judge Dalton’s gatekeeping analysis, however, did not stop with the finding of blatant plagiarism from the IARC monograph. Panigraphy’s report was further methodologically marred by reliance upon the IARC, and his confusion of the IARC hazard evaluation with the required determination of causation in the law of torts. Judge Dalton explained that

“the plagiarism here reflects even deeper methodological problems because the report lifts a great deal of its analysis from IARC in particular. As the Court discussed in the interim causation Order, research agencies like IARC are, understandably, focused on protecting public health and recommending protective standards, rather than evaluating causation from an expert standpoint in the litigation context. IARC determines qualitatively whether substances are carcinogenic to humans; its descriptors have “no quantitative significance” such as more likely than not. Troublingly, Dr. Panigrahy did not grasp this crucial distinction between IARC’s classifications and the general causation preponderance standard. Because so much of Dr. Panigrahy’s report is merely a wholesale adoption of IARC’s findings under the guise of his own expertise, and IARC’s findings in and of themselves are insufficient, he fails to reliably establish general causation.”[6]

Dr. Panigraphy was accepted into medical school at the age of 17. His accelerated education may have left him without a firm understanding of the ethical requirements of scholarship.

Earlier this month, Senior Judge Dalton excluded another expert witness’s opinion testimony, from Dr. Donald Mattison, on autism, multiple sclerosis, and Parkinson’s disease, but permitted opinions on the causation of various birth defects.[7] Judge Dalton’s decisions arise from a group of companion cases, brought by more than 60 claimants against Lockheed Martin for various health conditions alleged to have been caused by Lockheed’s supposed contamination of the air, soil, and groundwater, with chemicals from its weapons manufacturing plant.

The unreliability of Panigraphy’s report led to the entry of summary judgment against the 22 plaintiffs, whose cases turned on the Panigraphy report.

The putative plagiarist, Dr. Panigraphy, is an assistant professor of pathology, at Harvard Medical School, in the department of pathology, Beth Israel Deaconess Medical Center, in Boston.  “The Expert Institute” has a profile of Panigraphy, with a compilation of background information and litigation activities. His opinions were excluded in the federal multi-district litigation concerning Zantac/ranitidine.[8]  Very similar opinions were permitted over defense challenges, in a short, perfunctory order, even shorter on reasoning, in the valsartan multi-district litigation.[9]

[1] John O’Brien, “‘A mess’: Expert in Florida toxic tort plagiarizes cancer research of others, tries to submit it to court,” Legal News Line (Mar. 25, 2024).

[2] trichloroethylene, tetrachloroethylene, formaldehyde, arsenic, hexavalent chromium, trichloroethylene, and styrene.

[3] cancers of the kidney, breast, thyroid, pancreas, liver and bile duct, testicles, and anus, as well as Hodgkin’s lymphoma, non-Hodgkin’s lymphoma, and leukemia.

[4] Henderson v. Lockheed Martin Corp., case no. 6:21-cv-1363-RBD-DCI, document 399 (M.D. Fla. Mar. 18, 2024) (Dalton, S.J.).

[5] Henderson Order at 6, citing Moore v. BASF Corp., No. CIV.A. 11-1001, 2012 WL 6002831, at *7 (E.D. La. Nov. 30, 2012) (excluding expert testimony from Bhaskar Kura), aff’d, 547 F. App’x 513 (5th Cir. 2013); Spiral Direct, Inc. v. Basic Sports Apparel, Inc., No. 6:15-cv-641, 2017 WL 11457208, at *2 (M.D. Fla. Apr. 13, 2017); 293 F. Supp. 3d 1334, 1363 n. 20 (2017); Legier & Materne v. Great Plains Software, Inc., No. CIV.A. 03-0278, 2005 WL 2037346, at *4 (E.D. La. Aug. 3, 2005) (denying motion to exclude proffered testimony because expert witness plagiarized a paragraph in his report).

[6] Henderson Order at 8 -10 (internal citations omitted), citing McClain v. Metabolife Internat’l, Inc., 401 F.3d 1233, 1249 (11th Cir. 2005) (distinguishing agency assessment of risk from judicial assessment of causation); Williams v. Mosaic Fertilizer, LLC, 889 F.3d 1239, 1247 (11th Cir. 2018) (identifying “methodological perils” in relying extensively on regulatory agencies’ precautionary standards to determine causation); Allen v. Pennsylvania Eng’g Corp., 102 F.3d 194, 198 (5th Cir. 1996) (noting that IARC’s “threshold of proof is reasonably lower than that appropriate in tort law, which traditionally makes more particularized inquiries into cause and effect and requires a plaintiff to prove that it is more likely than not that another individual has caused him or her harm”); In re Roundup Prods. Liab. Litig., 390 F. Supp. 3d 1102, 1109 (N.D. Cal. 2018) (“IARC classification is insufficient to get the plaintiffs over the general causation hurdle.”), aff’d, 997 F.3d 941 (9th Cir. 2021).

[7] John O’Brien, “Autism plaintiffs rejected from Florida Lockheed Martin toxic tort,” Legal Newsline (Mar. 15, 2024).

[8][8] In re Zantac (ranitidine) Prods. Liab. Litig., MDL NO. 2924 20-MD-2924, 644 F. Supp. 3d 1075, 1100 (S.D. Fla. 2022).

[9] In re Valsartan, Losartan, and Irbesartan Prods. Liab. Litig., Case 1:19-md-02875-RBK-SAK, document 1958 (D.N.J. Mar. 4, 2022).

Posted in Causation, Expert Witness Hall of Shame, Expert Witnesses, Rule 702, Scientific Misconduct | 127 Comments »

A Π-Day Celebration of Irrational Numbers and Other Things – Philadelphia Glyphosate Litigation

March 14th, 2024

Science can often be more complicated and nuanced than we might like. Back in 1897, the Indiana legislature attempted to establish that π was equal to 3.2.[1] Sure, that was simpler and easier to use in calculations, but also wrong. The irreducible fact is that π is an irrational number, and Indiana’s attempt to change that fact was, well, irrational. And to celebrate irrationality, consider the lawsuit’s industry’s jihad against glyphosate, including its efforts to elevate a dodgy IARC evaluation, while suppressing evidence of glyphosate’s scientific exonerations

                                                 

After Bayer lost three consecutive glyphosate cases in Philadelphia last year, observers were scratching their heads over why the company had lost when the scientific evidence strongly supports the defense. The Philadelphia Court of Common Pleas, not to be confused with Common Fleas, can be a rough place for corporate defendants. The local newspapers, to the extent people still read newspapers, are insufferably slanted in their coverage of health claims.

The plaintiffs’ verdicts garnered a good deal of local media coverage in Philadelphia.[2] Defense verdicts generally receive no ink from sensationalist newspapers such as the Philadelphia Inquirer. Regardless, media accounts, both lay and legal, are generally inadequate to tell us what happened, or what went wrong in the court room. The defense losses could be attributable to partial judges or juries, or the difficulty in communicating subtle issues of scientific validity. Plaintiffs’ expert witnesses may seem more sure of themselves than defense experts, or plaintiffs’ counsel may connect better with juries primed by fear-mongering media. Without being in the courtroom, or at least studying trial transcripts, outside observers are challenged to explain fully jury verdicts that go against the scientific evidence. The one thing jury verdicts are not, however, are valid assessments of the strength of scientific evidence, inferences, and conclusions.

Although Philadelphia juries can be rough, they like to see a fight. (Remember Rocky.) It is not a place for genteel manners or delicate and subtle distinctions. Last week, Bayer broke its Philadelphia losing streak, with a win in Kline v. Monsanto Co.[3] Mr. Kline claimed that he developed Non-Hodgkin’s lymphoma (NHL) from his long-term use of Round-Up. The two-week trial, before Judge Ann Butchart, last week went to the jury, which deliberated two hours before returning a unanimous defense verdict. The jury found that the defendants, Monsanto and Nouryon Chemicals LLC, were not negligent, and that the plaintiff’s use of Roundup was not a factual cause of his lymphoma.[4]

Law360 reported that the Kline verdict was the first to follow a ruling on Valentine’s Day, February 14, 2024, which excluded any courtroom reference to the hazard evaluation of Glyphosate by the International Agency for Research on Cancer (IARC). The Law360 article indicated that the IARC found that glyphosate can cause cancer; except of course IARC has never reached such a conclusion.

The IARC working group evaluated the evidence for glyphosate and classified the substance as a category IIA carcinogen, which it labels as “probably” causing human cancer. This label sounds close to what might be useful in a courtroom, except that the IARC declares that “probably,” as used in is IIA classification does not mean what people generally, and lawyers and judges specifically, mean by the word probably.  For IARC, “probable” has no quantitative meaning.  In other words, for IARC, probable, a quantitative concept, which everyone understands to be measured on a scale from 0 to 1, or from 0% to 100%, is not quantitative. An IARC IIA classification could thus represent a posterior probability of 1% in favor of carcinogenicity (and 99% probable not a carcinogen). In other words, on whether glyphosate causes cancer in humans, IARC says maybe in its own made-up epistemic modality.

To find the idiosyncratic definition of “probable,” a diligent reader must go outside the monograph of interest to the so-called Preamble, a separate document, last revised in 2019. The first time the jury will hear of the IARC pronouncement will be in the plaintiff’s case, and if the defense wishes to inform the jury on the special, idiosyncratic meaning of IARC “probable,” they must do it on cross-examination of hostile plaintiffs’ witnesses, or wait longer until they present their own witnesses. Disclosing the IARC IIA classification hurts because the “probable” language lines up with what the trial judges will instruct the juries at the end of the case, when the jurors are told that they need not believe that the plaintiff has eliminated all doubt; they need only find that the plaintiff has shown that each element of his case is “probable,” or more likely than not, in order to prevail. Once the jury has heard “probable,” the defense will have a hard time putting the toothpaste back in the tube. Of course, this is why the lawsuit industry loves IARC evaluations, with its fallacies of semantical distortion.[5]

Although identifying the causes of a jury verdict is more difficult than even determining carcinogenicity, Rosemary Pinto, one of plaintiff Kline’s lawyers, suggested that the exclusion of the IARC evaluation sank her case:

“We’re very disappointed in the jury verdict, which we plan to appeal, based upon adverse rulings in advance of the trial that really kept core components of the evidence out of the case. These included the fact that the EPA safety evaluation of Roundup has been vacated, who IARC (the International Agency for Research on Cancer) is and the relevance of their finding that Roundup is a probable human carcinogen [sic], and also the allowance into evidence of findings by foreign regulatory agencies disguised as foreign scientists. All of those things collectively, we believe, tilted the trial in Monsanto’s favor, and it was inconsistent with the rulings in previous Roundup trials here in Philadelphia and across the country.”[6]

Pinto was involved in the case, and so she may have some insight into why the jury ruled as it did. Still, issuing this pronouncement before interviewing the jurors seems little more than wishcasting. As philosopher Harry Frankfurt explained, “the production of bullshit is stimulated whenever a person’s obligations or opportunities to speak about some topic exceed his knowledge of the facts that are relevant to that topic.”[7] Pinto’s real aim was revealed in her statement that the IARC review was “crucial evidence that juries should be hearing.”[8]  

What is the genesis of Pinto’s complaint about the exclusion of IARC’s conclusions? The Valentine’s Day Order, issued by Judge Joshua H. Roberts, who heads up the Philadelphia County mass tort court, provided that:

AND NOW, this 14th day of February, 2024, upon consideration of Defendants’ Motion to Clarify the Court’s January 4, 2024 Order on Plaintiffs Motion in Limine No. 5 to Exclude Foreign Regulatory Registrations and/or Approvals of Glyphosate, GBHs, and/or Roundup, Plaintiffs’ Response, and after oral argument, it is ORDERED as follows:

  1. The Court’s Order of January 4, 2024, is AMENDED to read as follows: [ … ] it is ORDERED that the Motion is GRANTED without prejudice to a party’s introduction of foreign scientific evidence, provided that the evidence is introduced through an expert witness who has been qualified pursuant to Pa. R. E. 702.

  2. The Court specifically amends its Order of January 4, 2024, to exclude reference to IARC, and any other foreign agency and/or foreign regulatory agency.

  3. The Court reiterates that no party may introduce any testimony or evidence regarding a foreign agency and/or foreign regulatory agency which may result in a mini-trial regarding the protocols, rules, and/or decision making process of the foreign agency and/or foreign regulatory agency. [fn1]

  4. The trial judge shall retain full discretion to make appropriate evidentiary rulings on the issues covered by this Order based on the testimony and evidence elicited at trial, including but not limited to whether a party or witness has “opened the door.”[9]

Now what was not covered in the legal media accounts was the curious irony that the exclusion of the IARC evaluation resulted from plaintiffs’ motion, an own goal of sorts. In previous Philadelphia trials, plaintiffs’ counsel vociferously objected to the defense counsel’s and experts’ references to the determinations by foreign regulators, such as European Union Assessment Group on Glyphosate (2017, 2022), Health Canada (2017), European Food Safety Authority (2017, 2023), Australian Pesticides and Veterinary Medicines Authority (2017), German Federal Institute for Risk Assessment (2019), and others, that rejected the IARC evaluation and reported that glyphosate has not been shown to be carcinogenic.[10]

The gravamen of the plaintiffs’ objection was that such regulatory determinations were hearsay, and that they resulted from various procedures, using various criteria, which would require explanation, and would be subject to litigants’ challenges.[11] In other words, for each regulatory agency’s determination, there would be a “mini-trial,” or a “trial within a trial,” about the validity and accuracy of the foreign agency’s assessment.

In the earlier Philadelphia trials, the plaintiffs’ objections were largely sustained, which created a significant evidentiary bias in the courtrooms. Plaintiffs’ expert witnesses could freely discuss the IARC glyphosate evaluation, but the defense and its experts could not discuss the many determinations of the safety of glyphosate. Jurors were apparently left with the erroneous impression that the IARC evaluation was a consensus view of the entire world’s scientific community.

Now plaintiffs’ objection has a point, even though it seems to prove too much and must ultimately fail. In a trial, each side has expert witnesses who can offer an opinion about the key causal issue, whether glyphosate can cause NHL, and whether it caused this plaintiff’s NHL. Each expert witness will have written a report that identifies the facts and data relied upon, and that explains the inferences drawn and conclusions reached. The adversary can challenge the validity of the data, inferences, and conclusions because the opposing expert witness will be subject to cross-examination.

The facts and data relied upon will, however, be “hearsay,” which will come from published studies not written by the expert witnesses at trial. There will be many aspects of the relied upon studies that will be taken on faith without the testimony of the study participants, their healthcare providers, or the scientists who collected the data, chose how to analyze the data, conducted the statistical and scientific analyses, and wrote up the methods and study findings. Permitting reliance upon any study thus allows for a “mini-trial” or a “trial within a trial,” on each study cited and relied upon by the testifying expert witnesses. This complexity involved in expert witness opinion testimony is one of the foundational reasons for Rule 702’s gatekeeping regime in federal court, and most state courts, but which is usually conspicuously absent in Pennsylvania courtrooms.

Furthermore, the plaintiffs’ objections to foreign regulatory determinations would apply to any review paper, and more important, it would apply to the IARC glyphosate monograph itself. After all, if expert witnesses are supposed to have reviewed the underlying studies themselves, and be competent to do so, and to have arrived at an opinion in some reliable way from the facts and data available, then they would have no need to advert to the IARC’s review on the general causation issue.  If an expert witness were allowed to invoke the IARC conclusion, presumably to bolster his or her own causation opinion, then the jury would need to resolve questions about:

  • who was on the working group;
  • how were working group members selected, or excluded;
  • how the working group arrived at its conclusion;
  • what did the working group rely upon, or not rely upon, and why,
  • what was the group’s method for synthesizing facts and data to reach its conclusion;
  • was the working group faithful to its stated methodology;
  • did the working group commit any errors of statistical or scientific judgment along the way;
  • what potential biases did the working group members have;
  • what is the basis for the IARC’s classificatory scheme; and
  • how are IARC’s key terms such as “sufficient,” “limited,” “probable,” “possible,” etc., defined and used by working groups.

Indeed, a very substantial trial could be had on the bona fides and methods of the IARC, and the glyphosate IARC working group in particular.

The curious irony behind the Valentine’s Day order is that plaintiffs’ counsel were generally winning their objections to the defense’s references to foreign regulatory determinations. But as pigs get fatter, hogs get slaughtered. Last year, plaintiffs’ counsel moved to “exclude foreign regulatory registrations and or approvals of glyphosate.”[12] To be sure, plaintiffs’ counsel were not seeking merely the exclusion of glyphosate registrations, but the scientific evaluations of regulatory agencies and their staff scientists and consulting scientists. Plaintiffs wanted trials in which juries would hear only about IARC, as though it was a scientific consensus. The many scientific regulatory considerations and rejections of the IARC evaluation would be purged from the courtroom.

On January 4, 2024, plaintiffs’ counsel obtained what they sought, an order that memorialized the tilted playing field they had largely been enjoying in Philadelphia courtrooms. Judge Roberts’ order was short and somewhat ambiguous:

“upon consideration of plaintiff’s motion in limine no. 5 to exclude foreign regulatory registrations and/or approvals of glyphosate, GBHs, and/or Roundup, any response thereto, the supplements of the parties, and oral argument, it is ORDERED that the motion is GRANTED without prejudice to a party’s introduction of foreign scientific evidence including, but not limited to, evidence from the International Agency for Research on Cancer (IARC), provided that such introduction does not refer to foreign regulatory agencies.”

The courtroom “real world” outcome after Judge Roberts’ order was an obscene verdict in the McKivison case. Again, there may have been many contributing causes to the McKivison verdict, including Pennsylvania’s murky and retrograde law of expert witness opinion testimony.[13] Mr. McKivison was in remission from NHL and had sustained no economic damages, and yet, on January 26, 2024, a jury in his case returned a punitive compensatory damages award of $250 million, and an even more punitive punitive damage award of $2 billion.[14] It seems at least plausible that the imbalance between admitting the IARC evaluation while excluding foreign regulatory assessments helped create a false narrative that scientists and regulators everywhere had determined glyphosate to be unsafe.

On February 2, 2024, the defense moved for a clarification of Judge Roberts’ January 4, 2024 order that applied globally in the Philadelphia glyphosate litigation. The defendants complained that in their previous trial, after Judge Roberts’ Order of January 4, 2024, they were severely prejudiced by being prohibited from referring to the conclusions and assessments of foreign scientists who worked for regulatory agencies. The complaint seems well founded.  If a hearsay evaluation of glyphosate by an IARC working group is relevant and admissible, the conclusions of foreign scientists about glyphosate are relevant and admissible, whether or not they are employed by foreign regulatory agencies. Indeed, plaintiffs’ counsel routinely complained about Monsanto/Bayer’s “influence” over the United States Environmental Protection Agency, but the suggestion that the European Union’s regulators are in the pockets of Bayer is pretty farfetched. Indeed, the complaint about bias is peculiar coming from plaintifs’ counsel, who command an out-sized influence within the Collegium Ramazzini,[15] which in turn often dominates IARC working groups. Every agency and scientific group, including the IARC, has its “method,” its classificatory schemes, its definitions, and the like. By privileging the IARC conclusion, while excluding all the other many agencies and groups, and allowing plaintiffs’ counsel to argue that there is no real-world debate over glyphosate, Philadelphia courts play a malignant role in helping to generate the huge verdicts seen in glyphosate litigation.

The defense motion for clarification also stressed that the issue whether glyphosate causes NHL or other human cancer is not the probandum for which foreign agency and scientific group statements are relevant.  Pennsylvania has a most peculiar, idiosyncratic law of strict liability, under which such statements may not be relevant to liability questions. Plaintiffs’ counsel, in glyphosate and most tort litigations, however, routinely assert negligence as well as punitive damages claims. Allowing plaintiffs’ counsel to create a false and fraudulent narrative that Monsanto has flouted the consensus of the entire scientific and regulatory community in failing to label Roundup with cancer warnings is a travesty of the rule of law.

What seems clever by halves in the plaintiffs’ litigation approach was that its complaints about foreign regulatory assessments applied equally, if not more so, to the IARC glyphosate hazard evaluation. The glyphosate litigation is not likely as interminable as π, but it is irrational.

*      *     *      *      *     * 

Post Script.  Ten days after the verdict in Kline, and one day after the above post, the Philadelphia Inquirer released a story about the defense verdict. See Nick Vadala, “Monsanto wins first Roundup court case in recent string of Philadelphia lawsuits,” Phila. Inq. (Mar. 15, 2024).

[1] Bill 246, Indiana House of Representatives (1897); Petr Beckmann, A History of π at 174 (1971).

[2] See Robert Moran, “Philadelphia jury awards $175 million after deciding 83-year-old man got cancer from Roundup weed killer,” Phila. Inq. (Oct. 27, 2023); Nick Vadala, “Philadelphia jury awards $2.25 billion to man who claimed Roundup weed killer gave him cancer,” Phila. Inq. (Jan. 29, 2024).

[3] Phila. Ct. C.P. 2022-01641.

[4] George Woolston, “Monsanto Nabs 1st Win In Philly’s Roundup Trial Blitz,” Law360 (Mar. 5, 2024); Nicholas Malfitano, “After three initial losses, Roundup manufacturers get their first win in Philly courtroom,” Pennsylvania Record (Mar. 6, 2024).

[5][5] See David Hackett Fischer, “ Fallacies of Semantical Distortion,” chap. 10, in Historians’ Fallacies: Toward a Logic of Historical Thought (1970); see also “IARC’s Fundamental Distinction Between Hazard and Risk – Lost in the Flood” (Feb. 1, 2024); “The IARC-hy of Evidence – Incoherent & Inconsistent Classification of Carcinogencity” (Sept. 19, 2023).

[6] Malfitano, note 2 (quoting Pinto); see also Law360, note 2 (quoting Pinto).

[7] Harry Frankfurt, On Bullshit at 63 (2005); see “The Philosophy of Bad Expert Witness Opinion Testimony” (Oct. 2, 2010).

[8] See Malifanto, note 2 (quoting Pinto).

[9] In re Roundup Prods. Litig., Phila. Cty. Ct. C.P., May Term 2022-0550, Control No. 24020394 (Feb. 14, 2024) (Roberts, J.). In a footnote, the court explained that “an expert may testify that foreign scientists have concluded that Roundup and· glyphosate can be used safely and they do not cause cancer. In the example provided, there is no specific reference to an agency or regulatory body, and the jury is free to make a credibility determination based on the totality of the expert’s testimony. It is, however, impossible for this Court, in a pre-trial posture, to anticipate every iteration of a question asked or answer provided; it remains within the discretion of the trial judge to determine whether a question or answer is appropriate based on the context and the trial circumstances.”

[10] See National Ass’n of Wheat Growers v. Bonta, 85 F.4th 1263, 1270 (9th Cir. 2023) (“A significant number of . . . organizations disagree with IARC’s conclusion that glyphosate is a probable carcinogen”; … “[g]lobal studies from the European Union, Canada, Australia, New Zealand, Japan, and South Korea have all concluded that glyphosate is unlikely to be carcinogenic to humans.”).

[11] See, e.g., In re Seroquel, 601 F. Supp. 2d 1313, 1318 (M.D. Fla. 2009) (noting that references to foreign regulatory actions or decisions “without providing context concerning the regulatory schemes and decision-making processes involved would strip the jury of any framework within which to evaluate the meaning of that evidence”)

[12] McKivison v. Monsanto Co., Phila. Cty. Ct. C.P., No. 2022- 00337, Plaintiff’s Motion in Limine No. 5 to Exclude Foreign Regulatory Registration and/or Approvals of Glyphosate, GHBs and/or Roundup.

[13] See Sherman Joyce, “New Rule 702 Helps Judges Keep Bad Science Out Of Court,” Law360 (Feb. 13, 2024) (noting Pennsylvania’s outlier status on evidence law that enables dodgy opinion testimony).

[14] P.J. D’Annunzio, “Monsanto Fights $2.25B Verdict After Philly Roundup Trial,” Law360 (Feb. 8, 2024).

[15] “Collegium Ramazzini & Its Fellows – The Lobby” (Nov. 19, 2023).

Posted in Causation, Rule 702, Scientific Evidence | 39 Comments »

Purging Compurgation

March 12th, 2024

“You could file briefs on a napkin right now and get it granted.”

Alan Lange & Tom Dawson, Kings of Torts 87 (2d ed. 2010) (quoting convicted former lawyer, Zach Scruggs)

Back in the 1980s, I started to see expert witnesses stray into the business of psychoanalysis of corporate defendants. Perhaps it took place earlier; it seemed to be a tactic when I first started to try cases. Not only did expert witnesses wish to indict products as causes of plaintiffs’ harms, they wanted to indict the motives and intentions of the manufacturers. Such “motive” testimony should have been cleared from courtrooms by the basic rule of expert witness opinion testimony; namely, the warrant for expert witness testimony is that the subject matter is “beyond the ken” of the jury. Given that the tendentious witnesses had no special skills in divining motives, and that jurors were routinely called upon to infer motives, the offending testimony should have been readily quashed. Almost 100 years ago, Judge Learned Hand, confronted with similar argumentative opinion testimony, held, in his magisterial way, that “[a]rgument is argument whether in the box or at the bar, and its proper place is the last.”[1]

What I found when I started trying cases was that many states had hard rules on expert witnesses, but soft judges. In some litigations, plaintiffs’ counsel offered a witness, such as the late Marc Lappé, not only to assess motives, but also to make ethical pronouncements about defendants’ conduct. More typically, the ethical judgments came from historian witnesses or regulatory expert witnesses. Occasionally, expert witnesses on health effects issues offered psychoanalytic opinions as well. Plaintiffs’ counsel typically argued that Federal Rule of Evidence 704, which declared that “[a]n opinion is not objectionable just because it embraces an ultimate issue,” green lighted their witnesses’ amateur or professional psychoanalysis. Defendants typically argued that the common law requirement that opinions be “beyond the ken” of jurors was carried forward in Rule 702’s requirement of relevant expertise, knowledge, and helpfulness to the trier of fact.  State court analogues to these rules replicated the debate in state courts around the country.

The attempt to deprecate the intentions or motives of a party were not necessarily enhanced when the expert witness compurgator had some semblance of subject-matter expertise. In one case, a frequent statistician testifier for the lawsuit industry, Martin Wells, expressed the opinion that the study at issue in the litigation “was seriously flawed by bad epidemiological practice. The combination of bias and poor epidemiologic practice is so rampant that one can easily conclude the study was intentionally designed to achieve a desired result regardless of the actual findings in the data.”[2]

Wells may have been entitled to his opinion about the quality of the study at issue, and if he had good grounds and a reliable methodology, perhaps he should have been permitted to share that opinion with a jury. The court, however, held that opinions based upon “inferences about the intent or motive of parties … lie outside the bounds of expert testimony, but are instead classic jury questions.”[3] The acceptability of the Wells’ compurgation was not improved or made more admissible by coating it with a patina of expertise about interpreting studies. The trial court found that:

“Dr. Wells’ statements represent his subjective beliefs regarding an alleged bad motive or intent on the part of defendants or others who designed the study. The Court finds that his speculation about the reason for alleged methodological issues in the study are not the product of reliable methods, and will be excluded.”[4]

By 2011, or so, the case law interpreting common law and statutory rules about ethics and motive opinion generally tilted in favor of the defense.[5] Courts routinely excluded expert witness opinions about corporate knowledge, motivations, and intent, as irrelevant and inadmissible under Rule 702.

As though ethicist and historian testimony were not bad enough, imagine an economist offering testimony to deprecate lobbying efforts that are protected first amendment speech. In one multi-district litigation, thinking that they could get away with most anything, plaintiffs’ counsel offered just such an expert witness.

The expert witness at issue was an economist, Glen W. Harrison, of no particular distinction, who sought to serve as a compurgator in litigation. Harrison is an accomplished litigation witness, who was developed and trained by the Motley Rice firm and others in many tobacco cases.[6] What is clear is that he was deployed, in MDL 1535, to lobby the fact-finder inappropriately, without any real expertise in the material science, toxicology, or epidemiology issues in the litigation.

The essence of Glen Harrison’s opinion is that the “manufacturing industry” saw itself as having an “economic incentive” to engage in lobbying. This opinion was either tautologically or trivially true, but plaintiffs sought the opportunity to cast lawful (and constitutionally protected) lobbying as nefarious and tortious. A disinterested observer might have thought that the important issue was whether the lobbying was unlawful and thus inappropriate, but Harrison was not an expert on the law governing stakeholders’ submissions to agencies or to organizations that promulgate standards.

Harrison’s opinion on “inappropriateness” was based upon his inexpert factual review of documents, with occasional inferences or comments about whether the documents were incomplete, or inconsistent with other pieces of evidence. What was remarkable about this bold attempt to subvert the MDL trial process was that Harrison had absolutely no expertise or competence to discuss documents that involved issues of epidemiology, risk assessment, neurology, neuropsychology, toxicology, or exposure measurements. Harrison tried to squeeze out some bare relevance by commenting upon documents with his personal, lay observations that they seemed inaccurate, or that they were incomplete. Of course, a lawyer could equally well argue the point to the jury in summation. Clearly, the goal of proffering Harrison was to have a summation from a witness, with a pleasant Australian accent, in the middle of the plaintiffs’ case in chief. If you listened closely, you could hear a roar of disapproval from the Albany Rural Cemetery.[7]

For some time, the MDL 1535 judge winked at the plaintiffs’ and Harrison’s improper ploy to demonize lawful, appropriate industry conduct, and the MDL resolved before the parties obtained a ruling on Harrison’s proffered testimony. While the issue was before the MDL court, it appeared unmoved by considerations of the First Amendment or of the Noerr-Pennington doctrine,[8] or even the statutory invitation and right to comment upon proposed regulations.[9] Of course, manufacturing and lawsuit industries have a right to participate in notice and comment periods of rulemaking. The courtroom asymmetry threatened by Harrison’s proffered testimony was that plaintiffs’ counsel could comment upon defendants’ lobbying, but defense counsel had no equivalent opportunity to comment upon the lawsuit’s extensive rent seeking.[10]

[1] Nichols v. Universal Pictures Corp., 45 F.2d 119, 123 (2d Cir. 1930) (Hand, J.).

[2] In re Trasylol Prods. Liab. Litig., 08-md-01928, 2010 WL 1489793, at *2 (S.D. Fla. Feb. 24, 2010) (quoting from Rule 26 report of Martin T. Wells ¶ 4, Van Steenburgh Affidavit, Exhibit B, Docket No. 1677).

[3] Id. at *8 (internal quotation marks omitted).

[4] Id. at *2.

[5] See Beck, “Experts Offering Evidence of Corporate Intent, Ethics, And The Like,” Drug & Device Law (May 19, 2011) (collecting cases). See, e.g., Kidder, Peabody & Co., Inc. v. IAG Int’l Acceptance Grp., N.V., 14 F.Supp. 2d 391, 404 (S.D.N.Y.1998); Crown Cork, 2013 WL 978980, at *7 (excluding expert opinions of parties’ knowledge, state of mind, and intent); DePaepe v. General Motors Corp., 141 F.3d 714, 720 (7th Cir. 1998) (disallowing opinion of expert witness, who “lacked any scientific basis for an opinion about … motives,” about defendant’s failure to add safety measure in order to “save money”); In re Diet Drugs Prods. Liab. Litig., 2000 WL 876900, at *9 (E.D.Pa. June 20, 2000) (noting that “question of intent is a classic jury question and not one for experts”); Smith v. Wyeth-Ayerst Laboratories Co., 278 F.Supp.2d 684, 700 (W.D.N.C. 2003) (expert witnesses may not opine about corporate intent and motive) (barring Dr. Moye from giving such testimony); In re Rezulin Products Liab. Litig., 309 F. Supp. 2d 531, 543, 545 n.37 (S.D.N.Y.2004) (excluding opinions on intent and motive, as well as historical narrative gleaned form otherwise admissible documentary evidence); In re Baycol Prods. Liab. Litig., 495 F. Supp. 2d 977, 1001 (D.Minn. 2007) (holding expert witness to have exceeded proper proffer “to the extent that he speculates as to Bayer’s motive, intent, or state of mind”); 532 F. Supp. 2d 1029, 1069 (D. Minn. 2007) (“[A]n expert may not testify as to ethical issues or to his personal views”; “[t]he question of corporate intent is one for the jury, not for an expert”); Reece v. Astrazeneca Pharms., LP, 500 F. Supp. 2d 736, 744-46 (S.D. Ohio 2007) (advisability of tests; warnings needed for particular medical conditions; lack of methodology); In re Guidant Corp. Implantable Defibrillators Prods. Liab. Litig., 2007 WL 1964337, at *8 (D. Minn. June 29, 2007); Singh v. Edwards Lifesciences Corp., 2008 WL 5758387, ¶ELS 6 (Wash. Super. Snohomish Cty. Jan. 31, 2008); In re Fosamax Prods. Liab. Litig., 645 F. Supp. 2d 164, 192 (S.D.N.Y. 2009) (granting defendant’s motion to exclude testimony from Dr. Furberg about purported general ethical standards for conducting clinical trials); In re Xerox Corp. Sec. Litig., 746 F. Supp. 2d 402, 415 (D. Conn. 2010) (“Inferences about the intent or motive of parties or others lie outside the bounds of expert testimony.”) (internal citations omitted); In re Gadolinium-Based Contrast Agents Prods. Liab. Litig., 2010 WL 1796334, at *13 (N.D. Ohio May 4, 2010); In re Levaquin Prods. Liab. Litig., 2010 WL 11470977 (D. Minn. Nov. 10, 2010); Deutsch v. Novartis Pharms. Corp., 768 F. Supp.2d 420, 467 (E.D.N.Y. 2011); In re Heparin Prods. Liab. Litig., 2011 WL 1059660, at *8 (N.D. Ohio March 21, 2011); Lemons v. Novartis Pharms. Corp., 849 F. Supp.2d 608, 615 (W.D.N.C. 2012); Hill v. Novartis Pharms. Corp., 2012 WL 5451809, at *2 (E.D. Cal. Nov. 7, 2012); Georges v. Novartis Pharms. Corp., 2012 WL 9064768, at *13 (C.D. Cal. Nov. 2, 2012); Johnson v. Wyeth LLC, 2012 WL 1204081, at *3 (D. Ariz. Apr. 11, 2012); Pritchett v. I-Flow Corp., 2012 WL 1059948, at *6 (D. Colo. Mar. 28, 2012); Chandler v. Greenstone Ltd., 2012 WL 882756, at *1 (W.D. Wash. Mar. 14, 2012); Winter v. Novartis Pharms. Corp., 2012 WL 827305, at *5 (W.D. Mo. March 8, 2012); Earp v. Novartis Pharms. Corp., 2013 WL 4854488, at *4 (E.D.N.C. Sept. 11, 2013).

[6] See, e.g., Group Health Plan, Inc. v. Philip Morris, Inc., 188 F. Supp. 2d 1122 (D. Minn. 2002); Blue Cross & Blue Shield of N.J. v. Philip Morris, 178 F. Supp. 2d 198 (E.D.N.Y. 2001); Rent-A-Center West Inc.  v. Dept. of Revenue, 418 S.C. 320, 792 S.E.2d 260 (2016).

[7] Where Judge Learned Hand was buried.

[8] See, e.g., Video Int’l Prod., Inc. v. Warner-Amex Cable Comm., Inc., 858 F.2d 1075, 1084 (5th Cir. 1988) (applying Noerr-Pennington doctrine to bar use of evidence of lobbying in tort case); Hamilton v. AccuTek, 935 F. Supp. 1307, 1321 (E.D.N.Y. 1996) (granting summary judgment to gun makers on product liability and fraud claims based upon their efforts to influence federal policies by lawful lobbying); In re Municipal Stormwater Pond, No. 18-cv-3495 (JNE/KMM), 2019 U.S. Dist. LEXIS 227887, at *12 (D. Minn. Dec. 20, 2019) (dismissing claims of fraudulent misrepresentation claims against maker of coal-tar sealant on grounds that the Noerr-Pennington doctrine protected manufacturer’s lobbying before state and local governments); Eiser v. Brown & Williamson Tobacco Corp., Phila. Ct. Com. Pleas LEXIS 43, *20, 2005 WL 1323030 (2005) (invoking Noerr-Pennington doctrine to bar evidence of defendant manufacturer’s lobbying in products liability case). See generally James M. Sabovich, “Petition without Exception: Against the Fraud Exception to Noerr-Pennington Immunity from the Toxic Tort Perspective,” 17 Penn State Envt’l L. Rev. 101 (2008).

[9] See Admin. Procedures Act, 5 U.S.C. § 553; Attorney General’s Manual on the Administrative Procedure Act 31 (1947) (“[t]he objective should be to assure informed administrative action and adequate protection to private interests”).

[10] Lawsuit industry certainly exercises its rent-seeking through legitimate lobbying, and occasionally through illegimate means.  See U.S. v. Scruggs, 691 F.3d 660 (5th Cir. 2012).

Posted in Expert Witnesses, Rule 702 | 4 Comments »

A Citation for Jurs & DeVito’s Unlawful U-Turn

February 27th, 2024

Antic proposals abound in the legal analysis of expert witness opinion evidence. In the courtroom, the standards for admitting or excluding such evidence are found in judicial decisions or in statutes. When legislatures have specified standards for admitting expert witness opinions, courts have a duty to apply the standards to the facts before them. Law professors are, of course, untethered from either precedent or statute, and so we may expect chaos to ensue when they wade into disputes about the proper scope of expert witness gatekeeping.

Andrew Jurs teaches about science and the law at the Drake University Law School, and Scott DeVito is an associate professor of law at the Jacksonville University School of Law. Together, they have recently produced one of the most antic of antic proposals in a fatuous call for the wholesale revision of the law of expert witnesses.[1]

Jurs and DeVito rightly point out that since the Supreme Court, in Daubert,[2] waded into the dispute whether the historical Frye decision survived the enactment of the Federal Rules of Evidence, we have seen lower courts apply the legal standard inconsistently and sometimes incoherently. These authors, however, like many other academics, incorrectly label one or the other standard, Frye or Daubert, as being stricter than the other. Applying the labels of stricter and weaker standards, ignores that they are standards that measure completely different things. Frye advances a sociological standard, and a Frye test challenge can be answered by conducting a survey. Rule 702, as interpreted by Daubert, and as since revised and adopted by the Supreme Court and Congress, is an epistemic standard. Jurs and DeVito, like many other legal academic writers, apply a single adjective to standards that measure two different, incommensurate things. The authors’ repetition of the now 30-plus year-old mistake is a poor start for a law review article that sets out to reform the widespread inconsistency in the application of Rule 702, in federal and in state courts.

In seeking greater adherence to the actual rule, and consistency among decisions, Jurs and DeVito might have urged for judicial education, or blue-ribbon juries, or science courts, or greater use of court-appointed expert witnesses. Instead they have put their marker down on abandoning all meaningful gatekeeping. Jurs and DeVito are intent upon repairing the inconsistency and incoherency in the application of Daubert, by removing the standard altogether.

“To resolve the problem, we propose that the Courts replace the multiple Daubert factors with a single factor—testability—and that once the evidence meets this standard the judge should provide the jury with a proposed jury instruction to guide their analysis of the fact question addressed by the expert evidence.”[3]

In other words, because lower federal courts have routinely ignored the actual statutory language of Rule 702, and Supreme Court precedents, Jurs and DeVito would have courts invent a new standard, that virtually excludes nothing as long as someone can imagine a test for the asserted opinion. Remarkably, although they carry on about the “rule of law,” the authors fail to mention that judges have no authority to ignore the requirements of Rule 702. And perhaps even more stunning is that they have advanced their nihilistic proposal in the face of the remedial changes in Rule 702, designed to address judicial lawlessness in ignoring previously enacted versions of Rule 702. This antic proposal would bootstrap previous judicial “flyblowing” of a Congressional mandate into a prescription for abandoning any meaningful standard. They have articulated the Cole Porter standard: anything goes. Any opinion that “can be tested is science”; end of discussion.  The rest is for the jury to decide as a question of fact, subject to the fact finder’s credibility determinations. This would be a Scott v. Sandford rule[4] for scientific validity; science has no claims of validity that the law is bound to respect.

Jurs and DeVito attempt a cynical trick. They argue that they would fix the problem of “an unpredictable standard” by reverting to what they say is Daubert’s first principle of ensuring the reliability of expert witness testimony, and limiting the evidentiary display at trial to “good science.” Cloaking their nihilism, the authors say that they want to promote “good science,” but advocate the admissibility of any and every opinion, as long as it is theoretically “testable.” In order to achieve this befuddled goal, they simply redefine scientific knowledge as “essentially” equal to testable propositions.[5]

Jurs and DeVito marshal evidence of judicial ignorance of key aspects of scientific method, such as error rate. We can all agree that judges frequently misunderstand key scientific concepts, but their misunderstandings and misapplications do not mean that the concepts are unimportant or unnecessary. Many judges seem unable to deliver an opinion that correctly defines p-value or confidence interval, but their inabilities do not allow us to dispense with the need to assess random error in statistical tests. Our faint-hearted authors never explain why the prevalence of judicial error must be a counsel of despair that drives us to bowdlerize scientific evidence into something it is not. We may simply need better training for judges, or better assistance for them in addressing complex claims. Ultimately, we need better judges.

For those judges who have taken their responsibility seriously, and who have engaged with the complexities of evaluating validity concerns raised in Rule 702 and 703 challenges, the Jurs and DeVito proposal must seem quite patronizing. The “Daubert” factors are simply too complex for you, so we will just give you crayons, or a single, meaningless factor that you cannot screw up.[6]

The authors set out a breezy, selective review of statements by a few scientists and philosophers of science. Rather than supporting the extreme reductionism, Jurs and DeVito’s review reveals that science is much more than identifying a “testable” proposition. Indeed, the article’s discussion of philosophy and practice of science weighs strongly against the authors’ addled proposal.[7]

The authors, for example, note that Sir Isaac Newton emphasized the importance of empirical method.[8] Contrary to the article’s radical reductionism, the authors note that Sir Karl Popper and Albert Einstein stressed that the failure to obtain a predicted experimental result may render a theory “untenable,” which of course requires data and valid tests and inferences to assess. Quite a bit of motivated reasoning has led Jurs and DeVito to confuse a criterion of testability with the whole enterprise of science, and to ignore the various criteria of validity for collecting data, testing hypotheses, and interpreting results.

The authors suggest that their proposal will limit the judicial inquiry to the the legal question of reliability, but this suggestion is mere farce. Reliability means obtaining the same or sufficiently similar results upon repeated testing, but these authors abjure testing itself.  Furthermore, reliability as contemplated by the Supreme Court, in 1993, and by FRE 702 ever since, has meant validity of the actual test that an expert witness argues in support of his or her opinion or claims.

Whimsically, and without evidence, Jurs and DeVito claim that their radical abandonment of gatekeeping will encourage scientists, in “fields that are testable, but not yet tested, to perform real, objective, and detailed research.” Their proposal, however, works to remove any such incentive because untested but testable research becomes freely admissible. Why would the lawsuit industry fund studies, which might not support their litigation claims, when the industry’s witnesses need only imagine a possible test to advance their claims, without the potential embarrassment by facts? The history of modern tort law teaches us that cheap speculation would quickly push out actual scientific studies.

The authors’ proposal would simply open the floodgates of speculation, conjecture, and untested hypothesis, and leave the rest to the vagaries of trials, mostly in front of jurors untrained in evaluating scientific and statistical evidence. Admittedly, some incurious and incompetent gatekeepers and triers of fact will be relieved to know that they will not have to evaluate actual scientific evidence, because it had been eliminated by the Jurs and DeVito proposal to make mere testability the touchstone of admissibility

To be sure, in Aristotelian terms, testability is logical and practically prior to testing, but these relationships do not justify holding out testability as the “essence” of science, and the alpha and omega of science.[9] Of course, one must have an hypothesis to engage in hypothesis testing, but science lies in the clever interrogation of nature, guided by the hypothesis. The scientific process lies in answering the question, not simply in formulating the question.

As for the authors’ professed concern about “rule of law,” readers should note that the Jurs and DeVito article completely ignores the remedial amendment to Rule 702, which went into effect on December 1, 2023, to address the myriad inconsistencies, and failures to engage, in required gatekeeping of expert witness opinion testimony.[10]

The new Rule 702 is now law, with its remedial clarification that the proponent of expert witness opinion must show the court that the opinion is sufficiently supported by facts or data, Rule 702(b), and that the opinion “reflects a reliable application of the principles and methods to the facts of the case,” Rule 702(d). The Rule prohibits deferring the evaluation of sufficiency of support or reliability of application of method to the trier of fact; there is no statutory support for suggesting that these inquires always or usually go to “weight and not admissibility.”

The Jurs and DeVito proposal would indeed be a U-Turn in the law of expert witness opinion testimony. Rather than promote the rule of law, they have issued an open, transparent call for licentiousness in the adjudication of scientific and technical issues.

[1] Andrew Jurs & Scott DeVito, “A Return to Rationality: Restoring the Rule of Law After Daubert’s Disasterous U-Turn,” 164 New Mexico L. Rev. 164 (2024) [cited below as U-Turn]

[2] Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993).

[3] U-Turn at 164, Abstract.

[4] 60 U.S. 393 (1857).

[5] U-Turn at 167.

[6] U-Turn at 192.

[7] See, e.g., U-Turn at 193 n.179, citing David C. Gooding, “Experiment,” in W.H. Newton-Smith, ed., A Companion to the Philosophy of Science 117 (2000) (emphasizing the role of actual experimentation, not the possibility of experimentation, in the development of science).

[8] U-Turn at 194.

[9] See U-Turn at 196.

[10] See Supreme Court Order, at 3 (Apr. 24, 2023); Supreme Court Transmittal Package (Apr. 24, 2023).

Posted in Expert Witnesses, Rule 702, Scientific Evidence | 247 Comments »

Grumpy Old Men

February 11th, 2024

This blog is not about politics, although sometimes I have wandered into the political thicket when the events of the day involved scientific and statistical issues.[1] Our current events today do not involve statistical evidence so much as political, moral, and practical judgment. At some point, however, I cannot avoid a sense of responsibility to speak on the issues. On the one side, we have a grumpy old man who has evidenced some senior moments. On the other side, we have a grumpy old man who has evidenced senior moments as well as a vile character and temperament, generally devoid of decency, historical knowledge, and practical judgment.

Some have argued that the first grumpy old man is propped up by Marxists, the ultra-woke, and “haters” of the USA. Others have argued that the second grumpy old man is a creature of a personality cult, and that he is a documented rapist, fraudster, liar, hustler, and worse.[2]

Well, I know I am not going to firm up the political quagmire on which we now stand. But I have a humble, and I hope constructive, suggestion. Supporters of both grumpy old men would, I think, agree that they care about strength, intelligence, tactical and strategic ability, mental acuity and stamina, and a desire to win.  My suggestion is instead of having debates between the main protagonists, which many would agree to be a waste of time and money, we should instead require the grumpy old men to face off in a chess match. It need not be long; it could be, say, best of three games. Their moves would be observable by all, including their ability to conform to the rules of game, which require, among other things, refraining from talking and making noise. The American public, and the world, would see whether one or the other, or both, or neither, of the grumpy old men have the cognitive and intellectual ability and a true “killer” instinct, to be a national leader in these troubled times. Professional psychologists could analyze the intelligence of the players in real time.

Perhaps the rules of the game might be modified to include shock collars to discipline either player who spoke during the game, other than to offer a draw, or to resign.

Perhaps I am grumpy old man myself.

[1] See “Pernicious Probabilities in the Supreme Court” (Nov. 21, 2020); “A Trumpence for Your Thoughts” (Nov. 21, 2020). And I have supported sound deployment of statistical evidence. See, e.g., Brief of 27 Election Law, Scientific Evidence, and Empirical Legal Scholars as Amici Curiae in Support of Appellees, in Rucho v. Common Cause, No. 18-422, Supreme Court of the United States (Mar. 9, 2019); Brief of 44 Election Law, Scientific Evidence, and Empirical Legal Scholars as Amici Curiae in Support of Appellees, in Gill v. Whitford, No. 16-1161, Supreme Court of the United States (Sept. 1, 2017).

[2] The supporters of the second grumpy old man have criticized those who criticize his supporters, on grounds that there are substantive issues at stake in their support.  See Julian Adorney, “Stop Dunking on Trump Supporters: No one is beyond reach—unless everyone around them refuses to reach out,” Quilette (Feb. 7, 2024). This of course misses the point that the supporters have had plenty of other viable candidates to support who would have actually been articulate voices on their behalf.  The supporters have shown, consistently over eight years, that they do not care about immorality or sycophancy to dictators, and that they prefer a sexual abuser and fraudster.

Posted in Uncategorized | 35 Comments »

IARC’S Fundamental Distinction Between Hazard and Risk – Lost in the Flood

February 1st, 2024

Socrates viewed philosophy as beginning in wonder,[1] but Socrates and his philosophic heirs recognized that philosophy does not get down to business until it starts to clarify the terms of discussion. By the middle of the last century, failure to understand the logic of language replaced wonder as the beginning of philosophy.[2] Even if philosophy could not cure all conceptual pathology, most writers came to see that clarifying terms, concepts, and usage was an essential starting point in thinking clearly about a subject.[3]

Hazard versus Risk

Precision in scientific exposition often follows from the use of measurements, using agreed upon quantitative units, and accepted, accurate, reliable procedures for measurements. When scientists substitute qualitative measures for what are inherently quantitative measures, they frequently lapse into error. For example, beware of rodent studies that proclaim harms at “low doses,” which turn out to low in comparison to other rodent studies, but orders of magnitude greater exposure than experienced by human beings.

Risk is a quantitative term meaning a rate of some specified outcome. A Dictionary of Epidemiology, for instance, defines risk as:

“The probability of an adverse or beneficial event in a defined population over a specified time interval. In epidemiology and in clinical research it is commonly measured through the cumulative incidence and the incidence proportion.”[4]

An increased risk thus requires a measurement of a rate or probability of an outcome greater than expected in the absence of the exposure of interest. We might be uncertain of the precise measure of the risk, or of an increased risk, but conceptually a risk connotes a rate or a probability that is, at least in theory, measurable.

Hazard is a categorical concept; something is, or is not, a hazard without regard to the rate or incidence of harm. The definition of “hazard,” in the Dictionary of Epidemiology provides a definition that captures the non-quantitative, categorical natural of some exposure’s being a hazard:

“The inherent capability of a natural or human-made agent or process to adversely affect human life, health, property, or activity, with the potential to cause a disease.”[5]

The International Agency for Research on Cancer (IARC) purports to set out a classification scheme, for human cancer hazards. As used by IARC, its classification scheme involves a set of epistemic modal terms: “known,” “probably,” “possibly,” and “indeterminate.” These epistemic modalities characterize the strength of the evidence that an agent is carcinogenic, and not the magnitude of quantitative risk of cancer from exposure at a given level. The IARC Preamble, which attempts to describe the Agency’s methodology, explains that the distinction between hazard and risk is “fundamental”:

“A cancer hazard is an agent that is capable of causing cancer, whereas a cancer risk is an estimate of the probability that cancer will occur given some level of exposure to a cancer hazard. The Monographs assess the strength of evidence that an agent is a cancer hazard. The distinction between hazard and risk is fundamental. The Monographs identify cancer hazards even when risks appear to be low in some exposure scenarios. This is because the exposure may be widespread at low levels, and because exposure levels in many populations are not known or documented.”[6]

This attempted explanation reveals an important problem in IARC’s project, as stated in the Preamble. First, there is an unproven assumption that there will be cancer hazards regardless of the exposure levels. The IARC contemplates that there may circumstances of low levels of risk from low levels of exposure, but it elides the important issue of thresholds of exposure, below which there is no risk. The Preamble suggests that IARC does not attempt to provide evidence for or against meaningful thresholds of hazardousness, but this failure greatly undermines the project.  Exposure circumstances may be such that there is no hazard at all, and so the risk is zero.

The purported distinction between hazard and risk, supposedly fundamental, is often blurred by the Agency, in the monographs produced by working groups on specific exposure circumstances. Consider for instance how a working group characterized the “hazard” of inhalation of respirable crystalline silica:

“ln making the overalI evaluation, the Working Group noted that carcinogenicity in humans was not detected in all industrial circumstances studied. Carcinogenicity may be dependent on inherent characteristics of the crystalline silica or on external factors affecting its biological activity or distribution of its polymorphs.

Crystalline silica inhaled in the form of quartz or cristobalite from occupational sources is carcinogenic to humans (Group 1)”[7]

So some IARC classifications actually do specify that exposure to a substance is not a hazard in all circumstances, a qualification that implies that the same exposure in some exposure circumstances is not a hazard, and so the risk is zero.

We know something about the deliberations of the crystalline silica working group. The members were deadlocked for some time, and the switch of one vote ultimately gave a bare majority to reclassifying crystalline silica as a Group I exposure. Here is how the working group member, Corbett McDonald described the situation:

“The IARC Working Group, in 1997, had considerable difficulty in reaching a decision and might well not have done so had it not been made clear that it was concerned with hazard identification, not risk.”[8]

It was indeed Professor McDonald who changed his vote based upon this linguistic distinction between hazard and risk. His own description of the dataset, however, suggests that the elderly McDonald was railroaded by more younger, more strident members of the group:

“Of the many studies reviewed by the Working Group … nine were identified as providing the least confounded evidence. Four studies which we considered positive included two of refractory brick workers, one in the diatomite industry and our own in pottery workers; the five which seemed negative or equivocal included studies of South Dakota gold miners, Danish stone workers, US stone workers and US granite workers. This further example that the truth is seldom pure and never simple underlines the difficulty of establishing a rational control policy for some carcinogenic materials.”[9]

In defense of his vote, McDonald meekly offered that

“[s]ome equally expert panel of scientists presented with the same information on another occasion could of course have reached a different verdict. The evidence was conflicting and difficult to assess and such judgments are essentially subjective. Of course, when the evidence is conflicting, it cannot be said to be sufficient. Not only was the epidemiologic evidence conflicting, but so was the whole animal toxicology, which found a risk of tumors in rats, but not in mice or hamsters.”

Aside from endorsing a Group I classification for crystalline silica, the working group ignored the purportedly fundamental distinction between hazard and risk, by noting that not all exposure circumstances posed a hazard of cancer. The same working group did even greater violence to the supposed distinction between risk and hazard in its evaluation of coal dust exposure and human cancer. Coal miners have been studied extensively for cancer risk, and the working group reviewed and evaluated the nature of their exposures and their cancer outcomes. Coal dust virtually always contains crystalline silica, often making up a sizable percentage of the total inhalational exposures (40% or so) of coal dust.[10] And yet, when the group studied the cancer rates among coal miners, and in animals, it concluded that there was “inadequate evidence in humans, and “in experimental animals,” for carcinogenicity. The same working group that agreed, on a divided vote, to place crystalline silica in Group 1, voted that “[c]oal dust cannot be classified as to its carcinogenicity to humans (Group 3).”[11]

The conceptual confusion between hazard and risk is compounded by the IARC’s use of epistemic modalities – known, probably, possibly, and indeterminate – to characterize the existence of a hazard. The Preamble, in Table 4, summarizes the categories and “the stream of evidence” needed to place any particular exposure in a one epistemic modal class or another. What is inexplicable is how and why a single substance such as crystalline silica goes from a known cancer hazard in some unspecified occupational setting but then its putative carcinogenicity becomes indeterminate when it makes up 40% of the inhaled dust in a coal mine.

 

The conceptual difficulty created by IARC’s fundamental distinction between hazard and risk is that risk might well vary across exposure circumstances, but there is no basis for varying the epistemic modality for the hazard assessment simply because coal dust is only say 40% crystalline silica. Some of the exposure circumstances evaluated for the Group I silica hazard classification actually were lower than the silica content of coal.  Granite quarrying, for example, involves exposure to rock dust that is roughly 30% crystalline silica.

The conceptual and epistemic confusion caused by IARC’s treatment of the same substance in different exposure circumstances is hardly unique to its treatment of crystalline silica and coal dust. Benzene has long been classified as a Group I human carcinogen, for its ability to cause a specific form of leukemia.[12] Gasoline contains, on average, about one percent benzene, and so gasoline exposure inevitably involves benzene exposure. And yet, benzene exposure in the form of inhaling gasoline fumes is only a “possible” human carcinogen, Group 2B.[13]

Similarly, in 2018, the IARC classified the evidence for the human carcinogenicity of coffee as “indeterminate,” Group 3.[14] And yet every drop of coffee inevitably contains acrylamide, which is, according to IARC, a Group 2A “probable human carcinogen.”[15] Rent-seeking groups, such as the Council for Education and Research on Toxics (founded by Carl Cranor and Martyn Smith) have tried shamelessly to weaponize the IARC 2A classification for acrylaminde by claiming a bounty against coffee sellers such as Star-Bucks in California Proposition 65 litigation.[16]

Similarly confusing, IARC designates acetaldehyde on its own a “possible” human carcinogen, group 2B, even though acetaldehyde is invariably associated with the metabolism of ethyl alcohol, which itself is a Group I human carcinogen.[17] There may well be other instances of such confusions, and I would welcome examples from readers.

These disparate conclusions strain credulity, and undermine confidence that the hazard-risk distinction does any work at all. Hazard and risk do have different meanings, and I would not want to be viewed as anti-semantic. IARC’s use of the hazard-risk distinction, however, lends itself to the interpretation that hazard is simply risk without the quantification. This usage actually is worse than having no distinction at all, because it ignores the existence of thresholds below which exposure carries no risk, as well as ignoring different routes of exposure and exposure circumstances that carry no risk at all. The vague and unquantified categorical determination that a substance is a hazard allows fear mongers to substitute subjective, emotive, and unscientific judgments for scientific assessment of carcinogenicity under realistic conditions of use and exposure.

[1] Plato, Theaetetus 155d (Fowler transl. 1921).

[2] Ludwig Wittgenstein, Philosophical Investigations (1953).

[3] See, e.g., Richard M. Rorty, ed., The Linguistic Turn: Essays in Philosophical Method (1992); Nicholas Rescher, Concept Audits: A Philosophical Method (2016); Timothy Williamson, Philosophical Method: A Very Short Introduction 32 (2020) (discussing the need to clarify terms).

[4] Miquel Porta, Sander Greenland, Miguel Hernán, Isabel dos Santos Silva, John M. Last, and Andrea Burón, A Dictionary of Epidemiology 250 (6th ed. 2014).

[5] Id. at 128.

[6] IARC Monographs on the Identification of Carcinogenic Hazards to Humans – Preamble (2019) (emphasis added).

[7] IARC Monograph on the Evaluation of Carcinogenic Risks to Humans: Volume 68, Silica, Some Silicates, Coal Dust, and para-Aramid Fibrils 210-211 (1997).

[8] Corbett McDonald & Nicola Cherry, “Crystalline Silica and Lung Cancer: The Problem of Conflicting Evidence,” 8 Indoor Built Environment 121, 121 (1999).

[9] Id.

[10] IARC Monograph on the Evaluation of Carcinogenic Risks to Humans: Volume 68, Silica, Some Silicates, Coal Dust, and para-Aramid Fibrils 340 (1997).

[11] Id. at 393.

[12] IARC Monograph, Volume 120: Benzene (2018).

[13] IARC Monographs on the Evaluation of Carcinogenic Risks to Humans: Volume 45, Occupational Exposures in Petroleum Refining; Crude Oil and Major Petroleum Fuels 194 (1989).

[14] IARC Monograph No. 116, Drinking Coffee, Mate, and Very Hot Beverages (2018).

[15] IARC Monograph no. 60, Some Industrial Chemicals (1994).

[16] See “Coffee with Cream, Sugar & a Dash of Acrylamide” (June 9, 2018); “The Council for Education and Research on Toxics” (July 9, 2013).

[17] IARC Monographs on the Evaluation of Carcinogenic Risks to Humans Volume 96 1278 (2010).

Posted in Causation, Scientific Evidence | 3 Comments »

« Older Entries
Newer Entries »

The opinions, statements, and asseverations expressed on Tortini are my own, or those of invited guests, and these writings do not necessarily represent the views of clients, friends, or family, even when supported by good and sufficient reason.