For those of us who litigate health effects claims, either as pursuers or defenders, the pathology of science is often as important and interesting as pristine methodology. Identifying the pathological epistemology (patho-epistemology) of our adversaries’ claims, in the facts and data relied upon, and the inferences drawn at every step in reasoning to a conclusion, are critical to getting to the truth, as well as prevailing in litigation.
To its credit, the Reference Manual on Scientific Evidence has addressed, since its first edition, some varieties of bias that threaten the validity of epidemiologic studies. The most recent edition has the most extensive discussion yet. The authors of the chapter on epidemiology provide a basic taxonomy of systematic biases into three categories: selection, information, and confounding bias, all of which can affect the internal validity of an epidemiologic study.[1] Importantly, the chapter authors advise that the inevitable limitations in studies “must be considered to interpret their results properly.”[2] The chapter authors seem, however, keen to give examples in which courts dismiss challenges to studies with actual biases and severe limitations, rather than exclude witnesses who rely upon seriously biased studies.
The chapter thus cites a district court’s pronouncement that “[w]here a positive association is observed, its validity is assessed by evaluating the role of possible alternative explanations, such as chance, bias, or confounding.”[3] The authors avoid acknowledging, however, that the quoted court found that the pursuers’ expert witnesses had failed to evaluate bias adequately. In a similar vein, the chapter authors downplay potent biases when the biases undermine the validity of studies relied upon by claimants. The chapter cites a notorious decision in the phenylpropanolamine litigation, and quotes from the MDL court’s decision that dismissed the defendant’s “ex post facto dissection” of a study, because all “scientific studies almost invariably contain flaws.”[4] The quoted language reveals that particular MDL court’s refusal to engage with the evidence of the seriousness of the specific flaws identified. All humans have flaws, but still we acknowledge some as saints and some as criminals. A lot more is required than shrugging off challenges because no study is perfect.
Another example of the epidemiology chapter’s apparent approval of toothless judicial review of biases in studies can be seen in citation to the RoundUp MDL. The presiding judge, Judge Chhabria, declared that “concerns about recall bias in these studies do not demand that a reliable expert opinion meaningfully discount the body of case-control studies when assessing causation.”[5] The basis for this curious judgment was the plaintiffs’ expert witnesses’ claim that concerns over recall bias were diminished when studies looked only at one particular outcome (Non-Hodgkins’ lymphoma (NHL)) as opposed to many different kinds of cancer. These claims were free of empirical support, and puzzling in that case-control studies typically involve only one outcome of interest.
In that same RoundUp litigation, an expert testified that because recall bias would be expected to affect reported exposures for people with any type of cancer, concerns about recall bias were diminished where “epidemiology studies on the whole observed associations only between” exposure to an herbicide and a particular type of cancer, rather than with “the other cancers about which participants were asked.” No empirical support was cited for this curious, counter-intuitive opinion. If the cancer for which the odds ratio was elevated was the subject of litigation and a good deal of sensational, misleading publicity involving RoundUp, then we might well expect cases – with NHL – to remember or even exaggerate exposure to RoundUp more than the controls who did not have NHL.
Other instances in the Reference Manual’s treatment of bias are equally skewed. The chapter authors point to the Selikoff asbestos insulator study of cancer mortality as an example of information bias that diminished identification of mesothelioma risk because of misdiagnosis of mesotheliomas as lung cancers.[6] Although this is indeed an example of information bias that arose because of the uncertainty in distinguishing mesothelioma from lung cancer at a time when the diagnostic criteria for mesothelioma were not well developed, the authors ignore how this bias inflated lung cancer mortality, and how it inflated colorectal cancer mortality as a result of misdiagnosed peritoneal mesotheliomas. If the chapter authors had looked at the structure of the Selikoff study, they would have seen that smoking was a covariate reported by post card survey of a population (insulators), who were very much aware of the litigation issues, who funded the study, and who were keen to reduce the role attributable to smoking in the study. The Manual authors missed an important opportunity to discuss bias created by conducting studies in a group of workers who were keen to support their union’s and their own litigation efforts.
The chapter acknowledges that identifying biases can be challenging for expert witnesses, and can require expertise in epidemiologic methodology and the outcomes under study.[7] Unfortunately, the chapter does not address the very low bar for qualifying expert witnesses, which results in courtroom presentation of testimony about epidemiologic evidence from witnesses with weak to non-existent expertise in epidemiology or the specific disease outcome at issue. Both plaintiffs’ and defense counsel have been known to recruit treating physicians as expert witnesses on general and specific causation, despite their lack of epidemiologic expertise, on the belief that juries will accord them greater credibility because of their “hands-on” experience with patients. Despite the Manual’s acknowledgment of the need for subject-matter expertise, the chapter on epidemiology has nothing to say about the ineffective standards for ensuring actual expertise.
The Reference Manual’s discussion of study bias has its strengths, and more than its fair share of weaknesses, but any exposition of ten or so pages would be inadequate to the task of preparing judges and lawyers to complete their task in specific cases. The Manual could have been improved by including some discussion of the many resources available on the subject of systematic and other biases in epidemiology.
There are innumerable (figuratively speaking) journal articles on the many types of systematic biases.[8] There are also important book-length discussions of the full range of systematic and other biases that can afflict and invalidate epidemiologic studies. Michael B. Bracken, now professor emeritus at Yale University, has just published an important book, Bias! How Systemic Error Threatens Biomedical Research.[9] Bracken provides a helpful synopsis of his work:
“Scientists are alert to the play of chance in their research findings, but systemic error, which defines bias, is a much more insidious player. There are few formal methods for assessing bias, and researchers are often unaware of how bias is influencing their study results. Bias produces the worst kind of study outcome: The result appears precise and free of random error but, because of systemic error, it is wrong. Bias operates at every stage of research:
- which hypotheses will be tested;
- how study participants are selected;
- the choice of comparison groups;statistical analysis, research synthesis, and meta-analysis;and the interpretation and dissemination of study results.
Bias is a root cause of inefficiencies and waste in biomedical research, and for well-documented failures in result reproducibility. This book describes cognitive biases that influence scientists and science teams, as well as bias inherent in how research is conducted. Selected types of research are examined: genetic, pharmacologic, pandemic, clinical trial, and animal studies. Historical and modem examples are provided throughout the book and suggestions offered for how scientists might immunize themselves against the systemic error that threatens their work.”
Bracken’s book is a crucial resource for lawyers who need to understand the varied biases that must be considered in the evaluation of any epidemiologic study. The book’s discussions of the origins, types, and effects of biases go far beyond the meager (and at times biased) discussion of bias in the Reference Manual’s chapter on epidemiology. Lawyers who litigate health effect cases who fail to consult Bracken’s work on bias are likely deviating from their own professional standard of care.
Bracken’s work on bias is not the first book-length treatment of the subject. Professor Timothy Lash, along with co-authors, have published an important work, now in its second edition, on the quantification of biases.[10] In 2024, the International Agency for Research on Cancer has published a book on bias assessment in observational cancer epidemiologic studies.[11] The book, which grew out of a workshop funded in part by the National Cancer Institute, is available for download as a free PDF file from IARC’s website. Although not as comprehensive as it might be, the IARC’s textbook on bias does describe some techniques to avoid, control, and measure the role of bias in the results of epidemiologic studies. Unfortunately, IARC does not prescribe the same level of analysis for its working groups involved in classifying agents as cancer hazards.[12]
Another important book that belongs within easy reach of health-effect litigation lawyers is the award-winning book by statistician Herbert Weisberg, Bias and Causation: Models and Judgment for Valid Comparisons.[13] Weisberg drills down on bias as one the key problems in the assessment of causality. Judea Pearl called Weisberg’s work “a thoughtful and well written book, covering important issues of causal inference in every field of applied data analysis.”[14] Pearl fussed that while “the book shines in the motivational and conceptual levels,” he was not satisfied with it because of its lack of attention to mathematical models. Lawyers, especially those who lack training in advanced mathematics, will find this lack a blessing.
Of course, all the major textbooks on epidemiology will treat systematic bias with greater care and intensity than the Reference Manual. Mark Elwood’s insightful text on evaluating epidemiological studies probably provides more practical assistance to the litigation bar than the general epidemiology textbooks.[15] The Manual’s treatment of systematic bias creates an impression that the authors would prefer judges not to look too carefully at the indicia of invalidity in the studies relied upon by expert witnesses.
[1] Steve C. Gold, Michael D. Green, Jonathan Chevrier, & Brenda Eskenazi, Reference Guide on Epidemiology 897, 928, in National Academies of Sciences, Engineering, and Medicine & Federal Judicial Center, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE (4th ed. 2025) [RMSE4th]
[2] Id. at 903, 929 n. 94.
[3] 929 n. 94, citing and quoting from Daniels-Feasel v. Forest Pharms., Inc., No. 17 CV 4188-LTS-JLC, 2021 WL 4037820, at *2 (S.D.N.Y. Sept. 3, 2021). The chapter ignores that the district court found, in a Rule 702 evaluation, that the plaintiffs’ expert witnesses had failed to consider these alternatives adequately. The chapter also failed to note that the Second Circuit had affirmed the district court’s exclusion of the plaintiffs’ expert witnesses. Daniels-Feasel v. Forest Pharms., Inc., 2023 U.S. App. LEXIS 19448, 2023 WL 4837521 (2d Cir. July 28, 2023) (per curiam).
[4] RMSE4th at 903 n. 12, citing In re Phenylpropanolamine (PPA) Prods. Liab. Litig., 289 F. Supp. 2d 1230, 1240 (W.D. Wash. 2003).
[5] RMSE4th at 948 & n. 145 citing and quoting In re Roundup Prods. Liab. Litig., 390 F. Supp. 3d 1102, 1121 (N.D. Cal. 2018).
[6] d. at 946 n.140 citing Irving John Selikoff, et al., Mortality Experience of Insulation Workers in the United States and Canada, 220 ANN. N.Y. ACAD. SCI. 91, 110–11 (1979); David E. Lilienfeld & Paul D. Gunderson, The “Missing Cases” of Pleural Malignant Mesothelioma in Minnesota, 1979–81: Preliminary Report, 101 PUB. HEALTH REP. 395, 397–98 (1986).
[7] RMSE4th at 943.
[8] See, e.g., David L. Sackett, Bias in Analytic Research, 32 J. CHRON. DIS. 51 (1979).
[9] Michael B. Bracken, BIAS! HOW SYSTEMIC ERROR THREATENS BIOMEDICAL RESEARCH (2026)
[10] Matthew P. Fox, Richard F. MacLehose & Timothy L. Lash, APPLYING QUANTIATIVE BIAS ANALYSIS TO EPIDEMIOLOGIC DATA (2d ed. 2021).
[11] Amy Berrington de González, David B. Richardson & Mary K. Schubauer-Berigan, eds., STATISTICAL METHODS IN CANCER RESEARCH, vol. V: BIAS ASSESSMENT IN CASE-CONTROL AND COHORT STUDIES FOR HAZARD IDENTIFICATION, IARC Scientific Publication No. 171 (2024).
[12] See generally Schachtman, IARC’s Precautionary Science: How the WHO Cancer Research Agency Misinforms Regulation and Litigation, WLF Monograph (2016), https://www.wlf.org/wp-content/uploads/2026/04/WLF-Precautionary-Science-monograph.pdf
[13] Herbert I. Weisberg, BIAS AND CAUSATION: MODELS AND JUDGMENT FOR VALID COMPARISONS (2010).
[14] Judea Pearl, Review: Models and Judgment for Valid Comparisons, 68 BIOMETRICS 659, 660 (2012).
[15] Mark Elwood, APPRAISAL OF EPIDEMIOLOGICAL STUDIES AND CLINICAL TRIALS (2017). See also Raj S. Bhopal, ERROR, BIAS, AND CONFOUNDING IN EPIDEMIOLOGY (2016); Oxford Centre for Evidence-Based Medicine (CEBM), Catalogue of Bias, https://catalogofbias.org/biases/.
