TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Small Relative Risks and Causation (General & Specific)

June 28th, 2022

The Bradford Hill Predicate: Ruling Out Random and Systematic Error

In two recent posts, I spent some time discussing a recent law review, which had some important things to say about specific causation.[1] One of several points from which I dissented was the article’s argument that Sir Austin Bradford Hill had not made explicit that ruling out random and systematic error was required before assessing his nine “viewpoints” on whether an association was causal. I take some comfort in the correctness of my interpretation of Sir Austin’s famous article by reading the analysis of no less than Sir Richard Doll’s own analysis of his friend and colleague’s views:

“In summary, we have to show, first, that the association cannot reasonably be explained by chance (bearing in mind that extreme chances do turn up from time to time or no one would buy a ticket in a national lottery), by methodological bias (which can have many sources), or by confounding (which needs to be explored but should not be postulated without some idea of what it might be). Second, we have to see whether the available evidence gives positive support to the concept of causality: that is to say, how it matches up to Hill’s (1965) guidelines (Table 1).”[2]

On the issue of whether small relative risks can establish general causation, the Differential Etiology  paper urged caution in interpreting results when “strength of a relationship is modest.” The strength of an association is, of course, one of the nine Bradford Hill viewpoints, which come into play after we have a “clear-cut” association beyond what we would care to attribute to chance. Additionally, strength of association is primarily a quantitive assessment, and the advice given about caution in the face of “modest” associations is not terribly helpful.  The scientific literature does better.

Sir Richard’s 2002 paper is in a sense a scientific autobiography about some successes in discerning causal associations from observational studies. Unlike expert witnesses for the lawsuit industry, Sir Richard’s essay is notably for its intellectual humility.  In addition to its clear and explicit articulation of the need to rule out random and systematic error before proceeding to a consideration of Sir Austin’s nine guidelines, Sir Richard Doll’s 2002 essay is instructive for judges and lawyers, for other reasons. For example, he raises and explains the problem encountered for causal inference by small relative risks:

“Small relative risks of the order of 2:1 or even less are what are likely to be observed, like the risk now recorded for childhood leukemia and exposure to magnetic fields of 0.4 µT or more (Ahlbom et al. 2000) that are seldom encountered in the United Kingdom. And here the problems of eliminating bias and confounding are immense.”[3]

Sir Richard opines that relative risks under two can be shown to be causal associations, but often with massive data, randomization, and a good deal of support from experimental work.

Another Sir Richard, Sir Richard Peto, along with Sir Richard Doll, raised this concern in their classic essay on the causes of cancer, where they noted that relative risks between one and two create extremely difficult problems of interpretation because the role of the association cannot be confidently disentangled from the contribution of biases.[4] Small relative risks are thus seen as raising a concern about bias and confounding.[5]

In the legal world, courts have recognized that the larger the relative risk, or the strength of association, the more likely a general causation inference can be drawn, even when they blithely ignored the role of actual or residual confounding.[6]

The chapters on statistics and on epidemiology in the current (third) edition of the Reference Manual on Scientific Evidence directly tie the magnitude of the association to the elimination of confounding as an alternative explanation for causality of an association. A larger “effect size,” such as for smoking and lung cancer (greater than ten-fold, and often higher than 30-fold), eliminates the need to worry about confounding:

“Many confounders have been proposed to explain the association between smoking and lung cancer, but careful epidemiological studies have ruled them out, one after the other.”[7]

*  *  *  *  *  *

“A relative risk of 10, as seen with smoking and lung cancer, is so high that it is extremely difficult to imagine any bias or confounding factor that might account for it. The higher the relative risk, the stronger the association and the lower the chance that the effect is spurious. Although lower relative risks can reflect causality, the epidemiologist will scrutinize such associations more closely because there is a greater chance that they are the result of uncontrolled confounding or biases.”[8]

The Reference Manual omits the converse: the lower relative risk, the weaker the association and the greater the chance that the apparent effect is spurious. The authors’ intent, however, is clear enough. In the Appendix, below, I have collected some pronouncements from the scientific literature that urge caution in drawing causal inferences in the face of weak associations, but with more quantitative guidance.

 Small RRs and Specific Causation

Sir Richard Doll was among the first generation of epidemiologists in the academic world. He eschewed the use of epidemiology for discerning the cause of an individual’s disease:

“That asbestos is a cause of lung cancer in this practical sense is incontrovertible, but we can never say that asbestos was responsible for the production of the disease in a particular patient, as there are many other etiologically significant agents to which the individual may have been exposed, and we can speak only of the extent to which the risk of the disease was increased by the extent of his or her exposure.”[9]

On the individual attribution issue, Sir Richard’s views do not hold up as well as his analysis of general causation. Epidemiologic study results are used to predict future disease in individuals, to guide screening and prophylaxis decisions, to determine pharmacologic and surgical interventions in individuals, and to provide prognoses to individuals. Just as confounding falls by the wayside in the analysis of general causation with relative risks greater than 20, so too do the concerns about equating increased risk with specific causation.

The urn model of probability, however, gives us some insight into attributability. If we expected 100 cases of a disease in a sample of a certain size, but we observed 200 cases, then we would have 100 expected and 100 excess cases. Attribution would be no better than a flip of a coin.  If, however, in a situation where the relative risk was 20, we might have 100 expected cases and 2,000 excess cases. The odds of a given case’s being an excess case are rather strong, and even the agnostics and dissenters from probabilistic reasoning in individual cases become weak kneed about denying recovery when the claimant is similar to the cases seen in the study sample.

******************Appendix*************************

Norman E. Breslow & N. E. Day, “Statistical Methods in Cancer Research,” in The Analysis of Case-Control Studies 36 (IARC Pub. No. 32, 1980) (“[r]elative risks of less than 2.0 may readily reflect some unperceived bias or confounding factor”)

Richard Doll & Richard Peto, The Causes of Cancer 1219 (1981) (“when relative risk lies between 1 and 2 … problems of interpretation may become acute, and it may be extremely difficult to disentangle the various contributions of biased information, confounding of two or more factors, and cause and effect.”)

Iain K. Crombie, “The limitations of case-control studies in the detection of environmental carcinogens,” 35 35 J. Epidem. & Community Health 281, 281 (1981) (“The case-control study is unable to detect very small relative risks (< 1.5) even where exposure is widespread and large numbers of cases of cancer are occurring in the population.”)

Ernst L. Wynder & Geoffrey C. Kabat, “Environmental Tobacco Smoke and Lung Cancer: A Critical Assessment,” in H. Kasuga, ed., Indoor Air Quality 5, 6 (1990) (“An association is generally considered weak if the odds ratio is under 3.0 and particularly when it is under 2.0, as is the case in the relationship of ETS and lung cancer. If the observed relative risk is small, it is important to determine whether the effect could be due to biased selection of subjects, confounding, biased reporting, or anomalies of particular subgroups.”).

Ernst L. Wynder, “Epidemiological issues in weak associations,” 19 Internat’l  J. Epidemiol. S5 (1990)

David Sackett, R. Haynes, Gordon Guyatt, and Peter Tugwell, Clinical  Epidemiology: A Basic Science for Clinical Medicine (2d ed. 1991)

Muin J. Khoury, Levy M. James, W. Dana Flanders, and David J. Erickson, “Interpretation of recurring weak associations obtained from epidemiologic studies of suspected human teratogens,” 46 Teratology 69 (1992);

Lynn Rosenberg, “Induced Abortion and Breast Cancer: More Scientific Data Are Needed,” 86 J. Nat’l Cancer Instit. 1569, 1569 (1994) (“A typical difference in risk (50%) is small in epidemiologic terms and severely challenges our ability to distinguish if it reflects cause and effect or if it simply reflects bias.”) (commenting upon Janet R. Daling, K. E. Malone, L. F. Voigt, E. White, and Noel S. Weiss, “Risk of breast cancer among young women: relationship to induced abortion,” 86 J. Nat’l Cancer Inst. 1584 (1994);

Linda Anderson, “Abortion and possible risk for breast cancer: analysis and inconsistencies,” (Wash. D.C., Nat’l Cancer Institute, Oct. 26,1994) (“In epidemiologic research, relative risks of less than 2 are considered small and are usually difficult to interpret. Such increases may be due to chance, statistical bias, or effects of confounding factors that are sometimes not evident.”); 

Washington Post (Oct. 27, 1994) (quoting Dr. Eugenia Calle, Director of Analytic Epidemiology for the American Cancer Society: “Epidemiological studies, in general are probably not able, realistically, to identify with any confidence any relative risks lower than 1.3 (that is a 30% increase in risk) in that context, the 1.5 [reported relative risk of developing breast cancer after abortion] is a modest elevation compared to some other risk factors that we know cause disease.”)

Gary Taubes, “Epidemiology Faces Its Limits,” 269 Science 164, 168 (July 14, 1995) (quoting Marcia Angell, former editor of the New England Journal of Medicine, as stating that “[a]s a general rule of thumb, we are looking for a relative risk of 3 or more [before accepting a paper for publication], particularly if it is biologically implausible or if it’s a brand new finding.”) (quoting John C. Bailar: “If you see a 10-fold relative risk and it’s replicated and it’s a good study with biological backup, like we have with cigarettes and lung cancer, you can draw a strong inference. * * * If it’s a 1.5 relative risk, and it’s only one study and even a very good one, you scratch your chin and say maybe.”)

Samuel Shapiro, “Bias in the evaluation of low-magnitude associations: an empirical perspective,” 151 Am. J. Epidemiol. 939 (2000)

David A. Freedman & Philip B. Stark, “The Swine Flu Vaccine and Guillain-Barré Syndrome: A Case Study in Relative Risk and Specific Causation,” 64 Law & Contemp. Probs. 49, 61 (2001) (“If the relative risk is near 2.0, problems of bias and confounding in the underlying epidemiologic studies may be serious, perhaps intractable.”).

S. Straus, W. Richardson, P. Glasziou, and R. Haynes, Evidence-Based Medicine. How to Teach and Practice EBM (3d ed. 2005)

David F. Goldsmith & Susan G. Rose, “Establishing Causation with Epidemiology,” in Tee L. Guidotti & Susan G. Rose, eds., Science on the Witness Stand: Evaluating Scientific Evidence in Law, Adjudication, and Policy 57, 60 (2001) (“There is no clear consensus in the epidemiology community regarding what constitutes a ‘strong’ relative risk, although, at a minimum, it is likely to be one where the RR is greater than two; i.e., one in which the risk among the exposed is at least twice as great as among the unexposed.”)

Samuel Shapiro, “Looking to the 21st century: have we learned from our mistakes, or are we doomed to compound them?” 13 Pharmacoepidemiol. & Drug Safety  257 (2004)

Mark Parascandola, Douglas L Weed & Abhijit Dasgupta, “Two Surgeon General’s reports on smoking and cancer: a historical investigation of the practice of causal inference,” 3 Emerging Themes in Epidemiol. 1 (2006)

Heinemann, “Epidemiology of Selected Diseases in Women,” chap. 4, in M.A. Lewis, M. Dietel, P.C. Scriba, W.K. Raff, eds., Biology and Epidemiology of Hormone Replacement Therapy 47, 48 (2006) (discussing the “small relative risks in relation to bias/confounding and causal relation.”)

Roger D. Peng, Francesca Dominici, and Scott L. Zeger, “Reproducible Epidemiologic Research,” 163 Am. J. Epidem. 783, 784 (2006) (“The targets of current investigations tend to have smaller relative risks that are more easily confounded.”)

R. Bonita, R. Beaglehole & T. Kjellström, Basic Epidemiology 93 (W.H.O. 2d ed. 2006) (“A strong association between possible cause and effect, as measured by the size of the risk ratio (relative risk), is more likely to be causal than is a weak association, which could be influenced by confounding or bias. Relative risks greater than 2 can be considered strong.”)

David A. Grimes & Kenneth F. Schulz, “False alarms and pseudo-epidemics: the limitations of observational epidemiology,” 120 Obstet. & Gynecol. 920 (2012) (“Most reported associations in observational clinical research are false, and the minority of associations that are true are often exaggerated. This credibility problem has many causes, including the failure of authors, reviewers, and editors to recognize the inherent limitations of these studies. This issue is especially problematic for weak associations, variably defined as relative risks (RRs) or odds ratios (ORs) less than 4.”)

Kenneth F. Schulz & David A. Grimes, Essential Concepts in Clinical Research:
Randomised Controlled Trials and Observational Epidemiology at 75 (2d ed. 2019) (“Even after attempts to minimise selection and information biases and after control for known potential confounding factors, bias often remains. These biases can easily account for small associations. As a result, weak associations (which dominate in published studies) must be viewed with circumspection and humility.43 Weak associations, defined as relative risks between 0.5 and 2.0, in a cohort study can readily be accounted for by residual bias (Fig. 7.2). Because case-control studies are more susceptible to bias than are cohort studies, the bar must be set higher. ln case-control studies, weak associations can be viewed as odds ratios between 0.33 and 3.0 (Fig. 7.3). Results that full within these zones may be due to bias. Results that full outside these bounds in either direction may deserve attention.”)

Brian L. Strom, “Basic Principles of Clinical Epidemiology Relevant to Pharmacoepidemiologic Studies,” chap. 3, in Brian L. Strom, Stephen E. Kimmel & Sean Hennessy, eds., Pharmacoepidemiology 48 (6th ed. 2020) (“Conventionally, epidemiologists consider an association with a relative risk of less than 2.0 a weak association.”)


[1] Joseph Sanders, David L. Faigman, Peter B. Imrey, and Philip Dawid, “Differential Etiology: Inferring Specific Causation in the Law from Group Data in Science,” 63 Ariz. L. Rev. 851 (2021) [Differential Etiology].

[2] Richard Doll, “Proof of Causality: deduction from epidemiological observation,” 45 Persp. Biology & Med. 499, 501 (2002) (emphasis added).

[3] Id. at 512.

[4] Richard Doll & Richard Peto, The Causes of Cancer 1219 (1981) (“when relative risk lies between 1 and 2 … problems of interpretation may become acute, and it may be extremely difficult to disentangle the various contributions of biased information, confounding of two or more factors, and cause and effect.”).

[5]Confounding in the Courts” (Nov. 2, 2018); “General Causation and Epidemiologic Measures of Risk Size” (Nov. 24, 2012). 

[6] See King v. Burlington Northern Santa Fe Railway Co., 762 N.W.2d 24, 40 (Neb. 2009) (“the higher the relative risk, the greater the likelihood that the relationship is causal”); Landrigan v. Celotex Corp., 127 N.J. 404, 605 A.2d 1079, 1086 (1992) (“The relative risk of lung cancer in cigarette smokers as compared to nonsmokers is on the order of 10:1, whereas the relative risk of pancreatic cancer is about 2:1. The difference suggests that cigarette smoking is more likely to be a causal factor for lung cancer than for pancreatic cancer.”).

[7] RMSE3d at 219.

[8] RMSE3d at 602. 

[9] Richard Doll, “Proof of Causality: deduction from epidemiological observation,” 45 Persp. Biology & Med. 499, 500 (2002).

Only Judges Can Change History

June 24th, 2022

Samuel Butler is credited with the quip that “God cannot alter the past, though historians can.”  Historians can only try to alter the past; judges can do it with authority. Butler also had an opinion on authority: “Authority intoxicates, and makes mere sots of magistrates. The fumes of it invade the brain, and make men giddy, proud, and vain.”

Yesterday, the Supreme Court handed down its latest decision on the Second Amendment, in New York State Rifle & Pistol Ass’n v. Bruen.[1] Justice Thomas wrote the opinion for the Court, with concurrences by Justices Alito, Roberts, and Kavanaugh. Justice Breyer, joined by Justices Kagan and Sotomayor, dissented. A predictable 6 to 3 split, along what some regard as partisan lines. There are aspects of the decision, however, that will keep legal scholars and public intellectuals busy for a long time. In an affront to “textualists,” the Court’s decision perpetuates the earlier decision in Heller[2] to write the word “militia” out of the Constitution.

The Court’s and the dissent’s opinions have fulsome discussions of the history of laws regarding open and concealed carry of firearm. The Bruen case came up from the United States Court of Appeals on a decision to deny an injunction against the New York state judge who denied the petitioners’ a firearm carry permit. Remarkably, the decision was on the pleadings, with no testimony taken from historian expert witnesses. Remarkably, the Supreme Court took the case without requiring the petitioners to exhaust their state court appellate remedies.

As for the legal history, I defer to the legal historians. Justice Thomas, drawing from George Orwell, teaches that not all history is created equal. “Constitutional rights are enshrined with the scope they were understood to have when the people adopted them.”[3] Perhaps we should take Thomas’s teching to heart. The Second Amendment was adopted in 1791, when there were no bullets in the form of metal casings, with shaped lead projectiles. The Second Amendment right may well ensconce the entitlement to use black powder pistols that took a minute to reload and were accurate up to five meters, but there were no handguns that could fire “bullets,” made of metal casings, repeatedly. Relevant handgun technology did not evolve until the period around 1830-40, and even then, it was a novelty.

Maybe we do need a better history? In the meanwhile, New York should promptly require permits for bullets, as we know them, since they were not in existence in 1791, when the Second Amendment became law.

 

[1] New York State Rifle & Pistol Ass’n v. Bruen, No.20-843, Slip op., U.S. Supreme Court (June 23, 2022).

[2] District of Columbia v. Heller, 554 U. S. 570 (2008).

[3] Slip op. at 25, quoting from Heller, 554 U. S., at 634–635.

Statistical Significance Test Anxiety

June 20th, 2022

Although lawyers are known as a querulous lot, statisticians may not be far behind. The famous statistician John Wilder Tukey famously remarked that the collective noun for the statistical profession should be a “quarrel” of statisticians.[1]

Recently, philosopher Deborah Mayo, who has written insightfully about the “statistics wars,”[2] published an important article that addressed an attempt by some officers of the American Statistical Association (ASA) to pass off their personal views of statistical significance testing as views of the ASA.[3] This attempt took not only the form of an editorial over the name of the Executive Director, without a disclaimer, but also an email campaign to push journal editors to abandon statistical significance testing. Professor Mayo’s recent article explores the interesting concept of intellectual conflicts of interest arising from journal editors and association leaders who use their positions to advance their personal views. As discussed in some of my own posts, the conflict of interest led another ASA officer to appoint a Task Force on statistical significance testing, which has now, finally, been published in multiple fora.

Last week, on January 11, 2022, Professor Mayo convened a Zoom forum, “Statistical Significance Test Anxiety,” moderated by David Hand, at which she and Yoav Benjamini, an author of the ASA President’s Task Force, presented. About 70 statisticians and scientists from around the world attended.

Professor Mayo has hosted several editorial commentaries on her editorial in Conservation Biology, including guest blog posts from:

Brian Dennis
Philip Stark
Kent Staley
Yudi Pawitan
Christian Hennig
Ionides and Ritov
Brian Haig
Daniël Lakens

and my humble post, which is set out in full, below. There are additional posts on “statistical test anxiety” coming; check Professor Mayo’s blog for additional commentaries.

     *     *     *     *     *     *     *     *     *     *     *     *     *     *     *

Of Significance, Error, Confidence, and Confusion – In the Law and In Statistical Practice

The metaphor of law as an “empty vessel” is frequently invoked to describe the law generally, as well as pejoratively to describe lawyers. The metaphor rings true at least in describing how the factual content of legal judgments comes from outside the law. In many varieties of litigation, not only the facts and data, but the scientific and statistical inferences must be added to the “empty vessel” to obtain a correct and meaningful outcome.

Once upon a time, the expertise component of legal judgments came from so-called expert witnesses, who were free to opine about the claims of causality solely by showing that they had more expertise than the lay jurors. In Pennsylvania, for instance, the standard for qualify witnesses to give “expert opinions” was to show that they had “a reasonable pretense to expertise on the subject.”

In the 19th and the first half of the 20th century, causal claims, whether of personal injuries, discrimination, or whatever, virtually always turned on a conception of causation as necessary and sufficient to bring about the alleged harm. In discrimination claims, plaintiffs pointed to the “inexorable zero,” in cases in which no Black citizen was ever seated on a grand jury, in a particular county, since the demise of Reconstruction. In health claims, the mode of reasoning usually followed something like Koch’s postulates.

The second half of the 20th century was marked by the rise of stochastic models in our understanding of the world. The consequence is that statistical inference made its way into the empty vessel. The rapid introduction of statistical thinking into the law did not always go well. In a seminal discrimination case, Casteneda v. Partida, 430 U.S. 432 (1977), in an opinion by Associate Justice Blackmun, the court calculated a binomial probability for observing the sample result (rather than a result at least as extreme as such a result), and mislabeled the measurement “standard deviations” rather than standard errors:

“As a general rule for such large samples, if the difference between the expected value and the observed number is greater than two or three standard deviations, then the hypothesis that the jury drawing was random would be suspect to a social scientist.  The II-year data here reflect a difference between the expected and observed number of Mexican-Americans of approximately 29 standard deviations. A detailed calculation reveals that the likelihood that such a substantial departure from the expected value would occur by chance is less than I in 10140.”

Id. at 430 U.S. 482, 496 n.17 (1977). Justice Blackmun was graduated from Harvard College, summa cum laude, with a major in mathematics.

Despite the extreme statistical disparity in the 11-year run of grand juries, Justice Blackmun’s opinion provoked a robust rejoinder, not only on the statistical analysis, but on the Court’s failure to account for obvious omitted confounding variables in its simplistic analysis. And then there were the inconvenient facts that Mr. Partida was a rapist, indicted by a grand jury (50% with “Hispanic” names), which was appointed by jury commissioners (3/5 Hispanic). Partida was convicted by a petit jury (7/12 Hispanic), in front a trial judge who was Hispanic, and he was denied a writ of habeas court by Judge Garza, who went on to be a member of the Court of Appeals. In any event, Justice Blackmun’s dictum about “two or three” standard deviations soon shaped the outcome of many thousands of discrimination cases, and was translated into a necessary p-value of 5%.

Beginning in the early 1960s, statistical inference became an important feature of tort cases that involved claims based upon epidemiologic evidence. In such health-effects litigation, the judicial handling of concepts such as p-values and confidence intervals often went off the rails.  In 1989, the United States Court of Appeals for the Fifth Circuit resolved an appeal involving expert witnesses who relied upon epidemiologic studies by concluding that it did not have to resolve questions of bias and confounding because the studies relied upon had presented their results with confidence intervals.[4] Judges and expert witnesses persistently interpreted single confidence intervals from one study as having a 95 percent probability of containing the actual parameter.[5] Similarly, many courts and counsel committed the transposition fallacy in interpreting p-values as posterior probabilities for the null hypothesis.[6]

Against this backdrop of mistaken and misrepresented interpretation of p-values, the American Statistical Association’s p-value statement was a helpful and understandable restatement of basic principles.[7] Within a few weeks, however, citations to the p-value Statement started to show up in the briefs and examinations of expert witnesses, to support contentions that p-values (or any procedure to evaluate random error) were unimportant, and should be disregarded.[8]

In 2019, Ronald Wasserstein, the ASA executive director, along with two other authors wrote an editorial, which explicitly called for the abandonment of using “statistical significance.”[9] Although the piece was labeled “editorial,” the journal provided no disclaimer that Wasserstein was not speaking ex cathedra.

The absence of a disclaimer provoked a great deal of confusion. Indeed, Brian Turran, the editor of Significancepublished jointly by the ASA and the Royal Statistical Society, wrote an editorial interpreting the Wasserstein editorial as an official ASA “recommendation.” Turran ultimately retracted his interpretation, but only in response to a pointed letter to the editor.[10] Turran adverted to a misleading press release from the ASA as the source of his confusion. Inquiring minds might wonder why the ASA allowed such a press release to go out.

In addition to press releases, some people in the ASA started to send emails to journal editors, to nudge them to abandon statistical significance testing on the basis of what seemed like an ASA recommendation. For the most part, this campaign was unsuccessful in the major biomedical journals.[11]

While this controversy was unfolding, then President Karen Kafadar of the ASA stepped into the breach to state definitively that the Executive Director was not speaking for the ASA.[12]  In November 2019, the ASA board of directors approved a motion to create a “Task Force on Statistical Significance and Replicability.”[8] Its charge was “to develop thoughtful principles and practices that the ASA can endorse and share with scientists and journal editors. The task force will be appointed by the ASA President with advice and participation from the ASA Board.”

Professor Mayo’s editorial has done the world of statistics, as well as the legal world of judges, lawyers, and legal scholars, a service in calling attention to the peculiar intellectual conflicts of interest that played a role in the editorial excesses of some of  the ASA’s leadership. From a lawyer’s perspective, it is clear that courts have been misled, and distracted by, some of the ASA officials who seem to have worked to undermine a consensus position paper on p-values.[13]

Curiously, the task force’s report did not find a home in any of the ASA’s several scholarly publications. Instead “The ASA President’s Task Force Statement on Statistical Significance and Replicability[14] appeared in the The Annals of Applied  Statistics, where it is accompanied by an editorial by ASA former President Karen Kafadar.[15]  In November 2021, the ASA’s official “magazine,” Chance, also published the Task Force’s Statement.[16]

Judges and litigants who must navigate claims of statistical inference need guidance on the standard of care scientists and statisticians should use in evaluating such claims. Although the Taskforce did not elaborate, it advanced five basic propositions, which had been obscured by many of the recent glosses on the ASA 2016 p-value statement, and the 2019 editorial discussed above:

  1. “Capturing the uncertainty associated with statistical summaries is critical.”
  2. “Dealing with replicability and uncertainty lies at the heart of statistical science. Study results are replicable if they can be verified in further studies with new data.”
  3. “The theoretical basis of statistical science offers several general strategies for dealing with uncertainty.”
  4. “Thresholds are helpful when actions are required.”
  5. “P-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data.”

Although the Task Force’s Statement will not end the debate or the “wars,” it will go a long way to correct the contentions made in court about the insignificance of significance testing, while giving courts a truer sense of the professional standard of care with respect to statistical inference in evaluating claims of health effects.


[1] David R. Brillinger, “. . . how wonderful the field of statistics is. . . ,” Chap. 4, 41, 44, in Xihong Lin, et al., eds., Past, Present, and Future of Statistical Science (2014).

[2] Deborah MayoStatistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (2018).

[3] Deborah Mayo, “The Statistics Wars and Intellectual Conflicts of Interest,” Conservation Biology (2021) (in press).

[4] Brock v. Merrill Dow Pharmaceuticals, Inc., 874 F.2d 307, 311-12 (5th Cir. 1989).

[5] Richard W. Clapp & David Ozonoff, “Environment and Health: Vital Intersection or Contested Territory?” 30 Am. J. L. & Med. 189, 210 (2004) (“Thus, a RR [relative risk] of 1.8 with a confidence interval of 1.3 to 2.9 could very likely represent a true RR of greater than 2.0, and as high as 2.9 in 95 out of 100 repeated trials.”) (Both authors testify for claimants cases involving alleged environmental and occupational harms.); Schachtman, “Confidence in Intervals and Diffidence in the Courts” (Mar. 4, 2012) (collecting numerous examples of judicial offenders).

[6] See, e.g., In re Ephedra Prods. Liab. Litig., 393 F.Supp. 2d 181, 191, 193 (S.D.N.Y. 2005) (Rakoff, J.) (credulously accepting counsel’s argument that the use of a critical value of less than 5% of significance probability increased the “more likely than not” burden of proof upon a civil litigant). The decision has been criticized in the scholarly literature, but it is still widely cited without acknowledging its error. See Michael O. Finkelstein, Basic Concepts of Probability and Statistics in the Law 65 (2009).

[7] Ronald L. Wasserstein & Nicole A. Lazar, “The ASA’s Statement on p-Values: Context, Process, and Purpose,” 70 The Am. Statistician 129 (2016); see “The American Statistical Association’s Statement on and of Significance” (March 17, 2016). The commentary beyond the “bold faced” principles was at times less helpful in suggesting that there was something inherently inadequate in using p-values. With the benefit of hindsight, this commentary appears to represent editorizing by the authors, and not the sense of the expert committee that agreed to the six principles.

[8] Schachtman, “The American Statistical Association Statement on Significance Testing Goes to Court, Part I” (Nov. 13, 2018), “Part II” (Mar. 7, 2019).

[9] Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar, “Editorial: Moving to a World Beyond ‘p < 0.05’,” 73 Am. Statistician S1, S2 (2019); see Schachtman,“Has the American Statistical Association Gone Post-Modern?” (Mar. 24, 2019).

[10] Brian Tarran, “THE S WORD … and what to do about it,” Significance (Aug. 2019); Donald Macnaughton, “Who Said What,” Significance 47 (Oct. 2019).

[11] See, e.g., David Harrington, Ralph B. D’Agostino, Sr., Constantine Gatsonis, Joseph W. Hogan, David J. Hunter, Sharon-Lise T. Normand, Jeffrey M. Drazen, and Mary Beth Hamel, “New Guidelines for Statistical Reporting in the Journal,” 381 New Engl. J. Med. 285 (2019); Jonathan A. Cook, Dean A. Fergusson, Ian Ford, Mithat Gonen, Jonathan Kimmelman, Edward L. Korn, and Colin B. Begg, “There is still a place for significance testing in clinical trials,” 16 Clin. Trials 223 (2019).

[12] Karen Kafadar, “The Year in Review … And More to Come,” AmStat News 3 (Dec. 2019); see also Kafadar, “Statistics & Unintended Consequences,” AmStat News 3,4 (June 2019).

[13] Deborah Mayo, “The statistics wars and intellectual conflicts of interest,” 36 Conservation Biology (2022) (in-press, online Dec. 2021).

[14] Yoav Benjamini, Richard D. DeVeaux, Bradly Efron, Scott Evans, Mark Glickman, Barry Braubard, Xuming He, Xiao Li Meng, Nancy Reid, Stephen M. Stigler, Stephen B. Vardeman, Christopher K. Wikle, Tommy Wright, Linda J. Young, and Karen Kafadar, “The ASA President’s Task Force Statement on Statistical Significance and Replicability,” 15 Annals of Applied Statistics (2021) (in press).

[15] Karen Kafadar, “Editorial: Statistical Significance, P-Values, and Replicability,” 15 Annals of Applied Statistics (2021).

[16] Yoav Benjamini, Richard D. De Veaux, Bradley Efron, Scott Evans, Mark Glickman, Barry I. Graubard, Xuming He, Xiao-Li Meng, Nancy M. Reid, Stephen M. Stigler, Stephen B. Vardeman, Christopher K. Wikle, Tommy Wright, Linda J. Young & Karen Kafadar, “ASA President’s Task Force Statement on Statistical Significance and Replicability,” 34 Chance 10 (2021).

Differential Etiologies – Part Two – Ruling Out

June 19th, 2022

Perhaps the most important point of this law review article, “Differential Etiology: Inferring Specific Causation in the Law from Group Data in Science,”  is that general causation is necessary but insufficient, standing alone, to show specific causation. To be sure, the authors proclaimed that strong evidence of general causation somehow reduces the burden to show specific causation, but this pronouncement turned out to be an ipse dixit, without supporting analysis or citation. On general causation itself, what the authors characterized as the “ruling in” part of differential etiology, the authors offered some important considerations for courts to consider. Not the least of the important advice on general causation was urging caution in interpreting results when “strength of a relationship is modest.”[1] Given that they were talking to judges and lawyers, the advice might have taken on greater saliency if the authors explicitly noted that modest strength of a putative relationship means small relative risks, such as those smaller than two or three.

Acute Onset Conditions

The authors’ stated goal of bringing clarity to the determination of differential-etiology is a laudable one. In seeking clarity, they brush away some “easy” cases, such as the causal determination of acute onset conditions. Even so, the authors do not give any concrete examples. A broken bone discovered immediately after a car crash would hardly give a court much pause, but something such as the onset of acute liver failure shortly after ingesting a new medication turns out to be much more complicated than many would anticipate. Viral infections and autoimmune disease must be eliminated, and so such events are clearly in the realm of differential etiology, despite the close temporal proximity.

So-Called Signature Diseases

The authors also try to brush aside the “easy” case of signature diseases as not requiring differential etiology. The complexity of such cases ultimately embarrasses everyone. The authors no doubt thought that they were on safe ground in proffering the example of mesothelioma as a signature cancer caused by only asbestos (without wading into the deeper complexity of what is asbestos and which minerals in what mineralogical habit actually cause the disease).[2] Unfortunately, mesothelioma has never been a truly signal disease. The authors nonetheless consider it as one, with the caveat that mesotheliomas not caused by asbestos are “very rare.” And what was the authority for this statement? The Pennsylvania Supreme Court! Now the Pennsylvania Supreme Court is no doubt, at times, an authority on Pennsylvania law, if only because the Court is the last word on this contorted body of law. The Justices of that Court, however, would probably be the first to disclaim any credibility on the causes of any disease.[3]

The authors further distort the notion of signature diseases by stating that “[v]aginal adenocarcinoma in young women appears to be a signature disease associated with maternal use of DES.”[4] This cannot be right because over 10% of vaginal cancers are adenocarcimas. The principle of charity requires us to assume that the authors meant to indicate clear cell vaginal adenocarcinoma, but even so, charity will not correct the mistake. DES daughters do indeed have an increased risk of developing developing clear cell adenocarcinoma, but this type of cancer was well described before DES was ever invented and prescribed to women.[5]

Perhaps the safest ground for signature diseases is in microbiology, where we have infectious disease defined by the microbial agent that is uniquely associated with the disease. Probably close to the infectious diseases are the nutritional deficiency diseases defined by the absence of an essential nutrient or vitamin. To be sure, there are non-infectious diseases such as the pneumoconioses, each defined by the nature of the inhaled particle. Contrary to the authors’ contention, these diseases no not necessarily remove differential etiology from the analysis. Silicosis has a distinctive radiographic appearance, and yet that radiographic appearance is the same in many cases of coccidioidomycosis (Valley Fever). Asbestosis has a different radiographic appearance of the lungs and pleura, but the radiographic patterns might well be confused with the sequelae of rheumatoid arthritis or other interstitial lung diseases. At low levels of profusion of radiographic opacities, diseases such asbestosis and silicosis have diagnostic criteria that are far from perfect sensitivity and specificity. In one of the very first asbestos cases I defended, the claimant was diagnosed, by no less than the late Dr. Irving Selikoff,[6] with asbestosis, 3/3 on the ILO scale of linear, irregular radiographic lung opacities. An autopsy, however, found that there was no asbestosis at all, or even an elevated tissue fiber burden; the claimant had died of bilateral lymphangenitic carcinomatosis.

Definitive Mechanistic Pathway to Individual Causation

The paper presents a limited discussion of genetic causation. In the instance of mutations of highly penetrant alleles, identifying the genetic mutation will provide the general and the specific cause in a case. The authors also acknowledge that there may be cases involving hypothetical biomarkers that reveals a well-documented causal pathway from exposure to disease.

Differential Etiologies

So what happens when the plaintiff is claiming that he has developed a disease of ordinary life, one that has multiple known causes? Disease onset is not acute, but rather after a lengthy latency period. The plaintiff wants to inculpate the supposedly tortious exposure (the tortogen), and avoid the conclusion that any or all of the known alternative causes participated in his case. If there are cases of the disease without known causes (idiopathogens), the claimant will need to exclude idiopathogens in favor of fingering the tortogen as responsible for his bad outcome.

The authors helpfully distinguish differential diagnosis from differential etiology. The confusion of the two concepts has led to courts’ generally over-endorsing the black box of clinical judgment in health effects litigation. At the very least, this article can perhaps help the judiciary to move on from this naïve confusion.[7]

The authors advance the vague notion that somehow “clinical information” can supplement a relative that is not greater than two to augment the specific causation inference. This was, to be sure, the assertion of the New Jersey Supreme Court, based upon the improvident concession of the defense lawyer who argued the case.[8] There was nothing in the record of the New Jersey case, however, that would support the relevance of clinical information to the causal analysis of the plaintiff’s colorectal cancer.

The authors also point to a talc ovarian cancer case as exemplifying the use of clinical data to supplement a relative risk below two.[9] The cited case, however, involved expert witnesses who claimed a relative risk greater than two for the tortogen, and who failed to show how clinical information (such as the presence of talc in ovarian tissue) made the claimant any more likely to have had a cancer caused by talc.

Adverting to “clinical information” to supplement the relative risk all-too-often is hand waving that offers no analytical support for the specific causal inference. The clinical factors often are covariates in the multivariate model that generated the relevant relative risk. As such, the relative risk represents an assessment of the strength of the relevant association, independent of the clinical factors that are captured in the co-variates, in the multivariate model.  In the New Jersey case, Landrigan, plaintiff had no asbestosis that would suggest he even had a serious exposure to asbestos. In a companion case, Caternicchio, the plaintiff claimed that he had asbestosis, and somehow this made the causal inference for his colorectal cancer stronger.[10]  The epidemiologic studies he relied upon, however, stratified their analyses by length of exposure, and by radiographic category of asbestosis, neither of which suggested any relationship between radiographic findings and colorectal cancer outcome.

Perhaps because the authors are academics, they had to ask questions no one has every raised in a serious way in litigation, such as whether in addition to the clinical information, claimants could assert that toxicological data could be used to supplement a low (not greater than two) relative risk. The authors state the obvious; namely, toxicologic evidence is best suited to the assessment of general causation. They do not stop there, as they might have. Throwing their stated task of explicating the scientific foundations for specific causation inferences to the wind, the authors tell us that “[t]here is no formula for when such toxicologic evidence can tip the scales on the question of specific causation.”[11] And they wind up telling us vacuously that if the relevant epidemiology showed a small effect size, such as a two percent increased risk (RR = 1.02), then it would be unclear “how any animal data could cause one to substantially alter the best estimate of a human effect to reach a more-likely-than-not threshold.”[12] At this point in their paper, the authors seem to be discussing specific causation, but they offer nothing in the way of scientific evidence or examples of how toxicologic data could supplement a low relative risk (less than or equal to two) to permit a specific causation inference.

Idiopathy

When the analysis of the putative risk is done in a multivariate model that fairly covers the other relevant risks, relative risks less than 100 or so, suggest that there is a substantial baseline or background risk for the outcome of concern. When the relative risks identified in such analyses are less than 5 or so, the studies will suggest a reasonable proportion of so-called background cases with idiopathic (unknown) causes. Differential etiologies will have to rule out those mysterious idiopathogens.

If the putative specific cause is the only substance established to cause the outcome of concern, and the RR is greater than 1.0 and less than or equal to 2.0, by definition, there is a large base rate of the disease. No amount of hokey pokey will rule out the background causes. The authors deal with this scenario under the heading of differential etiology in the face of idiopathic causes, and characterize it as a “problem.”

Long story short, the authors conclude that “perhaps it is reasonable for courts to disregard idiopathic causes in those cases where idiopathic causes comprise a relatively small percent of all injuries.”[13] Such cases, however, by definition will diseases for which most causes are known, and the attributable fractions collectively for the known risks will be very high (say greater than 80 or 90%). Conversely, when the attributable fraction for all known risks is lower than 80%, the unexplained portion of the disease cases will represent idiopathic cases and causes that cannot be rule out with any confidence.

Differential etiology cannot work in the situation with a substantial baseline risk because there will be a disjunct (idiopathogen(s)) in the first statement of the syllogism, which cannot be ruled out. Thus, even if every other putative cause can be eliminated, the claimant will be left with the either the tortogen or the baseline risk as the cause of his injury, and the claimant will never arrive at a conclusion that is free of a disjunction that precludes judgment in his favor. In this scenario, the claimant must lose as a matter of law.

In their discussion of this issue, the authors note that this indeterminancy resulted in the exclusion of plaintiff’s expert witnesses in the notorious case of Milward v. Acuity Specialty Products Group, Inc.[14] In Milward, plaintiff had developed a rare variety of acute myeloid leukemia (AML), which had a large attributable fraction for idiopathic causation. This factual setting simply means that no known cause exists with a large relative risk, or even a small relative risk of 1.3 or so. Remarkably, these authors state that Milward “had prevailed on the general-causation issue” but in fact, no trial was ever held. The defense prevailed at the trial court by way of Rule 702 exclusion of plaintiff’s causation expert witnesses, but the First Circuit reversed and remanded for trial. The only prevailing that took place was the questionable avoidance of exclusion and summary judgment.[15]

On remand, the defense moved again to exclude plaintiffs’ expert witnesses on specific causation. Given that about 75% of AML cases are idiopathic, the court held that the plaintiffs’ expert witnesses attempt to proffer a differential etiology was fatally flawed.[16]

The authors cite the Milward specific causation decision, which in turn channeled the Restatement (Third) by couching the argument in terms of probability. If the claimant is left with a disjunction, [tortogen OR idiopathogen], then they suggest a probability value be assigned to the idiopathogen to support the inference that the probability that the tortogen was responsible for the claimant’s outcome [(1 – P(idiopathogen) x 100%]. Or in Judge Woodlock’s words:

“When a disease has a discrete set of causes, eliminating some number of them significantly raises the probability that the remaining option or options were the cause-in-fact of the disease. Restatement (Third) of Torts: Phys. & Emot. Harm § 28, cmt. c (2010) (‘The underlying premise [of differential etiology] is that each of the [ ] known causes is independently responsible for some proportion of the disease in a given population. Eliminating one or more of these as a possible cause for a specific plaintiff’s disease increases the probability that the agent in question was responsible for that plaintiff’s disease.’). The same cannot be said when eliminating a few possible causes leaves not only fewer possible causes but also a high probability that a cause cannot be identified. (‘When the causes of a disease are largely unknown . . . differential etiology is of little assistance.’).”[17]

The Milward approach is thus a vague, indirect invocation of relative risks and attributable fractions, without specifying the probabilities involved in quantitative terms.  Like obscenity, judges are supposed to discern when the residual probability of idiopathy is too great to permit an inference of specific causation. Somehow, I have the sense we should be able to do better than this.

Multiple Risks

To their credit, the authors tackle the difficult cases that arise when multiple risks are present. Those multiple risks may be competing risks, including the tortogen, in which case not all participate in bringing about the outcome. Indeed, if there is a baseline risk, the result may still have come about from an idiopathogen. The discussion in Differential Etiologies take some twists and turns, and I will not discuss all of it here.

Strong tortogen versus one weak competing risk

The authors describe the scenario of strong tortogen versus a single competing risk as one of the “easy cases,” at least when the alternative cause appears to be de minimus:

“If the choice of whether one’s lung cancer was the result of a lifetime of heavy smoking or by a brief encounter with a substance for which there is a significant but weak correlation with lung cancer, in most situations it should be an easy task to rule out the other substance as the specific cause of the individual’s injury.”[18]

Unfortunately, the article’s discussion leaves everything rather vague, without quantifying the risks involved. We can, without too much effort, provide some numbers, although we cannot be sure that the authors would accept the resulting quantification. If the claimant’s lifetime of heavy smoking carried a relative risk of 30, and the claimant worked for a few years in a truck depot where he was exposed to diesel fumes that carried a relative risk of 1.2, it would seem that it should be “an easy task” to rule out diesel fumes and rule in smoking. Note however that ease of the inference is lubricated by the size of the relative risks involved, one much larger than two, and the other much smaller than two, and the absence of any suggestion of interaction or synergy between them. If the tortogen in this scenario is tobacco, the plaintiff wins readily. If the tortogen is diesel fumes, the plaintiff loses. Query, if this scenario arises in a case against the tobacco company, whether the alternative causation defense of exposure to diesel fumes fails as a matter of law?

Synergy between strong tortogen and strong competing risk

The authors cannot resist the temptation to cite the Mt. Sinai catechism[19] of multiplicative risk from smoking and asbestos exposure[20]:

“A well-known example of a synergistic effect is the combined effect of asbestos exposure and smoking on the likelihood of developing lung cancer. For long-term smokers, the relative risk of developing lung cancer compared to those who have never smoked is sometimes estimated to be in the range of 10.0. For individuals substantially exposed to asbestos, the relative risk of developing lung cancer compared to non-exposed individuals is in the range of 5.0. However, if one is unfortunate enough to have been exposed to asbestos and to have been a long-term smoker, the relative risk compared to those unexposed individuals who have not smoked exceeds the sum of the relative risks. One possibility is that the relationship is multiplicative, in the range of 50.0—i.e., a 49-fold risk increment.”[21]

The synergistic interaction is often raised in an attempt to defeat causal apportionment or avoid responsibility for the larger risk, as when smokers attempt to recover for lung cancer from asbestos exposure. Some courts have, however, permitted causal apportionment. In their analysis, the authors of Differential Etiologies simply wink and tell us that “[t]he calculation of synergistic effects is fairly complex.”[22]

Tortogen versus Multiple Risks

The scenario in which the tortogen has been “ruled in,” and is present in the claimant’s history, along with multiple other risks is more difficult than one might have imagined. The authors tell us that an individual claimant will fail to show that the tortogen is more likely than not a cause of her injury when one or more of the competing risks is stronger than the risk from the tortogen (assuming no synergy).[23] The authors’ analysis leaves unclear why the claimant does not similarly fail when the strength of the tortogen is equal to that of a competing risk. Similarly, the claimant would appear to have fallen short of the burden of proving the tortogen’s causal role when there are multiple competing risk factors that individually present smaller risks than the tortogen, but for which multiple subsets represent combined competing risks greater than the risk of the tortogen. 

Concluding Thoughts

If the authors had framed the differential enterprise by the logic of iterative disjunctive syllogism, they would have recognized that the premise of the argument must contain the disjunction of all general causes that might have been a cause of the claimant’s disease or injury. Furthermore, unless the idiopathogen(s) is eliminated, which rarely is the case, we are left with a disjunction in the conclusion that prevents judgment for the plaintiff. The extensive analysis provided in Differential Etiologies ultimately must equate risk with cause, and it must do so on a probabilistic basis, even when the probabilities are left vague, and unquantified. Indeed, the authors come close to confronting the reality that we often do not know the cause of many individual’s diseases. We do know something about the person’s antecedent risks, and we can quantify and compare those risks. Noncommittally, the authors note that courts have been receptive to the practical solution of judging whether the tortogen’s relative risk was greater than two as a measure of sufficiency for specific causation, and that they “agree that theoretically this intuition has appeal.”[24]

Although I have criticized many aspect of the article, it is an important contribution to the legal study of specific causation. Its taxonomy will not likely be the final word on the subject, but it is a major step toward making sense of an area of the law long dominated by clinical black boxes and ipse dixits.


[1] Differential Etiologies at 885. The authors noted that their advice was “especially true in those case-control studies where the cases and controls are not drawn from the same defined population at risk for the outcome under investigation.”

[2] Differential Etiologies at 895.

[3] Differential Etiologies at 895 & n. 154, citing Betz v. Pneumo Abex, LLC, 44 A.3d 27, 51 (Pa. 2012).

[4] Differential Etiologies at 895 at n. 156.

[5] American Cancer Soc’y website, last visited June 19, 2022.

[6] I did not know at the time that Selikoff had failed the B-reader examination.

[7] See, e.g., Bowers v. Norfolk Southern Corp., 537 F. Supp. 2d 1343, 1359–60 (M.D. Ga. 2007) (“The differential diagnosis method has an inherent reliability; the differential etiology method does not. This conclusion does not suggest that the differential etiology approach has no merit. It simply means that courts, when dealing with matters of reliability, should consider opinions based on the differential etiology method with more caution. It also means that courts should not conflate the two definitions.”)

[8] Differential Etiologies at 899 & n.176, citing Landrigan v. Celotex Corp., 127 N.J. 404, 605 A.2d 1079, 1087 (1992).

[9] Differential Etiologies at 899 & n.179, citing Johnson & Johnson Talcum Powder Cases, 249 Cal. Rptr. 3d 642, 671–72 (Cal. Ct. App. 2019).

[10] Caterinicchio v. Pittsburgh Corning Corp., 127 N.J. 428, 605 A.2d 1092 (1992).

[11] Differential Etiologies at 899.

[12] Differential Etiologies at 900.

[13] Differential Etiologies at 915.

[14] 639 F.3d 11 (1st Cir. 2011).

[15] Does it require pointing out that the reversal took place with a highly questionable, unethical amicus brief submitted by a not-for-profit that was founded by the two plaintiffs’ expert witnesses excluded by the trial court? Given that the First Circuit reversed and remanded, and then later affirmed the exclusion of plaintiffs’ expert witnesses on specific causation, and the entry of judgment, the first appellate decision became unnecessary to the final judgment and no longer a clear precedent.

[16] Differential Etiologies at 912, discussing Milward v. Acuity Specialty Prods. Group, Inc., 969 F. Supp. 2d 101, 109 (D. Mass. 2013), aff’d sub. nom., Milward v. Rust-Oleum Corp., 820 F.3d 469, 471, 477 (1st Cir. 2016).

[17] Id., quoting from Milward.

[18] Differential Etiologies at 901.

[19]  “The Mt. Sinai Catechism” (June 11, 2013).

[20] The mantra of 5-10-50 comes from early publications by Irving John Selikoff, and represents a misrepresentation of “never smoked regularly” as “never smoked,” and the use of a non-contemporaneous control group for the non-asbestos exposed, non-smoker base rate. When the external control group was updated to show a relative risk of 20, rather than 10 for smoking only, Selikoff failed to update his analysis. Selikoff’s protégés have recently updated the insulator cohort, repeating many of the original errors, but even so, finding only that “the joint effect of smoking and asbestos alone was additive.” See Steve Markowitz, Stephen Levin, Albert Miller, and Alfredo Morabia, “Asbestos, Asbestosis, Smoking and Lung Cancer: New Findings from the North American Insulator Cohort,” Am. J. Respir. & Critical Care Med. (2013).

[21] Differential Etiologies at 902. The authors do not cite the Selikoff publications, which repeated his dataset and his dubious interpretation endlessly, but rather cite David Faigman, et al., Modern Scientific Evidence: The Law and Science of Expert Testimony § 26.25. (West 2019–2020 ed.). To their credit, the authors describe multiplicative interaction as a possibility, but surely they known that plaintiffs’ expert witnesses recite the Mt. Sinai catechism in courtrooms all around the country, while intoning “reasonable degree of medical certainty.” The authors cite some contrary studies. Differential Etiologies at 902 n.188, citing several reviews including Darren Wraith & Kerrie Mengersen, “Assessing the Combined Effect of Asbestos Exposure & Smoking on Lung Cancer: A Bayesian Approach, 26 Stats. Med. 1150, 1150 (2007) (evidence supports more than an additive model and less than a multiplicative relation).”

[22] Differential Etiologies at 902 at n.189.

[23] Differential Etiologies at 905. The authors note that courts have admitted differential etiology testimony when the tortogen’s risk is greater than the risk from other known risks. Id. citing Cooper v. Takeda Pharms., 191 Cal. Rptr. 3d 67, 79 (Ct. App. 2015).

[24] Differential Etiologies at 896 & n.163.

Differential Etiologies – Part One – Ruling In

June 17th, 2022

You put your right foot in

You put your right foot out

You put your right foot in

And you shake it all about

You do the Hokey Pokey and you turn yourself around

That’s what it’s all about!

 

Ever since the United States Supreme Court decided Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993), legal scholars, judges, and lawyers have struggled with the structure and validity of expert opinion on specific causation. Professor David Faigman and others have attempted to articulate the scientific basis (if any) for opinion testimony in health-effects litigation that a give person’s disease has been caused by an exposure or condition.

In 2015, as part of a tribute to the late Judge Jack Weinstein, Professor Faigman offered the remarkable suggestion that in advancing differential etiologies, expert witnesses were inventing wholesale an approach that had no foundation or acceptance in their scientific disciplines:

 “Differential etiology is ostensibly a scientific methodology, but one not developed by, or even recognized by, physicians or scientists. As described, it is entirely logical, but has no scientific methods or principles underlying it. It is a legal invention and, as such, has analytical heft, but it is entirely bereft of empirical grounding. Courts and commentators have so far merely described the logic of differential etiology; they have yet to define what that methodology is.”[1]

Faigman is correct that courts often have left unarticulated exactly what the methodology is, but he does not quite make sense when he writes that the method of differential etiology is “entirely logical,” but has no “scientific methods or principles underlying it.” After all, Faigman starts off his essay with a quotation from Thomas Huxley that “science is nothing but trained and organized common sense.”[2] As I have written elsewhere, the form of reasoning involved in differential diagnosis is nothing other than iterative disjunctive syllogism.[3] Either-or reasoning occurs throughout the physical and biological sciences; it is not clear why Faigman declares it un- or extra-scientific.

The strength of Faigman’s claim about the made-up nature of differential etiology appears to be undermined and contradicted by an example that he provides from clinical allergy and immunology:

“Allergists, for example, attempt to identify the etiology of allergic reactions in order to treat them (or to advise the patient to avoid what caused them), though it might still be possible to treat the allergic reactions without knowing their etiology.”

Faigman at 437. Of course, not only allergists try to determine the cause of an individual patient’s disease. Psychiatrists, in the psychoanalytic tradition, certain do so as well. Physicians who use predictive regression models use group data, in multivariate analyses, to predict outcomes, risk, and mortality in individual patients. Faigman’s claim is similarly undermined by the existence of a few diseases (other than infectious diseases) that are defined by the causative exposure. Silicosis and manganism have played a large role in often bogus litigation, but they represent instances in which a differential diagnosis and puzzle may also be an etiological diagnosis and a puzzle. Of course, to the extent that a disease is defined in terms of causative exposures, there may be serious and even intractable problems caused by the lack of specificity and accuracy in the diagnostic criteria for the supposedly pathognomonic disease.

As I noted at the time of Faigman’s 2015 essay, his suggestion that the concept of “differential etiology” was not used in the sciences themselves, was demonstrably flawed and historically inaccurate.[4]

A year earlier, in a more sustained analysis of specific causation, Professor Faigman went astray in a different direction, this time by stating that:

“it is not customary in the ordinary practice of sociology, epidemiology, anthropology, and related fields (for example, cognitive and social psychology) for professionals to make individual diagnostic judgments derived from group-based data.”[5]

Faigman’s invocation of “ordinary practice” of epidemiology was seriously wide of the mark. Medical practitioners and scientists frequently use epidemiologic data, based upon “group-based data” to make individual diagnostic judgments. The inferences from group data to individual range abound in the diagnostic process itself, where the specificity and sensitivity of disease signs and symptoms are measured by group data. Physicians must rely upon group data to make prognoses for individual patients, and they rely upon group data to predict future disease risks for individual patients. Future disease risks, as in the Framingham risk score for hard coronary heart disease, or the Gale model for breast cancer risk, are, of course, based upon “group-based data.” Medical decisions to intervene, surgically, pharmacologically, or by some other method, all involve applying group data to the individual patient.

Faigman’s 2014 law review article was certainly correct, however, in noting that specific causation inferences and conclusions were often left “profoundly underdefined,” with glib identifications of risk with cause.[6] There was thus plenty of room for further elucidation of specific causation decisions, and I welcome Faigman’s most recent effort to nail conceptual jello to the wall, in a law review article that was published last year.[7]

This new article, “Differential Etiology: Inferring Specific Causation in the Law from Group Data in Science,” is the collaborative product of Professor Faigman and three other academics. Joseph Sanders will be immediately recognizable to the legal community as someone who long pondered causation issues, both general and specific, and who has contributed greatly to the law review literature on causation of health outcomes. In addition to the law professors, Peter B. Imrey, a professor of medicine at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, and Philip Dawid, an emeritus professor of statistics in Cambridge University, have joined the effort to make sense of specific causation in the law. The addition of medical and statistical expertise has added greatly to Faigman’s previous efforts, and it has corrected some of his earlier errors and added much nuance to the discussion. The resulting law review article is well worth reading for practitioners. In this post, however, I have not detailed every important insight, but rather I have tried to point out some of the continuing and new errors in the analysis.

The Sanders-Faigman-Imbrey-Dawid analysis begins with a lament that:

“there is no body of science to which experts can turn when addressing this issue. Ultimately, much of the evidence that can be brought to bear on this causal question is the same group-level data employed to prove general causation. Consequently, the expert testimony often feels jerry-rigged, an improvisation designed to get through a tough patch.”[8]

As an assessment of the judicial decisions on specific causation, there can be no dissent or appeal from the judgment of these authors. The authors use of the term “jerry-rigged” is curious. I had first I thought they were straining to avoid using the common phrase “jury rigged” or to avoid inventing a neologism such as “judge rigged.” The American Heritage and Merriam Webster dictionaries, however, describe the phrase “jerry-rigged” as a conflation of “jury-rigged,” a nautical term for a temporary but adequate repair, with “jerry-rigged,” a war-time pejorative term for makeshift devices put together by Germans. So jerry-rigged it is, and the authors are off and running to try to describe, clarify, and justify the process of drawing specific causation inferences by differential etiology. They might have called what passes for judicial decision making in this area as the “hokey pokey.”

The authors begin their analysis of specific causation with a brief acknowledgement that our legal system could abandon any effort to set standards or require rigorous thinking on the matter by simply leaving the matter to the jury.[9] After all, this laissez-faire approach had been the rule of law for centuries. Nevertheless, despite occasional retrograde, recidivist judicial opinions,[10] the authors realize that the law has evolved to a point that some judicial control over specific causation opinions is required. And if judges are going to engage in gatekeeping of specific-causation opinions, they need to explain and justify their decisions in a coherent and cogent fashion.

Having thus dispatched legal nihilism, the authors turn their attention to what they boldly describe as “the first full-scale effort to bring scientific sensibilities – and rigorous statistical thinking – to the legally imperative concept of specific causation.”[11] The claim is remarkable claim given that tort law has been dealing with the issue for decades, but probably correct given how frequently judges have swept the issue under a judicial rug of inpenetrable verbiage and shaggy thinking. The authors also walk back some of Faigman’s earlier claims that there is no science in the assessment of specific causation, although they acknowledge the obvious, that policy issues sometimes play a role in deciding both general and specific causation decisions. The authors also offer the insight, for which they claim novelty, that some of the Bradford Hill guidelines, although stated as part of assessing general causation, have some relevancy to decisions concerning specific causation.[12] Their insight is indeed important, although hardly novel.

Drawing upon some of the clearer judicial decisions, the authors identify three necessary steps to reach a conclusion of specific causation:

“(a) making a proper diagnosis;

(b) supporting (“ruling in”) the plausibility of the alleged cause of the injury on the basis of general evidence and logic; and

(c) particularization, i.e., excluding (‘ruling out’) competing causes in the specific instance under consideration.”[13]

Although this article is ostensibly about specific causation, the authors do not reach a serious discussion of the matter until roughly the 42nd page of a 72 page article. Having described a three-step approach, the authors feel compelled to discuss step one (describing or defining the “diagnosis,” or the outcome of interest), and step two, the “ruling in” process that requires an assessment of general causation.

Although ascertaining general causation is not the focus of this article, the authors give an extensive discourse on it. Indeed, the authors have some useful things to say about steps one and two, and I commend the article to readers for some of its learning. As much as the lawsuit industry might wish to do away with the general causation step, it is not going anywhere soon.[14] The authors also manage to say some things that range from wrong to not even wrong. One example of professoriate wish casting is the following assertion:

“Other things being equal, when the evidence for general causation is strong, and especially when the strength of the exposure–disease relationship as demonstrated in a body of research is substantial, the plaintiff faces a lower threshold in establishing the substance as the cause in a particular case than when the relationship is weaker.”[15]

This assertion appears, sans citation or analysis. The generalization fails in the face of counterexamples. The causal role for estrogen in many breast cancers is extremely strong. The International Agency for Cancer Research classifies estrogen as a Category I, known human carcinogen for breast cancer, even though estrogen is made naturally in the human female, and male, body. In the Women’s Health Initiative clinical trial, researchers reported a hazard ratio of 1.2,[16] but plaintiffs struggled to prevail on specific causation in litigation involving claims of breast cancer caused by post-menopausal hormone therapy. Perhaps the authors meant, by strength of exposure relationship, a high relative risk as well, but that point is taken up when the authors address the “ruling in” step of the three-step approach. In any event, the strength of the case for general causation is quite independent of the specific causation inference, especially in the face of small effect sizes.

On general causation itself, the authors begin their discussion with “threats to validity,” a topic that they characterize as mostly implicit in the Bradford Hill guidelines. But their suggestion that validity is merely implicit in the guidelines is belied by their citation to Dr. Woodside’s helpful article on the “forgotten predicate” to the nine Bradford Hill guidelines.[17] Bradford Hill explicitly noted that the starting point for considering an association to be causal occurred when “[o]ur observations reveal an association between two variables, perfectly clear-cut and beyond what we would care to attribute to the play of chance.”[18] Sir Austin told us in no uncertain terms that there is no need to consider the nine guidelines until random and systematic error have been rejected.[19]

In this article’s discussion of general causation, Professor’s Dawid’s influence can be seen in the unusual care to describe and define the p-value.[20] But the discussion devolves into more wish casting, when the authors state that p-values are not the only way to assess random error in research results.

They double down by stating that “[m]any prominent statisticians and other scientists have questioned it, and the need for change is increasingly accepted.”[21] The source for their statement, the American Statistical Association (ASA) 2016 p-value Statement, did not questioned the utility of the p-value for assessing random error, and this law review provides no other support for other unidentified methods to assess random error. For the most part, the ASA Statement identified misuses and misstatements of p-values, with the caveat that “[s]cientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.” This is hardly questioning the importance or utility of p-values in assessing random error.

When one of the cited authors, Ronald Wasserstein, published an editorial in 2019, proclaiming that it was time to move past the p-value, the then president of the ASA, Professor Karen Kafadar, commissioned a task force on the matter. That task force, consisting of many of the world’s leading statisticians, issued a short, but pointed rejection of Wasserstein’s advocacy, and by implication, the position asserted in this law review.[22] Several of the leading biomedical journals that were lobbied by Wasserstein to abandon statistical significance testing reassessed their statistical guidelines and reaffirmed the use of p-values and tests.[23]

Similarly, this law review’s statements that alternatives to frequentist tests (p-values) such as Bayesian inference are “ascendant” have no supporting citations, and generally are an inaccurate assessment of what most biomedical journals are currently publishing.

Despite the care with which this law review article has defined p-values, the authors run off the road when defining a confidence interval:

A 95% confidence interval … is a one-sided or two-sided interval from a data sample with 95% probability of bounding a fixed, unknown parameter, for which no nondegenerate probability distribution is conceived, under specified assumptions about the data distribution.”[24]

The emphasis added is to point out that the authors assigned a single confidence interval with the property of bounding the true parameter with 95% probability. That property, however, belongs to the infinite set of confidence intervals based upon repeated sampling of the same size from the same population, and constant variance. There is no probability statement to be made for the true parameter, as either in or not in a given confidence interval.

In an issue that is relevant to general and specific causation, the authors offer some ipse dixit on the issue of “thresholds”:

“with respect to some substance/injury relationships, it is thought that there is no safe threshold. Cancer is the injury for which it is most frequently thought that there is no safe threshold, but even here the mechanism of injury may lead to a different conclusion.”[25]

Here as elsewhere, the authors are repeating dogma, not science, and they ignore the substantial body of scientific evidence that undermines the so-called linear no threshold dose-response curve. The only citation offered is a judicial citation to a case that rejected the no threshold position![26]

So much for “ruling in.” In the next post, I will turn my attention to this law review’s handling of the “ruling out” step of differential etiology.


[1] David L. Faigman & Claire Lesikar, “Organized Common Sense: Some Lessons from Judge Jack Weinstein’s Uncommonly Sensible Approach to Expert Evidence,” 64 DePaul L. Rev. 421, 444 (2015).

[2] Thomas H. Huxley, “On the Education Value of the Natural History Sciences” (1854), in Lay Sermons, Addresses and Reviews 77 (1915).

[3] See, e.g., “Differential Etiology and Other Courtroom Magic” (June 23, 2014) (collecting cases); “Differential Diagnosis in Milward v. Acuity Specialty Products Group” (Sept. 26, 2013).

[4] See David Faigman’s Critique of G2i Inferences at Weinstein Symposium (Sept. 11, 2015); Kløve & D. Doehring, “MMPI in epileptic groups with differential etiology,” 18 J. Clin. Psychol. 149 (1962); Kløve & C. Matthews, “Psychometric and adaptive abilities in epilepsy with differential etiology,” 7 Epilepsia 330 (1966); Teuber & K. Usadel, “Immunosuppression in juvenile diabetes mellitus? Critical viewpoint on the treatment with cyclosporin A with consideration of the differential etiology,” 103 Fortschr. Med. 707 (1985); G.May & W. May, “Detection of serum IgA antibodies to varicella zoster virus (VZV)–differential etiology of peripheral facial paralysis. A case report,” 74 Laryngorhinootologie 553 (1995); Alan Roberts, “Psychiatric Comorbidity in White and African-American Illicity Substance Abusers” Evidence for Differential Etiology,” 20 Clinical Psych. Rev. 667 (2000); Mark E. Mullinsa, Michael H. Leva, Dawid Schellingerhout, Gilberto Gonzalez, and Pamela W. Schaefera, “Intracranial Hemorrhage Complicating Acute Stroke: How Common Is Hemorrhagic Stroke on Initial Head CT Scan and How Often Is Initial Clinical Diagnosis of Acute Stroke Eventually Confirmed?” 26 Am. J. Neuroradiology 2207 (2005);Qiang Fua, et al., “Differential Etiology of Posttraumatic Stress Disorder with Conduct Disorder and Major Depression in Male Veterans,” 62 Biological Psychiatry 1088 (2007); Jesse L. Hawke, et al., “Etiology of reading difficulties as a function of gender and severity,” 20 Reading and Writing 13 (2007); Mastrangelo, “A rare occupation causing mesothelioma: mechanisms and differential etiology,” 105 Med. Lav. 337 (2014).

[5] David L. Faigman, John Monahan & Christopher Slobogin, “Group to Individual (G2i) Inference in Scientific Expert Testimony,” 81 Univ. Chi. L. Rev. 417, 465 (2014).

[6] Id. at 448.

[7] Joseph Sanders, David L. Faigman, Peter B. Imrey, and Philip Dawid, “Differential Etiology: Inferring Specific Causation in the Law from Group Data in Science,” 63 Ariz. L. Rev. 851 (2021) [Differential Etiology]. I am indebted to Kirk Hartley for calling this new publication to my attention.

[8] Id. at 851, 855.

[9] Id. at 855 & n. 8 (citing A. Philip Dawid, David L. Faigman & Stephen E. Fienberg, “Fitting Science into Legal Contexts: Assessing Effects of Causes or Causes of Effects?,” 43 Sociological Methods & Research 359, 363–64 (2014). See also Barbara Pfeffer Billauer, “The Causal Conundrum: Examining the Medical-Legal Disconnect in Toxic Tort Cases from a Cultural Perspective or How the Law Swallowed the Epidemiologist and Grew Long Legs and a Tail,” 51 Creighton L. Rev. 319 (2018) (arguing for a standard-less approach that allows clinicians to offer their ipse dixit opinions on specific causation).

[10] Differential Etiology at 915 & n.231, 919 & n.244 (citing In re Round-Up Prods. Liab. Litig., 358 F. Supp. 3d 956, 960 (N.D. Cal. 2019).

[11] Differential Etiology at 856 (emphasis added).

[12] Differential Etiology at 857.

[13] Differential Etiology at 857 & n.14 (citing Best v. Lowe’s Home Ctrs., Inc., 563 F.3d 171, 180 (6th Cir. 2009)).

[14] See Margaret Berger, “Eliminating General Causation: Notes Toward a New Theory of Justice and Toxic Torts,” 97 Colum L. Rev. 2117 (1997).

[15] Differential Etiology at 864.

[16] Jacques E. Rossouw, et al.,Risks and benefits of estrogen plus progestin in healthy postmenopausal women: Principal results from the Women’s Health Initiative randomized controlled trial,” 288 J. Am. Med. Ass’n 321 (2002).

[17] Differential Etiology at 884 & n.104, citing Frank Woodside & Allison Davis, “The Bradford Hill Criteria: The Forgotten Predicate,” 35 Thomas Jefferson L. Rev. 103 (2013).

[18] Austin Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295, 295 (1965).  

[19] Differential Etiology at 865.

[20] Differential Etiology at 869.

[21] Differential Etiology at 872, citing Ronald L. Wasserstein and Nicole A. Lazar, “The ASA Statement on p-Values: Context, Process, and Purpose,” 72 Am. Statistician 129 (2016).

[22] Yoav Benjamini, Richard D. De Veaux, Bradley Efron, Scott Evans, Mark Glickman, Barry I. Graubard, Xuming He, Xiao-Li Meng, Nancy M. Reid, Stephen M. Stigler, Stephen B. Vardeman, Christopher K. Wikle, Tommy Wright, Linda J. Young, and Karen Kafadar, “ASA President’s Task Force Statement on Statistical Significance and Replicability,” 15 Ann. Applied Statistics 1084 (2021), 34 Chance 10 (2021).

[23] See “Statistical Significance at the New England Journal of Medicine” (July 19, 2019); See also Deborah G. Mayo, “The NEJM Issues New Guidelines on Statistical Reporting: Is the ASA P-Value Project Backfiring?” Error Statistics Philosophy  (July 19, 2019).

[24] Differential Etiology at 898 n.173 (emphasis added).

[25] Differential Etiology at 890.

[26] Differential Etiology at n.134, citing Chlorine Chemistry Council v. Envt’l Protection Agency, 206 F.3d 1286 (D.C. Cir. 2000), which rejected the agency’s assumption that the carcinogenic effects of chloroform in drinking water lacked a threshold.