TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

Carl Cranor’s Conflicted Jeremiad Against Daubert

September 23rd, 2018

Carl Cranor’s Conflicted Jeremiad Against Daubert

It seems that authors who have the most intense and refractory conflicts of interest (COI) often fail to see their own conflicts and are the most vociferous critics of others for failing to identify COIs. Consider the spectacle of having anti-tobacco activists and tobacco plaintiffs’ expert witnesses assert that the American Law Institute had an ethical problem because Institute members included some tobacco defense lawyers.1 Somehow these authors overlooked their own positional and financial conflicts, as well as the obvious fact that the Institute’s members included some tobacco plaintiffs’ lawyers as well. Still, the complaint was instructive because it typifies the abuse of ethical asymmetrical standards, as well as ethical blindspots.2

Recently, Raymond Richard Neutra, Carl F. Cranor, and David Gee published a paper on the litigation use of Sir Austin Bradford Hill’s considerations for evaluating whether an association is causal or not.3 See Raymond Richard Neutra, Carl F. Cranor, and David Gee, “The Use and Misuse of Bradford Hill in U.S. Tort Law,” 58 Jurimetrics 127 (2018) [cited here as Cranor]. Their paper provides a startling example of hypocritical and asymmetrical assertions of conflicts of interests.

Neutra is a self-styled public health advocate4 and the Chief of the Division of Environmental and Occupational Disease Control (DEODC) of the California Department of Health Services (CDHS). David Gee, not to be confused with the English artist or the Australian coin forger, is with the European Environment Agency, in Copenhagen, Denmark. He is perhaps best known for his precautionary principle advocacy and his work with trade unions.5

Carl Cranor is with the Center for Progressive Reform, and he teaches philosophy at one of the University of California campuses. Although he is neither a lawyer nor a scientist, he participates with some frequency as a consultant, and as an expert witness, in lawsuits, on behalf of claimants. Perhaps Cranor’s most notorious appearance as an expert witness resulted in the decision of Milward v. Acuity Specialty Products Group, Inc., 639 F.3d 11 (1st Cir. 2011), cert. denied sub nom., U.S. Steel Corp. v. Milward, 132 S. Ct. 1002 (2012). Probably less generally known is that Cranor was one of the founders of an organization, the Council for Education and Research on Toxics (CERT), which recently was the complaining party in a California case in which CERT sought money damages for Starbucks’ failure to label each cup of coffee sold as known to the State of California as causing cancer.6 Having a so-called not-for-profit corporation can also be pretty handy, especially when it holds itself out as a scientific organization and files amicus briefs in support of reversing Daubert exclusions of the founding members of the corporation, as CERT did on behalf of its founding member in the Milward case.7 The conflict of interest, in such an amicus brief, however, is no longer potential or subtle, and violates the duty of candor to the court.

In this recent article on Hill’s considerations for judging causality, Cranor followed CERT’s lead from Milward. Cranor failed to disclose that he has been a party expert witness for plaintiffs, in cases in which he was advocating many of the same positions put forward in the Jurimetrics article, including the Milward case, in which he was excluded from testifying by the trial court. Cranor’s lack of candor with the readers of the Jurimetrics article is all the more remarkable in that Cranor and his co-authors give conflicts of interest outsize importance in substantive interpretations of scholarship:

“the desired reliability for evidence evaluation requires that biases that derive from the financial interests and ideological commitments of the investigators and editors that control the gateways to publication be considered in a way that Hill did not address.”

Cranor at 137 & n.59. Well, we could add that Cranor’s financial interests and ideological commitments might well be considered in evaluating the reliability of the opinions and positions advanced in this most recent work by Cranor and colleagues. If you believe that COIs disqualify a speaker from addressing important issues, then you have all the reason you need to avoid reading Cranor’s recent article.

Dubious Scholarship

The more serious problem with Cranor’s article is not his ethically strained pronouncements about financial interests, but the dubious scholarship he and his colleagues advance to thwart judicial gatekeeping of even more dubious expert witness opinion testimony. To begin with, the authors disparage the training and abilities of federal judges to assess the epistemic warrant and reliability of proffered causation opinions:

“With their enhanced duties to review scientific and technical testimony federal judges, typically not well prepared by legal education for these tasks, have struggled to assess the scientific support for—and the reliability and relevance of—expert testimony.”

Cranor at 147. Their assessment is fair but hides the authors’ cynical agenda to remove gatekeeping and leave the assessment to lay juries, who are less well prepared for the task, and whose function ensures no institutional accountability, review, or public evaluation.

Similarly, the authors note the temporal context and limitations of Bradford Hill’s 1965 paper, which date and limit the advice provided over 50 years ago in a discipline that has changed dramatically with the advancement of biological, epidemiologic, and genetic science.8 Even at the time of its original publication in 1965, Bradford Hill’s paper, which was based upon an informal lecture, was not designed or intended to be a definitive treatment of causal inference. Cranor and his colleagues make no effort to review Bradford Hill’s many other publications, both before and after his 1965 dinner speech, for evidence of his views on the factors for causal inference, including the role of statistical testing and inference.

Nonetheless, Bradford Hill’s 1965 paper has become a landmark, even if dated, because of its author’s iconic status in the world of public health, earned for his showing that tobacco smoking causes lung cancer,9 and for advancing the role of double-blind randomized clinical trials.10 Cranor and his colleagues made no serious effort to engage with the large body of Bradford Hill’s writings, including his immensely important textbook, The Principles of Medical Statistics, which started as a series of articles in The Lancet, and went through 12 editions in print.11 Hill’s reputation will no doubt survive Cranor’s bowdlerized version of Sir Austin’s views.

Epidemiology is Dispensable When It Fails to Support Causal Claims

The egregious aspect of Cranor’s article is its bill of particulars against the federal judiciary for allegedly errant gatekeeping, which for these authors translates really into any gatekeeping at all. Cranor at 144-45. Indeed, the authors provide not a single example of what was a “proper” exclusion of an expert witness, who was contending for some doubtful causal claim. Perhaps they have never seen a proper exclusion, but doesn’t that speak volumes about their agenda and their biases?

High on the authors’ list of claimed gatekeeping errors is the requirement that a causal claim be supported with epidemiologic evidence. Although some causal claims may be supported by strong evidence of a biological process with mechanistic evidence, such claims are not common in United States tort litigation.

In support of the claim that epidemiology is dispensable, Cranor suggests that:

“Some courts have recognized this, and distinguished scientific committees often do not require epidemiological studies to infer harm to humans. For example, the International Agency for Research on Cancer (IRAC) [sic], the National Toxicology Program, and California’s Proposition 65 Scientific Advisory Panel, among others, do not require epidemiological data to support findings that a substance is a probable or—in some cases—a known human carcinogen, but it is welcomed if available.”

Cranor at 149. California’s Proposition 65!??? Even IARC is hard to take seriously these days with its capture by consultants for the litigation industry, but if we were to accept IARC as an honest broker of causal inferences, what substance “known” to IARC to cause cancer in humans (Category I) was branded as a “known carcinogen” without the support of epidemiologic studies? Inquiring minds might want to know, but they will not learn the answer from Cranor and his co-authors.

When it comes to adverting to legal decisions that supposedly support the authors’ claim that epidemiology is unnecessary, their scholarship is equally wanting. The paper cites the notorious Wells case, which was so roundly condemned in scientific circles, that it probably helped ensure that a decision such as Daubert would ultimately be handed down by the Supreme Court. The authors seemingly cannot read, understand, and interpret even the most straightforward legal decisions. Here is how they cite Wells as support for their views:

“Wells v. Ortho Pharm. Corp., 788 F.2d 741, 745 (11th Cir. 1986) (reviewing a district court’s decision deciding not to require the use of epidemiological evidence and instead allowing expert testimony).”

Cranor at 149-50 n.122. The trial judge in Wells never made such a decision; indeed, the case was tried by the bench, before the Supreme Court decided Daubert. There was no gatekeeping involved at all. More important, however, and contrary to Cranor’s explanatory parenthetical, both sides presented epidemiologic evidence in support of their positions.12

Cranor and his co-authors similarly misread and misrepresent the trial court’s decision in the litigation over maternal sertraline use and infant birth defects. Twice they cite the Multi-District Litigation trial court’s decision that excluded plaintiffs’ expert witnesses:

“In re Zoloft (Sertraline Hydrochloride) Prods. Liab. Litig., 26 F. Supp. 3d 449, 455 (E.D. Pa. 2014) (expert may not rely on nonstatistically significant studies to which to apply the [Bradford Hill] factors).”

Cranor at 144 n.85; 158 n.179. The MDL judge, Judge Rufe, decidedly never held that an expert witness may not rely upon a statistically non-significant study in a “Bradford Hill” analysis, and the Third Circuit, which affirmed the exclusions of the plaintiffs’ expert witnesses’ testimony, was equally clear in avoiding the making of such a pronouncement.13

Who Needs Statistical Significance

Part of Cranor’s post-science agenda is to intimidate judges into believing that statistical significance is unnecessary and a wrong-headed criterion for judging the validity of relied upon research. In their article, Cranor and friends suggest that Hill agreed with their radical approach, but nothing could be further from the truth. Although these authors parse almost every word of Hill’s 1965 article, they conveniently omit Hill’s views about the necessary predicates for applying his nine considerations for causal inference:

“Disregarding then any such problem in semantics we have this situation. Our observations reveal an association between two variables, perfectly clear-cut and beyond what we would care to attribute to the play of chance. What aspects of that association should we especially consider before deciding that the most likely interpretation of it is causation?”

Austin Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295, 295 (1965). Cranor’s radicalism leaves no room for assessing whether a putative association is “beyond what we would care to attribute to the play of chance,” and his poor scholarship ignores Hill’s insistence that this statistical analysis be carried out.14

Hill’s work certainly acknowledged the limitations of statistical method, which could not compensate for poorly designed research:

“It is a serious mistake to rely upon the statistical method to eliminate disturbing factors at the completion of the work.  No statistical method can compensate for a badly planned experiment.”

Austin Bradford Hill, Principles of Medical Statistics at 4 (4th ed. 1948). Hill was equally clear, however, that the limits on statistical methods did not imply that statistical methods are not needed to interpret a properly planned experiment or study. In the summary section of his textbook’s first chapter, Hill removed any doubt about his view of the importance, and the necessity, of statistical methods:

“The statistical method is required in the interpretation of figures which are at the mercy of numerous influences, and its object is to determine whether individual influences can be isolated and their effects measured.”

Id. at 10 (emphasis added).

In his efforts to eliminate judicial gatekeeping of expert witness testimony, Cranor has struggled with understanding of statistical inference and testing.15 In an early writing, a 1993 book, Cranor suggests that we “can think of type I and II error rates as “standards of proof,” which begs the question whether they are appropriately used to assess significance or posterior probabilities.16 Indeed, Cranor goes further, in confusing significance and posterior probabilities, when he described the usual level of alpha (5%) as the “95%” rule, and claimed that regulatory agencies require something akin to proof “beyond a reasonable doubt,” when they require two “statistically significant” studies.17

Cranor has persisted in this fallacious analysis in his writings. In a 2006 book, he erroneously equated the 95% coefficient of statistical confidence with 95% certainty of knowledge.18 Later in this same text, Cranor again asserted his nonsense that agency regulations are written when supported by “beyond a reasonable doubt.”19 Given that Cranor has consistently confused significance and posterior probability, he really should not be giving advice to anyone about statistical or scientific inference. Cranor’s persistent misunderstandings of basic statistical concepts do, however, explain his motivation for advocating the elimination of statistical significance testing, even if these misunderstandings make his enterprise intellectually unacceptable.

Cranor and company fall into a similar muddle when they offer advice on post-hoc power calculations, which advice ignores standard statistical learning for interpreting completed studies.20 Another measure of the authors’ failed scholarship is their omission of any discussion of recent efforts by many in the scientific community to lower the threshold for statistical significance, based upon the belief that the customary 5% p-value is an order of magnitude too high.21

 

Relative Risks Greater Than Two

There are other tendentious arguments and treatments in Cranor’s brief against gatekeeping, but I will stop with one last example. The inference of specific causation from study risk ratios has provoked a torrent of verbiage from Sander Greenland (who is cited copiously by Cranor). Cranor, however, does not even scratch the surface of the issue and fails to cite the work of epidemiologists, such as Duncan C. Thomas, who have defended the use of probabilities of (specific) causation. More important, however, Cranor fails to speak out against the abuse of using any relative risk greater than 1.0 to support an inference of specific causation, when the nature of the causal relationship is neither necessary nor sufficient. In this context, Kenneth Rothman has reminded us that someone can be exposed to, or have, a risk, and then develop the related outcome, without there being any specific causation:

“An elementary but essential principle to keep in mind is that a person may be exposed to an agent and then develop disease without there being any causal connection between the exposure and the disease. For this reason, we cannot consider the incidence proportion or the incidence rate among exposed people to measure a causal effect.”

Kenneth J. Rothman, Epidemiology: An Introduction at 57 (2d ed. 2012).

The danger in Cranor’s article in Jurimetrics is that some readers will not realize the extreme partisanship in its ipse dixit, and erroneous, pronouncements. Caveat lector! 

1 Elizabeth Laposata, Richard Barnes & Stanton Glantz, “Tobacco Industry Influence on the American Law Institute’s Restatements of Torts and Implications for Its Conflict of Interest Policies,” 98 Iowa L. Rev. 1 (2012).

2 The American Law Institute responded briefly. See Roberta Cooper Ramo & Lance Liebman, “The ALI’s Response to the Center for Tobacco Control Research & Education,” 98 Iowa L. Rev. Bull. 1 (2013), and the original authors’ self-serving last word. Elizabeth Laposata, Richard Barnes & Stanton Glantz, “The ALI Needs to Implement Modern Conflict of Interest Policies,” 98 Iowa L. Rev. Bull. 17 (2013).

3 Austin Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295 (1965).

4 Raymond Richard Neutra, “Epidemiology Differs from Public Health Practice,” 7 Epidemiology 559 (1996).

7 “From Here to CERT-ainty” (June 28, 2018).

8 Kristen Fedak, Autumn Bernal, Zachary Capshaw, and Sherilyn A Gross, “Applying the Bradford Hill Criteria in the 21st Century: How Data Integration Has Changed Causal Inference in Molecular Epidemiology,” Emerging Themes in Epidemiol. 12:14 (2015); John P. A. Ioannides, “Exposure Wide Epidemiology, Revisiting Bradford Hill,” 35 Stats. Med. 1749 (2016).

9 Richard Doll & Austin Bradford Hill, “Smoking and Carcinoma of the Lung,” 2(4682) Brit. Med. J. (1950).

10 Geoffrey Marshall (chairman), “Streptomycin Treatment of Pulmonary Tuberculosis: A Medical Research Council Investigation,” 2 Brit. Med. J. 769, 769–71 (1948).

11 Vern Farewell & Anthony Johnson, “The origins of Austin Bradford Hill’s classic textbook of medical statistics,” 105 J. Royal Soc’y Med. 483 (2012). See also Hilary E. Tillett, “Bradford Hill’s Principles of Medical Statistics,” 108 Epidemiol. Infect. 559 (1992).

13 In re Zoloft Prod. Liab. Litig., No. 16-2247 , __ F.3d __, 2017 WL 2385279, 2017 U.S. App. LEXIS 9832 (3d Cir. June 2, 2017) (affirming exclusion of biostatistician Nichols Jewell’s dodgy opinions, which involved multiple methodological flaws and failures to follow any methodology faithfully).

14 See “Bradford Hill on Statistical Methods” (Sept. 24, 2013).

16 Carl F. Cranor, Regulating Toxic Substances: A Philosophy of Science and the Law at 33-34 (1993) (arguing incorrectly that one can think of α, β (the chances of type I and type II errors, respectively and 1- β as measures of the “risk of error” or “standards of proof.”); see also id. at 44, 47, 55, 72-76. At least one astute reviewer called Cranor on his statistical solecisms. Michael D. Green, “Science Is to Law as the Burden of Proof is to Significance Testing: Book Review of Cranor, Regulating Toxic Substances: A Philosophy of Science and the Law,” 37 Jurimetrics J. 205 (1997) (taking Cranor to task for confusing significance and posterior (burden of proof) probabilities).

17 Id. (squaring 0.05 to arrive at “the chances of two such rare events occurring” as 0.0025, which impermissibly assumes independence between the two studies).

18 Carl F. Cranor, Toxic Torts: Science, Law, and the Possibility of Justice 100 (2006) (incorrectly asserting that “[t]he practice of setting α =.05 I call the “95% rule,” for researchers want to be 95% certain that when knowledge is gained [a study shows new results] and the null hypothesis is rejected, it is correctly rejected.”).

19 Id. at 266.

21 See, e.g., John P. A. Ioannidis, “The Proposal to Lower P Value Thresholds to .005,” 319 J. Am. Med. Ass’n 1429 (2018); Daniel J. Benjamin, James O. Berger, Valen E. Johnson, et al., “Redefine statistical significance,” 2 Nature Human Behavior 6 (2018).

Posted in Causation, Conflicts of Interest, Public Nuisances, Rule 702, Scientific Evidence, statistical evidence | Comments Off on Carl Cranor’s Conflicted Jeremiad Against Daubert

The Appeal of the Learned Treatise

August 16th, 2018

In many states, the so-called “learned treatise” doctrine creates a pseudo-exception to the rule against hearsay. The contents of such a treatise can be read to the jury, not for its truth, but for the jury to consider against the credibility of an expert witness who denies the truth of the treatise. Supposedly, some lawyers can understand the distinction between the treatise’s content’s being admitted for its truth as opposed to the credibility of an expert witness who denies its truth. Under the Federal Rules of Evidence, and in some states, the language of the treatise may be considered for its truth as well, but the physical treatise may not be entered into evidence. There are several serious problems with both the state and the federal versions of the doctrine.1

Legal on-line media recently reported about an appeal in the Pennsylvania Superior Court, which heard arguments in a case that apparently turned on allegations of trial court error in refusing to allow learned treatise cross-examination of a plaintiff’s expert witness in Pledger v. Janssen Pharms., Inc., Phila. Cty. Ct. C.P., April Term 2012, No. 1997. See Matt Fair, “J&J Urges Pa. Appeals Court To Undo $2.5M Risperdal Verdict,” Law360 (Aug. 8, 2018) (reporting on defendants’ appeal in Pledger, Pa. Super. Ct. nos. 2088 EDA 2016 and 2187 EDA 2016).

In Pledger, plaintiff claimed that he developed gynecomastia after taking the defendants’ antipsychotic medication Risperdal. Defendants warned about gynecomastia, but the plaintiff claimed that the defendants had not accurately quantified the rate of gynecomastia in its package insert.

From Mr. Fair’s reporting, readers can discern only one ground for appeal, namely whether the “trial judge improperly barred it from using a scientific article to challenge an expert’s opinion that the antipsychotic drug Risperdal caused an adolescent boy to grow breasts.” Without having heard the full oral argument, or having read the briefs, the reader cannot tell whether there were other grounds. According to Mr. Fair, defense counsel contended that the trial court’s refusal to allow the learned treatise “had allowed the [plaintiff’s] expert’s opinion to go uncountered during cross-examination.” The argument, according to Mr. Fair, continued:

“Instead of being able to confront the medical causation expert with an article that absolutely contradicted and undermined his opinion, the court instead admonished counsel in front of the jury and said, ‘In Pennsylvania, we don’t try cases by books, we try them by live witnesses’.”

The cross-examination at issue, on the other hand, related to whether gynecomastia could occur naturally in pre-pubertal boys. Plaintiffs’ expert witness, Dr. Mark Solomon, a plastic surgeon, opined that gynecomastia did not occur naturally, and the defense counsel attempted to confront him with a “learned treatise,” an article from the Journal of Endocrinology, which apparently stated to the contrary. Solomon, following the usual expert witness playbook, testified that he had not read the article (and why would a surgeon have read this endocrinology journal?) Defense counsel pressed, and according to Mr. Fair, the trial judge disallowed further inquiry on cross-examination. On appeal, the defendants argued that the trial judge violated the learned treatise rule that allows “scholarly articles to be used as evidence.” The plaintiffs contended, in defense of their judgment below, that the “learned treatise rule” does not allow “scholarly articles to simply be read verbatim into the record,” and that the defense had the chance to raise the article in the direct examination of its own expert witnesses.

The Law360 reporting is curious on several fronts. The assigned error would have only been in support of a challenge to the denial of a new trial, and in a Risperdal case, the defense would likely have made a motion for judgment notwithstanding the verdict, as well as for new trial. Although the appellate briefs are not posted online, the defense’s post-trial motions in Pledger v. Janssen Pharms., Inc., Phila. Cty. Ct. C.P., April Term 2012, No. 1997, are available. See Defendants’ Motions for Post-Trial Relief Pursuant to Pa.R.C.P. 227.1 (Mar. 6, 2015).

At least at the post-trial motion stage, the defendants clearly made both motions for judgment and for a new trial, as expected.

As for the preservation of the “learned treatise” issue, the entire assignment of error is described in a single paragraph (out of 116 paragraphs) in the post-trial motion, as follows:

27. Moreover, appearing to rely on Aldridge v. Edmunds, 750 A.2d 292 (Pa. 2000), the Court prevented Janssen from cross-examining Dr. Solomon with scientific authority that would undermine his position. See, e.g., Tr. 60:9-63:2 (p.m.). Aldridge, however, addresses the use of learned treatises in the direct examination, and it cites with approval the case of Cummings v. Borough of Nazareth, 242 A.2d 460, 466 (Pa. 1968) (plurality op.), which stated that “[i]t is entirely proper in examination and cross-examination for counsel to call the witness’s attention to published works on the matter which is the subject of the witness’s testimony.” Janssen should not have been so limited in its cross examination of Dr. Solomon.

In Cummings, the issue revolved around using manuals that contained industry standards for swimming pool construction, not the appropriateness of a learned scientific treatise. Cummings v. Nazareth Borough, 430 Pa. 255, 266-67 (Pa. 1968). The defense motion did not contend that the defense counsel had laid the appropriate foundation for the learned treatise to be used. In any event, the trial judge wrote an opinion on the post-trial motions, in which he did not appear to address the learned treatise issue at all. Pledger v Janssen Pharms, Inc., Phila. Ct. C.P., Op. sur post-trial motions (Aug. 10., 2017) (Djerassi, J.).

The Pennsylvania Supreme Court has addressed the learned treatise exception to the rule against hearsay on several occasions. Perhaps the leading case described the law as:

“well-settled that an expert witness may be cross-examined on the contents of a publication upon which he or she has relied in forming an opinion, and also with respect to any other publication which the expert acknowledges to be a standard work in the field. * * * In such cases, the publication or literature is not admitted for the truth of the matter asserted, but only to challenge the credibility of the witness’ opinion and the weight to be accorded thereto. * * * Learned writings which are offered to prove the truth of the matters therein are hearsay and may not properly be admitted into evidence for consideration by the jury.”

Majdic v. Cincinnati Mach. Co., 537 A. 2d 334, 621-22 (Pa. 1988) (internal citations omitted).

The Law360 report is difficult to assess. Perhaps the reporting by Mr. Fair was non-eponymously unfair? There is no discussion of how the defense had laid its foundation. Perhaps the defense had promised “to connect up” by establishing the foundation of the treatise through a defense expert witness. If there had been a foundation established, or promised to be established, the post-trial motion would have, in the normal course of events, cited the transcript for the proffer of a foundation. And why did Mr. Fair report on the oral argument as though the learned treatise issue was the only issue before the court? Inquiring minds want to know.

Judge Djerassi’s opinion on post-trial motions was perhaps more notable for embracing some testimony on statistical significance from Dr. David Kessler, former Commissioner of the FDA, and now a frequent testifier for the lawsuit industry on regulatory matters. Judge Djerassi, in his opinion, stated:

“This statistically significant measure is shown in Table 21 and was within a chi-square rate of .02, meaning within a 98% chance of certainty. In Dr. Kessler’s opinion this is a statistically significant finding. (N.T. 1/29/15, afternoon, pp. p. 27, ln. 2 10-11, p. 28, lns. 7-12).”

Post-trial opinion at p.11.2 Surely, the defense’s expert witnesses explained that the chi-square test did not yield a measure of certainty that the measured statistic was the correct value.

The trial court’s whopper was enough of a teaser to force me to track down Kessler’s testimony, which was posted to the internet by the plaintiffs’ law firm. Judge Djerassi’s erroneous interpretation of the p-value can indeed be traced to Kessler’s improvident testimony:

Q. And since 2003, what have you been doing at University of California San Francisco, sir?

A. Among other things, I am currently a professor of pediatrics, professor of epidemiology, professor of biostatistics.

Pledger Transcript, Thurs., Jan. 28, 2015, Vol. 3, Morning Session at 111:3-7.

A. What statistical significance means is it’s mathematical and scientific calculations, but when we say something is statistically significant, it’s unlikely to happen by chance. So that association is very likely to be real. If you redid this, general statistically significant says if I redid this and redid the analysis a hundred times, I would get the same result 95 of those times.

Pledger Transcript, Fri., Jan. 29, 2015, Vol. 4, Morning Session at 80:18 – 81:2.

Q. So, sir, if we see on a study — and by the way, do the investigators of a study decided in their own criteria what is statistically significant? Do they assign what’s called a P value?

A. Exactly. So you can set it at 95, you can set it at 98, you can set it at 90. Generally, 95 significance level, for those of you who are mathematicians or scientifically inclined, it’s a P less than .05.

Q. As a general rule?

A. Yes.

Q. So if I see a number that is .0158, next to a dataset, that would mean that it occurs by chance less than two in 100. Correct?

A. Yes, that’s what the P value is saying.

Pledger Transcript, Fri., Jan. 29, 2015, Vol. 4, Morning Session at 81:5-20

Q. … If someone — if something has a p-value of less than .02, the converse of it is that your 98 — .98, that would be 98 percent certain that the result is not by chance?

A. Yes. That’s a fair way of saying it.

Q. And if you have a p-value of .10, that means the converse of it is 90 percent, or 90 percent that it’s not by chance, correct?

A. Yes.

Pledger Transcript, Fri., Jan. 29, 2015, Vol. 4, Afternoon Session at 7:14-22.

Q. Okay. And the last thing I’d like to ask about — sorry to keep going back and forth — is so if the jury saw a .0158, that’s of course less than .02, which means that it is 90 — almost 99 percent not by chance.

A. Yes. It’s statistically significant, as I would call it.

Pledger Transcript, Fri., Jan. 29, 2015, Vol. 4, Afternoon Session at 8:7-13.

2 See also Djerassi opinion at p.13 n. 13 (“P<0.02 is the chi—square rate reflecting a data outcome within a 98% chance of certainty.”).

Posted in Causation, Expert Witnesses, statistical evidence | Comments Off on The Appeal of the Learned Treatise

N.J. Supreme Court Uproots Weeds in Garden State’s Law of Expert Witnesses

August 8th, 2018

The United States Supreme Court’s decision in Daubert is now over 25 years old. The idea of judicial gatekeeping of expert witness opinion testimony is even older in New Jersey state courts. The New Jersey Supreme Court articulated a reliability standard before the Daubert case was even argued in Washington, D.C. See Landrigan v. Celotex Corp., 127 N.J. 404, 414 (1992); Rubanick v. Witco Chem. Corp., 125 N.J. 421, 447 (1991). Articulating a standard, however, is something very different from following a standard, and in many New Jersey trial courts, until very recently, the standard was pretty much anything goes.

One counter-example to the general rule of dog-eat-dog in New Jersey was Judge Nelson Johnson’s careful review and analysis of the proffered causation opinions in cases in which plaintiffs claimed that their use of the anti-acne medication isotretinoin (Accutane) caused Crohn’s disease. Judge Johnson, who sits in the Law Division of the New Jersey Superior Court for Atlantic County held a lengthy hearing, and reviewed the expert witnesses’ reliance materials.1 Judge Johnson found that the plaintiffs’ expert witnesses had employed undue selectivity in choosing what to rely upon. Perhaps even more concerning, Judge Johnson found that these witnesses had refused to rely upon reasonably well-conducted epidemiologic studies, while embracing unpublished, incomplete, and poorly conducted studies and anecdotal evidence. In re Accutane, No. 271(MCL), 2015 WL 753674, 2015 BL 59277 (N.J.Super. Law Div., Atlantic Cty. Feb. 20, 2015). In response, Judge Johnson politely but firmly closed the gate to conclusion-driven duplicitous expert witness causation opinions in over 2,000 personal injury cases. “Johnson of Accutane – Keeping the Gate in the Garden State” (Mar. 28, 2015).

Aside from resolving over 2,000 pending cases, Judge Johnson’s judgment was of intense interest to all who are involved in pharmaceutical and other products liability litigation. Judge Johnson had conducted a pretrial hearing, sometimes called a Kemp hearing in New Jersey, after the New Jersey Supreme Court’s opinion in Kemp v. The State of New Jersey, 174 N.J. 412 (2002). At the hearing and in his opinion that excluded plaintiffs’ expert witnesses’ causation opinions, Judge Johnson demonstrated a remarkable aptitude for analyzing data and inferences in the gatekeeping process.

When the courtroom din quieted, the trial court ruled that the proffered testimony of Dr., Arthur Kornbluth and Dr. David Madigan did not meet the liberal New Jersey test for admissibility. In re Accutane, No. 271(MCL), 2015 WL 753674, 2015 BL 59277 (N.J.Super. Law Div. Atlantic Cty. Feb. 20, 2015). And in closing the gate, Judge Johnson protected the judicial process from several bogus and misleading “lines of evidence,” which have become standard ploys to mislead juries in courthouses where the gatekeepers are asleep. Recognizing that not all evidence is on the same analytical plane, Judge Johnson gave case reports short shrift.

“[u]nsystematic clinical observations or case reports and adverse event reports are at the bottom of the evidence hierarchy.”

Id. at *16. Adverse event reports, largely driven by the very litigation in his courtroom, received little credit and were labeled as “not evidentiary in a court of law.” Id. at 14 (quoting FDA’s description of FAERS).

Judge Johnson recognized that there was a wide range of identified “risk factors” for irritable bowel syndrome, such as prior appendectomy, breast-feeding as an infant, stress, Vitamin D deficiency, tobacco or alcohol use, refined sugars, dietary animal fat, fast food. In re Accutane, 2015 WL 753674, at *9. The court also noted that there were four medications generally acknowledged to be potential risk factors for inflammatory bowel disease: aspirin, nonsteroidal anti-inflammatory medications (NSAIDs), oral contraceptives, and antibiotics. Understandably, Judge Johnson was concerned that the plaintiffs’ expert witnesses preferred studies unadjusted for potential confounding co-variables and studies that had involved “cherry picking the subjects.” Id. at *18.

Judge Johnson had found that both sides in the isotretinoin cases conceded the relative unimportance of animal studies, but the plaintiffs’ expert witnesses nonetheless invoked the animal studies in the face of the artificial absence of epidemiologic studies that had been created by their cherry-picking strategies. Id.

Plaintiffs’ expert witnesses had reprised a common claimants’ strategy; namely, they claimed that all the epidemiology studies lacked statistical power. Their arguments often ignored that statistical power calculations depend upon statistical significance, a concept to which many plaintiffs’ counsel have virulent antibodies, as well as an arbitrarily selected alternative hypothesis of association size. Furthermore, the plaintiffs’ arguments ignored the actual point estimates, most of which were favorable to the defense, and the observed confidence intervals, most of which were reasonably narrow.

The defense responded to the bogus statistical arguments by presenting an extremely capable clinical and statistical expert witness, Dr. Stephen Goodman, to present a meta-analysis of the available epidemiologic evidence.

Meta-analysis has become an important facet of pharmaceutical and other products liability litigation[1]. Fortunately for Judge Johnson, he had before him an extremely capable expert witness, Dr. Stephen Goodman, to explain meta-analysis generally, and two meta-analyses he had performed on isotretinoin and irritable bowel outcomes.

Dr. Goodman explained that the plaintiffs’ witnesses’ failure to perform a meta-analysis was telling when meta-analysis can obviate the plaintiffs’ hyperbolic statistical complaints:

“the strength of the meta-analysis is that no one feature, no one study, is determinant. You don’t throw out evidence except when you absolutely have to.”

In re Accutane, 2015 WL 753674, at *8.

Judge Johnson’s judicial handiwork received non-deferential appellate review from a three-judge panel of the Appellate Division, which reversed the exclusion of Kornbluth and Madigan. In re Accutane Litig., 451 N.J. Super. 153, 165 A.3d 832 (App. Div. 2017). The New Jersey Supreme Court granted the isotretinoin defendants’ petition for appellate review, and the issues were joined over the appropriate standard of appellate review for expert witness opinion exclusions, and the appropriateness of Judge Johnson’s exclusions of Kornbluth and Madigan. A bevy of amici curiae joined in the fray.2

Last week, the New Jersey Supreme Court issued a unanimous opinion, which reversed the Appellate Division’s holding that Judge Johnson had “mistakenly exercised” discretion. Applying its own precedents from Rubanick, Landrigan, and Kemp, and the established abuse-of-discretion standard, the Court concluded that the trial court’s ruling to exclude Kornbluth and Madigan was “unassailable.” In re Accutane Litig., ___ N.J. ___, 2018 WL 3636867 (2018), Slip op. at 79.3

The high court graciously acknowledged that defendants and amici had “good reason” to seek clarification of New Jersey law. Slip op. at 67. In abandoning abuse-of-discretion as its standard of review, the Appellate Division had relied upon a criminal case that involved the application of the Frye standard, which is applied as a matter of law. Id. at 70-71. The high court also appeared to welcome the opportunity to grant review and reverse the intermediate court reinforce “the rigor expected of the trial court” in its gatekeeping role. Id. at 67. The Supreme Court, however, did not articulate a new standard; rather it demonstrated at length that Judge Johnson had appropriately applied the legal standards that had been previously announced in New Jersey Supreme Court cases.4

In attempting to defend the Appellate Division’s decision, plaintiffs sought to characterize New Jersey law as somehow different from, and more “liberal” than, the United States Supreme Court’s decision in Daubert. The New Jersey Supreme Court acknowledged that it had never formally adopted the dicta from Daubert about factors that could be considered in gatekeeping, slip op. at 10, but the Court went on to note what disinterested observers had long understood, that the so-called Daubert factors simply flowed from a requirement of sound methodology, and that there was “little distinction” and “not much light” between the Landrigan and Rubanick principles and the Daubert case or its progeny. Id at 10, 80.

Curiously, the New Jersey Supreme Court announced that the Daubert factors should be incorporated into the New Jersey Rules 702 and 703 and their case law, but it stopped short of declaring New Jersey a “Daubert” jurisdiction. Slip op. at 82. In part, the Court’s hesitance followed from New Jersey’s bifurcation of expert witness standards for civil and criminal cases, with the Frye standard still controlling in the criminal docket. At another level, it makes no sense to describe any jurisdiction as a “Daubert” state because the relevant aspects of the Daubert decision were dicta, and the Daubert decision and its progeny were superseded by the revision of the controlling statute in 2000.5

There were other remarkable aspects of the Supreme Court’s Accutane decision. For instance, the Court put its weight behind the common-sense and accurate interpretation of Sir Austin Bradford Hill’s famous articulation of factors for causal judgment, which requires that sampling error, bias, and confounding be eliminated before assessing whether the observed association is strong, consistent, plausible, and the like. Slip op. at 20 (citing the Reference Manual at 597-99), 78.

The Supreme Court relied extensively on the National Academies’ Reference Manual on Scientific Evidence.6 That reliance is certainly preferable to judicial speculations and fabulations of scientific method. The reliance is also positive, considering that the Court did not look only at the problematic epidemiology chapter, but adverted also to the chapters on statistical evidence and on clinical medicine.

The Supreme Court recognized that the Appellate Division had essentially sanctioned an anything goes abandonment of gatekeeping, an approach that has been all-too-common in some of New Jersey’s lower courts. Contrary to the previously prevailing New Jersey zeitgeist, the Court instructed that gatekeeping must be “rigorous” to “prevent[] the jury’s exposure to unsound science through the compelling voice of an expert.” Slip op. at 68-9.

Not all evidence is equal. “[C]ase reports are at the bottom of the evidence hierarchy.” Slip op. at 73. Extrapolation from non-human animal studies is fraught with external validity problems, and such studies “far less probative in the face of a substantial body of epidemiologic evidence.” Id. at 74 (internal quotations omitted).

Perhaps most chilling for the lawsuit industry will be the Supreme Court’s strident denunciation of expert witnesses’ selectivity in choosing lesser evidence in the face of a large body of epidemiologic evidence, id. at 77, and their unprincipled cherry picking among the extant epidemiologic publications. Like the trial court, the Supreme Court found that the plaintiffs’ expert witnesses’ inconsistent use of methodological criteria and their selective reliance upon studies (disregarding eight of the nine epidemiologic studies) that favored their task masters was the antithesis of sound methodology. Id. at 73, citing with approval, In re Lipitor, ___ F.3d ___ (4th Cir. 2018) (slip op. at 16) (“Result-driven analysis, or cherry-picking, undermines principles of the scientific method and is a quintessential example of applying methodologies (valid or otherwise) in an unreliable fashion.”).

An essential feature of the Supreme Court’s decision is that it was not willing to engage in the common reductionism that has “all epidemiologic studies are flawed,” and which thus privileges cherry picking. Not all disagreements between expert witnesses can be framed as differences in interpretation. In re Accutane will likely stand as a bulwark against flawed expert witness opinion testimony in the Garden State for a long time.

1 Judge Nelson Johnson is also the author of Boardwalk Empire: The Birth, High Times, and Corruption of Atlantic City (2010), a spell-binding historical novel about political and personal corruption.

2 In support of the defendants’ positions, amicus briefs were filed by the New Jersey Business & Industry Association, Commerce and Industry Association of New Jersey, and New Jersey Chamber of Commerce; by law professors Kenneth S. Broun, Daniel J. Capra, Joanne A. Epps, David L. Faigman, Laird Kirkpatrick, Michael M. Martin, Liesa Richter, and Stephen A. Saltzburg; by medical associations the American Medical Association, Medical Society of New Jersey, American Academy of Dermatology, Society for Investigative Dermatology, American Acne and Rosacea Society, and Dermatological Society of New Jersey, by the Defense Research Institute; by the Pharmaceutical Research and Manufacturers of America; and by New Jersey Civil Justice Institute. In support of the plaintiffs’ position and the intermediate appellate court’s determination, amicus briefs were filed by political action committee the New Jersey Association for Justice; by the Ironbound Community Corporation; and by plaintiffs’ lawyer Allan Kanner.

3 Nothing in the intervening scientific record called question upon Judge Johnson’s trial court judgment. See, e.g., I.A. Vallerand, R.T. Lewinson, M.S. Farris, C.D. Sibley, M.L. Ramien, A.G.M. Bulloch, and S.B. Patten, “Efficacy and adverse events of oral isotretinoin for acne: a systematic review,” 178 Brit. J. Dermatol. 76 (2018).

4 Slip op. at 9, 14-15, citing Landrigan v. Celotex Corp., 127 N.J. 404, 414 (1992); Rubanick v. Witco Chem. Corp., 125 N.J. 421, 447 (1991) (“We initially took that step to allow the parties in toxic tort civil matters to present novel scientific evidence of causation if, after the trial court engages in rigorous gatekeeping when reviewing for reliability, the proponent persuades the court of the soundness of the expert’s reasoning.”).

5 The Court did acknowledge that Federal Rule of Evidence 702 had been amended in 2000, to reflect the Supreme Court’s decision in Daubert, Joiner, and Kumho Tire, but the Court did not deal with the inconsistencies between the present rule and the 1993 Daubert case. Slip op. at 64, citing Calhoun v. Yamaha Motor Corp., U.S.A., 350 F.3d 316, 320-21, 320 n.8 (3d Cir. 2003).

6 See Accutane slip op. at 12-18, 24, 73-74, 77-78. With respect to meta-analysis, the Reference Manual’s epidemiology chapter is still stuck in the 1980s and the prevalent resistance to poorly conducted, often meaningless meta-analyses. See “The Treatment of Meta-Analysis in the Third Edition of the Reference Manual on Scientific Evidence” (Nov. 14, 2011) (The Reference Manual fails to come to grips with the prevalence and importance of meta-analysis in litigation, and fails to provide meaningful guidance to trial judges).

Posted in Evidence, Expert Witnesses, Frye, Meta-analysis, Reference Manual on Scientific Evidence, Rule 702, Scientific Evidence, statistical evidence, Statistical Power | Comments Off on N.J. Supreme Court Uproots Weeds in Garden State’s Law of Expert Witnesses

P-Values: Pernicious or Perspicacious?

May 12th, 2018

Professor Kingsley R. Browne, of the Wayne State University Law School, recently published a paper that criticized the use of p-values and significance testing in discrimination litigation. Kingsley R. Browne, “Pernicious P-Values: Statistical Proof of Not Very Much,” 42 Univ. Dayton L. Rev. 113 (2017) (cited below as Browne). Browne amply documents the obvious and undeniable, that judges, lawyers, and even some ill-trained expert witnesses, are congenitally unable to describe and interpret p-values properly. Most of Browne’s examples are from the world of anti-discrimination law, but he also cites a few from health effects litigation as well. Browne also cites from many of the criticisms of p-values in the psychology and other social science literature.

Browne’s efforts to correct judicial innumeracy are welcomed, but they take a peculiar turn in this law review article. From the well-known state of affairs of widespread judicial refusal or inability to discuss statistical concepts accurately, Browne argues for what seem to be two incongruous, inconsistent responses. Rejecting the glib suggestion of former Judge Posner that evidence law is not “fussy” about evidence, Browne argues that federal evidence law requires courts to be “fussy” about evidence, and that Rule 702 requires courts to exclude expert witnesses, whose opinions fail to “employ[] in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field.” Browne at 143 (quoting from Kumho Tire Co. v. Carmichael, 526 U.S. 137, 152 (1999). Browne tells us, with apparently appropriate intellectual rigor, that “[i]f a disparity that does not provide a p-value of less than 0.05 would not be accepted as meaningful in the expert’s discipline, it is not clear that the expert should be allowed to testify – on the basis of his expertise in that discipline – that the disparity is, in fact, meaningful.” Id.

In a volte face, Browne then argues that p-values do “not tell us much,” basically because they are dependent upon sample size. Browne suggests that the quantitative disparity between expected value and observed proportion or average can be assessed without the use of p-values, and that measuring a p-value “adds virtually nothing and just muddies the water.” Id. at 152. The prevalent confusion among judges and lawyers seems sufficient in Browne’s view to justify his proposal, as well as his further suggestion that Rule 403 should be invoked to exclude p-values:

“The ease with which reported p-values cause a trier of fact to slip into the transposition fallacy and the difficulty of avoiding that lapse of logic, coupled with the relatively sparse information actually provided by the p-value, make p-values prime candidates for exclusion under Federal Rule of Evidence 403. *** If judges, not to mention the statistical experts they rely on, cannot use the information without falling into fallacious reasoning, the likelihood that the jury will misunderstand the evidence is very high. Since the p-value actually provides little useful relevant information, the high risk of misleading the jury greatly exceeds its scant probative value, so it simply should not be presented to the jury.”

Id. at 152-53.

And yet, elsewhere in the same article, Browne ridicules one court and several expert witnesses who have argued in favor of conclusions that were based upon p-values up to 50%.1 The concept of p-values cannot be so flexible as to straddle the extremes of having no probative value, and yet capable of rendering an expert witness’s opinions ludicrous. P-values quantify an estimate of random error, even if that error rate varies with sample size. To be sure, the measure of random error depends upon the specified model and assumption of a null hypothesis, but the crucial point is that the estimate (whether mean, proportion, risk ratio, risk difference, etc.) is rather meaningless without some further estimate of random variability of the estimate. Of course, random error is not the only type of error, but the existence of other potential systematic errors is hardly a reason to ignore random error.

In the science of health effects, many applications of p-values have given way to the use of confidence intervals, which arguably provide more direct assessments of both sample estimates, along with ranges of potential outcomes that are reasonably compatible with the sample estimates. Remarkably, Browne never substantively discusses confidence intervals in his article.

Under the heading of other problems with p-values and significance testing, Browne advances four additional putative problems with p-values. First, Browne asserts with little to no support that “[t]he null hypothesis is unlikely a priori.” Id. at 155. He fails to tell us why the null hypothesis of no disparity is not a reasonable starting place in the absence of objective evidence of a prior estimate. Furthermore, a null hypothesis of no difference will have legal significance in claims of health effects, or of unlawful discrimination.

Second, Browne argues that significance testing will lead to “[c]onflation of statistical and practical (or legal) significance” in the minds of judges and jurors. Id. at 156-58. This charge is difficult to sustain. The actors in legal cases can probably best appreciate practical significance and its separation from statistical significance, most readily. If a large class action showed that the expected value of a minority’s proportion was 15%, and the observed proportion was 14.8%, p < 0.05, most innumerate judges and jurors would sense that this disparity was unimportant and that no employer would fine tune its discriminatory activities so closely as to achieve such a meaningless difference.

Third, Browne reminds us that the validity and the interpretation of a p-value turns on the assumption that the statistical model is perfectly specified. Id. at 158-59. His reminder is correct, but again, this aspect of p-values (or confidence intervals) is relatively easy to explain, as well as to defend or challenge. To be sure, there may be legitimate disputes about whether an appropriate model was used (say binomial versus hypergeometric), but such disputes are hardly the most arcane issues that judges and jurors will face.

Fourth, Browne claims that “the alternative hypothesis is seldom properly specified.” Id. at 159-62. Unless analysts are focused on measuring pre-test power or type II error, however, they need not advance an alternative hypothesis. Furthermore, it is hardly a flaw with significance testing that it does not account for systematic bias or confounding.

Browne does not offer an affirmative response such as urging courts to adopt a Bayesian program. A Bayesian response to prevalent blunders in interpreting statistical significance would introduce perhaps even more arcane and hard-to-discern blunders in court proceedings. Browne also leaves courts without a meaningful approach to evaluate random error other than to engage in crude comparisons between two means or proportions. The recommendations in this law review article appear to be a giant step, backwards, into an epistemic void.

1See Browne at 146, citing In re Photochromic Lens Antitrust Litig., 2014 WL 1338605 (M.D. Fla. April 3, 2014) (reversing magistrate judge’s exclusion of an expert witness who had advanced claims based upon p-value of 0.50); id. at 147 n. 116, citing In re High-Tech Employee Antitrust Litig., 2014 WL 1351040 (N.D. Cal. 2014).

Posted in Evidence, Rule 702, statistical evidence | Comments Off on P-Values: Pernicious or Perspicacious?

Statistical Deontology

March 2nd, 2018

In courtrooms across America, there has been a lot of buzzing and palavering about the American Statistical Association’s Statement on Statistical Significance Testing,1 but very little discussion of the Society’s Ethical Guidelines, which were updated and promulgated in the same year, 2016. Statisticians and statistics, like lawyers and the law, receive their fair share of calumny over their professional activities, but the statistician’s principal North American professional organization is trying to do something about members’ transgressions.

The American Statistical Society (ASA) has promulgated ethical guidelines for statisticians, as has the Royal Statistical Society,2 even if these organizations lack the means and procedures to enforce their codes. The ASA’s guidelines3 are rich with implications for statistical analyses put forward in all contexts, including in litigation and regulatory rule making. As such, the guidelines are well worth studying by lawyers.

The ASA Guidelines were prepared by the Committee on Professional Ethics, and approved by the ASA’s Board in April 2016. There are lots of “thou shall” and “thou shall nots,” but I will focus on the issues that are more likely to arise in litigation. What is remarkable about the Guidelines is that if followed, they probably are more likely to eliminate unsound statistical practices in the courtroom than the ASA State on P-values.

Defining Good Statistical Practice

“Good statistical practice is fundamentally based on transparent assumptions, reproducible results, and valid interpretations.” Guidelines at 1. The Guidelines thus incorporate something akin to the Kumho Tire standard that an expert witness ‘‘employs in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field.’’ Kumho Tire Co. v. Carmichael, 526 U.S. 137, 152 (1999).

A statistician engaged in expert witness testimony should provide “only expert testimony, written work, and oral presentations that he/she would be willing to have peer reviewed.” Guidelines at 2. “The ethical statistician uses methodology and data that are relevant and appropriate, without favoritism or prejudice, and in a manner intended to produce valid, interpretable, and reproducible results.” Id. Similarly, the statistician, if ethical, will identify and mitigate biases, and use analyses “appropriate and valid for the specific question to be addressed, so that results extend beyond the sample to a population relevant to the objectives with minimal error under reasonable assumptions.” Id. If the Guidelines were followed, a lot of spurious analyses would drop off the litigation landscape, regardless whether they used p-values or confidence intervals, or a Bayesian approach.

Integrity of Data and Methods

The ASA’s Guidelines also have a good deal to say about data integrity and statistical methods. In particular, the Guidelines call for candor about limitations in the statistical methods or the integrity of the underlying data:

“The ethical statistician is candid about any known or suspected limitations, defects, or biases in the data that may impact the integrity or reliability of the statistical analysis. Objective and valid interpretation of the results requires that the underlying analysis recognizes and acknowledges the degree of reliability and integrity of the data.”

Guidelines at 3.

The statistical analyst openly acknowledges the limits of statistical inference, the potential sources of error, as well as the statistical and substantive assumptions made in the execution and interpretation of any analysis,” including data editing and imputation. Id. The Guidelines urge analysts to address potential confounding not assessed by the study design. Id. at 3, 10. How often do we see these acknowledgments in litigation-driven analyses, or in peer-reviewed papers, for that matter?

Affirmative Actions Prescribed

In the aid of promoting data and methodological integrity, the Guidelines also urge analysts to share data when appropriate without revealing the identities of study participants. Statistical analysts should publicly correct any disseminated data and analyses in their own work, as well as working to “expose incompetent or corrupt statistical practice.” Of course, the Lawsuit Industry will call this ethical duty “attacking the messenger,” but maybe that’s a rhetorical strategy based upon an assessment of risks versus benefits to the Lawsuit Industry.

Multiplicity

The ASA Guidelines address the impropriety of substantive statistical errors, such as:

“[r]unning multiple tests on the same data set at the same stage of an analysis increases the chance of obtaining at least one invalid result. Selecting the one “significant” result from a multiplicity of parallel tests poses a grave risk of an incorrect conclusion. Failure to disclose the full extent of tests and their results in such a case would be highly misleading.”

Guidelines at 9.

There are some Lawsuit Industrialists who have taken comfort in the pronouncements of Kenneth Rothman on corrections for multiple comparisons. Rothman’s views on multiple comparisons are, however, much broader and more nuanced than the Industry’s sound bites.4 Given that Rothman opposes anything like strict statistical significance testing, it follows that he is relatively unmoved for the need for adjustments to alpha or the coefficient of confidence. Rothman, however, has never deprecated the need to consider the multiplicity of testing, and the need for researchers to be forthright in disclosing the the scope of comparisons originally planned and actually done.

2 Royal Statistical Society – Code of Conduct (2014); Steven Piantadosi, Clinical Trials: A Methodologic Perspective 609 (2d ed. 2005).

3 Shelley Hurwitz & John S. Gardenier, “Ethical Guidelines for Statistical Practice: The First 60 Years and Beyond,” 66 Am. Statistician 99 (2012) (describing the history and evolution of the Guidelines).

4 Kenneth J. Rothman, “Six Persistent Research Misconceptions,” 29 J. Gen. Intern. Med. 1060, 1063 (2014).

Posted in Data Sharing, Expert Witness Discovery, Scientific Misconduct, statistical evidence, Underlying Data | Comments Off on Statistical Deontology

The 5% Solution at the FDA

February 24th, 2018

The statistics wars rage on1, with Bayesians attempting to take advantage of the so-called replication crisis to argue it is all the fault of frequentist significance testing. In 2016, there was an attempted coup at the American Statistical Association, but the Bayesians did not get what they wanted, with little more than a consensus that p-values and confidence intervals should be properly interpreted. Patient advocacy groups have lobbied for the availability of unapproved and incompletely tested medications, and rent-seeking litigation has argued and lobbied for the elimination of statistical tests and methods in the assessment of causal claims. The battle continues.

Against this backdrop, a young Harvard graduate student has published a a paper with a brief history of significance testing, and the role that significance testing has taken on at the United States Food and Drug Administration (FDA). Lee Kennedy-Shaffer, “When the Alpha is the Omega: P-Values, ‘Substantial Evidence’, and the 0.05 Standard at FDA,” 72 Food & Drug L.J. 595 (2017) [cited below as K-S]. The paper presents a short but entertaining history of the evolution of the p-value from its early invocation in 1710, by John Arbuthnott, a Scottish physician and mathematician, who calculated the probability that male births would exceed female births 82 consecutive years if their true proportions were equal. K-S at 603. Kennedy-Shaffer notes the role of the two great French mathematicians, Pierre-Simon Laplace and Siméon-Denis Poisson, who used p-values (or their complements) to evaluate empirical propositions. As Kennedy-Shaffer notes, Poisson observed that the equivalent of what would be a modern p-value about 0.005, was sufficient in his view, back in 1830, to believe that the French Revolution of 1830 had caused the pattern of jury verdicts to be changed. K-S at 604.

Kennedy-Shaffer traces the p-value, or its equivalent, through its treatment by the great early 20th century statisticians, Karl Pearson and Ronald A. Fisher, through its modification by Jerzy Neyman and Egon Pearson, into the bowels of the FDA in Rockville, Maryland. It is a history well worth recounting, if for no other reason, to remind us that the p-value or its equivalent has been remarkably durable and reasonably effective in protecting the public against false claims of safety and efficacy. Kennedy-Shaffer provides several good examples in which the FDA’s use of significance testing was outcome dispositive of approval or non-approval of medications and devices.

There is enough substance and history here that everyone will have something to pick at this paper. Let me volunteer the first shot. Kennedy-Shaffer describes the co-evolution of the controlled clinical trial and statistical tests, and points to the landmark study by the Medical Research Council on streptomycin for tuberculosis. Geoffrey Marshall (chairman), “Streptomycin Treatment of Pulmonary Tuberculosis: A Medical Research Council Investigation,” 2 Brit. Med. J. 769, 769–71 (1948). This clinical trial was historically important, not only for its results and for Sir Austin Bradford Hill’s role in its design, but for the care with which it described randomization, double blinding, and multiple study sites. Kennedy-Shaffer suggests that “[w]hile results were presented in detail, few formal statistical tests were incorporated into this analysis.” K-S at 597-98. And yet, a few pages later, he tells us that “both chi-squared tests and t-tests were used to evaluate the responses to the drug and compare the control and treated groups,” and that “[t]he difference in mortality between the two groups is statistically significant.” K-S at 611. Although it is true that the authors did not report their calculated p-values for any test, the difference in mortality between the streptomycin and control groups was very large, and the standards for describing the results of such a clinical trial were in their infancy in 1948.

Kennedy-Shaffer’s characterization of Sir Austin Bradford Hill’s use of statistical tests and methods takes on out-size importance because of the mischaracterizations, and even misrepresentations, made by some representatives of the Lawsuit Industry, who contend that Sir Austin dismissed statistical methods as unnecessary. In the United States, some judges have been seriously misled by those misrepresentations, which have their way into published judicial decisions.

The operative document, of course, is the publication of Sir Austin’s famous after-dinner speech, in 1965, on the occasion of his election to the Presidency of the Royal Society of Medicine. Although the speech is casual and free of scholarly footnotes, Sir Austin’s message was precise, balanced, and nuanced. The speech is a classic in the history of medicine, which remains important even if rather dated in terms of its primary message about how science and medicine move from beliefs about associations to knowledge of causal associations. As everyone knows, Sir Austin articulated nine factors or viewpoints through which to assess any putative causal association, but he emphasized that before these nine factors are assessed, our starting point itself has prerequisites:

“Disregarding then any such problem in semantics we have this situation. Our observations reveal an association between two variables, perfectly clear-cut and beyond what we would care to attribute to the play of chance. What aspects of that association should we especially consider before deciding that the most likely interpretation of it is causation?”

Austin Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295, 295 (1965) [cited below as Hill]. The starting point, therefore, before the Bradford Hill nine factors come into play, is a “clear-cut” association, which is “beyond what we would care to attribute to the play of chance.”

In other words, consideration of random error is necessary.

Now for the nuance and the balance. Sir Austin acknowledged that there were some situations in which we simply do not need to calculate standard errors because the disparity between treatment and control groups is so large and meaningful. He goes on to wonder out loud:

“whether the pendulum has not swung too far – not only with the attentive pupils but even with the statisticians themselves. To decline to draw conclusions without standard errors can surely be just as silly? Fortunately I believe we have not yet gone so far as our friends in the USA where, I am told, some editors of journals will return an article because tests of significance have not been applied. Yet there are innumerable situations in which they are totally unnecessary – because the difference is grotesquely obvious, because it is negligible, or because, whether it be formally significant or not, it is too small to be of any practical importance. What is worse the glitter of the t table diverts attention from the inadequacies of the fare.”

Hill at 299. Now this is all true, but hardly the repudiation of statistical testing claimed by those who want to suppress the consideration of random error from science and judicial gatekeeping. There are very few litigation cases in which the difference between exposed and unexposed is “grotesquely obvious,” such that we can leave statistical methods at the door. Importantly, the very large differences between the streptomycin and placebo control groups in the Medical Council’s 1948 clinical trial were not so “grotesquely obvious” that statistical methods were obviated. To be fair, the differences were sufficiently great that statistical discussion could be kept to a minimum. Sir Austin gave extensive tables in the 1948 paper to let the reader appreciate the actual data themselves.

In his after-dinner speech, Hill also gives examples of studies that are so biased and confounded that no statistical method will likely ever save them. Certainly, the technology of regression and propensity-score analyses have progressed tremendously since Hill’s 1965 speech, but his point still remains. This point hardly excuses the lack of statistical apparatus in highly confounding or biased observations.

In addressing the nine factors he identified, which presumed a “clear-cut” association, with random error ruled out, Sir Austin did opine that for the factors raised questions and that:

“No formal tests of significance can answer those questions. Such tests can, and should, remind us of the effects that the play of chance can create, and they will instruct us in the likely magnitude of those effects. Beyond that they contribute nothing to the ‘proof’ of our hypothesis.”

Hill at 299. Again, the date and the context are important. Hill is addressing consideration of the nine factors, not the required predicate association beyond the play of chance or random error. The date is important as well, because it would be foolish to suggest that statistical methods have not grown in the last half century to address some of the nine factors. The existence and the nature of dose-response are the subject of extensive statistical methods, and meta-analysis and meta-regression are used to assess and measure consistency between studies.

Kennedy-Shaffer might well have pointed out the great influence Sir Austin’s textbook on medical statistics had had on medical research and practice. This textbook, which went through numerous editions, makes clear the importance of statistical testing and methods:

“Are simple methods of the interpretation of figures only a synonym for common sense or do they involve an art or knowledge which can be imparted? Familiarity with medical statistics leads inevitably to the conclusion that common sense is not enough. Mistakes which when pointed out look extremely foolish are quite frequently made by intelligent persons, and the same mistakes, or types of mistakes, crop up again and again. There is often lacking what has been called a ‘statistical tact, which is rather more than simple good sense’. That fact the majority of persons must acquire (with a minority it is undoubtedly innate) by a study of the basic principles of statistical method.”

Austin Bradford Hill, Principles of Medical Statistics at 2 (4th ed. 1948) (emphasis in original). And later in his text, Sir Austin notes that:

“The statistical method is required in the interpretation of figures which are at the mercy of numerous influences, and its object is to determine whether individual influences can be isolated and their effects measured.”

Id. at 10 (emphasis added).

Sir Austin’s work taken as a whole demonstrates the acceptance of the necessity of statistical methods in medicine, and causal inference. Kennedy-Shaffer’s paper covers much ground, but it short changes this important line of influence, which lies directly in the historical path between Sir Ronald Fisher and the medical regulatory community.

Kennedy-Shaffer gives a nod to Bayesian methods, and even suggests that Bayesian results are “more intuitive,” but he does not explain the supposed intuitiveness of how a parameter has a probability distribution. This might make sense at the level of quantum physics, but does not seem to describe the reality of a biomedical phenomenon such as relative risk. Kennedy-Shaffer notes the FDA’s expression of willingness to entertain Bayesian analyses of clinical trials, and the rare instances in which such analyses have actually been deployed. K-S at 629 (“e.g., Pravigard Pac for prevention of myocardial infarction”). He concedes, however, that Bayesian designs are still the exception to the rule, as well as the cautions of Robert Temple, a former FDA Director of Medical Policy, in 2005, that Bayesian proposals for drug clinical trials were at that time “very rare.2” K-S at 630.

2 Robert Temple, “How FDA Currently Makes Decisions on Clinical Studies,” 2 Clinical Trials 276, 281 (2005).

Posted in statistical evidence | Comments Off on The 5% Solution at the FDA

Scientific Evidence in Canadian Courts

February 20th, 2018

A couple of years ago, Deborah Mayo called my attention to the Canadian version of the Reference Manual on Scientific Evidence.1 In the course of discussion of mistaken definitions and uses of p-values, confidence intervals, and significance testing, Sander Greenland pointed to some dubious pronouncements in the Science Manual for Canadian Judges [Manual].

Unlike the United States federal court Reference Manual, which is published through a joint effort of the National Academies of Science, Engineering, and Medicine, the Canadian version, is the product of the Canadian National Judicial Institute (NJI, or the Institut National de la Magistrature, if you live in Quebec), which claims to be an independent, not-for-profit group, that is committed to educating Canadian judges. In addition to the Manual, the Institute publishes Model Jury Instructions and a guide, Problem Solving in Canada’s Courtrooms: A Guide to Therapeutic Justice (2d ed.), as well as conducting educational courses.

The NJI’s website describes the Instute’s Manual as follows:

“Without the proper tools, the justice system can be vulnerable to unreliable expert scientific evidence.

         * * *

The goal of the Science Manual is to provide judges with tools to better understand expert evidence and to assess the validity of purportedly scientific evidence presented to them. …”

The Chief Justice of Canada, Hon. Beverley M. McLachlin, contributed an introduction to the Manual, which was notable for its frank admission that:

“[w]ithout the proper tools, the justice system is vulnerable to unreliable expert scientific evidence.

****

Within the increasingly science-rich culture of the courtroom, the judiciary needs to discern ‘good’ science from ‘bad’ science, in order to assess expert evidence effectively and establish a proper threshold for admissibility. Judicial education in science, the scientific method, and technology is essential to ensure that judges are capable of dealing with scientific evidence, and to counterbalance the discomfort of jurists confronted with this specific subject matter.”

Manual at 14. These are laudable goals, indeed, but did the National Judicial Institute live up to its stated goals, or did it leave Canadian judges vulnerable to the Institute’s own “bad science”?

In his comments on Deborah Mayo’s blog, Greenland noted some rather cavalier statements in Chapter two that suggest that the conventional alpha of 5% corresponds to a “scientific attitude that unless we are 95% sure the null hypothesis is false, we provisionally accept it.” And he, pointed elsewhere where the chapter seems to suggest that the coefficient of confidence that corresponds to an alpha of 5% “constitutes a rather high standard of proof,” thus confusing and conflating probability of random error with posterior probabilities. Greenland is absolutely correct that the Manual does a rather miserable job of educating Canadian judges if our standard for its work product is accuracy and truth.

Some of the most egregious errors are within what is perhaps the most important chapter of the Manual, Chapter 2, “Science and the Scientific Method.” The chapter has two authors, a scientist, Scott Findlay, and a lawyer, Nathalie Chalifour. Findlay is an Associate Professor, in the Department of Biology, of the University of Ottawa. Nathalie Chalifour is an Associate Professor on the Faculty of Law, also in the University of Ottawa. Together, they produced some dubious pronouncements, such as:

Weight of the Evidence (WOE)

“First, the concept of weight of evidence in science is similar in many respects to its legal counterpart. In both settings, the outcome of a weight-of-evidence assessment by the trier of fact is a binary decision.”

Manual at 40. Findlay and Chalifour cite no support for their characterization of WOE in science. Most attempts to invoke WOE are woefully vague and amorphous, with no meaningful guidance or content.2  Sixty-five pages later, if any one is noticing, the authors let us in a dirty, little secret:

“at present, there exists no established prescriptive methodology for weight of evidence assessment in science.”

Manual at 105. The authors omit, however, that there are prescriptive methods for inferring causation in science; you just will not see them in discussions of weight of the evidence. The authors then compound the semantic and conceptual problems by stating that “in a civil proceeding, if the evidence adduced by the plaintiff is weightier than that brought forth by the defendant, a judge is obliged to find in favour of the plaintiff.” Manual at 41. This is a remarkable suggestion, which implies that if the plaintiff adduces the crummiest crumb of evidence, a mere peppercorn on the scales of justice, but the defendant has none to offer, that the plaintiff must win. The plaintiff wins notwithstanding that no reasonable person could believe that the plaintiff’s claims are more likely than not true. Even if there were the law of Canada, it is certainly not how scientists think about establishing the truth of empirical propositions.

Confusion of Hypothesis Testing with “Beyond a Reasonable Doubt”

The authors’ next assault comes in conflating significance probability with the probability connected with the burden of proof, a posterior probability. Legal proceedings have a defined burden of proof, with criminal cases requiring the state to prove guilt “beyond a reasonable doubt.” Findlay and Chalifour’s discussion then runs off the rails by likening hypothesis testing, with an alpha of 5% or its complement, 95%, as a coefficient of confidence, to a “very high” burden of proof:

“In statistical hypothesis-testing – one of the tools commonly employed by scientists – the predisposition is that there is a particular hypothesis (the null hypothesis) that is assumed to be true unless sufficient evidence is adduced to overturn it. But in statistical hypothesis-testing, the standard of proof has traditionally been set very high such that, in general, scientists will only (provisionally) reject the null hypothesis if they are at least 95% sure it is false. Third, in both scientific and legal proceedings, the setting of the predisposition and the associated standard of proof are purely normative decisions, based ultimately on the perceived consequences of an error in inference.”

Manual at 41. This is, as Greenland and many others have pointed out, a totally bogus conception of hypothesis testing, and an utterly false description of the probabilities involved.

Later in the chapter, Findlay and Chalifour flirt with the truth, but then lapse into an unrecognizable parody of it:

“Inferential statistics adopt the frequentist view of probability whereby a proposition is either true or false, and the task at hand is to estimate the probability of getting results as discrepant or more discrepant than those observed, given the null hypothesis. Thus, in statistical hypothesis testing, the usual inferred conclusion is either that the null is true (or rather, that we have insufficient evidence to reject it) or it is false (in which case we reject it). 16 The decision to reject or not is based on the value of p if the estimated value of p is below some threshold value a, we reject the null; otherwise we accept it.”

Manual at 74. OK; so far so good, but here comes the train wreck:

“By convention (and by convention only), scientists tend to set α = 0.05; this corresponds to the collective – and, one assumes, consensual – scientific attitude that unless we are 95% sure the null hypothesis is false, we provisionally accept it. It is partly because of this that scientists have the reputation of being a notoriously conservative lot, given that a 95% threshold constitutes a rather high standard of proof.”

Manual at 75. Uggh; so we are back to significance probability’s being a posterior probability. As if to atone for their sins, in the very next paragraph, the authors then remind the judicial readers that:

“As noted above, p is the probability of obtaining results at least as discrepant as those observed if the null is true. This is not the same as the probability of the null hypothesis being true, given the results.”

Manual at 75. True, true, and completely at odds with what the authors have stated previously. And to add to the reader’s now fully justified conclusion, the authors describe the standard for rejecting the null hypothesis as “very high indeed.” Manual at 102, 109. Any reader who is following the discussion might wonder how and why there is such a problem of replication and reproducibility in contemporary science.

Conflating Bayesianism with Frequentist Modes of Inference

We have seen how Findlay and Chalifour conflate significance and posterior probabilities, some of the time. In a section of their chapter that deals explicitly with probability, the authors tell us that before any study is conducted the prior probability of the truth of the tested hypothesis is 50%, sans evidence. This an astonishing creation of certainty out nothingness, and perhaps it explains the authors’ implied claim that the crummiest morsel of evidence on one side is sufficient to compel a verdict, if the other side has no morsels at all. Here is how the authors put their claim to the Canadian judges:

“Before each study is conducted (that is, a priori), the hypothesis is as likely to be true as it is to be false. Once the results are in, we can ask: How likely is it now that the hypothesis is true? In the first study, the low a priori inferential strength of the study design means that this probability will not be much different from the a priori value of 0.5 because any result will be rather equivocal owing to limitations in the experimental design.”

Manual at 64. This implied Bayesian slant, with 50% priors, in the world of science would lead anyone to believe “as many as six impossible things before breakfast,” and many more throughout the day.

Lest you think that the Manual is all rubbish, there are occasional gems of advice to the Canadian judges. The authors admonish the judges to

“be wary of individual ‘statistically significant’ results that are mined from comparatively large numbers of trials or experiments, as the results may be ‘cherry picked’ from a larger set of experiments or studies that yielded mostly negative results. The court might ask the expert how many other trials or experiments testing the same hypothesis he or she is aware of, and to describe the outcome of those studies.”

Manual at 87. Good advice, but at odds with the authors’ characterization of statistical significance as establishing the rejection of the null hypothesis well-nigh beyond a reasonable doubt.

When Greenland first called attention to this Manual, I reached to some people who had been involved in its peer review. One reviewer told me that it was a “living document,” and would likely be revised after he had the chance to call the NJI’s attention to the errors. But two years later, the errors remain, and so we have to infer that the authors meant to say all the contradictory and false statements that are still present in the downloadable version of the Manual.

2 See “WOE-fully Inadequate Methodology – An Ipse Dixit By Another Name” (May 1, 2012); “Weight of the Evidence in Science and in Law” (July 29, 2017); see also David E. Bernstein, “The Misbegotten Judicial Resistance to the Daubert Revolution,” 89 Notre Dame L. Rev. 27 (2013).

Posted in Reference Manual on Scientific Evidence, Scientific Evidence, statistical evidence | Comments Off on Scientific Evidence in Canadian Courts

Ninth Circuit Quashes Harkonen’s Last Chance

January 8th, 2018

With the benefit of hindsight, even the biggest whopper can be characterized as a strategic choice for trial counsel. As are result of this sort of thinking, the convicted have a very difficult time in pressing claims of ineffective assistance of counsel. After the fact, a reviewing or an appellate court can always imagine a strategic reason for trial counsel’s decisions, even if they contributed to the client’s conviction.

In the Harkonen case, a pharmaceutical executive was indicted and tried for wire fraud and misbranding. His crime was to send out a fax with a preliminary assessment of a recently unblinded clinical trial. In his fax, Dr Harkonen described the trial’s results as “demonstrating” a survival benefit in study participants with mild and moderate disease. Survival (or mortality) was not a primary outcome of the trial, but it was a secondary outcome, and arguably the most important one of all. The subgroup of “mild and moderate” was not pre-specified, but it was highly plausible.

Clearly, Harkonen’s post hoc analysis would not be sufficient normally to persuade the FDA to approve a medication, but Harkonen did not assert or predict that the company would obtain FDA approval. He simply claimed that the trial “demonstrated” a benefit. A charitable interpretation of his statement, which was several pages long, would include the prior successful clinical trial, as important context for Harkonen’s statement.

The United States government, however, was not interested in the principle of charity, the context, or even its own pronouncements on the issue of statistical significance. Instead, the United States Attorney pushed for draconian sentences under the Wire Fraud Act, and the misbranding sections of the Food, Drug, and Cosmetics Act. A jury acquitted on the misbranding charge, but convicted on wire fraud. The government’s request for an extreme prison term and fines was rebuffed by the trial court, which imposed a term of six months of house arrest, and a small fine.1 The conviction, however, effectively keeps Dr Harkonen from working again in the pharmaceutical industry.

In post-verdict challenges to the conviction, Harkonen’s lawyers were able to marshal support from several well-renown statisticians and epidemiologists, but the trial court was reluctant to consider these post-verdict opinions when the defense called no expert witness at trial. The trial situation, however, was complicated and confused by the government’s pre-trial position that it would not call expert witnesses on the statistical and clinical trial interpretative issues. Contrary to these representations, the government called Dr Thomas Fleming, as statistician, who testified at some length, and without objection, to strict criteria for assessing statistical significance and causation in clinical trials.

Having read Fleming’s testimony, I can say that the government got away with introducing a great deal of expert witness opinion testimony, without effective contradiction or impeachment. With the benefit of hindsight, the defense decision not to call an expert witness looks like a serious deviation from the standard of care. Fleming’s “facts” about how the FDA would evaluate the success or failure of the clinical trial were not relevant to whether Harkonen’s claim of a demonstrated benefit were true or false. More importantly, Harkonen’s claim involved an inference, which is not a fact, but an opinion. Fleming’s contrary opinion really did not turn Harkonen’s claim into a falsehood. A contrary rule would have many expert witnesses in civil and in criminal litigation behind bars on similar charges of wire or mail fraud.

After Harkonen exhausted his direct appeals,2 he petitioned for a writ of coram nobis. The trial court denied the petition,3 and in a non-precedential opinion [sic], the Ninth Circuit affirmed the denial of coram nobis.4 United States v. Harkonen, slip op., No. 15-16844 (9th Cir., Dec. 4, 2017) [cited below as Harkonen].

The Circuit rejected Harkonen’s contention that the Supreme Court had announced a new rule with respect to statistical significance, in Matrixx Initiatives, Inc. v. Siracusano, 563 U.S. 27 (2011), which change in law required that his conviction be vacated. Harkonen’s lawyer, like much of the plaintiffs’ tort bar, oversold the Supreme Court’s comments about statistical significance, which were at best dicta, and not very well considered or supported dicta, at that. Still, there was an obvious tension, and duplicity, between positions that the government, through the Solicitor General’s office, had taken in Siracusano, and positions the government took in the Harkonen case.5 Given the government’s opportunistic double-faced arguments about statistical significance, the Ninth Circuit held that Harkonen’s proffered evidence was “compelling, especially in light of Matrixx,” but the panel concluded that his conviction was not the result of a “manifest injustice” that requires the issuance of the writ of coram nobis. Harkonen at 2 (emphasis added). Apparently, Harkonen had suffered an injustice of a less obvious and blatant variety, which did not rise to the level of manifest injustice.

The Ninth Circuit gave similarly short shrift to Harkonen’s challenge to the competency of his counsel. His trial lawyers had averred that they thought that they were doing well enough not to risk putting on an expert witness, especially given that the defense’s view of the evidence came out in the testimony of the government’s witnesses. The Circuit thus acquiesced in the view that both sides had chosen to forgo expert witness testimony, and overlooked the defense’s competency issue for not having objected to Fleming’s opinion trial testimony. Harkonen at 2-4. Remarkably, the appellate court did not look at how Fleming was allowed to testify on statistical issues, without being challenged on cross-examination.

2 United States v. Harkonen, 510 F. App’x 633, 638 (9th Cir. 2013), cert. denied, 134 S. Ct. 824 (2013).

4 Dave Simpson, “9th Circuit Refuses To Rethink Ex-InterMune CEO’s Conviction,” Law360 (Dec. 5, 2017).

Posted in Causation, Scientific Evidence, statistical evidence | Comments Off on Ninth Circuit Quashes Harkonen’s Last Chance

Failed Gatekeeping in Ambrosini v. Labarraque (1996)

December 28th, 2017

The Ambrosini case straddled the Supreme Court’s 1993 Daubert decision. The case began before the Supreme Court clarified the federal standard for expert witness gatekeeping, and ended in the Court of Appeals for the District of Columbia, after the high court adopted the curious notion that scientific claims should be based upon reliable evidence and valid inferences. That notion has only slowly and inconsistently trickled down to the lower courts.

Given that Ambrosini was litigated in the District of Columbia, where the docket is dominated by regulatory controversies, frequently involving dubious scientific claims, no one should be surprised that the D.C. Court of Appeals did not see that the Supreme Court had read “an exacting standard” into Federal Rule of Evidence 702. And so, we see, in Ambrosini, this Court of Appeals citing and purportedly applying its own pre-Daubert decision in Ferebee v. Chevron Chem. Co., 552 F. Supp. 1297 (D.D.C. 1982), aff’d, 736 F.2d 1529 (D.C. Cir.), cert. denied, 469 U.S. 1062 (1984).1 In 2000, the Federal Rule of Evidence 702 was revised in a way that extinguishes the precedential value of Ambrosini and the broad dicta of Ferebee, but some courts and commentators have failed to stay abreast of the law.

Escolastica Ambrosini was using a synthetic progestin birth control, Depo-Provera, as well as an anti-nausea medication, Bendectin, when she became pregnant. The child that resulted from this pregnancy, Teresa Ambrosini, was born with malformations of her face, eyes, and ears, cleft lip and palate, and vetebral malformations. About three percent of all live births in the United States have a major malformation. Perhaps because the Divine Being has sovereign immunity, Escolastica sued the manufacturers of Bendectin and Depo-Provera, as well as the prescribing physician.

The causal claims were controversial when made, and they still are. The progestin at issue, medroxyprogesterone acetate (MPA), was embryotoxic in the cynomolgus monkey2, but not in the baboon3. The evidence in humans was equivocal at best, and involved mostly genital malformations4; the epidemiologic evidence for the MPA causal claim to this day remains unconvincing5.

At the close of discovery in Ambrosini, Upjohn (the manufacturer of the progestin) moved for summary judgment, with a supporting affidavit of a physician and geneticist, Dr. Joe Leigh Simpson. In his affidavit, Simpson discussed three epidemiologic studies, as well as other published papers, in support of his opinion that the progestin at issue did not cause the types of birth defects manifested by Teresa Ambrosini.

Ambrosini had disclosed two expert witnesses, Dr. Allen S. Goldman and Dr. Brian Strom. Neither Goldman nor Strom bothered to identify the papers, studies, data, or methodology used in arriving at an opinion on causation. Not surprisingly, the district judge was unimpressed with their opposition, and granted summary judgment for the defendant. Ambrosini v. Labarraque, 966 F.2d 1462, 1466 (D.C. Cir. 1992).

The plaintiffs appealed on the remarkable ground that Goldman’s and Strom’s crypto-evidence satisfied Federal Rule of Evidence 703. Even more remarkably, the Circuit, in a strikingly unscholarly opinion by Judge Mikva, opined that disclosure of relied-upon studies was not required for expert witnesses under Rules 703 and 705. Judge Mikva seemed to forget that the opinions being challenged were not given in testimony, but in (late-filed) affidavits that had to satisfy the requirement of Federal Rule of Civil Procedure 26. Id. at 1468-69. At trial, an expert witness may express an opinion without identifying its bases, but of course the adverse party may compel disclosure of those bases. In discovery, the proffered expert witness must supply all opinions and evidence relied upon in reach the opinions. In any event, the Circuit remanded the case for a hearing and further proceedings, at which the two challenged expert witnesses, Goldman and Strom, would have to identify the bases of their opinions. Id. at 1471.

Not long after the case landed back in the district court, the Supreme Court decided Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). With an order to produce entered, plaintiffs’ counsel could no longer hide Goldman and Strom’s evidentiary bases, and their scientific inferences came under judicial scrutiny.

Upjohn moved again to exclude Goldman and Strom’s opinions. The district court upheld Upjohn’s challenges, and granted summary judgment in favor of Upjohn for the second time. The Ambrosinis appealed again, but the second case in the D.C. Circuit resulted in a split decision, with the majority holding that the exclusion of Goldman and Strom’s opinions under Rule 702 was erroneous. Ambrosini v. Labarraque, 101 F.3d 129 (D.C. Cir. 1996).

Although issued two decades ago, the majority’s opinion remains noteworthy as an example of judicial resistance to the existence and meaning of the Supreme Court’s Daubert opinion. The majority opinion uncritically cited the notorious Ferebee6 and other pre-Daubert decisions. The court embraced the Daubert dictum about gatekeeping being limited to methodologic consideration, and then proceeded to interpret methodology as superficially as necessary to sustain admissibility. If an expert witness claimed to have looked at epidemiologic studies, and epidemiology was an accepted methodology, then the opinion of the expert witness must satisfy the legal requirements of Daubert, or so it would seem from the opinion of the U.S. Court of Appeals for the District of Columbia.

Despite the majority’s hand waving, a careful reader will discern that there must have been substantial gaps and omissions in the explanations and evidence cited by plaintiffs’ expert witnesses. Seeing anything clearly in the Circuit’s opinion is made difficult, however, by careless and imprecise language, such as its descriptions of studies as showing, or not showing “causation,” when it could have meant only that such studies showed associations, with more or less random and systematic error.

Dr. Strom’s report addressed only general causation, and even so, he apparently did not address general causation of the specific malformations manifested by the plaintiffs’ child. Strom claimed to have relied upon the “totality of the data,” but his methodologic approach seems to have required him to dismiss studies that failed to show an association.

“Dr. Strom first set forth the reasoning he employed that led him to disagree with those studies finding no causal relationship [sic] between progestins and birth defects like Teresa’s. He explained that an epidemiologist evaluates studies based on their ‘statistical power’. Statistical power, he continued, represents the ability of a study, based on its sample size, to detect a causal relationship. Conventionally, in order to be considered meaningful, negative studies, that is, those which allege the absence of a causal relationship, must have at least an 80 to 90 percent chance of detecting a causal link if such a link exists; otherwise, the studies cannot be considered conclusive. Based on sample sizes too small to be reliable, the negative studies at issue, Dr. Strom explained, lacked sufficient statistical power to be considered conclusive.”

Id. at 1367.

Putting aside the problem of suggesting that an observational study detects a “causal relationship,” as opposed to an association in need of further causal evaluation, the Court’s précis of Strom’s testimony on power is troublesome, and typical of how other courts have misunderstood and misapplied the concept of statistical power. Statistical power is a probability of observing an association of a specified size at a specified level of statistical significance. The calculation of statistical power turns indeed on sample size, the level of significance probability preselected for “statistical significance, an assumed probability distribution of the sample, and, critically, an alternative hypothesis. Without a specified alternative hypothesis, the notion of statistical power is meaningless, regardless of what probability (80% or 90% or some other percentage) is sought for finding the alternative hypothesis. Furthermore, the notion that the defense must adduce studies with “sufficient statistical power to be considered conclusive” creates an unscientific standard that can never be met, while subverting the law’s requirement that the claimant establish causation.

The suggestion that the studies that failed to find an association cannot be considered conclusive because they “lacked sufficient statistical power” is troublesome because it distorts and misapplies the very notion of statistical power. No attempt was made to describe the confidence intervals surrounding the point estimates of the null studies; nor was there any discussion whether the studies could be aggregated to increase their power to rule out meaningful associations.

The Circuit court’s scientific jurisprudence was thus seriously flawed. Without a discussion of the end points observed, the relevant point estimates of risk ratios, and the confidence intervals, the reader cannot assess the strength of the claims made by Goldman and Strom, or by defense expert Simpson, in their reports. Without identifying the study endpoints, the reader cannot evaluate whether the plaintiffs’ expert witnesses relied upon relevant outcomes in formulating their opinions. The court viewed the subject matter from 30,000 feet, passing over at 600 mph, without engagement or care. A strong dissent, however, suggested serious mischaracterizations of the plaintiffs’ evidence by the majority.

The only specific causation testimony to support plaintiff’s claims came from Goldman, in what appears to have been a “differential etiology.” Goldman purported to rule out a genetic cause, even though he had not conducted a critical family history or ordered a state-of-the-art chromosomal study. Id. at 140. Of course, nothing in a differential etiology approach would allow a physician to rule out “unknown” causes, which, for birth defects, make up the most prevalent and likely causes to explain any particular case. The majority acknowledged that these were short comings, but rhetorically characterized them as substantive, not methodologic, and therefore as issues for cross-examination, not for consideration by a judicial gatekeeping. All this is magical thinking, but it continues to infect judicial approaches to specific causation. See, e.g., Green Mountain Chrysler Plymouth Dodge Jeep v. Crombie, 508 F. Supp. 2d 295, 311 (D.Vt. 2007) (citing Ambrosini for the proposition that “the possibility of uneliminated causes goes to weight rather than admissibility, provided that the expert has considered and reasonably ruled out the most obvious”). In Ambrosini, however, Dr. Goldman had not ruled out much of anything.

Circuit Judge Karen LeCraft Henderson dissented in a short, but pointed opinion that carefully marshaled the record evidence. Drs. Goldman and Strom had relied upon a study by Greenberg and Matsunaga, whose data failed to show a statistically significant association between MPA and cleft lip and palate, when the crucial issue of timing of exposure was taken into consideration. Ambrosini, 101 F.3d at 142.

Beyond the specific claims and evidence, Judge Henderson anticipated the subsequent Supreme Court decisions in Joiner, Kumho Tire, and Weisgram, and the year 2000 revision of Rule 702, in noting that the majority’s acceptance of glib claims to have used a “traditional methodology” would render Daubert nugatory. Id. at 143-45 (characterizing Strom and Goldman’s methodologies as “wispish”). Even more importantly, Judge Henderson refused to indulge the assumption that somehow the length of Goldman’s C.V. substituted for evidence that his methods satisfied the legal (or scientific) standard of reliability. Id.

The good news is that little or nothing in Ambrosini survives the 2000 amendment to Rule 702. The bad news is that not all federal judges seem to have noticed, and that some commentators continue to cite the case, as lovely.

Probably no commentator has promiscuously embraced Ambrosini as warmly as Carl Cranor, a philosopher, and occasional expert witness for the lawsuit industry, in several publications and presentations.8 Cranor has been particularly enthusiastic about Ambrosini’s approval of expert witness’s testimony that failed to address “the relative risk between exposed and unexposed populations of cleft lip and palate, or any other of the birth defects from which [the child] suffers,” as well as differential etiologies that exclude nothing.9 Somehow Cranor, as did the majority in Ambrosini, believes that testimony that fails to identify the magnitude of the point estimate of relative risk can “assist the trier of fact to understand the evidence or to determine a fact in issue.”10 Of course, without that magnitude given, the trier of fact could not evaluate the strength of the alleged association; nor could the trier assess the probability of individual causation to the plaintiff. Cranor also has written approvingly of lumping unrelated end points, which defeats the assessment of biological plausibility and coherence by the trier of fact. When the defense expert witness in Ambrosini adverted to the point estimates for relevant end points, the majority, with Cranor’s approval, rejected the null findings as “too small to be significant.”11 If the null studies were, in fact, too small to be useful tests of the plaintiffs’ claims, intellectual and scientific honesty required an acknowledgement that the evidentiary display was not one from which a reasonable scientist would draw a causal conclusion.

1Ambrosini v. Labarraque, 101 F.3d 129, 138-39 (D.C. Cir. 1996) (citing and applying Ferebee), cert. dismissed sub nom. Upjohn Co. v. Ambrosini, 117 S.Ct. 1572 (1997) See also David E. Bernstein, “The Misbegotten Judicial Resistance to the Daubert Revolution,” 89Notre Dame L. Rev. 27, 31 (2013).

2 S. Prahalada, E. Carroad, M. Cukierski, and A.G. Hendrickx, “Embryotoxicity of a single dose of medroxyprogesterone acetate (MPA) and maternal serum MPA concentrations in cynomolgus monkey (Macaca fascicularis),” 32 Teratology 421 (1985).

3 S. Prahalada, E. Carroad, and A.G. Hendrick, “Embryotoxicity and maternal serum concentrations of medroxyprogesterone acetate (MPA) in baboons (Papio cynocephalus),” 32 Contraception 497 (1985).

4 See, e.g., Z. Katz, M. Lancet, J. Skornik, J. Chemke, B.M. Mogilner, and M. Klinberg, “Teratogenicity of progestogens given during the first trimester of pregnancy,” 65 Obstet Gynecol. 775 (1985); J.L. Yovich, S.R. Turner, and R. Draper, “Medroxyprogesterone acetate therapy in early pregnancy has no apparent fetal effects,” 38 Teratology 135 (1988).

5 G. Saccone, C. Schoen, J.M. Franasiak, R.T. Scott, and V. Berghella, “Supplementation with progestogens in the first trimester of pregnancy to prevent miscarriage in women with unexplained recurrent miscarriage: a systematic review and meta-analysis of randomized, controlled trials,” 107 Fertil. Steril. 430 (2017).

6 Ferebee v. Chevron Chemical Co., 736 F.2d 1529, 1535 (D.C. Cir.), cert. denied, 469 U.S. 1062 (1984).

7 Dr. Strom was also quoted as having provided a misleading definition of statistical significance: “whether there is a statistically significant finding at greater than 95 percent chance that it’s not due to random error.” Ambrosini at 101 F.3d at 136. Given the majority’s inadequate description of the record, the description of witness testimony may not be accurate, and error cannot properly be allocated.

8 Carl F. Cranor, Toxic Torts: Science, Law, and the Possibility of Justice 320, 327-28 (2006); see also Carl F. Cranor, Toxic Torts: Science, Law, and the Possibility of Justice 238 (2d ed. 2016).

9 Carl F. Cranor, Toxic Torts: Science, Law, and the Possibility of Justice 320 (2006).

10 Id.

11 Id. ; see also Carl F. Cranor, Toxic Torts: Science, Law, and the Possibility of Justice 238 (2d ed. 2016).

Posted in Causation, Expert Witness Discovery, Rule 702, Rule 703, Scientific Evidence, statistical evidence, Statistical Power | Comments Off on Failed Gatekeeping in Ambrosini v. Labarraque (1996)

Ferebee Revisited

December 28th, 2017

The following post was originally published on November 8, 2012, but was hacked, no doubt by the lawsuit industry, and replaced with mindless fluff as is its wont. It is now restored.

Ferebee Revisited

I used to think of the infamous Ferebee decision as the Dred Scott decision of scientific evidence; in essence, declaring that science has no validity issues that the law is bound to respect. Ferebee v. Chevron Chem. Co., 552 F. Supp. 1297 (D.D.C. 1982), aff’d, 736 F.2d 1529 (D.C. Cir.), cert. denied, 469 U.S. 1062 (1984). The rhetoric on expert witnesses, from the district and circuit courts in this case is sometimes jarring, but the facts of the case make the holding, rather than the expansive dicta, not so unreasonable, under all the facts and circumstances of the case.

On rereading Ferebee, I was struck by several aspects of the case that rarely are discussed when Ferebee is cited. On sober second thought, Ferebee may not be such a bad decision, especially considering that it has no continuing validity as a rule of decision for expert witness admissibility in federal court.

1. Ferebee is a government negligence case.

The plaintiff worked for the federal government when he was exposed to the herbicide paraquat. Richard Ferebee began working for the Department of Agriculture’s Beltsville Agricultural Research Center (BARC), in Beltsville, Maryland. He started spraying paraquat in the summer of 1977, and used the herbicide regularly through the time he was diagnosed with pulmonary fibrosis, in November 1979. 736 F.2d at 1531-32. Ferebee brought a failure to warn claim against the supplier of paraquat, Chevron Chemical Company. The allegations of actual or constructive knowledge of a hazard, however, could just as readily be asserted against the federal government, which owned the BARC facility, employed Ferebee, controlled and supervised his use of paraquat, and failed to comply with Chevron’s instructions. The federal government further regulated the sale and use of paraquat extensively, first by the Department of Agriculture, and later by the Environmental Protection Agency. Id. at 1532.

2. The exposure.

Ferebee filed suit in 1981, he died in 1982. His case was tried twice. In the first trial, the jury deadlocked; in the second trial, the jury returned a verdict in favor of his estate, and for his family, for $60,000. In his deposition testimony, Ferebee described how sprayed paraquat, in the summer of 1977. The chemical was diluted for use, per Chevron’s instructions. There was no evidence that Ferebee ever had direct contact with undiluted paraquat, or that the paraquat he was exposed to was not diluted according to the proportions recommended on Chevron’s label. 552 F. Supp. at 1295 & n. 3. Ferebee frequently got the chemical on his hands. 552 F. Supp. at 1294-95. Ferebee further described an occasion when he was drenched with paraquat when he walked behind a tractor that was spraying the chemical, and another incident when he used a defective sprayer that leaked paraquat “all over his pants.” 736 F.2d at 1532. On both occasions, Ferebee did not wash, and apparently went home contaminated, where he fell asleep, tired and dizzy, without showering. Id. As we will see, the exposure that Ferebee described would not have occurred had his federal employer followed the instructions on the label that it mandated. In 1978, the federal Occupational Health & Safety Administration published Guidelines on the need for protective clothing, respirators, immediate washing of contaminated skin, etc. Ferebee’s federal employer recklessly disregarded its own guidelines.

3. The warnings.

Paraquat could be sold in the United States only when labeled in accordance with EPA regulations, promulgated pursuant to the Federal Insecticide, Fungicide, and Rodenticide Act, 7 U.S.C. § 136, et seq. (FIFRA) The statute bars EPA from allowing sale of regulated herbicides, such as paraquat, unless the chemicals, as labeled, will not cause “unreasonable adverse effects on the environment.” 7 U.S.C. § 136a(c)(5)(C). Such effects are in turn defined as any unreasonable risk to man or the environment, taking into account the economic, social, and environmental costs and benefits of the use of [the] pesticide. 7 U.S.C. § 136(bb). FIFRA further requires the EPA to require labeling that is “adequate to protect health and the environment” and that is “likely to be read and understood.” 7 U.S.C. § 136(q)(1)(E). 736 F.2d at 1539-40.

Unfortunately, the courts failed to provide the complete warning label and the material data safety sheets. There are “snippets” provided, which make clear that the federal government was largely to blame for failing to comply with the directions required under FIFRA. For instance, the district court, in a footnote, acknowledged:

“For example, the label advised the user spraying paraquat to wear waterproof clothing and goggles, to avoid working in spray mist, and to wash splashes on the skin or eyes immediately with water.”

552. F. Supp. at 1304 n.40. The Court of Appeals reported that “the label, in large bold letters states:

DANGER

CAN KILL IF SWALLOWED

HARMFUL TO THE EYES AND SKIN

736 F.2d at 1536. The label also informed users to wash any exposed areas immediately, and to remove contaminated clothing. Id.

4. The Stipulation.

A key fact, rarely described or explained in discussions of the Ferebee case, is the parties’ stipulation

“that Mr. Ferebee’s only significant exposure to paraquat was on his intact skin; i.e., there was no evidence that Mr. Ferebee swallowed or inhaled paraquat, or that he spilled or sprayed it on an area of his skin upon which he had any apparent cuts or scrapes. The jury was not, of course, precluded from concluding that a person engaged in Mr. Ferebee’s line of work could have had some, or even many, minor cuts or abrasions not readily discernible to the naked eye or likely to be remembered some time later.”

552. F. Supp. at 1295 & n. 3.

Why did the plaintiffs try to present their case solely as a dermal exposure cases? As we will see, this stratagem made their medical causation case more difficult, but it avoided serious misuse and lack of proximate cause issues. Ferebee had been instructed by his co-workers and supervisors that paraquat was extremely dangerous if swallowed, and probably also if inhaled. The warning label was unequivocal in detailing the dangers and the need to avoid ingestion. (Without the full label, it is difficult to evaluate how well the label warned against inhalation, but the 1978 OSHA guidelines address the use of a proper respirator for situations in which paraquat may be inhaled.) On the other hand, the label had a weakness, which could be exploited, as long as the preemption defense could be held at bay: the label urged protective clothing, goggles, and immediate washing of contaminated skin, but it failed to describe the consequence of dermal exposure other than irritation. Ferebee could thus avoid his culpable conduct, as well as a sophisticated intermediary defense, by claiming that his exposure was only dermal.

Why did Chevron agree to the stipulation? The defendant probably felt sanguine about its preemption defense, and thus also about the adequacy of its warnings overall. The stipulation limited the plaintiff’s medical causation case to a route of exposure that put it into an arguable “first instance” case report. Chevron stood to gain a claim of “lack of notice,” and thus lack of actual or constructive knowledge of the risk of lung disease from dilute dermal exposure. The clinical presentation itself differed from many of the cases of known paraquat poisoning, see infra, and Chevron probably believed that it could deal with the medical causation claim better if exposure was limited to transdermal absorption. Curiously, Chevron did not argue that Ferebee must have had some inhalational exposure, which he almost certainly did. I suspect that Chevron’s position on inhalation was hedged because its warning label did not specify respirator usage for ordinary work exposures of applicators (as opposed to workers who handled undiluted paraquat, worked in confined spaces, etc.).

5. Medical causation

Chevron took a strident position, standing on the fact that there had been no previous documented cases of pulmonary fibrosis in workers exposed to diluted paraquat through their skin. The following facts were uncontroverted:

  • Paraquat causes pulmonary fibrosis in humans.
  • The evidence that established paraquat as a cause of pulmonary fibrosis was largely case series of acute onset of pulmonary fibrosis after ingestion.
  • Paraquat induces pulmonary fibrosis relatively rapidly.
  • Paraquat can be absorbed through the skin.
  • The parties agreed that any type of exposure – ingestion, inhalation, or dermal absorption – could cause lung damage. 552. F. Supp. at 1300 & n.28.
  • Once paraquat is ingested, inhaled, or absorbed, it can travel to the lungs.
  • Lung fibrosis caused by dermal absorption of paraquat had been described previously only with skin lesions before or after the injury. 736 F.2d at 1538.
  • The lungs are the target organ for paraquat.
  • There are numerous causes of pulmonary fibrosis (such as asbestosis, scleroderma, rheumatoid arthritis, etc.).
  • The variants of pulmonary fibrosis do not all look alike, present alike, or progress alike.
  • Mr. Ferebee had no known other disease or exposure that could account for his pulmonary fibrosis.
  • There is are cases of pulmonary fibrosis with no identifiable cause, known as idiopathic pulmonary fibrosis (IPF).
  • IPF is relatively rare; it too has a rapid onset and progression, although not as fast as the cases described after exposure to undiluted paraquat.
  • Mr. Ferebee’s medical history was largely unhelpful in explaining his clinical course.
  • Ferebee had some shortness of breath before starting to use paraquat. 552. F. Supp. at 1295.
  • Ferebee used paraquat occasionally over three years before he was diagnosed with pulmonary fibrosis.

Some observations about these facts. General causation in a sense was not contested. Paraquat causes pulmonary fibrosis. The issue was whether dilute dermal exposure over three years causes pulmonary fibrosis. Chevron stridently asserted that the “scientific method” required controlled experimental or observational (epidemiologic) studies. The problem with Chevron’s position was that general causation had already been established, and not by analytical epidemiologic studies.

6. The expert witnesses.

Ferebee was initially treated by Dr. Muhammed Yusuf, a pulmonary specialist, who diagnosed pulmonary fibrosis. Dr. Yusef referred Ferebee to the National Institutes of Health (NIH), where he came under the care of Dr. Ronald G. Crystal of the Heart, Lung, and Blood Institute. (Dr. Crystal is now at Cornell-Weill, where he is Chairman of Genetic Medicine, and he practices pulmonary medicine.)

Chevron called Dr. Carrington, who diagnosed Ferebee with IPF. Dr. Carrington challenged the plaintiffs’ expert witnesses’ opinions for lacking reliance upon controlled observational or experimental studies. 552. F. Supp. at 1301. Dr. Carrington, however, acknowledged that dermal cases are too rare for observational epidemiologic analysis, but emphasized that no animal studies of sufficient size had been done to support plaintiffs’ hypothesis. Chevron also called a Dr. Fisher, who presented a toxicokinetic (TK) analysis of Ferebee’s dermal absorption. Based upon his TK analysis, Dr. Fisher concluded that the maximal amount of paraquat absorbed by Ferebee was too small, based upon known cases and animal studies, to have caused paraquat toxicity. Id.

7. Chevron’s challenge to plaintiffs’ expert witnesses’ causation opinion.

None of the defendant’s expert witnesses examined Ferebee. The courts thought this was relevant, but they never articulated what would have been observed on physical examination that was important to resolving the differential diagnosis of paraquat toxicity versus IPF. There was no dispute that Ferebee had rapidly progressing pulmonary fibrosis. The expert witnesses on both sides evaluated Ferebee’s clinical data, presentation, clinical course, and arrived at different diagnoses. The plaintiffs’ expert witnesses’ diagnosis, however, involved a causal attribution to paraquat exposure.

The Ferebee case was litigated under Maryland law because federal statutory law requires state law to control in a wrongful death action arising out of the neglect or wrongful act of another on a federal enclave. 16 U.S.C. § 457. 736 F.2d at 1533. (Maryland law is actually favorable to a sophisticated intermediary defense, although the key decisions post-date Ferebee.) Chevron appears to have relied upon Maryland’s articulation of the Frye general acceptance doctrine, and the courts analyzed Chevron’s arguments as a Frye challenge. 552 F. Supp. at 1301; 736 F.2d at 1535. Although the use of Maryland law to determine an evidentiary issue seems suspect, Chevron pressed apparently pressed its challenge in terms of Maryland’s version of Frye, and not based upon Federal Rule of Evidence 702. The infamous language used by both the district and the circuit courts was, therefore, not an interpretation of federal law. Rule 702 was never cited or discussed in either the trial or the appellate court’s opinion.

My re-reading of Ferebee has softened my criticisms of state courts that had relied upon the case, even after the Supreme Court’s decision in Daubert. Softened but not eliminated my criticism — Ferebee is still a case largely confined to its facts, and the language quoted as a standard of admissibility is really a statement of the appellate standard of review for the jury’s determination of medical causation.

8. The judicial resolution of Chevron’s Frye challenge

The district court insightfully recognized that Chevron was demanding a level of evidence, which had never been required to establish paraquat’s generally accepted ability to cause pulmonary fibrosis. This recognition led to the district court’s colorful language:

“It is true that medical expert testimony must be grounded in proper scientific methodology, but the extremely stringent standard that defendant suggests is beyond reason. Product liability law, especially as it relates to relatively new products or those with a relatively rare yet significant danger, would be rendered next to meaningless if a plaintiff could prove he was injured by a product only after a ‘statistically significant’ number of other people were also injured. A civilized legal system does not require that much human sacrifice before it can intervene. The fact that this is the first case of this exact type-or at least the first of its exact type in which the involvement of paraquat was discovered by alert doctors — cannot be enough by itself to shield defendant from liability. Defendant’s experts were not able to fault Dr. Crystal for his basic diagnostic methodology; in fact, they used the same kinds of test results, consultations, and other tools that he did. What they disagreed with chiefly were his conclusions.”

552 F. Supp. at 1301. The important observation is that general causation had been established case series and reports of human exposure. There never was statistical evidence that had been evaluated for “significance,” to establish general causation for undiluted paraquat, and the trial court refused, under Maryland law, to require such evidence for general causation for diluted paraquat. In this context, we can see that the trial court’s suggestion that statistical significance was not required has little bearing upon, cases in which general causation could only be established using epidemiologic evidence, with its attendant statistical inferences.

Of course, the matter only became worse when Chevron persisted in its argument and presented it to a liberal panel of the D.C. Circuit. (Judge Mikva wrote the opinion for a panel that included Judge Wald, and Senior Judge Bazelon.) The panel’s decision ratcheted up the rhetoric:

“Thus, a cause-effect relationship need not be clearly established by animal or epidemiological studies before a doctor can testify that, in his opinion, such a relationship exists. As long as the basic methodology employed to reach such a conclusion is sound, such as use of tissue samples, standard tests, and patient examination, product liability does not preclude recovery until a ‘statistically significant’ number of people have been injured or until science has had the time and resources to complete sophisticated laboratory studies of the chemical. In a courtroom, the test for allowing a plaintiff to recover is not scientific certainty, but legal sufficiency; if reasonable jurors could conclude from the expert testimony that paraquat more likely than not caused Ferebee’s injury, the fact that another jury might reach the opposite conclusion or that science would require more evidence before conclusively considering the causation question resolved is irrelevant. That Ferebee’s case may have been the first of its exact type, or that his doctors may have been the first alert enough to recognize such a case, does not mean that the testimony of those doctors, who are concededly well qualified in their fields, should not have been admitted.”

736 F.2d at 1535-36 (emphasis in original).

Again, the dismissive attitude towards statistically significant evidence is limited to the context of a causal analysis that had been made, to everyone’s satisfaction, for undiluted paraquat, without the need for epidemiologic, statistical evidence. Statistical significance was never at issue. In this way, Ferebee resembles the untoward language on statistical significance from Matrixx Initiatives Inc. v. Siracusano. In both cases, statistical significance was never really at issue. In Ferebee, there was no statistical evidence needed or used to reach causal conclusions about paraquat’s ability to induce pulmonary fibrosis. In Matrixx Initiatives, allegations of statistical significance and causation were not necessary because the plaintiffs needed only to allege materiality of the facts suppressed by the company in order to plead a securities fraud case. Materiality could be established without causation, and thus neither causation nor statistical significance needed to be alleged.

As for Chevron’s Frye challenge, the district court rejected the implied call for a vote on the general acceptance of Dr. Crystal’s reasoning. Frye may require “vote counting” of some sort, but the process becomes irrelevant when virtually no one has registered to vote. Otherwise, the defense and the plaintiffs’ expert witnesses appeared to be using the same technique of arguing by analogy to accepted cases of paraquat poisoning or IPF. Dr. Crystal opined that Ferebee’s case was “similar” to three other cases he had identified. Dr. Carrington argued that Ferebee’s case was more like IPF cases, although IPF cases themselves have some clinical heterogeneity as well. Paraquat cases described onset to death as a very rapid process. Ferebee did not present with significant symptoms for three years after his first exposure, and then he survived for another two plus years. Ferebee did not report skin lesions, which had been reported in previous cases of dermal exposure leading up to pulmonary fibrosis. The case presented, on the diagnostic level, a difficult call, but it is easy to see the courts’ impatience with the defendant’s insistence upon more stringent criteria and evidence than was used to establish the causal connection with undiluted paraquat.

9. Expert witness qualifications.

Chevron never challenged Dr. Yusuf’s or Dr. Crystal’s qualifications. The oft-quoted comments about expert witness qualifications were made in the context of describing the appellate court’s standard of review, and the court’s role in not assessing credibility or weighing the evidence:

“These admonitions apply with special force in the context of the present action, in which an admittedly dangerous chemical is alleged through long-term exposure to have caused disease. Judges, both trial and appellate, have no special competence to resolve the complex and refractory causal issues raised by the attempt to link low-level exposure to toxic chemicals with human disease. On questions such as these, which stand at the frontier of current medical and epidemiological inquiry, if experts are willing to testify that such a link exists, it is for the jury to decide whether to credit such testimony.”

736 F.2d at 1534.

This procedural posture is obviously very different from the initial determination of admissibility. As far as credentials are concerned, Drs. Yusuf and Crystal were hardly “hired guns”; both physicians were well qualified. Dr. Crystal had outstanding qualifications, and Chevron wisely never challenged them. Remarkably, this language has been mistakenly invoked as a standard for trial courts to use in determining the admissibility of expert witness opinion testimony. It is no such thing.

10. Preemption and Warnings Causation.

Ultimately, Chevron’s preemption defense was rejected by both the district and the circuit court. FIFRA preemption has had its ups and downs; no surprise there. More interesting is the emphasis that both courts gave to the important role of the employer in the case. The evidence overwhelming showed that Ferebee had never read the warning label, and thus the element of proximate causation between allegedly inadequate warning and harm was in jeopardy of going unproved. The courts, however, emphasized the role that the employer, through its supervisors and responsible co-workers, play in the complex organizational situation of a modern workplace:

“Mr. Ferebee’s situation was quite different, however. He did not purchase paraquat for his personal use; rather, it was provided to him by his employer for use on the job. The evidence showed that his principal source of information about paraquat was the oral instructions of his supervisors and co-workers, not the written label. He learned from them how to mix the product and how to spray it. It was also from this source that he learned of the danger of getting the product in his mouth: one of his co-workers warned him that if he accidently swallowed paraquat, it would ‘get in his blood’ and poison him. This is a common pattern of instruction and use of occupational materials in the workplace. Learning by doing and learning by oral instruction are tried and true methods of educating manual workers in their jobs. Therefore, although it is crucial to plaintiff’s case that someone would have read the label, it was not necessary for Mr. Ferebee to have done so. And it is obvious that one or more employees at BARC did read the label, since information did reach Mr. Ferebee about the proportions for diluting the product and about the dangers about which the label did warn. It was appropriate for the jury to infer that a warning about the danger of fatal lung disease from dermal exposure would also have been communicated to Mr. Ferebee. See Restatement (Second) of Torts § 388 comment n (seller normally entitled to assume that adequate warning will be passed on by purchaser to ultimate user); cf. Chambers v. G.D. Searle & Co., 441 F.Supp. at 381 (in product liability case involving prescription drug, relevant warning is the one given to doctor, not patient).”

552 F. Supp. at 1303-04 (internal citations omitted). So here we have Ferebee, the subject of so much derision and aspersion from defense counsel, embracing the Section 388, comment n, as well as applying learned intermediary principles to a case not involving prescription drugs. The appellate court was waxed enthusiastic about the principles of Section 388, and went so far as to cite Victor Schwartz in support:

“We live in an organizational society in which traditional common-law limitations on an actor’s duty must give way to the realities of society. *** In this case, Mr. Ferebee did not purchase the paraquat for his personal use, and there was substantial evidence that workplace communication about the dangers associated with various chemicals usually took the form of oral instructions from supervisors to workers, the latter of whom then retransmitted the information to co-workers. This, rather than individual reading of product warnings, is a typical method by which information is disseminated in the modern workplace. See Schwartz & Driver, “Warnings in the Workplace: The Need for a Synthesis of Law and Communication Theory,” 52 U. Cinn. L. Rev. 38, 66-83 (1983). The requirement that an improper warning proximately ‘cause’ the injury should be elaborated against this background. We believe Maryland would construe its tort law in this case to require only that someone in the workplace have read the label, not that Mr. Ferebee personally have read it. Because there is no dispute that one or more employees at BARC did read the label, we hold that the jury could properly have inferred that, had a warning about the danger of disease from dermal exposure been included on the label, that warning would have been communicated to Mr. Ferebee and that he would as a result have acted differently. Alternatively, the jury could have inferred that an adequate warning would have led Ferebee’s employers to undertake steps that would have protected him from paraquat poisoning-for example, provision of showers for use after spraying.”

736 F.2d at 1539 (emphasis in original; internal citation omitted). Judge Mikva’s prediction, of course, was absolutely accurate; Maryland tort law did, soon thereafter, embrace the sophisticated intermediary defense to exculpate the defendant in such remote supplier situations. See, e.g., Kennedy v. Mobay Corp., 84 Md. App. 397 (1990) (applying sophisticated user defense to bar claims against manufacturers of toluene diisocyanate), aff’d, 325 Md. 385 (1992); Higgins v. E.I. DuPont de Nemours, Inc., 671 F. Supp. 1055 (D. Md. 1987) (Maryland law; holding that manufacturer of paint was in better position than bulk supplier to communicate warnings to customers’ employees), aff’d, 863 F.2d 1162 (4th Cir. 1988). The principle invoked to excuse plaintiff from reading the warning label also works to exculpate the defendant when that warning label is otherwise adequate, or when the intermediary knows of the hazard in any event.

Posted in Causation, Rule 702, Scientific Evidence, statistical evidence | Comments Off on Ferebee Revisited

« Older Entries
Newer Entries »