TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Hysterical Histortions

June 12th, 2014

Ramses Delafontaine is a young, aspiring historian. In his graduate thesis, Historicizing the Forensification of History: A Study of Historians as Expert Witnesses in Tobacco Litigation in the United States of America (Univ. Ghent 2013), discusses my commentary on Marxist historians, David Rosner and Gerald Markowitz, and suggests that I claim that lawyers without historical training or experience can do the job of historians.  Id. at 98-100.

Given their training and skills in documenting and recounting narratives, lawyers do, indeed, often do the job of historians, and they often do it very well. Of course, lawyers are often guided, inspired, and assisted by professional historians. Sometimes that guidance is necessary. Lawyers’ narratives, unlike historians’, are also subject to judicial control in the form of evidentiary rules about speculation, relevance, reliability, authentication, and trustworthiness.

Regurgitating Historical Evidence

American courts have thus appropriately limited the use of expert witnesses to present historical narratives in judicial proceedings.  See In re Fosamax Prods. Liab. Litig., 645 F. Supp. 2d 164, 192 (S.D.N.Y. 2009) (‘‘[A]n expert cannot be presented to the jury solely for the purpose of constructing a factual narrative based upon record evidence.’’) (internal citations omitted).[1] In Fosamax, Judge Keenan excluded Dr. Susan Parisian’s proffered narrative account of the development and approval of a bisphosphonate medication for osteoporosis in a case involving a claim of osteonecrosis of the jaw (“phossy jaw”).  Judge Keenan detailed the problems that arise from using expert witnesses to present partisan historical narratives:

“In detailing the factual basis for her opinions, Dr. Parisian’s report presents a narrative of select regulatory events through the summary or selective quotation from internal Merck documents, regulatory filings, and the deposition testimony of Merck employees.

The Court agrees with Merck that, to the extent such evidence is admissible, it should be presented to the jury directly. Dr. Parisian’s commentary on any documents and exhibits in evidence will be limited to explaining the regulatory context in which they were  created, defining any complex or specialized terminology, or drawing inferences that would not be apparent without the benefit of experience or specialized knowledge. She will not be permitted to merely read, selectively quote from, or ‘regurgitate’ the evidence.”

Id.[2]

Ramses Delafontaine is wrong, however, to opine that my rants against Rosner and Markowitz suggest that I have ruled out any role for historians in litigation.   Lisa K. Walker is an historian, who trained at the University of California, Berkeley.  In the welding fume litigation, plaintiffs’ counsel weaved a complex narrative of conspiracy allegations, based in large measure upon the absence of evidence. At the request of Metropolitan Life Insurance Company, Professor Walker researched the dates of publication for various editions of a booklet, Health Protection in Welding, which formed the basis for the plaintiffs’ speculations. Walker found and analyzed eight separate editions, and dated each by internal and external references.  Based upon her research, Walker submitted a declaration, which ultimately was immensely helpful to the resolution of the issues. See In re Welding Rod Prods. Liab. Litig., Case No. 1:03-CV-17000 (MDL Docket No. 1535) (N.D.Ohio Nov. 24, 2004; In re Welding Fume Prods. Liab. Litig., 2007 WL 1087605 (N.D. Ohio April 9, 2007)  (O’Malley, J.) (granting summary judgment in favor of Metropolitan Life Insurance Company).

Although Metropolitan Life should not have had to disprove the allegations, Walker’s research showed the plaintiffs’ historical speculation to be clearly wrong. Sometimes historians can and do contribute valuably to the resolution of legal issues, but the issues are usually more modest than the ones that social and labor historians want to see resolved in favor of their pet theories.

Delafontaine also has a website on the role of historians as expert witnesses in United States tobacco cases.  One of his theses is that “historians who have been involved as expert witnesses for the tobacco industry have been in it for the money and have sold their professional integrity as a historian and an academic.” Delafontaine’s approach is a bit one-sided in that he sees only defendants’ expert witnesses as being “in it for the money,” despite the substantial billings of plaintiffs’ expert witnesses, and their ideological biases. Somehow Delafontaine has missed how even the feel-good advocacy of anti-tobacco activists occasionally outruns its evidentiary headlights.  See, e.g., Michael Siegel, “Is the tobacco control movement misrepresenting the acute cardiovascular health effects of secondhand smoke exposure? An analysis of the scientific evidence and commentary on the implications for tobacco control and public health practice,” 4 Epidem. Persp. & Innov. 12 (2007).


[1] See also In re Prempro Prods. Liab. Litig., 554 F.Supp. 2d 871, 880, 886 (E.D.Ark.2008) (overturning a punitive damages award based on Dr. Parisian’s testimony in part because she ‘‘did not explain the documents, provide summaries, or tie them in to her proposed regulatory testimony’’ and ‘‘did not provide analysis, opinion, or expertise’’); Highland Capital Management, L.P. v. Schneider, 379 F.Supp. 2d 461, 469 (S.D.N.Y.2005)(‘‘[A]n expert cannot be presented to the jury solely for the purpose of constructing a factual narrative based upon record evidence.’’); In re Rezulin Products Liab. Litig., 309 F.Supp. 2d 531, 546 (S.D.N.Y.2004) (rejecting portion of expert report presenting history of Rezulin for no purpose but to ‘‘provid[e] an historical commentary of what happened,’’ along with subjective assessments of intent, motives, and states of mind); In re Diet Drugs Prods. Liab. Litig., MDL No. 1203, 2000 WL 876900, at *9 (E.D.Pa. June 20, 2000) (same); Taylor v. Evans, 1997 WL 154010, at *2 (S.D.N.Y. Apr.1, 1997) (rejecting portions of expert report on the ground that the testimony consisted of ‘‘a narrative of the case which a lay juror is equally capable of constructing’’).

[2] Dr. Parisian appears to be a serial narrative abuser, who has been repeatedly but not consistently excluded. See Scheinberg v. Merck & Co. 924 F.Supp. 2d 477, 497 (S.D.N.Y. 2013); Pritchett v. I-Flow Corp., 2012 WL 1059948, at *7 (D. Colo. Mar. 28, 2012); Miller v. Stryker Instruments, 2012 WL 1718825, at *10-12 (D. Ariz. Mar. 29, 2012) (excluding narrative testimony); Kaufman v. Pfizer Pharms., Inc., 2011 WL 7659333, at *6-10 (S.D. Fla. Aug. 4, 2011), reh’g denied, 2011 WL 10501233 (S.D. Fla. Aug. 10, 2011)(narrative testimony); Hines v. Wyeth, 2011 WL 2680842, at *7 (S.D.W. Va. July 8, 2011), reh’g granted in part, 2011 WL 2730908, at *2 (S.D.W. Va. July 13, 2011); In re Heparin Prods. Liab. Litig., 2011 WL 1059660, at *8 (N.D. Ohio March 21, 2011); Lopez v. I-Flow Inc., 2011 WL 1897548, at *9-10 (D. Ariz. Jan. 26, 2011) (narrative testimony); In re Trasylol Prods. Liab. Litig., 709 F.Supp.2d 1323, 1351 (S.D. Fla. 2010)(tendentious narrative testimony) (“Plainly stated, Dr. Parisian is an advocate, presented with the trappings of an expert but with no expectation or intention of abiding by the opinion constraints of Rule 702.”), reh’g denied, 2010 WL 2541892 (S.D. Fla. June 22, 2010); In re Gadolinium-Based Contrast Agents Prods. Liab. Litig., 2010 WL 1796334, at *13 (N.D. Ohio May 4, 2010); Bessemer v. Novartis Pharms. Corp., 2010 WL 2300222 (N.J. Super. Law Div. April 30, 2010); In re Prempro Prods. Liab. Litig., 554 F. Supp. 2d 871, 879-87 (E.D. Ark. 2008)(reversing judgment on grounds of erroneous admission of narrative testimony), aff’d  in relevant part, 586 F.3d 547, 571 (8th Cir. 2009). Occasionally, Dr. Parisian slips through the gate.  See, e.g., Block v. Woo Young Medical Co., 937 F.Supp.2d 1028, 1044-47 (2013)

Goodman v Viljoen – Meeting the Bayesian Challenge Head On

June 11th, 2014

Putting Science On Its Posterior

Plaintiffs’ and Defendants’ counsel both want the scientific and legal standard to be framed as a very high posterior probability of the truth of a claim. Plaintiffs want the scientific posterior probability to be high because they want to push the legal system in the direction of allowing weak or specious claims that are not supported by sufficient scientific evidence to support a causal conclusion.  By asserting that the scientific posterior probability for a causal claim is high, and that the legal and scientific standards are different, they seek to empower courts and juries to support judgments of causality that are deemed inconclusive, speculative, or worse, by scientists themselves.

Defendants want the scientific posterior probability to be high, and claim that the legal standard should be at least as high as the scientific standard.

Both Plaintiffs and Defendants thus find common cause in committing the transposition fallacy by transmuting the coefficient of confidence, typically 95%, into a minimally necessary posterior probability for scientific causal judgments.  “One wanders to the left, another to the right ; both are equally in error, but are seduced by different delusions.”[1]

In the Goodman v. Viljoen[2] case, both sides, plaintiffs and defendants, embraced the claim that science requires a high posterior probability, and that the p-value provided evidence of the posterior probability of the causal claim at issue.  The error came mostly from the parties’ clinical expert witnesses and from the lawyers themselves; the parties’ statistical expert witnesses appeared to try to avoid the transposition fallacy. Clearly, no text would support the conflation of confidence with certainty. No scientific text, treatise, or authority was cited for the notion that scientific “proof” required 95% certainty. This notion was simply an opinion of testifying witnesses.

The principal evidence that antenatal corticosteroid (ACS) therapy can prevent cerebral palsy (CP) came from a Cochrane review and meta-analysis[3] of clinical trials.  The review examined a wide range of outcomes, only one of which was CP.  The trials were apparently not designed to assess CP risk, and they varied significantly in case definition, diagnostic criteria, and length of follow up for case ascertainment. Of the five included studies, four ascertained CP at follow up from two to six years, and the length of follow up was unknown in the fifth study.

Data were sparse in the Cochrane review, as expected for a relatively rare outcome.  The five studies encompassed 904 children, with 490 in the treatment group, and 414 in the control group. There was a total of 48 CP cases, with 20 in the treatment, and 28 in the control, groups. Blinding was apparently not maintained over the extended reporting period.

Professor Andrew Willan, plaintiffs’ testifying expert witness on statistics, sponsored a Bayesian statistical analysis, with which he concluded that there was between a 91 and 97% probability that there was an increased risk of CP from not providing ACS in pre-term labor (or, a decreased risk of CP from administering ACS).[4] Willan’s posterior probabilities was for any increased risk, based upon the Cochrane data.  Willan’s calculations were not provided in his testimony, and no information about his prior probability, was given. The data came from clinical trials, but the nature of the observations and the analyses made these trials little more than observational studies conducted within the context of clinical trials designed to look at other outcomes. The Bayesian analysis did not account for the uncertainty in the case definitions, variations in internal validity and follow up, and biases in the clinical trials. Willan’s posterior probabilities thus described a maximal probability for general causation, which surely needed to be discounted for validity and bias issues.

There was a further issue of external validity. The Goodman twins developed CP from having sustained periventricular leukomalacia (PVL), which is one among several mechanistic pathways by which CP can develop in pre-term infants.  The Cochrane data did not address PVL, and the included trials were silent as to whether any of the CP cases involved PVL mechanisms.  There was no basis for assuming that ACS reduced risk of CP from all mechanisms equally, or even at all.[5] The Willan posterior probabilities did not address the external validity issues as they pertained to the Goodman case itself.

Although Dr. Viljoen abandoned the challenge to the Bayesian analysis at trial, his statistical expert witness, Dr. Robert Platt went further to opine that he agreed with Willan’s calculations.  To agree with his calculations, and the posterior probabilities that came out of those calculations, Platt had to have agreed with the analyses themselves. This agreement seems ill considered given that elsewhere in his testimony, Platt appears to advance important criticisms of the Cochrane data in the form of validity and bias issues.

Certainly, Platt’s concession about the correctness of Willan’s calculations greatly undermined Dr. Viljoen’s position with the trial and appellate court. Dr. Viljoen maintained those criticisms throughout the trial, and on appeal.  See, e.g., Defendant (Appellant) Factum, 2012 CCLTFactum 20936, at ¶14(a)

(“(a) antenatal corticosteroids have never been shown to reduce the incidence or effect of PVL”); id. at ¶14(d)(“at best, even taking the Bayesian approach at face value, the use of antenatal corticosteroids showed only a 40% reduction in the incidence of cerebral palsy, but not PVL”).

How might have things gone better for Dr. Vijoen? For one thing, Platt’s concession about the correctness of Willan’s calculations had to be explained and qualified as conceding only the posterior probability on the doubtful and unproven assumptions made by Willan. Willan’s posterior, as big as it was, represented only an idealized maximal posterior probability, which in reality had to be deeply discounted by important uncertainties, biases, and validity concerns.  The inconclusiveness of the data were “provable” on either a frequentist or a Bayesian analysis.


[1] Horace, in Wood, Dictionary of Quotations 182 (1893).

[2] Goodman v. Viljoen, 2011 ONSC 821 (CanLII), aff’d, 2012 ONCA 896 (CanLII), leave appeal den’d, Supreme Court of Canada No. 35230 (July 11, 2013).

[3] Devender Roberts & Stuart R Dalziel “Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth,” Cochrane Database of Systematic Reviews, at 8, Issue 3. Art. No. CD004454 (2006)

[4] Notes of Testimony of Andrew Willan at 34 (April 9, 2010) (concluding that ACS reduces risk of CP, with a probability of 91 to 97 percent, depending upon whether random effects or fixed effect models are used).

[5] See, e.g., Olivier Baud, Laurence Laurence Foix l’Hélias, et al., “Antenatal Glucocorticoid- Treatment and Cystic Periventricular Leukomalacia in Very Premature Infants,” 341 New Engl. J. Med. 1190, 1194 (1999) (“Our results suggest that exposure to betamethasone but not dexamethasone is associated with a decreased risk of cystic periventricular leukomalacia.”).

 

Goodman v Viljoen – Statistical Fallacies from Both Sides

June 8th, 2014

There was a deep irony to the Goodman[1] case.  If a drug company, in 1995, marketed antenatal corticosteroid (ACS) for the prevention of cerebral palsy (CP) in the United States, the government might well have prosecuted the company for misbranding.  The company might also be subject to a False Claims Act case as well. No clinical trial had found ACS efficacious for the prevention of CP at the significance level typically required by the FDA; no meta-analysis had found ACS statistically significantly better than placebo for this purpose.  In the Goodman case, however, failure to order a full course of ACS was malpractice with respect to the claimed causation of CP in the Goodman twins.

The Goodman case also occasioned a well-worn debate over the difference between scientific and legal evidence, inference, and standards of “proof.” The plaintiffs’ case rested upon a Cochrane review of ACS with respect to various outcomes. For CP, the Cochrane meta-analyzed only clinical trial data, and reported:

“a trend towards fewer children having cerebral palsy (RR 0.60, 95% CI 0.34 to 1.03, five studies, 904 children, age at follow up two to six years in four studies, and unknown in one study).”[2]

The defendant, Dr. Viljoen, appeared to argue that the Cochrane meta-analysis must be disregarded because it did not provide a showing of efficacy for ACS in preventing CP, at a significance probability less than 5 percent.  Here is the trial court’s characterization of Dr. Viljoen’s argument:

“[192] The argument that the Cochrane data concerning the effects of ACS on CP must be ignored because it fails to reach statistical significance rests on the flawed premise that legal causation requires the same standard of proof as medical/scientific causation. This is of course not the case; the two standards are in fact quite different. The law is clear that scientific certainty is not required to prove causation to the legal standard of proof on a balance of probabilities (See: Snell v. Farrell, [1990] 2 S.C.R. 311, at para. 34). Accordingly, the defendant’s argument in this regard must fail and for the purposes of this court, I accept the finding of the Cochrane analysis that ACS reduces the instance [sic] of CP by 40%.”

“Disregard” seems extreme for a meta-analysis that showed a 40% reduction in risk of a serious central nervous system disorder, with p = 0.065.  Perhaps Dr. Viljoen might have tempered his challenge some by arguing that the Cochrane analysis was insufficient.  One problem with Dr. Viljoen’s strident argument about statistical significance was that it overshadowed the more difficult, qualitative arguments about threats to validity in the Cochrane finding from loss to follow up in the aggregated trial data. These threats were probably stronger arguments against accepting the Cochrane “trend” as a causal conclusion. Indeed, the validity and the individual studies and the meta-analyses, along with questions about the accuracy of data, were not reflected in Bayesian analysis.

Another problem is that Dr. Viljoen’s strident assertion that p < 0.05 was absolutely necessary fed plaintiffs’ argument that the defendant was attempting to change the burden of proof for plaintiffs from greater than 50% to 95% or greater.  Given the defendant’s position, great care was required to prevent the trial court from committing the transposition fallacy.

Justice Walters rejected the suggestion that a meta-analysis with a p-value of 6.5% should be disregarded, but the court’s discussion skirts the question whether and how the Cochrane data can be sufficient to support a conclusion of ACS efficacy. Aside from citing a legal case, however, Justice Walters provided no basis for suggesting that the scientific standard of proof was different from the legal standard. From the trial court’s opinion, the parties or their expert witnesses appeared to conflate “confidence,” a technical term when used to describe intervals or random error around sample statistics, with “level of certainty” in the obtained result.

Justice Walters is certainly not the first judge to fall prey to the fallacious argument that the scientific burden of proof is 95%.[3]  The 95% is, of course, the coefficient of confidence for the confidence interval that is based upon a p-value of 5%. No other explanation for why 95% is a “scientific” standard of proof was offered in Goodman; nor is it likely that anyone could point to an authoritative source for the claim that scientists actually adjudge facts and theories by this 95 percent probability level.

Justice Walters’ confusion was led by the transposition fallacy, which confuses posterior and significance probabilities.  Here is a sampling from Her Honor’s opinion, first from Dr. Jon Barrett, one of the plaintiffs’ expert witnesses, an obstetrician and fetal maternal medicine specialist at Sunnybrook Hospital, in Toronto, Ontario:

“[85] Dr. Barrett’s opinion was not undermined during his lengthy cross-examination. He acknowledged that the scientific standard demands 95% certainty. He is, however, prepared to accept a lower degree of certainty. To him, 85 % is not merely a chance outcome.

                                                                                        * * *

[87] He acknowledged that scientific evidence in support of the use of corticosteroids has never shown statistical significance with respect to CP. However, he explained it is very close at 93.5%. He cautioned that if you use a black and white outlook and ignore the obvious trends, you will falsely come to the conclusion that there is no effect.”

Dr. Jon (Yoseph) Barrett is a well-respected physician, who specializes in high-risk pregnancies, but his characterization of a black-white outlook on significance testing as leading to a false conclusion of no effect was statistically doubtful.[4]  Dr. Barrett may have to make divinely inspired choices in surgery, but in a courtroom, expert witnesses are permitted to say that they just do not know. Failure to achieve statistical significance, with p < 0.05, does support a conclusion that there is no effect.

Professor Andrew Willan was plaintiffs’ testifying expert witness on statistics.  Here is how Justice Walters summarized Willan’s testimony:

“[125] Dr. Willan described different statistical approaches and in particular, the frequentist or classical approach and the Bayesian approach which differ in their respective definitions of probability. Simply, the classical approach allows you to test the hypothesis that there is no difference between the treatment and a placebo. Assuming that there is no difference, allows one to make statements about the probability that the results are not due to chance alone.

To reach statistical significance, a standard of 95% is required. A new treatment will not be adopted into practice unless there is less than a 5% chance that the results are due to chance alone (rather than due to true treatment effect).

[127] * * * The P value represents the frequentist term of probability. For the CP analysis [from the Cochrane meta-analysis], the P value is 0.065. From a statistical perspective, that means that there is a 6.5% chance that the differences that are being observed between the treatment arm versus the non-treatment arm are due to chance rather than the treatment, or conversely, a 93.5% chance that they are not.”

Justice Walters did not provide transcript references for these statements, but they are clear examples of the transposition fallacy. The court’s summary may have been unfair to Professor Willan, who seems to have taken care to avoid the transposition fallacy in his testimony:

“And I just want to draw your attention to the thing in parenthesis where it says, “P = 0.065.” So, basically that is the probability of observing data this extremely, this much in favor of ACS given, if, if in fact the no [sic, null] hypothesis was true. So, if, if the no hypothesis was true, that is there was no difference, then the probability of observing this data is only 6.5 percent.”

Notes of Testimony of Andrew Willan at 26 (April , 2010). In this quote, Professor Willan might have been more careful to point out that the significance probability of 6.5%  is a cumulative probability by describing the data observed “this extremely” and more. Nevertheless, Willan certainly made clear that the probability measure was based upon assuming the correctness of the null hypothesis. The trial court, alas, erred in stating the relevant statistical concepts.

And then there was the bizarre description by Justice Walters, of the Cochrane data, as embodying a near-uniform distribution represented by the Cochrane data:

“[190] * * * The Cochrane analysis found that ACS reduced the risk of CP (in its entirety) by 40%, 93.5% of the time.”

The trial court did not give the basis for this erroneous description of the Cochrane ACS/CP data.[5] To be sure, if the Cochrane result were true, then 40% reduction might be the expected value for all trials, but it would be a remarkable occurrence for 93.5% of the trials to obtain the same risk ratio as the one observed in the meta-analysis.

The defendant’s expert witness on statistical issues, Prof. Robert Platt, similarly testified that the significance probability reported by the Cochrane was dependent upon an assumption of the null hypothesis of no association:

“What statistical significance tells us, and I mentioned at the beginning that it refers to the probability of a chance finding could occur under the null-hypothesis of no effect. Essentially, it provides evidence in favour of there being an effect.  It doesn’t tell us anything about the magnitude of that effect.”

Notes of Testimony of Robert Platt at 11 (April 19, 2010)

Perhaps part of the confusion resulted from Prof. Willan’s sponsored Bayesian analysis, which led him to opine that the Cochrane data permitted him to state that there was a 91 to 97 percent probability of an effect, which might have appeared to the trial court to be saying the same thing as interpretation of the Cochrane’s p-value of 6.5%.  Indeed, Justice Walters may have had some assistance in this confusion from the defense statistical expert witness, Prof. Platt, who testified:

“From the inference perspective the p-value of 0.065 that we observe in the Cochrane review versus a 91 to 97 percent probability that there is an effect, those amount to the same thing.”

Notes of Testimony of Robert Platt at 50 (April 19, 2010).  Now the complement of the p-value, 93.5%, may have fallen within the range of posterior probabilities asserted by Professor Willan, but these probabilities are decidedly not the same thing.

Perhaps Prof. Platt was referring only to the numerical equivalence, but his language, “the same thing,” certainly could have bred misunderstanding.  The defense apparently attacked the reliability of the Bayesian analysis before trial, only to abandon the challenge by the time of trial.  At trial, defense expert witness Prof. Platt testified that he did not challenge Willan’s Bayesian analysis, or the computation of posterior probabilities.  Platt’s acquiescence in Willan’s Bayesian analysis is unfortunate because the parties never developed testimony exactly as to how Willan arrived at his posterior probabilities, and especially as to what prior probability he employed.

Professor Platt went on to qualify his understanding of Willan’s Bayesian analysis as providing a posterior probability that there is an effect, or in other words, that the “effect size” is greater than 1.0.  At trial, the parties spent a good deal of time showing that the Cochrane risk ratio of 0.6 represented the decreased risk for CP of administering a full course of ACS, and that this statistic could be presented as an increased CP risk ratio of 1.7, for not having administered a full course of ACS.  Platt and Willan appeared to agree that the posterior probability described the cumulative posterior probabilities for increased risks above 1.0.

“[T]he 91% is a probability that the effect is greater than 1.0, not that it is 1.7 relative risk.”

Notes of Testimony of Robert Platt at 51 (April 19, 2010); see also Notes of Testimony of Andrew Willan at 34 (April 9, 2010) (concluding that ACS reduces risk of CP, with a probability of 91 to 97 percent, depending upon whether random effects or fixed effect models are used).[6]

One point on which the parties’ expert witnesses did not agree was whether the failure of the Cochrane’s meta-analysis to achieve statistical significance was due solely to the sparse data aggregated from the randomized trials. Plaintiffs’ witnesses appeared to have testified that had the Cochrane been able to aggregate additional clinical trial data, the “effect size” would have remained constant, and the p-value would have shrunk, ultimately to below the level of 5 percent.  Prof. Platt, testifying for the defense, appropriately criticized this hand-waving excuse:

“Q. and the probability factor, the P value, was 0.065, which the previous witness had suggested is an increase in probability of our reliability on the underlying data.  Is it reasonable to assume that this data that a further increase in the sample size will achieve statistical significance?

A. No, that’s not a reasonable assumption….”

Notes of Testimony of Robert Platt at 29 (April 19, 2010).

Positions on Appeal

Dr. Viljoen continued to assert the need for significance on appeal. As appellant, he challenged the trial court’s finding that the Cochrane review concluded that there was a 40% risk reduction. See Goodman v. Viljoen, 2011 ONSC 821, at ¶192 (CanLII) (“I accept the finding of the Cochrane analysis that ACS reduces the instance of CP by 40%”). Dr. Viljoen correctly pointed out that the Cochrane review never reached such a conclusion. Appellant’s Factum, 2012 CCLTFactum 20936, ¶64.  It was the plaintiffs’ expert witnesses, not the Cochrane reviewers, who reached the conclusion of causality from the Cochrane data.

On appeal, Dr. Viljoen pressed the point that his expert witnesses described statistical significance in the Cochrane analysis would have been “a basic and universally accepted standard” for showing that ACS was efficacious in preventing CP or PVL. Id. at ¶40. The appellant’s brief then commits to the very error that Dr. Barrett complained would follow from a finding that did not have statistical significance; Dr. Viljoen maintained that the “trend” of reduced CP reduced CD rates from ACS administration “is the same as a chance occurrence.” Defendant (Appellant), 2012 CCLTFactum 20936, at ¶40; see also id. at ¶14(e) (arguing that the Cochrane result for ACS/CP “should be treated as pure chance given it was not a statistically significant difference”).

Relying upon the Daubert decision from the United States, as well as Canadian cases, Dr. Viljoen framed one of his appellate issues as whether the trial court had “erred in relying upon scientific evidence that had not satisfied the benchmark of statistical significance”:

“101. Where a scientific effect is not shown to a level of statistical significance, it is not proven. No study has demonstrated a reduction in cerebral palsy with antenatal corticosteroids at a level of statistical significance.

102. The Trial Judge erred in law in accepting that antenatal corticosteroids reduce the risk of cerebral palsy based on Dr. Willan’s unpublished Bayesian probability analysis of the 48 cases of cerebral palsy reviewed by Cochrane—an analysis prepared for the specific purpose of overcoming the statistical limitations faced by the Plaintiffs on causation.”

Defendant (Appellant), 2012 CCLTFactum 20936. The use of the verb “proven” is problematic because it suggests a mathematical demonstration, which is never available for empirical propositions about the world, and especially not for the biological world.  The use of a mathematical standard begs the question whether the Cochrane data were sufficient to establish a scientific conclusion of the efficacy of ACS in preventing CP.

In opposing Dr. Viljoen’s appeal, the plaintiffs capitalized upon his assertion that science requires a very high level of posterior probability for establishing a causal claim, by simply agreeing with it. See Plaintiffs’ (Respondents’) Factum,  2012 CCLTFactum 20937, at ¶31 (“The scientific method requires statistical significance at a 95% level.”).  By accepting the idealized notion that science somehow requires 95% certainty (as opposed to 95% confidence levels as a test for assessing random error), the plaintiffs made the defendant’s legal position untenable.

In order to keep the appellate court thinking that the defendant was imposing an extra-legal, higher burden of proof upon plaintiffs, the plaintiffs went so far as to misrepresent the testimony of their own expert witness, Professor Willan, as having committed the transposition fallacy:

“49. Dr. Willan provided the frequentist explanation of the Cochrane analysis on CP:

a. The risk ratio (RR) is .060 which means that there is a 40% risk reduction in cerebral palsy where there has been administration of antenatal corticosteroids;

b. The upper limit of the confidence interval (CI) barely crosses 1 so it just barely fails to meet the rigid test of statistical significance;

c. The p value represents the frequentist term of probability;

d. In this case the p value is .065;

e. From a statistical perspective that means that there is a 6.5% chance that the difference observed in CP rates is due to chance alone;

f. Conversely there is a 93.5% chance that the result (the 40% reduction in CP) is due to a true treatment effect of ACS.”

2012 CCLTFactum 20937, at ¶49 (citing Evidence of Dr. Willan, Respondents’ Compendium, Tab 4, pgs. 43-52).

Although Justice Doherty dissented from the affirmance of the trial court’s judgment, he succumbed to the parties’ misrepresentations about scientific certainty, and their prevalent commission of the transposition fallacy. Goodman v. Viljoen, 2012 ONCA 896 (CanLII) at ¶36 (“Scientists will draw a cause and effect relationship only when a result follows at least 95 per cent of the time. The results reported in the Cochrane analysis fell just below that standard.”), leave appeal den’d, Supreme Court of Canada No. 35230 (July 11, 2013).

The statistical errors on both sides redounded to the benefit of the plaintiffs.


[1] Goodman v. Viljoen, 2011 ONSC 821 (CanLII), aff’d, 2012 ONCA 896 (CanLII), leave appeal den’d, Supreme Court of Canada No. 35230 (July 11, 2013).

[2] Devender Roberts & Stuart R Dalziel “Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth,” Cochrane Database of Systematic Reviews, at 8, Issue 3. Art. No. CD004454 (2006).

[3] See, e.g., In re Ephedra Prods. Liab. Litig., 393 F.Supp. 2d 181, 191, 193 (S.D.N.Y. 2005) (fallaciously arguing that the use of a critical value of less than 5% of significance probability increased the “more likely than not” burden of proof upon a civil litigant.  Id. at 188, 193.  See also Michael O. Finkelstein, Basic Concepts of Probability and Statistics in the Law 65 (2009) (criticizing the Ephedra decision for confusing posterior probability with significance probability).

[4] I do not have the complete transcript of Dr. Barrett’s testimony, but the following excerpt from April 9, 2010, at page 100, suggests that he helped lead Justice Walters into error: “When you say statistical significance, if you say that something is statistically significance, it means you’re, for the scientific notation, 95 percent sure. That’s the standard we use, 95 percent sure that that result could not have happened by chance. There’s still a 5 percent chance it could. It doesn’t mean for sure, but 95 percent you’re sure that the result you’ve got didn’t happen by chance.”

[5] On appeal, the dissenting judge erroneously accepted Justice Walters’ description of the Cochrane review as having supposedly reported a 40% reduction in CP incidence, 93.5% of the time, from use of ACS. Goodman v. Viljoen, 2012 ONCA 896 (CanLII) at ¶36, leave appeal den’d, Supreme Court of Canada No. 35230 (July 11, 2013).

[6] The Bayesian analysis did not cure the attributability problem with respect to specific causation.

 

Goodman v Viljoen – Subterfuge to Circumvent Relative Risks Less Than 2

June 6th, 2014

Back in March, I wrote about a “Black Swan” case, in which litigants advanced a Bayesian analysis to support their claims. Goodman v. Viljoen, 2011 ONSC 821 (CanLII), aff’d, 2012 ONCA 896 (CanLII), leave appeal den’d, Supreme Court of Canada No. 35230 (July 11, 2013).

Goodman was a complex medical practice case in which Mrs. Goodman alleged that her obstetrician, Dr. Johan Viljoen, deviated from the standard of care by failing to prescribe antenatal corticosteroids (ACS) sufficiently in advance of delivery to reduce the risks attendant early delivery for her twin boys, of early delivery. Both boys developed cerebral palsy (CP). The parties and their experts agreed that the administration of ACS reduced the risks of respiratory distress and other complications of pre-term birth, but they disputed the efficacy of ACS to avoid or diminish the risk of CP.

According to the plaintiffs, ACS would have, more probably than not, prevented the twins from developing cerebral palsy, or would have diminished the severity of their condition.  Dr. Viljoen disputed both general and specific causation. Evidence of general causation came from both randomized clinical trials (RCTs) and observational studies.

Limitations Issue

There were many peculiar aspects to the Goodman case, not the least of which was that the twins sued Dr. Viljoen over a decade after they were born.  Dr. Viljoen had moved his practice in the passage of time, and he was unable to produce crucial records that supported his account of how his staff responded to Mrs. Goodman’s telephone call about signs and symptoms of labor. The prejudice to Dr. Viljoen illustrates the harshness of broad tolling statutes, the unfairness of which could be reduced by requiring infant plaintiffs to give notice of their intent to sue, even if they wait until the age of majority before filing their complaints.

State of the Art Issue

Dr. Viljoen suffered perhaps a more serious prejudice in the form of hindsight bias that resulted from the evaluation of his professional conduct by evidence that was unavailable when the twins were born in 1995. The following roughly contemporaneous statement from the New England Journal of Medicine is typical of serious thinking at the time of the alleged malpractice:

“Antenatal glucocorticoid therapy decreases the incidence of several complications among very premature infants. However, its effect on the occurrence of cystic periventricular leukomalacia, a major cause of cerebral palsy, remains unknown.”

Olivier Baud, Laurence Laurence Foix l’Hélias, et al., “Antenatal Glucocorticoid- Treatment and Cystic Periventricular Leukomalacia in Very Premature Infants,” 341 New Engl. J. Med. 1190, 1190 (1999) (emphasis added). The findings of this observational study illustrate some of the difficulties with the claim that Dr. Viljoen failed to prevent an avoidable consequence of pre-term delivery:

“Our results suggest that exposure to betamethasone but not dexamethasone is associated with a decreased risk of cystic periventricular leukomalacia.”

Id. at 1194. Results varied among various corticosteroids, among doses, among timing regimens.  There hardly seemed enough data in 1995 to dictate a standard of care.

Meta-Analysis Issues

Over ten years after the Goodman twins were born, the Cochrane collaboration published a meta-analysis that was primarily concerned with the efficacy of ACS for lung maturation. Devender Roberts & Stuart R Dalziel “Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth,” Cochrane Database of Systematic Reviews Issue 3. Art. No. CD004454 (2006). The trials included mostly post-dated the birth of the twins, and the alleged malpractice. The relevance of the trials to address the causation of CP in infants who experienced periventricular leukomalacia (PVL) was hotly disputed, but for now, I will gloss over the external validity problem of the Cochrane meta-analysis.

The Cochrane Collaboration usually limits its meta-analyses to the highest quality evidence, or RCTs, but in this instance, the RCTs did not include CP in its primary pre-specified outcomes. Furthermore, the trials were generally designed to ascertain short-term benefits from ACS, and the data in the trials were uncertain with respect to longer-term outcomes, which may have been ascertained differentially. Furthermore, the trials were generally small and were plagued by sparse data.  None of the individual trials was itself statistically significant at the 5 percent level.  The meta-analysis did not show a statistically significant decrease in CP from ACS treatment.  The authors reported:

“a trend towards fewer children having cerebral palsy (RR 0.60, 95% CI 0.34 to 1.03, five studies, 904 children, age at follow up two to six years in four studies, and unknown in one study).”

 Id. at 8 (emphasis added).

The Cochrane authors were appropriately cautious in interpreting the sparse data:

“Results suggest that antenatal corticosteroids result in less neurodevelopmental delay and possibly less cerebral palsy in childhood.”

Id. at 13-14 (emphasis added).

The quality of the trials included in the Cochrane meta-analysis varied, as did the trial methodologies.  Despite the strong clinical heterogeneity, the Cochrane authors performed their meta-analysis with a fixed-effect model. The confidence interval, which included 1.0, reflected a p-value of 0.065, but that p-value would have certainly increased if a more appropriate random-effects model had been used.

Furthermore, the RCTs were often no better than observational studies on the CP outcome. The RCTs here perhaps should not have been relied upon to the apparent exclusion of observational epidemiology.

Relative Risk Less Than Two

There is much to be said about the handling of statistical significance, the Bayesian analysis, the arguments about causal inference, but for now, let us look at one of the clearest errors in the case:  the inference of specific causation from a relative risk less than two.  To be sure, the Cochrane meta-analysis reported a non-statistically significant 40% decrease, but if we were to look at this outcome in terms of the increase in risk of CP from the physician’s failure to administer ACS timely, then the risk ratio would be 1.67, or a 67% increase.  On either interpretation, fewer than half the cases of CP can be attributed to the failure to administer ACS fully and timely in the case.

The parties tried their case before Justice Walters, in St. Catherines, Ontario. Goodman v. Viljoen, 2011 ONSC 821 (CanLII).  Justice Walters recognized that specific causation was essential and at the heart of the parties’ disagreement:

“[47] In order to succeed, the plaintiffs must establish that the failure to receive a full course of ACS materially affected the twins’ outcome. That is, they must establish that “but for” the failureto receive a full course of ACS, the twins would not have suffered from the conditions they now do, or that the severity of these afflictions would have been materially reduced.

[48] Not surprisingly, this was the most contentious issue at trial and the court heard a good deal of evidence with respect to the issue of causation.”

One of the defendant’s expert witnesses, Robert Platt, a professor of statistics at McGill University School of Medicine, testified, according to Justice Walters:

“[144] Dr. Platt also stated that the absolute risk in and of itself does not tell us anything about what might have happened in a specific case absent clinical and mechanistic explanations for that specific case.”

The plaintiffs’ expert witnesses apparently conceded the point.  Professor Andrew Willan, a statistician, testifying for the plaintiffs, attempted to brush Platt’s point aside by suggesting it would render clinical research useless, but that was hardly the point.  Platt embraced clinical research for what it could show about the “averages” in a sample of the population, even if we cannot discern causal efficacy retrospectively in a specific patient:

“[133] Dr. Willan also responded to Dr. Platt’s criticism that it was impossible to determine the distribution of the effect across the population. Professor Willan felt this issue was a red herring, and if it were valid, it would render most clinical research useless. There is really no way of knowing who will benefit from a treatment and who will not. Unless there are reasons to believe otherwise, it is best to apply the population average effect to each person.”

Although Willan labeled Platt’s point as cold-blooded and fishy, he ultimately concurred that the population average effect should be applied to each person in the absence of evidence of risk being sequestered in a subgroup.

A closer look at Willan’s testimony at trial is instructive. Willan acknowledged, on direct examination, that the plaintiffs were at increased risk, even if their mother had received a full course of ACS.  All he would commit to, on behalf of the plaintiffs, was that their risk would have been less had the ACS been given earlier:

“All we can say is that there’s a high probability that that risk would be reduced and that this is probably the best estimate of the excess risk for not being treated and I would say that puts that in the 70 percent range of excess risk and I would say the probability that the risk would have been reduced is into the 90 percentage points.”

Notes of Testimony of Andrew Willan at 62 (April 6, 2010).  The 90 percentage points reference here was Willan’s posterior probability that the claimed effect was real.

On cross-examination, the defense pressed the point:

Q. What you did not do in this, in this report, is provide any quantification for the reduction in the risk, true?

A. That’s correct.

Notes of Testimony of Andrew Willan at 35 (April 9, 2010)

Q. And you stated that there is no evidence that the benefits of steroids is restricted to any particular subgroup of patients?

A. I wasn’t given any. I haven’t seen any evidence of that.

Id. at 43.

Q. And what you’re suggesting with that statement, is that the statistics should be generally, should be considered by the court to be generally applicable, true?

A. That’s correct.

Id. at 44.

Q. But given your report, you can’t offer assistance on the clinical application to the statistics, true?

A. That’s true.

Id. at 46.

With these concessions in hand, defense counsel elicited the ultimate concession relevant to the “but for” standard of causation:

Q. And to do that by looking at an increase in risk, the risk ratio from the data must achieve 2 in order for there to be a 50 percent change in the underlying data, true?

A. Yeah, to double the risk, the risk ratio would have to be 2, to double the risk.

Id. at 63.

* * *

Q. So, none of this data achieves the threshold of a 50 percent change in the underlying data, whether you look at it as an increase in risk or …

A. Sure.

Q …. a decrease in risk …

A. Yeah.

Id. at 66.

Leaping Inferences

The legal standard for causation in Canada is the same counterfactual requirement that applies in most jurisdictions in the United States.  Goodman v. Viljoen, 2011 ONSC 821 (CanLII), at ¶14, 47. The trial court well understood that the plaintiffs’ evidence left them short of showing that their CP would not have occurred but for the delay in administering ACS. Remarkably, the court permitted the plaintiffs to use non-existing evidence to bridge the gap.

According to Dr. Max Perlman, plaintiffs’ expert witness on neonatology and pediatrics, CP is not a dichotomous condition, but a spectrum that is manifested on a continuum of signs and symptoms.  The RCTs relied upon had criteria for ascertaining CP and including it as an outcome.  The result of these criteria was that CP was analyzed as a binary outcome.  Dr. Perlman, however, held forth that “common sense and clinical experience” told him that CP is not a condition that is either present or not, but rather presented on a continuum. Id. at [74].

Without any evidence, Perlman testified that when CP is not avoided by ACS, “it is likely that it is less severe for those who do go on to develop it.” Id. [75].  Indeed, Perlman made the absence of evidence a claimed virtue; with all his experience and common sense, he “could not think of a single treatment which affects a basic biological process that has a yes or no effect; they are all on a continuum.” Id. From here, Perlman soared to his pre-specified conclusion that “that it is more likely than not that the twins would have seen a material advantage had they received the optimal course of steroids.” Id. at [76].

Perlman’s testimony is remarkable for inventing a non-existing feature of biological evidence:  everything is a continuum. Justice Walters could not resist this seductive testimony:

“[195] The statistical information is but one piece of the puzzle; one way of assessing the impact of ACS on CP. Notably, the 40% reduction in CP attributable to ACS represents an all or nothing proposal. In other words, 93.5% of the time, CP is reduced in its entirety by 40%. It was the evidence of Dr. Perlman, which I accept, that CP is not a black and white condition, and, like all biological processes, it can be scaled on a continuum of severity. It therefore follows that in those cases where CP is not reduced in its entirety, it is likely to be less severe for those who go on to develop it. Such cases are not reflected in the Cochrane figure.

[196] Since the figure of 40% represents an all or nothing proposal, it does not accurately reflect the total impact of ACS on CP. Based on this evidence, it is a logical  conclusion that if one were able to measure the total effect of ACS on CP, the statistical measure of that effect would be inflated beyond 40%.

[197] Unfortunately, this common sense conclusion has never and can never be tested by science. As Dr. Perlman testified, such a study would be impossible to conduct because it would require pre-identification of those persons who go on to develop CP.  Furthermore, because the short term benefits of ACS are now widely accepted, it would be unethical to withhold steroids to conduct further studies on long term outcomes.”

Doubly unfortunate, because Perlman’s argument was premised on a counterfactual assumption.  Many biological phenomena are dichotomous.  Pregnancy, for instance, does not admit of degrees.  Disease states are frequently dichotomous, and no evidence was presented that CP was not dichotomous. Threshold effects abound in living organisms. Perlman’s argument further falls apart when we consider that the non-experimental arm of the RCTs would also have had additional “less-severe” CP cases, with no evidence that they occurred disproportionately in the control arms of these RCTs. Furthermore, high-quality observational studies might have greater validity than post-hoc RCTs in this area, and there have been, and likely will continue to be, such studies to attempt better understanding of the efficacy of ACS, as well as differing effects among the various corticosteroids, doses, and patterns of administration.

On appeal, the Justice Walters’ verdict for plaintiffs was affirmed, but over a careful, thoughtful dissent. Goodman v. Viljoen, 2012 ONCA 896 (CanLII) (Doherty, J., dissenting). Justice Doherty caught the ultimate futility of Dr. Perlman’s opinion based upon non-existent evidence: even if there were additional sub-CP cases in the treatment arms of the RCTs, and if they occurred disporportionately more often in the treatment than in the placebo arms, we are still left guessing about the quantitative adjustment to make to the 40% decrease, doubtful as it was, which came from the Cochrane review.

Recrudescence of Traumatic Cancer Claims

June 4th, 2014

In 1991, Peter Huber, discussing traumatic cancer claims, wrote:

“After years of floundering in the junk science morass of traumatic cancer, judges slowly abandoned sequence-of-events logic, turned away from the sympathetic speculations of family doctors, and struggled on to the higher and firmer ground of epidemiology and medical science.  Eventually, the change of heart among appellate judges was communicated back down to trial judges and worker’s compensation boards, and traumatic cancer went into almost complete remission.”

Peter W. Huber, Galileo’s Revenge: Junk Science in the Courtroom 55-56 (1991).

With the advent of Daubert and meaningful gatekeeping of expert witness opinion testimony, the traumatic cancer claims did recede. For a while. Plaintiffs’ counsel, and stalwart opponent of epistemic standards for scientific claims in court, Kenneth Chesebro attacked Huber’s précis of the traumatic cancer law and science. Kenneth J. Chesebro, “Galileo’s Retort: Peter Huber’s Junk Scholarship,” 42 Am. Univ. L. Rev. 1637 (1993). Defenses of the dubious science continue to appear, although mostly in non-peer-reviewed publications.[1]

One of the more disturbing implications of the West Virginia Supreme Court’s decision in Harris v. CSX Transportation, Inc., 232 W.Va. 617, 753 S.E.2d 275 (2013), was the Court’s reliance upon its own, recent approval of traumatic cancer claims.  The Harris Court cited, with approval, a 2002 traumatic cancer case, State ex rel. Wiseman v. Henning, 212 W.Va. 128, 569 S.E.2d 204 (2002).  The Wiseman case involved a specious claim that a traumatic rib injury caused multiple myeloma, a claim at odds with scientific method and observation.  The West Virginia Supreme Court blinked at the challenge to the physician expert witness who advanced the causal claim in Wiseman; and in Harris, the Court made clear that blinking is what trial courts should do when confronted with methodological challenges to far-fetched causal opinions.

A couple of years ago, the New York Times ran an article about traumatic cancer. C. Claiborne Ray, “Injury and Insult” (Nov. 5, 2012), responding to the question “Is it possible for cancer to develop as a result of an injury?” Here is how Times science reporter responded:

A.It’s a common myth that injuries can cause cancer,” the American Cancer Society says on its Web site. Until the 1920s, some doctors believed trauma did cause cancer, “despite the failure of injury to cause cancer in experimental animals.” But most medical authorities, including the cancer society and the National Cancer Institute, see no such link. The more likely explanation, the society suggests, is that a visit to the doctor for an injury could lead to finding an existing cancer.

Other possibilities are that scar tissue from an old trauma could look like a cancerous lesion and that an injured breast or limb would be more closely watched for cancer to develop.

Ms. Ray went on to note a published study, in which would-be myth-busters presented observational data purportedly showing a relationship between physical injury and subsequent breast cancer.  The paper cited by Ms. Ray was a report on a small case-control study done by investigators at the Department of Geography, Lancaster University. See Jan Rigby, et al., “Can physical trauma cause breast cancer?” 11 Eur. J. Cancer. Prev. 307 (2002). The study consisted of 67 breast cancer cases and 134 controls, matched on age, family history, age of menarche, parity, age at first birth, and menopausal status.

Not surprisingly, considering its small size, the Rigby study reported no statistically significant differences for several factors known to be associated with breast cancer: social class, education, residence, smoking and alcohol consumption.  Although lacking power to detect differences of known risk factors, this study turned up a large, statistically significant association between physical trauma and breast cancer:

“Women with breast carcinoma were more likely to report physical trauma to the breast in the previous 5 years than were the controls (odds ratio (OR) 3.3, 95% confidence interval (CI) 1.3-10.8, P < 0.0001).”

* * * * *

“More likely to [self-]report” hardly implies causation, but the authors jumped not only to a causal explanation but to a causal conclusion:

* * * * *

“In conclusion, recall bias is an unlikely explanation for these results in view of the nature and severity of physical trauma. Models of epithelial cell generation indicate that a causal link between physical trauma and cancer is plausible. A latent interval between cancer onset and presentation of under 5 years is also plausible. The most likely explanation of the findings is that physical trauma can cause breast cancer.”

Rigby at 307.

The Rigby study is a valuable demonstration of how malleable researchers can be in discovering plausible explanations for their data.  The authors fail to discuss the natural history of breast carcinoma, such as tumor doubling time, which would make their five-year window decidedly implausible.  The Rigby paper also demonstrates how strident researchers can be in claiming that they have produced a study that has eliminated bias in observational research, when they have barely scratched the surface of bias or confounding. Magical thinking is not the exclusive domain of lawyers.

Until reading the Harris and Wiseman cases, I had thought that the legal system had graduated from the “mythology” of traumatic cancer cases.[2]  To be sure, in the past, any number of physicians have supported traumatic cancer claims, in print and in the courtroom.[3] Some authors attempted to put some rational limits on the extent of the traumatic cancer claims.[4] By 1947, at least, the trauma theory was criticized in leading texts.[5]  In 1974, the Mayo Clinic published a review that emphasized the lack of experimental evidence to support the claim that uncomplicated trauma causes cancer.[6] The law review literature attempted to make sense of the compensation-frenzied courts, without much success.[7]

Many cases from most jurisdictions have approved traumatic cancer claims.  Some are set out below. Some courts heroically resisted the pro-compensation Zeitgeist, usually on case-specific evidentiary issues.[8]

In New York, judges seem to be well aware that post hoc ergo propter hoc is a fallacy.  Cassano v. Hagstrom, 5 N.Y.2d 643, 159 N.E.2d 348, 187 N.Y.S.2d 1 (1959) (affirming dismissal of case based because of plaintiffs’ attempt to use fallacious reasoning in the form of  “post hoc ergo propter hoc”); Holzberg v. Flower & Fifth Ave. Hosps., 39 AD 2d 526 (N.Y. 1st Dep’t 1972). Still, the New York courts struggled with traumatic cancer claims, and appeared to oscillate wildly without clear guidance on whether or to what extent the courts could reject specious claiming supported by speculative or unreliable expert witness opinion testimony.[9] Given the current hostility to gatekeeping of expert witness opinion, a recrudescence of traumatic cancer claims is likely.

Opinions Approving Causation in Traumatic Cancer Cases

California

Santa Ana Sugar Co. v. Industrial Accid. Comm’n, 170 P. 630, 630 (Cal. Dist. Ct. App. 1917)

Colorado

Canon Reliance Coal Co. v. Indus. Comm’n, 72 Colo. 477, 211 P. 868, 869-70 (1922) (cancer caused by being hit on cheek with a lump of coal)

Georgia

National Dairy Prods. Corp. v. Durham, 154 S.E.2d 752, 753-54 (Ga. Ct. App. 1967)

Kentucky

Louisville Ry v. Steubing’s Adm’r, 136 S.W. 634, 634 (Ky. Ct. App. 1911)

Louisiana

Reed v. Mullin Wood Co., 274 So. 2d 845, 846-47 (La. Ct. App. 1972), cert. denied, 275 So. 2d 729, 791 (La. 1973);

Thompson v. New Orleans Ry. & Light Co., 83 So. 19, 20 (La. 1919)

Michigan

Wilson v. Doehler-Jarvis Div. of Nat’l Lead Co., 353 Mich. 363, 91 N.W.2d 538, 539-40 (1958) (blow to lip caused cancer)

Mooney v. Copper Range RR, 27 N.W.2d 603, 604 (Mich. 1947)

Minnesota

Daly v. Bergstedt, 267 Minn. 244, 126 N.W.2d 242, 247–48 (1964) (affirming jury finding of causation between traumatic leg fracture and breast cancer; six physicians testified against causation; one stated cancer “could” result from trauma; imagining that scientific and legal standards of causation differ)

Pittman v. Pillsbury Flour Mills, Inc., 48 N.W.2d 735, 736 (Minn. 1951)

Hertz v. Watab Pulp & Paper Co., 237 N.W. 610, 611 (Minn. 1931)

Austin v. Red Wing Sewer Pipe Co., 163 Minn. 397, 204 N.W. 323, 323-24 (Minn. 1925) (cancer developed one year after worker was hit in the face with coal)

Gaetz v. City of Melrose, 193 N.W. 691, 692 (Minn. 1923)

Missouri

Vitale v. Duerbeck, 338 Mo. 536, 92 S.W.2d 691, 695 (1936)

New Hampshire

Jewell v. Grand Trunk Ry, 55 N.H. 84 (1874) (reversing traumatic cancer verdict on other grounds)

New Mexico

White v. Valley Land Co., P.2d 707, 708-10 (N.M. 1957)

Ohio

Hanna v. Aetna Ins., 24 Ohio Misc. 27, 52 Ohio Op. 2d 316, 259 N.E.2d 177, 177-79 (Ohio Mun. Ct. Dayton 1970)(breast lump found three months after car accident)

Glenn v. National Supply, 129 N.E.2d 189, 190-91 (Ohio Ct. App. 1954)

Oregon

Devine v. Southern Pacific Co., 207 Or. 261, 295 P.2d 201 (1956) (holding that physician’s testimony as to “probable” causation between shoulder fracture and lung cancer was sufficient; jury verdict for plaintiff reversed on other grounds).

Pennsylvania

Baker v. DeRosa, 413 Pa. 164, 196 A.2d 387, 389–90 (Pa. 1964)

Menarde v. Philadelphia Transp. Co., 376 Pa. 497, 103 A.2d 681, 684(1954) (the fact that breast cancer was found in the same place as the injury-caused bruise helped establish causation);

Southern S.S. Co. v. Norton, 41 F. Supp. 103 (E.D. Pa. 1940) (trauma to skull and lower back held to have caused lung cancer)

Tennessee

Koehring-Southern & Am. Mut. Ins. Co. v. Burnette, 464 S.W.2d 820, 821 (Tenn. 1970)

Boyd v. Young, 193 Tenn. 272, 246 S.W.2d 10, 10 (Tenn. 1951)

Rhode Island

Valente v. Bourne Mills, 77 R.I. 274, 278-79, 75 A.2d 191, 193-94 (1950) (adopting house of cards position in which any rational inference suffices even if not supported by expert medical opinion)

Emma v. A.D. Julliard & Co., 75 R.I. 94, 63 A.2d 786, 787-89 (R.I. 1949)(plaintiff had malignant tumor removed from her breast seven weeks after being hit with a can of juice)

Texas

Traders & General Insur. Co. v. Turner, 149 S.W.2d 593, 597-98 (Tex. Civ. App. 1941) (testicular cancer)

Virginia

Ellis v. Commonwealth Dep’t of Highways, 28 S.E.2d 730, 731-32, 735 (Va. 1944) (accepting post-hoc reasoning “[f]acts prevail over possibilities or probabilities”)

Winchester Milling Corp. v. Sencindiver, 138 S.E. 479, 480-81 (Va. 1927)


[1] See, e.g., Melvin A. Shiffman, Can Trauma Cause or Accelerate the Growth of Cancer? Forensic Examiner 6 (Fall 2004).

[2] See Manasco v. Insurance Co. of State of Pennsylvania, 89 S.W.3d 239 (Tex. App. Texarkana 2002) (affirming denial of benefits to worker who claimed head injury caused brain tumor; citing to epidemiological studies that failed to show an association between trauma and brain tumors).

[3] See, e.g., George R. Parsons, “Sufficiency of Proof in Traumatic Cancer Cases,” 2 Tort & Med. Year Book 335 (1962); Stoll & Crissey, “Epithelioma from Single Trauma,” 62 N.Y. St. J. Med. 496 (Feb. 15, 1962); Wilhelm C. Hueper, Trauma and Cancer (1959); Arden R. Hedge, “Can a Single Injury Cause Cancer?” 90 Calif. Med. 55 (1959); R. Crane, “The Relationship of a Single Act of Trauma to Subsequent Malignancy,” in Alan R. Moritz & David S. Helberg, eds., Trauma and Disease 147 (1959); Shields Warren, M.D., “Minimal criteria required to prove causation of traumatic or occupational neoplasms,” Ann. Surgery 585 (1943); Bishop, “Cancer, Trauma, and Compensation,” 32 So. Med. J. 302 (1939); Knox, “Trauma and Malignant Tumors, 26 Am. J. Surg. 66, 69-70 (1934); William B. Coley & Norman L. Higinbotham, “Injury as a causative factor in the development of malignant tumors,” 98 Ann. Surg. 991 (1933); Wainwright, “Single Trauma, Carcinoma and Workman’s Compensation,” 5 Am. J. Surg. 433 (1928); Alson R. Kilgore & Curtis E. Smith, “Industrial liability for cancer,” 25 Calif. & Western Med. 70 (1926); Charles Phelps, “The relation of trauma to cancer formation,” 51 Ann. Surgery 609 (1910).

[4] James Ewing, “Modern Attitudes Toward Traumatic Cancer,” 19 Arch. Path. 690, 692 (1935); James Ewing, “The Relation of Trauma to Malignant Tumors,” Am. J. Surg. 30, 31-34 (Feb. 1926).

[5] See, e.g., James A. Tobey, Public Health Law 321 (3ed 1947) (“Although there is little, if any, scientific evidence to prove conclusively that malignant growths such as carcinoma, sarcoma, and other forms of cancer are ever caused by single blows, wounds, injuries, or other forms of trauma, the courts have awarded damages in a number of instances to persons who have developed cancers following single injuries.”) (internal citations omitted).

[6] George R. Monkman, Gregg Orwoll & John C. Ivins, “Trauma and Oncogenesis,” 49 Mayo Clinic Proc. 157 (1974).

[7] The trauma theory of carcinogenesis was discussed and questioned in several law review articles.  See, e.g., Orrin E. Tilevitz, “Judicial Attitudes Towards Legal and Scientific Proof of Cancer Causation,” 3 Colum. J. Envt’l L. 344 (1977); Donald J. Ladanyi, “Impact Trauma As ‘Legal Cause’ of Cancer,” 20 Cleveland State L. Rev. 409 (1971); Theodore Dyke, “Traumatic Cancer?” 15 Clev.-Marshall L. Rev. 472 (1966); Jerry G. Elliott, “Traumatic cancer and ‘an old misunderstanding between doctors and lawyers’,” 13 U. Kan. L. Rev. 79 (1964); Comment, Sufficiency of Proof in Traumatic Cancer: A Medico-Legal Quandary, 16 Ark. L. Rev. 243 (1962); Comment, “Sufficiency of Proof in Traumatic Cancer Cases,” 46 Cornell L.Q. 581 (1961); Adelson, Injury and Cancer, 5 Western Res. L. Rev. 150 (1954).

[8] State Compensation Ins. Fund v. Kindig, 445 P.2d 72 (Colo. 1968) (head injury held not to have caused leukemia 68 days later); Slack v. C.L. Percival Co., 198 Iowa 54, 199 N.W. 323, 326 (1924) (anticipating Daubert by rejecting expert witness opinion that was “wholly in the realm of conjecture, speculation, and surmise”); Ortner v. Zenith Carburetor Co., 207 Mich. 610, 175 N .W. 122 (1919) (holding that 30 months was too long for a claim that accident that crushed worker’s fingers caused blood poisoning and penile cancer); Stordahl v. Rush Implement Co., 417 P.2d 95 (Mont. 1966) (rejecting traumatic causation of malignant tumor); Tonkovich v. Dep’t of Lab. & Indus., 31 Wash. 2d 220, 195 P.2d 638 (1948) (injury to foot held not to have caused abdominal cancer)

[9] See Dennison v. Wing, 279 App. Div. 494, 110 N.Y.S.2d 811, 813 (1952) (rejecting cancer claim when latency was two months on grounds that cancer took longer to develop); Sikora v. Apex Beverage Corp., 282 App. Div. 193, 196-97 (1953) (reversing judgment for plaintiff based upon jury’s finding that slip and fall accelerated breast cancer based upon lack of evidentiary support), aff’d, 306 N.Y. 917, 119 N.E.2d 601 (1954); Frankenheim v. B. Altman & Co., 13 Misc. 2d 1079, 1080-81, 177 N.Y.S.2d 2 (Bronx Cty. S.Ct. 1958) (granting motion to set aside verdict for plaintiff based upon traumatic cancer claim on grounds of insufficient evidence), app. dism’d, 8 App. Div. 2d 809 (First Dep’t 1959). But see McGrath v. Irving, 24 App. Div. 2d 236, 265 N.Y.S.2d 376 (1965) (affirming jury verdict based upon claim that plaintiff’s swallowing glass in car accident caused or accelerated development of laryngeal cancer); Mattfield v. Ward Baking Co., 14 App. Div. 2d 942, 221 N.Y.S.2d 224, 224 (1st Dep’t 1961) (affirming award for traumatic cancer based upon the “usual” conflicting expert witness testimony) Mattfield v. Ward Baking Co., 14 App. Div. 2d 942, 942 (1961) (affirming workman’s compensation award for “aggravation” of cancer, which resulted after “the usual conflict of medical opinion”); Pezzolanti v. Green Bus Lines, 114 App. Div. 2d 553, 553-54, 494 N.Y.S.2d 168, 169 (1985) (affirming workman’s compensation award for disability to wrist, which resulted from “trauma” of hitting pothole, which in turn injured asymptomatic wrist destabilized by pre-existing cancer).

Intellectual Due Process in West Virginia and Beyond

June 1st, 2014

Harris v. CSX Transportation

I have borrowed and modified the phrase “Intellectual Due Process” from earlier writers because of its obvious implications for the presentation, interpretation, synthesis, and evaluation of scientific evidence in court. See Scott Brewer, “Scientific Expert Testimony and Intellectual Due Process,” 107 Yale L. J. 1535 (1998). The major reason courts write opinions is to explain and justify their decisions to litigants, present and future, and to a wider audience of lawyers, scholars, and the general public. Judicial opinions involving scientific evidence, whether in legislation, regulation, or litigation must satisfy the societal need to explain and justify the acceptance and rejection of scientific claims. Despite a great deal of hand waving that law and science are somehow different, in the end, when courts describe their acceptance or rejection of scientific claims, they are addressing the same epistemic warrant that scientists themselves employ. Even a cursory review of the judicial output reveals an unsatisfactory state of affairs in which many courts mangle scientific and statistical evidence and inference.  There is much that is needed to correct the problem.

One proposal would be to require that the parties file proposed findings of facts in connection with Rule 702 gatekeeping challenges.  Courts should file detailed findings of facts that underlie their decisions to admit or to exclude expert witness opinion testimony.  Another proposal would require courts to cite properly the scientific studies that they discuss in reaching a legal conclusion about sufficiency or admissibility.  These are small steps, but ones that would help reduce the gross inaccuracies and the glib generalizations, while increasing the opportunity for public scrutiny and criticism.

We do not think anything is amiss with special courts for tax, patent, family law, national security, equity, or commercial matters.  There is an even greater need for scientific skill, knowledge, and aptitude in a specialized science court.  The time has come for special courts to hear cases involving scientific claims in health effects and other litigation.

*   *   *   *   *   *   *

A decision of the West Virginia Supreme Court, late last year, illustrates the need for substantial reform of how claiming based upon “scientific evidence” is permitted and evaluated in court.  Mrs. Harris sued the railroad for the wrongful death of her husband, who died of multiple myeloma. Mr. Harris had been exposed, in his railroad workplace, to diesel exhaust, which Mrs. Harris claimed caused his cancer. See Harris v. CSX Transportation, Inc., 232 W.Va. 617, 753 S.E.2d 275 (2013). The trial court excluded Mrs. Harris’s expert witnesses. Harris v. CSX Transportation, Inc., No. 12-1135, 2012 WL 8899119 (Cir. Ct. Marshall Cty., W.Va. Aug. 21, 2012).

1. The West Virginia Supreme Court reversed the trial court’s exclusion of witnesses on the basis of an asymmetrical standard of review, which would allow de novo review of trial court decisions to exclude expert witness opinions, but which would privilege trial court decisions to admit opinions by limiting appellate review to abuse of discretion. This asymmetry was, of course, the same dodge that the Third and Eleventh Circuits had used to keep the “gates open,” regardless of validity or reliability concerns, and the same dodge that the Supreme Court shut down in General Electric v. Joiner. A single judge dissented in Harris, Justice Loughry, who took the majority to task for twisting facts and law to get to a desired result.

2. The Harris Court cited a federal court case for dicta that “Rule 702 reflects an attempt to liberalize the rules governing the admissibility of expert testimony.” See Harris, 753 S.E.2d at 279 (citing and quoting from Weisgram v. Marley Co., 169 F.3d 514, 523 (8th Cir.1999). Remarkably, the Harris Court omitted reference to the United States Supreme Court’s unanimous affirmance of Weisgram, which saw Justice Ginsburg write that “[s]ince Daubert, moreover, parties relying on expert evidence have had notice of the exacting standards of reliability such evidence must meet.” Weisgram v. Marley Co., 528 U.S. 440, 442 (2000).  The Harris Court’s lack of scholarship is telling.

3. Meta-analysis appeared to play a role in the case, but the judicial decisions in Harris fail to describe the proffered evidence. The majority in Harris noted that one of plaintiff’s expert witnesses, Dr. Infante, relied upon a meta-analysis referred to as “Sonoda 2001.” Harris, 753 S.E.2d at 309. Neither the Court nor the dissent cited the published meta-analysis in a way that would help an interested reader in finding the paper.  One could imagine the hue and cry if courts cited judicial cases or statutes by short-hand names without providing enough information to access the relied upon source.  In this case, a PubMed search reveals the source so perhaps the error is harmless. Tomoko Sonoda, Yoshie Nagata, Mitsuru Mori, Tadao Ishida & Kohzoh Imai, “Meta-analysis of multiple myeloma and benzene exposure,” 11. J. Epidemiol. 249 (2001).  Still, the time has come for courts to describe and report the scientific evidence with the same care and detail that they would use in a car collision case.

4. A quick read shows that the Sonoda meta-analysis supports the dissent’s assessment:

“‘Dr. Infante testified on direct examination that Sonoda 2001 considered 8 case-control studies specific to engine exhaust and stated it concluded that diesel and non-diesel engine exhaust causes multiple myeloma.’ Yet, as the trial court found, ‘[o]n cross examination Dr. Infante acknowledged that none of the 8 papers included in the Sonoda meta-analysis mention diesel exhaust’.”

Harris, 753 S.E.2d at 309.  The dissent would have been considerably more powerful had it actually adverted to the language of Sonoda 2001:

“These results suggested that benzene exposure itself was not likely to be a risk factor of MM [multiple myeloma]. It is thought that several harmful chemical agents in engine exhaust, other than benzene, could be etiologically related to the risk of MM. Further case-control studies on MM are needed to obtain more information about detailed occupational exposure to toxic substances.”

Sonoda at 249 (2001) (emphasis added).  Contrary to Infante’s asseveration, Sonoda and colleagues never concluded that diesel exhaust causes multiple myeloma.  The state of scholarship and “intellectual due process” makes it impossible to tell whether or not Dr. Infante was telling the truth or the Harris Court badly misunderstood the record. Either way, something must give.

The dissent went on to note that Dr. Infante conducted his own meta-analysis, which included studies that did not mention diesel exhaust. Harris, 753 S.E.2d at 309.  The railroad complained that some of the studies were small and had limited power, but that is exactly why a meta-analysis would be appropriate.  The more disturbing complaints were that the meta-analysis left out important studies, and that it included irrelevant studies of benzene exposure and myeloma, which raised insuperable problems of external validity.

5. A half empty glass that is always full.  According to the Harris Court, the West Virginia shadow of Rule 702 is a rule of “admissibility rather than exclusion.” Harris, 753 S.E.2d at 279 (citing and quoting from In re Flood Litig. Coal River Watershed, 222 W.Va. 574, 581, 668 S.E.2d 203, 210 (2008), which in turn quoted a federal case, Arcoren v. United States, 929 F.2d 1235, 1239 (8th Cir. 1991), decided before the Supreme Court decided Daubert.)  This is just silly hand waving and blatant partisanship.  A rule that sets out criteria or bases for admissibility also demarcates the inadmissible.

6. Cherry Picking. Dr. Infante was permitted by the Harris Court to aggregate data from studies that did not observe diesel exposure, while he failed to include, or he deliberately excluded data from, a large, powerful, exonerative study conducted by scientists from the National Cancer Institute, the International Agency for Research on Cancer (IARC), and the Karolinska Institute. See Paolo Boffetta, Mustafa Dosemeci, Gloria Gridley, Heather Bath, Tahere Moradi and Debra Silverman, “Occupational exposure to diesel engine emissions and risk of cancer in Swedish men and women,” 12 Cancer Causes Control 365 (2001). Dr. Infante inexplicably excluded this study, which found a risk ratio for men exposed to diesel exhaust that was below one, 0.98, with a very narrow 95% confidence interval, 0.92-1.05. Boffetta at 368, Table 2.

7. The West Virginia articulated an incohorent definition of “reliable,” designed to give itself the ability to reject gatekeeping completely. Citing its earlier decision in Flood, the Court offered its own ipse dixit:

“The assessment of whether scientifically-based expert testimony is “reliable,” as that term is used in [Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993), and Wilt v. Buracker, 191 W.Va. 39, 443 S.E.2d 196 (1993)], does not mean an assessment of whether the testimony is persuasive, convincing, or well-founded. Rather, assessing ‘reliability’ is a shorthand term of art for assessing whether the testimony is to a reasonable degree based on the use of knowledge and procedures that have been arrived at using the methods of science — rather than being based on irrational and intuitive feelings, guesses, or speculation. If the former is the case, then the jury may (or may not, in its sole discretion) ‘rely upon’ the testimony. In re Flood Litig., 222 W.Va. at 582 n. 5, 668 S.E.2d at 211 n. 5.”

Harris, 753 S.E.2d at 279-80. Surely, this is circular or vacuous or both. Opinions not “well-founded” will be ones that are based upon guesses or speculation.  Opinions arrived at by the “methods of science” will be ones that have an epistemic warrant that will survive a claim that they are not “well-founded.”

8. The Harris Court evidenced its hostility to scientific evidence by dredging up one of its own decisions involving a multiple myeloma causation claim, State ex rel. Wiseman v. Henning, 212 W.Va. 128, 569 S.E.2d 204 (2002).  Wiseman involved a specious claim that a traumatic rib injury caused multiple myeloma, a claim at odds with scientific method and observation:

“Some research has suggested that people in some jobs may have an increased risk of developing multiple myeloma because they are exposed to certain chemicals. But the International Agency for Research on Cancer (IARC) states that the evidence is limited overall. It has been suggested that people may have an increased risk if they work in the petrol or oil industry, farming, wood working, the leather industry, painting and decorating, hairdressing, rubber manufacturing or fire fighting. But there is no evidence to prove that any of these occupations carry an increased risk of myeloma.”

Cancer Research UK, “Myeloma risks and causes” (last visited May 28, 2014). Even the most non-progressive jurisdictions have generally eradicated specious claiming for trauma-induced cancers, but West Virginia has carved out a place second to none in its race to the bottom.

9. WOE.  Not surprisingly, the Harris Court relied heavily on the First Circuit’s “weight of the evidence” end-run around the notion of epistemic warrant for scientific claims, citing Milward v. Acuity Specialty Products Group, Inc., 639 F.3d 11 (1st Cir.2011), cert. denied sub nom., U.S. Steel Corp. v. Milward, ___ U.S. ___, 2012 WL 33303 (2012). The Harris Court went on to conflate and confuse WOE with Bradford Hill, and cited a recent New York case that confidently saw through WOE hand waving, while ignoring its devasting critique of expert witnesses’ attempts to pass off WOE for scientific, epistemic warrant.  Reeps ex rel. Reeps v. BMW of N. Am., LLC, No. 100725/08,

2013 WL 2362566, at *3, 2012 N.Y. Misc. LEXIS 5788; 2012 NY Slip Op 33030U  (N.Y. Sup. Ct. May 10, 2013).

10.  Link.  Dr. Infante links a lot, even when his sources do not:

“Dr. Infante testified that the International Agency for Research on Cancer issued Technical Publication Number 42 in 2009, and that the publication stated that diesel exhaust exposures have been linked to multiple myeloma and leukemia.”

Harris, 753 S.E.2d at 294. The Harris Court neglected to give the title of the publication, which tells a different story.  Identification of research needs to resolve the carcinogenicity of high-priority IARC carcinogens. The dissent was willing to go behind the conclusory and false characterization that Dr. Infante and plaintiff gave to this publication.  Harris, 753 S.E.2d at 309. The trial court’s finding (and the dissent’s assertion) that the IARC Technical Publication 42 intended to express a research agenda, not to make a causation statement, seems unassailable.  Furthermore, it appears to be precisely the sort of specious claim that a court should keep from a jury.  The cited IARC source actually notes that the then current IARC classification of diesel exhaust was of inadequate evidence for human carcinogenicity, with a focus on lung cancer, and barely a mention of multiple myeloma.

11.  The Benzene Connection. Plaintiffs’ expert witnesses, including Dr. Infante, argued that benzene was a component of diesel exhaust, and benzene caused multiple myeloma.  This move ignored not only the lack of evidence to implicate benzene in the causation of multiple myeloma, but it also ignored the large quantitative differences between the benzene occupational exposure studies and the very small amounts of benzene in diesel exhaust.  The Harris Court held that the trial court acted improperly by inquiring into and finding the following facts, which were “exclusively” for the jury:

  • “There is substantially more benzene in cigarette smoke than diesel exhaust.
  • Benzene is present only in trivial doses in diesel exhaust.
  • The hypothesis that diesel exhaust causes multiple myeloma is confounded by the fact that cigarette smoking does not.”

The Harris majority further chastised the trial court for adverting to the ten or so studies that failed to find a statistically significant association between benzene exposure and multiple myeloma.  Harris, 753 S.E.2d at 305-06.  This inquiry directly calls into question, however, Dr. Infante’s methodology.

If these facts, found by the trial court, were reasonably established, then Dr. Infante’s argument was less than bogus, and a major underpinning for inclusion of benzene studies in his meta-analysis was refuted.  These are precisely the sort of foundational facts that must be part of an inquiry into the methodological grounds of an expert witness’s opinion.

12.  The Harris Court confused “proving causation” with “showing a methodology that provides an epistemic warrant for concluding.” Harris, 753 S.E.2d at 300. The Harris Court asserted that the trial court exceeded its gatekeeping function by inquiring into whether Mrs. Harris’s expert witnesses “proved” causation. Harris, 753 S.E.2d at 300. Speaking of “proof of” or “proving” causation is an affectation of lawyers, who refer to their evidence as their “proofs.”  Epidemiologic articles and meta-analyses do not end with quod erat demonstrandum. Beyond the curious diction, there is a further issue in the majority’s suggestion that the trial court set the bar too high in declaring that the plaintiff failed to “prove” causation.  Even if we were to accept the continuous nature of strength of evidence for a causal conclusion, Dr. Infante and the other plaintiff’s witnesses, would be fairly low on the curve, and their lowly position must of necessity speak to the merits of the defense motion to exclude under Rule 702.

13. Purely Matters for Jury. The Harris Court criticized the trial court for conducting a “mini-trial,” which set out to “resolve issues that were purely matters for jury consideration.” Harris, 753 S.E.2d at 305. In holding that the matters addressed in the pre-trial hearing were “exclusively grist for the jury and which had no relevancy to the limited role the trial court had under the facts of this case,” the Harris Court displayed a profound disregard for what facts would be relevant for a challenge to the plaintiff’s expert witnesses’ methodology. Many of the facts found by the trial court were directly relevant to “general acceptance,” validity (internal and external) of studies relied upon, and reliability of reasoning and inferences drawn. Aside from the lack of general acceptance and peer review of the plaintiff’s claimed causal relationship, the proffered testimony was filled with gaps and lacunae, which are very much at issue in methodological challenges to an opinion of causality.

*   *   *   *   *   *   *

The Harris case has taken its place next to Milward in the litigation industry’s arsenal of arguments for abandoning meaningful judicial supervision and gatekeeping of expert witness opinion testimony.  See Andrew S. Lipton, “Proving Toxic Harm: Getting Past Slice and Dice Tactics,” 45 McGeorge L. Rev. 707, 731 (2014) (plaintiffs’ bar cheerleading for the Harris decision as “a lengthy and thoughtful analysis”, and for the Milward case as roadmap to evade meaningful judicial oversight).  Not all was perfect with the trial court’s opinion.  The defense seemed to have misled the court by asserting that “a difference between a case group and control group is not statistically significant then there is no difference at all.”  See Respondent’s Brief at 5, Harris v. CSX Transportation, Inc., 2013 WL 4747999 (filed (Feb. 4, 2013) (citing  App. 169, 228-230 (Shields) as having explained that the p-values greater than 0.05 do not support a causal association).

This is hardly true, and indeed, the lack of statistical significance does not lead to a claim that the null hypothesis of no association between exposure and outcome is correct.  The defense, however, did not have a burden of showing the null to be correct; only that there was no reliable method deployed to reject the null in favor an alternative that the risk ratio for myeloma was raised among workers exposed to diesel exhaust.

Still, the trial court did seem to understand the importance of replication, in studies free of bias and confounding. Courts generally will have to do better at delineating what are “positive” and “negative” studies, with citations to the data and the papers, so that judicial opinions provide a satisfactory statement of reasons for judicial decisions.

Biostatistics and FDA Regulation: The Convergence of Science and Law

May 29th, 2014

On May 20, 2014, the Food and Drug Law Institute (FDLI), the Drug Information Association (DIA), and the Harvard Law School’s Petrie-Flom Center for Health Law Policy, Biotechnology, and Bioethics, in collaboration with the Harvard School of Public Health Department of Biostatistics and Harvard Catalyst | The Harvard Clinical and Translational Science Center, presented a symposium on“Biostatistics and FDA Regulation: The Convergence of Science and Law.”

The symposium might just as well have been described as the collision of science and law.

The Symposium agenda addressed several cutting-edge issues on statistical evidence in the law, criminal, civil, and regulatory. Names of presenters are hyperlinked to presentations slides that are available.

I. Coleen Klasmeier, of Sidley Austin LLP, introduced and moderated the first section, “Introduction to Statistics and Regulatory Law,” which focused on current biostatistical issues in regulation of drugs, devices, and foods by the Food and Drug Administration (FDA). Qi Jiang, Executive Director of Amgen, Robert T. O’Neill, retired from the FDA, and now Statistical Advisor in CDER, and Jerald S. Schindler, of Merck Research Laboratories, presented.

II. Qi Jiang moderated and introduced the second section on safety issues, and the difficulties presented by meta-analysis and other statistical assessments of safety outcomes in clinical trials and in marketing of drugs and devices. Lee-Jen Wei, of the Harvard School of Public Health, Geoffrey M. Levitt, an Associate General Counsel of Pfizer, Inc., and Janet Wittes, of the Statistics Collaborative, presented.

III. Aaron Katz, of Ropes & Gray LLP, introduced the third section, on “Statistical Disputes in Life Sciences Litigation,” which addressed recent developments in expert witness gatekeeping, the Avandia litigation, and the role of statistics in two recent cases, Matrixx, Inc. v. Siracusano, and United States v. HarkonenAnand Agneshwar, of Arnold & Porter LLP, Lee-Jen Wei, Christina L. Diaz, Assistant General Counsel of GlaxoSmithKline, and Nathan A. Schachtman presented.

IV. Christopher Robertson, a law professor now visiting at Harvard Law School, moderated a talk by Robert O’Neill on “Emerging Issues,” at the FDA.

V. Dr. Wittes moderated a roundtable discussion on “Can We Handle the Truth,” which explored developments in First Amendment and media issues involved in regulation and litigation. Anand Agneshwar, and Freddy A. Jimenez, Assistant General Counsel, Johnson & Johnson, presented.

The Outer Limits (and Beyond?) of Ex Parte Advocacy of Federal Judges

May 23rd, 2014

As every trial lawyer knows, people sometimes reveal important facts in curious ways, incorporated in their own biased narrative of events.  Recently, I heard a recorded lecture about expert witnesses, by a plaintiffs’ lawyer, who revealed a damning fact about a judge.  The lawyer clearly thought that this fact was commendatory, but in fact revealed another effort of scientific advocates and zealots to subvert the neutrality of federal judges.  See In re School Asbestos Litigation, 977 F.2d 764 (3d Cir. 1992) (describing effort by plaintiffs’ lawyers and the late Dr. Irving Selikoff to corrupt state and federal judges with one-sided ex parte presentations of their views at the so-called Third-Wave Conference).

Anthony Z. Roisman is the Managing Partner of the National Legal Scholars Law Firm.  This firm has a roster of affiliated law professors who serve as consultants for plaintiffs in environmental and tort cases. (Some other participants in this law firm include Jay M. Feinman, Lucinda M. Finley, Neil Vidmar, and Richard W. Wright.) Roisman has been active in various plaintiff organizations, including serving as the head of the ATLA Section on Toxic, Environmental & Pharmaceutical Torts (STEP). 

Roisman lectures frequently for the American Law Institute on expert witness issues. Recently, I was listening to an mp3 recording of one of Roisman’s lectures on expert witnesses in environmental litigation.  Given Roisman’s practice and politics, I was not surprised to hear him praise Judge Rothstein’s opinion that refused to exclude plaintiffs’ expert witnesses’ causation opinions in the PPA litigation.  See In re Phenylpropanolamine Prod. Liab. Litig., 289 F. 2d 1230 (2003).  What stunned me, however, was his statement that Judge Rothstein issued her opinion “fresh from a seminar at the Tellus Institute,” which he described as “organization set up by scientist trying to bring common sense to interpretation of science.”

Post hoc; ergo propter hoc?

Judge Rothstein’s PPA decision stands as a landmark of judicial gullibility.  Judge Rothstein conducted hearings and entertaining extensive briefings on the reliability of plaintiffs’ expert witnesses’ opinions, which were based largely upon one epidemiologic study, known as the “Yale Hemorrhagic Stroke Project (HSP).”  In the end, publication in a prestigious peer-reviewed journal proved to be a proxy for independent review: “The prestigious NEJM published the HSP results, further substantiating that the research bears the indicia of good science.” Id. at 1239 (citing Daubert II for the proposition that peer review shows the research meets the minimal criteria for good science). The admissibility challenges were refused.

Ultimately, the HSP study received much more careful analysis before juries, which uniformly returned verdicts for the defense. After one of the early defense verdicts, plaintiffs’ counsel challenged the defendant’s reliance upon underlying data in the HSP, which went behind the peer-reviewed publication, and which showed that the peer review failed to prevent serious errors.  The trial court rejected the plaintiffs’ request for a new trial, and spoke to the significance of challenging the superficial significance of peer review of the key study relied upon by plaintiffs in the PPA litigation:

“I mean, you could almost say that there was some unethical activity with that Yale Study.  It’s real close.  I mean, I — I am very, very concerned at the integrity of those researchers.”

“Yale gets — Yale gets a big black eye on this.”

O’Neill v. Novartis AG, California Superior Court, Los Angeles Cty., Transcript of Oral Argument on Post-Trial Motions, at 46 -47 (March 18, 2004) (Hon. Anthony J. Mohr)

Roisman’s endorsement of the PPA decision may have been purely result-oriented jurisprudence, but what of his enthusiasm for the “learning” that Judge Rothstein received at the Tellus Institute.  Tell us, what is this Tellus Institute?

In 2003, roughly contemporaneously with Judge Rothstein’s PPA decision, SKAPP published a jeremiad against the Daubert decision, with support from none other than the Tellus Group. See Daubert: The Most Influential Supreme Court Ruling You’ve Never Heard Of;  A Publication of the Project on Scientific Knowledge and Public Policy, coordinated by the Tellus Institute (2003). The Tellus Institute website tells us very little specific detail about the Institute’s projects, other than stating some vague and pious goals.  The alignment, however, of the Tellus Institute with David Michael’s SKAPP, which was created with plaintiffs’ lawyers’ funding, certainly seems like a dubious indicator of neutrality and scientific commitment.  SeeSkapp a Lot” (April 30, 2010).

We might get a better idea of the organization from the Tellus membership.

Richard Clapp and David Ozonoff are both regular testifiers for plaintiffs in so-called toxic tort and environmental litigation. In an article published about the time of the PPA decision, Clapp and Ozonoff acknowledged having benefited from discussions with colleagues at the Tellus Institute.  See Richard W. Clapp & David Ozonoff, “Environment and Health: Vital Intersection or Contested Territory?” 30 Am. J. L. & Med. 189, 189 (2004) (“This Article also benefited from discussions with colleagues in the project on Scientific Knowledge and Public Policy at Tellus Institute, in Boston, Massachusetts.”).

In the infamous case of Selikoff and Motley and their effort to subvert the neutrality of Judge James M. Kelly in the school district asbestos litigation, the conspiracy was detected in time for a successful recusal effort. In re School Asbestos Litigation, 977 F.2d 764 (3d Cir. 1992).  Unfortunately, in the PPA litigation, there was no disclosure of the efforts by the advocacy group, Tellus Institute, to undermine the neutrality of a federal judge. 

Outside observers will draw their own inferences about whether Tellus was an “honest broker” of scientific advice to Judge Rothstein. One piece of evidence may be SKAPP’s website, which contains a page about Richard Clapp’s courtroom advocacy in the PPA litigation. Additional evidence comes from Clapp’s leadership role in Physicians for Social Responsibility, and his own characterization of himself as a healthcare professional advocate. Clapp, a member of Tellus, was an expert witness for plaintiffs in PPA cases.

Was Clapp present at the Tellus Institute meeting attended by Judge Rothstein? History will judge whether the Tellus Institute participated in corrupting the administration of justice.

The Fallacy of Cherry Picking As Seen in American Courtrooms

May 3rd, 2014

After a long winter, the cherry trees are finally managing to blossom.  Before we know it, it will be cherry-picking time.

Cherry picking is a good thing; right?  Cherry picking yields cherries, and cherries are good.  Selective cherry picking yields the best, ripest, sweetest, tastiest cherries. Cherry picking data no doubt yields the best, unbiased, unconfounded, most probative data to be had.  Well, maybe not.

What could be wrong with picking cherries?  At the end of the process you have cherries, and if you do it right, you have all ripe, and no rotten, cherries.  Your collection of ripe cherries, however, will be unrepresentative of the universe of cherries, but at least we understand how and why your cherries were selected.

Elite colleges cherry pick the best high school students; leading law schools cherry pick the top college students; and top law firms and federal judges cherry pick the best graduates from the best law schools.  Lawyers are all-too-comfortable with “cherry picking.”  Of course, the cherry-picking process here has at least some objective criteria, which can be stated in advance of the selection.

In litigation, each side is expected to “cherry pick” the favorable evidence, and ignore or flyblow the contrary evidence.  Perhaps this aspect of the adversarial system induces complacency in judges about selectivity in the presentation of evidence by parties and their witnesses.  In science, this kind of adversarial selectivity is a sure way to inject bias and subjectivity into claims of knowledge.  And even in law, there are limits to this adversarial system. Undue selectivity in citing precedent can land a lawyer in a heap of trouble. See Thul v. OneWest Bank, FSB, No. 12 C 6380, 2013 WL 212926 (N.D. Ill. Jan. 18, 2013) (failure to cite relevant judicial precedent constitutes an ethical offense)

In science, the development of the systematic review, in large measure, has been supported by the widespread recognition that studies cannot be evaluated with post hoc, subjective evaluative criteria. See generally Matthias Egger, George Davey Smith, and Douglas Altman, Systematic Reviews in Health Care: Meta-Analysis in Context (2001).

Farmers pick the cherries they want to go to market, to make money and satisfy customers. The harvesters’ virtue lies in knowing what to pick to obtain the best crop.  The scientist’s virtue lies in the disinterested acquisition of data pursuant to a plan, and the evaluation of the data pursuant to pre-specified criteria.

The scientist’s virtue is threatened by motivations that are all-too human, and all-too common. The vice in science is wanting data that yields marketable publications, grants, promotions, awards, prizes, and perhaps a touch of fame. Picking data based upon a desired outcome is at the very least scientific fallacy if not scientific fraud. Cherry picking does not necessarily imply scienter, but in science, it is a strict liability offense.

The metaphor of cherry picking, mixed as it may be, thus gives us a label for fallacy and error.  Cherry picking incorporates sampling bias, selection bias,  confirmation bias, hasty generalization, and perhaps others as well. As explained recently, in Nature:

“Data can be dredged or cherry picked. Evidence can be arranged to support one point of view. * * * The question to ask is: ‘What am I not being told?’”

William J. Sutherland, David Spiegelhalter & Mark Burgman, “Policy: Twenty tips for interpreting scientific claims,” 503 Nature 335, 337 (2013).

Cherry picking in the orchard may be a good thing, but in the scientific world, it refers to the selection of studies or data within studies to yield results desired results, however misleading or counterfactual.  See Ben Goldacre, Bad Science 97-99 (2008). The selective use of evidence is not a fallacy unique to science. Cherry picking is widely acknowledged to seriously undermine the quality of public debate See Gary Klass, “Just Plain Data Analysis: Common Statistical Fallacies in Analyses of Social Indicator Data” (2008).  See generally Bradley Dowden, “Fallacies,” in James Fieser & Bradley Dowden, eds., Internet Encyclopedia of Philosophy.

The International Encyclopedia of Philosophy describes “cherry picking” as a fallacy, “a kind of error in reasoning.”  Cherry-picking the evidence, also known as “suppressed evidence,” is:

“[i]ntentionally failing to use information suspected of being relevant and significant is committing the fallacy of suppressed evidence. This fallacy usually occurs when the information counts against one’s own conclusion. * * * If the relevant information is not intentionally suppressed but rather inadvertently overlooked, the fallacy of suppressed evidence also is said to occur, although the fallacy’s name is misleading in this case.”

Bradley Dowden, “Suppressed Evidence,” International Encyclopedia of Philosophy (Last updated: December 31, 2010). See alsoCherry picking (fallacy),” Wikipedia (describing cherry picking as the pointing to data that appears to confirm one’s opinion, while ignoring contradictory data).

In 1965, in his landmark paper, Sir Austin Bradford Hill described some important factors to consider in determining whether a clear-cut association, beyond that which we would attribute to chance, was a causal association. Hill, Austin Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295, 295 (1965).

One of the key Hill factors is, of course, consistent, replicated results.  Surely, an expert witness should not be permitted to manufacture a faux consistency by conducting a partial review.  In birth defects litigation, the problem of  “cherry picking” is so severe that one of the leading professional societies concerned with birth defects has issued a position paper to remind its members, other scientists, and the public that “[c]ausation determinations are made using all the scientific evidence”:

Causation determinations are made using all the scientific evidence. This evidence is derived from correctly interpreted papers that have been published in the peer-reviewed literature. Unpublished data may be useful if available in sufficient detail for an evaluation and if derived from a source that is known to use reliable internal or external review standards. A National Toxicology program report would be an example of an unpublished source that is typically reliable. All available papers are considered in a scientific deliberation; selective consideration of the literature is not a scientific procedure.”

The Public Affairs Committee of the Teratology Society, “Teratology Society Public Affairs Committee Position Paper Causation in Teratology-Related Litigation,” 73 Birth Defects Research (Part A) 421, 422 (2005) (emphasis added).

* * * * * *

Cherry picking is a main rhetorical device for the litigator. Given the pejorative connotations of “cherry picking,” no one should be very surprised that lawyers and judges couch their Rule 702 arguments and opinions in terms of whether expert witnesses engaged in this fulsome fruitful behavior.

The judicial approach to cherry picking is a just a little schizophrenic. Generally, in the context of exercising its gatekeeping function for expert witnesses, the elimination of cherry picking is an important goal. Lust v. Merrell Dow Pharmaceuticals, Inc., 89 F.3d 594, 596-98 (9th Cir. 1996) (affirming exclusion of Dr. Done in a Chlomid birth defects case; district court found that “Dr. Done has seen fit to ‘pick and chose’ [sic] from the scientific landscape and present the Court with what he believes the final picture looks like. This is hardly scientific.”) (internal citation omitted); Barber v. United Airlines, Inc., 17 Fed. Appx. 433, 437 (7th Cir. 2001) (holding that a “selective use of facts fails to satisfy the scientific method and Daubert”). See also Crawford v. Indiana Harbor Belt Railroad Co., 461 F.3d 844 (7th Cir. 2006) (affirming summary judgment in disparate treatment discharge case, and noting judicial tendency to require “comparability” between plaintiffs and comparison group as a “natural response to cherry-picking by plaintiffs”); Miller v. Pfizer, Inc., 196 F. Supp. 2d 1062, (D. Kan. 2002) (excluding, with aid of independent, court-appointed expert witnesses, a party expert witness, David Healy, who failed to reconcile the fact that other research is contrary to his conclusion), aff’d, 356 F.3d 1326 (10th Cir.), cert denied, 125 S. Ct. 40 (2004).

In Ellis v. Barnhart, the Eighth Circuit affirmed a district court’s reversal of an Administrative Law Judge for “cherry picking” the record in a disability case.  392 F.3d 988 (8th Cir. 2005).  Clearly cherry picking was a bad thing for a judicial officer to do when charged with the administration of justice. Several years later, however, the Eighth Circuit held that a trial court erred in excluding an expert witness for having offered an opinion that ignored the witness’s own prior, contrary opinions, a key National Institutes of Health clinical trial, and multiple other studies.  The adversary’s charges of  “cherry picking” were to no avail. Kuhn v. Wyeth, Inc., 686 F.3d 618, 633 (8th Cir. 2012) (“There may be several studies supporting Wyeth’s contrary position, but it is not the province of the court to choose between the competing theories when both are supported by reliable scientific evidence.”), rev’g Beylin v. Wyeth, 738 F.Supp. 2d 887, 892 (E.D.Ark. 2010) (MDL court) (Wilson, J. & Montgomery, J.) (excluding proffered testimony of Dr. Jasenka Demirovic who appeared to have “selected study data that best supported her opinion, while downplaying contrary findings or conclusions.”).

But wait, the court in Kuhn did not cite its own published opinion on cherry picking in Ellis.  Some might say that the Circuit cherry picked its own precedents to get to a desired result. Anthony Niblett, “Do Judges Cherry Pick Precedents to Justify Extralegal Decisions?: A Statistical Examination,” 70 Maryland L. Rev. 234 (2010) (reviewing charges of cherry picking, and examining data [cherry picked?] from California).

The situation in the federal trial courts is chaotic. Most of the caselaw recognizes the fallacy of an expert witness’s engaging in ad hoc selection of studies upon which to rely.  Federal courts, clear on their gatekeeping responsibilities and aware of the selection fallacy, have condemned cherry-picking expert witnesses. Judge Lewis Kaplan, in the Southern District of New York, expressed the proper judicial antipathy to cherry picking:

“[A]ny theory that fails to explain information that otherwise would tend to cast doubt on that theory is inherently suspect,” and “courts have excluded expert testimony ‘where the expert selectively chose his support from the scientific landscape.’”

In re Rezulin Prod. Liab. Litig., 369 F. Supp. 2d 398, 425 & n.164 (S.D.N.Y. 2005) (citation omitted).

Judge Breyer, of the Northern District of California, expressed similar sentiments in ruling on Rule 702 motions in the Celebrex personal injury litigation:

“these experts ignore the great weight of the observational studies that contradict their conclusion and rely on the handful that appear to support their litigation-created opinion.”

In re Bextra & Celebrex Mktg. Sales Pracs. & Prods. Liab. Litig., 524 F. Supp. 2d 1166, 1181 (N.D. Cal. 2007).  The “cherry-picking” of favorable data “does not reflect scientific knowledge, is not derived by the scientific method, and is not ‘good science.’” Id. at 1176.

Other illustrative federal cases include:

In re Bausch & Lomb, Inc., 2009 WL 2750462 at *13-14 (D.S.C. 2009) (“Dr. Cohen did not address [four contradictory] studies in her expert reports or affidavit, and did not include them on her literature reviewed list [. . .] This failure to address this contrary data renders plaintiffs’ theory inherently unreliable.”)

Rimbert v. Eli Lilly & Co., No. 06-0874, 2009 WL 2208570, *19 (D.N.M. July 21, 2009) )(“Even more damaging . . . is her failure to grapple with any of the myriad epidemiological studies that refute her conclusion.”), aff’d, 647 F.3d 1247 (10th Cir. 2011) (affirming exclusion but remanding to permit plaintiff to find a new expert witness)

LeClercq v. The Lockformer Co., No. 00C7164, 2005 WL 1162979, at *4, 2005 U.S. Dist. LEXIS 7602, at *15 (N.D. Ill. Apr. 28, 2005) (“failure to discuss the import of, or even mention … material facts in [expert] reports amounts to ‘cherry-pick[ing]’ … and such selective use of facts fail[s] to satisfy the scientific method and Daubert.”) (internal citations and quotations omitted)

Contractors Ass’n of E. Pa. Inc. v. City of Philadelphia, 893 F. Supp. 419, 436 (E.D. Pa., 1995) (holding that expert witness opinion was unreliable when witness’s conclusions rested on incomplete factual data)

Galaxy Computer Servs. Inc. v. Baker, 325 B.R. 544 (E.D. Va. 2005) (excluding expert witness when witness relied upon incomplete data in reaching a valuation assessment).

Dwyer v. Sec’y of Health & Human Servs., No. 03-1202V, 2010 WL 892250, at *14 (Fed. Cl. Spec. Mstr. Mar. 12, 2010)(recommending rejection of thimerosal autism claim)(“In general, respondent’s experts provided more responsive answers to such questions.  Respondent’s experts were generally more careful and nuanced in their expert reports and testimony. In contrast, petitioners’ experts were more likely to offer opinions that exceeded their areas of expertise, to “cherry-pick” data from articles that were otherwise unsupportive of their position, or to draw conclusions unsupported by the data cited… .”)

Holden Metal & Aluminum Works, Ltd. v. Wismarq Corp., No. 00C0191, 2003 WL 1797844, at *2 (N.D. Ill. Apr. 3, 2003) (“Essentially, the expert ‘cherrypicked’ the facts he considered to render his opinion, and such selective use of facts failed to satisfy the scientific method and Daubert.”) (internal citation omitted).

Flue-Cured Tobacco Cooperative Stabilization Corp. v. EPA, 4 F. Supp. 2d 435, 459 – 60  (M.D.N.C. 1998) (finding that  EPA’s selection of studies for inclusion in a meta-analysis to be “disturbing,” and that agency’s selective, incomplete inclusion of studies violated its own guidelines for conducting risk assessments), rev’d on other grounds, 313 F.3d 852, 862 (4th Cir. 2002) (Widener, J.) (holding that the issuance of the report was not “final agency action”)

Fail-Safe, LLC v. AO Smith Corp., 744 F. Supp. 2d 870, 889 (E.D. Wis. 2010) (“the court also finds the witness’s methodology unreliable because of how Dr. Keegan uniformly treated all evidence that undermined his underlying conclusion: unwarranted dismissal of the evidence or outright blindness to contrary evidence. In fact, it is readily apparent that Dr. Keegan all but ‘cherry picked’ the data he wanted to use, providing the court with another strong reason to conclude that the witness utilized an unreliable methodology. * * * Dr. Keegan’s two reports are rich with examples of his ‘cherry picking’ of the evidence.”)

As noted, however, there are federal trial courts that are all too willing to suspend judgment and kick the case to the jury.  Here is a sampler of cases that found cherry picking to be an acceptable methodology, or at least a methodology sufficient to require that the case be submitted to the finder of fact.

In Berg v. Johnson & Johnson, the district court noted the defendants’ argument that proffered testimony is unreliable because witness “cherry-picked” data in order to form an opinion solely for purposes of litigation. 940 F.Supp. 2d 983, 991-92 (D.S.D. 2013). The trial judge, however, was not willing to look particularly closely at what was excluded or why:

“The only difference between his past and present research seems to exist in how he categorized his data. Defendants label this ‘cherry-picking’. The court views it as simply looking at the existing data from a different perspective.”

Id.  Of course, expert witnesses on opposite sides look at the case from different perspectives, but the question begged was whether the challenged expert witness had categorized data in an unprincipled way. Other cases of this ilk include:

United States v. Paracha, 2006 WL 12768, at *20 (S.D. N.Y. Jan. 3, 2006) (rejecting challenge to terrorism expert witness on grounds that he cherry picked evidence in conspiracy prosecution involving al Queda)

In re Chantix (Varenicline) Products Liab. Litig., 889 F. Supp. 2d 1272, 1288 (N.D. Ala. 2012) (“Why Dr. Kramer chose to include or exclude data from specific clinical trials is a matter for cross-examination, not exclusion under Daubert.“)

Bouchard v. Am. Home Prods. Corp., 2002 WL 32597992 at *7 (N.D. Ohio May 24, 2002) (“If Bouchard believes that [the expert]… ignored evidence that would have required him to substantially change his opinion, that is a fit subject for cross-examination, not a grounds for wholesale rejection of an expert opinion.”)

In re Celexa & Lexapro Prods. Liab. Litig., 927 F. Supp. 2d 758, 2013 WL 791780, at *5, *7, *8 (E.D. Mo. 2013) (Sippel, J.) (rejecting challenge to David Healy in antidepressant suicide case)

Allen v. Takeda Pharms., MDL No. 6:11-md-2299, No. 12-cv-00064, 2013 WL 6825953, at *11 (W.D. La. Dec. 20, 2013) (challenged expert witness in Actos litigation sufficiently explained his choices to be exonerated from charges of cherry picking)

In re NuvaRing Prods. Liab. Litig., No. 4:08–MD–1964 RWS, 2013 WL 791787 (E.D. Mo. Mar. 4, 2013) (“As to cherry picking data, the Eighth Circuit has recently made clear that such allegations should be left for crossexamination.”)

McClellan v. I-Flow Corp., 710 F. Supp. 2d 1092, 1114 (D. Ore. 2010) (“Defendants are correct that plaintiffs’ experts must elucidate how the relevant evidence lends support to their opinions by explaining…..”) (rejecting cherry picking but denying Rule 702 challenge based in part upon alleged cherry picking)

Rich v. Taser Internat’l, Inc., No. 2:09–cv–02450–ECR–RJJ, 2012 WL 1080281, at *6 (D. Nev. March 30, 2012) (noting the objection to cherry picking but holding that it was an issue for cross-examination)

In re Urethane Antitrust Litig., No. 04-1313-JWL, MDL No. 1616, 2012 WL 6681783, at *3 (D. Kan. Dec. 21, 2012) (allowing expert testimony that “certain events are consistent with collusion”; “the extent to which [an expert] considered the entirety of the evidence in the case is a matter for cross-examination.”)

In re Titanium Dioxide Antitrust Litig., No. RDB-10-0318, 2013 WL 1855980, 2013 U.S. Dist. LEXIS 62394 (D. Md. May 1, 2013) (rejecting Rule 702 cherry-picking challenge to an expert who cherry picked; witness’s selection of documents upon which to rely from a record that exceeded 14 million pages was not unreliable. “ If important portions of the record were overlooked, then the Defendants may address that issue at trial.”)

STATE COURTS

The situation in state courts is similarly chaotic and fragmented.

In Lakey v. Puget Sound Energy, Inc., the Washington Supreme Court resoundingly rejected “cherry picking” by expert witnesses in a public and private nuisance case against a local utility for fear of future illnesses from exposure to electro-magnetic frequency radiation (EMF).  Lakey v. Puget Sound Energy, Inc., 176 Wn.2d 909 (2013). The court held that the plaintiffs’ expert witnesses’ cherry-picking approach to data and studies was properly excluded under Rule 702. Their selective approach vitiated the reliability of his opinion with the consequence of :

“seriously tainting his conclusions because epidemiology is an iterative science relying on later studies to refine earlier studies in order to reach better and more accurate conclusions. Carpenter refused to account for the data from the toxicological studies, which epidemiological methodology requires unless the evidence for the link between exposure and disease is unequivocal and strong, which is not the case here. Carpenter also selectively sampled data within one of the studies he used, taking data indicating an EMF-illness link and ignoring the larger pool of data within the study that showed no such link, Carpenter’s treatment of this data created an improper false impression about what the study actually showed.”

Id.; see alsoWashington Supreme Court Illustrates the Difference Between Frye and Rule 702” (April 15, 2013).

Other state Supreme Courts have recognized and rejected the gerrymandering of scientific evidence.  Betz v. Pneumo Abex LLC, 2012 WL 1860853, *16 (May 23, 2012 Pa. S. Ct.)(“According to Appellants, moreover, the pathologist’s self-admitted selectivity in his approach to the literature is decidedly inconsistent with the scientific method. Accord Brief for Amici Scientists at 17 n.2 (“‘Cherry picking’ the literature is also a departure from ‘accepted procedure’.”)); George v. Vermont League of Cities and Towns, 2010 Vt. 1, 993 A.2d 367, 398 (Vt. 2010)(expressing concern about how and why plaintiff’s expert witnesses selected some studies to include in their “weight of evidence” methodology.  Without an adequate explanation of selection and weighting criteria, the choices seemed “arbitrary” “cherry picking.”); Bowen v. E.I. DuPont de Nemours & Co., 906 A.2d 787, 797 (Del. 2006) (noting that expert witnesses cannot ignore studies contrary to their opinions).

Lower state courts have also quashed the cherry-picking harvest. Scaife v. AstraZeneca LP, 2009 WL 1610575, at *8 (Del. Super. June 9, 2009) (“Simply stated, the expert cannot accept some but reject other data from the medical literature without explaining the bases for her acceptance or rejection.”); see also In re Bextra & Celebrex Prod. Liab. Litig., No. 762000/2006, 2008 N.Y. Misc. LEXIS 720, at *47 (Sup. Ct. N.Y. Co. Jan 7, 2008) (stating that plaintiffs must show that their experts “do not ignore contrary data”).

The Nebraska Supreme Court appears to recognize the validity of considering the existence of cherry-picking in expert witness gatekeeping.  In practice, however, that Court has shown an unwillingness to tolerate close scrutiny into what was included and excluded from the expert witness’s consideration.  King v. Burlington No. Santa Fe Ry, ___N.W.2d___, 277 Neb. Reports 203, 234 (2009)(noting that the law does “not preclude a trial court from considering as part of its reliability inquiry whether an expert has cherry-picked a couple of supporting studies from an overwhelming contrary body of literature,” but ignoring the force of the fallacious expert witness testimony by noting that the questionable expert witness (Frank) had some studies that showed associations between exposure to diesel exhaust or benzene and multiple myeloma).


“Of all the offspring of time, Error is the most ancient, and is so old and familiar an acquaintance, that Truth, when discovered, comes upon most of us like an intruder, and meets the intruder’s welcome.”

Charles MacKay, Extraordinary Popular Delusions and the Madness of Crowds (1841)

On The Quaint Notion That Gatekeeping Rules Do Not Apply to Judges

April 27th, 2014

In In re Zurn Pex Plumbing Prods. Liab. Litig., 644 F.3d 604 (8th Cir. 2011), the United States Court of Appeals for the Eighth Circuit rejected the defendant’s argument that a “full and conclusive” Rule 702 gatekeeping procedure was required before a trial court could certify a class action under the Federal Rules. The Circuit remarked that “[t]he main purpose of Daubert exclusion is to protect juries from being swayed by dubious scientific testimony,” an interest “not implicated at the class certification stage where the judge is the decision maker.”  Id. at 613.

Surely, one important purpose of Rule 702 is to protect juries against dubious scientific testimony, but judges are not universally less susceptible to dubious testimony.  There are many examples of judges being misled by fallacious scientific evidence, especially when tendentiously presented by advocates in court.  No jury need be present for dubious science testimony + “zealous” advocacy to combine to create major errors and injustice.  See, e.g., Wells v. Ortho Pharmaceutical Corp., 615 F. Supp. 262 (N.D. Ga. 1985)(rendering verdict for plaintiffs after bench trial), aff’d and rev’d in part on other grounds, 788 F.2d 741 (11th Cir.), cert. denied, 479 U.S.950 (1986); Hans Zeisel & David Kaye, Prove It With Figures: Empirical Methods in Law and Litigation § 6.5 n.3, at 271 (1997) (characterizing Wells as “notorious,” and noting that the case became a “lightning rod for the legal system’s ability to handle expert evidence.”).  Clearly Rule 702 does not exist only to protect juries.

Nemo iudex in causa sua! Perhaps others should judge the competence of judges’ efforts at evaluating scientific evidence.  At the very least, within the institutional framework of our rules of civil procedure and evidence, Rule 702 creates a requirement of structured inquiry into expert opinion testimony before the court.  That gatekeeping inquiry, and its requirement of a finding, subject to later appellate review and to public and professional scrutiny, are crucial to the rendering of intellectual due process in cases that involve scientific and technical issues.  The Eighth Circuit was unduly narrow in its statement of the policy bases for Rule 702, and their applicability to class certification.

The case of Obrey v. Johnson, 400 F.3d 691 (9th Cir. 2005) provides another cautionary tale about the inadequacies of judges in the evaluation of scientific and statistical evidence.  The plaintiff, Mr. Obrey, sued the Navy on a claim of race discrimination in promoting managers at the Pearl Harbor Naval Shipyard.  The district court refused plaintiff’s motion to admit the testimony of a statistician, Mr. James Dannemiller, President of the SMS Research & Marketing Services, Inc. The district court also excluded much of plaintiff’s anecdotal evidence, and entered summary judgment.  Id. at 691 – 93.

On appeal, Obrey claimed that Dannemiller’s report showed “a correlation between race and promotion.” Id. at 693. This vague claim seemed good enough for the Ninth Circuit, which reversed the district court’s grant of summary judgment and remanded for trial.

The Ninth Circuit’s opinion does not tell us what sort of correlation was supposedly shown by Mr. Dannemiller. Was it Pearson’s r?  Or Jaspen’s multi-serial coefficient? Spearman’s ρ?  Perhaps Kendall’s τ? Maybe the appellate court was using correlation loosely, and Mr. Dannemiller had conducted some other sort of statistical analysis. The district court’s opinion is not published and is not available on Westlaw.  It is all a mystery. More process is due the litigants and the public.

Even more distressing than the uncertainty as to the nature of the correlation is that the Ninth Circuit does not tell us what the correlation “effect size” was, or whether the correlation was statistically significant.  If the Circuit did not follow strict hypothesis testing, perhaps it might have told us the extent of random error in the so-called correlation.  The Circuit did not provide any information about the extent or the precision of the claim of a “correlation”; nor did the Circuit assess the potential for bias or confounding in Mr. Dannemiller’s analysis.

Indeed, the Ninth Circuit seemed to suggest that Mr. Dannemiller never even showed a correlation; rather the court described Mr. Dannemiller as having opined that there was “no statistical evidence in these data that the selection process for GS-13 through GS-15 positions between 1999 and 2002 was unbiased with respect to race.” Id. at 694. Reading between the lines, it seems that the statistical evidence was simply inconclusive, and Mr. Dannemiller surreptitiously shifted the burden of proof and offered an opinion that the Navy had not ruled out bias. The burden, of course, was on Mr. Obrey to establish a prima facie case, but the appellate court glossed over this fatal gap in plaintiff’s evidence.

On appeal, the Navy pressed its objections to the relevance and reliability of Mr. Dannemiller’s opinions. Brief of the Navy, 2004 WL 1080083, at *1 (April 7, 2004).  There seemed to be no dispute that Mr. Dannemiller’s “study” was based entirely upon “statistical disparities,” which failed to take into account education, experience, and training.  Mr. Dannemiller appeared to have simplistically compared race make up of the promoted workers, ignoring the Navy’s showing of the relevancy of education, experience, and training.  Id. at *13, 18.

The Ninth Circuit not only ignored the facts of the case, it ignored its own precedents.  See Obrey v. Johnson, 400 F.3d at 696 (citing and quoting from Coleman v. Quaker Oats Co., 232 F.3d 1271, 1283 (9th Cir. 2000) (“Because [the statistics] failed to account for many factors pertinent to [the plaintiff], we conclude that the statistics are not enough to take this case to trial.”). The court, in Obrey, made no effort to distinguish its treatment of the parties in Coleman, or to justify its decision as to why the unspecified, unquantified, mysterious statistical analysis of Mr. Dannemiller sufficed under Rule 702. The Circuit cryptically announced that “Obrey’s evidence was not rendered irrelevant under Rule 402 simply because it failed to account for the relative qualifications of the applicant pool.”  Obrey, 400 F.3d at 695.  Citing pre-Daubert decisions for the most part (such as Bazemore), the Ninth Circuit persuaded itself that Rule 702 requires nothing more than simple relevancy. Had the Circuit taken even a cursory look at Bazemore, it would have seen that the case involved a much more involved multiple regression than whatever statistical analysis Mr. Dannemiller propounded.  And the Ninth Circuit would have seen that even the Bazemore decision acknowledged that there may be

“some regressions so incomplete as to be inadmissible as irrelevant… .”

478 U.S. 385, 400 n.10 (1986). It is difficult to imagine a discrimination claim analysis more incomplete than one that did not address education, training, and experience.

Sadly, neither the Navy’s nor Mr. Obrey’s brief, 2004 WL 545873 (Feb. 4, 2004) provided any discussion of the nature, quality, findings, or limits of Mr. Dannemiller’s statistical analysis.  The Navy’s brief referred to Mr. Dannemiller as a “purported” expert.  His resume, available online, shows that Mr. Dannemiller studied history as an undergraduate, and has a master’s degree in sociology. He is the president of SMS Research, a consulting company.

The taxpayers deserved better advocacy from the Department of Justice, and greater attention to statistical methodology from its appellate judges.  See ATA Airlines, Inc. v. Federal Exp. Corp., 665 F.3d 882, 888-96 (2011) (Posner, J.) (calling for lawyers and judges to do better in understanding and explaining, in plain English, the statistical analyses that are essential to their cases). Judges at level need to pay greater attention to the precepts of Rule 702, even when there is no jury around to be snuckered.

The opinions, statements, and asseverations expressed on Tortini are my own, or those of invited guests, and these writings do not necessarily represent the views of clients, friends, or family, even when supported by good and sufficient reason.