TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

The Fallacy of Cherry Picking As Seen in American Courtrooms

May 3rd, 2014

After a long winter, the cherry trees are finally managing to blossom.  Before we know it, it will be cherry-picking time.

Cherry picking is a good thing; right?  Cherry picking yields cherries, and cherries are good.  Selective cherry picking yields the best, ripest, sweetest, tastiest cherries. Cherry picking data no doubt yields the best, unbiased, unconfounded, most probative data to be had.  Well, maybe not.

What could be wrong with picking cherries?  At the end of the process you have cherries, and if you do it right, you have all ripe, and no rotten, cherries.  Your collection of ripe cherries, however, will be unrepresentative of the universe of cherries, but at least we understand how and why your cherries were selected.

Elite colleges cherry pick the best high school students; leading law schools cherry pick the top college students; and top law firms and federal judges cherry pick the best graduates from the best law schools.  Lawyers are all-too-comfortable with “cherry picking.”  Of course, the cherry-picking process here has at least some objective criteria, which can be stated in advance of the selection.

In litigation, each side is expected to “cherry pick” the favorable evidence, and ignore or flyblow the contrary evidence.  Perhaps this aspect of the adversarial system induces complacency in judges about selectivity in the presentation of evidence by parties and their witnesses.  In science, this kind of adversarial selectivity is a sure way to inject bias and subjectivity into claims of knowledge.  And even in law, there are limits to this adversarial system. Undue selectivity in citing precedent can land a lawyer in a heap of trouble. See Thul v. OneWest Bank, FSB, No. 12 C 6380, 2013 WL 212926 (N.D. Ill. Jan. 18, 2013) (failure to cite relevant judicial precedent constitutes an ethical offense)

In science, the development of the systematic review, in large measure, has been supported by the widespread recognition that studies cannot be evaluated with post hoc, subjective evaluative criteria. See generally Matthias Egger, George Davey Smith, and Douglas Altman, Systematic Reviews in Health Care: Meta-Analysis in Context (2001).

Farmers pick the cherries they want to go to market, to make money and satisfy customers. The harvesters’ virtue lies in knowing what to pick to obtain the best crop.  The scientist’s virtue lies in the disinterested acquisition of data pursuant to a plan, and the evaluation of the data pursuant to pre-specified criteria.

The scientist’s virtue is threatened by motivations that are all-too human, and all-too common. The vice in science is wanting data that yields marketable publications, grants, promotions, awards, prizes, and perhaps a touch of fame. Picking data based upon a desired outcome is at the very least scientific fallacy if not scientific fraud. Cherry picking does not necessarily imply scienter, but in science, it is a strict liability offense.

The metaphor of cherry picking, mixed as it may be, thus gives us a label for fallacy and error.  Cherry picking incorporates sampling bias, selection bias,  confirmation bias, hasty generalization, and perhaps others as well. As explained recently, in Nature:

“Data can be dredged or cherry picked. Evidence can be arranged to support one point of view. * * * The question to ask is: ‘What am I not being told?’”

William J. Sutherland, David Spiegelhalter & Mark Burgman, “Policy: Twenty tips for interpreting scientific claims,” 503 Nature 335, 337 (2013).

Cherry picking in the orchard may be a good thing, but in the scientific world, it refers to the selection of studies or data within studies to yield results desired results, however misleading or counterfactual.  See Ben Goldacre, Bad Science 97-99 (2008). The selective use of evidence is not a fallacy unique to science. Cherry picking is widely acknowledged to seriously undermine the quality of public debate See Gary Klass, “Just Plain Data Analysis: Common Statistical Fallacies in Analyses of Social Indicator Data” (2008).  See generally Bradley Dowden, “Fallacies,” in James Fieser & Bradley Dowden, eds., Internet Encyclopedia of Philosophy.

The International Encyclopedia of Philosophy describes “cherry picking” as a fallacy, “a kind of error in reasoning.”  Cherry-picking the evidence, also known as “suppressed evidence,” is:

“[i]ntentionally failing to use information suspected of being relevant and significant is committing the fallacy of suppressed evidence. This fallacy usually occurs when the information counts against one’s own conclusion. * * * If the relevant information is not intentionally suppressed but rather inadvertently overlooked, the fallacy of suppressed evidence also is said to occur, although the fallacy’s name is misleading in this case.”

Bradley Dowden, “Suppressed Evidence,” International Encyclopedia of Philosophy (Last updated: December 31, 2010). See also “Cherry picking (fallacy),” Wikipedia (describing cherry picking as the pointing to data that appears to confirm one’s opinion, while ignoring contradictory data).

In 1965, in his landmark paper, Sir Austin Bradford Hill described some important factors to consider in determining whether a clear-cut association, beyond that which we would attribute to chance, was a causal association. Hill, Austin Bradford Hill, “The Environment and Disease: Association or Causation?” 58 Proc. Royal Soc’y Med. 295, 295 (1965).

One of the key Hill factors is, of course, consistent, replicated results.  Surely, an expert witness should not be permitted to manufacture a faux consistency by conducting a partial review.  In birth defects litigation, the problem of  “cherry picking” is so severe that one of the leading professional societies concerned with birth defects has issued a position paper to remind its members, other scientists, and the public that “[c]ausation determinations are made using all the scientific evidence”:

“Causation determinations are made using all the scientific evidence. This evidence is derived from correctly interpreted papers that have been published in the peer-reviewed literature. Unpublished data may be useful if available in sufficient detail for an evaluation and if derived from a source that is known to use reliable internal or external review standards. A National Toxicology program report would be an example of an unpublished source that is typically reliable. All available papers are considered in a scientific deliberation; selective consideration of the literature is not a scientific procedure.”

The Public Affairs Committee of the Teratology Society, “Teratology Society Public Affairs Committee Position Paper Causation in Teratology-Related Litigation,” 73 Birth Defects Research (Part A) 421, 422 (2005) (emphasis added).

* * * * * *

Cherry picking is a main rhetorical device for the litigator. Given the pejorative connotations of “cherry picking,” no one should be very surprised that lawyers and judges couch their Rule 702 arguments and opinions in terms of whether expert witnesses engaged in this fulsome fruitful behavior.

The judicial approach to cherry picking is a just a little schizophrenic. Generally, in the context of exercising its gatekeeping function for expert witnesses, the elimination of cherry picking is an important goal. Lust v. Merrell Dow Pharmaceuticals, Inc., 89 F.3d 594, 596-98 (9th Cir. 1996) (affirming exclusion of Dr. Done in a Chlomid birth defects case; district court found that “Dr. Done has seen fit to ‘pick and chose’ [sic] from the scientific landscape and present the Court with what he believes the final picture looks like. This is hardly scientific.”) (internal citation omitted); Barber v. United Airlines, Inc., 17 Fed. Appx. 433, 437 (7th Cir. 2001) (holding that a “selective use of facts fails to satisfy the scientific method and Daubert”). See also Crawford v. Indiana Harbor Belt Railroad Co., 461 F.3d 844 (7th Cir. 2006) (affirming summary judgment in disparate treatment discharge case, and noting judicial tendency to require “comparability” between plaintiffs and comparison group as a “natural response to cherry-picking by plaintiffs”); Miller v. Pfizer, Inc., 196 F. Supp. 2d 1062, (D. Kan. 2002) (excluding, with aid of independent, court-appointed expert witnesses, a party expert witness, David Healy, who failed to reconcile the fact that other research is contrary to his conclusion), aff’d, 356 F.3d 1326 (10th Cir.), cert denied, 125 S. Ct. 40 (2004).

In Ellis v. Barnhart, the Eighth Circuit affirmed a district court’s reversal of an Administrative Law Judge for “cherry picking” the record in a disability case.  392 F.3d 988 (8th Cir. 2005).  Clearly cherry picking was a bad thing for a judicial officer to do when charged with the administration of justice. Several years later, however, the Eighth Circuit held that a trial court erred in excluding an expert witness for having offered an opinion that ignored the witness’s own prior, contrary opinions, a key National Institutes of Health clinical trial, and multiple other studies.  The adversary’s charges of  “cherry picking” were to no avail. Kuhn v. Wyeth, Inc., 686 F.3d 618, 633 (8th Cir. 2012) (“There may be several studies supporting Wyeth’s contrary position, but it is not the province of the court to choose between the competing theories when both are supported by reliable scientific evidence.”), rev’g Beylin v. Wyeth, 738 F.Supp. 2d 887, 892 (E.D.Ark. 2010) (MDL court) (Wilson, J. & Montgomery, J.) (excluding proffered testimony of Dr. Jasenka Demirovic who appeared to have “selected study data that best supported her opinion, while downplaying contrary findings or conclusions.”).

But wait, the court in Kuhn did not cite its own published opinion on cherry picking in Ellis.  Some might say that the Circuit cherry picked its own precedents to get to a desired result. Anthony Niblett, “Do Judges Cherry Pick Precedents to Justify Extralegal Decisions?: A Statistical Examination,” 70 Maryland L. Rev. 234 (2010) (reviewing charges of cherry picking, and examining data [cherry picked?] from California).

The situation in the federal trial courts is chaotic. Most of the caselaw recognizes the fallacy of an expert witness’s engaging in ad hoc selection of studies upon which to rely.  Federal courts, clear on their gatekeeping responsibilities and aware of the selection fallacy, have condemned cherry-picking expert witnesses. Judge Lewis Kaplan, in the Southern District of New York, expressed the proper judicial antipathy to cherry picking:

“[A]ny theory that fails to explain information that otherwise would tend to cast doubt on that theory is inherently suspect,” and “courts have excluded expert testimony ‘where the expert selectively chose his support from the scientific landscape.’”

In re Rezulin Prod. Liab. Litig., 369 F. Supp. 2d 398, 425 & n.164 (S.D.N.Y. 2005) (citation omitted).

Judge Breyer, of the Northern District of California, expressed similar sentiments in ruling on Rule 702 motions in the Celebrex personal injury litigation:

“these experts ignore the great weight of the observational studies that contradict their conclusion and rely on the handful that appear to support their litigation-created opinion.”

In re Bextra & Celebrex Mktg. Sales Pracs. & Prods. Liab. Litig., 524 F. Supp. 2d 1166, 1181 (N.D. Cal. 2007).  The “cherry-picking” of favorable data “does not reflect scientific knowledge, is not derived by the scientific method, and is not ‘good science.’” Id. at 1176.

Other illustrative federal cases include:

In re Bausch & Lomb, Inc., 2009 WL 2750462 at *13-14 (D.S.C. 2009) (“Dr. Cohen did not address [four contradictory] studies in her expert reports or affidavit, and did not include them on her literature reviewed list [. . .] This failure to address this contrary data renders plaintiffs’ theory inherently unreliable.”)

Rimbert v. Eli Lilly & Co., No. 06-0874, 2009 WL 2208570, *19 (D.N.M. July 21, 2009) )(“Even more damaging . . . is her failure to grapple with any of the myriad epidemiological studies that refute her conclusion.”), aff’d, 647 F.3d 1247 (10th Cir. 2011) (affirming exclusion but remanding to permit plaintiff to find a new expert witness)

LeClercq v. The Lockformer Co., No. 00C7164, 2005 WL 1162979, at *4, 2005 U.S. Dist. LEXIS 7602, at *15 (N.D. Ill. Apr. 28, 2005) (“failure to discuss the import of, or even mention … material facts in [expert] reports amounts to ‘cherry-pick[ing]’ … and such selective use of facts fail[s] to satisfy the scientific method and Daubert.”) (internal citations and quotations omitted)

Contractors Ass’n of E. Pa. Inc. v. City of Philadelphia, 893 F. Supp. 419, 436 (E.D. Pa., 1995) (holding that expert witness opinion was unreliable when witness’s conclusions rested on incomplete factual data)

Galaxy Computer Servs. Inc. v. Baker, 325 B.R. 544 (E.D. Va. 2005) (excluding expert witness when witness relied upon incomplete data in reaching a valuation assessment).

Dwyer v. Sec’y of Health & Human Servs., No. 03-1202V, 2010 WL 892250, at *14 (Fed. Cl. Spec. Mstr. Mar. 12, 2010)(recommending rejection of thimerosal autism claim)(“In general, respondent’s experts provided more responsive answers to such questions.  Respondent’s experts were generally more careful and nuanced in their expert reports and testimony. In contrast, petitioners’ experts were more likely to offer opinions that exceeded their areas of expertise, to “cherry-pick” data from articles that were otherwise unsupportive of their position, or to draw conclusions unsupported by the data cited… .”)

Holden Metal & Aluminum Works, Ltd. v. Wismarq Corp., No. 00C0191, 2003 WL 1797844, at *2 (N.D. Ill. Apr. 3, 2003) (“Essentially, the expert ‘cherrypicked’ the facts he considered to render his opinion, and such selective use of facts failed to satisfy the scientific method and Daubert.”) (internal citation omitted).

Flue-Cured Tobacco Cooperative Stabilization Corp. v. EPA, 4 F. Supp. 2d 435, 459 – 60  (M.D.N.C. 1998) (finding that  EPA’s selection of studies for inclusion in a meta-analysis to be “disturbing,” and that agency’s selective, incomplete inclusion of studies violated its own guidelines for conducting risk assessments), rev’d on other grounds, 313 F.3d 852, 862 (4th Cir. 2002) (Widener, J.) (holding that the issuance of the report was not “final agency action”)

Fail-Safe, LLC v. AO Smith Corp., 744 F. Supp. 2d 870, 889 (E.D. Wis. 2010) (“the court also finds the witness’s methodology unreliable because of how Dr. Keegan uniformly treated all evidence that undermined his underlying conclusion: unwarranted dismissal of the evidence or outright blindness to contrary evidence. … In fact, it is readily apparent that Dr. Keegan all but ‘cherry picked’ the data he wanted to use, providing the court with another strong reason to conclude that the witness utilized an unreliable methodology. * * * Dr. Keegan’s two reports are rich with examples of his ‘cherry picking’ of the evidence.”)

As noted, however, there are federal trial courts that are all too willing to suspend judgment and kick the case to the jury.  Here is a sampler of cases that found cherry picking to be an acceptable methodology, or at least a methodology sufficient to require that the case be submitted to the finder of fact.

In Berg v. Johnson & Johnson, the district court noted the defendants’ argument that proffered testimony is unreliable because witness “cherry-picked” data in order to form an opinion solely for purposes of litigation. 940 F.Supp. 2d 983, 991-92 (D.S.D. 2013). The trial judge, however, was not willing to look particularly closely at what was excluded or why:

“The only difference between his past and present research seems to exist in how he categorized his data. Defendants label this ‘cherry-picking’. The court views it as simply looking at the existing data from a different perspective.”

Id.  Of course, expert witnesses on opposite sides look at the case from different perspectives, but the question begged was whether the challenged expert witness had categorized data in an unprincipled way. Other cases of this ilk include:

United States v. Paracha, 2006 WL 12768, at *20 (S.D. N.Y. Jan. 3, 2006) (rejecting challenge to terrorism expert witness on grounds that he cherry picked evidence in conspiracy prosecution involving al Queda)

In re Chantix (Varenicline) Products Liab. Litig., 889 F. Supp. 2d 1272, 1288 (N.D. Ala. 2012) (“Why Dr. Kramer chose to include or exclude data from specific clinical trials is a matter for cross-examination, not exclusion under Daubert.“)

Bouchard v. Am. Home Prods. Corp., 2002 WL 32597992 at *7 (N.D. Ohio May 24, 2002) (“If Bouchard believes that [the expert]… ignored evidence that would have required him to substantially change his opinion, that is a fit subject for cross-examination, not a grounds for wholesale rejection of an expert opinion.”)

In re Celexa & Lexapro Prods. Liab. Litig., 927 F. Supp. 2d 758, 2013 WL 791780, at *5, *7, *8 (E.D. Mo. 2013) (Sippel, J.) (rejecting challenge to David Healy in antidepressant suicide case)

Allen v. Takeda Pharms., MDL No. 6:11-md-2299, No. 12-cv-00064, 2013 WL 6825953, at *11 (W.D. La. Dec. 20, 2013) (challenged expert witness in Actos litigation sufficiently explained his choices to be exonerated from charges of cherry picking)

In re NuvaRing Prods. Liab. Litig., No. 4:08–MD–1964 RWS, 2013 WL 791787 (E.D. Mo. Mar. 4, 2013) (“As to cherry picking data, the Eighth Circuit has recently made clear that such allegations should be left for crossexamination.”)

McClellan v. I-Flow Corp., 710 F. Supp. 2d 1092, 1114 (D. Ore. 2010) (“Defendants are correct that plaintiffs’ experts must elucidate how the relevant evidence lends support to their opinions by explaining…..”) (rejecting cherry picking but denying Rule 702 challenge based in part upon alleged cherry picking)

Rich v. Taser Internat’l, Inc., No. 2:09–cv–02450–ECR–RJJ, 2012 WL 1080281, at *6 (D. Nev. March 30, 2012) (noting the objection to cherry picking but holding that it was an issue for cross-examination)

In re Urethane Antitrust Litig., No. 04-1313-JWL, MDL No. 1616, 2012 WL 6681783, at *3 (D. Kan. Dec. 21, 2012) (allowing expert testimony that “certain events are consistent with collusion”; “the extent to which [an expert] considered the entirety of the evidence in the case is a matter for cross-examination.”)

In re Titanium Dioxide Antitrust Litig., No. RDB-10-0318, 2013 WL 1855980, 2013 U.S. Dist. LEXIS 62394 (D. Md. May 1, 2013) (rejecting Rule 702 cherry-picking challenge to an expert who cherry picked; witness’s selection of documents upon which to rely from a record that exceeded 14 million pages was not unreliable. “ If important portions of the record were overlooked, then the Defendants may address that issue at trial.”)

STATE COURTS

The situation in state courts is similarly chaotic and fragmented.

In Lakey v. Puget Sound Energy, Inc., the Washington Supreme Court resoundingly rejected “cherry picking” by expert witnesses in a public and private nuisance case against a local utility for fear of future illnesses from exposure to electro-magnetic frequency radiation (EMF).  Lakey v. Puget Sound Energy, Inc., 176 Wn.2d 909 (2013). The court held that the plaintiffs’ expert witnesses’ cherry-picking approach to data and studies was properly excluded under Rule 702. Their selective approach vitiated the reliability of his opinion with the consequence of :

“seriously tainting his conclusions because epidemiology is an iterative science relying on later studies to refine earlier studies in order to reach better and more accurate conclusions. Carpenter refused to account for the data from the toxicological studies, which epidemiological methodology requires unless the evidence for the link between exposure and disease is unequivocal and strong, which is not the case here. Carpenter also selectively sampled data within one of the studies he used, taking data indicating an EMF-illness link and ignoring the larger pool of data within the study that showed no such link, Carpenter’s treatment of this data created an improper false impression about what the study actually showed.”

Id.; see also “Washington Supreme Court Illustrates the Difference Between Frye and Rule 702” (April 15, 2013).

Other state Supreme Courts have recognized and rejected the gerrymandering of scientific evidence.  Betz v. Pneumo Abex LLC, 2012 WL 1860853, *16 (May 23, 2012 Pa. S. Ct.)(“According to Appellants, moreover, the pathologist’s self-admitted selectivity in his approach to the literature is decidedly inconsistent with the scientific method. Accord Brief for Amici Scientists at 17 n.2 (“‘Cherry picking’ the literature is also a departure from ‘accepted procedure’.”)); George v. Vermont League of Cities and Towns, 2010 Vt. 1, 993 A.2d 367, 398 (Vt. 2010)(expressing concern about how and why plaintiff’s expert witnesses selected some studies to include in their “weight of evidence” methodology.  Without an adequate explanation of selection and weighting criteria, the choices seemed “arbitrary” “cherry picking.”); Bowen v. E.I. DuPont de Nemours & Co., 906 A.2d 787, 797 (Del. 2006) (noting that expert witnesses cannot ignore studies contrary to their opinions).

Lower state courts have also quashed the cherry-picking harvest. Scaife v. AstraZeneca LP, 2009 WL 1610575, at *8 (Del. Super. June 9, 2009) (“Simply stated, the expert cannot accept some but reject other data from the medical literature without explaining the bases for her acceptance or rejection.”); see also In re Bextra & Celebrex Prod. Liab. Litig., No. 762000/2006, 2008 N.Y. Misc. LEXIS 720, at *47 (Sup. Ct. N.Y. Co. Jan 7, 2008) (stating that plaintiffs must show that their experts “do not ignore contrary data”).

The Nebraska Supreme Court appears to recognize the validity of considering the existence of cherry-picking in expert witness gatekeeping.  In practice, however, that Court has shown an unwillingness to tolerate close scrutiny into what was included and excluded from the expert witness’s consideration.  King v. Burlington No. Santa Fe Ry, ___N.W.2d___, 277 Neb. Reports 203, 234 (2009)(noting that the law does “not preclude a trial court from considering as part of its reliability inquiry whether an expert has cherry-picked a couple of supporting studies from an overwhelming contrary body of literature,” but ignoring the force of the fallacious expert witness testimony by noting that the questionable expert witness (Frank) had some studies that showed associations between exposure to diesel exhaust or benzene and multiple myeloma).

“Of all the offspring of time, Error is the most ancient, and is so old and familiar an acquaintance, that Truth, when discovered, comes upon most of us like an intruder, and meets the intruder’s welcome.”

Charles MacKay, Extraordinary Popular Delusions and the Madness of Crowds (1841)

Posted in Causation, Rule 702 | Comments Off on The Fallacy of Cherry Picking As Seen in American Courtrooms

On The Quaint Notion That Gatekeeping Rules Do Not Apply to Judges

April 27th, 2014

In In re Zurn Pex Plumbing Prods. Liab. Litig., 644 F.3d 604 (8th Cir. 2011), the United States Court of Appeals for the Eighth Circuit rejected the defendant’s argument that a “full and conclusive” Rule 702 gatekeeping procedure was required before a trial court could certify a class action under the Federal Rules. The Circuit remarked that “[t]he main purpose of Daubert exclusion is to protect juries from being swayed by dubious scientific testimony,” an interest “not implicated at the class certification stage where the judge is the decision maker.”  Id. at 613.

Surely, one important purpose of Rule 702 is to protect juries against dubious scientific testimony, but judges are not universally less susceptible to dubious testimony.  There are many examples of judges being misled by fallacious scientific evidence, especially when tendentiously presented by advocates in court.  No jury need be present for dubious science testimony + “zealous” advocacy to combine to create major errors and injustice.  See, e.g., Wells v. Ortho Pharmaceutical Corp., 615 F. Supp. 262 (N.D. Ga. 1985)(rendering verdict for plaintiffs after bench trial), aff’d and rev’d in part on other grounds, 788 F.2d 741 (11th Cir.), cert. denied, 479 U.S.950 (1986); Hans Zeisel & David Kaye, Prove It With Figures: Empirical Methods in Law and Litigation § 6.5 n.3, at 271 (1997) (characterizing Wells as “notorious,” and noting that the case became a “lightning rod for the legal system’s ability to handle expert evidence.”).  Clearly Rule 702 does not exist only to protect juries.

Nemo iudex in causa sua! Perhaps others should judge the competence of judges’ efforts at evaluating scientific evidence.  At the very least, within the institutional framework of our rules of civil procedure and evidence, Rule 702 creates a requirement of structured inquiry into expert opinion testimony before the court.  That gatekeeping inquiry, and its requirement of a finding, subject to later appellate review and to public and professional scrutiny, are crucial to the rendering of intellectual due process in cases that involve scientific and technical issues.  The Eighth Circuit was unduly narrow in its statement of the policy bases for Rule 702, and their applicability to class certification.

The case of Obrey v. Johnson, 400 F.3d 691 (9th Cir. 2005) provides another cautionary tale about the inadequacies of judges in the evaluation of scientific and statistical evidence.  The plaintiff, Mr. Obrey, sued the Navy on a claim of race discrimination in promoting managers at the Pearl Harbor Naval Shipyard.  The district court refused plaintiff’s motion to admit the testimony of a statistician, Mr. James Dannemiller, President of the SMS Research & Marketing Services, Inc. The district court also excluded much of plaintiff’s anecdotal evidence, and entered summary judgment.  Id. at 691 – 93.

On appeal, Obrey claimed that Dannemiller’s report showed “a correlation between race and promotion.” Id. at 693. This vague claim seemed good enough for the Ninth Circuit, which reversed the district court’s grant of summary judgment and remanded for trial.

The Ninth Circuit’s opinion does not tell us what sort of correlation was supposedly shown by Mr. Dannemiller. Was it Pearson’s r?  Or Jaspen’s multi-serial coefficient? Spearman’s ρ?  Perhaps Kendall’s τ? Maybe the appellate court was using correlation loosely, and Mr. Dannemiller had conducted some other sort of statistical analysis. The district court’s opinion is not published and is not available on Westlaw.  It is all a mystery. More process is due the litigants and the public.

Even more distressing than the uncertainty as to the nature of the correlation is that the Ninth Circuit does not tell us what the correlation “effect size” was, or whether the correlation was statistically significant.  If the Circuit did not follow strict hypothesis testing, perhaps it might have told us the extent of random error in the so-called correlation.  The Circuit did not provide any information about the extent or the precision of the claim of a “correlation”; nor did the Circuit assess the potential for bias or confounding in Mr. Dannemiller’s analysis.

Indeed, the Ninth Circuit seemed to suggest that Mr. Dannemiller never even showed a correlation; rather the court described Mr. Dannemiller as having opined that there was “no statistical evidence in these data that the selection process for GS-13 through GS-15 positions between 1999 and 2002 was unbiased with respect to race.” Id. at 694. Reading between the lines, it seems that the statistical evidence was simply inconclusive, and Mr. Dannemiller surreptitiously shifted the burden of proof and offered an opinion that the Navy had not ruled out bias. The burden, of course, was on Mr. Obrey to establish a prima facie case, but the appellate court glossed over this fatal gap in plaintiff’s evidence.

On appeal, the Navy pressed its objections to the relevance and reliability of Mr. Dannemiller’s opinions. Brief of the Navy, 2004 WL 1080083, at *1 (April 7, 2004).  There seemed to be no dispute that Mr. Dannemiller’s “study” was based entirely upon “statistical disparities,” which failed to take into account education, experience, and training.  Mr. Dannemiller appeared to have simplistically compared race make up of the promoted workers, ignoring the Navy’s showing of the relevancy of education, experience, and training.  Id. at *13, 18.

The Ninth Circuit not only ignored the facts of the case, it ignored its own precedents.  See Obrey v. Johnson, 400 F.3d at 696 (citing and quoting from Coleman v. Quaker Oats Co., 232 F.3d 1271, 1283 (9th Cir. 2000) (“Because [the statistics] failed to account for many factors pertinent to [the plaintiff], we conclude that the statistics are not enough to take this case to trial.”). The court, in Obrey, made no effort to distinguish its treatment of the parties in Coleman, or to justify its decision as to why the unspecified, unquantified, mysterious statistical analysis of Mr. Dannemiller sufficed under Rule 702. The Circuit cryptically announced that “Obrey’s evidence was not rendered irrelevant under Rule 402 simply because it failed to account for the relative qualifications of the applicant pool.”  Obrey, 400 F.3d at 695.  Citing pre-Daubert decisions for the most part (such as Bazemore), the Ninth Circuit persuaded itself that Rule 702 requires nothing more than simple relevancy. Had the Circuit taken even a cursory look at Bazemore, it would have seen that the case involved a much more involved multiple regression than whatever statistical analysis Mr. Dannemiller propounded.  And the Ninth Circuit would have seen that even the Bazemore decision acknowledged that there may be

“some regressions so incomplete as to be inadmissible as irrelevant… .”

478 U.S. 385, 400 n.10 (1986). It is difficult to imagine a discrimination claim analysis more incomplete than one that did not address education, training, and experience.

Sadly, neither the Navy’s nor Mr. Obrey’s brief, 2004 WL 545873 (Feb. 4, 2004) provided any discussion of the nature, quality, findings, or limits of Mr. Dannemiller’s statistical analysis.  The Navy’s brief referred to Mr. Dannemiller as a “purported” expert.  His resume, available online, shows that Mr. Dannemiller studied history as an undergraduate, and has a master’s degree in sociology. He is the president of SMS Research, a consulting company.

The taxpayers deserved better advocacy from the Department of Justice, and greater attention to statistical methodology from its appellate judges.  See ATA Airlines, Inc. v. Federal Exp. Corp., 665 F.3d 882, 888-96 (2011) (Posner, J.) (calling for lawyers and judges to do better in understanding and explaining, in plain English, the statistical analyses that are essential to their cases). Judges at level need to pay greater attention to the precepts of Rule 702, even when there is no jury around to be snuckered.

Posted in Expert Witnesses, Rule 702, statistical evidence | Comments Off on On The Quaint Notion That Gatekeeping Rules Do Not Apply to Judges

Duty of Objectivity for Expert Witnesses – Up North and Abroad

April 19th, 2014

In the United States, we talk of the requirements for admissibility of expert witness opinion testimony; proffered testimony must be relevant and reliable.  If the requirements go unsatisfied, the legal consequence is usually limited to the preclusion of the expert witness’s challenged opinion.  If the opinion is necessary to support the sponsoring party’s claim or defense, the further legal consequence may be the entry of judgment adverse to the retaining party.

A few states have permitted a party to sue its own expert witness for “expert malpractice,” committed in the scope of the witness’s engagement as an expert witness. See, e.g., LLMD of Michigan Inc. v. Jackson-Cross Co., 740 A.2d 186 (Pa. 1991). Fewer states permit the adverse party to sue its adversary’s expert witness. Davis v. Wallace, 565 S.E.2d 386 (W. Va. 2002).

In the United Kingdom and Canada, courts impose duties directly upon expert witnesses themselves.  The following enumeration is frequently cited as setting forth the independent duties, owed to the court, by expert witnesses:

“1. Expert evidence presented to the Court should be, and should be seen to be, the independent product of the expert uninfluenced as to form or content by the exigencies of litigation.

2. An expert witness should provide independent assistance to the Court by way of objective unbiased opinion in relation to matters within his expertise. An expert witness in the High Court should never assume the role of an advocate.)

3.  An expert witness should state the facts or assumption upon which his opinion is based. He should not omit to consider material facts which could detract from his concluded opinion.

4.  An expert witness should make it clear when a particular question or issue falls outside his expertise.

5.  If an expert’s opinion is not properly researched because he considers that insufficient data is available, then this must be stated with an indication that the opinion is no more than a provisional one.  In cases where an expert witness who has prepared a report could not assert that the report contained the truth, the whole truth and nothing but the truth without some qualification, that qualification should be stated in the report.

6.  If, after exchange of reports, an expert witness changes his view on a material matter having read the other side’s expert’s report or for any other reason, such change of view should be communicated (through legal representatives) to the other side without delay and when appropriate to the Court.

7.   Where expert evidence refers to photographs, plans, calculations, analyses, measurements, survey reports or other similar documents, these must be provided to the opposite party at the same time as the exchange of reports.”

National Justice Compania Naviera S.A. v. Prudential Assurance Co. Ltd., (“The Ikarian Reefer”), [1993] 2 Lloyd’s Rep. 68 at 81-82 (Q.B.D.), rev’d on other grounds [1995] 1 Lloyd’s Rep. 455 at 496 (C.A.)(embracing the enumeration of duties).

It is unclear, however, what the consequences of breach of these duties are.  Often the sponsoring party may be complicit in the breach, and the harm will be to the court and the adverse party. 

In the United States, perhaps the heavy lifting of judicial gatekeeping, required by Federal Rule of Evidence 702, might be assisted in recognizing these independent duties of expert witnesses.

The duties of expert witnesses, set out in the The Ikarian Reefer, have been generally accepted by courts in Ontario and throughout Canada. See, e.g., Frazer v. Haukioja, 2008 CanLII 42207, at ¶141 (O.S.C.) (Moore, J.) (quoting from Ikarian Reefer).  The Ontario court system decided not to leave compliance with these duties to chance or instructions from counsel. Starting in 2010, Ontario’s New Rule 4.1 of its Rules of Civil Procedure went into effect to define explicitly the duties of an expert witness:

RULE 4.1 DUTY OF EXPERT

4.1.01 (1)

It is the duty of every expert engaged by or on behalf of a party to provide evidence in relation to a proceeding under these rules,

(a) to provide opinion evidence that is fair, objective and non-partisan;

(b) to provide opinion evidence that is related only to matters that are within the expert’s area of expertise; and

(c) to provide such additional assistance as the court may reasonably require to determine a matter in issue.

Duty Prevails

The duty stated in the Ontario Rule 4.1 trumps any contractual or positional obligations expert witnesses may owe to the parties that engaged them. Remarkably, the Ontario courts do not leave to chance whether expert witnesses will understand and act upon their mandated obligations.  Ontario Rule 53,  subrule 53.03(2.1), requires expert witnesses to submit signed acknowledgment forms (Form 53, below), which recite their understand of their duties.

FORM 53

 Courts of Justice Act

 ACKNOWLEDGMENT OF EXPERT’S DUTY

 1. My name is _______________________________ (name). I live at

___________________ (address), in the __________________ (name of city) of _________________________ (name of province/state).

2. I have been engaged by or on behalf of ___________________ (name of party/parties) to provide evidence in relation to the above-noted court proceeding.

3. I acknowledge that it is my duty to provide evidence in relation to this proceeding as follows:

a. To provide opinion evidence that is fair, objective and non-partisan;

b. To provide opinion evidence that is related only to matters that are within my area of expertise; and

c. To provide such additional assistance as the court may reasonably require, to determine a matter in issue.

4. I acknowledge that the duty referred to above prevails over any obligation which I may owe to any party by whom or on whose behalf I am engaged.

 

Date: ___________________  ___________________________                                                                                             (signature)

Posted in Expert Witnesses, Rule 702 | Comments Off on Duty of Objectivity for Expert Witnesses – Up North and Abroad

Relative Risk of Two in Medical Malpractice Cases

April 14th, 2014

Counsel for plaintiffs and defendants in toxic tort cases are well aware of the need to show a sufficiently large relative risk, greater than two, to have sufficient evidence to satisfy the burden of proof on proximate causation between a known causal exposure and a specific plaintiff’s injury.  As Judge Jack Weinstein wrote 30 years ago, “[a] government administrative agency may regulate or prohibit the use of toxic substances through rulemaking, despite a very low probability of any causal relationship.  A court, in contrast, must observe the tort law requirement that a plaintiff establish a probability of more than 50% that the defendant’s action injured him. … This means that at least a two-fold increase in incidence of the disease attributable to Agent Orange exposure is required to permit recovery if epidemiological studies alone are relied upon.” In re Agent Orange Product Liab. Litig., 597 F. Supp. 740, 785, 836 (E.D.N.Y. 1984), aff’d 818 F.2d 145, 150-51 (2d Cir. 1987)(approving district court’s analysis), cert. denied sub nom. Pinkney v. Dow Chemical Co., 487 U.S. 1234 (1988).

In toxic tort cases, the risk ratio at issue allegedly results from a higher incidence of the disease in exposed persons compared to the incidence in unexposed persons.  A similar risk ratio issue occurs in medical malpractice cases when a healthcare provider negligently fails to administer a therapy, or fails to administer a therapy in a timely fashion, to the detriment of the plaintiff.  In instances in which the therapy is almost always efficacious, the risk ratio of a bad patient outcome will be very high, and the corresponding probability that the bad outcome would have been avoided by proper or timely therapy will be close to 100 percent.  On the other hand, for some therapies, even timely administration is efficacious in a limited number of cases, less often than the 50-plus percent of cases that would support a proximate cause opinion between the allegedly negligent failure to administer therapy and the patient’s bad health outcome.

Unfortunately, the relative risk issue goes unlitigated in many cases, in New York and elsewhere. One recurring malpractice claim involves the alleged failure to administer clot-busting drugs to ischemic stroke patients.  One such drug, tissue plasminogen activator (t-PA), which was approved by the Food and Drug Administration in 1996, can substantially reduce brain damage if administered within three hours of stroke onset.  Even if administered within the crucial therapeutic time window, however, t-PA will benefit only about 30 percent of patients, and there is no medical “fingerprint”that identifies who has benefitted from the t-PA. In Samaan v. St. Joseph Hospital, 670 F.3d 21 (1st Cir. 2012), the First Circuit acted on its gatekeeping responsibilities to perscrutate the risk evidence and found that it fell short of what is required by Federal Rule of Evidence 702, and the “more likely than not” standard for civil cases. See also Smith v. Bubak, 643 F.3d 1137, 1141–42 (8th Cir.2011) (rejecting relative benefit testimony and suggesting in dictum that absolute benefit “is the measure of a drug’s overall effectiveness”); Young v. Mem’l Hermann Hosp. Sys., 573 F.3d 233, 236 (5th Cir.2009) (holding that Texas law requires a doubling of the relative risk of an adverse outcome to prove causation), cert. denied, ___ U.S. ___, 130 S.Ct. 1512 (2010).

Samaan and the cases cited by the First Circuit are hardly unique; the size of the relative risk issue has helped the defense prevail in other t-PA and similar malpractice cases around the country. Kava v. Peters, 450 Fed.Appx. 470, 478-79 (6th Cir. 2011) (affirming summary judgment for defendants when plaintiffs expert witnesses failed to provide clear testimony that plaintiff specific condition would have been improved by timely administration of therapy); Bonesmo v. The Nemours Foundation, 253 F.Supp. 2d 801, 809 (D.Del. 2003); Joshi v. Providence Health System of Oregon Corp., 342 Or. 152, 156, 149 P. 3d 1164, 1166 (2006) (affirming directed verdict for defendants when expert witness testified that he could not state, to a reasonable degree of medical probability, beyond 30%, that administering t-PA, or other anti-coagulant would have changed the outcome and prevented death); Ensink v. Mecosta County Gen. Hosp., 262 Mich.App. 518, 687 N.W.2d 143 (Mich.App. 2004) (affirming summary judgment for hospital and physicians when patient could not greater than 50% probability of obtaining a better result had emergency physician administered t-PA within three hours of stroke symptoms); Merriam v. Wanger, 757 A.2d 778, 2000 Me. 159 (2000) (reversing judgment on jury verdict for plaintiff on grounds that plaintiff failed to show that defendant failure to act were, more likely than not, a cause of harm). In Michigan, the holding of the t-PA and similar medical malpractice cases has been codified by statute:

“In an action alleging medical malpractice, the plaintiff has the burden of proving that he or she suffered an injury that more probably than not was proximately caused by the negligence of the defendant or defendants. In an action alleging medical malpractice, the plaintiff cannot recover for loss of an opportunity to survive or an opportunity to achieve a better result unless the opportunity was greater than 50%.”

Mich. Comp. Laws § 600.2912a(2) (2009).  But see O’Neal v. St. John Hosp. & Med. Ctr., 487 Mich. 485, 791 N.W.2d 853 (Mich. 2010) (affirming denial of summary judgment when failure to administer therapy (not t-PA) in a timely fashion more than doubled the risk of stroke). In one unpublished Kentucky case, involving t-PA, the court seemed to acknowledge the general principle, but became confused as to whether 30 percent, was a reasonable probability. Lake Cumberland, LLC v. Dishman, 2007 WL 1229432, *5 (Ky. Ct. App. 2007) (unpublished) (citing without critical discussion an apparently innumerate opinion of expert witness Dr. Lawson Bernstein).

Despite the success of medical malpratice defense counsel in litigating dispositive motions in t-PA cases, the issue seems to go unnoticed in New York cases. For instance, in Gyani v. Great Neck Medical Group, a stroke victim sued on various allegations of medical malpractice, including failure to administer t-PA.   N.Y. S.Ct. for Nassau Cty, 2011 WL 1430037 (April 4, 2011). The trial court denied summary judgment on proximate cause grounds, and noted that

“[t]he plaintiffs’ expert ultimately opines that the failure to administer t-PA allowed Gyani’s stroke to go untreated and progress to the point of her being locked-in permanently which would not have happened had t-PA been administered.”

From the court’s opinion, it would appear that defense counsel never pressed beyond this conclusory opinion, devoid of quantified relative risk. Behind the curtain of “ultimate” opinion is an expert without a meaningful basis for his opinion.  It is time to pull the curtain.

Posted in Causation, Expert Witnesses, Risk and Risk Factor, Rule 702 | Comments Off on Relative Risk of Two in Medical Malpractice Cases

“Dummkopf! You’re Fired” – Judge Posner on Expert Witness Gatekeeping

February 15th, 2014

“Equity is a roguish thing, for law we have a measure, know what to trust to. equity is according to the conscience of him that is chancellor, and as that is larger or narrower so is equity. ’Tis all one as if they should make the standard for the measure we call a foot, a chancellor’s foot. What an uncertain measure would this be. One chancellor has a long foot, another a short foot, a third an indifferent foot; ’tis the same thing in the chancellor’s conscience.”

John Selden, The Table Talk of John Selden (1689), at 61 (Samuel Harvey Reynolds, ed., Oxford 1892).

*  *  *  *  *  *  *  *

As Equity in days of old varied with the size of the Chancelor’s foot, today the quality of judicial gatekeeping of expert witness opinion testimony varies with the acumen of the trial judge in the area of the challenged witness’s expertise.  In Apple Inc. v. Motorola, Inc., 2012 WL 1959560 (N.D. Ill. 2012), the parties challenged each other’s damages expert witnesses under Federal Rule 702, only to find that the trial judge was considerably more astute than their expert witnesses. When it came to assessing the validity and reliability of the damages opinions, the trial judge was a veritable “big foot,” kicking the courthouse door closed to some dodgy damage calculations.

The Hon. Richard Posner is a judge of the United States Court of Appeals, for the Seventh Circuit.  Judge Posner is also an economist and a stalwart of law-and-economics jurisprudence. In Apple v. Motorola, Judge Posner sat by designation as a trial judge.   Instead of judging whether a trial judge had abused his or her discretion in admitting or excluding expert witness testimony, Judge Posner had to put his own discretion on the line. 

Judge Posner identified the biggest challenge in gatekeeping as:

“distinguish[ing] between disabling problems with the proposed testimony, which are a ground for excluding it, and weaknesses in the testimony, which are properly resolved at the trial itself on the basis of evidence and crossexamination.”

Apple Inc. v. Motorola, Inc., 2012 WL 1959560, *1. Posner cites old caselaw, arguably superseded by the current Rule 702, for the chestnut that:

“the judge should not exclude evidence simply because he or she thinks that there is a flaw in the expert’s investigative process which renders the expert’s conclusions incorrect. The judge should only exclude the evidence if the flaw is large enough that the expert lacks ‘good grounds’ for his or her conclusions.”

Id. (quoting In re Paoli R.R. Yard PCB Litigation, 35 F.3d 717, 746 (3d Cir.1994)). Of course, flawed reasoning or methodology is the essence of what deprives anyone from making an claim to knowledge; this little chestnut is not very nourishing.

Judge Posner does better in “operationalizing” Kumho Tire for making the distinction between flaws that weaken, and those that vitiate, the epistemic bases for opinions.  Whether an expert witness “employs in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field” is a key test for determining on which side of the distinction a challenged opinion falls. Id. at *2 (quoting Kumho Tire Co. v. Carmichael, 526 U.S. 137, 152 (1999)).

“The Kumho principle implies that an expert witness must provide reasons for rejecting alternative hypotheses ‘using scientific methods and procedure’ and the elimination of those hypotheses must be founded on more than ‘subjective beliefs or unsupported speculation, … .’ ”

Id. at *8 (internal citations omitted).

Posner tempers, and dilutes, Kumho by qualifying the Kumho principle to require a testifying expert to use the same approach as used in the relevant field “if it is feasible for him to do so.” Id. at *3. It is always feasible, but rarely seen, for an expert witness to profess insufficient knowledge, facts, or data to give an opinion. Posner goes further and rewrites the statute, Rule 702, which was designed to keep uncertainty from being masqueraded as certainty:

“when the plaintiff has done his best to prove damages his inability to dispel uncertainty concerning the accuracy of his claim is not fatal. But if an expert witness fails to conduct a responsible inquiry that would have been feasible to conduct, his failure cannot be excused by reference to the principle that speculation is permitted in the calculation of damages; that permission presupposes the exhaustion of feasible means of dispelling uncertainty. Uncertainty is a bad; it is tolerated only when the cost of eliminating it would exceed the benefit.”

Id. at *5.  Sometimes the best efforts to eliminate uncertainty will leave us uncertain.  And the issue of acceptable uncertainty is not necessarily tied to the cost of eliminating it.

Nevertheless, Posner goes on to identify multiple unreasonable assumptions, alternative inferences, missing data, and flawed methods that vitiated most of the opinions before him in Apple v. Motorola.

Judge Posner applies the Kumho principle in the context of damages with a series of counterfactual Gedanken experiments.  He asks what if the plaintiff’s expert witness were working for the defendant (and vice versa), and charged with ascertaining the lowest cost to avoid infringing the plaintiff’s patent.  If the expert submitted the most expensive approach, or an extremely speculative, answer, the defendant would indeed fire the expert: “Dummkopf! You’re fired.” Id. at *9. And if the expert offered unverified evidence that came from an interested, adverse party, the expert’s opinion would again be worthy of no consideration.

Judge Posner is at home in the world of assessing economic damages, and as a “Chancellor,” he proved to have a very big foot indeed. The parties’ expert witnesses came up short on almost every damages opinion examined.

Not all evidentiary issues can be resolved by Judge Posner’s economic reductionism as neatly as the damages issues in this patent infringement case.  Posner’s approach is less satisfying in the context of health effects litigation, where expert witnesses will often have the option of proclaiming inadequate knowledge or method to ascertain general or specific causation.  Still Posner’s Gedanken experiments are contagious.  Suppose we were confronted with a birth defect case in which plaintiffs claimed that the mother’s use of a medication in early pregnancy caused the child’s birth defect.  It is generally conceded that most such birth defects have no known cause, but the plaintiffs’ expert witnesses contend that they have conducted a differential etiology and ascribed causation of the baby’s defect to the mother’s use of the defendant’s   medication.  Suppose there was a serious economic (or life-and-death) consequence to the expert’s opinion.  If the defect were drug-induced, there was no surgical or other correction, but if the defect were “idiopathic,” it could be readily repaired surgically.  Would the expert witness, acting as a treating physician, withhold the treatment because he was “reasonably medically certain” that the defect was caused by the drug?  I don’t think so.  Dr. Dummkopf, you’re fired!

Posted in Expert Witnesses, Rule 702 | Comments Off on “Dummkopf! You’re Fired” – Judge Posner on Expert Witness Gatekeeping

The Capture of the Public Health Community by the Litigation Industry

February 10th, 2014

The American Public Health Association (APHA) is a significant organization ostensibly committed to the improvement of public health. Among its many activities, the APHA publishes a journal, the American Journal of Public Health.  Here is how the APHA describes itself and its activities to advance public health:

“The American Public Health Association champions the health of all people and all communities. We strengthen the profession of public health, share the latest research and information, promote best practices and advocate for public health issues and policies grounded in research. We are the only organization that combines a 140-plus year perspective, a broad-based member community and the ability to influence federal policy to improve the public’s health.”

How could anyone be against the APHA?

A dubious development in the APHA’s history was its evolution as a tool of the litigation industry.  In 2004, after several years of lobbying, agents of the litigation industry managed to push a policy statement past the Association’s leadership, to condemn the requirement of evidence-based reasoning in federal courts in the United States.

The litigation industry’s victory is memorialized in the “Final Minutes of Meetings of the APHA Governing Council, ” held in November 2004, when the industry’s attack on evidence-based science and data transparency, known as “Policy Number: 2004-11 Threats to Public Health Science,” was adopted as an APHA policy statement.

“2004-11” was published in the American Journal of Public Health and is still available on the APHA website, as Policy Number: 2004-11 Threats to Public Health Science.  I have excerpted contentions and recommendations from the APHA policy, in the left column of the chart, below.  My comments are on the right.

 

APHA Policy

Comment
“Acknowledging that within science, absolute proof and perfect information are rare;” Notice the false dichotomy between absolute proof and perfect information and the entire remaining spectrum of scientific information.  This dichotomization has been part of the litigation strategy of passing off hypotheses, preliminary conclusions, unreplicated findings, etc., as though they were acceptable bases for causal conclusions.
“Recognizing that special interests have exploited the nature of science, specifically scientific uncertainty, to delay protective legal and/or regulatory action;” Notice the asymmetry of the accusations; the APHA apparently has no concern for “special interests” that exploit the nature of science by passing off hypotheses as conclusions, and seek to accelerate protective legal and regulatory action by manufacturing faux scientific conclusions.
“Acknowledging that some public health decisions must be made in the absence of perfect scientific information;” “Le mieux est l’ennemi du bien.” But isn’t the good also the enemy of the shabby, dodgy, and fraudulent? Notice again the false dichotomy between “perfect” information and everything else, as though our failing to achieve the perfect opens the door to the worst. True, of course, that sometimes action is needed on incomplete records, but such action is rarely needed for compensation claims.
“Recognizing that special interests, under the guise of a call for “sound science” have sponsored and promoted changes in public policy that have weakened and continue to threaten public health protections;” If the call for sound science cannot be sustained, then this rhetorical gambit will blowback hard on those “special interests.”  Why are these putative scientists, at APHA, so afraid of sound science?
“Recognizing that special interests have challenged highly regarded public health research and researchers, and inappropriately characterized established scientific methods as ‘junk science’;” Mon Dieu!  How cheeky of those special interests.  See the discussion of Dr. Barry S. Levy, below.
“Recognizing that the Daubert decision has propagated misinterpretations and misapplications of scientific principles relied upon throughout the public health sciences, such as insisting that any epidemiologic study that is relied on to support causation demonstrate a twofold increase in risk as well as a reliance on significance testing to determine which scientific findings are to be allowed as evidence;” This contention misunderstands the basic nature of evidence law. Studies, whether they have statistically significant results, or not, are rarely admissible in evidence.  What is admissible, or not, are opinions of duly qualified expert witnesses, who explain and justify the epistemic warrant for their opinions.  With respect to general causation opinions, expert witnesses will often have to show that they have properly ruled out chance, bias, and confounding to arrive at a causal conclusions.  Significance testing can be abused, in both directions, but the APHA ignores the need for having some quantitative approach to assess random variability. As for relative risks greater than two, the APHA is correct that general causation may often be found with small relative risks, but the attribution of causation in an individual claimant often can be made only on probabilistic inferences that will require relative risks greater than two, or even larger.
“Recognizing that special interests are engaged in a campaign to extend Daubert’s reach to those states that have not embraced prescriptive definitions of scientific reliability.” So the APHA makes common cause with those “special interests,” which would abolish all limits on the admissibility of expert witness opinions, and all normative assessments of scientific research.  This position ignores the prescriptive aspect of methodology, and the nature of epistemic warrant in a methodology.

 

What follows from these contentions? 

“Therefore, APHA:”

“Opposes legislation or administrative policies that attempt to define the characteristics of valid public health science, or dictate prescriptive scientific methodologies; and” Admittedly, defining good science is very difficult, but the law often works like science as defining health as the absence of disease.  There are obviously some well-known pathologies of scientific method, and it hardly seems extravagant to urge courts to avoid flaws, fallacies, and fraud.  
“Supports the efforts of other scientific organizations to promote the government’s ability to utilize the best available science to protect the public’s health; and” Of course, sometimes the “best” available science is rather shabby. 
“Urges friend of the court briefs that address the problem inherent in the adoption of Daubert and Daubert-like court rulings, the application of Daubert in regulatory proceedings, and when judges misinterpret scientific evidence in their implementation of the Daubert ruling.” We do not see many APHA-types deploring jury verdicts that offend scientific sensibilities; and so the APHA’s urging here seems again rather one-sided and partisan.  The fact, however, that judges’ misinterpretations of scientific evidence can be criticized publicly is one of the key differences that separates judicial gatekeeping from the black box of jury determinations.

In 2005, the APHA published, in its journal, APJH, a special supplement, “Scientific Evidence and Public Policy,” with

“academic analysis of the conflicts arising in the use of science in regulatory, civil and criminal proceedings. This special issue examines how recent developments in the legal and regulatory arenas have emboldened corporations involved in civil litigation and regulatory proceedings to accuse adversaries of practicing ‘junk science’.”

Apparently, the APHA was not, and is not, concerned with the emboldening the  litigation industry and its efforts to subvert the truth-finding function of civil litigation. 

David Michaels served as the guest editor for the APJH supplement.  Michaels repeated many of the contentions of the 2004 Policy Statement, above, and he added some new ones of his own:

  • Judges are no better than juries in assessing scientific evidence.
  • Scientists evaluate all the evidence by applying a “weight-of-the-evidence” approach.
  • Uncertainty in science is normal and does not mean the underlying science flawed.

David Michaels, “Editorial: Scientific Evidence and Public Policy,” 95 (Supp. 1) Am. J. Pub. Health S5 (2005). These are all serious half truths.  Many judges are quite astute when evaluating scientific evidence, but even the lowest aptitude judges must give articulated reasons for their decisions, which opens up a public process of comment, correction, and criticism.  Juries vote in secret, without having to explain or justify their verdicts.  Scientists, metaphorically speaking, weigh evidence, as do non-scientists, but this opaque metaphor hardly explicates the process of how scientists arrive at conclusions about causal relationships.  And uncertainty is a condition of many scientific fields, but the error lies in trying to pass off tentative, uncertain, preliminary observations and findings as knowledge.

Michaels sees the development of judicial gatekeeping as favoring “the powerful,” and hurting “the weak and vulnerable.” Id. Michaels did not seem to mind if his editorial recommendations favored the litigation industry and hurt the truth.  He now heads up the Occupational Health & Safety Administration.

Here is how Michaels and the APHA described the funding for the AJPH supplement:

“Support for the supplement was provided through unrestricted funding to the Project on Scientific Knowledge and Public Policy (SKAPP) from the Common Benefit Litigation Trust, a fund established by court order in the Silicone Gel Breast Implant Products Liability Litigation. SKAPP is an initiative of scholars that examines the application of scientific evidence in the legal and regulatory arenas. SKAPP is based at the George Washington University School of Public Health and Health Services; more information is available at www.DefendingScience.org.”

See APHA website <http://www.apha.org/about/news/pressreleases/2005/05arenas.htm>, last visited on February 10, 2014.

This pseudo-disclosure is perhaps the most fraudulent aspect of the entire APHA enterprise.  The Common Benefit Trust was a fund that was held back from settlement monies paid by defendants in the silicone gel breast implant litigation.  The Trust was nothing more than the Plaintiffs’ Steering Committee’s war chest, from which it could advance litigation goals within MDL 926 (silicone breast implant cases).  Ironically, the appointment of neutral, court-appointed expert witnesses led to the triumph of “sound science,” and the collapse of the plaintiffs’ counsel house of cards.  Rather than returning their litigation expense fund to the claimants, the plaintiffs’ counsel found a more worthwhile recipient — SKAPP — to advance their litigation goals, if not for MDL 926, then for the next MDL, and the next, and the next….  See SKAPP A LOT; and Conflicted Public Interest Groups.

* * * * * * *

The same year that the APHA published the SKAPP-inspired and funded challenges to Federal Rules of Evidence 702, the APHA awarded its most prestigious award, the Sedgwick Medal, to a physician whose opinions had routinely been found to be unreliable and irrelevant in various litigation industry efforts. “Barry Levy Wins APHA’s Oldest and Most Prestigious Award, the Sedgwick Medal.” (December 11, 2005).

Perhaps the APHA had Levy in mind when it complained that “special interests have challenged highly regarded public health … researchers….”  Dr. Levy seems to have less favorable accolades from trial and appellate judges.  For instance, one federal judge found Levy engaged in a dubious enterprise to manufacture silicosis claims in Mississippi.  In re Silica Products Liability Litigation, 398 F. Supp. 2d 563, 611-16, 622 & n.100 (S.D. Texas 2005) (expressing particular disappointment with Dr. Barry Levy, who although not the worst offender of a bad lot of physicians, betrayed his “sterling credentials” in a questionable enterprise to manufacture diagnoses of silicosis for litigation).[1] Interestingly, Judge Jack’s opinion was not mentioned in the APHA press release for Dr. Levy’s award ceremony.


[1] See Schachtman, Silica Litigation: Screening, Scheming & Suing; Washington Legal Foundation Critical Legal Issues Working Paper Series No. 135 (Dec. 2005) (exploring the ethical and legal implications of the entrepreneurial litigation in which Levy and others were involved). See also Lofgren v. Motorola, Inc., 1998 WL 299925, No. CV 93-05521 (Ariz. Super. Ct., Maricopa Cty. June 1, 1998); Harman v. Lipari, N.J. L. Div. GLO-L-1375-95, Order of Nov. 3, 2000 (Tomasello, J.) (barring the use of Barry Levy in class action for medical monitoring damages); Castellow v. Chevron USA, 97 F.Supp. 2d 780, 793-95 (S.D. Tex. 2000); Knight v. Kirby Inland Marine Inc., 482 F.3d 347 (5th Cir. 2007); Watts v. Radiator Specialty Co., 990 So. 2d 143 (Miss. 2008); Aurand v. Norfolk So. Ry., 802 F.Supp.2d 950 (2011); Mallozzi v. Ecosmart Technologies, Inc., 2013 WL 2415677, No. 11-CV-2884 (SJF)(ARL) (E.D.N.Y. May 31, 2013).

Posted in Conflicts of Interest, Expert Witnesses, Rule 702 | Comments Off on The Capture of the Public Health Community by the Litigation Industry

The Not-So-Elite Defense Bar’s Approach to Gatekeeping

January 2nd, 2014

A couple of months ago, Professor David Bernstein posted to the Volokh Conspiracy, a short piece about some of the missteps and mistakes committed by “elite defense counsel” in litigating expert witness issues.  See David Bernstein, “The Elite Defense Bar and Expert Evidence” (Nov. 6, 2013).  Professor Bernstein makes some interesting points about questionable positions taken by “elite defense counsel” (read: “highly paid, large firm lawyers”).  For instance, according to Bernstein:

1. elite defense lawyers missed the boat early on by arguing that statistical evidence (observational epidemiology) was inadmissible or insufficient to prove general or specific causation;

2. defense counsel missed the significance of the Supreme Court’s opinion in Daubert;

3. defense counsel continued to press for Frye rule in state courts, although the Frye rule had been shown inadequate and unavailing as a rule to control medical causation opinions; and

4. defense bar has grown soft on Rule 702.

Although the charges seem at points overstated, Bernstein has presented an important indictment of the defense bar.  At the very least, the charges deserve a full exploration by a wider audience. Defense lawyers who are self-critical about their practice should certainly be concerned that someone as persistently pro-702 has taken aim at them.

On the first point, many of the early scientific causation battles were fought in tobacco litigation, in which defendants and their counsel were forced to deny and contest the obvious, the causal role for tobacco in carcinogenesis, at all costs.  The tobacco defense bar, however, should not be confused with the defense bar, generally.  Defense lawyers in Bendectin, silicone, and asbestos cases developed arguments against specious use of epidemiologic evidence, as well as sophisticated, affirmative use of epidemiologic evidence to show lack of association.  Even so, we should keep in mind that it often requires a large body of epidemiologic evidence to show “no association,” and it is not the defense’s burden to do so. 

The use of statistical or probabilistic evidence for inferring specific causation has been, and remains, problematic. See, e.g., Richard Doll, “Proof of Causality: Deduction from Epidemiological Observation,” 45 Perspectives in Biology & Medicine 499, 500 (2002) (“That asbestos is a cause of lung cancer in this practical sense is incontrovertible, but we can never say that asbestos was responsible for the production of the disease in a particular patient, as there are many other etiologically significant agents to which the individual may have been exposed, and we can speak only of the extent to which the risk of the disease was increased by the extent of his or her exposure.”) Notwithstanding the controversy, defense counsel and some astute judges have understood the indeterminate nature of statistical evidence for specific causation, and have advanced a pragmatic position of resolving specific causation controversies against plaintiffs when risk ratios failed to exceed two. See, e.g., In re Agent Orange Product Liab. Litig., 597 F. Supp. 740, 785, 836 (E.D.N.Y. 1984) (“A government administrative agency may regulate or prohibit the use of toxic substances through rulemaking, despite a very low probability of any causal relationship.  A court, in contrast, must observe the tort law requirement that a plaintiff establish a probability of more than 50% that the defendant’s action injured him. … This means that at least a two-fold increase in incidence of the disease attributable to Agent Orange exposure is required to permit recovery if epidemiological studies alone are relied upon.”), aff’d 818 F.2d 145, 150-51 (2d Cir. 1987)(approving district court’s analysis), cert. denied sub nom. Pinkney v. Dow Chemical Co., 487 U.S. 1234 (1988).

Early use of meta-analysis by plaintiffs’ expert witnesses elicited generalized attacks on meta-analysis by defense counsel.  See, e.g., In re Paoli Railroad Yard PCB Litigation, 706 F. Supp. 358, 373 (E.D. Pa. 1988) (Kelly, R., J.) (excluding plaintiffs’ expert witness Dr. William Nicholson and his testimony based upon his unpublished meta-analysis of health outcomes among PCB-exposed workers), rev’d 916 F.2d 829, 856-57 (3d Cir. 1990), cert. denied, 499 U.S. 961 (1991).  Again, it took some time for the defense to understand the potency of meta-analysis in synthesizing and presenting a summary point estimate that essentially rules out any meaningful play of chance. In the silicone gel breast implant litigation, one defense expert witness conducted and published his meta-analysis of autoimmune disease outcomes. Otto Wong, “A Critical Assessment of the Relationship between Silicone Breast Implants and Connective Tissue Diseases,” 23 Regulatory Toxicol. & Pharmacol. 74 (1996).  When the MDL 926 court got around to appointing court-appointed expert witnesses, they too picked up on the approach and made meta-analyses the hallmark of their reports. See Barbara Hulka, Betty Diamond, Nancy Kerkvliet & Peter Tugwell, “Silicone Breast Implants in Relation to Connective Tissue Diseases and Immunologic Dysfunction:  A Report by a National Science Panel to the Hon. Sam Pointer Jr., MDL 926 (Nov. 30, 1998)”; Barbara Hulka, Nancy Kerkvliet & Peter Tugwell, “Experience of a Scientific Panel Formed to Advise the Federal Judiciary on Silicone Breast Implants,” 342 New Engl. J. Med. 812 (2000).

On the second charge, many defense lawyers missed, and continue to miss, the significance of a shift to evidence-based scientific testimony, as opposed to the authority-based worldview.  When I first started trying cases, senior trial lawyers instructed me not to engage plaintiffs’ expert witnesses on substantive issues, but to limit cross-examination to “collateral attack” on bias and related issues.  The problem that I saw was that when both sides limited the attack to the other side’s expert witness’s bias, the plaintiffs won because juries were often all too willing to think the worst of defense experts, and forgive plaintiffs’ experts.  Nothing short of dramatically confronting the jury with the rubbish, inconsistencies, and incoherence spouted by the plaintiffs’ expert witnesses worked.  I quickly learned to ignore the old timers’ advice, and most of them have now dropped off.

Bernstein gives the example of the Dow Corning lawyers’ declining the Ninth Circuit’s invitation to reframe their appeal in Hopkins v. Dow Corning, 33 F.3d 1116 (9th Cir. 1994), after the Supreme Court handed down Daubert.  He may well be right about that case, but the lawyers may have been inhibited by positions that they had taken earlier in the case, before the Supreme Court breathed life into Rule 702.  Still, the Hopkins decision remains a derelict on the jurisprudential sea of expert witness law. Truth be told, there are other cases that turned out badly because of overstated or poorly framed defense arguments. 

As for their persistent affection for Frye, the defense lawyers referenced by Bernstein certainly lacked imagination, and maybe even a full measure of zeal.  The Frye case had never proven itself to be an important defense against specious expert witness opinion testimony on medical causation issues.  Daubert held out the promise that trial judges would actually have to engage with the evidence, rather than counting noses for “general acceptance,” or kicking the can, after hearing qualifications. I still recall how my colleagues and friends viewed Daubert, in 1993.  Bristol Myers Squibb and other companies were inundated by silicone breast implant cases, and plaintiffs had managed to snucker a few juries into returning large verdicts against the defendants.  Daubert was a lifeline, a way to focus trial judges’ attention on the plaintiffs’ expert witnesses and the fatally flawed, even fraudulent scientific studies that had found their way into the peer-reviewed journals, and into the courtroom.

What has happened to Daubert?  Well, of course, it is no longer Daubert, but now a much more potent statutory rule, Federal Rule of Evidence 702, as amended in 2000, and recently “restylized.”  Professor Bernstein may be on to something in calling out the hubris of big firm lawyers, who think that their prowess in  litigating scientific issues comes from charging high fees.  The sad truth is that the level of scientific and statistical acumen of lawyers generally, whether at the bench or at the bar, is low.  There is much work for law schools to do to change this situation. 

There are other forces at work in creating the phenomena described by Professor Bernstein.  Many product liability cases involve multiple defendants.  Defending such cases from the defense perspective is often like herding house cats.  Every defense counsel thinks he or she knows the best course; no one wants to do anything that raises the profile of his client or increases the cost of defense. Furthermore, products liability defense work has become increasingly “commoditized” in the last two decades.  Clients and third-party payors increasingly impose budgets that do not allow defense lawyers sufficient time to develop the necessary expertise and learning to discover, understand, and challenge plaintiffs’ scientific expert witnesses fully and effectively. There is, sadly and unfortunately, a contingent of defense lawyers who would rather see speculative claims flourish and mature into full-scale litigation, with lots of trials and appeals (and large fees), than to see specious claiming pretermitted by pre-trial judicial gatekeeping.

Another factor that may have stunted the gatekeeping process is the growth of federal (and state) multi-district litigation (MDL).  By centralizing the pre-trial handling of large, multi-plaintiff tort litigation in the hands of a single judge, the MDL process has raised the stakes for the trial judge gatekeeping.  The MDL judge is no longer deciding a single case, but the fate of thousands of claimants.  The Rule 702 process suffers in several ways.  First, many MDL judges are clearly intimidated by the life-or-death control they have over hundreds, if not thousands, of claims.  These judges may well now be inclined to deny the Rule 702 challenges, in the fond hope that juries and appellate courts will take care of the problem. Second, there is no longer the incremental decision making of case-by-case exploration of the issues.  There are economies of scale, to be sure, but the single MDL judge cannot learn from previous trial judges’ decisions and records.  The evolutionary approach of the common-law is undermined by the efficiencies of MDL procedure. Third, the MDL process blunts one of the virtues of judicial gatekeeping in requiring the transparent statement of reasons for the grant or denial of a Rule 702 motion to exclude expert evidence.  By the time the scientific, legal academic, and legal practitioner communities can weigh in on the gatekeeping process, the Rule 702 decision is a fait accompli, in the form of a single judicial decision. Compare  In re Avandia Marketing, Sales Practices and Product Liability Litigation, 2011 WL 13576, *12 (E.D. Pa. 2011) (Rufe, J.) (denying defendant’s Rule 702 motions with respect to plaintiffs’ expert witnesses’ opinions that Avandia causes heart attacks) with “FDA Drug Safety Communication: FDA requires removal of some prescribing and dispensing restrictions for rosiglitazone-containing diabetes medicines” (Nov. 25, 2013) (re-evaluating evidence that Avandia does not increase risk of heart attacks).  See also “Learning to Embrace Flawed Evidence – The Avandia MDL’s Daubert Opinion” (Jan. 10, 2011).

Not surprisingly, plaintiffs’ counsel have gotten better at dressing up weak and fatally flawed evidence as “inference to the best explanation,” or “weight of the evidence” evaluations that have the appearance without the reality of scientific conclusions. And as evidenced by the Third Edition of the Reference Manual on Scientific Evidence, and many recent 702 decisions in the Circuits (and Matrixx Initiatives in the Supreme Court), the federal judiciary has lost its will and its way in applying Rule 702. An activist core of federal trial and appellate courts have shown increasing hostility and outright disregard for the gatekeeping process, and a willingness to disregard the language of the statute, Supreme Court precedent, and sound scientific and statistical methodology.  Rule 702 in its present form is a strong rule, but unfortunately it is neutralized in the hands of weak judges who cannot or will not explore the intricate methodological issues raised by Rule 702 and 703 motions.

The phenomenon identified by Professor Bernstein may be real, but like many natural (and unnatural) phenomena, the causes are not always easily discerned. The phenomenon and its causes are worth a deeper exploration.

Posted in Expert Witnesses, Rule 702 | Comments Off on The Not-So-Elite Defense Bar’s Approach to Gatekeeping

Gastwirth on the Wells Case

December 27th, 2013

One of the high points of 2013 for me was the Practicing Law Institute seminar on expert witness gatekeeping.  I am indebted to my co-chair David Cohen, and the talented participants, Dr. David Garabrant, Joe Cecil, Mary Wells, and Mike Williams.  I learned a great deal from all of them, even when we disagreed.

At the seminar, which addressed the slippage in the federal judiciary’s fidelity to Rule 702 text and principles, Joe Cecil kindly reminded me of Professor Gastwirth’s contribution on the Wells v. Ortho Pharmaceutical Corp., 615 F. Supp. 262, 298 (N.D. Ga. 1985), aff’d and rev’d in part on other grounds, 788 F.2d 741 (11th Cir.), cert. denied, 479 U.S.950 (1986).  Joseph L. Gastwirth, “The need for careful evaluation of epidemiological evidence in product liablility cases: a reexamination of Wells v. Ortho and Key Pharmaceuticals,” 2 Law, Probability and Risk 151 (2003).

The Supreme Court recently reinvigorated the Wells case, and like Frankenstein’s monster, it walks again.  Matrixx Initiatives, Inc. v. Siracusano, 131 S. Ct. 1309 (2011).  See also “Matrixx Unloaded.”  As a result of the Matrixx decision, I revisited the untoward Wells decision several times this year. See “Wells v. Ortho Pharmaceutical Corp. Reconsidered – Part 1”; and here, here, here, here, and here.  Joe’s reminder, however, provoked me to look one more time at Wells, through the lens of Gastwirth’s review.

Gastwirth is a well-known statistician, not a lawyer.  His article undertakes some interesting sensitivity analyses of the potential role of one known confounder in the epidemiologic studies of spermicides and birth defects, cited by witnesses in Wells. Not surprisingly, Gastwirth confuses important legal concepts, when he contends that science and law have different goals because science has the luxury of waiting for additional studies to resolve indeterminate datasets.  Gastwirth at 154. Gastwirth suggests that in law, causality determinations turn on fairness, but in science, they turn on the truth.  Gastwirth offers no support for his assertion, and there is much law to the contrary.  Federal Rule of Evidence 102, for instance, explains the “Purpose” of the Federal Rules and how they should be construed:

“These rules should be construed so as to administer every proceeding fairly, eliminate unjustifiable expense and delay, and promote the development of evidence law, to the end of ascertaining the truth and securing a just determination.”

Truth does count, at least in some courts, and it is hard to imagine a case in which the goals of truth and justice are bifurcated. Gastwirth sees law courts as unable to avoid judgment when the evidentiary display is weakly developed or not replicated, but he is wrong.  Courts can, do, and should avoid permitting judgments based upon indeterminate datasets.

Gastwirth also contends that the duty to warn in tort is based upon a possibility of causation, and he opines that this is perfectly satisfactory.  Gastwirth at 155.  Gastwirth cites the Restatement (Third) a/Torts for his view, but the Restatement does not couch the duty to warn as predicated upon merely possible risk:

“A defendant will not be liable under an implied warranty of merchantability for failure to warn about risks that were not reasonably foreseeable at the time of sale or could not have been discovered by way of reasonable testing prior to marketing the product. A manufacturer will be held to the standard of knowledge of an expert in the appropriate field, and will remain subject to a continuing duty to warn of risks discovered following the sale of the product at issue.”

Id. (emphasis added).  Gastwirth fails to explain why the epistemic standard of the Restatement should be diluted to include merely possible risks; nor does he explain why the law should impose a duty because a minority of so-called experts claim knowledge of the existence of a risk, whether potential or known.

Risks are based upon known causal relationships, viewed ex ante rather than post hoc.  Gastwirth attempts to defend the Wells decision on the basis that the trial judge, Judge Shoob, could have correctly found a possible risk, but this was not then or now the standard for imposing a duty to warn.  Even if possibilities were the basis for a duty to warn, the courts in Wells had to find sufficient evidence of cause in fact between the spermicidal jelly and the birth defect present in the infant plaintiff.  No merely possible risk could have satisfied this burden, and Gastwirth avoids this important issue in his review.  Ultimately, Gastwirth fails to come to grips with how the Wells courts artificially distinguished two causation standards – warning-harm and product-harm.  Gastwirth at 184.

When courts are required to work with an epistemic standard for evaluating opinions about causal claims, they will often confront sincere beliefs supported by “putative” justifications, but which turn out to be untrue.  The obvious implication is that the causal opinions were not “true” when offered, and the asserted, putative justification was no justification at all.  Twenty years post-Daubert should have made our courts more sensitive to the pathology of claiming, or pathoepistemology, as I call it.

Although Gastwirth is intent to show that the district and circuit court decisions in Wells were reasonable given the record evidence at the time of their decisions, Gastwirth’s defense is tepid at best. Gastwirth concedes that the plaintiffs’ evidence in the Wells case was not very convincing, and he is quick to point out that he does not conclude that spermicides cause limb reduction defects (although he does believe that the data were sufficient for his idiosyncratic diluted duty-to-warn legal standard:

 “The epidemiological evidence in the Wells case surely was not very convincing and it is understandable that scientists and legal scholars might desire stronger scientific evidence before holding a defendant liable. What seems unfortunate is the lack of discussion of the part of the record most supportive of the legal decisions from the published criticisms.201 While it is reasonable to ask how much evidence should be required before a warning is required, is it fair to rely on studies that were published after the time of exposure to assess the producer’s duty at the time?”

Gastwirth at 184-85.  The equivocation in this passage between assessing causation for a duty to warn and for product harm is typical of Gastwirth’s confusion throughout this article.  As noted, Gastwirth appears to believe it is sufficient to show the plausibility of the causal relationship such that the trial court’s findings of a duty to warn, and a breach of that duty, were reasonably supported:

“It should be stressed that our examination of the studies does not conclude that exposure to spermicides causes limb reduction defects. 202 Indeed, the studies published in the 1980s are sensitive to the potential effect of unmeasured known potential risk factors that were not controlled for in the analysis. Many studies observed that it is difficult to rule out small but meaningful risks given their sample size or that more studies including potential confounding variables be carried out before a scientific conclusion can be reached.203”

Gastwirth at 185 & n.202.  So small increased risks were difficult to rule in or rule out, but then how can a judgment for plaintiffs be sustained unless, as I have shown elsewhere, Judge Shoob improperly placed the burden of proof upon the defendant?  The studies published in the 1980s may have been sensitive to the potential effect of unmeasured and uncontrolled known potential risks, but Judge Shoob clearly was not sensitive to the problem at all.

Other problems abound. The studies involved in the Wells case not only failed to account for the few known confounders at the time, but they could not and did not sort out the residual confounding.  Gastwirth concedes the point in a footnote, in which he cites a paper by Professor Gary Shaw on the role of multivitamin use in preventing birth defects. See Gastwirth at 185 & n. 202 (citing Shaw et al., “Maternal Periconceptional Use of Multivitamins and Reduced Risk for Conotruncal Heart Defects and Limb Deficiencies Among Offspring,” 59 Am. J. Med. Genetics 536 (1995) (showing that children of mothers who had taken vitamins and folic acid had a 30-35% lower risk of limb defects).  Gastwirth sees the discovery of causes not previously controlled for in the spermicide/birth defect studies of the 1980s as example of the differences between scientific and legal judgments.  Another way to look at such discoveries is to impose some standard of intellectual modesty upon courts in basing their findings upon speculative causal claims.

Gastwirth’s casual dismissiveness of the need for replication in a legal standard of causation is difficult to square with his acknowledgment that early studies often prove misleading about the existence of a true risk.  Gastwirth, for instance, acknowledges that later publications quite commonly do not support initial hypotheses.  Gastwirth at 179, 184 (citing Michael B. Bracken, “Spermicidal Contraceptives and Poor Reproductive Outcomes: The Epidemiologic Evidence Against an Association,” 151 Am. J. Obstet. & Gyncecol. 552, 555 (1985)).  Gastwirth channels the anti-manufacturer Zeitgeist for his claim that “firms” may not report studies that show risk, but he ignores the pervasive, prevalent practice of academic and advocacy scientists’ not publishing negative studies in the first place, and even later in the natural history of scientific controversies.

Ultimately, however, Gastwirth’s own judgment is a self-fulfilling prophecy of how the law and science should part company over false-positive assertions of causality.  Fortunately, the law has evolved in a different direction, towards convergence of scientific and legal modes of assessing causal claims.  Now judges must evolve as well.

Posted in Expert Witnesses, Rule 702, Scientific Evidence | Comments Off on Gastwirth on the Wells Case

How Law Professors Think — About Expert Evidence

December 19th, 2013

In a recent law review article, two University of Virginia law professors question whether expert evidence should be subject to its own exclusionary rules.  Frederick Schauer and Barbara A. Spellman, “Is Expert Evidence Really Different?” 89 Notre Dame L. Rev. 1 (2013)[Schauer & Spellman]. Professors Schauer and Spellman argue that expert evidence is not really different from other kinds of evidence, and they suggest that the exclusionary procedures introduced by Daubert and its progeny are ill conceived.

Gedankenexperiment

In order to understand the exact nature of the harms of “junk science,” the authors conduct an interesting Gedanken experiment:

“Suppose ten witnesses testify that they had never been sick a day in their lives, that they then moved in middle age to a community in close proximity to a defendant’s chemical plant, and that they were all diagnosed with the same form of cancer within a year. And suppose that this is the only evidence of causation.”

The authors conclude that this evidence is relevant under Federal Rule of Evidence 401, and sufficient to raise a triable issue of fact.  From their conclusion, the authors argue further all the dangers of mass tort causation evidence are independent of junk science because a jury would be free to reach a verdict for plaintiffs based upon pure sympathy, anti-corporate animus, white-hat bias, or Robin Hood motives.  The authors see, in their hypothetical, a jury reaching a judgment against the defendant

“regardless of any strong evidence of causation, and without any junk science whatsoever.”

Schauer and Spellman’s conclusions, however, are wrong.  Their hypothetical evidentiary display is not even minimally logically relevant.  They are correct that there is no strong evidence of causation, but whence comes the conclusion that no “junk science” would be involved in the jury’s determination?  That determination would not have even the second-hand support of an expert witness opinion, but it would be jury’s first-hand, jejune interpretation of a completely indeterminate fact pattern.

These authors, after all, do not specify what kind of cancer is involved. Virtually no cancer has an induction period of less than a year.  Their hypothetical does not specify what chemicals are “released,” in what route of exposure, in what level, and for what duration. Furthermore, the suggestion of logical relevance implies that the described occurrence is beyond what we would care to ascribe to chance alone, but we do not know the number of people involved, or the baseline risk of the cancer at issue.  One in a million happens eight times a day, in New York City. Flipping a coin ten times, and observing 6 heads and 4 tails, would not support an inference that the best evidence we have is that the coin will always favor heads to tails in a ratio of 1.5.

Schauer and Spellman might improve their hypothetical, but they are unlikely to free themselves of the need for expertise beyond the ken of a lay jury to evaluate the clinical, epidemiologic, scientific, and statistical issues raised by their supposed disease “outbreak.”  And although they have taken the expert witness as the usual purveyor of junk science out of the hypothetical by design, they have simply left the jury to draw a bogus inference.  The junk is still at work, only the jury is no longer a passive recipient of the inference; they are themselves the authors of the junk-science inference.

Schauer and Spellman’s Claim That Expert Evidence Is Not Really Different

The authors make the case that there are some instances in which expert witness opinion testimony is not so different from lay opinions about intoxication or speed of a car or eyewitness identification.  But Schauer and Spellman are wrong to allow themselves to be fooled about expert witness testimony in many complex mass tort cases.

Such cases commonly involve several scientific disciplines, such as toxicology, epidemiology, exposure assessment, neuropsychology, and others. The expert witness for each discipline might have a few dozen studies that are germane to the issues in the case, and each one of those studies might cite or rely upon several papers for their background, methods  and inferences.  Reference lists for each expert witness might run into the hundreds of articles, and in some cases, the experts might need to access underlying data and materials to understand fully the papers upon which they have relied.  A careful reading of each paper might require an hour or more for the expert to understand the claims and limitations of the study.  The expertise to understand the articles fully may have taken years or decades of education.

Juries do not have the time, the interest, the aptitude, the training, the experience, to read, understand, and synthesize the data and information in the studies themselves.  Our trials involving complex technical issues are much like Plato’s allegory of the cave; the jury never sees the actual evidence, only shadows cast by evidence they are usually not permitted to see, and don’t care to see when they have the chance. Juries decide technical issues based upon mostly the appearance of expertise, not upon evidence.

Some years ago, I tried an asbestos case against Charles “Joey” Grant, in front of Judge Armand Della Porter and a jury in Philadelphia Court of Common Pleas.  Joey represented a man who claimed that he had asbestosis from working at the Philadelphia Naval Shipyard. His chest X-ray interpretation was disputed, but he has small-airways obstruction, which his expert witness attributed to asbestosis.  The defense expert thought smoking was a much more likely cause of the obstruction, but the plaintiff had denied smoking in his deposition.  In order to test his assertion, the defense asked a private investigator to conduct surveillance of the plaintiff to determine whether or not he was a smoker.

The investigator, retired Alcohol Tobacco and Firearms agent Frank Buenas, tailed the plaintiff and observed and photographed him smoking.  Plaintiff’s counsel, Joey Grant, seized on my not having provided Buenas an authenticated photograph of the plaintiff, and challenged the identification and every aspect of the surveillance.  The direct examination lasted no more than 25 minutes; the cross-examination lasted about four hours.

Joey was a very good trial lawyer.  He had just come out of the Philadelphia District Attorney’s office, after having successfully prosecuted Nicodemo “Little Nicky” Scarfo.  Joey was also a good looking African American man who played well to our all female, all African American jury.  The issues of the surveillance, and of whether or not he plaintiff was a smoker, were understandable and accessible to the jurors, who were riveted by Joey’s cross-examination. Ultimately, the issues were resolved for the jury in a dramatic fashion.  The plaintiff, who continued to work at the Navy Yard, returned to court at the end of his shift, towards the end of the day in court.  Over objection, I called him back to the stand. He had not heard the investigator’s testimony, but when I showed him Buenas’ photographs, he exclaimed “that’s my bald head!”  The jurors practically lunged out of their seats when I published the photographs to the jury over Joey’s objection.

The point of the war story is to recount how the jury followed a protracted examination, and ignored their bias in favor of Joey, and their prejudice against me, the white guy representing the “asbestos companies” in this reverse-bifurcated trial.  The testimony involved a predicate issue whether Buenas had followed and photographed the right man in the act of smoking.  Would a jury, any jury, follow the testimony of a scientist who was being challenged on methodological details of a single study, for four hours?  Would any jury remain attentive and engaged in the testimony of expert witnesses who testified on direct and cross-examination in similar detail about hundreds of studies?

Having tried cases that involve both the simple, straightforward issue such as Buenas’ investigation and surveillance, and also complex scientific issues, I believe the answer is obvious.  None of the studies cited by Schauer and Spellman address the issue of complexity and how it is represented in the courtroom. 

Most judges, when acting as the trier of fact, lack the interest, time, training, and competence to try complex cases. Consider the trial judge who conducted bench trial of a claim of autoimmune disease caused by silicone.  The trial judge was completely unable to assess the complex immunological, toxicological, and epidemiologic evidence in the case.  See Barrow v. Bristol Myers Squibb Co., No. 96 689 CIV ORL 19B, 1998 WL 812318 (M.D. Fla. Oct. 29, 1998).  In another courtroom, not far away, Judge Sam Pointer had appointed four distinguished expert witnesses under Rule 706. In re Silicone Gel Breast Implant Prods. Liab. Litig. (MDL 926), 793 F. Supp. 1098 (J.P.M.L. 1992) (No. CV92-P-10000-S), Order No. 31, 31B, 31E.  In November 1998, a month after the Middle District of Florida trial court decided the Barrow case, the four court-appointed expert witnesses filed their joint and individual reports that concluded that silicone in breast implants “do[es] not alter incidence or severity of autoimmune disease” and that women who have had such implants  “do[] not display a silicone-induced systemic abnormality in the . . . cells of the immune system.” National Science Panel, Silicone Breast Implants in Relation to Connective Tissue Diseases and Immunologic Dysfunction, Executive Summary at 5-6  (Nov. 17, 1998).  The Panel also found that “[n]o association was evident between breast implants and any of the . . . connective tissue diseases . . . or the other autoimmune/rheumatic conditions” claimed by the thousands of women who had filed lawsuits. Id. at 6-7. 

Another case, in the causal chain that produced the Daubert decision, might also have cautioned Professors Schauer and Spellman against oversimplifying the distinctions between expert and other evidence.  Wells v. Ortho Pharmaceutical Corp., 615 F. Supp. 262 (N.D. Ga. 1985), aff’d and rev’d in part on other grounds, 788 F.2d 741 (11th Cir.), cert. denied, 479 U.S.950 (1986).  The Wells case was a bench trial, in which the trial judge honestly professed not to have understood the epidemiologic evidence, and to have decided the case on his assessment of expert witness demeanor and atmospherics.  And like the Barrow case, the Wells case was clearly wrongly decided. See “Wells v. Ortho Pharmaceutical Corp. Reconsidered – Part 1” (Nov. 12th, 2012) (footnote 1 collects a sample of citations criticizing Judge Shoob’s failure to engage with the scientific evidence).

Jury determinations, unlike the clearly erroneous determinations in Barrow and Wells, are “black boxes.”  The public and the scientific community cannot really criticize jury decisions because the jury does not explain their inferences from evidence .  The lack of explanation, however, does not save them from engaging in junk science.  Outside the Gedanken experiment above, jurors can blame expert witnesses for their errors.  In the experiment’s hypothetical, junk science is still taking place.

The Discomfort With Daubert

The authors recount many of the charges against and criticisms of the Daubert decision.  Schauer & Spellman at 2. They note that some commentators assert that the Justices in Daubert embraced a “clumsy philosophy of science.” But at least Justice Blackmun engaged with the philosophy of science, and the epistemic requirements of Rule 702, and made some attempt to reconcile the truth-finding process in court with what happens in science.  The attempted reconciliation was long overdue.

The authors also point out that some commentators have expressed concern that Daubert burdens mass tort and employment discriminations who “non-traditional experts and expertise.” Id. To paraphrase Tim Minchin, there is non-traditional opinion known not to be true, and non-traditional opinion not known to be true, but if non-traditional opinion is known to be true, then we call it … knowledge.  Perhaps Schauer and Spellman think that our courts should be more open and inclusive, warmer and fuzzier, for clinical ecologists, naturopaths, aromatherapists, homeopaths, and other epistemiopaths.  My sense is that these non-traditional experts should be relegated to live in their own world of magical thinking.

The authors give voice to “the broad worry that law should not outsource its own irreducibly legal determinations to science and scientists with different goals and consequently different standards.” But science and the law are both engaged in truth determinations.  Maybe the law should worry more about having jurors and judges making their determinations with different goals and standards.  Maybe the law should be more concerned with truth and scientific accuracy, and should not outsource its “irreducibly fact determinations” to the epistemiopaths.

Expert witness testimony is clearly different in many important respects from lay witness testimony and other evidence.  The expert witness testimony is largely opinion.  It relies upon many layers of hearsay, with many of the layers not subject to ready scrutiny and testing for veracity in a courtroom.  Many layers of the “onion” represent evidence that would not be admissible under any likely evidentiary scenario in a courtroom.

And jurors are reduced to using proxies for assessing the truth of claims and defenses.  Decisions in a courtroom are often made based upon witness demeanor, style, presentation, qualifications, and appearance of confidence and expertise. The authors lament the lack of empirical support for the last point, but this misses the larger point that the issue is not primarily an empirical one, and the point is not limited to jury competence.  Hon. Jed S. Rakoff, “Lecture: Are federal judges competent? Dilettantes in an age of economic expertise,” 17 Fordham J. Corp. & Fin. L. 4 (2012).

The Monte Hall Problem and Cognitive Biases

The “Monty Hall problem” was originally framed by statistician Steve Selvin in publications in the American Statistician in 1975 (Selvin 1975a), (Selvin 1975b).  The problem was then popularized by an exposition in Parade magazine in 1990, by Marilyn vos Savant.  The problem, based upon Monty Hall’s television game show Let’s Make a Deal, and Monty’s practice of asking contestants whether they wanted to switch.  A full description of the problem can be found elsewhere.

For our purposes here, the interesting point is that the correct answer was not intuitively obvious.  After vos Savant published the answer, many readers wrote indignant letters, claiming she was wrong.  Some of these letters were written by college and university professors of mathematics.  Vos Savant’s popularization of Selvin’s puzzle illustrates that there are cognitive biases, flaws in reasoning, and paradoxes, the avoidance of which requires some measure of intelligence and specialty training in probability and statistics.  See also Amos Tversky and Daniel Kahneman, “Judgment under Uncertainty: Heuristics and Biases,” 185 Science 1124 (1974).

Our system of justice suffers enough from the current regime of trial by sound-bite.  It should not be further undermined by the abandonment of judicial gatekeeping and judicial review of the quality and quantity of scientific evidence in our civil and criminal trials. Opponents of gatekeeping imply that the Federal Rules of Evidence should be “liberally” construed, by which they usually mean, construed consistent to their biases and prejudices.  The adjective “liberal,” both traditionally and today, however, connotes enlightened, free from superstition, bias, prejudice, and promoting accuracy in judgments that are important to institutional respect and prestige.  Let’s hope that 2014 is a better year for science in the courtroom.

Posted in Rule 702 | Comments Off on How Law Professors Think — About Expert Evidence

Supreme Court Denies Certiorari in Harkonen v. United States

December 16th, 2013

The Supreme Court took up Dr. Harkonen’s petition for certiorari on Friday, December 13, 2013.  The SCOTUS Blog made the petition its “Petition of the Day.”

Unfortunately, the blog’s attention did not carry over to the Court’s conference. The Court release its orders this morning, which included a denial of certiorari in Harkonen v. United States.

Although there was a good deal of noise about Dr. Harkonen’s intent, that issue is uniquely case specific.  The real issue going forward would seem to be the reach of the government’s position, and now the Ninth Circuit’s position, that failure to disclose multiple testing or deviation from a protocol is demonstrably false or misleading.  In Dr. Harkonen’s case, the causal inference was reasonably supported despite the non-protocol analysis.  The use of the verb “demonstrate,” however, is often used carelessly, and the Harkonen decision may well breathe life into Rule 702 gatekeeping.

 

Posted in Rule 702, Scientific Misconduct | Comments Off on Supreme Court Denies Certiorari in Harkonen v. United States

« Older Entries
Newer Entries »