There is much to digest in the new Reference Manual on Scientific Evidence, third edition (RMSE 3d).  Much of what is covered is solid information on the individual scientific and technical disciplines covered.  Although the information is easily available from other sources, there is some value in collecting the material in a single volume for the convenience of judges.  Of course, given that this information is provided to judges from an ostensibly neutral, credible source, lawyers will naturally focus on what is doubtful or controversial in the RMSE.

I have already noted some preliminary concerns, however, with some of the comments in the Preface, by Judge Kessler and Dr. Kassirer.  See “New Reference Manual’s Uneven Treatment of Conflicts of Interest.”  In addition, there is a good deal of overlap among the chapters on statistics, epidemiology, and medical testimony.  This overlap is at first blush troubling because the RMSE has the potential to confuse and obscure issues by having multiple authors address them inconsistently.  This is an area where reviewers should pay close attention.

From first looks at the RMSE 3d, there is a good deal of equivocation between encouraging judges to look at scientific validity, and discouraging them from any meaningful analysis by emphasizing inaccurate proxies for validity, such as conflicts of interest.  (As I have pointed out, the new RSME did not do quite so well in addressing its own conflicts of interest.  See “Toxicology for Judges – The New Reference Manual on Scientific Evidence (2011).”)

The strengths of the chapter on statistical evidence, updated from the second edition, remain, as do some of the strengths and flaws of the chapter on epidemiology.  I hope to write more about each of these important chapters at a later date.

The late Professor Margaret Berger has an updated version of her chapter from the second edition, “The Admissibility of Expert Testimony,” RSME 3d 11 (2011).  Berger’s chapter has a section criticizing “atomization,” a process she describes pejoratively as a “slicing-and-dicing” approach.  Id. at 19.  Drawing on the publications of Daubert-critic Susan Haack, Berger rejects the notion that courts should examine the reliability of each study independently. Id. at 20 & n. 51 (citing Susan Haack, “An Epistemologist in the Bramble-Bush: At the Supreme Court with Mr. Joiner,” 26 J. Health Pol. Pol’y & L. 217–37 (1999).  Berger contends that the “proper” scientific method, as evidenced by works of the International Agency for Research on Cancer, the Institute of Medicine, the National Institute of Health, the National Research Council, and the National Institute for Environmental Health Sciences, “is to consider all the relevant available scientific evidence, taken as a whole, to determine which conclusion or hypothesis regarding a causal claim is best supported by the body of evidence.” Id. at 19-20 & n.52.  This contention, however, is profoundly misleading.  Of course, scientists undertaking a systematic review should identify all the relevant studies, but some of the “relevant” studies may well be insufficiently reliable (because of internal or external validity issues) to answer the research question at hand. All the cited agencies, and other research organizations and researchers, exclude studies that are fundamentally flawed, whether as a result of bias, confounding, erroneous data analyses, or related problems.  Berger cites no support for the remarkable suggestion that scientists do not make “reliability” judgments about available studies when assessing the “totality of the evidence.”

Professor Berger, who had a distinguished career as a law professor and evidence scholar, died in November 2010.  She was no friend of Daubert, but remarkably her antipathy has outlived her.  Her critical discussion of “atomization” cites the notorious decision in Milward v. Acuity Specialty Products Group, Inc., 639 F.3d 11, 26 (1st Cir. 2011), which was decided four months after her passing. Id. at 20 n.51. (The editors note that the published chapter was Berger’s last revision, with “a few edits to respond to suggestions by reviewers.”)

Professor Berger’s contention about the need to avoid assessments of individual studies in favor of the whole gamish must also be rejected because Federal Rule of Evidence 703 requires that each study considered by an expert witness “qualify” for reasonable reliance by virtue of the study’s containing facts or data that are “of a type reasonably relied upon by experts in the particular field forming opinions or inferences upon the subject.”  One of the deeply troubling aspects of the Milward decision is that it reversed the trial court’s sensible decision to exclude a toxicologist, Dr. Martyn Smith, who outran his headlights on issues having to do with a field in which he was clearly inexperienced – epidemiology.

Scientific studies, and especially epidemiologic studies, involve multiple levels of hearsay.  A typical epidemiologic study may contain hearsay leaps from patient to clinician, to laboratory technicians, to specialists interpreting test results, back to the clinician for a diagnosis, to a nosologist for disease coding, to a national or hospital database, to a researcher querying the database, to a statistician analyzing the data, to a manuscript that details data, analyses, and results, to editors and peer reviewers, back to study authors, and on to publication.  Those leaps do not mean that the final results are untrustworthy, only that the study itself is not likely admissible in evidence.

The inadmissibility of scientific studies is not problematic because Rule 703 permits testifying expert witnesses to formulate opinions based upon facts and data, which are not themselves admissible in evidence. The distinction between relied upon, and admissible, studies is codified in the Federal Rules of Evidence, and in virtually every state’s evidence law.

Referring to studies, without qualification, as admissible in themselves is wrong as a matter of evidence law.  The error has the potential to encourage carelessness in gatekeeping expert witnesses’ opinions for their reliance upon inadmissible studies.  The error is doubly wrong if this approach to expert witness gatekeeping is taken as license to permit expert witnesses to rely upon any marginally relevant study of their choosing.  It is therefore disconcerting that the new Reference Manual on Science Evidence (RMSE 3d) fails to make the appropriate distinction between admissibility of studies and admissibility of expert witness opinion that has reasonably relied upon appropriate studies.

Consider the following statement from the chapter on epidemiology:

“An epidemiologic study that is sufficiently rigorous to justify a conclusion that it is scientifically valid should be admissible,184 as it tends to make an issue in dispute more or less likely.¹⁸⁵“

RMSE 3d at 610.  Curiously, the authors of this chapter have ignored Professor Berger’s caution against slicing and dicing, and speak to a single study’s ability to justify a conclusion. The authors of the epidemiology chapter seem to be stressing that scientifically valid studies should be admissible.  The footnote emphasizes the point:

“See DeLuca v. Merrell Dow Pharms., Inc., 911 F.2d 941, 958 (3d Cir. 1990); cf. Kehm v. Procter & Gamble Co., 580 F. Supp. 890, 902 (N.D. Iowa 1982) (“These [epidemiologic] studies were highly probative on the issue of causation—they all concluded that an association between tampon use and menstrually related TSS [toxic shock syndrome] cases exists.”), aff’d, 724 F.2d 613 (8th Cir. 1984). Hearsay concerns may limit the independent admissibility of the study, but the study could be relied on by an expert in forming an opinion and may be admissible pursuant to Fed. R. Evid. 703 as part of the underlying facts or data relied on by the expert. In Ellis v. International Playtex, Inc., 745 F.2d 292, 303 (4th Cir. 1984), the court concluded that certain epidemiologic studies were admissible despite criticism of the methodology used in the studies. The court held that the claims of bias went to the studies’ weight rather than their admissibility. Cf. Christophersen v. Allied-Signal Corp., 939 F.2d 1106, 1109 (5th Cir. 1991) (“As a general rule, questions relating to the bases and sources of an expert’s opinion affect the weight to be assigned that opinion rather than its admissibility. . . .”).”

RMSE 3d at 610 n.184 (emphasis in bold, added).  This statement, that studies relied upon by an expert in forming an opinion may be admissible pursuant to Rule 703, is unsupported by Rule 703 and the overwhelming weight of case law interpreting and applying the rule.  (Interestingly, the authors of this chapter seem to abandon their suggestion that studies relied upon “might qualify for the learned treatise exception to the hearsay rule, Fed. R. Evid. 803(18), or possibly the catchall exceptions, Fed. R. Evid. 803(24) & 804(5),” which was part of their argument in the Second Edition of the RMSE.  RMSE 2d at 335 (2000).)  See also RMSE 3d at 214 (discussing statistical studies as generally “admissible,” but acknowledging that admissibility may be no more than permission to explain the basis for an expert’s opinion).

The cases cited by the epidemiology chapter, Kehm and Ellis, both involved “factual findings” in public investigative or evaluative reports, which were independently admissible under Federal Rule of Evidence 803(8)(C).  See Ellis, 745 F.2d at 299-303; Kehm, 724 F.2d at 617-18.  As such, the cases hardly support the chapter’s suggestion that Rule 703 is a rule of admissibility for epidemiologic studies.

Here the RMSE, in one sentence, confuses Rule 703 with an exception to the rule against hearsay, which would prevent the statistical studies from being received in evidence.  The point is reasonably clear, however, that the studies “may be offered” to explain an expert witness’s opinion.  Under Rule 705, that offer may also be refused. The offer, however, is to “explain,” not to have the studies admitted in evidence.

The RMSE is certainly not alone in advancing this notion that studies are themselves admissible.  Other well-respected evidence scholars lapse into this position:

“Well conducted studies are uniformly admitted.”

David L. Faigman, et al., Modern Scientific Evidence:  The Law and Science of Expert Testimony v.1, § 23:1,at 206 (2009)

Evidence scholars should not conflate admissibility of the epidemiologic (or other) studies with the ability of an expert witness to advert to a study to explain his or her opinion.  The testifying expert witness really has no need to become a conduit for off-hand comments and opinions in the introduction or discussion section of relied upon articles, and the wholesale admission of such hearsay opinions undermines the court’s control over opinion evidence.  Rule 703 authorizes reasonable reliance upon “facts and data,” not every opinion that creeps into the published literature.

The new, third edition of the Reference Manual on Scientific Evidence (RMSE) appears to get off to a good start in the Preface by Judge Kessler and Dr. Kassirer, when they note that the Supreme Court mandated federal courts to

“examine the scientific basis of expert testimony to ensure that it meets the same rigorous standard employed by scientific researchers and practitioners outside the courtroom.”

RMSE at xiii.  The preface falters, however, on two key issues, causation and conflicts of interest, which are taken up as an introduction to the new volume.

1. CAUSATION

The authors tell us in squishy terms that causal assessments are judgments:

“Fundamentally, the task is an inferential process of weighing evidence and using judgment to conclude whether or not an effect is the result of some stimulus. Judgment is required even when using sophisticated statistical methods. Such methods can provide powerful evidence of associations between variables, but they cannot prove that a causal relationship exists. Theories of causation (evolution, for example) lose their designation as theories only if the scientific community has rejected alternative theories and accepted the causal relationship as fact. Elements that are often considered in helping to establish a causal relationship include predisposing factors, proximity of a stimulus to its putative outcome, the strength of the stimulus, and the strength of the events in a causal chain.”

RMSE at xiv.

The authors leave the inferential process as a matter of “weighing evidence,” but without saying anything about how the scientific community does its “weighing.”  Language about “proving” causation is also unclear because “proof” in scientific parlance connotes a demonstration, which we typically find in logic or in mathematics.  Proving empirical propositions suggests a bar set too high such that the courts must inevitable lower the bar considerably.  The question is, of course, how low will judges go to admit evidence.

The authors thus introduce hand waving and excuses for why evidence can be weighed differently in court proceedings from the world of science:

“Unfortunately, judges may be in a less favorable position than scientists to make causal assessments. Scientists may delay their decision while they or others gather more data. Judges, on the other hand, must rule on causation based on existing information. Concepts of causation familiar to scientists (no matter what stripe) may not resonate with judges who are asked to rule on general causation (i.e., is a particular stimulus known to produce a particular reaction) or specific causation (i.e., did a particular stimulus cause a particular consequence in a specific instance). In the final analysis, a judge does not have the option of suspending judgment until more information is available, but must decide after considering the best available science.”

RMSE at xiv.  But the “best available science” may be pretty crummy, and the temptation to turn desperation into evidence (“well, it’s the best we have now”) is often severe.  The authors of the Preface signal that “inconclusive” is not a judgment open to judges charged with expert witness gatekeeping.  If the authors truly mean to suggest that judges should go with whatever is dished out as “the best available science,” then they have overlooked the obvious:  Rule 702 opens the door to “scientific, technical, or other specialized knowledge,” not to hunches, suggestive but inconclusive evidence, and wishful thinking about how the science may turn out when further along.  Courts have a choice to exclude expert witness opinion testimony that is based upon incomplete or inconclusive evidence.

2. CONFLICTS OF INTEREST

Surprisingly, given the scope of the scientific areas covered in the RMSE, the authors discuss conflicts of interest (COI) at some length.  Conflicts of interest are a fact of life in all endeavors, and it is understandable counsel judges and juries to try to identify, assess, and control them.  COIs, however, are weak proxies for unreliability.  The emphasis given here is undue because federal judges are misled into thinking that they can discern unreliability from COI, when they should be focused on the data and the analysis.

The authors of the Preface set about to use COI as a basis for giving litigation plaintiffs a pass, and for holding back studies sponsored by corporate defendants.

“Conflict of interest manifests as bias, and given the high stakes and adversarial nature of many courtroom proceedings, bias can have a major influence on evidence, testimony, and decisionmaking. Conflicts of interest take many forms and can be based on religious, social, political, or other personal convictions. The biases that these convictions can induce may range from serious to extreme, but these intrinsic influences and the biases they can induce are difficult to identify. Even individuals with such prejudices may not appreciate that they have them, nor may they realize that their interpretations of scientific issues may be biased by them. Because of these limitations, we consider here only financial conflicts of interest; such conflicts are discoverable. Nonetheless, even though financial conflicts can be identified, having such a conflict, even one involving huge sums of money, does not necessarily mean that a given individual will be biased. Having a financial relationship with a commercial entity produces a conflict of interest, but it does not inevitably evoke bias. In science, financial conflict of interest is often accompanied by disclosure of the relationship, leaving to the public the decision whether the interpretation might be tainted. Needless to say, such an assessment may be difficult. The problem is compounded in scientific publications by obscure ways in which the conflicts are reported and by a lack of disclosure of dollar amounts.

Judges and juries, however, must consider financial conflicts of interest when assessing scientific testimony. The threshold for pursuing the possibility of bias must be low. In some instances, judges have been frustrated in identifying expert witnesses who are free of conflict of interest because entire fields of science seem to be co-opted by payments from industry. Judges must also be aware that the research methods of studies funded specifically for purposes of litigation could favor one of the parties. Though awareness of such financial conflicts in itself is not necessarily predictive of bias, such information should be sought and evaluated as part of the deliberations.”

RMSE at xiv-xv.  All in all, rather misleading advice.  Financial conflicts are not the only conflicts that can be “discovered.”  Often expert witnesses will have political and organizational alignments, which will show deep-seated ideological alignments with the party for which they are testifying.  For instance, in one silicosis case, an expert witness in the field of history of medicine testified, at an examination before trial, that his father suffered from a silica-related disease.  This witness’s alignment with Marxist historians and his identification with radical labor movements made his non-financial conflicts obvious, although these COI would not necessarily have been apparent from his scholarly publications alone.

How low will the bar be set for discovering COI?  If testifying expert witnesses are relying upon textbooks, articles, essays, will federal courts open the authors/hearsay declarants up to searching discovery of their finances?

Also misleading is the suggestion that “entire fields of science seem to be co-opted by payments from industry.”  Do the authors mean to exclude the plaintiffs’ lawyer litigation industry, which has grown so large and politically powerful in this country?  In litigations in which I have been involved, I have certainly seen plaintiffs’ counsel, or their proxies – labor unions or “victim support groups” provide substantial funding for studies.  The Preface authors themselves show an untoward bias by their pointing out industry payments without giving balanced attention to other interested parties’ funding of scientific studies.

The attention to COI is also surprising given that one of the key chapters, for toxic tort practitioners, was written by Dr. Bernard D. Goldstein, who has testified in toxic tort cases, mostly (but not exclusively) for plaintiffs.  See, e.g., Parker v. Mobil Oil Corp., 7 N.Y.3d 434, 857 N.E.2d 1114, 824 N.Y.S.2d 584 (2006); Exxon Corp. v. Makofski, 116 SW 3d 176 (Tex. Ct. App. 2003).  The Makofsky case is particularly interesting because Dr. Goldstein was forced to explain why he was willing to opine that benzene caused acute lymphocytic leukemia, despite the plethora of published studies finding no statistically significant relationship.  Dr. Goldstein resorted to the inaccurate notion that scientific “proof” of causation requires 95 percent certainty, whereas he imposed only a 51 percent certainty for his medico-legal testimonial adventures. Dr. Goldstein also attempted to justify the discrepancy from the published literature by adverting to the lower standards used by federal regulatory agencies and treating physicians. Id.

These explanations are particularly concerning because they reflect basic errors in statistics and in causal reasoning.  The 95 percent derives from the use of the same percentage in confidence intervals, but the probability involved there is not the probability of the association’s being correct, and it has nothing to do with the probability in the belief that an association is real or is causal.  (Thankfully the RMSE chapter on statistics gets this right, but my fear is that judges will skip over the more demanding chapter on statistics and place undue weight on the toxicology chapter, written by Dr. Goldstein.)  The reference to federal agencies (OSHA, EPA, etc.) and to treating physicians was meant, no doubt, to invoke precautionary principle concepts as a justification for some vague, ill-defined, lower standard of causal assessment.

The Preface authors might well have taken their own counsel and conducted a more searching assessment of COI among authors of Reference Manual.  Better yet, the authors might have focused the judiciary on the data and the analysis.

In “the Top Reason that the ALI’s Restatement of Torts Should Steer Clear of Partisan Conflicts,” I pointed out the inappropriateness of advertising the ALI’s Restatement of Torts to the organized plaintiffs’ bar, much as the plaintiffs’ bar advertises potential huge recoveries for the latest tort du jour.  See Michael D. Green & Larry S. Stewart, “The New Restatement’s Top 10 Tort Tools,” Trial 44 (April 2010).

Some of the authors’ tort tool kit may be unexceptionable.  Among these authors’ top ten tort tools, however, is the new Restatement’s edict that “reasonable degree of medical certainty” means, or should mean, nothing more than saying “more likely than not.”  The authors criticize the reasonable certainty standard with an abbreviated rendition of the Restatement’s critique:

“Many courts hold that expert opinion must be expressed in terms of medical or scientific certainty’. Requiring certainty seems to impose a criminal law-like burden of proof that is inconsistent with civil burdens of preponderance of the evidence to establish a fact. Such a requirement is also problematic at best because medical and scientific communities have no such ‘reasonable certainty’ standard. The standard then becomes whatever the attorney who hired the expert tells the expert it means or, absent that, whatever the expert imagines it means. Section 28, comment e, of the Restatement criticizes this standard and makes clear that the same preponderance standard (or ‘more likely than not’ standard), which is universally applied in all aspects of civil cases, also applies to expert testimony.”

Id. at 46-47.

Well, the more likely than not standard is not “universally applied in all aspects of civil cases,” because several states require exemplary damages to be proven by “clear and convincing” or greater evidence.  In some states, the burden of proof in fraud cases is higher than a mere preponderance of the evidence. This premise of the authors’ article is incorrect.

But even if the authors were correct that the preponderance standard applied “in all aspects” of civil cases, their scholarship would remain suspect, as others and I have previously pointed out.  See “Reasonable Degree of Medical Certainty,” and “More Uncertainty About Reasonable Degree of Medical Certainty.”

1. The Restatement’s Treatment of Expert Witness Evidentiary Rules Exceeded the Scope of the Tort Restatement.

The most peculiar aspect of this “top tool,” is that it has nothing to do with the law of torts.  The level of certitude required of an expert witness is an evidentiary and a procedural issue. Of course the issue comes up in tort cases, which frequently involve medical and scientific causation opinions, as well as other expert witness opinions.  The issue, however, comes up in all cases that involve expert witnesses:  trust and estates, regulatory, environmental, securities fraud, commercial, and other cases.

The Restatement of Torts weakly acknowledges its frolic and detour in treating a procedural issue concerning the admissibility of expert witness opinion testimony, by noting that it does “not address any other requirements for the admissibility of an expert witness’s testimony, including qualifications, expertise, investigation, methodology, or reasoning.” Restatement (Third) of Torts: Liability for Physical and Emotional Harm § 28, cmt. e (2010).  The certitude issue has nothing special to do with the substantive law of torts, and should not have been addressed in the torts restatement.

2. The Restatement’s Treatment of “Reasonable Degree of Medical Certainty” Has No Relevance to the Burden of Proof in Tort Cases.

The expert witness certitude issue has nothing to do with the burden of proof, and the Restatement should not have confused and conflated the burden of proof with the standard of certitude for expert witnesses.  The clear but unacceptable implication is that expert witnesses in criminal cases must testify to certitude “beyond a reasonable doubt,” and in claims for equitable relief, expert witnesses may share only opinions that are made, in their minds, by “clear and convincing evidence.”  There is no support in law or logic for the identification of witness certitude with parties’ burdens of proof.

Comment e states the critique more fully:

“If courts do interpret the reasonable-certainty standard to require a level of certitude greater than the preponderance-of-the-evidence standard requires, this creates a troubling inconsistency between standards for the admissibility of evidence and the threshold required for sufficiency of proof. The threshold for admissibility should not be higher than the threshold to sufficiency.  Moreover, the reasonable-certainty standard provides no assurance of the quality of the expert’s qualifications, expertise, investigation, methodology, or reasoning.  Thus, the Section adopts the same preponderance standard that is universally adopted in civil cases.  Direct and cross-examination can be employed to flesh out the degree of certainty with which an expert’s opinion is held and to identify opinions that are speculative and therefore inadmissible.”

Id. The critique badly misfires because there is no inconsistency and no trouble in having different standards for the admissibility of opinion evidence and the burden of proof.  As noted, expert witnesses testify on causation and other issues in criminal, equity, and tort cases, all with different burdens of proof.  Juries in criminal and tort cases must apply instructions on burdens of proof to an entire evidentiary display, not just the expert witnesses’ opinions.  In logic and law, there ultimately must be different burdens for admissibility of expert witness testimony and for sufficiency of a party’s proofs.

3. The Restatement’s Treatment of “Reasonable Degree of Medical Certainty” Incoherently Confuses Two Different Standards.

We can see that Comment e’s approach to legislating an equivalence between expert witness certitude and the burden must fail even on its own terms.  Consider the legal consequences of tort claimants, with the burden of proof, who produce expert witnesses to opine about key elements (e.g., causation) of torts by stating that their opinions were held by a mere “preponderance of the evidence.”

If this probability is understood to be only infinitesimally greater than 50%, then courts would have to direct verdicts in many (and perhaps most) cases.

Courts must ensure that a rational jury can find for the party with the burden of proof.  Juries must evaluate the credibility and reliability of expert witnesses, their opinions, as well as the predicate facts for those opinions.  If those expert witness opinions were barely greater than 50% probable on an essential element, then unless the witnesses had perfect credibility, and all predicate facts were as probable as claimed by the witnesses, then juries would frequently have to reject the witnesses’ opinions.  The bare preponderance of the expert witnesses’ opinions would result in an overall probability of the essential element less than 50%.

4. The Restatement Incorrectly Implies that Expert Witnesses Can Quantify Their Opinions in Probabilistic Terms.

There are even more far-reaching problems with simply substituting “more likely than not” for RDMC as a threshold requirement of expert witness testimony.  Comment e implies that expert witnesses can discern the difference between an opinion that they believe is “more likely than not” and another which is “as likely as not.” On some occasions, there may be opinions that derive from quantitative reasoning, for which an expert witness could truly say, with some level of certainty, that his or her opinion is “more likely than not.” On most occasions, an expert witness’s degree of certainty is a qualitative opinion that simply does not admit of a quantitative characterization. The Restatement’s comment perpetuates this confusion by casting the reasonable certainty standard as a bare probability.

Comment e further suggests that expert witnesses are themselves expert in assessing their own level of certainty, and that they have the training and experience to distinguish an opinion that is 50.1% likely from another that is only 50% likely. The assignment of precise mathematical probabilities to personal, subjective beliefs is a doubtful exercise, at best. See, e.g., Daniel Kahneman and Amos Tversky, “Judgment under Uncertainty: Heuristics and Biases,” 185 Science 1124 (1974).

5. The Restatement Incorrectly Labels “Reasonable Degree of Medical Certainty” As An Empty Formalism.

Comment e ignores the epistemic content of reasonable certainty, which bears an uncanny resemblance to the knowledge requirement of Rule 702.  The “mantra” is helpful to the extent it imposes an objective epistemic standard, especially in states that have failed to impose, or that have abrogated, expert witness gatekeeping.  In some states, there is no meaningful expert witness gatekeeping under either the Frye standard or Rule 702. See, e.g., “Expert Evidence Free-for-All in Washington State.”  See also Joseph Sanders, “Science, Law, and the Expert Witness,” 72 Law & Contemporary Problems 63, 87 & n. 118 (2009) (noting that the meaning of “reasonable degree of scientific certainty” is unclear, but that it can be understood as an alternative formulation of Kumho’s “same intellectual rigor” test).

Some of these “top” tools may be defective.  The authors may need good defense counsel.