TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

The Treatment of Meta-Analysis in the Third Edition of the Reference Manual on Scientific Evidence

November 14th, 2011

Meta-analysis is a statistical procedure for aggregating data and statistics from individual studies into a single summary statistical estimate of the population measurement of interest.  The first meta-analysis is typically attributed to Karl Pearson, circa 1904, who sought a method to overcome the limitations of small sample size and low statistical power.  Statistical methods for meta-analysis, however, did not mature until the 1970s.  Even then, the biomedical scientific community remained skeptical of, if not out rightly hostile to, meta-analysis until relatively recently.

The hostility to meta-analysis, especially in the context of observational epidemiologic studies, was colorfully expressed by Samuel Shapiro and Alvan Feinstein, as late as the 1990s:

“Meta-analysis begins with scientific studies….  [D]ata from these studies are then run through computer models of bewildering complexity which produce results of implausible precision.”

* * * *

“I propose that the meta-analysis of published non-experimental data should be abandoned.”

Samuel Shapiro, “Meta-analysis/Smeta-analysis,” 140 Am. J. Epidem. 771, 777 (1994).  See also Alvan Feinstein, “Meta-Analysis: Statistical Alchemy for the 21st Century,” 48 J. Clin. Epidem. 71 (1995).

The professional skepticism about meta-analysis was reflected in some of the early judicial assessments of meta-analysis in court cases.  In the 1980s and early 1990s, some trial judges erroneously dismissed meta-analysis as a flawed statistical procedure that claimed to make something out of nothing. Allen v. Int’l Bus. Mach. Corp., No. 94-264-LON, 1997 U.S. Dist. LEXIS 8016, at *71–*74 (suggesting that meta-analysis of observational studies was controversial among epidemiologists).

In In re Paoli Railroad Yard PCB Litigation, Judge Robert Kelly excluded plaintiffs’ expert witness Dr. William Nicholson and his testimony based upon his unpublished meta-analysis of health outcomes among PCB-exposed workers.  Judge Kelly found that the meta-analysis was a novel technique, and that Nicholson’s meta-analysis was not peer reviewed.  Furthermore, the meta-analysis assessed health outcomes not experienced by any of the plaintiffs before the trial court.  706 F. Supp. 358, 373 (E.D. Pa. 1988).

The Court of Appeals for the Third Circuit reversed the exclusion of Dr. Nicholson’s testimony, and remanded for reconsideration with instructions.  In re Paoli R.R. Yard PCB Litig., 916 F.2d 829, 856-57 (3d Cir. 1990), cert. denied, 499 U.S. 961 (1991); Hines v. Consol. Rail Corp., 926 F.2d 262, 273 (3d Cir. 1991).  The Circuit noted that meta-analysis was not novel, and that the lack of peer-review was not an automatic disqualification.  Acknowledging that a meta-analysis could be performed poorly using invalid methods, the appellate court directed the trial court to evaluate the validity of Dr. Nicholson’s work on his meta-analysis.

In one of many squirmishes over colorectal cancer claims in asbestos litigation, Judge Sweet in the Southern District of New York was unimpressed by efforts to aggregate data across studies.  Judge Sweet declared that “no matter how many studies yield a positive but statistically insignificant SMR for colorectal cancer, the results remain statistically insignificant. Just as adding a series of zeros together yields yet another zero as the product, adding a series of positive but statistically insignificant SMRs together does not produce a statistically significant pattern.”  In In re Joint E. & S. Dist. Asbestos Litig., 827 F. Supp. 1014, 1042 (S.D.N.Y. 1993).  The plaintiffs’ expert witness who had offered the unreliable testimony, Dr. Steven Markowitz, like Nicholson, another foot soldier in Dr. Irving Selikoff’s litigation machine, did not offer a formal meta-analysis to justify his assessment that multiple non-significant studies, taken together, rule out chance as a likely explanation for an aggregate finding of an increased risk.

Judge Sweet was quite justified in rejecting this back of the envelope, non-quantitative meta-analysis.  His suggestion, however, that multiple non-significant studies could never collectively serve to rule out chance as an explanation for an overall increased rate of disease in the exposed groups is wrong.  Judge Sweet would have better focused on the validity issues in key studies, the presence of bias and confounding, and the completeness of the proffered meta-analysis.  The Second Circuit reversed the entry of summary judgment, and remanded the colorectal cancer claim for trial.  52 F.3d 1124 (2d Cir. 1995).  Over a decade later, with even more accumulated studies and data, the Institute of Medicine found the evidence for asbestos plaintiffs’ colorectal cancer claims to be scientifically insufficient.  Institute of Medicine, Asbestos: Selected Cancers (Wash. D.C. 2006).

Courts continue to go astray with an erroneous belief that multiple studies, all without statistically significant results, cannot yield a statistically significant summary estimate of increased risk.  See, e.g., Baker v. Chevron USA, Inc., 2010 WL 99272, *14-15 (S.D.Ohio 2010) (addressing a meta-analysis by Dr. Infante on multiple myeloma outcomes in studies of benzene-exposed workers).  There were many sound objections to Infante’s meta-analysis, but the suggestion that multiple studies without statistical significance could not yield a summary estimate of risk with statistical significance was not one of them.

In the last two decades, meta-analysis has emerged as an important technique for addressing random variation in studies, as well as some of the limitations of frequentist statistical methods.  In 1980s, articles reporting meta-analyses were rare to non-existent.  In 2009, there were over 2,300 articles with “meta-analysis” in their title, or in their keywords, indexed in the PubMed database of the National Library of Medicine.  See Michael O. Finkelstein and Bruce Levin, “Meta-Analysis of ‘Sparse’ Data: Perspectives from the Avandia Cases” (2011) (forthcoming in Jurimetrics).

The techniques for aggregating data have been studied, refined, and employed extensively in thousands of methods and application papers in the last decade. Consensus guideline papers have been published for meta-analyses of clinical trials as well as observational studies.  See Donna Stroup, et al., “Meta-analysis of Observational Studies in Epidemiology: A Proposal for Reporting,” 283 J. Am. Med. Ass’n 2008 (2000) (MOOSE statement); David Moher, Deborah Cook, Susan Eastwood, Ingram Olkin, Drummond Rennie, and Donna Stroup, “Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement,” 354 Lancet 1896 (1999).  See also Jesse Berlin & Carin Kim, “The Use of Meta-Analysis in Pharmacoepidemiology,” in Brian Strom, ed., Pharmacoepidemiology 681, 683–84 (4th ed. 2005); Zachary Gerbarg & Ralph Horwitz, “Resolving Conflicting Clinical Trials: Guidelines for Meta-Analysis,” 41 J. Clin. Epidemiol. 503 (1988).

Meta-analyses, of observational studies and of randomized clinical trials, routinely are relied upon by expert witnesses in pharmaceutical and so-called toxic tort litigation. Id. See also In re Bextra and Celebrex Marketing Sales Practices and Prod. Liab. Litig., 524 F. Supp. 2d 1166, 1174, 1184 (N.D. Cal. 2007) (holding that reliance upon “[a] meta-analysis of all available published and unpublished randomized clinical trials” was reasonable and appropriate, and criticizing the expert witnesses who urged the complete rejection of meta-analysis of observational studies)

The second edition of the Reference Manual on Scientific Evidence gave very little attention to meta-analysis.  With this historical backdrop, it is interesting to see what the new third edition provides for guidance to the federal judiciary on this important topic.

STATISTICS CHAPTER

The statistics chapter of the third edition gives continues to give scant attention to meta-analysis.  The chapter notes, in a footnote, that there are formal procedures for aggregating data across studies, and that the power of the aggregated data will exceed the power of the individual, included studies.  The footnote then cautions that meta-analytic procedures “have their own weakness,” without detailing what that one weakness is.  RMSE 3d at 254 n. 107.

The glossary at the end of the statistics chapter offers a definition of meta-analysis:

“meta-analysis. Attempts to combine information from all studies on a certain topic. For example, in the epidemiological context, a meta-analysis may attempt to provide a summary odds ratio and confidence interval for the effect of a certain exposure on a certain disease.”

Id. at 289.

This definition is inaccurate in ways that could yield serious mischief.  Virtually all meta-analyses are built upon a systematic review that sets out to collect all available studies on a research issue of interest.  It is a rare meta-analysis, however, that includes “all” studies in its quantitative analysis.  The meta-analytic process involves a pre-specification of inclusionary and exclusionary criteria for the quantitative analysis of the summary estimate of risk.  Those criteria may limit the quantitative analysis to randomized trials, or to analytical epidemiologic studies.  Furthermore, meta-analyses frequently and appropriately have pre-specified exclusionary criteria that relate to study design or quality.

On a more technical note, the offered definition suggests that the summary estimate of risk will be an odds ratio, which may or may not be true.  Meta-analyses of risk ratios may yield summary estimates of risk in terms of relative risk or hazard ratios, or even of risk differences.  The meta-analysis may combine data of means rather than proportions as well.

EPIDEMIOLOGY CHAPTER

The chapter on epidemiology delves into meta-analysis in greater detail than the statistics chapter, and offers apparently inconsistent advice.  The overall gist of the chapter, however, can perhaps best be summarized by the definition offered in this chapter’s glossary:

“meta-analysis. A technique used to combine the results of several studies to enhance the precision of the estimate of the effect size and reduce the plausibility that the association found is due to random sampling error.  Meta-analysis is best suited to pooling results from randomly controlled experimental studies, but if carefully performed, it also may be useful for observational studies.”

Reference Guide on Epidemiology, RSME3d at 624.  See also id. at 581 n. 89 (“Meta-analysis is better suited to combining results from randomly controlled experimental studies, but if carefully performed it may also be helpful for observational studies, such as those in the epidemiologic field.”).  The epidemiology chapter appropriately notes that meta-analysis can help address concerns over random error in small studies.  Id. at 579; see also id. at 607 n. 171.

Having told us that properly conducted meta-analyses of observational studies can be helpful, the chapter hedges considerably:

“Meta-analysis is most appropriate when used in pooling randomized experimental trials, because the studies included in the meta-analysis share the most significant methodological characteristics, in particular, use of randomized assignment of subjects to different exposure groups. However, often one is confronted with nonrandomized observational studies of the effects of possible toxic substances or agents. A method for summarizing such studies is greatly needed, but when meta-analysis is applied to observational studies – either case-control or cohort – it becomes more controversial.174 The reason for this is that often methodological differences among studies are much more pronounced than they are in randomized trials. Hence, the justification for pooling the results and deriving a single estimate of risk, for example, is problematic.175

Id. at 607.  The stated objection to pooling results for observational studies is certainly correct, but many research topics have sufficient studies available to allow for appropriate selectivity in framing inclusionary and exclusionary criteria to address the objection.  The chapter goes on to credit the critics of meta-analyses of observational studies.  As they did in the second edition of the RSME, the authors repeat their cites to, and quotes from, early papers by John Bailar, who was then critical of such meta-analyses:

“Much has been written about meta-analysis recently and some experts consider the problems of meta-analysis to outweigh the benefits at the present time. For example, John Bailar has observed:

‘[P]roblems have been so frequent and so deep, and overstatements of the strength of conclusions so extreme, that one might well conclude there is something seriously and fundamentally wrong with the method. For the present . . . I still prefer the thoughtful, old-fashioned review of the literature by a knowledgeable expert who explains and defends the judgments that are presented. We have not yet reached a stage where these judgments can be passed on, even in part, to a formalized process such as meta-analysis.’

John Bailar, “Assessing Assessments,” 277 Science 528, 529 (1997).”

Id. at 607 n.177.  Bailar’s subjective preference for “old-fashioned” reviews, which often cherry picked the included studies is, well, “old fashioned.”  More to the point, it is questionable science, and a distinctly minority viewpoint in the light of substantial improvements in the conduct and reporting of meta-analyses of observational studies.  Bailar may be correct that some meta-analyses should have never left the protocol stage, but the RMSE 3d fails to provide the judiciary with the tools to appreciate the distinction between good and bad meta-analyses.

This categorical rejection, cited with apparent approval, is amplified by a recitation of some real or apparent problems with meta-analyses of observational studies.  What is missing is a discussion of how many of these problems can be and are dealt with in contemporary practice:

“A number of problems and issues arise in meta-analysis. Should only published papers be included in the meta-analysis, or should any available studies be used, even if they have not been peer reviewed? Can the results of the meta-analysis itself be reproduced by other analysts? When there are several meta-analyses of a given relationship, why do the results of different meta-analyses often disagree? The appeal of a meta-analysis is that it generates a single estimate of risk (along with an associated confidence interval), but this strength can also be a weakness, and may lead to a false sense of security regarding the certainty of the estimate. A key issue is the matter of heterogeneity of results among the studies being summarized.  If there is more variance among study results than one would expect by chance, this creates further uncertainty about the summary measure from the meta-analysis. Such differences can arise from variations in study quality, or in study populations or in study designs. Such differences in results make it harder to trust a single estimate of effect; the reasons for such differences need at least to be acknowledged and, if possible, explained.176 People often tend to have an inordinate belief in the validity of the findings when a single number is attached to them, and many of the difficulties that may arise in conducting a meta-analysis, especially of observational studies such as epidemiologic ones, may consequently be overlooked.177

Id. at 608.  The authors are entitled to their opinion, but their discussion leaves the judiciary uninformed about current practice, and best practices, in epidemiology.  A categorical rejection of meta-analyses of observational studies is at odds with the chapter’s own claim that such meta-analyses can be helpful if properly performed.  What was needed, and is missing, is a meaningful discussion to help the judiciary determine whether a meta-analysis of observational studies was properly performed.

MEDICAL TESTIMONY CHAPTER

The chapter on medical testimony is the third pass at meta-analysis in RMSE 3d.   The second edition’s chapter on medical testimony ignored meta-analysis completely; the new edition addresses meta-analysis in the context of the hierarchy of study designs:

“Other circumstances that set the stage for an intense focus on medical evidence included

(1) the development of medical research, including randomized controlled trials and other observational study designs;

(2) the growth of diagnostic and therapeutic interventions;141

(3) interest in understanding medical decision making and how physicians reason;142 and

(4) the acceptance of meta-analysis as a method to combine data from multiple randomized trials.143

RMSE 3d at 722-23.

The chapter curiously omits observational studies, but the footnote reference (note 143) then inconsistently discusses two meta-analyses of observational, rather than experimental, studies:

“143. Video Software Dealers Ass’n v. Schwarzenegger, 556 F.3d 950, 963 (9th Cir. 2009) (analyzing a meta-analysis of studies on video games and adolescent behavior); Kennecott Greens Creek Min. Co. v. Mine Safety & Health Admin., 476 F.3d 946, 953 (D.C. Cir. 2007) (reviewing the Mine Safety and Health Administration’s reliance on epidemiological studies and two meta-analyses).”

Id. at 723 n.143.

The medical testimony chapter then provides further confusion by giving a more detailed listing of the hierarchy of medical evidence in the form of different study designs:

3. Hierarchy of medical evidence

With the explosion of available medical evidence, increased emphasis has been placed on assembling, evaluating, and interpreting medical research evidence.  A fundamental principle of evidence-based medicine (see also Section IV.C.5, infra) is that the strength of medical evidence supporting a therapy or strategy is hierarchical.  When ordered from strongest to weakest, systematic review of randomized trials (meta-analysis) is at the top, followed by single randomized trials, systematic reviews of observational studies, single observational studies, physiological studies, and unsystematic clinical observations.150 An analysis of the frequency with which various study designs are cited by others provides empirical evidence supporting the influence of meta-analysis followed by randomized controlled trials in the medical evidence hierarchy.151 Although they are at the bottom of the evidence hierarchy, unsystematic clinical observations or case reports may be the first signals of adverse events or associations that are later confirmed with larger or controlled epidemiological studies (e.g., aplastic anemia caused by chloramphenicol,152 or lung cancer caused by asbestos153). Nonetheless, subsequent studies may not confirm initial reports (e.g., the putative association between coffee consumption and pancreatic cancer).154

Id. at 723-24.  This discussion further muddies the water by using a parenthetical to suggest that meta-analyses of randomized clinical trials are equivalent to systematic reviews of such studies — “systematic review of randomized trials (meta-analysis).” Of course, systematic reviews are not meta-analyses, although they are a necessary precondition for conducting a meta-analysis.  The relationship between the procedures for a systematic review and a meta-analysis are in need of clarification, but the judiciary will not find it in the new Reference Manual.

New-Age Levellers – Flattening Hierarchy of Evidence

October 30th, 2011

The Levelers were political dissidents in England, in the middle of the 17th century.  Among their causes, Levelers advanced popular sovereignty, equal protection of the law, and religious tolerance.

The political agenda of the Levelers sounds quite noble to 21st century Americans, but their ideals have no place in the world of science:  not all opinions or scientific studies are created equally; not all opinions are worthy of being taken seriously in scientific discourse or in courtroom presentations of science; and not all opinions should be tolerated, especially when they claim causal conclusions based upon shoddy or inadequate evidence.

In some litigations, legal counsel set out to obscure the important quantitative and qualitative distinctions among scientific studies.  Sometimes, lawyers find cooperative expert witnesses, willing to engage in hand waving about “the weight of the evidence,” where the weights are assigned post hoc, in a highly biased fashion.  No study (that favors the claim) left behind.  This is not science, and it is not how science operates, even though some expert witnesses, such as Professor Cranor in the Milward case, have been able to pass off their views as representative of scientific practice.

A sound appreciation of how scientists evaluate studies, and of why not all studies are equal, is essential to any educated evaluation of scientific controversies.  Litigants who face high-quality studies, with results inconsistent with their litigation claims, may well resort to “leveling” of studies.  This leveling may be advanced out of ignorance, but more likely the leveling is an attempt to snooker courts with evidence from exploratory, preliminary, and hypothesis-generating studies as somehow equal to, or greater than, the value of hypothesis-testing studies.

Some of the leveling tactics that have become commonplace in litigation include asserting that:

  • All experts witnesses are the same;
  • All expert witnesses conduct the same analysis;
  • All expert witnesses read articles, interpret them, and offer opinions;
  • All expert witnesses are inherently biased;
  • All expert witnesses select the articles to read and interpret in line with their biases;
  • All epidemiologic studies are the same;
  • All studies are flawed; and
  • All opinions are, in the final analysis, subjective.

This leveling strategy can be seen in Professor Margaret Berger’s introduction to the Reference Manual on Scientific Evidence (RMSE 3d), where she supported an ill-defined “weight-of-the-evidence” approach to causal judgments. SeeLate Professor Berger’s Introduction to the Reference Manual on Scientific Evidence” (Oct. 23, 2011).

Other chapters in the RMSE 3d are at odds with Berger’s introduction.  The epidemiology chapter does not explicitly address the hierarchy of studies, but it does describe cross-sectional, ecological, and secular trend studies are less able to support causal conclusions.  Cross-sectional studies are described as “rarely useful in identifying toxic agents,” RMSE 3d at 556, and as “used infrequently when the exposure of interest is an environmental toxic agent,” RMSE 3d at 561.  Cross-sectional studies are described as hypothesis-generating as opposed to hypothesis testing, although not in those specific terms.  Id. (describing cross-sectional studies as providing valuable leads for future research).  Ecological studies are described as useful for identifying associations, but not helpful in determining whether such associations are causal; and ecological studies are identified as a fertile source of error in the form of the “ecological fallacy.”  Id. at 561 -62.

The epidemiology chapter perhaps weakens its helpful description of the limited role of ecological studies by citing, with apparent approval, a district court that blinked at its gatekeeping responsibility to ensure that testifying expert witnesses did, in fact, rely upon “sufficient facts or data,” as well as upon studies that are “of a type reasonably relied upon by experts in the particular field in forming opinions or inferences upon the subject.” Rule 703. RMSE 3d at 561 n.34 (citing Cook v. Rockwell International Corp., 580 F. Supp. 2d 1071, 1095–96 (D. Colo. 2006), where the district court acknowledged the severe limitations of ecological studies in supporting causal inferences, but opined that the limitations went to the weight of the study). Of course, the insubstantial weight of an ecological study is precisely what may result in the study’s failure to support a causal claim.

The ray of clarity in the epidemiology chapter about the hierarchical nature of studies is muddled by an attempt to level epidemiology and toxicology.  The chapter suggests that there is no hierarchy of disciplines (as opposed to studies within a discipline).  RMSE 3d at 564 & n.48 (citing and quoting symposium paper that “[t]here should be no hierarchy [among different types of scientific methods to determine cancer causation]. Epidemiology, animal, tissue culture and molecular pathology should be seen as integrating evidences in the determination of human carcinogenicity.” Michele Carbone et al., “Modern Criteria to Establish Human Cancer Etiology,” 64 Cancer Res. 5518, 5522 (2004).)  Carbone, of course, is best known for his advocacy of a viral cause (SV40), of human mesothelioma, a claim unsupported, and indeed contradicted, by epidemiologic studies.  His statement does not support the chapter’s leveling of epidemiology and toxicology, and Carbone is, in any event, an unlikely source to cite.

The epidemiology chapter undermines its own description of the role of study design in evaluating causality by pejoratively asserting that most epidemiologic studies are “flawed”:

“It is important to emphasize that all studies have ‘flaws’ in the sense of limitations that add uncertainty about the proper interpretation of the results.9 Some flaws are inevitable given the limits of technology, resources, the ability and willingness of persons to participate in a study, and ethical constraints. In evaluating epidemiologic evidence, the key questions, then, are the extent to which a study’s limitations compromise its findings and permit inferences about causation.”

RSME 3d at 553.  This statement is actually a significant improvement over the second edition, where the authors of the epidemiology chapter asserted, without qualification:

“It is important to emphasize that most studies have flaws.”

RMSE 2d 337.  The “flaws” language from the earlier chapter was used on occasion by courts that were set on ignoring competing interpretations of epidemiologic studies.  Since all or most studies are flawed, why bother figuring out what is valid and reliable?  Just let the jury sort it out.  This is not an aid to gatekeeping, but rather a prescription for allowing the gatekeeper to call in sick.

The current epidemiology chapter essentially backtracks from the harsh connotations of its use of the term “flaws,” by now equating the term with “limitations.”  Flaws and limitations, however, are quite different from one another.  What is left out in the third edition’s description is the sense that there are indeed some studies that are so flawed that they must be disregarded altogether.  There may also be limitations in studies, especially observational studies, which is why the party with the burden of proof should generally not be allowed to proceed with only one or two epidemiologic studies.  Rule 702, after all, requires that an expert opinion to be based upon “sufficient facts or data.”

The RSME 3d chapter on medical evidence is a refreshing break from the leveling approach seen elsewhere.  Here at least, the chapter authors devote several pages to explaining the role of study design in assessing an etiological issue:

3. Hierarchy of medical evidence

With the explosion of available medical evidence, increased emphasis has been placed on assembling, evaluating, and interpreting medical research evidence.  A fundamental principle of evidence-based medicine (see also Section IV.C.5, infra) is that the strength of medical evidence supporting a therapy or strategy is hierarchical.

When ordered from strongest to weakest, systematic review of randomized trials (meta-analysis) is at the top, followed by single randomized trials, systematic reviews of observational studies, single observational studies, physiological studies, and unsystematic clinical observations.150 An analysis of the frequency with which various study designs are cited by others provides empirical evidence supporting the influence of meta-analysis followed by randomized controlled trials in the medical evidence hierarchy.151 Although they are at the bottom of the evidence hierarchy, unsystematic clinical observations or case reports may be the first signals of adverse events or associations that are later confirmed with larger or controlled epidemiological studies (e.g., aplastic anemia caused by chloramphenicol,152 or lung cancer caused by asbestos153). Nonetheless, subsequent studies may not confirm initial reports (e.g., the putative association between coffee consumption and pancreatic cancer).154

John B. Wong, Lawrence O. Gostin, and Oscar A. Cabrera, “Reference Guide on Medical Testimony,” RMSE 3d 687, 723 -24 (2011).  The third edition’s chapter is a significant improvement of the second edition’s chapter on medical testimony, which does not mention the hierarchy of evidence.  Mary Sue Henifin, Howard M. Kipen, and Susan R. Poulter, ” Reference Guide on Medical Testimony,” RMSE 2d 440 (2000).  Indeed, the only time the word “hierarchy” appeared in the entire second edition was in connection with the hierarchy of the federal judiciary.

The tension, contradictions, and differing emphases among the various chapters of the RSME 3d point to an important “flaw” in the new edition.  The chapters appear to have been written largely in isolation, and without much regard for what the other chapters contain.  The chapters overlap, and indeed contradict one another on key points.  Witness Berger’s rejection of the hierarchy of evidence, the epidemiology chapter’s inconstant presentation of the concept without mentioning it by name, and the medical testimony chapter’s embrace and explicit presentation of the hierarchical nature of medical study evidence.  Fortunately, the laissez-faire editorial approach allowed the disagreement to remain, without censoring any position, but the federal judiciary is not aided by the contradiction and tension in the approaches.

Given the importance of the concept, even the medical testimony chapter in RSME 3d may seem to be too little, too late to be helpful to the judiciary.  There are book-length treatments of systematic reviews and “evidence-based medicine”: the three pages in Wong’s chapter barely scratch the surface of this important topic of how evidence is categorized, evaluated, and synthesized in making judgments of causality.

There are many textbooks and articles available to judges and lawyers on how to assess medical studies.  Recently, John Cherrie has posted on his blog, OH-world, about a series of 17 articles, in the journal Aerzteblatt International, on the proper evaluation of medical and epidemiologic studies.

These papers, overall, make the point that not all studies are equal, and that not all evidentiary displays are adequate to support conclusions of causal association.  The papers are available without charge from the journal’s website:

01. Critical Appraisal of Scientific Articles

02. Study Design in Medical Research

03. Types of Study in Medical Research

04. Confidence Interval or P-Value?

05. Requirements and Assessment of Laboratory Tests: Inpatient Admission Screening

06. Systematic Literature Reviews and Meta-Analyses

07. The Specification of Statistical Measures and Their Presentation in Tables and Graphs

08. Avoiding Bias in Observational Studies

09. Interpreting Results in 2×2 Tables

10. Judging a Plethora of p-Values: How to Contend With the Problem of Multiple Testing

11. Data Analysis of Epidemiological Studies

12. Choosing statistical tests

13. Sample size calculation in clinical trials

14. Linear regression analysis

15. Survival analysis

16. Concordance analysis

17. Randomized controlled trials

This year, the Journal of Clinical Epidemiology began publishing a series of papers, known by the acronym GRADE, which aim to provide guidance on how studies are categorized and assessed for their evidential quality in supporting treatments and intervention.  The GRADE project is led by Gordon Guyatt, who is known for having coined the term “evidence-based medicine,” and written widely on the subject.  Guyatt, along with his colleagues including Peter Tugwell (who was one of the court-appointed expert witnesses in MDL 926), has described the GRADE project:

“The ‘Grades of Recommendation, Assessment, Development, and Evaluation’ (GRADE) approach provides guidance for rating quality of evidence and grading strength of recommendations in health care. It has important implications for those summarizing evidence for systematic reviews, health technology assessment, and clinical practice guidelines. GRADE provides a systematic and transparent framework for clarifying questions, determining the outcomes of interest, summarizing the evidence that addresses a question, and moving from the evidence to a recommendation or decision. Wide dissemination and use of the GRADE approach, with endorsement from more than 50 organizations worldwide, many highly influential   http://www.gradeworkinggroup.org/), attests to the importance of this work. This article introduces a 20-part series providing guidance for the use of GRADE methodology that will appear in the Journal of Clinical Epidemiology.”

Gordon Guyatt, Andrew D. Oxman, Holger Schünemann, Peter Tugwell, Andre Knottnerus, “GRADE guidelines – new series of articles in Journal of Clinical Epidemiology,” 64 J. Clin. Epidem. 380 (2011).  See also Gordon Guyatt, Andrew Oxman, et al., for the GRADE Working Group, “Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations,” 336 Brit. Med. J. 924 (2008).  [pdf]

Of the 20 papers planned, 9 of the GRADE papers have been published to date in the Journal of Clinical Epidemiology:

01 Intro – GRADE evidence profiles & summary of findings tables

02 Framing question & deciding on important outcomes

03 Rating quality of evidence

04 Rating quality of evidence – study limitations (risk of bias)

05 Rating the quality of evidence—publication bias

06 Rating up quality of evidence – imprecision

07 Rating quality of evidence – inconsistency

08 Rating quality of evidence – indirectness

09 Rating up quality of evidence

The GRADE guidance papers focus on the efficacy of treatments and interventions, but in doing so, they evaluate “effects” and are thus applicable to the etiologic issues of alleged harm that find their way into court.  The papers build on other grading systems advanced previously by the Oxford Center for Evidence-Based Medicine, the U.S. Preventive Services Task Force (Agency for Healthcare Research and Quality AHRQ), the Cochrane Collaboration, as well as many individual professional organizations.

GRADE has had some success in harmonizing disparate grading systems, and forging a consensus among organizations that had been using their own systems, such as the  World Health Organization, the American College of Physicians, the American Thoracic Society, the Cochrane Collaboration, the American College of Chest Physicians, the British Medical Journal, and Kaiser Permanente.

There are many other important efforts to provide consensus support for improving the quality of the design, conduct, and reporting of published studies, as well as the interpretation of those studies once published.  Although the RSME 3d does a good job of introducing its readers to the basics of study design, it could have done considerably more to help judges become discerning critics of scientific studies and of conclusions based upon individual or multiple studies.

Historians As Expert Witnesses – A Wiki

October 28th, 2011

“The one duty we owe to history is to rewrite it.”

Oscar Wilde, The Critic As Artist (1891)

“What will history say?  History, sir, will tell lies as usual.”

George Bernard Shaw, The Devil’s Disciple (1901)

* * * * * * * * * * * * * * * * * * * * * * * * *

The Defense Research Institute recently announced that Bill Childs, a professor at the Western New England University School of Law, will be speaking the use of historians as expert witnesses in litigation.  Having puzzled about this very issue in previous writings, I look forward to Professor Childs’ contributions on the issue.  The announcement also noted Professor Childs’ creation, “the Historians as Experts Wiki,” which I knew about, but had not previously visited.

The wiki is a valuable resource of information about historians who have participated in the litigation process in all manner of cases, including art, asbestos, creationism, native Americans, holocaust, products liability, intellectual property, and voting rights.  There are pages for each historian witness, including expert witnesses in other fields, who have given testimony of an explicitly historical nature. The website is still in its formative stages, but it holds great promise as a resource to lawyers who are researching historians who have been listed as expert witnesses in their cases.

Most of my musings about historians as expert witnesses have been provoked by those who have testified about the history of silicosis.  Last year, I presented at a conference sponsored by the International Commission on Occupational Health (ICOH), about such historians.  See “A Walk on the Wild Side,” July 16, 2010.  My presentation abstract, along with all the proceedings of that conference, will be published next year as  “Courting Clio:  Historians and Their Testimony in Products Liability Action,” in: Brian Dolan and Paul Blanc, eds., At Work in the World: Proceedings of the Fourth International Conference on the History of Occupational and Environmental Health, Perspectives in Medical Humanities, University of California Medical Humanities Consortium, University of California Press (2012)(in press).

New Reference Manual on Scientific Evidence Short Shrifts Rule 703

October 16th, 2011

In “RULE OF EVIDENCE 703 — Problem Child of Article VII (Sept. 19, 2011),” I wrote about how Federal Rule of Evidence 703 is generally ignored and misunderstood in current federal practice.  The Supreme Court, in deciding Daubert, shifted the focus to Rule 702, as the primary tool to deploy in admitting, as well as limiting and excluding, expert witness opinion testimony.  The Court’s decision, however, did not erase the need for an additional, independent rule to control the quality of inadmissible materials upon which expert witnesses rely.  Indeed, Rule 702 as amended in 2000, incorporated much of the learning of the Daubert decision, and then some, but it does not address the starting place of any scientific opinion:  the data, the analyses (usually statistical) of data, and the reasonableness of relying upon those data and analyses.  Instead, Rule 702 asks whether the proffered testimony is based upon:

  1. sufficient facts or data,
  2. the product of reliable principles and methods, and
  3. a reliable application of principles and methods to the facts of the case

Noticeably absent from Rule 702, in its current form, is any directive to determine whether the proffered expert witness opinion is based upon facts or data of the sort upon which experts in the pertinent field would reasonably rely.  Furthermore,  Daubert did not address the fulsome importation and disclosure of untrustworthy hearsay opinions through Rule 703.  See Problem Child (discussing the courts’ failure to appreciate the structure of peer-reviewed articles, and the need to ignore the discussion and introduction sections of such articles as often containing speculative opinions and comments).  See also Luciana B. Sollaci & Mauricio G. Pereira, “The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey,” 92 J. Med. Libr. Ass’n 364 (2004); Montori, et al., “Users’ guide to detecting misleading claims in clinical research reports,” 329 Br. Med. J. 1093, 1093 (2004) (advising readers on how to avoid being misled by published literature, and counseling readers to “Read only the Methods and Results sections; bypass the Discuss section.”)  (emphasis added).

Given this background, it is disappointing but not surprising that the new Reference Manual on Scientific Evidence severely slights Rule 703.  Using either a word search in the PDF version or the index at end of book tells the story:  There are five references to Rule 703 in the entire RMSE!  The statistics chapter has an appropriate but fleeting reference:

“Or the study might rest on data of the type not reasonably relied on by statisticians or substantive experts and hence run afoul of Federal Rule of Evidence 703. Often, however, the battle over statistical evidence concerns weight or sufficiency rather than admissibility.”

RMSE 3d at 214. At least this chapter acknowledges, however briefly, the potential problem that Rule 703 poses for expert witnesses.  The chapter on survey research similarly discusses how the data collected in a survey may “run afoul” of Rule 703.  RMSE 3d at 361, 363-364.

The chapter on epidemiology takes a different approach by interpreting Rule 703 as a rule of admissibility of evidence:

“An epidemiologic study that is sufficiently rigorous to justify a conclusion that it is scientifically valid should be admissible,184 as it tends to make an issue in dispute more or less likely.185

Id. at 610.  This view is mistaken.  Sufficient rigor in an epidemiologic study is certainly needed for reliance by an expert witness, but such rigor does not make the study itself admissible; the rigor simply permits the expert witness to rely upon a study that is typically several layers of inadmissible hearsay.  See Reference Manual on Scientific Evidence v3.0 – Disregarding Study Validity in Favor of the “Whole Gamish” (Oct. 14, 2011) (discussing the argument put forward by the epidemiology chapter for considering Rule 703 as an exception to the rule against hearsay).

While the treatment of Rule 703 in the epidemiology chapter is troubling, the introductory chapter on the admissibility of expert witness opinion testimony by the late Professor Margaret Berger really sets the tone and approach for the entire volume. See Berger, “The Admissibility of Expert Testimony,” RSME 3d 11 (2011).  Professor Berger never mentions Rule 703 at all!  Gone and forgotten. The omission is not, however, an oversight.  Rule 703, with its requirement of qualifying each study relied upon as having been “reasonably relied upon,” as measured by what experts in the appropriate discipline, is the refutation of Berger’s argument that somehow a pile of weak, flawed studies, taken together can yield a scientifically reliable conclusion. SeeWhole Gamish,” (Oct. 14th, 2011).

Rule 703 is not merely an invitation to trial judges; it is a requirement to look at the discrete studies relied upon to determine whether the building blocks are sound.  Only then can the methods and procedures of science begin to analyze the entire evidentiary display to yield reliable scientific opinions and conclusions.

Reference Manual on Scientific Evidence v3.0 – Disregarding Study Validity in Favor of the “Whole Gamish”

October 14th, 2011

There is much to digest in the new Reference Manual on Scientific Evidence, third edition (RMSE 3d).  Much of what is covered is solid information on the individual scientific and technical disciplines covered.  Although the information is easily available from other sources, there is some value in collecting the material in a single volume for the convenience of judges.  Of course, given that this information is provided to judges from an ostensibly neutral, credible source, lawyers will naturally focus on what is doubtful or controversial in the RMSE.

I have already noted some preliminary concerns, however, with some of the comments in the Preface, by Judge Kessler and Dr. Kassirer.  See “New Reference Manual’s Uneven Treatment of Conflicts of Interest.”  In addition, there is a good deal of overlap among the chapters on statistics, epidemiology, and medical testimony.  This overlap is at first blush troubling because the RMSE has the potential to confuse and obscure issues by having multiple authors address them inconsistently.  This is an area where reviewers should pay close attention.

From first looks at the RMSE 3d, there is a good deal of equivocation between encouraging judges to look at scientific validity, and discouraging them from any meaningful analysis by emphasizing inaccurate proxies for validity, such as conflicts of interest.  (As I have pointed out, the new RSME did not do quite so well in addressing its own conflicts of interest.  SeeToxicology for Judges – The New Reference Manual on Scientific Evidence (2011).”)

The strengths of the chapter on statistical evidence, updated from the second edition, remain, as do some of the strengths and flaws of the chapter on epidemiology.  I hope to write more about each of these important chapters at a later date.

The late Professor Margaret Berger has an updated version of her chapter from the second edition, “The Admissibility of Expert Testimony,” RSME 3d 11 (2011).  Berger’s chapter has a section criticizing “atomization,” a process she describes pejoratively as a “slicing-and-dicing” approach.  Id. at 19.  Drawing on the publications of Daubert-critic Susan Haack, Berger rejects the notion that courts should examine the reliability of each study independently. Id. at 20 & n. 51 (citing Susan Haack, “An Epistemologist in the Bramble-Bush: At the Supreme Court with Mr. Joiner,” 26 J. Health Pol. Pol’y & L. 217–37 (1999).  Berger contends that the “proper” scientific method, as evidenced by works of the International Agency for Research on Cancer, the Institute of Medicine, the National Institute of Health, the National Research Council, and the National Institute for Environmental Health Sciences, “is to consider all the relevant available scientific evidence, taken as a whole, to determine which conclusion or hypothesis regarding a causal claim is best supported by the body of evidence.” Id. at 19-20 & n.52.  This contention, however, is profoundly misleading.  Of course, scientists undertaking a systematic review should identify all the relevant studies, but some of the “relevant” studies may well be insufficiently reliable (because of internal or external validity issues) to answer the research question at hand. All the cited agencies, and other research organizations and researchers, exclude studies that are fundamentally flawed, whether as a result of bias, confounding, erroneous data analyses, or related problems.  Berger cites no support for the remarkable suggestion that scientists do not make “reliability” judgments about available studies when assessing the “totality of the evidence.”

Professor Berger, who had a distinguished career as a law professor and evidence scholar, died in November 2010.  She was no friend of Daubert, but remarkably her antipathy has outlived her.  Her critical discussion of “atomization” cites the notorious decision in Milward v. Acuity Specialty Products Group, Inc., 639 F.3d 11, 26 (1st Cir. 2011), which was decided four months after her passing. Id. at 20 n.51. (The editors note that the published chapter was Berger’s last revision, with “a few edits to respond to suggestions by reviewers.”)

Professor Berger’s contention about the need to avoid assessments of individual studies in favor of the whole gamish must also be rejected because Federal Rule of Evidence 703 requires that each study considered by an expert witness “qualify” for reasonable reliance by virtue of the study’s containing facts or data that are “of a type reasonably relied upon by experts in the particular field forming opinions or inferences upon the subject.”  One of the deeply troubling aspects of the Milward decision is that it reversed the trial court’s sensible decision to exclude a toxicologist, Dr. Martyn Smith, who outran his headlights on issues having to do with a field in which he was clearly inexperienced – epidemiology.

Scientific studies, and especially epidemiologic studies, involve multiple levels of hearsay.  A typical epidemiologic study may contain hearsay leaps from patient to clinician, to laboratory technicians, to specialists interpreting test results, back to the clinician for a diagnosis, to a nosologist for disease coding, to a national or hospital database, to a researcher querying the database, to a statistician analyzing the data, to a manuscript that details data, analyses, and results, to editors and peer reviewers, back to study authors, and on to publication.  Those leaps do not mean that the final results are untrustworthy, only that the study itself is not likely admissible in evidence.

The inadmissibility of scientific studies is not problematic because Rule 703 permits testifying expert witnesses to formulate opinions based upon facts and data, which are not themselves admissible in evidence. The distinction between relied upon, and admissible, studies is codified in the Federal Rules of Evidence, and in virtually every state’s evidence law.

Referring to studies, without qualification, as admissible in themselves is wrong as a matter of evidence law.  The error has the potential to encourage carelessness in gatekeeping expert witnesses’ opinions for their reliance upon inadmissible studies.  The error is doubly wrong if this approach to expert witness gatekeeping is taken as license to permit expert witnesses to rely upon any marginally relevant study of their choosing.  It is therefore disconcerting that the new Reference Manual on Science Evidence (RMSE 3d) fails to make the appropriate distinction between admissibility of studies and admissibility of expert witness opinion that has reasonably relied upon appropriate studies.

Consider the following statement from the chapter on epidemiology:

“An epidemiologic study that is sufficiently rigorous to justify a conclusion that it is scientifically valid should be admissible,184 as it tends to make an issue in dispute more or less likely.185

RMSE 3d at 610.  Curiously, the authors of this chapter have ignored Professor Berger’s caution against slicing and dicing, and speak to a single study’s ability to justify a conclusion. The authors of the epidemiology chapter seem to be stressing that scientifically valid studies should be admissible.  The footnote emphasizes the point:

See DeLuca v. Merrell Dow Pharms., Inc., 911 F.2d 941, 958 (3d Cir. 1990); cf. Kehm v. Procter & Gamble Co., 580 F. Supp. 890, 902 (N.D. Iowa 1982) (“These [epidemiologic] studies were highly probative on the issue of causation—they all concluded that an association between tampon use and menstrually related TSS [toxic shock syndrome] cases exists.”), aff’d, 724 F.2d 613 (8th Cir. 1984). Hearsay concerns may limit the independent admissibility of the study, but the study could be relied on by an expert in forming an opinion and may be admissible pursuant to Fed. R. Evid. 703 as part of the underlying facts or data relied on by the expert. In Ellis v. International Playtex, Inc., 745 F.2d 292, 303 (4th Cir. 1984), the court concluded that certain epidemiologic studies were admissible despite criticism of the methodology used in the studies. The court held that the claims of bias went to the studies’ weight rather than their admissibility. Cf. Christophersen v. Allied-Signal Corp., 939 F.2d 1106, 1109 (5th Cir. 1991) (“As a general rule, questions relating to the bases and sources of an expert’s opinion affect the weight to be assigned that opinion rather than its admissibility. . . .”).”

RMSE 3d at 610 n.184 (emphasis in bold, added).  This statement, that studies relied upon by an expert in forming an opinion may be admissible pursuant to Rule 703, is unsupported by Rule 703 and the overwhelming weight of case law interpreting and applying the rule.  (Interestingly, the authors of this chapter seem to abandon their suggestion that studies relied upon “might qualify for the learned treatise exception to the hearsay rule, Fed. R. Evid. 803(18), or possibly the catchall exceptions, Fed. R. Evid. 803(24) & 804(5),” which was part of their argument in the Second Edition of the RMSE.  RMSE 2d at 335 (2000).)  See also RMSE 3d at 214 (discussing statistical studies as generally “admissible,” but acknowledging that admissibility may be no more than permission to explain the basis for an expert’s opinion).

The cases cited by the epidemiology chapter, Kehm and Ellis, both involved “factual findings” in public investigative or evaluative reports, which were independently admissible under Federal Rule of Evidence 803(8)(C).  See Ellis, 745 F.2d at 299-303; Kehm, 724 F.2d at 617-18.  As such, the cases hardly support the chapter’s suggestion that Rule 703 is a rule of admissibility for epidemiologic studies.

Here the RMSE, in one sentence, confuses Rule 703 with an exception to the rule against hearsay, which would prevent the statistical studies from being received in evidence.  The point is reasonably clear, however, that the studies “may be offered” to explain an expert witness’s opinion.  Under Rule 705, that offer may also be refused. The offer, however, is to “explain,” not to have the studies admitted in evidence.

The RMSE is certainly not alone in advancing this notion that studies are themselves admissible.  Other well-respected evidence scholars lapse into this position:

“Well conducted studies are uniformly admitted.”

David L. Faigman, et al., Modern Scientific Evidence:  The Law and Science of Expert Testimony v.1, § 23:1,at 206 (2009)

Evidence scholars should not conflate admissibility of the epidemiologic (or other) studies with the ability of an expert witness to advert to a study to explain his or her opinion.  The testifying expert witness really has no need to become a conduit for off-hand comments and opinions in the introduction or discussion section of relied upon articles, and the wholesale admission of such hearsay opinions undermines the court’s control over opinion evidence.  Rule 703 authorizes reasonable reliance upon “facts and data,” not every opinion that creeps into the published literature.

New Reference Manual’s Uneven Treatment of Conflicts of Interest

October 12th, 2011

The new, third edition of the Reference Manual on Scientific Evidence (RMSE) appears to get off to a good start in the Preface by Judge Kessler and Dr. Kassirer, when they note that the Supreme Court mandated federal courts to

“examine the scientific basis of expert testimony to ensure that it meets the same rigorous standard employed by scientific researchers and practitioners outside the courtroom.”

RMSE at xiii.  The preface falters, however, on two key issues, causation and conflicts of interest, which are taken up as an introduction to the new volume.

1. CAUSATION

The authors tell us in squishy terms that causal assessments are judgments:

“Fundamentally, the task is an inferential process of weighing evidence and using judgment to conclude whether or not an effect is the result of some stimulus. Judgment is required even when using sophisticated statistical methods. Such methods can provide powerful evidence of associations between variables, but they cannot prove that a causal relationship exists. Theories of causation (evolution, for example) lose their designation as theories only if the scientific community has rejected alternative theories and accepted the causal relationship as fact. Elements that are often considered in helping to establish a causal relationship include predisposing factors, proximity of a stimulus to its putative outcome, the strength of the stimulus, and the strength of the events in a causal chain.”

RMSE at xiv.

The authors leave the inferential process as a matter of “weighing evidence,” but without saying anything about how the scientific community does its “weighing.”  Language about “proving” causation is also unclear because “proof” in scientific parlance connotes a demonstration, which we typically find in logic or in mathematics.  Proving empirical propositions suggests a bar set too high such that the courts must inevitable lower the bar considerably.  The question is, of course, how low will judges go to admit evidence.

The authors thus introduce hand waving and excuses for why evidence can be weighed differently in court proceedings from the world of science:

“Unfortunately, judges may be in a less favorable position than scientists to make causal assessments. Scientists may delay their decision while they or others gather more data. Judges, on the other hand, must rule on causation based on existing information. Concepts of causation familiar to scientists (no matter what stripe) may not resonate with judges who are asked to rule on general causation (i.e., is a particular stimulus known to produce a particular reaction) or specific causation (i.e., did a particular stimulus cause a particular consequence in a specific instance). In the final analysis, a judge does not have the option of suspending judgment until more information is available, but must decide after considering the best available science.”

RMSE at xiv.  But the “best available science” may be pretty crummy, and the temptation to turn desperation into evidence (“well, it’s the best we have now”) is often severe.  The authors of the Preface signal that “inconclusive” is not a judgment open to judges charged with expert witness gatekeeping.  If the authors truly mean to suggest that judges should go with whatever is dished out as “the best available science,” then they have overlooked the obvious:  Rule 702 opens the door to “scientific, technical, or other specialized knowledge,” not to hunches, suggestive but inconclusive evidence, and wishful thinking about how the science may turn out when further along.  Courts have a choice to exclude expert witness opinion testimony that is based upon incomplete or inconclusive evidence.

2. CONFLICTS OF INTEREST

Surprisingly, given the scope of the scientific areas covered in the RMSE, the authors discuss conflicts of interest (COI) at some length.  Conflicts of interest are a fact of life in all endeavors, and it is understandable counsel judges and juries to try to identify, assess, and control them.  COIs, however, are weak proxies for unreliability.  The emphasis given here is undue because federal judges are misled into thinking that they can discern unreliability from COI, when they should be focused on the data and the analysis.

The authors of the Preface set about to use COI as a basis for giving litigation plaintiffs a pass, and for holding back studies sponsored by corporate defendants.

“Conflict of interest manifests as bias, and given the high stakes and adversarial nature of many courtroom proceedings, bias can have a major influence on evidence, testimony, and decisionmaking. Conflicts of interest take many forms and can be based on religious, social, political, or other personal convictions. The biases that these convictions can induce may range from serious to extreme, but these intrinsic influences and the biases they can induce are difficult to identify. Even individuals with such prejudices may not appreciate that they have them, nor may they realize that their interpretations of scientific issues may be biased by them. Because of these limitations, we consider here only financial conflicts of interest; such conflicts are discoverable. Nonetheless, even though financial conflicts can be identified, having such a conflict, even one involving huge sums of money, does not necessarily mean that a given individual will be biased. Having a financial relationship with a commercial entity produces a conflict of interest, but it does not inevitably evoke bias. In science, financial conflict of interest is often accompanied by disclosure of the relationship, leaving to the public the decision whether the interpretation might be tainted. Needless to say, such an assessment may be difficult. The problem is compounded in scientific publications by obscure ways in which the conflicts are reported and by a lack of disclosure of dollar amounts.

Judges and juries, however, must consider financial conflicts of interest when assessing scientific testimony. The threshold for pursuing the possibility of bias must be low. In some instances, judges have been frustrated in identifying expert witnesses who are free of conflict of interest because entire fields of science seem to be co-opted by payments from industry. Judges must also be aware that the research methods of studies funded specifically for purposes of litigation could favor one of the parties. Though awareness of such financial conflicts in itself is not necessarily predictive of bias, such information should be sought and evaluated as part of the deliberations.”

RMSE at xiv-xv.  All in all, rather misleading advice.  Financial conflicts are not the only conflicts that can be “discovered.”  Often expert witnesses will have political and organizational alignments, which will show deep-seated ideological alignments with the party for which they are testifying.  For instance, in one silicosis case, an expert witness in the field of history of medicine testified, at an examination before trial, that his father suffered from a silica-related disease.  This witness’s alignment with Marxist historians and his identification with radical labor movements made his non-financial conflicts obvious, although these COI would not necessarily have been apparent from his scholarly publications alone.

How low will the bar be set for discovering COI?  If testifying expert witnesses are relying upon textbooks, articles, essays, will federal courts open the authors/hearsay declarants up to searching discovery of their finances?

Also misleading is the suggestion that “entire fields of science seem to be co-opted by payments from industry.”  Do the authors mean to exclude the plaintiffs’ lawyer litigation industry, which has grown so large and politically powerful in this country?  In litigations in which I have been involved, I have certainly seen plaintiffs’ counsel, or their proxies – labor unions or “victim support groups” provide substantial funding for studies.  The Preface authors themselves show an untoward bias by their pointing out industry payments without giving balanced attention to other interested parties’ funding of scientific studies.

The attention to COI is also surprising given that one of the key chapters, for toxic tort practitioners, was written by Dr. Bernard D. Goldstein, who has testified in toxic tort cases, mostly (but not exclusively) for plaintiffs.  See, e.g., Parker v. Mobil Oil Corp., 7 N.Y.3d 434, 857 N.E.2d 1114, 824 N.Y.S.2d 584 (2006); Exxon Corp. v. Makofski, 116 SW 3d 176 (Tex. Ct. App. 2003).  The Makofsky case is particularly interesting because Dr. Goldstein was forced to explain why he was willing to opine that benzene caused acute lymphocytic leukemia, despite the plethora of published studies finding no statistically significant relationship.  Dr. Goldstein resorted to the inaccurate notion that scientific “proof” of causation requires 95 percent certainty, whereas he imposed only a 51 percent certainty for his medico-legal testimonial adventures. Dr. Goldstein also attempted to justify the discrepancy from the published literature by adverting to the lower standards used by federal regulatory agencies and treating physicians. Id.

These explanations are particularly concerning because they reflect basic errors in statistics and in causal reasoning.  The 95 percent derives from the use of the same percentage in confidence intervals, but the probability involved there is not the probability of the association’s being correct, and it has nothing to do with the probability in the belief that an association is real or is causal.  (Thankfully the RMSE chapter on statistics gets this right, but my fear is that judges will skip over the more demanding chapter on statistics and place undue weight on the toxicology chapter, written by Dr. Goldstein.)  The reference to federal agencies (OSHA, EPA, etc.) and to treating physicians was meant, no doubt, to invoke precautionary principle concepts as a justification for some vague, ill-defined, lower standard of causal assessment.

The Preface authors might well have taken their own counsel and conducted a more searching assessment of COI among authors of Reference Manual.  Better yet, the authors might have focused the judiciary on the data and the analysis.

Diluting “Reasonable Degree of Medical Certainty” – An AAJ-Restatement “Tool” to Help Plaintiffs

October 3rd, 2011

In “the Top Reason that the ALI’s Restatement of Torts Should Steer Clear of Partisan Conflicts,” I pointed out the inappropriateness of advertising the ALI’s Restatement of Torts to the organized plaintiffs’ bar, much as the plaintiffs’ bar advertises potential huge recoveries for the latest tort du jour.  See Michael D. Green & Larry S. Stewart, “The New Restatement’s Top 10 Tort Tools,” Trial 44 (April 2010).

Some of the authors’ tort tool kit may be unexceptionable.  Among these authors’ top ten tort tools, however, is the new Restatement’s edict that “reasonable degree of medical certainty” means, or should mean, nothing more than saying “more likely than not.”  The authors criticize the reasonable certainty standard with an abbreviated rendition of the Restatement’s critique:

“Many courts hold that expert opinion must be expressed in terms of medical or scientific certainty’. Requiring certainty seems to impose a criminal law-like burden of proof that is inconsistent with civil burdens of preponderance of the evidence to establish a fact. Such a requirement is also problematic at best because medical and scientific communities have no such ‘reasonable certainty’ standard. The standard then becomes whatever the attorney who hired the expert tells the expert it means or, absent that, whatever the expert imagines it means. Section 28, comment e, of the Restatement criticizes this standard and makes clear that the same preponderance standard (or ‘more likely than not’ standard), which is universally applied in all aspects of civil cases, also applies to expert testimony.”

Id. at 46-47.

Well, the more likely than not standard is not “universally applied in all aspects of civil cases,” because several states require exemplary damages to be proven by “clear and convincing” or greater evidence.  In some states, the burden of proof in fraud cases is higher than a mere preponderance of the evidence. This premise of the authors’ article is incorrect.

But even if the authors were correct that the preponderance standard applied “in all aspects” of civil cases, their scholarship would remain suspect, as others and I have previously pointed out.  SeeReasonable Degree of Medical Certainty,” and “More Uncertainty About Reasonable Degree of Medical Certainty.”

1. The Restatement’s Treatment of Expert Witness Evidentiary Rules Exceeded the Scope of the Tort Restatement.

The most peculiar aspect of this “top tool,” is that it has nothing to do with the law of torts.  The level of certitude required of an expert witness is an evidentiary and a procedural issue. Of course the issue comes up in tort cases, which frequently involve medical and scientific causation opinions, as well as other expert witness opinions.  The issue, however, comes up in all cases that involve expert witnesses:  trust and estates, regulatory, environmental, securities fraud, commercial, and other cases.

The Restatement of Torts weakly acknowledges its frolic and detour in treating a procedural issue concerning the admissibility of expert witness opinion testimony, by noting that it does “not address any other requirements for the admissibility of an expert witness’s testimony, including qualifications, expertise, investigation, methodology, or reasoning.” Restatement (Third) of Torts: Liability for Physical and Emotional Harm § 28, cmt. e (2010).  The certitude issue has nothing special to do with the substantive law of torts, and should not have been addressed in the torts restatement.

2. The Restatement’s Treatment of “Reasonable Degree of Medical Certainty” Has No Relevance to the Burden of Proof in Tort Cases.

The expert witness certitude issue has nothing to do with the burden of proof, and the Restatement should not have confused and conflated the burden of proof with the standard of certitude for expert witnesses.  The clear but unacceptable implication is that expert witnesses in criminal cases must testify to certitude “beyond a reasonable doubt,” and in claims for equitable relief, expert witnesses may share only opinions that are made, in their minds, by “clear and convincing evidence.”  There is no support in law or logic for the identification of witness certitude with parties’ burdens of proof.

Comment e states the critique more fully:

“If courts do interpret the reasonable-certainty standard to require a level of certitude greater than the preponderance-of-the-evidence standard requires, this creates a troubling inconsistency between standards for the admissibility of evidence and the threshold required for sufficiency of proof. The threshold for admissibility should not be higher than the threshold to sufficiency.  Moreover, the reasonable-certainty standard provides no assurance of the quality of the expert’s qualifications, expertise, investigation, methodology, or reasoning.  Thus, the Section adopts the same preponderance standard that is universally adopted in civil cases.  Direct and cross-examination can be employed to flesh out the degree of certainty with which an expert’s opinion is held and to identify opinions that are speculative and therefore inadmissible.”

Id. The critique badly misfires because there is no inconsistency and no trouble in having different standards for the admissibility of opinion evidence and the burden of proof.  As noted, expert witnesses testify on causation and other issues in criminal, equity, and tort cases, all with different burdens of proof.  Juries in criminal and tort cases must apply instructions on burdens of proof to an entire evidentiary display, not just the expert witnesses’ opinions.  In logic and law, there ultimately must be different burdens for admissibility of expert witness testimony and for sufficiency of a party’s proofs.

3. The Restatement’s Treatment of “Reasonable Degree of Medical Certainty” Incoherently Confuses Two Different Standards.

We can see that Comment e’s approach to legislating an equivalence between expert witness certitude and the burden must fail even on its own terms.  Consider the legal consequences of tort claimants, with the burden of proof, who produce expert witnesses to opine about key elements (e.g., causation) of torts by stating that their opinions were held by a mere “preponderance of the evidence.”

If this probability is understood to be only infinitesimally greater than 50%, then courts would have to direct verdicts in many (and perhaps most) cases.

Courts must ensure that a rational jury can find for the party with the burden of proof.  Juries must evaluate the credibility and reliability of expert witnesses, their opinions, as well as the predicate facts for those opinions.  If those expert witness opinions were barely greater than 50% probable on an essential element, then unless the witnesses had perfect credibility, and all predicate facts were as probable as claimed by the witnesses, then juries would frequently have to reject the witnesses’ opinions.  The bare preponderance of the expert witnesses’ opinions would result in an overall probability of the essential element less than 50%.

4. The Restatement Incorrectly Implies that Expert Witnesses Can Quantify Their Opinions in Probabilistic Terms.

There are even more far-reaching problems with simply substituting “more likely than not” for RDMC as a threshold requirement of expert witness testimony.  Comment e implies that expert witnesses can discern the difference between an opinion that they believe is “more likely than not” and another which is “as likely as not.” On some occasions, there may be opinions that derive from quantitative reasoning, for which an expert witness could truly say, with some level of certainty, that his or her opinion is “more likely than not.” On most occasions, an expert witness’s degree of certainty is a qualitative opinion that simply does not admit of a quantitative characterization. The Restatement’s comment perpetuates this confusion by casting the reasonable certainty standard as a bare probability.

Comment e further suggests that expert witnesses are themselves expert in assessing their own level of certainty, and that they have the training and experience to distinguish an opinion that is 50.1% likely from another that is only 50% likely. The assignment of precise mathematical probabilities to personal, subjective beliefs is a doubtful exercise, at best. See, e.g., Daniel Kahneman and Amos Tversky, “Judgment under Uncertainty: Heuristics and Biases,” 185 Science 1124 (1974).

5. The Restatement Incorrectly Labels “Reasonable Degree of Medical Certainty” As An Empty Formalism.

Comment e ignores the epistemic content of reasonable certainty, which bears an uncanny resemblance to the knowledge requirement of Rule 702.  The “mantra” is helpful to the extent it imposes an objective epistemic standard, especially in states that have failed to impose, or that have abrogated, expert witness gatekeeping.  In some states, there is no meaningful expert witness gatekeeping under either the Frye standard or Rule 702. See, e.g., “Expert Evidence Free-for-All in Washington State.”  See also Joseph Sanders, “Science, Law, and the Expert Witness,” 72 Law & Contemporary Problems 63, 87 & n. 118 (2009) (noting that the meaning of “reasonable degree of scientific certainty” is unclear, but that it can be understood as an alternative formulation of Kumho’s “same intellectual rigor” test).

Some of these “top” tools may be defective.  The authors may need good defense counsel.

A New Day – A New Edition of the Reference Manual of Scientific Evidence

September 28th, 2011

It’s an event that happens about once every ten years – a new edition of the Reference Manual on Scientific Evidence.  This sort of thing gets science-geek-nerd lawyers excited.  Today, the National Academies of Science released the new, third edition of the Manual today.  The work has grown from the second edition into a hefty volume, now over 1,000 pages. Shorter than War and Peace, but easier to read than Gravity’s Rainbow.  Paperback volumes are available for $71.96, but a PDF file is available for free.

Unlike the first two editions, which were products of the Federal Judicial Center, the new edition was produced under the supervision of an ad hoc committee – the Committee on the Development of the Third Edition of the Reference Manual on Scientific Evidence – under the auspices of the National Academies’ Committee on Science, Technology and the Law.  See Media Release from the National Academies.

The release of the Third Edition of the Manual was accompanied by a ceremony today at the National Academies’ Keck Center in Washington, DC.   Dr. Cicerone, President of the National Academies, and Judge Barbara Rothstein, director of the Federal Judicial Center, gave inspirational talks on the rightful, triumphant, aspirational role of science in the courtroom, and the Manual’s role of ensuring that science gets its day in the adversarial push and shove of litigation.

Co-chairs, Judge Kessler and Dr. Kassirer introduced the other members of the ad hoc committee, and the substantive work that makes up the Third Edition.

Other members of the ad hoc committee were:

  • Hon. Ming Chin, Associate Justice of the Supreme Court of California.
  • Hon. Pauline Newman, Judge, U.S. Court of Appeals for the Federal Circuit.
  • Hon. Kathleen O’Malley, Judge, U.S. Court of Appeals for the Federal Circuit.
  • Hon. Jed S. Rakoff, Judge, U.S. District Court for the Southern District of New York.
  • Channing R. Robertson, Ph.D., School of Engineering, Stanford University.
  • Joseph V. Rodricks, Ph.D., Environ Corp.
  • Allen J. Wilcox, Ph.D., Senior Investigator in the Epidemiology Branch, of the NIEHS.
  • Sandy L. Zabell, Professor of Statistics and Mathematics, Northwestern University.

The Third Edition has some notable new chapters on “Forensic Identification Expertise” (Paul Giannelli, Edward Imwinkelried, and Joseph Peterson), on “Neuroscience” (Henry Greely and Anthony Wagner), on “Mental Health Evidence” (Paul Appelbaum), and on “Exposure Science” (Joseph Rodricks).

Other chapters that were present in the Second Edition are substantial revised and updated, including the chapters on “Statistics” (David Kaye and the late David Freedman), on “DNA Identification Evidence” (David Kaye and George Sensabaugh), on “Epidemiology” (Michael Green, Michal Freedman, and Leon Gordis), and on “Engineering” (Channing R. Robertson, John E. Moalli, and David L. Black).

The chapter on “Medical Testimony” (John Wong, Lawrence Gostin, and Oscar Cabrera) is also substantially revised and expanded, with new authors, all for a welcomed change.

A substantial portion of the new addition incorporates chapters from the second addition with little or no change:

Justice Stephen Breyer’s “Introduction,” the late Professor Margaret Berger’s essay on “The Admissibility of Expert Testimony,” David Goodstein’s primer on “How Science Works,” as well the chapters on “Multiple Regression” (Daniel L. Rubinfeld), on “Survey Research” (Shari Diamond), and on “Toxicology” (Bernard Goldstein and Mary Sue Henifin).

Also new in this edition is the support of private organizations, the Starr Foundation and the Carnegie Corporation of New York.

Judge Kessler explained how the Supreme Court’s decision in Daubert introduced “rigorous standards” for scientific evidence in federal courtrooms.  Expert witnesses must follow the reasoning, principles, and procedures of modern science.  Lovely sentiments, and wonderful if heeded.

The public release ceremony ended with questions from the audience, both live and from the webcast.

To provide some comic relief to the serious encomia to science in the law, Dr. Tee Guidotti rose to give a deadpan imitation of Emily Litella.  Dr. Guidotti started by lecturing the gathering on the limitations of frequentist statistics and the need to accommodate Bayesian statistical analyses in medical testimony.  Dr. Kassirer politely interrupted Dr. Guidotti to point out that Bayesian analyses were covered in some detail in the chapters on statistics and on medical testimony.  Without missing a beat, Dr. Guidotti shifted blame to the index, which he claimed failed to identify these discussions for him.  Judge Kessler delivered the coup de grace by pointing out that the discussions of Bayesian statistics were amply referenced in the index.  Ooops; this was a hot court.  A chorus of sotto voce “never minds” rippled through the audience.

One question not asked was whether there are mandatory minimum sentences for judges who fail to bother to read the relevant sections before opining on science issues.

The Third Edition of the Manual appears to be a substantial work.  Lawyers will ignore this work at their peril.  Trial judges of course have immunity except to the extent appellate judges are paying attention.  Like its predecessors, the new edition is likely to have a few quibbles, quirks, and quarrels lurking in the 1,000 plus pages here, and I am sure that the scrutiny of the bar and the academy will find them.  Let’s hope that it takes fewer than a decade to see them corrected in a Fourth Edition.

Expert Evidence Free-for-All in Washington State

September 23rd, 2011

Daubert/Frye issues are fact specific. Meaningful commentary about expert witness decisions requires a close familiarity with the facts and data in the case under scrutiny.  A recent case in point comes from the Washington Supreme Court.   The plaintiff alleged that her child was born with birth defects as a result of her workplace exposure to solvents from mixing paints.  The trial court dismissed the case on summary judgment, after excluding plaintiff’s expert witnesses’ causation opinions. On appeal, the Court, en banc, reversed the summary judgment, and remanded for trail.  Anderson v. Akzo Nobel Coatings Inc., No. 82264-6, Wash. Sup.; 2011 Wash. LEXIS 669 (Sept. 8, 2011).

Anderson worked for Akzo Nobel Coatings, Inc., until the time she was fired, which occurred shortly after she filed a safety complaint.  Her last position was plant environmental coordinator for health and safety. Her job occasionally required her to mix paints.  Akzo’s safety policies required respirator usage when mixing paints, although Anderson claimed that enforcement was lax.  Slip op. at 2.  Anderson gave birth to a son, who was diagnosed with congenital nervous and renal system defects.  Id. at 3.

Anderson apparently had two expert witnesses:  one of her child’s treating physicians and Dr. Khattak, an author of an epidemiologic study on birth defects in women exposed to organic solvents. Sohail Khattak, et. al., “Pregnancy Outcome Following Gestational Exposure to Organic Solvents,” 281 J. Am. Med. Ass’n 1106 (1999). See Slip op. at 3.

The conclusions of the published paper were modest, and no claim to causality was made from either the study alone or from the study combined with the prior knowledge in the field.  When the author, Dr. Khattak donned the mantle of expert witness, intellectual modest went out the door:  He opined that the association was causal.  The treating physician echoed Dr. Khattak’s causal opinion.

The fact-specific nature of the decision makes it difficult to assess the accuracy or validity of the plaintiff’s expert witnesses’ opinions.  The claimed teratogenicity of paint solvents is an interesting issue, but I confess it is one with which I am not familiar.  Perhaps others will address the claim.  Regardless whether or not the claim has scientific merit, the Anderson decision is itself seriously defective.  The Washington Supreme Court’s opinion shows that it did little to familiarize itself with the factual issue, and holds that judges need not tax themselves very much to understand the application of scientific principles to the facts and data of their cases.  Indeed, what is disturbing about this decision is that it sets the bar so low for medical causation claims. Although Anderson does not mark a reversion to the old Ferebee standard, which would allow any qualified, willing expert witness to testify to any conclusion, the decision does appear to permit any opinion based upon a generally accepted methodology, without gatekeeping analysis of whether the expert has actually faithfully and appropriately applied the claimed methodology.  The decision eschews the three subparts of Federal Rule of Evidence 702, which requires that the proffered opinion:

(1) … is based upon sufficient facts or data,

(2) … is the product of reliable principles and methods, and

(3) …[is the product of the application of] the principles and methods reliably to the facts of the case.

Federal Rule of Evidence 702.

In abrogating standards for expert witness opinion testimony, the Washington Supreme Court manages to commit several important errors about the nature of scientific and medical testimony.  These errors are much more serious than any possible denial of intellectual due process in the Anderson case because they virtually ensure that meaningful gatekeeping will not take place in future Washington state court cases.

I. The Court Confused Significance Probability with Expert Witnesses’ Subjective Assessment of Posterior Probability

The Washington Supreme Court advances two grounds for abrogating gatekeeping in medical causation cases.  First, the Court mistakenly states that the degree of certainty for scientific propositions is greater in the scientific world than it is in a civil proceeding:

“Generally, the degree of certainty required for general acceptance in the scientific community is much higher than the concept of probability used in civil courts.  While the standard of persuasion in criminal cases is “beyond a reasonable doubt,” the standard in most civil cases is a mere “preponderance.”

Id. at 14.  No citation is provided for the proposition that the scientific degree of certainty is “much higher,” other than a misleading reference to a book by Marcia Angell, former editor of the New England Journal of Medicine:

“By contrast, “[f]or a scientific finding to be accepted, it is customary to require a 95 percent probability that it is not due to chance alone.”  Marcia Angell, M.D., Science on Trial: The Clash of Medical Evidence and the Law in the Breast Implant Case 114 (1996).  The difference in degree of confidence to satisfy the Frye “general acceptance” standard and the substantially lower standard of “preponderance” required for admissibility in civil matters has been referred to as “comparing apples to oranges.” Id. To require the exacting level of scientific certainty to support opinions on causation would, in effect, change the standard for opinion testimony in civil cases.”

Id. at 15.  This popular press book hardly supports the Court’s contention. The only charitable interpretation of the 95% probability is that the Court, through Dr. Angell, is taking an acceptable rate of false positive errors to be no more than the customary 5%, and is looking at a confidence interval based upon this specified error rate of 1 – α. This error rate, however, is not the probability that the null hypothesis is true.  If the Court would have read the very next sentence, after the first sentence it quotes from Dr. Angell, it would have seen:

“(I am here giving a shorthand version of a much more complicated statistical concept.)”

Science on Trial at 114 (1996).  The Court failed to note that Dr. Angell was talking about significance probability, which is used to assess the strength of the evidence in a single study against the null hypothesis of no association.  Dr. Angell was well aware that she was simplifying the meaning of significance probability in order to distinguish it from a totally different concept, the probability of attribution of a specific case to a known cause of the disease.  It is the probability of attribution that has some relevance to the Court’s preponderance standard; and the probability of attribution standard is not different from the civil preponderance standard.

The Court’s citation of Dr. Angell for the proposition that the “degree of confidence” and the “preponderance” standard are like “comparing apples to oranges,” is a complete distortion of Dr. Angell’s book.  She is comparing the attributable risk based upon an effect size – the relative risk, which need be only greater than 50% for specific causation, with a significance probability for the interpretation of the data from a single, based upon the assumption of the null hypothesis:

“Comparing the size of an effect with the probability that a given finding isn’t due to chance is comparing apples and oranges.”

Id. This statement is a far cry from the Court’s misleading paraphrase, and is no support at all for the Court’s statistical solecism. Implicit in the Court’s error is its commission of the transpositional fallacy; it has confused significance probability (the probability of the evidence given the null hypothesis) with Bayesian posterior probabilities (the probability of the null hypothesis given all the data and evidence in the case).

Having misunderstood significance probability to be at odds with the preponderance standard, the Court notes that the “absence of a statistically significant basis” for an expert witness’s opinion does not implicate Frye or render the expert witness’s opinion inadmissible.  Id. at 16.  In the Anderson case, this musing is pure dictum because Dr. Khattak’s study showed a highly statistically significant difference in the rate of birth defects among women with solvent exposures compared with women without such exposures.

II.  The Court Abandons Evidence or Data as Necessary to Support Judgments of Causality

The Anderson Court did not stop with its misguided distinction between burdens of proof in science and in law.  The Court went on to offer the remarkable suggestion that gatekeeping is unnecessary for medical opinions because they are not, in any event, evidence-based:

“Many expert medical opinions are pure opinions and are based on experience and training rather than scientific data.  We only require that ‘medical expert testimony . . . be based upon ‘a reasonable degree of medical certainty’ or probability.”

Slip op. at16 -17 (internal citations omitted).  There may be some opinions that are experientially based, but the Court did not, and could not, adduce any support for the proposition that judgments of teratogenic causation do not require scientific data.  Troublingly, the Court appears to allow medical expert opinions to be “pure opinions,” unsupported by empirical, scientific data.

Presumably as an example of non-evidence based medical opinions, the Anderson Court offers the example of differential diagnosis:

“Many medical opinions on causation are based upon differential diagnoses. A physician or other qualified expert may base a conclusion about causation through a process of ruling out potential causes with due consideration to temporal factors, such as events and the onset of symptoms.”

Id. at 17. This example, however, does not explain or justify anything the Court  claimed.  Differential diagnoses, or more accurately “differential etiology,” is a process of reasoning by iterative disjunctive syllogism to the most likely cause of a particular patient’s disease.  The syllogism assumes that any disjunct – possible cause of this specific case – has previously, independently been shown to be capable of causing the outcome in question.  There is no known methodology by which this syllogism itself can show general causation.

Not surprisingly, the Court makes no attempt to support its mistaken claim that differential diagnosis permits the assessment of general causation without the necessity of “scientific data.”

The Court’s confusion between significance probability (1 – α)% and posterior probability based upon all the evidence, as well as its confusion between differential diagnosis and evidence-based assessments of general causation, allowed the Court to take a short way with medical causation evidence.  The denial of scientific due process followed inevitably.

III.  The Court Abandoned All Gatekeeping for Expert Witness Opinion Testimony

The Anderson Court suggested that gatekeeping was required by Washington’s continued adherence to the stringent Frye test, but the Court then created an exception bigger than the rule:

“Once a methodology is accepted in the scientific community, then application of the science to a particular case is a matter of weight and admissibility under ER 702, the Frye test is only implicated where the opinion offered is based upon novel science.  It applies where either the theory and technique or method of arriving at the data relied upon is so novel that it is not generally accepted by the relevant scientific community.  There is nothing novel about the theory that organic solvent exposure may cause brain damage and encephalopathy.  See, e.g., Berry v. CSX Transp., Inc., 709 So. 2d 552, 568 & n.12, 571-72 (Fla. Dist. Ct. App. 1998) (surveying medical literature). Nor does it appear that there is anything novel about the methods of the study about which Dr. Khattak wrote. Khattak, supra, at 1106. Frye does not require that the specific conclusions drawn from the scientific data upon which Dr. Khatta relied be generally accepted in the scientific community.  Frye does not require every deduction drawn from generally accepted theories to be generally accepted.”

Slip op. at 18-19 (internal citations omitted).

By excepting the specific inferences and conclusions from judicial review, the Court has sanctioned any nonsense as long as the expert witness can proclaim that he used the methods of “toxicology,” or of “epidemiology,” or some other generally accepted branch of science.  The Court left no room to challenge whether the claim is correct at any other than the most general level.  The studies cited in support of a causation may completely lack internal or external validity, but if they are of a class of studies that are “scientific,” and purport to use a method that is generally accepted (e.g., cohort or case-control studies), then the inquiry is over. Indeed, the Court left no room at all for challenges to expert witnesses who give dubious opinions about medical causation.

IV. Fault Issues

Not content to banish science from the judicial assessment of scientific causality judgments, the Anderson Court went further to take away any defense based upon the mother’s fault in engaging in unprotected mixing of paints while pregnant, or the mother’s fault in smoking while pregnant.   Slip op. at 20.  Suing the mother as a tortfeasor may not be an attractive litigation option to the defendant in a case arising out of workplace exposure to an alleged teratogen, but clearly the mother could be at fault with respect to the causation of her child’s harm. She was in charge of environmental health and safety, and she may well have been aware of the hazards of solvent exposures.  In this case, there were grounds to assert the mother’s fault both in failing to comply with workplace safety rules, and in smoking during her pregnancy (assuming that there was evidence, at the same level as paint fumes, for the teratogenicity of smoking).

RULE OF EVIDENCE 703 — Problem Child of Article VII

September 19th, 2011

With the exception of a few evidence scholars, Federal Rule of Evidence 703 is ignored or misunderstood in practice.  There was a time when virtually every motion to exclude an expert witness’s opinion was framed on Rule 703 concepts, either alone, or in conjunction with Rule 702 requirements.  The Supreme Court’s decision in Daubert changed practice by holding that Rule 702 required gatekeeping, and by generally slighting Rule 703.

I.  Reform of the Common Law

Federal Rule of Evidence 703 formally abandoned the common-law requirement that expert witnesses base their opinions upon evidence of record, either personal observations or facts admitted into evidence.  The first sentence of Rule 703, which has remained unchanged since its original adoption, makes clear that an expert witness may rely upon facts or data that are never admitted into evidence.  This sentence details three methods of putting “facts or data” before expert witnesses.  First, expert witnesses may themselves be percipient witnesses to the facts or data upon which they rely.  Second, expert witnesses may learn of facts or data at the trial by observing other witnesses testify or by being asked to assume facts or data for purposes of giving an opinion.  Third, expert witnesses may come to learn of “facts or data” before the hearing.  It is this third method that represents a departure from the common law, and which raises the issue whether the expert witness has relied upon facts or data, which are themselves inadmissible.

The rationale for Rule 703 was the recognition that much of the expert witness’s understanding of an area of science, medicine, or technology was governed by training, prior experience, professional collaborations, and extensive reading, all of which represented the basis, often in large part, of the case-specific opinions that are then offered in the courtroom.  These bases are mostly hearsay, and mostly inadmissible if expert witnesses were to try to articulate any particular aspect of their personal learning.  The rationale for Rule 703, however, also included the economy and convenience of presenting expert testimony without the need of formal proof of predicate “facts or data,” at least if those facts or data were of the type reasonably relied upon by experts in the relevant field.  Not surprisingly, advocates responded by using Rule 703 to inject all manner of hearsay into their trials, including opinion testimony from witnesses that would never testify at trial.  Courts and commentators responded with confusion over whether Rule 703 created a new exception to the rule against hearsay.

II.  Conduit for Inadmissible Evidence

Much academic, judicial, and professional criticism of Rule 703, before its amendment in 2000, centered on the mischief created by expert witnesses’ reliance upon inadmissible evidence and the disclosure of this information to the jury.  To be sure, Federal Rule 705 made clear that the expert witness need not disclose any basis; the expert opinion could be elicited as a conclusory opinion, or the expert could disclose some but not all bases.  Parties, however, were often intent to use Rule 703 to present, at least selectively, those relied upon facts and data (and sometimes opinions) that would aid their case, regardless of the admissibility of the disclosed expert witness bases.  If the other side were foolish enough to request a limiting instruction, the proponent would revel in the emphasis that the Court gave to their inadmissible facts and data.[i]

Of course, the presentation of expert opinion without requiring disclosure of bases is hardly calculated to permit jurors or trial judges to assess the validity or correctness of the opinions that they must weigh at trial.  Furthermore, Rule 703 shifted the burden to opposing counsel to elicit bases in order to show flaws or weaknesses in reasoning and inference.  This crossexamination frequently could not take place without eliciting inadmissible evidence.

In 2000, Rule 703 was amended to include its third, last sentence, which creates a presumption against disclosure of inadmissible facts or data to the jury.  The presumption against disclosure may be overcome by a judicial finding that the probative value in helping the jury evaluate the opinion is outweighed by the prejudice of injecting inadmissible evidence into the trial.  Nothing in the revised rule makes the inadmissible “facts or data” admissible, although at one point, the Advisory Committee Notes confuse admissibility and disclosure when it writes in terms of relied upon information that is “admissible only for the purpose of assisting the jury in evaluating an expert’s opinion.”  Such evidence is not admissible at all, which is exactly why the presumption is against disclosure and the alternative is disclosure, along with consideration of a limiting instruction.

III.  Expert Witness Opinions – Castles in the Air

Whether underlying facts are disclosed or not, Rule 703, as currently applied in federal courts, raises serious concerns about whether expert witness opinion testimony has a reliable foundation.  The law in most states is that an expert witness’s opinion can rise no higher than the facts upon which the opinion is based.  If the jury does not hear the bases of the opinion, it cannot meaningfully evaluate the opinion.  Furthermore, the jury cannot make sense of an expert witness’s opinion, when it is bound by a limiting instruction, which explains that it may consider the basis in evaluating the expert witness’s opinion, but it may not consider the basis as evidence that has been established in the case. If this basis is not otherwise established in the case, then the jury would be compelled to reject the testimony as unsupported by facts or data in the case.  If the jury must consider the opinion because the expert witness claims to have relied reasonably upon inadmissible “facts or data,” then the expert witness has been given important fact-finding power in the case.

Perhaps Rule 702, with its imposition of gatekeeping responsibilities upon the trial court, is supposed to solve this problem.  Many of the Circuits appear to be moving toward a requirement of pretrial hearings for Rule 702 challenges, at least when requested, and sometimes even when not.  In some instances, the lack of a proper factual predicate, or unreasonableness in reliance upon an inadmissible factual predicate, can be developed in a pretrial hearing that allows the parties to join issue over the reasonableness of reliance and proof of the predicate facts or data.

IV.  Who Decides Reasonable Reliance?

Some of the earlier case law suggested that the expert witness could validate his or her own reliance upon “facts or data,” as “reasonable.”[ii] Judges, like most people, glibly assumed that what people normally or customarily do is reasonable.  Extending this assumption to the law of expert witnesses, courts have equated the reasonable reliance of Rule 703 with what experts customarily do in their field.[iii] Other courts appeared to go further, especially in the context of forensic expert witness opinion, to equate reasonable reliance with what experts do in their courtroom testimony.

The current view, influenced no doubt by the Supreme Court’s holdings in Daubert, Joiner, and Kumho Tire, has settled on requiring the trial court to make an independent assessment, based upon a factual showing, that the “facts or data” in question may be reasonably relied upon by experts in the relevant field.[iv] One of the important implications of this shift is that courts may now accept an expert witness’s testimony about what he normally does, but if opposing counsel challenges the reasonableness of the practice, with affidavits, testimony, learned treatises, and the like, then the court will be required to make a preliminary determination of the reasonableness of the expert’s “normal practice.”  Given that litigation often involves unusual situations outside both the statistical and prescriptive “norms” of ordinary life, the abandonment of extreme deference to expert witnesses as the ultimate arbiters of reasonableness is a significant advance in the evolution of the Federal Rules of Evidence.

V.  Reasonable Reliance and Reliability:  The Intersection Between Rules 702 and 703

Some of the early enthusiasm for Rule 703 as a speed bump for unreliable expert witness testimony came from the explicit use of the concept of “reasonable reliance” in the second sentence of the Rule.  The original Advisory Committee Note encouraged this view by giving an example, without much analysis, of an accident reconstruction expert whose testimony would not be reasonably based upon the statements of bystanders.  Before the advent of Daubert, this example was a tease to lawyers who were looking for some way to limit the flood of unreliable expert witness opinion testimony.  The Advisors, however, did not explain why such reliance would be unreasonable.  We could certainly imagine situations in which bystanders’ statements were essential to recreating an accident.  Furthermore, the statements of bystanders might be admissible under various exceptions to the rule against hearsay, and the Note thus seems to contradict the actual language of the Rule, which limits the reasonableness requirement to reliance upon inadmissible evidence.  In any event, the Advisory Committee’s example of an “accidentologist” seemed to imply a requirement of trustworthiness, which might apply to both admissible and inadmissible “facts or data.”

Perhaps because of the original Advisory Committee Note, litigants, in challenging the reliability of expert witness opinion testimony, frequently invoked both Rules 702 and 703 in support of exclusion.  Indeed, cases that focus on only Rule 703 are relatively uncommon; most cases note that they are addressing motions to bar expert witnesses, made under both rules.  After the Supreme Court’s Quartet on Rule 702 (Daubert, Joiner, Kumho, and Weisgram), the need to frame an exclusionary motion on Rule 703 has been largely dispelled.

One case that gave rise to much of the enthusiasm for Rule 703 as a basis for expert witness preclusion was Judge Weinstein’s decision in In re Agent Orange.[v] Some of the expert witnesses in the Agent Orange litigation relied upon checklists of symptoms prepared by the litigants.  Invoking Rule 703 to support exclusion of the expert witnesses’ opinions, the trial court observed that “no reputable physician relies on hearsay checklists by litigants to reach a conclusion with respect to the cause of their affliction.”[vi]

The lesson of Agent Orange was that Rule 703 could serve as a basis for excluding expert witness testimony.  If the expert witness relied unreasonably upon “facts or data,” then that expert witness’s testimony was fatally flawed under the Rules and had to be excluded.  The Court in Agent Orange avoided the obvious conclusion that an expert witness’s opinion, which was not reasonably based upon “facts or data,” could not be helpful to the trier of fact, and thus the opinion would necessarily offend Rule 702, as well.

Practitioners, faced with dubious expert witness opinion testimony after Agent Orange, increasingly relied upon Rule 703, along with Rules 702 and 403, in stating their challenges to proffered opinions.  Many Courts, in ruling upon these challenges, did not separate out their holdings or reasoning, in applying Rules 702 and 703 to exclude opinions.[vii] Some courts, especially before the Supreme Court’s decision in Daubert, framed reliability challenges almost exclusively in terms of compliance with Rule 703.[viii]

The early enthusiasm for an expansive role for Rule 703 as a tool for broad gatekeeping was problematic from the beginning.  Rule 703 has always required “reasonableness” for an expert witness’s reliance upon inadmissible “facts or data.”  The Rule is, and has always been, silent about reliance upon admissible “facts or data.”  As a result, Rule 703 could never have aspired to the principal role of limiting the flow of unreliable expert testimony.  The Rules of Evidence provide ample bases for expert witnesses to formulate unreliable opinions based solely, and unreasonably, upon admissible “facts or data,” such as inadvertent and false party admissions, self-serving statements made to examining physicians, or vanity press publications elevated to “learned treatise” status.  The resulting opinions have little or no epistemic warrant or claim to reliable methodology, but they may readily pass muster under Rule 703.  Furthermore, even if Rule 703 were applied to eliminate all unreasonable reliance upon “facts or data,” the Rule would not have guarded against unreliability that crept into the opinions as a result of invalid inferences or reasoning from “facts or data,” which themselves were beyond reproach.

The Advisory Committee Note to Rule 702, from 2000, attempts to answer some of the questions about the proper scope of Rule 703:

There has been some confusion over the relationship between Rules 702 and 703. The amendment makes clear that the sufficiency of the basis of an expert’s testimony is to be decided under Rule 702. Rule 702 sets forth the overarching requirement of reliability, and an analysis of the sufficiency of the expert’s basis cannot be divorced from the ultimate reliability of the expert’s opinion. In contrast, the ”reasonable reliance” requirement of Rule 703 is a relatively narrow inquiry. When an expert relies on inadmissible information, Rule 703 requires the trial court to determine whether that information is of a type reasonably relied on by other experts in the field. If so, the expert can rely on the information in reaching an opinion. However, the question whether the expert is relying on a sufficient basis of information – whether admissible information or not – is governed by the requirements of Rule 702.

This Note leaves a large gap in the analysis of expert witness opinion evidence.  The question of the sufficiency of an expert’s bases is understandably different from whether the “facts or data” are themselves reasonably (and thus presumably also reliably) relied upon by experts in the field.  Rule 702 provides guidance about the sufficiency of “facts or data,” as well the reliable application of reliable principles and methods to the facts of the case.  Rule 702, however, is silent about the reliability of the starting point in the scientific or technical knowledge:  the data.  Perhaps the Advisory Committee meant to imply that reliable methodology requires obtaining “facts or data” in a reliable way, but it failed to address the issue in the recent amendments to Rule 702.

There is another problem that amended Rules 702 and 703, along with the Advisory Notes, fail to address.  This problem further illustrates the gap in the coverage of the rules, and perhaps it explains why courts have strained at times to include Rule 703 as part of their analysis of the reliability of expert witness opinion testimony.  Consider what happens when a proffered expert witness’s opinion has already been held to satisfy the relevance and reliability requirements of Rule 702.  The Court has explicitly ruled that the expert’s opinion has a sufficient factual basis, and that the expert has reached the opinion by reliably applying reliable methods to the facts of a case.  After the Court’s Rule 702 ruling, the expert witness amends her report to add reliance upon a new study.  The study is unfinished, and unpublished.  The paper has yet to be peer-reviewed.  Furthermore, the study is written in a foreign language, and the expert has relied upon a translation that appears to have errors, with analyses that are at least partially incoherent or incorrect.  This new study no longer raises questions about sufficiency of data, and the expert’s overall opinion, ex hypothesi, satisfies Rule 702.  This new study appears to raise fresh questions under Rule 703, not provided for in the Advisory Committee’s allocation of issues between Rules 702 and 703.[ix] Some courts might think that the addition of another study, even if the study were scientifically questionable, in support of an already 702-sufficient opinion could not be harmful error.  Yet, the additional study would give the jury the sense that the expert witness had a surfeit of support for his opinion.  Furthermore, the additional study would prejudice the adverse party by requiring more cross-examination on details that may test the patience of the factfinder.

VI.  “Facts or Data” versus “Opinions”

Rule 703 describes the condition for permitting expert witnesses to rely upon inadmissible “facts or data.”  The Rule is silent about reliance upon others’ opinions.  Of course, the distinction between facts (or data) and opinions may occasionally be blurred or difficult to discern, but the entirety of Article VII is predicated upon the existence of the distinction.[x] The conspicuous absence of “opinions” from the Rule’s conditional allowance of expert testimony based upon inadmissible “facts or data” would seem to mean that such reliance upon extra-record opinion was not authorized under Rule 703.[xi] Other courts, especially the Third Circuit, have given their blessing to the wholesale backdoor introduction of opinions, and they have not distinguished facts or data from opinions, as the potentially reasonably relied upon inadmissible evidence under Rule 703.[xii] `

The Advisory Committee Note to the 2000 amendment to Rule 702 purports to answer the question of the scope of “facts or data” under Rule 703:

The term ”data” is intended to encompass the reliable opinions of other experts. See the original Advisory Committee Note to Rule 703. The language ”facts or data” is broad enough to allow an expert to rely on hypothetical facts that are supported by the evidence.

Id.

The original Advisory Committee Note to Rule 703, however, refers to opinions as within the scope of “facts or data” in just one single passage, and in a relatively narrow context:

Thus a physician in his own practice bases his diagnosis on information from numerous sources and of considerable variety, including statements by patients and relatives, reports and opinions from nurses, technicians and other doctors, hospital records, and X rays.  Most of them are admissible in evidence, but only with the expenditure of substantial time in producing and examining various authenticating witnesses. The physician makes life-and-death decisions in reliance upon them. His validation, expertly performed and subject to cross-examination, ought to suffice for judicial purposes. Rheingold, supra, at 531; McCormick § 15. A similar provision is California Evidence Code § 801(b).[xiii]

The original Note to Rule 703 is highly misleading because opinions that are recorded in medical records would be admissible in any event as business records.  Furthermore, even if physicians must sometimes make life-or-death decisions on the basis of limited, incomplete, undocumented opinions offered by another medical care provider, that in extremis scenario is hardly a propitious basis for opinion testimony at a judicial hearing where the trier of fact is charged with making a deliberate evaluation of the evidence.  Courts and juries are charged with trying to ascertain the truth, and they do not have a warrant to abridge the fact-finding process because a physician, or any other “expert,” at time past was acting under exigent circumstances.

This more recent attempt to endorse Rule 703 as a conduit for other expert “opinions” should fail for several reasons.  First, the entire Article VII concerns itself with opinions and opinion testimony.  To suggest that Rule 703 used “facts or data” to include “opinions” ignores the context of Article VII and the limited exception that Rule 703 was making to common-law procedure.  Second, the original Advisory Committee note spoke only, in one sentence, to opinions of medical-care providers.  These opinions would normally be recorded in the patient’s medical charts and records, and they would be admissible in any event.[xiv] There is nothing in the notes to Rule 703 to support the wholesale inclusion of hearsay opinion testimony.  Third, the expansion of Rule 703 to include opinions should not circumvent the reliability requirements of Rule 702.  Fourth, the rationale of convenience used to support the expansion of the common law through Rule 703 is stood on its head by this expansion to include opinions.  The Rule puts a heavy burden to ferret out reliance upon opinions of other non-testifying experts, and to take adequate discovery of those persons or organizations. This is a heavy price to pay for the “convenience” of having an opinion introduced without the usual safeguards of critical examination of the qualifications of the expert, or the reliability of his opinion.

Until this extension of Rule 703 is checked, practitioners must inquire of their adversary’s expert witnesses, either in interrogatories or in depositions, whether the witnesses have consulted with and relied upon the writings or oral discussions with any other person regarded by the testifying expert witness as an expert.  If the testifying expert witness has relied upon these non-testifying expert’s statements or opinions, opposing counsel may have to entertain the expensive, inconvenient resort to additional discovery of the out-of-court declarant.

VII.  Fulsome Importation of Untrustworthy Opinions Through Rule 703

One prevalent and problematic practice is for expert witnesses to rely upon a study in order to pass through the study’s authors’ conclusions. Most published studies have a basic ordered structure (IMRAD) :

  • Introduction – identifies the purpose and scientific context of the study;
  • Methods – identifies the materials used, the identification, organization, collection of data and controls;
  • Results – reports the data obtained and any statistical analyses of the data; and
  • Discussion – reports the study authors’ interpretation of the results and how they fit within the larger array of data from other studies.

See Luciana B. Sollaci & Mauricio G. Pereira, “The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey,” 92 J. Med. Libr. Ass’n 364 (2004).

What becomes clear is that the testifying expert witnesses needs to have access to the methods and the results of published (and unpublished) papers in order to formulate and express their own opinions.  The introduction and discussion sections of relied upon papers are the scholarship and opinions of hearsay declarants, who in modern day publications are often quite untrustworthy.  The first and last section of most articles would rarely satisfy the procedural requirements of Federal Rule of Civil Procedure 26; nor would they satisfy the evidential reliability requirements of Rule 702. Rule 703’s limitation to “facts and data” should exclude the flood of hearsay opinion from relied upon studies by forcing expert witnesses to rely upon what is really necessary to their opinions.  If the testifying expert witness cannot testify without the scholarship and opinions of the relied upon studies, then he is probably not sufficiently expert to be giving an opinion in court.

There are many clear statements in the medical literature, which caution the consumers of medical studies against misleading claims.  Several years ago, the British Medical Journal published a paper by Montori, et al., “Users’ guide to detecting misleading claims in clinical research reports,” 329 Br. Med. J. 1093 (2004).  The authors distill their advice down to six suggestions in a “[g]uide to avoid being misled by biased presentation and interpretation of data, the first [suggestion] of which is to:  “Read only the Methods and Results sections; bypass the Discuss section.”  Id. at 1093 (emphasis added).

The federal courts have generally been oblivious to the problem of permitting fulsome presentation of hearsay opinions from the discussion and conclusion sections of articles relied upon by testifying expert witnesses.

The Supreme Court’s decision in Joiner provides a striking example.  The Court correctly assessed that plaintiffs’ expert witnesses in that case were relying upon pathologically deficient and unreliable evidence.  (Some of the expert witnesses in Joiner are known repeated offenders against Rule 702.)  In reaching the right result, and in advancing the jurisprudence of the reliability of expert witness opinion testimony, Joiner, however, stumbled in its analysis of the role of reliance upon published studies.  In his opinion in Joiner, Chief Justice Rehnquist gave considerable weight to the consideration that the plaintiffs’ expert witnesses relied upon studies, the authors of which explicitly refused to interpret as supporting a conclusion of human disease causation.  See General Electric Co. v. Joiner, 522 U.S. 136, 145-46 (1997) (noting that the PCB studies at issue did not support expert witnesses’ conclusion that PCB exposure caused cancer because the study authors, who conducted the research, were not willing to endorse a conclusion of causation).

Although the PCB study authors were well justified in their respective papers in refraining from over-interpreting their data and analyses, this consideration is of doubtful general value in evaluating the reliability of an expert witness’s proposed testimony.  First, as some plaintiffs’ counsel have argued, the testifying expert witness may be relying upon a more extensive and supportive evidentiary display than considered by the study authors.  The study, standing alone, might not support causation, but when considered with other evidence, the study could take on some importance in supporting a causal conclusion.  (This consideration would surely not save the sadly deficient opinions challenged in Joiner.) Second, as I have pointed out above, the Discussion sections of published papers of little value.  They are almost never comprehensive reviews of the subject matter, and they are often little more than the personal opinions of the study authors.  Peer reviewers may call for some acknowledgments of the weaknesses of the study, but the authors are generally allowed to press their speculations unchecked.

The use of a paper’s Discussion section to measure the reliability of a proffered expert testimony runs contrary to how scientists generally read and interpret papers.  Chief Justice Rehnquist’s emphasis upon the study authors’ Discussion of their own studies ignores the first important principal of interpreting medical studies, in an evidence-based world view:  In critically reading and evaluating a study, one should ignore anything in the paper other than the Methods and Results sections.

Joiner’s misplaced emphasis upon study authors’ Discussion sections has gained a foothold in the case law interpreting Rule 702.  In Huss v. Gayden, 571 F.3d 442  (5th Cir. 2009), for example, the Court declared:

“It is axiomatic that causation testimony is inadmissible if an expert relies upon studies or publications, the authors of which were themselves unwilling to conclude that causation had been proven.”

Id. (citing Vargas v. Lee, 317 F.3d 498, 501-01 (5th Cir. 2003) (noting that studies that did not themselves embrace causal conclusions undermined the reliability of the plaintiffs’ expert witness’s testimony that trauma caused fibromyalgia), and McClain v. Metabolife Int’l, Inc., 401 F.3d 1233, 1247-48 (11th Cir. 2005) (expert witnesses’ reliance upon studies that did not reach causal conclusions about ephedrine supported the challenge to the reliability of their proffered opinions).

The reference to what authors of relied upon papers state, in Joiner, perpetuates an authority-based view of science to the detriment of requiring good and sufficient reasons to support the testifying expert witnesses’ opinions.  The problem with Joiner’s suggestion that expert witness opinion should not be admissible if it disagrees with the study authors’ Discussion section is that sometimes study authors grossly over-interpret their data.  When it comes to scientific studies written by “political scientists” (scientists who see their work as advancing a political cause or agenda), then the Discussion section often becomes a fertile source of unreliable, speculative opinions that should not be given credence in Rule 104(a) contexts, and certainly should not be admissible in trials.

Perhaps the Discussion section, in the context of a Rule 104(a) proceeding, has some role in evaluating the challenged expert witness’s opinion, but surely it is a weak factor at best.  And clearly, the disagreement with the study authors’ conclusions or opinions, as reflected by speculative Discussion sections, can cut both ways.  Study authors may downplay their findings – appropriately or inappropriately, but study authors often overplay their findings and distort or misinterpret how their findings fit into the full picture of other studies and other evidence.  The quality of peer-reviewed publications is simply too irregular and unpredictable to make the subjective, evaluative comments in hearsay papers the touchstone for admissibility or inadmissibility.

There have been, and will continue to be, occasions in which published studies contain data, relevant and important to the causation issue, but which studies also contain speculative, personal opinions expressed in the Introduction and Discussion sections.  The parties’ expert witnesses may disagree with those opinions, but such disagreements hardly reflect poorly upon the testifying witnesses.  Neither sides’ expert witnesses should be judged by those out-of-court opinions.  Perhaps the hearsay Discussion section may be considered under Rule 104(a), which suspends the application of the Rules of Evidence, but it should hardly be an important or dispositive factor, other than raising questions for the reviewing court.

Expert witnesses should not be constrained or excluded for relying upon study data, when they disagree with the hearsay authors’ conclusions or discussions.  Given how many journals cater to advocacy scientists, and how variable the quality of peer review is, testifying expert witnesses should be required to have the expertise to interpret the data without substantial reliance upon, or reference to, the interpretative comments in the published literature.

VIII.  The Relationship Between Rules 703 and 705

Rule 705 simply provides:

The expert may testify in terms of opinion or inference and give reasons therefor without first testifying to the underlying facts or data, unless the court requires otherwise. The expert may in any event be required to disclose the underlying facts or data on cross-examination.

Rule 705, despite its brevity and apparent simplicity, encourages radical changes in the presentation of expert witness testimony in the courtroom.  Rule 705 permits expert witnesses to give opinions in the most conclusory terms.  Combined with Rule 703’s removal of admissibility as a requirement for materials “reasonably relied upon” by the expert witness, Rule 705 achieves the collapsing of difficult, technical issues into sound bites for juries and judges who increasingly suffer from inability to give sustained attention to such matters.  Under the banner of “convenience” and “economy,” these Rules operate to shift the burden to the crossexaminer to elicit the bases of an expert’s opinion as well as to then engage the expert witness on the reasonableness of his reliance, his methodology, and his application of method to the facts of the case, admissible or not.

The upshot of these changes is that the direct examination of an expert witness can often be very short, and it can be filled up with details of the expert witness’s qualifications and thinly veiled attempts to accredit the witness, even in advance of any attack on credibility.  The expert can then state his opinion as a conclusion, without any of the “messy” research facts or data, or other details.  The crossexaminer is left to dig through the bases, with judge and jury looking impatiently at the clock.  This imbalance creates practical and equitable hardships in how the Federal Rules allocate responsibility for developing factual bases for expert witness opinion between presenting and opposing counsel.  When courts impose time limits in trial of complex matters, the inequity created by the modern Rule 703 is compounded.[xv] Rule 705 gives trial courts discretion to require disclosure of bases.  In the proper case, counsel must be vigilant to motions to require this disclosure before the expert witness delivers his opinion.

Conclusion

Although Rule 703 successfully addresses some evidentiary problems in presenting expert witness opinion testimony, serious problems remain.  The Rule continues to permit expert witnesses to serve as conduits for inadmissible evidence, including opinion evidence that may escape the gatekeeping of Rule 702.  As legal scholars have pointed out, the Rule raises basic issues of fundamental fairness and constitutionality in both civil and criminal proceedings.[xvi] It is time for the Advisory Committee to go beyond restyling the Rule, and to reconsider its substance.

{This post is a revision of my article in article in 7 Proof 3 (Spring 2009).   An earlier version of that article was presented as part of an ALI-ABA Course of Study, “Opinion and Expert Testimony in Federal and State Courts,” on February 15, 2008, in San Diego, California.}


[i] Although elsewhere in the Federal Rules, the Advisory Committee disparaged limiting instructions, commentators and some courts engaged in the “judicial deception” of instructing the jury to accept the inadmissible basis as part of the explanation for the expert witness’s opinion, but not to accept or consider the basis for its truth.  See United States v. Grunewald, 233 F.2d 556, 574 (2d Cir. 1956)(“judicial deception”)(Frank, J.); Nash v. United States, 54 F.2d 1006, 1007 (2d Cir. 1932)(“mental gymnastic”)( Hand, J.).  Not only would such limiting instructions aggravate the problem by giving emphasis to the inadmissible evidence, the instructions surely would confuse most reasonable people who are trying to understand whether an expert witness has applied a reliable method to correctly ascertained “facts or data.”  The bases of an expert witness’s opinion are irrelevant if they do not have some evidential support.

[ii] Peteet v. Dow Chemical Co., 868 F.2d 1428, 1432 (8th Cir. 1989)(“[T]he trial court should defer to the expert’s opinion of what they find reasonably reliable.”); United States v. Sims, 514 F.2d 147 (9th Cir. 1975)(Rule 703 enacted, but not yet in effect)(affirming trial court’s allowing government’s psychologist to rely upon I.R.S. agent’s statement that defendant had previous “legal difficulties” to counter defendant’s claim of recent insanity against tax enforcement).

[iii] International Adhesive Coating Co. v. Bolton Emerson International, Inc., 851 F.2d 540, 544-45 (1st Cir. 1988)(equating reasonableness with “normal practice”).

[iv] United States v. Locascio, 6 F.3d 924, 938 (2d Cir. 1993).  The Third Circuit, which had adopted an extremely laissez-faire approach to expert witness testimony, signaled its compliance with the Supreme Court’s decision in Daubert, in In re Paoli Railroad Yard PCB Litigation:

We now make clear that it is the judge who makes the determination of reasonable reliance, and that for the judge to make the factual determination under Rule 104(a) that an expert is basing his or her opinion on a type of data reasonably relied upon by experts, the judge must conduct an independent evaluation into reasonableness.  The judge can of course take into account the particular expert’s opinion that experts reasonably rely on that type of data, as well as the opinions of other experts as to its reliability, but the judge can also take into account other factors he or she deems relevant.

35 F.3d 717, 748 (3d Cir. 1994)(emphasis in original).

[v] In re Agent Orange Product Liability Lit., 611 F. Supp. 1223 (E.D.N.Y. 1985), aff’d on other grounds, 818 F.2d 187 (2d Cir. 1987), cert. denied, 487 U.S. 1234 (1988).

[vi] 611 F. Supp. at 1246.  But see Fed. R. Evid. 803(4).

[vii] See, e.g., Soldo v. Sandoz Pharm. Corp., 244 F.Supp. 2d 434, 572 (W.D.Pa. 2003)(barring expert witness opinion testimony, under Rule 702 and 703).

[viii] See, e.g., Ealy v. Richardson-Merrell, Inc., 897 F.2d 1159, 1161-62 (D.C. Cir. 1990)(affirming exclusion of an expert whose opinion lacked scientific foundation, and ignored extensive contrary, published data); Lima v. United States, 708 F.2d 502, 508 (10th Cir. 1983)(affirming exclusion of epidemiologist who relied upon data not reasonably relied upon by experts in the fields of epidemiology and neurology).

[ix] See, e.g., Opinion, N.J. Super. Ct., Middlesex Cty., Docket L-5532-01-MT (denying motion to preclude expert witnesses from relying, in part, on unpublished study)(Garruto, J.).  This issue was anticipated in one of the leading cases on expert witness opinion testimony.  In re Paoli, 35 F.3d 717, 749 n. 19 (3d Cir. 1994)(pointing out that Rules 702 and 703 were not redundant, and that reliable opinions might be partially based upon unreliable data).

[x]See Beech Aircraft v. Rainey, 488 U.S. 153, 168 (1988)(“The distinction between statements of facts and opinion is, at best, one of degree.”)

[xi] American Key Corp. v. Cole Nat’l Corp., 762 F.2d 1569, 1580 (11th Cir. 1985)(“Expert opinions ordinarily cannot be based upon the opinions of others whether those opinions are in evidence or not.”); see also TK-7 Corp. v. Estate of Barbouti, 993 F.2d 722, 732 (10th Cir. 1993)(affirming exclusion of expert testimony under Rule 703 “where the expert failed to demonstrate any basis for concluding that another individual’s opinion on a subjective financial prediction was reliable, other than the fact that it was the opinion of someone he believed to be an expert who had a financial interest in making an accurate prediction”).

[xii] See, e.g., Lewis v. Rego Co.,757 F.2d 66, 73-74 (3d Cir. 1985)(holding that trial court had erred in excluding a testifying expert witness’s recounting of, and reliance upon, an out-of-court conversation with a non-testifying expert).  See also Barris v. Bob’s Drag Chutes & Safety Equipment, 685 F. 94, 102 n.10 (3d Cir. 1982)(“Under Rule 703, an expert’s testimony may be formulated by the use of the facts, data and conclusions of other experts.”); Seese v. Volkswagenwerk A.G., 648 F.2d 833, 845 (affirming admissibility, under Rule 703, of accident-reconstruction expert, whose opinion was based upon facts, data, and conclusions of a physician).

[xiii] Advisory Committee Note to Rule 703 (emphasis added).

[xiv] Fed. R. Evid. 803(6).

[xv] Evidentiary rules in state courts, even those states that have adopted Rule 703, vary considerably in how disclosure is required or allowed.  Pennsylvania, for instance, has adopted its Rule 703 verbatim from the Federal Rules, but it handles disclosure very differently under its version of Rule 705:

The expert may testify in terms of opinion or inference and give reasons therefore; however the expert must testify as to the facts or data on which the opinion or inference is based.

Pa. R. Evid. 705 (emphasis added).  See, e.g., Hansen v. Wyeth, Inc., 72 Pa. D. & C. 4th 225, 2005 WL 1114512, at *13, *19 (Phila. Ct. Common Pleas 2005)(Bernstein, J.)(granting new trial to verdict loser as result of expert witness’s failure or inability to provide all bases for his opinion).

[xvi] See, e.g., Seaman, “Triangulating Testimonial Hearsay:  The Constitutional Boundaries of Expert Opinion Testimony,” 96 Georgetown L.J. 827 (2008).