CONFOUNDING
Back in 2000, several law professors wrote an essay, in which they detailed some of the problems faced in expert witness gatekeeping. They noted that judges easily grasped the problem of generalizing from animal evidence to human experience, and thus they simplistically emphasized human (epidemiologic) data. But in their emphasis, the judges missed problems of internal validity, such as confounding, in epidemiologic studies:
“Why do courts have such a preference for human epidemiological studies over animal experiments? Probably because the problem of external validity (generalizability) is one of the most obvious aspects of research methodology, and therefore one that non-scientists (including judges) are able to discern with ease – and then give excessive weight to (because whether something generalizes or not is an empirical question; sometimes things do and other times they do not). But even very serious problems of internal validity are harder for the untrained to see and understand, so judges are slower to exclude inevitably confounded epidemiological studies (and give insufficient weight to that problem). Sophisticated students of empirical research see the varied weaknesses, want to see the varied data, and draw more nuanced conclusions.”
David Faigman, David Kaye, Michael Saks, Joseph Sanders, “How Good is Good Enough? Expert Evidence Under Daubert and Kumho,” 50 Case Western Reserve L. Rev. 645, 661 n.55 (2000). I am not sure that the problems are dependent in the fashion suggested by the authors, but their assessment that judges may be slow and frequently lack the ability to draw nuanced conclusions seems fair enough. Judges continue to miss important validity issues, perhaps because the adversarial process levels all studies to debating points in litigation. See, e.g., In re Welding Fume Prods. Liab. Litig., 2006 WL 4507859, *33 (N.D.Ohio 2006)(reducing all studies to one level, and treating all criticisms as though they rendered all studies invalid).
[This discussion of confounding has been updated; see here and there.]
