TORTINI

For your delectation and delight, desultory dicta on the law of delicts.

Archives »

Ethics and Statistics

January 21st, 2012

Chance magazine has started a new feature, the “Ethics and Statistics column, which is likely to be of interest to lawyers and to statisticians who work on litigation issues.  The column is edited by Andrew Gelman.  Judging from the Gelman’s first column, I think that the column may well become a valuable forum for important scientific and legal issues arising from studies used in public policy formulation, and in reaching conclusions that are the bases for scientific expert witnesses’ testimony in court.

Andrew Gelman is a professor of statistics and political science in Columbia University.  He is also the director of the University’s Applied Statistics Center.   Gelman’s inaugural column touches on some issues of great importance to legal counsel who litigate scientific issues involving scientific studies:  access to underlying data in the studies that are the bases for expert witness opinions.  See Andrew Gelman, “Open Data and Open Methods,” 24 Chance 51 (2011).

Gelman acknowledges that conflicts are not only driven by monetary gain; they can be potently raised by positions or causes espoused by the writer:

“An ethics problem arises when you are considering an action that

(a) benefits you or some cause you support,

(b) hurts or reduces benefits to others, and

(c) violates some rule.”

Id. at 51a.

Positional conflicts among scientists whose studies touch upon policy issues give rise to “the ethical imperative to share data.”  Id. at 51c.  Naming names, Professor Gelman relates an incident in which he wrote to an  EPA scientist, Carl Blackman, who had presented a study on the supposed health effects of EMF radiation.   Skeptical of how Blackman had analyzed data, Gelman wrote to Blackman to request his data to carry out additional, alternative statistical analyses.  Blackman answered that he did not think these other analyses were needed, and he declined to share his data.

This sort of refusal is all too common, and typical of the arrogance of scientists who do not want others to be able to take a hard look at how they arrived at their conclusions.  Gelman reminds us that:

“Refusing to share your data is improper… .”

* * * *

“[S]haring data is central to scientific ethics.  If you really believe your results, you should want your data out in the open. If, on the other hand, you have a sneaking suspicion that maybe there’s something there you don’t want to see, and then you keep your raw data hidden, it’s a problem.”

* * * *

“Especially for high-stakes policy questions (such as the risks of electric power lines), transparency is important, and we support initiatives for automatically making data public upon publication of results so researchers can share data without it being a burden.”

Id. at 53.

To be sure, there are some problems with sharing data, but none that is insuperable, and none that should be an excuse for withholding data.  The logistical, ethical, and practical problems of data sharing should now be anticipated long before publication and the requests for data sharing arrive.

Indeed, the National Institutes of Health requires data sharing plans to be part of a protocol for a federally funded study.  See Final NIH Statement on Sharing Research Data (Feb. 26, 2003). Unfortunately, the NIH’s implementation and enforcement of its data-sharing policy is as spotty as a Damien Hirst painting.  See “Seeing Spots” The New Yorker (Jan. 23, 2012).

Posted in Conflicts of Interest, Data Sharing, Rule 702, Scientific Evidence, statistical evidence | No Comments »

Defendants’ Petition for Certiorari in Milward – DENIED

January 9th, 2012

The Supreme Court reported this morning that the defendants petition for certiorari in U.S. Steel Corp. v. Milward, Docket No.. 11-316, was denied.

While unfortunate for the parties involved, the denial was not a surprise.  The Supreme Court does not sit to review factual errors and distortions, such as those that pervaded the First Circuit’s decision below.  Furthermore, most of the justices are at sea when it comes to scientific evidence, as shown by Justice Sotomayor’s incredible discussion of causal concepts, in Mattrix Initiatives v. Siracusano, ___ U.S. ___, 131 S.Ct. 1309 (2011).  See “Matrixx Unloaded.”

Indeed, there were great dangers involved in seeking this discretionary review in the Supreme Court.  As I have written, the SKAPP-a-lites have larded up the most recent edition of the Reference Manual on Scientific Evidence with language that could easily be marshaled in favor of a loosey-goosey interpretation of Rule 702.  See “Reference Manual on Scientific Evidence v3.0 – Disregarding Study Validity in Favor of the ‘Whole Gamish’.”

What is needed is not Supreme Court review, but a thorough dismemberment of the philosophy behind the Circuit’s decision in Milward, and the wayward, or the Milward, trend towards anything goes in the latest edition of the Reference Manual on Scientific Evidence.  See “Milward — Unhinging the Courthouse Door to Dubious Scientific Evidence.”

It was shame and humiliation that drove the Daubert decision in the Supreme Court, and ultimately the revision of Federal Rule of Evidence 702.   When the Courts suddenly realized that the scientific community was looking at their aberrant judgments,  they changed up.  The silicone gel breast implant litigation illustrates the phenomenon of how the courts react to the medical and scientific communties’ condemnation.

The Milward decision calls for a similar collateral attack on the unprincipled use of so-called “weight of the evidence” thinking.  Some evidence, after all, is a mere feather’s weight, and not an appropriate basis for a scientific conclusion.

FW: Defendants’ Petition for Certiorari in Milward – DENIED

Inbox
 x

Nathan A. Schachtman
11:37 AM (12 minutes ago)
to me

From: Nathan A. Schachtman [mailto:Nathan@SchachtmanLaw.com]
Sent: Monday, January 09, 2012 11:22 AM
To: ‘Nathan A. Schachtman’
Subject: Defendants’ Petition for Certiorari in Milward – DENIED

The Supreme Court reported this morning that the defendants petition for certiorari in U.S. Steel Corp. v. Milward, Docket No.. 11-316, was denied.

While unfortunate for the parties involved, the denial was not a surprise.  The Supreme Court does not sit to review factual errors and distortions, such as those that pervaded the First Circuit’s decision below.  Furthermore, most of the justices are at sea when it comes to scientific evidence, as shown by Justice Sotomayor’s incredible discussion of causal concepts, in Mattrix Initiatives v. Siracusano, ___ U.S. ___, 131 S.Ct. 1309 (2011).  See “Matrixx Unloaded.”

http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf

http://schachtmanlaw.com/matrixx-unloaded/

Indeed, there were great dangers involved in seeking this discretionary review in the Supreme Court.  As I have written, the SKAPP-a-lites have larded up the most recent edition of the Reference Manual on Scientific Evidence with language that could easily be marshaled in favor of a loosey-goosey interpretation of Rule 702.

What is needed is not Supreme Court review, but a thorough dismemberment of the philosophy behind the Circuit’s decision in Milward, and the wayward, or the Milward, trends towards anything goes in the latest edition of the Reference Manual on Scientific Evidence.  See “Milward — Unhinging the Courthouse Door to Dubious Scientific Evidence.”

http://schachtmanlaw.com/milward-unhinging-the-courthouse-door-to-dubious-scientific-evidence/

The courts need to be made to feel ashamed of their judgments with respect to scientific matters.

It was the shame and humiliation of Bendectin litigation and others that moved the Court in Daubert, and later Joiner.

Reply
Forward
Click here to Reply or Forward
Ads – Why this ad?
Bancarrota – Tarifa Plana Mas de 285 comentarios.

Posted in Rule 702, Scientific Evidence | No Comments »

The Integrity of Facts in Judicial Decisions

December 21st, 2011

One of the usual tasks of an appellate judge’s law clerk is to read the record – the entire record.  In my clerking experience, the law clerk who had the assignment for a case in which the judge was writing an opinion was responsible for knowing every detail of the record.  The judge believed that fidelity to the factual record was an absolute.

Not so for other appellate judges.  See, e.g., Jacoby, “Judicial Opinions as “Minefields of Misinformation: Antecedents, Consequences and Remedies,” University Public Law and Legal Theory Working Papers Paper 35 (N.Y. 2006).

Some important cases turn on facts misunderstood or misrepresented by appellate courts.  A few days ago, Kyle Graham blogged about a startling discovery in the Summers v. Tice case, which is covered in every first-year torts class.  Kyle Graham, “Summers v. Tice: The Rest of the Story” (Dec. 1, 2011).

Summers v. Tice, 33 Cal.2d 80, 199 P.2d 1 (1948), is a leading California tort law case that shifted the burden of proof on causation to the two defendants.  The rationale for shifting the burden was the gross negligence of both defendants, and the plaintiff’s faultless inability to identify which of the two defendants, Simonson or Tice, was responsible for shooting the plaintiff with a shotgun in their ill-fated quail hunt.

Professor Graham did something unusual:  he actually read the record of the bench trial.  It turns out that the facts were different from, and much more interesting than, those presented by the California Supreme Court.  Simonson admitted shooting Summers, and implicated Tice.  Tice denied shooting.  The trial judge resolved credibility issue against Tice, although it seems to have been a close issue.

More important, Tice testified that his gun was loaded with No. 6 shot, whereas Simonson had used No. 7.5 shot.  Summers admitted that the pellets had been given to him after his medical treatment, but he could not find them at the time of trial.  Had he kept the pellets, Summers would have been able to distinguish between the gunfeasors.

Spoliation anyone?  Missing evidence?  Adverse inference?

Even if the trial judge was unimpressed with Tice’s denial of having discharged his shotgun, Tice’s lack of credibility could not turn into affirmative evidence that he had used number 7.5 shot, as had Simonson.  This was a contested issue, on which the plaintiff could have adduced evidence.  The plaintiff’s failure to do so was the result of his own post-accident carelessness (or worse) in not keeping important evidence.  Tice’s testimony on the size of the shot in his gun was undisputed, even if the trial court thought that he was not a credible witness.

Thus, on the real facts, the shifting of the burden of proof, on the rationale that the plaintiff was without fault for his inability to produce evidence against Summers or Tice, was quite unjustified.  The plaintiff was culpable for the failure of proof, and there was no affirmative evidence that the two potential causative agents were indistinguishable. The defendants were not in a better position than the plaintiff to identify who had been the cause of plaintiff’s wounds.

The trial court’s credibility assessment of Tice, for having denied a role in shooting, did not turn the absence of evidence into affirmative evidence that both defendants used the same size pellets in their shotguns.  What makes for a great law school professor’s hypothetical was the result of an obviously fallacious inference, and a factual fabrication, borne of sloppy judicial decision making.

We can see a similar scenario play out in the New Jersey decisions that reversed directed verdicts in asbestos colorectal cancer cases.  Landrigan v. Celotex Corp., 127 NJ. 404, 605 A2d 1079 (1992); Caterinicchio v. Pittsburgh Corning Corp., 127 NJ. 428, 605 A.2d 1092 (1992). In both cases, the trial courts directed verdicts, assuming arguenda that asbestos can cause colorectal cancer (a dubious proposition), on the ground that the low relative risk cited by plaintiffs’ expert witnesses (about 1.5) was factually insufficient to support a verdict for plaintiffs on specific causation.  Indeed, the relative risk suggested that the odds were about 2 to 1 in defendants’ favor that the plaintiffs’ colorectal cancers were not caused by asbestos.

The intermediate appellate courts affirmed the directed verdicts, but the New Jersey Supreme Court reversed and remanded both judgments on curious grounds.  According to the Court, there were other probative factors that the juries could have used to make out specific causation:

“Dr. Wagoner did not rely exclusively on epidemiological studies in addressing that issue.   In addition to relying on such studies, he, like Dr. Sokolowski, reviewed specific evidence about decedent’s medical and occupational histories.   Both witnesses also excluded certain known risk factors for colon cancer, such as excessive alcohol consumption, a high-fat diet, and a positive family history.   From statistical population studies to the conclusion of causation in an individual, however, is a broad leap, particularly for a witness whose training, unlike that of a physician, is oriented toward the study of groups and not of individuals.   Nonetheless, proof of causation in toxic-tort cases depends largely on inferences derived from statistics about groups.”

Landrigan, 127 N.J. at 422.  The NJ Supreme Court held that the plaintiffs’ failure to show a relative risk in excess of 2.0 was not fatal to their cases, when there was other evidence that the jury could consider, in addition to the relative risks.

Well, actually there was no expert witness support for the assertion.  Completely absent from the evidentiary displays in both the Landrigan and Caterinicchio cases was any evidence, apart from plaintiffs’ expert witnesses’ hand waving, that a higher relative risk existed among the subcohort of asbestos insulators who had had heavier exposure or who had concomitant pulmonary disease.  There was no evidence that those exposed workers who lacked “excessive alcohol consumption, a high-fat diet, and a positive family history” had any increase risk.  Indeed, the Selikoff study relied upon extensively by plaintiffs’ expert witnesses failed to make any adjustment for the noted risk factors, as well as for the greater prevalence of smoking histories among the insulators than among the unexposed comparator population.  The Court turned the absence of evidence into the factual predicate for its holding that defendants were not entitled to judgment.

Now that’s judicial activism.

Posted in Rule 702, Scientific Evidence | No Comments »

COURTING CLIO: HISTORIANS UNDER OATH – Part 2

December 17th, 2011

Continued from Part 1:

Court-Appointed Historians

One lawyer, Jonathan Martin, trained in historical scholarship in Princeton University, has argued that historian expert witness opinion testimony is both unavoidable and refractory to the protections of judicial gatekeeping.  Martin, Historians at the Gate:  Accommodating Expert Historical Testimony in Federal Courts.” 78 N.Y.U.L. Rev. 1518 (2003).  Mr. Martin acknowledges that historians are beholding to an objective methodology, but when they are in the employ of lawyers, historians abridge or abrogate their commitment to objectivity:

Just as scientific testimony must adhere to the scientific method so too must historical testimony adhere to the historical method.  Unfortunately, historians often neglect the conventional method of their craft when offering expert testimony.  Outside the courtroom, historians generally expect one another to formulate complex, nuanced, and balanced arguments that take into account all available evidence, including any countervailing evidence.  At trial, however, the pressures of the adversary system routinely push historians toward interpretations of the past that are compressed and categorical . . .  .  As a result, historians now frequently offer unreliable evidence.

Id. at 1521.  Mr. Martin proposes to remedy the frequent, unreliable testimony from historians by the routine appointment of court-appointed expert witnesses.

In passing, Mr. Martin notes that others have urged judicial gatekeeping, under Daubert or Frye, to address unreliable historian testimony, but he rejects gatekeeping of adversarial expert witnesses as insufficient.  Id. at 1522 n.23.  Given the dearth of reported cases of such gatekeeping, this rejection seems premature.  Perhaps more important, Mr. Martin, in his rush to advocate court-appointed historians, fails to address how and why historians’ opinions are different from the opinions of experts in other fields, which are successfully subjected to cross-examination and to reliability analysis.  Historians are not alone, certainly, in succumbing to the temptation to stray from objective methodology.  Mr. Martin is correct, however, in his implicit acknowledgment that historian opinion testimony warrants increased judicial scrutiny.

One way historians differ from other fields of objective study is that historical scholarship is perfused with argument.  In biomedical and physical sciences, the presentation of research is carefully and routinely segregated into hypothesis, materials and methods, findings, and discussion.  Research findings are neatly presented without inferences to conclusions.  If conclusions can be reliably reached from the research or experiment, the investigators present their conclusions, with appropriate qualifications and caveats, in the discussion sections of their writings.  Readers understand that the discussion section is often the least important part of a published article.

Lawyering is similarly segregated into proofs and argument.  The trial lawyers’ evidence, whether real, documentary, or testimonial, is confined to a portion of the trial open for proof of facts in issue.  The trial court has the responsibility to prevent argument, argumentative questioning, and argumentative testimony in the proof-phase of the trial.  Only in closing argument, may the trial lawyers urge inferences and conclusions that assist the trier of fact to resolve the factual disputes in the case.  To be sure, trial lawyers try and sometimes succeed in advancing their argument in the proof phase of trial, either by clever juxtaposition in presenting facts, by adducing opinions in carefully defined exceptions (such as character evidence), or by successfully evading the trial court’s supervision.

Historians, in their scholarship, may acknowledge an objective method in their fact-finding, but they are under no professional constraint to separate their fact-finding and argument.  Both popular and academic historical scholarship blend fact and opinion in a manner antithetical to the sciences.  The strength and persuasiveness of historical scholarship often turns on how well the historian creates a complex narrative of fact, inference, argument, and opinion.  And the greatest art is that which conceals itself.

The pervasive role of argument is a relatively small problem compared to the dominance and legitimacy of subjective perspective in historical narrative.  Historians write from a point of view.  Openly and honestly, they narrate historical facts and events from a Marxist, labor, feminist, free-market, religious, or other point of view.  Sometimes, their point of view is covert, but it still colors the narrative.  Importantly, the point of view is often not scientific in that the scholars would likely refuse to count any empirical evidence as refuting the “truth” of their narrative.

The problems and excesses of historian opinion testimony are thus not likely to be remedied by having a court-appointed historian weigh in on the issues.  Such a court-appointed historian would present a challenge to the parties, who would need to cross examine vigorously, and to the court, which would be obligated to review and pass on the reliability of its own expert witness.  The prestige and imprimatur of court appointment would just as likely thwart as promote the truth-finding function of trial.  The argumentativeness of historical narrative would escape meaningful detection and confrontation.  Court appointments of historian witnesses might well have the effect of ending the dispute, but not in a way that advances the just resolution of the parties’ claims.

Appointment of “neutral” expert witnesses may appear to be an attractive judicial strategy to a trial court faced with party expert witnesses that are “too extreme.”  Trial judges, especially in federal Multidistrict Litigation (MDL), hear capable advocates present highly credentialed expert witnesses.  Often the opinions of the parties’ expert witnesses are diametrically opposed in ways that do not let the trial court gauge their competing claims to truth.  If trial courts find assessment of these expert witnesses’ opinions to be difficult, juries are not likely to fare better.  In perplexity, judges may try to align themselves in the middle, and comfort themselves with the belief that the trust must lie somewhere between the parties’ polar views of the world.

In the silicone gel breast implant litigation, MDL 926, Judge Sam Pointer found himself in the “middle.”  He had refused Daubert challenges to plaintiffs’ expert witnesses, and stated that the parties’ expert witnesses were too extreme.  After Judge Jack Weinstein sua sponte raised the issue of court-appointed experts in breast implant cases, plaintiffs’ counsel petitioned Judge Pointer to appoint expert witnesses in all the federal cases.  Over defendants’ objections, Judge Pointer appointed a toxicologist, a rheumatologist, an immunologist, and an epidemiologist to address the plaintiffs’ claims that silicone causes systemic autoimmune and connective tissue diseases.  After a lengthy, expensive, complex proceeding, the MDL court-appointed expert witnesses filed reports and gave testimony that rejected plaintiffs’ claims.  Much to Judge Pointer’s surprise, but not the scientific community’s, the Court’s expert witnesses opined that plaintiffs’ claims were not supported and shown by sound scientific evidence.  Subsequently, a committee of the Institute of Medicine, of the National Academy of Sciences, reached the same exculpatory conclusion.

In MDL 926, the resort to court-appointed witnesses was necessitated by that trial court’s refusal or failure to engage in meaningful gatekeeping.  Remarkably, before the MDL Court even embarked upon the expensive detour of four Rule 706 witnesses, another federal court, employing expert witness advisors, reached the same conclusion in Daubert proceedings.  Hall v. Baxter, 947 F.Supp. 1387 (D. Or. 1996).  Judge Weinstein, sitting on all federal cases in the Eastern and Southern District of New York, had already granted partial summary judgment to defendants on plaintiffs’ systemic injury claims.  In re Breast Implant Cases, 942 F. Supp. 958 (E. & S.D.N.Y. 1996).  Rule 706 was used by plaintiffs’ counsel to prolong and protract the federal proceedings, in the hope that they would be saved by research that they were sponsoring through their expert witnesses.

In looking at disputes of historical scholarship, we can easily imagine that judges will see the parties’ expert witnesses as too extreme.  The time-consuming, expensive resort to court-appointed witnesses, however, will not likely advance the resolution of issues of historical scholarship.  Unlike the selection process in MDL 926, where Judge Pointer could relatively quickly find his way to well-qualified, credible, and disinterested witnesses, the selection of an historian would stumble over the disinterestedness criterion.  Historians, by the nature of their craft, are permitted, and are encouraged, to advance a point of view that is out of place in the judicial process.

Historian Witnesses on State-of-the-Art in Tort Cases

In products liability litigation over designs or warnings, a supplier or manufacturer is typically held to the knowledge and expertise of an expert in the field.  Unfortunately, the law offers little help in answering the obvious question of which expert, of all the experts in the world, sets the appropriate standard.  In litigation over the quality of medical care, the law in many states resolves this issue by providing a defense under the “Two Schools of Thought Doctrine.”  See, e.g., “Two Schools of Thought and Informed Consent Doctrine in Pennsylvania.”  98 Dickenson L. Rev. 713 (1994).  A physician does not deviate from the standard of care simply because many or most physicians reject the approach he or she took to the patient’s problem.  As long as a substantial minority of physicians would have concurred in the judgment of the defendant physician, the claim of malpractice fails.  The Two School Doctrine has obvious implications for the standard of design or warning in products cases.

What is clear in products liability cases is that the standard of expertise must be assessed at a given time, when the product or material enters the stream of commerce.  In silicosis cases, which may involve long latency periods between exposure and manifestation of claimed disease, the parties may face historical issues of what experts knew at the legally relevant time of the sale.  Intellectual historians may indeed provide helpful insights into what was actually believed by experts in the past, but such historical data about past “beliefs” can answer the state-of-the-art inquiry only in part.  Knowledge requires at least true, justified belief.  Robert Nozick, Philosophical Explanations 167-288 (Cambridge 1981).  Hunches, suspicions, and hypotheses, even when published in respected books or journals, do not rise to the level of scientific knowledge that can be charged to the manufacturer or the supplier defendants.  Historians, unless adequately trained and expert in scientific method and research, will be inadequate to the task of explaining whether a given belief was justified and true.  Historians, motivated by politics or ideology, may try to advance their causes by trumpeting some past scientific findings, but in the last analysis, scientific theories cannot be chosen the way one chooses to be a Democrat or a Republican.  Proof of “state of the art,” or who knew what when, will require substantial expertise in science and medicine.  Historians may have to emote on the sidelines of these debates.

Posted in Historian Testimony, Rule 702 | No Comments »

COURTING CLIO: HISTORIANS UNDER OATH – Part 1

December 17th, 2011

The role of historians as expert witnesses is precarious.  They purport to marshal factual evidence from reliable sources, but that is exactly what lawyers are supposed to do.  Historians argue reasonable inferences from facts to opinions about causation, motivation, and intent.  Again, that is what lawyers are supposed to do in the argument phase of a trial.  Generally speaking, historians are superfluous in trials.

Last year, John Ulizio and I submitted an abstract, “Courting Clio,” to the International Commission on Occupational Health (ICOH), for its Fourth International Conference on the History of Occupational and Environmental Health June 19 – 22, 2010.  My presentation received mixed reviews, with the negative reaction coming from a peanut gallery of a testifying historian expert witness.  See “A Walk on the Wild Side” (July 16, 2010).  The abstract, “Courting Clio:  Historians and Their Testimony in Products Liability Action,” will appear next year in Brian Dolan and Paul Blanc, eds., At Work in the World: Proceedings of the Fourth International Conference on the History of Occupational and Environmental Health, Perspectives in Medical Humanities, University of California Medical Humanities Consortium, University of California Press (2012)(in press).

For an illustrative example of the gratuitousness of much of expert historian testimony, consider the following exchange in a deposition in a case involving claims of failure to warn, and of silicosis:

Q. Would you agree that in the 1930s, you would have to be quite a knucklehead – that’s a technical term – not to know that silica under some circumstances would cause silicosis?

A. I would agree with that characterization and the use of that technical term.

Gerald Markowitz Deposition, Mendez, at p. 115 (2005)

I believe you would have to be knucklehead to think that a jury needed Professor Markowitz to address historical knowledge about the hazards of excessive silica exposure, but plaintiffs’ lawyers persist in that practice in their silica cases.

What follows is the first part of an expanded version of the abstract presented at the ICOH.

COURTING CLIO:  HISTORIANS UNDER OATH

Introduction

Every trial involves a dispute about past facts and events.  The trier of fact must resolve the factual disputes based upon the evidence marshaled at trial.  In the common law system, the parties are primarily responsible for adducing evidence that will support their claims about historical facts.  A mature law of evidence governs admissibility of evidence, and what is reserved to counsel’s argument and the trier of fact’s reasonable inference.  The body of rules and principles that make up the law of evidence has evolved toward securing fairness, eliminating undue expense and delay, and ascertaining the truth.  Fed. R. Evid. 102.  Many would question the perfection of the current law of evidence, but few would agree to turn control over evidence and inference to historians appearing in court as expert witnesses.

Trial lawyers, using admissible evidence at trial, and fair comment in summation, attempt to persuade the triers of fact of the correctness of their version of historical facts relevant to the dispute.  In courthouses throughout the common-law world, trying cases ranging from car wrecks to complex anti-trust conspiracies, lawyers research, document, and adduce evidence of historical fact.  Judges regulate the proof of relevant historical facts by a system of rules that has been refined over centuries in the crucible of judicial experience.

At first blush, historians would appear to have little or no role in the litigation process.  Historians, in marshaling evidence and inference, are largely redundant to the lawyering process.  Indeed, reported decisions involving historians are relatively uncommon.  For the most part, historians are discussed as witnesses in only a few cases, involving such issues as land boundary disputes, navigability of rivers and riparian rights, Native American tribal status, or Nazi deportations.  See, e.g., Cayuga Indian Nation v. Pataki, 165 F.Supp. 2d 266, 300 (N.D.N.Y. 2001) (testimony on state’s purchase of tribal land); Denson v. Stack, 997 F.2d 1356, 1363-68 (11th Cir. 1993) (Clark, J., dissenting) (navigability of river in mid-19th century); Naujalis v. INS, 240 F.3d 642, 645 n.7 (7th Cir. 2001) (Nazi deportation); United States v. Dailide, 227 F.3d 385, 387 (6th Cir. 2000) (historian’s testimony in deportation proceeding).  The common themes to those cases are the arcane proofs, serious authenticity issues, and foreign language of the documentary evidence.

Historian As Advocate

There is a disturbing tendency for historians, as well as other academics, to view service as an expert witness as a way to effect social change.  Some historians have honestly acknowledged that they became lawyers manqué in their role as experts.  Professor Alfred Kelly, noted historian of the Reconstruction era, who served on the NAACP’s legal team in Brown v. Board of Education, 347 U.S. 483 (1954), strained to make out the historical case that the ratifiers of the Fourteenth Amendment had intended or anticipated to desegregate public schools.  In Richard Kluger’s history of Brown, Professor Kelly is quoted as saying that he “was caught between [his] ideals as an historian and what these people in New York [NAACP] wanted and needed.”  R. Kluger, Simple Justice:  The History of Brown v. Board of Education and Black America’s Struggle for Equality, 802 (N.Y. 1975).

Historian David Rothman recently acknowledged advocacy as the basic motivation for historians to serve as expert witnesses.  In his article, “Serving Clio and Client:  The Historian as Expert Witness,” 77 Bull. Hist. Med. 25 (2003), Rothman notes and embraces the tendentious nature of historical scholarship.  “Historians,” he tell us, “are no more or less ‘objective’ in the courtroom than they are in the lecture hall or in print.”  Id. at 44.  That assessment, alone, should give trial courts serious pause before allowing historians to testify.

In arguing why historians should serve as expert witnesses, Rothman advances the unexceptional point that litigants should have access to the expertise to have their day in court.  Id.  To the extent that historical expertise is the proper subject of opinion testimony, and the opinion is reliably based, Rothman’s point is well taken.  Emboldened, however, by the prospect of turning the witness chair into the bully pulpit, Rothman waxes expansively about the attraction of forays into the courtroom.  Historians, as expert witnesses, “may wish to bring their expertise to the support of a cause.”  Id (emphasis added).  Rothman sees historian expert witnesses as “advocates and agents of change.”  Id.

Rothman’s view of the historian as an advocate is by no means unique.  The American Historical Association (AHA), chartered by the United States Congress in 1889, has proclaimed that political views may “inform their historical practice.”  AHA, Statement on Standards of Professional Conduct, 2005 Edition, available at http://www.historians.org/pubs/Free/ProfessionalStandards.cfm (last revised January 2011).

The AHA urges hopefully that historians’ political views, when “applied with integrity,” can inform their historical researches and writing.  Id.  One of the public arenas contemplated for historical practice, by the AHA, is the courtroom, where historians will serve as expert witnesses.  Id.  In its Internet website guide to job opportunities for college graduates with degrees in history, the AHA describes “historians as advocates,” and encourages young historians that they may play an important part in the legal process by serving as expert witnesses.  AHA, http://www.historians.org/pubs/Free/careers/Index.htm (last visited on Dec.17, 2011).

Rothman’s and the AHA’s conception of the historian’s role in the trial process is inconsistent with long-established law of expert witness opinion.  Expert witnesses are simply not supposed to be advocates.  In their enthusiasm for an expansive role for historians, these historians have helped to identify much that is wrong with historian witness testimony.  Venality is a relatively weak motive, and one that is easily documented, compared with the corrupting influence of the zeal of advocacy, disguised as expert opinion.

We can find no clearer statement of judicial antipathy to expert witness advocacy than the famous copyright decision by Judge Learned Hand in Nichols v. Universal Pictures Corp., 45 F.2d 119 (2d Cir. 1930).  Both sides in Nichols presented expert testimony on “dramatic writing” in an effort to prove or disprove a claim that one screenplay infringed upon another.  Deprecating the lengthy, argumentative testimony from both sides’ experts, Judge Hand wrote that “[i]t ought not be allowed at all . . . .”  Id. at 123.  Judge Hand explained with his usual magisterial authority:  “Argument is argument whether in the box or at the bar, and its proper place is the last.”  Id.

Other areas of expertise, besides historical scholarship, fail to satisfy the basic requirements of expert witness testimony.  For instance, Judge Hand’s complaints about the “literary critic” expert witness in Nichols, have been re-lodged against witnesses with expertise in ethics.  In GST v. Telecommunications, Inc., 192 F.R.D. 109 (S.D.N.Y. 2000), both parties offered expert witness on the ethics of the conduct of corporate officers.  Invoking the requirement of helpfulness embodied in Federal Rule of Evidence 702, the Court found the proffered testimony would not aid the jury.  “It is evident that the contentious advocacy of the experts – illustrated by conclusions on the credibility of explanations regarding the business judgment of the board of directors . . . in clearly expressed, biased viewpoints – do little to aid the triers of fact on the underlying transactions.”  Id. at 110.  The trial court discerned a serious danger that expert testimony on ethics would usurp the jury’s role in applying the law to the facts found in the case.  Id.  Permitting such testimony would allow expert witnesses to attempt to substitute their judgment for the jury’s.  Id.; see, e.g., Pan American World Airways, Inc. v. Aetna Casualty & Surety, 505 F.2d 989, 998 (2d Cir. 1974) (noting disapprovingly that, evidence consisted “largely of hearsay, propaganda, speculation, and conjecture”).  See also Imwinkelried, “Expert Testimony by Ethicists:  What Should Be the Norm?” 76 Temple L. Rev. 91, 114, 128 (2003) (noting that normative testimony will virtually always be inadmissible).

Of course, the substitution of an expert witness’s judgment for that of a jury is precisely what all trial lawyers hope to accomplish.  Lawyers can select and present expert witnesses based upon their opinions and conclusion, whereas the conclusions of juries are all too unpredictable.  Trial courts must be vigilant to police expert witness opinion testimony in the area of history as much as, if not more than, in the area of scientific testimony.  Rule 702’s requirement of knowledge that will assist the trier of fact is designed to prevent expert witnesses from testifying about matters within the common understanding of the jury, and about which the jurors can reason without help from an expert witness.  In the Rezulin Multidistrict Litigation (MDL), testimony that was “a narrative of the case which a juror is equally capable of constructing” was precluded.  In re Rezulin Products Liability Litig., 309 F.Supp. 2d 531, 541 (S.D.N.Y. 2004).  Judge Kaplan explained that expert witnesses were not the appropriate conduit for a narrative of events, meetings, regulations, and documents, when they were not percipient witnesses.  Such testimony was within the jury’s lay understanding, and the law prohibited the attempt to have expert witnesses “supplant the role of counsel in making argument at trial, and the role of the jury in interpreting the evidence.”  Id. at 551.


Barring Unreliable Opinion Testimony From Historians

Stripping the argument and advocacy from historian testimony would go a long way toward remedying what is unreliable and objectionable from such opinions.  Indeed, in many cases, lawyers may find themselves with little or no reason to call on historian witnesses.

If juries should be subjected to the opinion testimony of historians, serving as expert witnesses, are there any protections against unreliable historical opinions?  We are all familiar with the extravagant claims of revisionist historians, who endeavor to reinvent the past for some political purpose.  Consider, for example, the cottage industry that decimated our forests in an effort to exculpate the late Alger Hiss.  The prospect of similar opinion testimony in the forum of tort cases is no less daunting.

The American Historical Association (A.H.A.) has recently promulgated updated standards necessary for the productions of reliable history.  Statement on Standards, supra at http://www.historians.org/pubs/Free/ProfessionalStandards.cfm (last revised January 2011).  These standards raise the important question whether courts, to the extent they permit historical testimony at all, will insist upon a showing of “reliability” before allowing widely disparate historical opinions to be presented to juries.  Surely, historical opinion that is unreliable, like unreliable scientific opinion, would not be helpful to the trier of fact.  Courts, employing various evidentiary standards, routinely exclude unhelpful testimony.  Furthermore, few lawyers would contend that they have a right, constitutional or otherwise, to present unreliable or unhelpful testimony.

The AHA’s Standards may provide a starting point for judicial consideration of the reliability of proffered historical testimony.  The Association views itself as having a special obligation to address “dilemmas and concerns about the practice of history that historians have regularly brought to the A.H.A. seeking guidance and counsel.”  Id.  The AHA Standards address “forms of professional misconduct that are especially troubling to historians,” and “identify a core set of shared values that professional historians strive to honor in the course of their work.”  Id.  The AHA thus acts as have many medical and scientific societies that have put forward guidelines and consensus statements on methodologic issues.

The AHA Standards explicitly acknowledge that historical scholarship is pursued and presented in many venues and formats, including as expert witness testimony in public arenas:

Practicing history in the public realm presents important challenges, for when historians communicate with a wide public, they must represent not just a particular interpretation or body of facts, but the best practices of the discipline of history itself.  This means they must inevitably walk a tightrope in balancing their desire to present a particular point of view with their responsibility to uphold the standards and values that underpin their professional authority as historians.  This challenge can be especially complex for . . . historians working in advocacy roles.

Id.  The Standards in many respects adopt a standard of care for historians.  Practicing history with integrity is defined in terms of both duties of aspiration and prohibition.  Among those standards relevant to a reliability assessment of historian testimony is the exhortation to acknowledge “the limits and uncertainties” of one’s arguments, as well as not to omit evidence that tends to counter or undermine one’s interpretations.  Id.  Although the Standards acknowledge the legitimacy of “point of view,” they define professional integrity in terms of, among other things, “a readiness to follow sound method and analysis wherever they may lead.”  Id.

The limits suggested by the AHA can certainly help courts evaluate historians’ testimony when carrying out their gatekeeping role for these expert witnesses.  The Standards, however, should not usurp the judicial function to define and apply criteria for the reliability and helpfulness of opinion testimony.

(to be continued)

Posted in Historian Testimony, Rule 702 | No Comments »

Federal Rules Get a Makeover

December 2nd, 2011

Bellbottoms are out; cuffs are in.  Robert Frost is out; Philip Levine is in.

So too with the Federal Rules.

The Federal Rules of Evidence have been “restyled.” Yesterday, the new, restyled Federal Rules of Evidence went into effect.

A PDF of the new rules is available at several places on the web, including the Federal Evidence Review website, which also has also links to the legislative history and guiding principles for this restyling.   The Legal Information Institute (LII) at Cornell Law School helpfully has posted ebooks, as ePub or mobi files, of the restyled Federal Rules of Civil Procedure, Criminal Procedure, and Evidence.

The legislative history of the restyled Evidence Rules 101-1103 make clear that the changes were designed to make the rules simpler, more readable and understandable, without changing their substantive meaning.  Was this effort worth the time and money?

The rules on expert witness opinion testimony are my particular interest.

Rule 703. Bases of an Expert’s Opinion Testimony

An expert may base an opinion on facts or data in the case that the expert has been made aware of or personally observed. If experts in the particular field would reasonably rely on those kinds of facts or data in forming an opinion on the subject, they need not be admissible for the opinion to be admitted. But if the facts or data would otherwise be inadmissible, the proponent of the opinion may disclose them to the jury only if their probative value in helping the jury evaluate the opinion substantially outweighs their prejudicial effect.

(Legislative History: Pub. L. 93-595, Jan. 2, 1975; Mar. 2, 1987, eff. Oct. 1, 1987; Apr. 17, 2000, eff. Dec. 1, 2000; Apr. 26, 2011, eff. Dec. 1, 2011.)

The rule specifies what happens “[i]f experts in the particular field would reasonably rely on those kinds of facts or data in forming an opinion on the subject,” but what happens “if not“?  The common reading interpolates “only” before “if,” but Rule 703 before and after restyling misses this drafting point.

So too does Rule 702:

Rule 702. Testimony by Expert Witnesses

A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if:

(a) the expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue;

(b) the testimony is based on sufficient facts or data;

(c) the testimony is the product of reliable principles and methods; and

(d) the expert has reliably applied the principles and methods to the facts of the case.

(Legislative History: Pub. L. 93-595, Jan. 2, 1975; Apr. 17, 2000, eff. Dec. 1, 2000; Apr. 26, 2011, eff. Dec. 1, 2011.)

And if not?

The enumeration of (a) through (d) in Rule 702, however, is an improvement for reading and comprehension, especially with the conjunction connecting the last member of the series.

I suppose at age 36, everyone is entitled to a makeover.

Posted in Rule 702, Rule 703, Scientific Evidence | No Comments »

The Treatment of Meta-Analysis in the Third Edition of the Reference Manual on Scientific Evidence

November 14th, 2011

Meta-analysis is a statistical procedure for aggregating data and statistics from individual studies into a single summary statistical estimate of the population measurement of interest.  The first meta-analysis is typically attributed to Karl Pearson, circa 1904, who sought a method to overcome the limitations of small sample size and low statistical power.  Statistical methods for meta-analysis, however, did not mature until the 1970s.  Even then, the biomedical scientific community remained skeptical of, if not out rightly hostile to, meta-analysis until relatively recently.

The hostility to meta-analysis, especially in the context of observational epidemiologic studies, was colorfully expressed by Samuel Shapiro and Alvan Feinstein, as late as the 1990s:

“Meta-analysis begins with scientific studies….  [D]ata from these studies are then run through computer models of bewildering complexity which produce results of implausible precision.”

* * * *

“I propose that the meta-analysis of published non-experimental data should be abandoned.”

Samuel Shapiro, “Meta-analysis/Smeta-analysis,” 140 Am. J. Epidem. 771, 777 (1994).  See also Alvan Feinstein, “Meta-Analysis: Statistical Alchemy for the 21st Century,” 48 J. Clin. Epidem. 71 (1995).

The professional skepticism about meta-analysis was reflected in some of the early judicial assessments of meta-analysis in court cases.  In the 1980s and early 1990s, some trial judges erroneously dismissed meta-analysis as a flawed statistical procedure that claimed to make something out of nothing. Allen v. Int’l Bus. Mach. Corp., No. 94-264-LON, 1997 U.S. Dist. LEXIS 8016, at *71–*74 (suggesting that meta-analysis of observational studies was controversial among epidemiologists).

In In re Paoli Railroad Yard PCB Litigation, Judge Robert Kelly excluded plaintiffs’ expert witness Dr. William Nicholson and his testimony based upon his unpublished meta-analysis of health outcomes among PCB-exposed workers.  Judge Kelly found that the meta-analysis was a novel technique, and that Nicholson’s meta-analysis was not peer reviewed.  Furthermore, the meta-analysis assessed health outcomes not experienced by any of the plaintiffs before the trial court.  706 F. Supp. 358, 373 (E.D. Pa. 1988).

The Court of Appeals for the Third Circuit reversed the exclusion of Dr. Nicholson’s testimony, and remanded for reconsideration with instructions.  In re Paoli R.R. Yard PCB Litig., 916 F.2d 829, 856-57 (3d Cir. 1990), cert. denied, 499 U.S. 961 (1991); Hines v. Consol. Rail Corp., 926 F.2d 262, 273 (3d Cir. 1991).  The Circuit noted that meta-analysis was not novel, and that the lack of peer-review was not an automatic disqualification.  Acknowledging that a meta-analysis could be performed poorly using invalid methods, the appellate court directed the trial court to evaluate the validity of Dr. Nicholson’s work on his meta-analysis.

In one of many squirmishes over colorectal cancer claims in asbestos litigation, Judge Sweet in the Southern District of New York was unimpressed by efforts to aggregate data across studies.  Judge Sweet declared that “no matter how many studies yield a positive but statistically insignificant SMR for colorectal cancer, the results remain statistically insignificant. Just as adding a series of zeros together yields yet another zero as the product, adding a series of positive but statistically insignificant SMRs together does not produce a statistically significant pattern.”  In In re Joint E. & S. Dist. Asbestos Litig., 827 F. Supp. 1014, 1042 (S.D.N.Y. 1993).  The plaintiffs’ expert witness who had offered the unreliable testimony, Dr. Steven Markowitz, like Nicholson, another foot soldier in Dr. Irving Selikoff’s litigation machine, did not offer a formal meta-analysis to justify his assessment that multiple non-significant studies, taken together, rule out chance as a likely explanation for an aggregate finding of an increased risk.

Judge Sweet was quite justified in rejecting this back of the envelope, non-quantitative meta-analysis.  His suggestion, however, that multiple non-significant studies could never collectively serve to rule out chance as an explanation for an overall increased rate of disease in the exposed groups is wrong.  Judge Sweet would have better focused on the validity issues in key studies, the presence of bias and confounding, and the completeness of the proffered meta-analysis.  The Second Circuit reversed the entry of summary judgment, and remanded the colorectal cancer claim for trial.  52 F.3d 1124 (2d Cir. 1995).  Over a decade later, with even more accumulated studies and data, the Institute of Medicine found the evidence for asbestos plaintiffs’ colorectal cancer claims to be scientifically insufficient.  Institute of Medicine, Asbestos: Selected Cancers (Wash. D.C. 2006).

Courts continue to go astray with an erroneous belief that multiple studies, all without statistically significant results, cannot yield a statistically significant summary estimate of increased risk.  See, e.g., Baker v. Chevron USA, Inc., 2010 WL 99272, *14-15 (S.D.Ohio 2010) (addressing a meta-analysis by Dr. Infante on multiple myeloma outcomes in studies of benzene-exposed workers).  There were many sound objections to Infante’s meta-analysis, but the suggestion that multiple studies without statistical significance could not yield a summary estimate of risk with statistical significance was not one of them.

In the last two decades, meta-analysis has emerged as an important technique for addressing random variation in studies, as well as some of the limitations of frequentist statistical methods.  In 1980s, articles reporting meta-analyses were rare to non-existent.  In 2009, there were over 2,300 articles with “meta-analysis” in their title, or in their keywords, indexed in the PubMed database of the National Library of Medicine.  See Michael O. Finkelstein and Bruce Levin, “Meta-Analysis of ‘Sparse’ Data: Perspectives from the Avandia Cases” (2011) (forthcoming in Jurimetrics).

The techniques for aggregating data have been studied, refined, and employed extensively in thousands of methods and application papers in the last decade. Consensus guideline papers have been published for meta-analyses of clinical trials as well as observational studies.  See Donna Stroup, et al., “Meta-analysis of Observational Studies in Epidemiology: A Proposal for Reporting,” 283 J. Am. Med. Ass’n 2008 (2000) (MOOSE statement); David Moher, Deborah Cook, Susan Eastwood, Ingram Olkin, Drummond Rennie, and Donna Stroup, “Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement,” 354 Lancet 1896 (1999).  See also Jesse Berlin & Carin Kim, “The Use of Meta-Analysis in Pharmacoepidemiology,” in Brian Strom, ed., Pharmacoepidemiology 681, 683–84 (4th ed. 2005); Zachary Gerbarg & Ralph Horwitz, “Resolving Conflicting Clinical Trials: Guidelines for Meta-Analysis,” 41 J. Clin. Epidemiol. 503 (1988).

Meta-analyses, of observational studies and of randomized clinical trials, routinely are relied upon by expert witnesses in pharmaceutical and so-called toxic tort litigation. Id. See also In re Bextra and Celebrex Marketing Sales Practices and Prod. Liab. Litig., 524 F. Supp. 2d 1166, 1174, 1184 (N.D. Cal. 2007) (holding that reliance upon “[a] meta-analysis of all available published and unpublished randomized clinical trials” was reasonable and appropriate, and criticizing the expert witnesses who urged the complete rejection of meta-analysis of observational studies)

The second edition of the Reference Manual on Scientific Evidence gave very little attention to meta-analysis.  With this historical backdrop, it is interesting to see what the new third edition provides for guidance to the federal judiciary on this important topic.

STATISTICS CHAPTER

The statistics chapter of the third edition gives continues to give scant attention to meta-analysis.  The chapter notes, in a footnote, that there are formal procedures for aggregating data across studies, and that the power of the aggregated data will exceed the power of the individual, included studies.  The footnote then cautions that meta-analytic procedures “have their own weakness,” without detailing what that one weakness is.  RMSE 3d at 254 n. 107.

The glossary at the end of the statistics chapter offers a definition of meta-analysis:

“meta-analysis. Attempts to combine information from all studies on a certain topic. For example, in the epidemiological context, a meta-analysis may attempt to provide a summary odds ratio and confidence interval for the effect of a certain exposure on a certain disease.”

Id. at 289.

This definition is inaccurate in ways that could yield serious mischief.  Virtually all meta-analyses are built upon a systematic review that sets out to collect all available studies on a research issue of interest.  It is a rare meta-analysis, however, that includes “all” studies in its quantitative analysis.  The meta-analytic process involves a pre-specification of inclusionary and exclusionary criteria for the quantitative analysis of the summary estimate of risk.  Those criteria may limit the quantitative analysis to randomized trials, or to analytical epidemiologic studies.  Furthermore, meta-analyses frequently and appropriately have pre-specified exclusionary criteria that relate to study design or quality.

On a more technical note, the offered definition suggests that the summary estimate of risk will be an odds ratio, which may or may not be true.  Meta-analyses of risk ratios may yield summary estimates of risk in terms of relative risk or hazard ratios, or even of risk differences.  The meta-analysis may combine data of means rather than proportions as well.

EPIDEMIOLOGY CHAPTER

The chapter on epidemiology delves into meta-analysis in greater detail than the statistics chapter, and offers apparently inconsistent advice.  The overall gist of the chapter, however, can perhaps best be summarized by the definition offered in this chapter’s glossary:

“meta-analysis. A technique used to combine the results of several studies to enhance the precision of the estimate of the effect size and reduce the plausibility that the association found is due to random sampling error.  Meta-analysis is best suited to pooling results from randomly controlled experimental studies, but if carefully performed, it also may be useful for observational studies.”

Reference Guide on Epidemiology, RSME3d at 624.  See also id. at 581 n. 89 (“Meta-analysis is better suited to combining results from randomly controlled experimental studies, but if carefully performed it may also be helpful for observational studies, such as those in the epidemiologic field.”).  The epidemiology chapter appropriately notes that meta-analysis can help address concerns over random error in small studies.  Id. at 579; see also id. at 607 n. 171.

Having told us that properly conducted meta-analyses of observational studies can be helpful, the chapter hedges considerably:

“Meta-analysis is most appropriate when used in pooling randomized experimental trials, because the studies included in the meta-analysis share the most significant methodological characteristics, in particular, use of randomized assignment of subjects to different exposure groups. However, often one is confronted with nonrandomized observational studies of the effects of possible toxic substances or agents. A method for summarizing such studies is greatly needed, but when meta-analysis is applied to observational studies – either case-control or cohort – it becomes more controversial.174 The reason for this is that often methodological differences among studies are much more pronounced than they are in randomized trials. Hence, the justification for pooling the results and deriving a single estimate of risk, for example, is problematic.175“

Id. at 607.  The stated objection to pooling results for observational studies is certainly correct, but many research topics have sufficient studies available to allow for appropriate selectivity in framing inclusionary and exclusionary criteria to address the objection.  The chapter goes on to credit the critics of meta-analyses of observational studies.  As they did in the second edition of the RSME, the authors repeat their cites to, and quotes from, early papers by John Bailar, who was then critical of such meta-analyses:

“Much has been written about meta-analysis recently and some experts consider the problems of meta-analysis to outweigh the benefits at the present time. For example, John Bailar has observed:

‘[P]roblems have been so frequent and so deep, and overstatements of the strength of conclusions so extreme, that one might well conclude there is something seriously and fundamentally wrong with the method. For the present . . . I still prefer the thoughtful, old-fashioned review of the literature by a knowledgeable expert who explains and defends the judgments that are presented. We have not yet reached a stage where these judgments can be passed on, even in part, to a formalized process such as meta-analysis.’

John Bailar, “Assessing Assessments,” 277 Science 528, 529 (1997).”

Id. at 607 n.177.  Bailar’s subjective preference for “old-fashioned” reviews, which often cherry picked the included studies is, well, “old fashioned.”  More to the point, it is questionable science, and a distinctly minority viewpoint in the light of substantial improvements in the conduct and reporting of meta-analyses of observational studies.  Bailar may be correct that some meta-analyses should have never left the protocol stage, but the RMSE 3d fails to provide the judiciary with the tools to appreciate the distinction between good and bad meta-analyses.

This categorical rejection, cited with apparent approval, is amplified by a recitation of some real or apparent problems with meta-analyses of observational studies.  What is missing is a discussion of how many of these problems can be and are dealt with in contemporary practice:

“A number of problems and issues arise in meta-analysis. Should only published papers be included in the meta-analysis, or should any available studies be used, even if they have not been peer reviewed? Can the results of the meta-analysis itself be reproduced by other analysts? When there are several meta-analyses of a given relationship, why do the results of different meta-analyses often disagree? The appeal of a meta-analysis is that it generates a single estimate of risk (along with an associated confidence interval), but this strength can also be a weakness, and may lead to a false sense of security regarding the certainty of the estimate. A key issue is the matter of heterogeneity of results among the studies being summarized.  If there is more variance among study results than one would expect by chance, this creates further uncertainty about the summary measure from the meta-analysis. Such differences can arise from variations in study quality, or in study populations or in study designs. Such differences in results make it harder to trust a single estimate of effect; the reasons for such differences need at least to be acknowledged and, if possible, explained.176 People often tend to have an inordinate belief in the validity of the findings when a single number is attached to them, and many of the difficulties that may arise in conducting a meta-analysis, especially of observational studies such as epidemiologic ones, may consequently be overlooked.177“

Id. at 608.  The authors are entitled to their opinion, but their discussion leaves the judiciary uninformed about current practice, and best practices, in epidemiology.  A categorical rejection of meta-analyses of observational studies is at odds with the chapter’s own claim that such meta-analyses can be helpful if properly performed.  What was needed, and is missing, is a meaningful discussion to help the judiciary determine whether a meta-analysis of observational studies was properly performed.

MEDICAL TESTIMONY CHAPTER

The chapter on medical testimony is the third pass at meta-analysis in RMSE 3d.   The second edition’s chapter on medical testimony ignored meta-analysis completely; the new edition addresses meta-analysis in the context of the hierarchy of study designs:

“Other circumstances that set the stage for an intense focus on medical evidence included

(1) the development of medical research, including randomized controlled trials and other observational study designs;

(2) the growth of diagnostic and therapeutic interventions;141

(3) interest in understanding medical decision making and how physicians reason;142 and

(4) the acceptance of meta-analysis as a method to combine data from multiple randomized trials.143“

RMSE 3d at 722-23.

The chapter curiously omits observational studies, but the footnote reference (note 143) then inconsistently discusses two meta-analyses of observational, rather than experimental, studies:

“143. … Video Software Dealers Ass’n v. Schwarzenegger, 556 F.3d 950, 963 (9th Cir. 2009) (analyzing a meta-analysis of studies on video games and adolescent behavior); Kennecott Greens Creek Min. Co. v. Mine Safety & Health Admin., 476 F.3d 946, 953 (D.C. Cir. 2007) (reviewing the Mine Safety and Health Administration’s reliance on epidemiological studies and two meta-analyses).”

Id. at 723 n.143.

The medical testimony chapter then provides further confusion by giving a more detailed listing of the hierarchy of medical evidence in the form of different study designs:

“3. Hierarchy of medical evidence

With the explosion of available medical evidence, increased emphasis has been placed on assembling, evaluating, and interpreting medical research evidence.  A fundamental principle of evidence-based medicine (see also Section IV.C.5, infra) is that the strength of medical evidence supporting a therapy or strategy is hierarchical.  When ordered from strongest to weakest, systematic review of randomized trials (meta-analysis) is at the top, followed by single randomized trials, systematic reviews of observational studies, single observational studies, physiological studies, and unsystematic clinical observations.150 An analysis of the frequency with which various study designs are cited by others provides empirical evidence supporting the influence of meta-analysis followed by randomized controlled trials in the medical evidence hierarchy.151 Although they are at the bottom of the evidence hierarchy, unsystematic clinical observations or case reports may be the first signals of adverse events or associations that are later confirmed with larger or controlled epidemiological studies (e.g., aplastic anemia caused by chloramphenicol,152 or lung cancer caused by asbestos153). Nonetheless, subsequent studies may not confirm initial reports (e.g., the putative association between coffee consumption and pancreatic cancer).154“

Id. at 723-24.  This discussion further muddies the water by using a parenthetical to suggest that meta-analyses of randomized clinical trials are equivalent to systematic reviews of such studies — “systematic review of randomized trials (meta-analysis).” Of course, systematic reviews are not meta-analyses, although they are a necessary precondition for conducting a meta-analysis.  The relationship between the procedures for a systematic review and a meta-analysis are in need of clarification, but the judiciary will not find it in the new Reference Manual.

Posted in Causation, Rule 702, Scientific Evidence, statistical evidence | 1 Comment »

New-Age Levellers – Flattening Hierarchy of Evidence

October 30th, 2011

The Levelers were political dissidents in England, in the middle of the 17th century.  Among their causes, Levelers advanced popular sovereignty, equal protection of the law, and religious tolerance.

The political agenda of the Levelers sounds quite noble to 21st century Americans, but their ideals have no place in the world of science:  not all opinions or scientific studies are created equally; not all opinions are worthy of being taken seriously in scientific discourse or in courtroom presentations of science; and not all opinions should be tolerated, especially when they claim causal conclusions based upon shoddy or inadequate evidence.

In some litigations, legal counsel set out to obscure the important quantitative and qualitative distinctions among scientific studies.  Sometimes, lawyers find cooperative expert witnesses, willing to engage in hand waving about “the weight of the evidence,” where the weights are assigned post hoc, in a highly biased fashion.  No study (that favors the claim) left behind.  This is not science, and it is not how science operates, even though some expert witnesses, such as Professor Cranor in the Milward case, have been able to pass off their views as representative of scientific practice.

A sound appreciation of how scientists evaluate studies, and of why not all studies are equal, is essential to any educated evaluation of scientific controversies.  Litigants who face high-quality studies, with results inconsistent with their litigation claims, may well resort to “leveling” of studies.  This leveling may be advanced out of ignorance, but more likely the leveling is an attempt to snooker courts with evidence from exploratory, preliminary, and hypothesis-generating studies as somehow equal to, or greater than, the value of hypothesis-testing studies.

Some of the leveling tactics that have become commonplace in litigation include asserting that:

  • All experts witnesses are the same;
  • All expert witnesses conduct the same analysis;
  • All expert witnesses read articles, interpret them, and offer opinions;
  • All expert witnesses are inherently biased;
  • All expert witnesses select the articles to read and interpret in line with their biases;
  • All epidemiologic studies are the same;
  • All studies are flawed; and
  • All opinions are, in the final analysis, subjective.

This leveling strategy can be seen in Professor Margaret Berger’s introduction to the Reference Manual on Scientific Evidence (RMSE 3d), where she supported an ill-defined “weight-of-the-evidence” approach to causal judgments. See “Late Professor Berger’s Introduction to the Reference Manual on Scientific Evidence” (Oct. 23, 2011).

Other chapters in the RMSE 3d are at odds with Berger’s introduction.  The epidemiology chapter does not explicitly address the hierarchy of studies, but it does describe cross-sectional, ecological, and secular trend studies are less able to support causal conclusions.  Cross-sectional studies are described as “rarely useful in identifying toxic agents,” RMSE 3d at 556, and as “used infrequently when the exposure of interest is an environmental toxic agent,” RMSE 3d at 561.  Cross-sectional studies are described as hypothesis-generating as opposed to hypothesis testing, although not in those specific terms.  Id. (describing cross-sectional studies as providing valuable leads for future research).  Ecological studies are described as useful for identifying associations, but not helpful in determining whether such associations are causal; and ecological studies are identified as a fertile source of error in the form of the “ecological fallacy.”  Id. at 561 -62.

The epidemiology chapter perhaps weakens its helpful description of the limited role of ecological studies by citing, with apparent approval, a district court that blinked at its gatekeeping responsibility to ensure that testifying expert witnesses did, in fact, rely upon “sufficient facts or data,” as well as upon studies that are “of a type reasonably relied upon by experts in the particular field in forming opinions or inferences upon the subject.” Rule 703. RMSE 3d at 561 n.34 (citing Cook v. Rockwell International Corp., 580 F. Supp. 2d 1071, 1095–96 (D. Colo. 2006), where the district court acknowledged the severe limitations of ecological studies in supporting causal inferences, but opined that the limitations went to the weight of the study). Of course, the insubstantial weight of an ecological study is precisely what may result in the study’s failure to support a causal claim.

The ray of clarity in the epidemiology chapter about the hierarchical nature of studies is muddled by an attempt to level epidemiology and toxicology.  The chapter suggests that there is no hierarchy of disciplines (as opposed to studies within a discipline).  RMSE 3d at 564 & n.48 (citing and quoting symposium paper that “[t]here should be no hierarchy [among different types of scientific methods to determine cancer causation]. Epidemiology, animal, tissue culture and molecular pathology should be seen as integrating evidences in the determination of human carcinogenicity.” Michele Carbone et al., “Modern Criteria to Establish Human Cancer Etiology,” 64 Cancer Res. 5518, 5522 (2004).)  Carbone, of course, is best known for his advocacy of a viral cause (SV40), of human mesothelioma, a claim unsupported, and indeed contradicted, by epidemiologic studies.  His statement does not support the chapter’s leveling of epidemiology and toxicology, and Carbone is, in any event, an unlikely source to cite.

The epidemiology chapter undermines its own description of the role of study design in evaluating causality by pejoratively asserting that most epidemiologic studies are “flawed”:

“It is important to emphasize that all studies have ‘flaws’ in the sense of limitations that add uncertainty about the proper interpretation of the results.9 Some flaws are inevitable given the limits of technology, resources, the ability and willingness of persons to participate in a study, and ethical constraints. In evaluating epidemiologic evidence, the key questions, then, are the extent to which a study’s limitations compromise its findings and permit inferences about causation.”

RSME 3d at 553.  This statement is actually a significant improvement over the second edition, where the authors of the epidemiology chapter asserted, without qualification:

“It is important to emphasize that most studies have flaws.”

RMSE 2d 337.  The “flaws” language from the earlier chapter was used on occasion by courts that were set on ignoring competing interpretations of epidemiologic studies.  Since all or most studies are flawed, why bother figuring out what is valid and reliable?  Just let the jury sort it out.  This is not an aid to gatekeeping, but rather a prescription for allowing the gatekeeper to call in sick.

The current epidemiology chapter essentially backtracks from the harsh connotations of its use of the term “flaws,” by now equating the term with “limitations.”  Flaws and limitations, however, are quite different from one another.  What is left out in the third edition’s description is the sense that there are indeed some studies that are so flawed that they must be disregarded altogether.  There may also be limitations in studies, especially observational studies, which is why the party with the burden of proof should generally not be allowed to proceed with only one or two epidemiologic studies.  Rule 702, after all, requires that an expert opinion to be based upon “sufficient facts or data.”

The RSME 3d chapter on medical evidence is a refreshing break from the leveling approach seen elsewhere.  Here at least, the chapter authors devote several pages to explaining the role of study design in assessing an etiological issue:

“3. Hierarchy of medical evidence

With the explosion of available medical evidence, increased emphasis has been placed on assembling, evaluating, and interpreting medical research evidence.  A fundamental principle of evidence-based medicine (see also Section IV.C.5, infra) is that the strength of medical evidence supporting a therapy or strategy is hierarchical.

When ordered from strongest to weakest, systematic review of randomized trials (meta-analysis) is at the top, followed by single randomized trials, systematic reviews of observational studies, single observational studies, physiological studies, and unsystematic clinical observations.150 An analysis of the frequency with which various study designs are cited by others provides empirical evidence supporting the influence of meta-analysis followed by randomized controlled trials in the medical evidence hierarchy.151 Although they are at the bottom of the evidence hierarchy, unsystematic clinical observations or case reports may be the first signals of adverse events or associations that are later confirmed with larger or controlled epidemiological studies (e.g., aplastic anemia caused by chloramphenicol,152 or lung cancer caused by asbestos153). Nonetheless, subsequent studies may not confirm initial reports (e.g., the putative association between coffee consumption and pancreatic cancer).154“

John B. Wong, Lawrence O. Gostin, and Oscar A. Cabrera, “Reference Guide on Medical Testimony,” RMSE 3d 687, 723 -24 (2011).  The third edition’s chapter is a significant improvement of the second edition’s chapter on medical testimony, which does not mention the hierarchy of evidence.  Mary Sue Henifin, Howard M. Kipen, and Susan R. Poulter, ” Reference Guide on Medical Testimony,” RMSE 2d 440 (2000).  Indeed, the only time the word “hierarchy” appeared in the entire second edition was in connection with the hierarchy of the federal judiciary.

The tension, contradictions, and differing emphases among the various chapters of the RSME 3d point to an important “flaw” in the new edition.  The chapters appear to have been written largely in isolation, and without much regard for what the other chapters contain.  The chapters overlap, and indeed contradict one another on key points.  Witness Berger’s rejection of the hierarchy of evidence, the epidemiology chapter’s inconstant presentation of the concept without mentioning it by name, and the medical testimony chapter’s embrace and explicit presentation of the hierarchical nature of medical study evidence.  Fortunately, the laissez-faire editorial approach allowed the disagreement to remain, without censoring any position, but the federal judiciary is not aided by the contradiction and tension in the approaches.

Given the importance of the concept, even the medical testimony chapter in RSME 3d may seem to be too little, too late to be helpful to the judiciary.  There are book-length treatments of systematic reviews and “evidence-based medicine”: the three pages in Wong’s chapter barely scratch the surface of this important topic of how evidence is categorized, evaluated, and synthesized in making judgments of causality.

There are many textbooks and articles available to judges and lawyers on how to assess medical studies.  Recently, John Cherrie has posted on his blog, OH-world, about a series of 17 articles, in the journal Aerzteblatt International, on the proper evaluation of medical and epidemiologic studies.

These papers, overall, make the point that not all studies are equal, and that not all evidentiary displays are adequate to support conclusions of causal association.  The papers are available without charge from the journal’s website:

01. Critical Appraisal of Scientific Articles

02. Study Design in Medical Research

03. Types of Study in Medical Research

04. Confidence Interval or P-Value?

05. Requirements and Assessment of Laboratory Tests: Inpatient Admission Screening

06. Systematic Literature Reviews and Meta-Analyses

07. The Specification of Statistical Measures and Their Presentation in Tables and Graphs

08. Avoiding Bias in Observational Studies

09. Interpreting Results in 2×2 Tables

10. Judging a Plethora of p-Values: How to Contend With the Problem of Multiple Testing

11. Data Analysis of Epidemiological Studies

12. Choosing statistical tests

13. Sample size calculation in clinical trials

14. Linear regression analysis

15. Survival analysis

16. Concordance analysis

17. Randomized controlled trials

This year, the Journal of Clinical Epidemiology began publishing a series of papers, known by the acronym GRADE, which aim to provide guidance on how studies are categorized and assessed for their evidential quality in supporting treatments and intervention.  The GRADE project is led by Gordon Guyatt, who is known for having coined the term “evidence-based medicine,” and written widely on the subject.  Guyatt, along with his colleagues including Peter Tugwell (who was one of the court-appointed expert witnesses in MDL 926), has described the GRADE project:

“The ‘Grades of Recommendation, Assessment, Development, and Evaluation’ (GRADE) approach provides guidance for rating quality of evidence and grading strength of recommendations in health care. It has important implications for those summarizing evidence for systematic reviews, health technology assessment, and clinical practice guidelines. GRADE provides a systematic and transparent framework for clarifying questions, determining the outcomes of interest, summarizing the evidence that addresses a question, and moving from the evidence to a recommendation or decision. Wide dissemination and use of the GRADE approach, with endorsement from more than 50 organizations worldwide, many highly influential   http://www.gradeworkinggroup.org/), attests to the importance of this work. This article introduces a 20-part series providing guidance for the use of GRADE methodology that will appear in the Journal of Clinical Epidemiology.”

Gordon Guyatt, Andrew D. Oxman, Holger Schünemann, Peter Tugwell, Andre Knottnerus, “GRADE guidelines – new series of articles in Journal of Clinical Epidemiology,” 64 J. Clin. Epidem. 380 (2011).  See also Gordon Guyatt, Andrew Oxman, et al., for the GRADE Working Group, “Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations,” 336 Brit. Med. J. 924 (2008).  [pdf]

Of the 20 papers planned, 9 of the GRADE papers have been published to date in the Journal of Clinical Epidemiology:

01 Intro – GRADE evidence profiles & summary of findings tables

02 Framing question & deciding on important outcomes

03 Rating quality of evidence

04 Rating quality of evidence – study limitations (risk of bias)

05 Rating the quality of evidence—publication bias

06 Rating up quality of evidence – imprecision

07 Rating quality of evidence – inconsistency

08 Rating quality of evidence – indirectness

09 Rating up quality of evidence

The GRADE guidance papers focus on the efficacy of treatments and interventions, but in doing so, they evaluate “effects” and are thus applicable to the etiologic issues of alleged harm that find their way into court.  The papers build on other grading systems advanced previously by the Oxford Center for Evidence-Based Medicine, the U.S. Preventive Services Task Force (Agency for Healthcare Research and Quality AHRQ), the Cochrane Collaboration, as well as many individual professional organizations.

GRADE has had some success in harmonizing disparate grading systems, and forging a consensus among organizations that had been using their own systems, such as the  World Health Organization, the American College of Physicians, the American Thoracic Society, the Cochrane Collaboration, the American College of Chest Physicians, the British Medical Journal, and Kaiser Permanente.

There are many other important efforts to provide consensus support for improving the quality of the design, conduct, and reporting of published studies, as well as the interpretation of those studies once published.  Although the RSME 3d does a good job of introducing its readers to the basics of study design, it could have done considerably more to help judges become discerning critics of scientific studies and of conclusions based upon individual or multiple studies.

Posted in Rule 702, Scientific Evidence | No Comments »

Historians As Expert Witnesses – A Wiki

October 28th, 2011

“The one duty we owe to history is to rewrite it.”

Oscar Wilde, The Critic As Artist (1891)

“What will history say?  History, sir, will tell lies as usual.”

George Bernard Shaw, The Devil’s Disciple (1901)

* * * * * * * * * * * * * * * * * * * * * * * * *

The Defense Research Institute recently announced that Bill Childs, a professor at the Western New England University School of Law, will be speaking the use of historians as expert witnesses in litigation.  Having puzzled about this very issue in previous writings, I look forward to Professor Childs’ contributions on the issue.  The announcement also noted Professor Childs’ creation, “the Historians as Experts Wiki,” which I knew about, but had not previously visited.

The wiki is a valuable resource of information about historians who have participated in the litigation process in all manner of cases, including art, asbestos, creationism, native Americans, holocaust, products liability, intellectual property, and voting rights.  There are pages for each historian witness, including expert witnesses in other fields, who have given testimony of an explicitly historical nature. The website is still in its formative stages, but it holds great promise as a resource to lawyers who are researching historians who have been listed as expert witnesses in their cases.

Most of my musings about historians as expert witnesses have been provoked by those who have testified about the history of silicosis.  Last year, I presented at a conference sponsored by the International Commission on Occupational Health (ICOH), about such historians.  See “A Walk on the Wild Side,” July 16, 2010.  My presentation abstract, along with all the proceedings of that conference, will be published next year as  “Courting Clio:  Historians and Their Testimony in Products Liability Action,” in: Brian Dolan and Paul Blanc, eds., At Work in the World: Proceedings of the Fourth International Conference on the History of Occupational and Environmental Health, Perspectives in Medical Humanities, University of California Medical Humanities Consortium, University of California Press (2012)(in press).

Posted in Historian Testimony, Rule 702 | No Comments »

New Reference Manual on Scientific Evidence Short Shrifts Rule 703

October 16th, 2011

In “RULE OF EVIDENCE 703 — Problem Child of Article VII (Sept. 19, 2011),” I wrote about how Federal Rule of Evidence 703 is generally ignored and misunderstood in current federal practice.  The Supreme Court, in deciding Daubert, shifted the focus to Rule 702, as the primary tool to deploy in admitting, as well as limiting and excluding, expert witness opinion testimony.  The Court’s decision, however, did not erase the need for an additional, independent rule to control the quality of inadmissible materials upon which expert witnesses rely.  Indeed, Rule 702 as amended in 2000, incorporated much of the learning of the Daubert decision, and then some, but it does not address the starting place of any scientific opinion:  the data, the analyses (usually statistical) of data, and the reasonableness of relying upon those data and analyses.  Instead, Rule 702 asks whether the proffered testimony is based upon:

  1. sufficient facts or data,
  2. the product of reliable principles and methods, and
  3. a reliable application of principles and methods to the facts of the case

Noticeably absent from Rule 702, in its current form, is any directive to determine whether the proffered expert witness opinion is based upon facts or data of the sort upon which experts in the pertinent field would reasonably rely.  Furthermore,  Daubert did not address the fulsome importation and disclosure of untrustworthy hearsay opinions through Rule 703.  See Problem Child (discussing the courts’ failure to appreciate the structure of peer-reviewed articles, and the need to ignore the discussion and introduction sections of such articles as often containing speculative opinions and comments).  See also Luciana B. Sollaci & Mauricio G. Pereira, “The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey,” 92 J. Med. Libr. Ass’n 364 (2004); Montori, et al., “Users’ guide to detecting misleading claims in clinical research reports,” 329 Br. Med. J. 1093, 1093 (2004) (advising readers on how to avoid being misled by published literature, and counseling readers to “Read only the Methods and Results sections; bypass the Discuss section.”)  (emphasis added).

Given this background, it is disappointing but not surprising that the new Reference Manual on Scientific Evidence severely slights Rule 703.  Using either a word search in the PDF version or the index at end of book tells the story:  There are five references to Rule 703 in the entire RMSE!  The statistics chapter has an appropriate but fleeting reference:

“Or the study might rest on data of the type not reasonably relied on by statisticians or substantive experts and hence run afoul of Federal Rule of Evidence 703. Often, however, the battle over statistical evidence concerns weight or sufficiency rather than admissibility.”

RMSE 3d at 214. At least this chapter acknowledges, however briefly, the potential problem that Rule 703 poses for expert witnesses.  The chapter on survey research similarly discusses how the data collected in a survey may “run afoul” of Rule 703.  RMSE 3d at 361, 363-364.

The chapter on epidemiology takes a different approach by interpreting Rule 703 as a rule of admissibility of evidence:

“An epidemiologic study that is sufficiently rigorous to justify a conclusion that it is scientifically valid should be admissible,184 as it tends to make an issue in dispute more or less likely.185“

Id. at 610.  This view is mistaken.  Sufficient rigor in an epidemiologic study is certainly needed for reliance by an expert witness, but such rigor does not make the study itself admissible; the rigor simply permits the expert witness to rely upon a study that is typically several layers of inadmissible hearsay.  See “Reference Manual on Scientific Evidence v3.0 – Disregarding Study Validity in Favor of the “Whole Gamish” (Oct. 14, 2011) (discussing the argument put forward by the epidemiology chapter for considering Rule 703 as an exception to the rule against hearsay).

While the treatment of Rule 703 in the epidemiology chapter is troubling, the introductory chapter on the admissibility of expert witness opinion testimony by the late Professor Margaret Berger really sets the tone and approach for the entire volume. See Berger, “The Admissibility of Expert Testimony,” RSME 3d 11 (2011).  Professor Berger never mentions Rule 703 at all!  Gone and forgotten. The omission is not, however, an oversight.  Rule 703, with its requirement of qualifying each study relied upon as having been “reasonably relied upon,” as measured by what experts in the appropriate discipline, is the refutation of Berger’s argument that somehow a pile of weak, flawed studies, taken together can yield a scientifically reliable conclusion. See “Whole Gamish,” (Oct. 14th, 2011).

Rule 703 is not merely an invitation to trial judges; it is a requirement to look at the discrete studies relied upon to determine whether the building blocks are sound.  Only then can the methods and procedures of science begin to analyze the entire evidentiary display to yield reliable scientific opinions and conclusions.

Posted in Rule 702, Rule 703, Scientific Evidence, Underlying Data | No Comments »

« Older Entries
Newer Entries »